1 2018-09-01T00:00:50  *** Chris_Stewart_5 has quit IRC
  2 2018-09-01T00:27:21  *** sipa has joined #bitcoin-core-dev
  3 2018-09-01T00:30:36  *** IGHOR has quit IRC
  4 2018-09-01T00:31:06  <jimpo> gmaxwell: By "like the undo data", do you mean just that there are flat file storing the large values and disk positions stored in LevelDB or are you suggesting specifically that the filters be computed in validation code and referenced by the block index?
  5 2018-09-01T00:32:59  *** IGHOR has joined #bitcoin-core-dev
  6 2018-09-01T00:33:19  <jimpo> in #14121
  7 2018-09-01T00:33:21  <gribble> https://github.com/bitcoin/bitcoin/issues/14121 | Index for BIP 157 block filters by jimpo · Pull Request #14121 · bitcoin/bitcoin · GitHub
  8 2018-09-01T00:43:36  *** ghost43 has quit IRC
  9 2018-09-01T00:44:17  *** ghost43 has joined #bitcoin-core-dev
 10 2018-09-01T00:45:54  *** a5m0 has quit IRC
 11 2018-09-01T00:55:57  *** a5m0 has joined #bitcoin-core-dev
 12 2018-09-01T01:04:02  *** d9b4bef9 has quit IRC
 13 2018-09-01T01:08:23  *** Guest79908 has joined #bitcoin-core-dev
 14 2018-09-01T01:10:07  *** d9b4bef9 has joined #bitcoin-core-dev
 15 2018-09-01T01:13:33  <gmaxwell> jimpo: the former.
 16 2018-09-01T01:13:45  <gmaxwell> Flat files with the filters, indexed by position.
 17 2018-09-01T01:33:45  <jimpo> Yeah, that makes sense to me. I'll code that up and compare read/write perf.
 18 2018-09-01T01:46:25  <sipa> i doubt it matters much here; we're not throughput limited
 19 2018-09-01T01:46:47  <sipa> leveldb writes all data twice, which is a reason against writing huge things like blocks and undo data
 20 2018-09-01T01:49:06  <gmaxwell> also has varrious caching behaviors, also stores it somewhat inefficiently.  I think that for recent blocks the filters are about 30KB per block? in any case if you think its okay it probably is.
 21 2018-09-01T01:49:15  *** drexl has quit IRC
 22 2018-09-01T01:49:35  <gmaxwell> I was also thinking about future dependency on leveldb... since we got non-atomic flushing, there are many other things possible for the chainstate.
 23 2018-09-01T01:52:43  *** peevsie has joined #bitcoin-core-dev
 24 2018-09-01T01:54:09  <echeveria> mongodb?
 25 2018-09-01T01:56:32  <gmaxwell> it has webscale
 26 2018-09-01T01:57:19  <gmaxwell> I don't think that even with non-atomic-flushing would mongo's consistency behavior be acceptable. :P
 27 2018-09-01T01:58:39  *** Chris_Stewart_5 has joined #bitcoin-core-dev
 28 2018-09-01T02:24:36  *** justanotheruser has quit IRC
 29 2018-09-01T02:26:38  *** justanotheruser has joined #bitcoin-core-dev
 30 2018-09-01T02:28:22  *** plankers has joined #bitcoin-core-dev
 31 2018-09-01T02:32:54  <jimpo> mongo write consistency could be a decent entropy source
 32 2018-09-01T03:18:45  *** unholymachine has quit IRC
 33 2018-09-01T03:33:14  *** kinlo has quit IRC
 34 2018-09-01T03:39:23  *** kinlo has joined #bitcoin-core-dev
 35 2018-09-01T03:46:12  *** Krellan has quit IRC
 36 2018-09-01T04:23:56  *** peevsie has quit IRC
 37 2018-09-01T04:25:27  *** sipa has quit IRC
 38 2018-09-01T04:29:52  *** plankers has quit IRC
 39 2018-09-01T05:06:47  *** Chris_Stewart_5 has quit IRC
 40 2018-09-01T05:11:01  *** d9b4bef9 has quit IRC
 41 2018-09-01T05:14:07  *** d9b4bef9 has joined #bitcoin-core-dev
 42 2018-09-01T05:27:15  *** Krellan has joined #bitcoin-core-dev
 43 2018-09-01T05:46:08  *** sipa has joined #bitcoin-core-dev
 44 2018-09-01T06:09:37  *** sipa has quit IRC
 45 2018-09-01T06:22:54  *** luke-jr has quit IRC
 46 2018-09-01T06:23:15  *** luke-jr has joined #bitcoin-core-dev
 47 2018-09-01T06:25:08  *** luke-jr has quit IRC
 48 2018-09-01T06:25:27  *** luke-jr has joined #bitcoin-core-dev
 49 2018-09-01T06:41:15  *** luke-jr has quit IRC
 50 2018-09-01T06:41:43  *** luke-jr has joined #bitcoin-core-dev
 51 2018-09-01T06:45:49  *** luke-jr has quit IRC
 52 2018-09-01T06:46:03  *** luke-jr has joined #bitcoin-core-dev
 53 2018-09-01T06:47:30  *** luke-jr has quit IRC
 54 2018-09-01T06:48:28  *** luke-jr has joined #bitcoin-core-dev
 55 2018-09-01T06:50:07  *** luke-jr has quit IRC
 56 2018-09-01T06:50:23  *** luke-jr has joined #bitcoin-core-dev
 57 2018-09-01T06:54:22  *** luke-jr has quit IRC
 58 2018-09-01T06:54:40  *** luke-jr has joined #bitcoin-core-dev
 59 2018-09-01T07:02:27  *** SopaXorzTaker has joined #bitcoin-core-dev
 60 2018-09-01T07:07:24  *** sipa has joined #bitcoin-core-dev
 61 2018-09-01T07:18:47  *** Victorsueca has quit IRC
 62 2018-09-01T07:20:02  *** Victorsueca has joined #bitcoin-core-dev
 63 2018-09-01T07:34:29  *** Krellan has quit IRC
 64 2018-09-01T07:35:21  *** Krellan has joined #bitcoin-core-dev
 65 2018-09-01T07:58:29  *** promag has joined #bitcoin-core-dev
 66 2018-09-01T08:08:15  *** luke-jr has quit IRC
 67 2018-09-01T08:08:32  *** luke-jr has joined #bitcoin-core-dev
 68 2018-09-01T08:18:42  *** luke-jr has quit IRC
 69 2018-09-01T08:19:05  *** luke-jr has joined #bitcoin-core-dev
 70 2018-09-01T08:44:52  *** lukedashjr has joined #bitcoin-core-dev
 71 2018-09-01T08:46:17  *** SopaXorzTaker has quit IRC
 72 2018-09-01T08:47:18  *** luke-jr has quit IRC
 73 2018-09-01T08:49:19  *** lukedashjr is now known as luke-jr
 74 2018-09-01T08:52:01  *** d9b4bef9 has quit IRC
 75 2018-09-01T08:53:04  *** profmac has quit IRC
 76 2018-09-01T08:53:08  *** d9b4bef9 has joined #bitcoin-core-dev
 77 2018-09-01T08:55:23  *** promag has quit IRC
 78 2018-09-01T09:14:42  *** Victorsueca has quit IRC
 79 2018-09-01T09:15:57  *** Victorsueca has joined #bitcoin-core-dev
 80 2018-09-01T09:55:20  *** profmac has joined #bitcoin-core-dev
 81 2018-09-01T10:04:20  *** savil has quit IRC
 82 2018-09-01T10:05:19  *** promag has joined #bitcoin-core-dev
 83 2018-09-01T10:09:08  *** savil has joined #bitcoin-core-dev
 84 2018-09-01T10:14:40  *** promag has quit IRC
 85 2018-09-01T10:15:51  *** profmac has quit IRC
 86 2018-09-01T10:33:22  *** profmac has joined #bitcoin-core-dev
 87 2018-09-01T10:49:24  *** lukedashjr has joined #bitcoin-core-dev
 88 2018-09-01T10:50:28  *** luke-jr has quit IRC
 89 2018-09-01T10:50:28  *** Krellan has quit IRC
 90 2018-09-01T10:53:51  *** lukedashjr is now known as luke-jr
 91 2018-09-01T10:56:48  *** hebasto has joined #bitcoin-core-dev
 92 2018-09-01T11:05:28  *** lnostdal has quit IRC
 93 2018-09-01T11:17:19  *** lnostdal has joined #bitcoin-core-dev
 94 2018-09-01T11:24:09  *** lnostdal has quit IRC
 95 2018-09-01T11:36:30  *** lnostdal has joined #bitcoin-core-dev
 96 2018-09-01T11:41:12  *** lnostdal has quit IRC
 97 2018-09-01T11:41:44  *** lnostdal has joined #bitcoin-core-dev
 98 2018-09-01T11:47:25  *** Guyver2 has joined #bitcoin-core-dev
 99 2018-09-01T11:49:27  *** lnostdal has quit IRC
100 2018-09-01T11:53:50  *** opdenkamp has quit IRC
101 2018-09-01T12:00:56  *** lnostdal has joined #bitcoin-core-dev
102 2018-09-01T12:12:29  *** itaseski has joined #bitcoin-core-dev
103 2018-09-01T12:16:05  *** owowo has joined #bitcoin-core-dev
104 2018-09-01T12:39:06  *** hebasto has quit IRC
105 2018-09-01T13:14:37  *** opdenkamp has joined #bitcoin-core-dev
106 2018-09-01T13:23:11  *** Chris_Stewart_5 has joined #bitcoin-core-dev
107 2018-09-01T13:52:33  *** lnostdal has quit IRC
108 2018-09-01T13:58:33  *** g687 has joined #bitcoin-core-dev
109 2018-09-01T14:05:31  *** lnostdal has joined #bitcoin-core-dev
110 2018-09-01T14:09:13  *** Chris_Stewart_5 has quit IRC
111 2018-09-01T14:16:21  *** Chris_Stewart_5 has joined #bitcoin-core-dev
112 2018-09-01T14:22:53  *** ken2812221 has quit IRC
113 2018-09-01T14:57:24  *** g687 has quit IRC
114 2018-09-01T15:04:51  *** ken2812221 has joined #bitcoin-core-dev
115 2018-09-01T15:35:36  *** opdenkamp has quit IRC
116 2018-09-01T15:40:00  *** opdenkamp has joined #bitcoin-core-dev
117 2018-09-01T15:40:28  *** jarthur has joined #bitcoin-core-dev
118 2018-09-01T16:01:11  *** Krellan has joined #bitcoin-core-dev
119 2018-09-01T16:21:11  *** jarthur has quit IRC
120 2018-09-01T16:43:48  *** owowo has quit IRC
121 2018-09-01T16:45:09  *** Victorsueca has quit IRC
122 2018-09-01T16:46:27  *** Victorsueca has joined #bitcoin-core-dev
123 2018-09-01T16:48:32  *** jhfrontz has joined #bitcoin-core-dev
124 2018-09-01T16:48:45  *** owowo has joined #bitcoin-core-dev
125 2018-09-01T17:02:04  <jimpo> hmm, seems the whole block tree db could be moved to flat files since it's all read into memory on startup anyway
126 2018-09-01T17:43:42  *** g687 has joined #bitcoin-core-dev
127 2018-09-01T17:56:31  *** Jbaczuk has quit IRC
128 2018-09-01T17:56:31  *** stevenroose has quit IRC
129 2018-09-01T17:56:31  *** cncr04s has quit IRC
130 2018-09-01T17:56:32  *** exit70 has quit IRC
131 2018-09-01T17:56:32  *** bosma has quit IRC
132 2018-09-01T17:56:32  *** epic has quit IRC
133 2018-09-01T17:56:32  *** mturquette has quit IRC
134 2018-09-01T17:56:32  *** bad_duck has quit IRC
135 2018-09-01T17:57:14  *** bad_duck has joined #bitcoin-core-dev
136 2018-09-01T18:01:46  *** stevenroose has joined #bitcoin-core-dev
137 2018-09-01T18:01:49  *** cncr04s has joined #bitcoin-core-dev
138 2018-09-01T18:01:57  *** Jbaczuk has joined #bitcoin-core-dev
139 2018-09-01T18:04:50  *** promag has joined #bitcoin-core-dev
140 2018-09-01T18:05:41  *** promag has quit IRC
141 2018-09-01T18:06:13  *** promag has joined #bitcoin-core-dev
142 2018-09-01T18:06:52  *** rafalcpp has quit IRC
143 2018-09-01T18:07:31  *** profmac has quit IRC
144 2018-09-01T18:08:04  *** Krellan has quit IRC
145 2018-09-01T18:10:36  *** promag has quit IRC
146 2018-09-01T18:19:28  *** rafalcpp has joined #bitcoin-core-dev
147 2018-09-01T18:23:40  *** promag has joined #bitcoin-core-dev
148 2018-09-01T18:24:10  *** profmac has joined #bitcoin-core-dev
149 2018-09-01T18:27:43  *** ExtraCrispy has quit IRC
150 2018-09-01T18:27:58  *** promag has quit IRC
151 2018-09-01T18:35:28  <sipa> jimpo: i guess!
152 2018-09-01T18:37:58  *** opdenkamp has quit IRC
153 2018-09-01T19:07:35  *** Guyver2 has quit IRC
154 2018-09-01T19:07:40  *** Krellan has joined #bitcoin-core-dev
155 2018-09-01T19:18:02  *** savil has quit IRC
156 2018-09-01T19:24:11  *** savil has joined #bitcoin-core-dev
157 2018-09-01T19:25:39  *** savil has quit IRC
158 2018-09-01T19:32:16  *** savil has joined #bitcoin-core-dev
159 2018-09-01T19:33:50  *** rafalcpp has quit IRC
160 2018-09-01T19:34:19  *** rafalcpp has joined #bitcoin-core-dev
161 2018-09-01T19:46:48  *** vexbuy has joined #bitcoin-core-dev
162 2018-09-01T19:46:49  *** Victorsueca has quit IRC
163 2018-09-01T19:47:57  *** Victorsueca has joined #bitcoin-core-dev
164 2018-09-01T19:50:24  *** hebasto has joined #bitcoin-core-dev
165 2018-09-01T20:00:03  *** g687 has quit IRC
166 2018-09-01T20:01:29  *** vexbuy has quit IRC
167 2018-09-01T20:15:47  *** Chris_Stewart_5 has quit IRC
168 2018-09-01T20:22:33  *** hebasto has quit IRC
169 2018-09-01T20:38:43  *** intcat has quit IRC
170 2018-09-01T20:41:14  *** intcat has joined #bitcoin-core-dev
171 2018-09-01T20:44:46  <wumpus> I think the idea is to not read it all into memory at some point
172 2018-09-01T20:44:56  <wumpus> just like with the wallet, FWIW
173 2018-09-01T20:45:58  <wumpus> for the block index, the pointers could be handles that prompt fetching some more specific data only on demand
174 2018-09-01T20:46:47  <wumpus> ken2812221: yes, that is funny
175 2018-09-01T20:46:59  <wumpus> ken2812221: the argument handling code is pretty weird in some regards, now
176 2018-09-01T20:47:14  <wumpus> I tried to document it, but I guess I failed
177 2018-09-01T20:48:34  <wumpus> I guess I'm going to untag #14105 and #14100 from 0.17.0
178 2018-09-01T20:48:36  <gribble> https://github.com/bitcoin/bitcoin/issues/14105 | util: Report parse errors in configuration file by laanwj · Pull Request #14105 · bitcoin/bitcoin · GitHubAsset 1Asset 1
179 2018-09-01T20:48:37  <gribble> https://github.com/bitcoin/bitcoin/issues/14100 | doc: Change documentation for =0 for non-boolean options by laanwj · Pull Request #14100 · bitcoin/bitcoin · GitHub
180 2018-09-01T20:49:14  <wumpus> we're never going to do a release if we try to solve this first
181 2018-09-01T20:53:18  <luke-jr> wumpus: aren't block indexes so small that it wouldn't be worth doing fetch-on-demand handles? (as opposed to fetch-on-demand map)
182 2018-09-01T20:56:06  <wumpus> luke-jr: there is certainly some minimum state that would make no sense to fetch on deman
183 2018-09-01T20:56:37  <wumpus> luke-jr: on the other hand, the structure per block is growing every release, I'm sure there are also things that don't make sense to read and store persistently
184 2018-09-01T20:58:58  <wumpus> luke-jr: I just meant I don't want to commit to a flat file because of that; also for updates, that would be much harder to manage
185 2018-09-01T20:59:29  <wumpus> having a block index database makes sense, no matter how exactly it's managed now
186 2018-09-01T21:02:44  <luke-jr> sure
187 2018-09-01T21:04:16  <wumpus> what is wrong with travis on master: https://travis-ci.org/bitcoin/bitcoin/builds/423430634?utm_source=email&utm_medium=notification
188 2018-09-01T21:04:31  <wumpus> the linting stage is failing but there are no errors
189 2018-09-01T21:11:20  <wumpus> of course it all passes perfectly locally
190 2018-09-01T21:14:04  *** Krellan has quit IRC
191 2018-09-01T21:14:42  *** Krellan has joined #bitcoin-core-dev
192 2018-09-01T21:15:19  *** maitra has joined #bitcoin-core-dev
193 2018-09-01T21:29:06  <gmaxwell> luke-jr: so sizeof(CBlockIndex) is 144 bytes, so thats 78MB (and slowly growing) of memory used for little particular purpose, excluding malloc overheads (which I guess are probably at least another 16 bytes per header).  The fact that we also keep so many of them in memory means a longer start time, and constant pressure to not add things to those objects with a result of reducing
194 2018-09-01T21:29:07  <gmaxwell> functionality.
195 2018-09-01T21:29:23  <gmaxwell> so I think it would make sense to eventually not keep them in memory.
196 2018-09-01T21:31:31  <gmaxwell> there should be no particular reason that someone couldn't run a fully functional bitcoin node using a few tens of MB of ram... though obviously not one with the lowest possible latency.
197 2018-09-01T21:31:56  <luke-jr> gmaxwell: sure, I'm just saying, a handle wouldn't be a big improvement
198 2018-09-01T21:32:36  <luke-jr> seems to make more sense to just create the indexx object itself on demand
199 2018-09-01T21:32:54  <luke-jr> and not store anything in memory per-block
200 2018-09-01T21:33:36  <gmaxwell> ah, I think I agree with that.
201 2018-09-01T21:34:42  <gmaxwell> Well really the access to the block index could be intermediated through a caching layer, so that the policy of what is in memory vs not is hidden from the rest of the code.
202 2018-09-01T21:34:55  <luke-jr> sure
203 2018-09-01T21:35:05  *** commavir has quit IRC
204 2018-09-01T21:39:25  *** commavir has joined #bitcoin-core-dev
205 2018-09-01T21:40:34  *** JackH has quit IRC
206 2018-09-01T21:47:34  <wumpus> "so I think it would make sense to eventually not keep them in memory" exactly
207 2018-09-01T21:48:54  <wumpus> I just meant we shouldn't be making any code changes in the direction of making that more difficult
208 2018-09-01T21:49:00  <wumpus> not so much 'we should be doing that now'
209 2018-09-01T21:49:32  <wumpus> I'd agree it's certainly not the biggest memory sink at the moent
210 2018-09-01T21:52:57  <gmaxwell> maybe one of the least useful ones, however.
211 2018-09-01T21:55:15  <wumpus> true
212 2018-09-01T22:00:02  *** d9b4bef9 has quit IRC
213 2018-09-01T22:08:50  *** michaelsdunn1 has joined #bitcoin-core-dev
214 2018-09-01T22:33:43  *** grubles has quit IRC
215 2018-09-01T22:34:15  *** grubles has joined #bitcoin-core-dev
216 2018-09-01T22:44:43  *** maitra has quit IRC
217 2018-09-01T22:54:09  *** spinza has quit IRC
218 2018-09-01T22:54:34  *** polydin has joined #bitcoin-core-dev
219 2018-09-01T22:56:13  *** polydin has quit IRC
220 2018-09-01T22:57:01  *** polydin has joined #bitcoin-core-dev
221 2018-09-01T23:02:06  *** no_input_found has quit IRC
222 2018-09-01T23:02:27  *** no_input_found has joined #bitcoin-core-dev
223 2018-09-01T23:04:02  *** spinza has joined #bitcoin-core-dev
224 2018-09-01T23:17:06  *** promag has joined #bitcoin-core-dev
225 2018-09-01T23:40:43  *** plankers has joined #bitcoin-core-dev
226 2018-09-01T23:41:04  *** promag has quit IRC
227 2018-09-01T23:53:24  *** michaelsdunn1 has quit IRC
228 2018-09-01T23:57:44  *** dendisuhubdy has joined #bitcoin-core-dev