1 2018-09-01T00:00:50 *** Chris_Stewart_5 has quit IRC
2 2018-09-01T00:27:21 *** sipa has joined #bitcoin-core-dev
3 2018-09-01T00:30:36 *** IGHOR has quit IRC
4 2018-09-01T00:31:06 <jimpo> gmaxwell: By "like the undo data", do you mean just that there are flat file storing the large values and disk positions stored in LevelDB or are you suggesting specifically that the filters be computed in validation code and referenced by the block index?
5 2018-09-01T00:32:59 *** IGHOR has joined #bitcoin-core-dev
6 2018-09-01T00:33:19 <jimpo> in #14121
7 2018-09-01T00:33:21 <gribble> https://github.com/bitcoin/bitcoin/issues/14121 | Index for BIP 157 block filters by jimpo · Pull Request #14121 · bitcoin/bitcoin · GitHub
8 2018-09-01T00:43:36 *** ghost43 has quit IRC
9 2018-09-01T00:44:17 *** ghost43 has joined #bitcoin-core-dev
10 2018-09-01T00:45:54 *** a5m0 has quit IRC
11 2018-09-01T00:55:57 *** a5m0 has joined #bitcoin-core-dev
12 2018-09-01T01:04:02 *** d9b4bef9 has quit IRC
13 2018-09-01T01:08:23 *** Guest79908 has joined #bitcoin-core-dev
14 2018-09-01T01:10:07 *** d9b4bef9 has joined #bitcoin-core-dev
15 2018-09-01T01:13:33 <gmaxwell> jimpo: the former.
16 2018-09-01T01:13:45 <gmaxwell> Flat files with the filters, indexed by position.
17 2018-09-01T01:33:45 <jimpo> Yeah, that makes sense to me. I'll code that up and compare read/write perf.
18 2018-09-01T01:46:25 <sipa> i doubt it matters much here; we're not throughput limited
19 2018-09-01T01:46:47 <sipa> leveldb writes all data twice, which is a reason against writing huge things like blocks and undo data
20 2018-09-01T01:49:06 <gmaxwell> also has varrious caching behaviors, also stores it somewhat inefficiently. I think that for recent blocks the filters are about 30KB per block? in any case if you think its okay it probably is.
21 2018-09-01T01:49:15 *** drexl has quit IRC
22 2018-09-01T01:49:35 <gmaxwell> I was also thinking about future dependency on leveldb... since we got non-atomic flushing, there are many other things possible for the chainstate.
23 2018-09-01T01:52:43 *** peevsie has joined #bitcoin-core-dev
24 2018-09-01T01:54:09 <echeveria> mongodb?
25 2018-09-01T01:56:32 <gmaxwell> it has webscale
26 2018-09-01T01:57:19 <gmaxwell> I don't think that even with non-atomic-flushing would mongo's consistency behavior be acceptable. :P
27 2018-09-01T01:58:39 *** Chris_Stewart_5 has joined #bitcoin-core-dev
28 2018-09-01T02:24:36 *** justanotheruser has quit IRC
29 2018-09-01T02:26:38 *** justanotheruser has joined #bitcoin-core-dev
30 2018-09-01T02:28:22 *** plankers has joined #bitcoin-core-dev
31 2018-09-01T02:32:54 <jimpo> mongo write consistency could be a decent entropy source
32 2018-09-01T03:18:45 *** unholymachine has quit IRC
33 2018-09-01T03:33:14 *** kinlo has quit IRC
34 2018-09-01T03:39:23 *** kinlo has joined #bitcoin-core-dev
35 2018-09-01T03:46:12 *** Krellan has quit IRC
36 2018-09-01T04:23:56 *** peevsie has quit IRC
37 2018-09-01T04:25:27 *** sipa has quit IRC
38 2018-09-01T04:29:52 *** plankers has quit IRC
39 2018-09-01T05:06:47 *** Chris_Stewart_5 has quit IRC
40 2018-09-01T05:11:01 *** d9b4bef9 has quit IRC
41 2018-09-01T05:14:07 *** d9b4bef9 has joined #bitcoin-core-dev
42 2018-09-01T05:27:15 *** Krellan has joined #bitcoin-core-dev
43 2018-09-01T05:46:08 *** sipa has joined #bitcoin-core-dev
44 2018-09-01T06:09:37 *** sipa has quit IRC
45 2018-09-01T06:22:54 *** luke-jr has quit IRC
46 2018-09-01T06:23:15 *** luke-jr has joined #bitcoin-core-dev
47 2018-09-01T06:25:08 *** luke-jr has quit IRC
48 2018-09-01T06:25:27 *** luke-jr has joined #bitcoin-core-dev
49 2018-09-01T06:41:15 *** luke-jr has quit IRC
50 2018-09-01T06:41:43 *** luke-jr has joined #bitcoin-core-dev
51 2018-09-01T06:45:49 *** luke-jr has quit IRC
52 2018-09-01T06:46:03 *** luke-jr has joined #bitcoin-core-dev
53 2018-09-01T06:47:30 *** luke-jr has quit IRC
54 2018-09-01T06:48:28 *** luke-jr has joined #bitcoin-core-dev
55 2018-09-01T06:50:07 *** luke-jr has quit IRC
56 2018-09-01T06:50:23 *** luke-jr has joined #bitcoin-core-dev
57 2018-09-01T06:54:22 *** luke-jr has quit IRC
58 2018-09-01T06:54:40 *** luke-jr has joined #bitcoin-core-dev
59 2018-09-01T07:02:27 *** SopaXorzTaker has joined #bitcoin-core-dev
60 2018-09-01T07:07:24 *** sipa has joined #bitcoin-core-dev
61 2018-09-01T07:18:47 *** Victorsueca has quit IRC
62 2018-09-01T07:20:02 *** Victorsueca has joined #bitcoin-core-dev
63 2018-09-01T07:34:29 *** Krellan has quit IRC
64 2018-09-01T07:35:21 *** Krellan has joined #bitcoin-core-dev
65 2018-09-01T07:58:29 *** promag has joined #bitcoin-core-dev
66 2018-09-01T08:08:15 *** luke-jr has quit IRC
67 2018-09-01T08:08:32 *** luke-jr has joined #bitcoin-core-dev
68 2018-09-01T08:18:42 *** luke-jr has quit IRC
69 2018-09-01T08:19:05 *** luke-jr has joined #bitcoin-core-dev
70 2018-09-01T08:44:52 *** lukedashjr has joined #bitcoin-core-dev
71 2018-09-01T08:46:17 *** SopaXorzTaker has quit IRC
72 2018-09-01T08:47:18 *** luke-jr has quit IRC
73 2018-09-01T08:49:19 *** lukedashjr is now known as luke-jr
74 2018-09-01T08:52:01 *** d9b4bef9 has quit IRC
75 2018-09-01T08:53:04 *** profmac has quit IRC
76 2018-09-01T08:53:08 *** d9b4bef9 has joined #bitcoin-core-dev
77 2018-09-01T08:55:23 *** promag has quit IRC
78 2018-09-01T09:14:42 *** Victorsueca has quit IRC
79 2018-09-01T09:15:57 *** Victorsueca has joined #bitcoin-core-dev
80 2018-09-01T09:55:20 *** profmac has joined #bitcoin-core-dev
81 2018-09-01T10:04:20 *** savil has quit IRC
82 2018-09-01T10:05:19 *** promag has joined #bitcoin-core-dev
83 2018-09-01T10:09:08 *** savil has joined #bitcoin-core-dev
84 2018-09-01T10:14:40 *** promag has quit IRC
85 2018-09-01T10:15:51 *** profmac has quit IRC
86 2018-09-01T10:33:22 *** profmac has joined #bitcoin-core-dev
87 2018-09-01T10:49:24 *** lukedashjr has joined #bitcoin-core-dev
88 2018-09-01T10:50:28 *** luke-jr has quit IRC
89 2018-09-01T10:50:28 *** Krellan has quit IRC
90 2018-09-01T10:53:51 *** lukedashjr is now known as luke-jr
91 2018-09-01T10:56:48 *** hebasto has joined #bitcoin-core-dev
92 2018-09-01T11:05:28 *** lnostdal has quit IRC
93 2018-09-01T11:17:19 *** lnostdal has joined #bitcoin-core-dev
94 2018-09-01T11:24:09 *** lnostdal has quit IRC
95 2018-09-01T11:36:30 *** lnostdal has joined #bitcoin-core-dev
96 2018-09-01T11:41:12 *** lnostdal has quit IRC
97 2018-09-01T11:41:44 *** lnostdal has joined #bitcoin-core-dev
98 2018-09-01T11:47:25 *** Guyver2 has joined #bitcoin-core-dev
99 2018-09-01T11:49:27 *** lnostdal has quit IRC
100 2018-09-01T11:53:50 *** opdenkamp has quit IRC
101 2018-09-01T12:00:56 *** lnostdal has joined #bitcoin-core-dev
102 2018-09-01T12:12:29 *** itaseski has joined #bitcoin-core-dev
103 2018-09-01T12:16:05 *** owowo has joined #bitcoin-core-dev
104 2018-09-01T12:39:06 *** hebasto has quit IRC
105 2018-09-01T13:14:37 *** opdenkamp has joined #bitcoin-core-dev
106 2018-09-01T13:23:11 *** Chris_Stewart_5 has joined #bitcoin-core-dev
107 2018-09-01T13:52:33 *** lnostdal has quit IRC
108 2018-09-01T13:58:33 *** g687 has joined #bitcoin-core-dev
109 2018-09-01T14:05:31 *** lnostdal has joined #bitcoin-core-dev
110 2018-09-01T14:09:13 *** Chris_Stewart_5 has quit IRC
111 2018-09-01T14:16:21 *** Chris_Stewart_5 has joined #bitcoin-core-dev
112 2018-09-01T14:22:53 *** ken2812221 has quit IRC
113 2018-09-01T14:57:24 *** g687 has quit IRC
114 2018-09-01T15:04:51 *** ken2812221 has joined #bitcoin-core-dev
115 2018-09-01T15:35:36 *** opdenkamp has quit IRC
116 2018-09-01T15:40:00 *** opdenkamp has joined #bitcoin-core-dev
117 2018-09-01T15:40:28 *** jarthur has joined #bitcoin-core-dev
118 2018-09-01T16:01:11 *** Krellan has joined #bitcoin-core-dev
119 2018-09-01T16:21:11 *** jarthur has quit IRC
120 2018-09-01T16:43:48 *** owowo has quit IRC
121 2018-09-01T16:45:09 *** Victorsueca has quit IRC
122 2018-09-01T16:46:27 *** Victorsueca has joined #bitcoin-core-dev
123 2018-09-01T16:48:32 *** jhfrontz has joined #bitcoin-core-dev
124 2018-09-01T16:48:45 *** owowo has joined #bitcoin-core-dev
125 2018-09-01T17:02:04 <jimpo> hmm, seems the whole block tree db could be moved to flat files since it's all read into memory on startup anyway
126 2018-09-01T17:43:42 *** g687 has joined #bitcoin-core-dev
127 2018-09-01T17:56:31 *** Jbaczuk has quit IRC
128 2018-09-01T17:56:31 *** stevenroose has quit IRC
129 2018-09-01T17:56:31 *** cncr04s has quit IRC
130 2018-09-01T17:56:32 *** exit70 has quit IRC
131 2018-09-01T17:56:32 *** bosma has quit IRC
132 2018-09-01T17:56:32 *** epic has quit IRC
133 2018-09-01T17:56:32 *** mturquette has quit IRC
134 2018-09-01T17:56:32 *** bad_duck has quit IRC
135 2018-09-01T17:57:14 *** bad_duck has joined #bitcoin-core-dev
136 2018-09-01T18:01:46 *** stevenroose has joined #bitcoin-core-dev
137 2018-09-01T18:01:49 *** cncr04s has joined #bitcoin-core-dev
138 2018-09-01T18:01:57 *** Jbaczuk has joined #bitcoin-core-dev
139 2018-09-01T18:04:50 *** promag has joined #bitcoin-core-dev
140 2018-09-01T18:05:41 *** promag has quit IRC
141 2018-09-01T18:06:13 *** promag has joined #bitcoin-core-dev
142 2018-09-01T18:06:52 *** rafalcpp has quit IRC
143 2018-09-01T18:07:31 *** profmac has quit IRC
144 2018-09-01T18:08:04 *** Krellan has quit IRC
145 2018-09-01T18:10:36 *** promag has quit IRC
146 2018-09-01T18:19:28 *** rafalcpp has joined #bitcoin-core-dev
147 2018-09-01T18:23:40 *** promag has joined #bitcoin-core-dev
148 2018-09-01T18:24:10 *** profmac has joined #bitcoin-core-dev
149 2018-09-01T18:27:43 *** ExtraCrispy has quit IRC
150 2018-09-01T18:27:58 *** promag has quit IRC
151 2018-09-01T18:35:28 <sipa> jimpo: i guess!
152 2018-09-01T18:37:58 *** opdenkamp has quit IRC
153 2018-09-01T19:07:35 *** Guyver2 has quit IRC
154 2018-09-01T19:07:40 *** Krellan has joined #bitcoin-core-dev
155 2018-09-01T19:18:02 *** savil has quit IRC
156 2018-09-01T19:24:11 *** savil has joined #bitcoin-core-dev
157 2018-09-01T19:25:39 *** savil has quit IRC
158 2018-09-01T19:32:16 *** savil has joined #bitcoin-core-dev
159 2018-09-01T19:33:50 *** rafalcpp has quit IRC
160 2018-09-01T19:34:19 *** rafalcpp has joined #bitcoin-core-dev
161 2018-09-01T19:46:48 *** vexbuy has joined #bitcoin-core-dev
162 2018-09-01T19:46:49 *** Victorsueca has quit IRC
163 2018-09-01T19:47:57 *** Victorsueca has joined #bitcoin-core-dev
164 2018-09-01T19:50:24 *** hebasto has joined #bitcoin-core-dev
165 2018-09-01T20:00:03 *** g687 has quit IRC
166 2018-09-01T20:01:29 *** vexbuy has quit IRC
167 2018-09-01T20:15:47 *** Chris_Stewart_5 has quit IRC
168 2018-09-01T20:22:33 *** hebasto has quit IRC
169 2018-09-01T20:38:43 *** intcat has quit IRC
170 2018-09-01T20:41:14 *** intcat has joined #bitcoin-core-dev
171 2018-09-01T20:44:46 <wumpus> I think the idea is to not read it all into memory at some point
172 2018-09-01T20:44:56 <wumpus> just like with the wallet, FWIW
173 2018-09-01T20:45:58 <wumpus> for the block index, the pointers could be handles that prompt fetching some more specific data only on demand
174 2018-09-01T20:46:47 <wumpus> ken2812221: yes, that is funny
175 2018-09-01T20:46:59 <wumpus> ken2812221: the argument handling code is pretty weird in some regards, now
176 2018-09-01T20:47:14 <wumpus> I tried to document it, but I guess I failed
177 2018-09-01T20:48:34 <wumpus> I guess I'm going to untag #14105 and #14100 from 0.17.0
178 2018-09-01T20:48:36 <gribble> https://github.com/bitcoin/bitcoin/issues/14105 | util: Report parse errors in configuration file by laanwj · Pull Request #14105 · bitcoin/bitcoin · GitHubAsset 1Asset 1
179 2018-09-01T20:48:37 <gribble> https://github.com/bitcoin/bitcoin/issues/14100 | doc: Change documentation for =0 for non-boolean options by laanwj · Pull Request #14100 · bitcoin/bitcoin · GitHub
180 2018-09-01T20:49:14 <wumpus> we're never going to do a release if we try to solve this first
181 2018-09-01T20:53:18 <luke-jr> wumpus: aren't block indexes so small that it wouldn't be worth doing fetch-on-demand handles? (as opposed to fetch-on-demand map)
182 2018-09-01T20:56:06 <wumpus> luke-jr: there is certainly some minimum state that would make no sense to fetch on deman
183 2018-09-01T20:56:37 <wumpus> luke-jr: on the other hand, the structure per block is growing every release, I'm sure there are also things that don't make sense to read and store persistently
184 2018-09-01T20:58:58 <wumpus> luke-jr: I just meant I don't want to commit to a flat file because of that; also for updates, that would be much harder to manage
185 2018-09-01T20:59:29 <wumpus> having a block index database makes sense, no matter how exactly it's managed now
186 2018-09-01T21:02:44 <luke-jr> sure
187 2018-09-01T21:04:16 <wumpus> what is wrong with travis on master: https://travis-ci.org/bitcoin/bitcoin/builds/423430634?utm_source=email&utm_medium=notification
188 2018-09-01T21:04:31 <wumpus> the linting stage is failing but there are no errors
189 2018-09-01T21:11:20 <wumpus> of course it all passes perfectly locally
190 2018-09-01T21:14:04 *** Krellan has quit IRC
191 2018-09-01T21:14:42 *** Krellan has joined #bitcoin-core-dev
192 2018-09-01T21:15:19 *** maitra has joined #bitcoin-core-dev
193 2018-09-01T21:29:06 <gmaxwell> luke-jr: so sizeof(CBlockIndex) is 144 bytes, so thats 78MB (and slowly growing) of memory used for little particular purpose, excluding malloc overheads (which I guess are probably at least another 16 bytes per header). The fact that we also keep so many of them in memory means a longer start time, and constant pressure to not add things to those objects with a result of reducing
194 2018-09-01T21:29:07 <gmaxwell> functionality.
195 2018-09-01T21:29:23 <gmaxwell> so I think it would make sense to eventually not keep them in memory.
196 2018-09-01T21:31:31 <gmaxwell> there should be no particular reason that someone couldn't run a fully functional bitcoin node using a few tens of MB of ram... though obviously not one with the lowest possible latency.
197 2018-09-01T21:31:56 <luke-jr> gmaxwell: sure, I'm just saying, a handle wouldn't be a big improvement
198 2018-09-01T21:32:36 <luke-jr> seems to make more sense to just create the indexx object itself on demand
199 2018-09-01T21:32:54 <luke-jr> and not store anything in memory per-block
200 2018-09-01T21:33:36 <gmaxwell> ah, I think I agree with that.
201 2018-09-01T21:34:42 <gmaxwell> Well really the access to the block index could be intermediated through a caching layer, so that the policy of what is in memory vs not is hidden from the rest of the code.
202 2018-09-01T21:34:55 <luke-jr> sure
203 2018-09-01T21:35:05 *** commavir has quit IRC
204 2018-09-01T21:39:25 *** commavir has joined #bitcoin-core-dev
205 2018-09-01T21:40:34 *** JackH has quit IRC
206 2018-09-01T21:47:34 <wumpus> "so I think it would make sense to eventually not keep them in memory" exactly
207 2018-09-01T21:48:54 <wumpus> I just meant we shouldn't be making any code changes in the direction of making that more difficult
208 2018-09-01T21:49:00 <wumpus> not so much 'we should be doing that now'
209 2018-09-01T21:49:32 <wumpus> I'd agree it's certainly not the biggest memory sink at the moent
210 2018-09-01T21:52:57 <gmaxwell> maybe one of the least useful ones, however.
211 2018-09-01T21:55:15 <wumpus> true
212 2018-09-01T22:00:02 *** d9b4bef9 has quit IRC
213 2018-09-01T22:08:50 *** michaelsdunn1 has joined #bitcoin-core-dev
214 2018-09-01T22:33:43 *** grubles has quit IRC
215 2018-09-01T22:34:15 *** grubles has joined #bitcoin-core-dev
216 2018-09-01T22:44:43 *** maitra has quit IRC
217 2018-09-01T22:54:09 *** spinza has quit IRC
218 2018-09-01T22:54:34 *** polydin has joined #bitcoin-core-dev
219 2018-09-01T22:56:13 *** polydin has quit IRC
220 2018-09-01T22:57:01 *** polydin has joined #bitcoin-core-dev
221 2018-09-01T23:02:06 *** no_input_found has quit IRC
222 2018-09-01T23:02:27 *** no_input_found has joined #bitcoin-core-dev
223 2018-09-01T23:04:02 *** spinza has joined #bitcoin-core-dev
224 2018-09-01T23:17:06 *** promag has joined #bitcoin-core-dev
225 2018-09-01T23:40:43 *** plankers has joined #bitcoin-core-dev
226 2018-09-01T23:41:04 *** promag has quit IRC
227 2018-09-01T23:53:24 *** michaelsdunn1 has quit IRC
228 2018-09-01T23:57:44 *** dendisuhubdy has joined #bitcoin-core-dev