12018-09-01T00:00:50 *** Chris_Stewart_5 has quit IRC
22018-09-01T00:27:21 *** sipa has joined #bitcoin-core-dev
32018-09-01T00:30:36 *** IGHOR has quit IRC
42018-09-01T00:31:06 <jimpo> gmaxwell: By "like the undo data", do you mean just that there are flat file storing the large values and disk positions stored in LevelDB or are you suggesting specifically that the filters be computed in validation code and referenced by the block index?
52018-09-01T00:32:59 *** IGHOR has joined #bitcoin-core-dev
62018-09-01T00:33:19 <jimpo> in #14121
72018-09-01T00:33:21 <gribble> https://github.com/bitcoin/bitcoin/issues/14121 | Index for BIP 157 block filters by jimpo · Pull Request #14121 · bitcoin/bitcoin · GitHub
82018-09-01T00:43:36 *** ghost43 has quit IRC
92018-09-01T00:44:17 *** ghost43 has joined #bitcoin-core-dev
102018-09-01T00:45:54 *** a5m0 has quit IRC
112018-09-01T00:55:57 *** a5m0 has joined #bitcoin-core-dev
122018-09-01T01:04:02 *** d9b4bef9 has quit IRC
132018-09-01T01:08:23 *** Guest79908 has joined #bitcoin-core-dev
142018-09-01T01:10:07 *** d9b4bef9 has joined #bitcoin-core-dev
152018-09-01T01:13:33 <gmaxwell> jimpo: the former.
162018-09-01T01:13:45 <gmaxwell> Flat files with the filters, indexed by position.
172018-09-01T01:33:45 <jimpo> Yeah, that makes sense to me. I'll code that up and compare read/write perf.
182018-09-01T01:46:25 <sipa> i doubt it matters much here; we're not throughput limited
192018-09-01T01:46:47 <sipa> leveldb writes all data twice, which is a reason against writing huge things like blocks and undo data
202018-09-01T01:49:06 <gmaxwell> also has varrious caching behaviors, also stores it somewhat inefficiently. I think that for recent blocks the filters are about 30KB per block? in any case if you think its okay it probably is.
212018-09-01T01:49:15 *** drexl has quit IRC
222018-09-01T01:49:35 <gmaxwell> I was also thinking about future dependency on leveldb... since we got non-atomic flushing, there are many other things possible for the chainstate.
232018-09-01T01:52:43 *** peevsie has joined #bitcoin-core-dev
242018-09-01T01:54:09 <echeveria> mongodb?
252018-09-01T01:56:32 <gmaxwell> it has webscale
262018-09-01T01:57:19 <gmaxwell> I don't think that even with non-atomic-flushing would mongo's consistency behavior be acceptable. :P
272018-09-01T01:58:39 *** Chris_Stewart_5 has joined #bitcoin-core-dev
282018-09-01T02:24:36 *** justanotheruser has quit IRC
292018-09-01T02:26:38 *** justanotheruser has joined #bitcoin-core-dev
302018-09-01T02:28:22 *** plankers has joined #bitcoin-core-dev
312018-09-01T02:32:54 <jimpo> mongo write consistency could be a decent entropy source
322018-09-01T03:18:45 *** unholymachine has quit IRC
332018-09-01T03:33:14 *** kinlo has quit IRC
342018-09-01T03:39:23 *** kinlo has joined #bitcoin-core-dev
352018-09-01T03:46:12 *** Krellan has quit IRC
362018-09-01T04:23:56 *** peevsie has quit IRC
372018-09-01T04:25:27 *** sipa has quit IRC
382018-09-01T04:29:52 *** plankers has quit IRC
392018-09-01T05:06:47 *** Chris_Stewart_5 has quit IRC
402018-09-01T05:11:01 *** d9b4bef9 has quit IRC
412018-09-01T05:14:07 *** d9b4bef9 has joined #bitcoin-core-dev
422018-09-01T05:27:15 *** Krellan has joined #bitcoin-core-dev
432018-09-01T05:46:08 *** sipa has joined #bitcoin-core-dev
442018-09-01T06:09:37 *** sipa has quit IRC
452018-09-01T06:22:54 *** luke-jr has quit IRC
462018-09-01T06:23:15 *** luke-jr has joined #bitcoin-core-dev
472018-09-01T06:25:08 *** luke-jr has quit IRC
482018-09-01T06:25:27 *** luke-jr has joined #bitcoin-core-dev
492018-09-01T06:41:15 *** luke-jr has quit IRC
502018-09-01T06:41:43 *** luke-jr has joined #bitcoin-core-dev
512018-09-01T06:45:49 *** luke-jr has quit IRC
522018-09-01T06:46:03 *** luke-jr has joined #bitcoin-core-dev
532018-09-01T06:47:30 *** luke-jr has quit IRC
542018-09-01T06:48:28 *** luke-jr has joined #bitcoin-core-dev
552018-09-01T06:50:07 *** luke-jr has quit IRC
562018-09-01T06:50:23 *** luke-jr has joined #bitcoin-core-dev
572018-09-01T06:54:22 *** luke-jr has quit IRC
582018-09-01T06:54:40 *** luke-jr has joined #bitcoin-core-dev
592018-09-01T07:02:27 *** SopaXorzTaker has joined #bitcoin-core-dev
602018-09-01T07:07:24 *** sipa has joined #bitcoin-core-dev
612018-09-01T07:18:47 *** Victorsueca has quit IRC
622018-09-01T07:20:02 *** Victorsueca has joined #bitcoin-core-dev
632018-09-01T07:34:29 *** Krellan has quit IRC
642018-09-01T07:35:21 *** Krellan has joined #bitcoin-core-dev
652018-09-01T07:58:29 *** promag has joined #bitcoin-core-dev
662018-09-01T08:08:15 *** luke-jr has quit IRC
672018-09-01T08:08:32 *** luke-jr has joined #bitcoin-core-dev
682018-09-01T08:18:42 *** luke-jr has quit IRC
692018-09-01T08:19:05 *** luke-jr has joined #bitcoin-core-dev
702018-09-01T08:44:52 *** lukedashjr has joined #bitcoin-core-dev
712018-09-01T08:46:17 *** SopaXorzTaker has quit IRC
722018-09-01T08:47:18 *** luke-jr has quit IRC
732018-09-01T08:49:19 *** lukedashjr is now known as luke-jr
742018-09-01T08:52:01 *** d9b4bef9 has quit IRC
752018-09-01T08:53:04 *** profmac has quit IRC
762018-09-01T08:53:08 *** d9b4bef9 has joined #bitcoin-core-dev
772018-09-01T08:55:23 *** promag has quit IRC
782018-09-01T09:14:42 *** Victorsueca has quit IRC
792018-09-01T09:15:57 *** Victorsueca has joined #bitcoin-core-dev
802018-09-01T09:55:20 *** profmac has joined #bitcoin-core-dev
812018-09-01T10:04:20 *** savil has quit IRC
822018-09-01T10:05:19 *** promag has joined #bitcoin-core-dev
832018-09-01T10:09:08 *** savil has joined #bitcoin-core-dev
842018-09-01T10:14:40 *** promag has quit IRC
852018-09-01T10:15:51 *** profmac has quit IRC
862018-09-01T10:33:22 *** profmac has joined #bitcoin-core-dev
872018-09-01T10:49:24 *** lukedashjr has joined #bitcoin-core-dev
882018-09-01T10:50:28 *** luke-jr has quit IRC
892018-09-01T10:50:28 *** Krellan has quit IRC
902018-09-01T10:53:51 *** lukedashjr is now known as luke-jr
912018-09-01T10:56:48 *** hebasto has joined #bitcoin-core-dev
922018-09-01T11:05:28 *** lnostdal has quit IRC
932018-09-01T11:17:19 *** lnostdal has joined #bitcoin-core-dev
942018-09-01T11:24:09 *** lnostdal has quit IRC
952018-09-01T11:36:30 *** lnostdal has joined #bitcoin-core-dev
962018-09-01T11:41:12 *** lnostdal has quit IRC
972018-09-01T11:41:44 *** lnostdal has joined #bitcoin-core-dev
982018-09-01T11:47:25 *** Guyver2 has joined #bitcoin-core-dev
992018-09-01T11:49:27 *** lnostdal has quit IRC
1002018-09-01T11:53:50 *** opdenkamp has quit IRC
1012018-09-01T12:00:56 *** lnostdal has joined #bitcoin-core-dev
1022018-09-01T12:12:29 *** itaseski has joined #bitcoin-core-dev
1032018-09-01T12:16:05 *** owowo has joined #bitcoin-core-dev
1042018-09-01T12:39:06 *** hebasto has quit IRC
1052018-09-01T13:14:37 *** opdenkamp has joined #bitcoin-core-dev
1062018-09-01T13:23:11 *** Chris_Stewart_5 has joined #bitcoin-core-dev
1072018-09-01T13:52:33 *** lnostdal has quit IRC
1082018-09-01T13:58:33 *** g687 has joined #bitcoin-core-dev
1092018-09-01T14:05:31 *** lnostdal has joined #bitcoin-core-dev
1102018-09-01T14:09:13 *** Chris_Stewart_5 has quit IRC
1112018-09-01T14:16:21 *** Chris_Stewart_5 has joined #bitcoin-core-dev
1122018-09-01T14:22:53 *** ken2812221 has quit IRC
1132018-09-01T14:57:24 *** g687 has quit IRC
1142018-09-01T15:04:51 *** ken2812221 has joined #bitcoin-core-dev
1152018-09-01T15:35:36 *** opdenkamp has quit IRC
1162018-09-01T15:40:00 *** opdenkamp has joined #bitcoin-core-dev
1172018-09-01T15:40:28 *** jarthur has joined #bitcoin-core-dev
1182018-09-01T16:01:11 *** Krellan has joined #bitcoin-core-dev
1192018-09-01T16:21:11 *** jarthur has quit IRC
1202018-09-01T16:43:48 *** owowo has quit IRC
1212018-09-01T16:45:09 *** Victorsueca has quit IRC
1222018-09-01T16:46:27 *** Victorsueca has joined #bitcoin-core-dev
1232018-09-01T16:48:32 *** jhfrontz has joined #bitcoin-core-dev
1242018-09-01T16:48:45 *** owowo has joined #bitcoin-core-dev
1252018-09-01T17:02:04 <jimpo> hmm, seems the whole block tree db could be moved to flat files since it's all read into memory on startup anyway
1262018-09-01T17:43:42 *** g687 has joined #bitcoin-core-dev
1272018-09-01T17:56:31 *** Jbaczuk has quit IRC
1282018-09-01T17:56:31 *** stevenroose has quit IRC
1292018-09-01T17:56:31 *** cncr04s has quit IRC
1302018-09-01T17:56:32 *** exit70 has quit IRC
1312018-09-01T17:56:32 *** bosma has quit IRC
1322018-09-01T17:56:32 *** epic has quit IRC
1332018-09-01T17:56:32 *** mturquette has quit IRC
1342018-09-01T17:56:32 *** bad_duck has quit IRC
1352018-09-01T17:57:14 *** bad_duck has joined #bitcoin-core-dev
1362018-09-01T18:01:46 *** stevenroose has joined #bitcoin-core-dev
1372018-09-01T18:01:49 *** cncr04s has joined #bitcoin-core-dev
1382018-09-01T18:01:57 *** Jbaczuk has joined #bitcoin-core-dev
1392018-09-01T18:04:50 *** promag has joined #bitcoin-core-dev
1402018-09-01T18:05:41 *** promag has quit IRC
1412018-09-01T18:06:13 *** promag has joined #bitcoin-core-dev
1422018-09-01T18:06:52 *** rafalcpp has quit IRC
1432018-09-01T18:07:31 *** profmac has quit IRC
1442018-09-01T18:08:04 *** Krellan has quit IRC
1452018-09-01T18:10:36 *** promag has quit IRC
1462018-09-01T18:19:28 *** rafalcpp has joined #bitcoin-core-dev
1472018-09-01T18:23:40 *** promag has joined #bitcoin-core-dev
1482018-09-01T18:24:10 *** profmac has joined #bitcoin-core-dev
1492018-09-01T18:27:43 *** ExtraCrispy has quit IRC
1502018-09-01T18:27:58 *** promag has quit IRC
1512018-09-01T18:35:28 <sipa> jimpo: i guess!
1522018-09-01T18:37:58 *** opdenkamp has quit IRC
1532018-09-01T19:07:35 *** Guyver2 has quit IRC
1542018-09-01T19:07:40 *** Krellan has joined #bitcoin-core-dev
1552018-09-01T19:18:02 *** savil has quit IRC
1562018-09-01T19:24:11 *** savil has joined #bitcoin-core-dev
1572018-09-01T19:25:39 *** savil has quit IRC
1582018-09-01T19:32:16 *** savil has joined #bitcoin-core-dev
1592018-09-01T19:33:50 *** rafalcpp has quit IRC
1602018-09-01T19:34:19 *** rafalcpp has joined #bitcoin-core-dev
1612018-09-01T19:46:48 *** vexbuy has joined #bitcoin-core-dev
1622018-09-01T19:46:49 *** Victorsueca has quit IRC
1632018-09-01T19:47:57 *** Victorsueca has joined #bitcoin-core-dev
1642018-09-01T19:50:24 *** hebasto has joined #bitcoin-core-dev
1652018-09-01T20:00:03 *** g687 has quit IRC
1662018-09-01T20:01:29 *** vexbuy has quit IRC
1672018-09-01T20:15:47 *** Chris_Stewart_5 has quit IRC
1682018-09-01T20:22:33 *** hebasto has quit IRC
1692018-09-01T20:38:43 *** intcat has quit IRC
1702018-09-01T20:41:14 *** intcat has joined #bitcoin-core-dev
1712018-09-01T20:44:46 <wumpus> I think the idea is to not read it all into memory at some point
1722018-09-01T20:44:56 <wumpus> just like with the wallet, FWIW
1732018-09-01T20:45:58 <wumpus> for the block index, the pointers could be handles that prompt fetching some more specific data only on demand
1742018-09-01T20:46:47 <wumpus> ken2812221: yes, that is funny
1752018-09-01T20:46:59 <wumpus> ken2812221: the argument handling code is pretty weird in some regards, now
1762018-09-01T20:47:14 <wumpus> I tried to document it, but I guess I failed
1772018-09-01T20:48:34 <wumpus> I guess I'm going to untag #14105 and #14100 from 0.17.0
1782018-09-01T20:48:36 <gribble> https://github.com/bitcoin/bitcoin/issues/14105 | util: Report parse errors in configuration file by laanwj · Pull Request #14105 · bitcoin/bitcoin · GitHubAsset 1Asset 1
1792018-09-01T20:48:37 <gribble> https://github.com/bitcoin/bitcoin/issues/14100 | doc: Change documentation for =0 for non-boolean options by laanwj · Pull Request #14100 · bitcoin/bitcoin · GitHub
1802018-09-01T20:49:14 <wumpus> we're never going to do a release if we try to solve this first
1812018-09-01T20:53:18 <luke-jr> wumpus: aren't block indexes so small that it wouldn't be worth doing fetch-on-demand handles? (as opposed to fetch-on-demand map)
1822018-09-01T20:56:06 <wumpus> luke-jr: there is certainly some minimum state that would make no sense to fetch on deman
1832018-09-01T20:56:37 <wumpus> luke-jr: on the other hand, the structure per block is growing every release, I'm sure there are also things that don't make sense to read and store persistently
1842018-09-01T20:58:58 <wumpus> luke-jr: I just meant I don't want to commit to a flat file because of that; also for updates, that would be much harder to manage
1852018-09-01T20:59:29 <wumpus> having a block index database makes sense, no matter how exactly it's managed now
1862018-09-01T21:02:44 <luke-jr> sure
1872018-09-01T21:04:16 <wumpus> what is wrong with travis on master: https://travis-ci.org/bitcoin/bitcoin/builds/423430634?utm_source=email&utm_medium=notification
1882018-09-01T21:04:31 <wumpus> the linting stage is failing but there are no errors
1892018-09-01T21:11:20 <wumpus> of course it all passes perfectly locally
1902018-09-01T21:14:04 *** Krellan has quit IRC
1912018-09-01T21:14:42 *** Krellan has joined #bitcoin-core-dev
1922018-09-01T21:15:19 *** maitra has joined #bitcoin-core-dev
1932018-09-01T21:29:06 <gmaxwell> luke-jr: so sizeof(CBlockIndex) is 144 bytes, so thats 78MB (and slowly growing) of memory used for little particular purpose, excluding malloc overheads (which I guess are probably at least another 16 bytes per header). The fact that we also keep so many of them in memory means a longer start time, and constant pressure to not add things to those objects with a result of reducing
1942018-09-01T21:29:07 <gmaxwell> functionality.
1952018-09-01T21:29:23 <gmaxwell> so I think it would make sense to eventually not keep them in memory.
1962018-09-01T21:31:31 <gmaxwell> there should be no particular reason that someone couldn't run a fully functional bitcoin node using a few tens of MB of ram... though obviously not one with the lowest possible latency.
1972018-09-01T21:31:56 <luke-jr> gmaxwell: sure, I'm just saying, a handle wouldn't be a big improvement
1982018-09-01T21:32:36 <luke-jr> seems to make more sense to just create the indexx object itself on demand
1992018-09-01T21:32:54 <luke-jr> and not store anything in memory per-block
2002018-09-01T21:33:36 <gmaxwell> ah, I think I agree with that.
2012018-09-01T21:34:42 <gmaxwell> Well really the access to the block index could be intermediated through a caching layer, so that the policy of what is in memory vs not is hidden from the rest of the code.
2022018-09-01T21:34:55 <luke-jr> sure
2032018-09-01T21:35:05 *** commavir has quit IRC
2042018-09-01T21:39:25 *** commavir has joined #bitcoin-core-dev
2052018-09-01T21:40:34 *** JackH has quit IRC
2062018-09-01T21:47:34 <wumpus> "so I think it would make sense to eventually not keep them in memory" exactly
2072018-09-01T21:48:54 <wumpus> I just meant we shouldn't be making any code changes in the direction of making that more difficult
2082018-09-01T21:49:00 <wumpus> not so much 'we should be doing that now'
2092018-09-01T21:49:32 <wumpus> I'd agree it's certainly not the biggest memory sink at the moent
2102018-09-01T21:52:57 <gmaxwell> maybe one of the least useful ones, however.
2112018-09-01T21:55:15 <wumpus> true
2122018-09-01T22:00:02 *** d9b4bef9 has quit IRC
2132018-09-01T22:08:50 *** michaelsdunn1 has joined #bitcoin-core-dev
2142018-09-01T22:33:43 *** grubles has quit IRC
2152018-09-01T22:34:15 *** grubles has joined #bitcoin-core-dev
2162018-09-01T22:44:43 *** maitra has quit IRC
2172018-09-01T22:54:09 *** spinza has quit IRC
2182018-09-01T22:54:34 *** polydin has joined #bitcoin-core-dev
2192018-09-01T22:56:13 *** polydin has quit IRC
2202018-09-01T22:57:01 *** polydin has joined #bitcoin-core-dev
2212018-09-01T23:02:06 *** no_input_found has quit IRC
2222018-09-01T23:02:27 *** no_input_found has joined #bitcoin-core-dev
2232018-09-01T23:04:02 *** spinza has joined #bitcoin-core-dev
2242018-09-01T23:17:06 *** promag has joined #bitcoin-core-dev
2252018-09-01T23:40:43 *** plankers has joined #bitcoin-core-dev
2262018-09-01T23:41:04 *** promag has quit IRC
2272018-09-01T23:53:24 *** michaelsdunn1 has quit IRC
2282018-09-01T23:57:44 *** dendisuhubdy has joined #bitcoin-core-dev