12015-10-22T00:12:54 <Luke-Jr> re sigop-limit flooding, can we just merge acceptnonstdtxn (mainnet option, off by default) finally? this would not be a problem if people were using code with it merged..
22015-10-22T00:15:32 <Luke-Jr> https://github.com/bitcoin/bitcoin/pull/559
32015-10-22T00:16:36 *** PaulCapestany has quit IRC
42015-10-22T00:19:02 <gmaxwell> Luke-Jr: explain your logic?
52015-10-22T00:19:32 <Luke-Jr> gmaxwell: the top commit limits (in policy) tx sigop count based on its size
62015-10-22T00:19:55 <Luke-Jr> so the spam would need to be larger for it to use 15 sigops
72015-10-22T00:20:21 <Luke-Jr> thus hitting the block size limit approximately at the same time as the sigop limit
82015-10-22T00:20:49 <gmaxwell> I ~think~ the suggestion I made of max(size, sigops*1e6/20000) is more general than the limit thing.
92015-10-22T00:20:54 <Luke-Jr> (and triggering higher fees, of course)
102015-10-22T00:21:28 <Luke-Jr> gmaxwell: yes, my point is that the bug wouldn't exist in the first place had this already been merged
112015-10-22T00:21:42 *** jl2012 has quit IRC
122015-10-22T00:23:13 *** PaulCapestany has joined #bitcoin-core-dev
132015-10-22T00:54:19 *** Ylbam has quit IRC
142015-10-22T00:54:27 *** PaulCapestany has quit IRC
152015-10-22T00:55:38 *** PaulCapestany has joined #bitcoin-core-dev
162015-10-22T01:10:07 <BlueMatt> morcos: you asked about tx cache? https://github.com/TheBlueMatt/bitcoin/tree/limitucache
172015-10-22T01:10:27 <BlueMatt> actually, I'll just pr
182015-10-22T01:11:47 <GitHub110> [bitcoin] TheBlueMatt opened pull request #6868: Limit txn cache growth (master...limitucache) https://github.com/bitcoin/bitcoin/pull/6868
192015-10-22T01:13:17 <gmaxwell> BlueMatt: for your don't fork from here-- I suggest https://github.com/gmaxwell/secp256k1
202015-10-22T01:13:50 <BlueMatt> gmaxwell: yes, but that takes effort....
212015-10-22T01:14:44 <gmaxwell> well if someone does it to the repo first you can check out theirs and force push it into yours. :P
222015-10-22T01:14:51 <gmaxwell> I guess I should do bitcoin core too.
232015-10-22T01:16:48 <GitHub26> [bitcoin] TheBlueMatt closed pull request #6868: Limit txn cache growth (master...limitucache) https://github.com/bitcoin/bitcoin/pull/6868
242015-10-22T01:46:28 <morcos> BlueMatt: interesting idea.. i'm not sure what i think of it yet... you don't thinks also potentially inefficient
252015-10-22T01:48:26 <morcos> did you see the conversation wumpus and I were having earlier
262015-10-22T01:48:52 <morcos> one thing we could do in ATMP, is not pull into the pcoinstip cache coins for txs which aren't actually accepted into the memory pool
272015-10-22T01:56:03 <sipa> yeah, we should do that
282015-10-22T01:56:35 <sipa> i see sometimes huge increases in utxo cache size after a transaction
292015-10-22T01:58:24 <sipa> however, with bith limited utxo cache and mempool, memory usage actually remains *very* low
302015-10-22T01:58:50 <sipa> have a node with 100 MB utxo and 200 MB mempool, and total memory is staying below 570 M
312015-10-22T01:59:08 *** amiller has quit IRC
322015-10-22T02:01:59 <morcos> just don't run GetBlockTemplate! :)
332015-10-22T02:02:17 <morcos> sipa: why does CreateNewBlock need to hold cs_main the whole time
342015-10-22T02:02:35 <sipa> no idea
352015-10-22T02:02:50 <sipa> also no idea why it actually needs to verify the result
362015-10-22T02:03:03 <morcos> it seems like after the first round of calling HaveCoins on all the txins of all the txs.. you now have a view with all of of your coins in it
372015-10-22T02:03:16 <morcos> and you could release cs_main
382015-10-22T02:03:27 <sipa> the mempool could change then, though?
392015-10-22T02:03:35 <morcos> i took a first stab at a background miner, but realized its useless if its holding cs_main
402015-10-22T02:04:39 <morcos> well... what do you mean change? i think we don't want to lose txs that we might be trying to mine
412015-10-22T02:04:54 <morcos> but maybe you can handle that by not removing things
422015-10-22T02:05:29 <morcos> a smaller lock that just guards actually removing txs
432015-10-22T02:06:13 <morcos> why would it not need to verify the result, you mean because it shouldn't have been able to create something invalid
442015-10-22T02:06:20 <sipa> yes
452015-10-22T02:06:21 <morcos> i do think thats a nice double check
462015-10-22T02:06:33 <sipa> i don't think it's worth the cost
472015-10-22T02:06:41 <sipa> given that it hasn't failed in years
482015-10-22T02:06:54 <morcos> well it will be if we're chaning the mining algorithm!
492015-10-22T02:06:59 <morcos> changing
502015-10-22T02:06:59 <sipa> maybe a spot check would be useful
512015-10-22T02:07:08 <sipa> run it once every 1000 calls
522015-10-22T02:07:13 <jgarzik> since it's in the background the cost is mitigated
532015-10-22T02:07:25 <morcos> holds cs_main
542015-10-22T02:07:31 <sipa> block validation will not be background
552015-10-22T02:07:44 <sipa> it needs the utxo set
562015-10-22T02:08:08 <sipa> though i guess you are right that of it actually does make a full copy of all inlits, it can release the lock afterwards
572015-10-22T02:08:10 <jgarzik> yes it needs utxo set but doesn't need cs_main long term
582015-10-22T02:08:21 <sipa> well it needs the utxo lock, whatever that is
592015-10-22T02:08:28 <morcos> yeah i think we could be smart about it
602015-10-22T02:08:43 <sipa> i think it shouldn't need the utxo set or copies at all
612015-10-22T02:08:49 <sipa> the mempool should be consistent
622015-10-22T02:09:01 <sipa> do a spot check occasionally to make sure the code isn't broken
632015-10-22T02:09:10 <morcos> right now it already pulls the potentially used utxos into caches TWICE. so you could use one for creation and the other for validation without needing to grab cs_main again
642015-10-22T02:09:17 <sipa> yup
652015-10-22T02:09:26 <sipa> it should be 0, imho
662015-10-22T02:09:45 <morcos> oh
672015-10-22T02:09:52 <morcos> thats what you mean by mempool should be consistent
682015-10-22T02:09:55 *** MarcoFalke has quit IRC
692015-10-22T02:09:55 <sipa> the mempool is known to not double spend
702015-10-22T02:09:56 <morcos> yikes!
712015-10-22T02:10:03 <sipa> not contain invalid transactions
722015-10-22T02:10:04 <morcos> but thats also been broken
732015-10-22T02:10:10 <sipa> so we should fix it
742015-10-22T02:10:32 <morcos> but what i'm saying is that breaks often enough that you wouldn't want it to cause you to mine an invalid block?
752015-10-22T02:10:42 <morcos> at least one check should still exist
762015-10-22T02:10:43 <sipa> that's like saying "we're not sure about the utxo cache, let's run checkblockchain on every update"
772015-10-22T02:10:49 <sipa> sure the check should exist
782015-10-22T02:11:00 <sipa> but it should not be slowing down GBT by seconds!
792015-10-22T02:11:21 <morcos> ok ok
802015-10-22T02:11:29 <sipa> every single call
812015-10-22T02:12:04 <morcos> its just scary when you have a giant block of code protected by cs_main.. trying to reason about what in there really cares about it
822015-10-22T02:12:28 <sipa> well you don't want the best block or the mempool to change underneath it
832015-10-22T02:12:57 <sipa> or you will be pulling in transactions that may conflict with each other
842015-10-22T02:13:00 <Luke-Jr> sipa: GBT verifies the result because it's worth it. No miner wants to find out after the fact they mined an invalid block.
852015-10-22T02:13:01 <morcos> if the best block changes i've got other problems.. i don't want to be calculating this block anyway
862015-10-22T02:13:17 <sipa> Luke-Jr: the code should not produce invalid blocks, period
872015-10-22T02:13:30 <Luke-Jr> also, spot checking just leads to people /expecting/ a lower time than they can rely on
882015-10-22T02:13:38 <sipa> Luke-Jr: if you run the check once every 1000 calls you will equally well detect broken code
892015-10-22T02:13:48 <Luke-Jr> not equally well, no.
902015-10-22T02:13:58 <sipa> if the code is broken, it will fail badly
912015-10-22T02:14:01 <Luke-Jr> and it will mislead people into thinking it's faster than it's necessarily
922015-10-22T02:14:22 <Luke-Jr> an edge case will not necessarily fail badly.
932015-10-22T02:14:40 <sipa> so run with -checkmempool on then
942015-10-22T02:14:50 <Luke-Jr> without the verification, there is no consensus code involved in producing blocks here..
952015-10-22T02:14:54 <Luke-Jr> valid transactions != valid block
962015-10-22T02:14:54 <sipa> then you will also not get an invalid memlool state
972015-10-22T02:15:13 <sipa> just get that check out of the mining path
982015-10-22T02:15:14 *** amiller has joined #bitcoin-core-dev
992015-10-22T02:15:27 <sipa> i wonder how much gbt delay has encourage validationless mining...
1002015-10-22T02:15:55 <Luke-Jr> removing the check *is* validationless.
1012015-10-22T02:16:00 <morcos> I'm with Luke-Jr that I think consensus code should happen at least once in the path
1022015-10-22T02:16:07 <sipa> it is not if the code is not broken
1032015-10-22T02:16:08 <morcos> but it doesn't have to happen twice as it does now
1042015-10-22T02:16:19 <Luke-Jr> morcos: wait, twice?
1052015-10-22T02:16:31 <sipa> once for building, once for checking
1062015-10-22T02:16:37 <morcos> CheckInputs on the transactions and TestBlockValidity
1072015-10-22T02:16:59 *** amiller has quit IRC
1082015-10-22T02:17:17 <Luke-Jr> CheckInputs doesn't check the entire block, but maybe it can be removed
1092015-10-22T02:17:40 <sipa> it does check the whole block eventually, as it's called for all transactions that end up in it
1102015-10-22T02:18:02 <Luke-Jr> sipa: no, because all the transactions can be valid without the block itself being valid
1112015-10-22T02:18:05 <morcos> but i think we're talking about different things here
1122015-10-22T02:18:15 <sipa> Luke-Jr: i know that
1132015-10-22T02:18:28 <morcos> i'm concerned with having the block-template thread hold cs_main too much...
1142015-10-22T02:18:36 <morcos> CheckInputs doesn't need cs_main once you have a view
1152015-10-22T02:18:50 <sipa> but checkinputs is called for the equivalent amount of work of validating the whole block
1162015-10-22T02:18:51 <morcos> sipa your concern seems to be latency in GBT
1172015-10-22T02:18:58 <morcos> but thats solvable in other ways
1182015-10-22T02:18:59 <sipa> yes
1192015-10-22T02:19:02 <sipa> ok
1202015-10-22T02:19:40 <morcos> as long as you think its ok to have brief periods of mining on an empty block or something
1212015-10-22T02:19:50 <Luke-Jr> GBT isn't supposed to be time-critical anyway (but I already plan to improve it)
1222015-10-22T02:20:01 *** belcher has quit IRC
1232015-10-22T02:20:09 <sipa> well it can't take seconds...
1242015-10-22T02:20:44 <Luke-Jr> maybe ideally it wouldn't, but it shouldn't hurt much
1252015-10-22T02:20:55 <morcos> forget about how long it takes, what you care about is how long after a new block comes in that you have a header you can mine that builds off it right?
1262015-10-22T02:21:08 <Luke-Jr> also afaik right now it only takes seconds when the miner is neglecting to keep a clean mempool
1272015-10-22T02:21:12 <morcos> it doesn't matter if it then takes seconds to generate a new one with txs right?
1282015-10-22T02:21:31 <Luke-Jr> since it needs to go over every tx in the mempool
1292015-10-22T02:21:37 <Luke-Jr> not because of the test afterward
1302015-10-22T02:21:39 <sipa> morcos: as long as fees are negligablez yes
1312015-10-22T02:21:40 <morcos> although seconds is really absurdly slow and we should be able to get it under that even for that case
1322015-10-22T02:21:57 <morcos> Luke-Jr: yeah thats one of the problems, is that now it loops all the txs twice
1332015-10-22T02:22:01 <morcos> once putting them all in vecPriority
1342015-10-22T02:22:15 <morcos> and then trying to go through vecPriority and put them all in a block
1352015-10-22T02:22:18 <Luke-Jr> yeah
1362015-10-22T02:22:19 <morcos> with a sorted mempool
1372015-10-22T02:22:30 <morcos> both become unnecessary (the 2nd was already unnecessary)
1382015-10-22T02:22:34 <Luke-Jr> sorted mempool is essentially what I'm working on
1392015-10-22T02:22:43 <Luke-Jr> although I need to rebase on top of the packages stuff I expect
1402015-10-22T02:22:55 <morcos> sdaftuar is very close to finishing the ancestor package tracking
1412015-10-22T02:23:16 <morcos> if you want to build a more general sorting priority score, thats probably what you should build off of
1422015-10-22T02:23:26 <Luke-Jr> not priority score
1432015-10-22T02:23:36 <Luke-Jr> sorting based on the user-defined policy ;)
1442015-10-22T02:23:42 <morcos> yeah thats what i meant
1452015-10-22T02:24:04 <morcos> don't you have to somehow turn their policy into a score?
1462015-10-22T02:24:09 <Luke-Jr> no
1472015-10-22T02:24:18 <morcos> how can you sort without a score?
1482015-10-22T02:24:25 <Luke-Jr> the policy is implemented mostly by a method that compares two transactions
1492015-10-22T02:25:12 *** amiller has joined #bitcoin-core-dev
1502015-10-22T02:25:24 <morcos> and if A < B and B < C then A < C right?
1512015-10-22T02:27:05 <Luke-Jr> right
1522015-10-22T02:27:16 <Luke-Jr> or at least, that's assumed by the mempool
1532015-10-22T02:27:34 <Luke-Jr> it might be fun to some day make blocks with a policy that just returns rand() :P
1542015-10-22T02:27:37 <morcos> ok well the point i was going to make was that if you could look at it as a score
1552015-10-22T02:27:47 <morcos> and A has child A2 and B has child B2
1562015-10-22T02:28:04 <sipa> you're defining a total ordering, that is equivalent to a score function
1572015-10-22T02:28:09 <Luke-Jr> yes, you could look at it that way; my point was that actually using a score is overcomplex
1582015-10-22T02:28:28 <morcos> then if you use suhas's ancestor package tracking with your score instead of fee you'll get CPFP in terms of your policy
1592015-10-22T02:31:02 <morcos> sipa: what are your thoughts on maintaining ancestor package tracking on nodes that aren't mining
1602015-10-22T02:31:46 <morcos> its not totally useless and it hopefully shouldn't be too expensive. but certainly adds more mememory/cpu for something they are basically not using
1612015-10-22T02:32:47 <morcos> the one thing i think it could be used for is fee estimation? but thats only a maybe, i haven't really figured out how or if that would be worth it. but estimating what miners are going to do seems like potentially a valuable tool for fee estimation.
1622015-10-22T02:45:24 *** CodeShark has quit IRC
1632015-10-22T02:45:43 *** CodeShark has joined #bitcoin-core-dev
1642015-10-22T02:48:04 *** CodeShark has quit IRC
1652015-10-22T02:48:50 *** CodeShark has joined #bitcoin-core-dev
1662015-10-22T03:01:16 *** Arnavion has quit IRC
1672015-10-22T03:01:40 *** Arnavion has joined #bitcoin-core-dev
1682015-10-22T03:39:25 <gmaxwell> cfields_: good catch, apparently it's not inhereted reliably and there is a bunch of broken software out there.
1692015-10-22T03:50:02 <gmaxwell> (by reliably I mean it's system dependant)
1702015-10-22T04:59:59 *** harding has quit IRC
1712015-10-22T05:54:15 *** Thireus has joined #bitcoin-core-dev
1722015-10-22T06:21:58 *** Thireus has quit IRC
1732015-10-22T06:22:10 *** Ylbam has joined #bitcoin-core-dev
1742015-10-22T06:35:39 *** fkhan has quit IRC
1752015-10-22T06:41:06 *** Thireus has joined #bitcoin-core-dev
1762015-10-22T06:47:25 *** fkhan has joined #bitcoin-core-dev
1772015-10-22T07:38:12 *** Thireus has quit IRC
1782015-10-22T07:40:32 *** ParadoxSpiral_ has joined #bitcoin-core-dev
1792015-10-22T07:43:54 *** ParadoxSpiral has quit IRC
1802015-10-22T08:11:48 *** Thireus has joined #bitcoin-core-dev
1812015-10-22T08:13:53 <phantomcircuit> can confirm bitcoin core requires reindex on power failure under windows
1822015-10-22T08:14:05 <phantomcircuit> probably leveldb env driver is stupid
1832015-10-22T08:15:16 <gmaxwell> phantomcircuit: fixy fixy? unfortunately I think it's unmaintained.
1842015-10-22T08:15:35 <gmaxwell> as chrome uses some chrome specific abstractions for IO. :(
1852015-10-22T08:19:37 <phantomcircuit> gmaxwell, i can take a look at it, probably it should just be replaced with the current posix version which is just fopen/fread/fwrite basically
1862015-10-22T08:24:07 <GitHub16> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/0fbfc5106cd9...a09297010e17
1872015-10-22T08:24:07 <GitHub16> bitcoin/master 579b863 Wladimir J. van der Laan: devtools: Add security-check.py...
1882015-10-22T08:24:08 <GitHub16> bitcoin/master a092970 Wladimir J. van der Laan: Merge pull request #6854...
1892015-10-22T08:24:18 <GitHub129> [bitcoin] laanwj closed pull request #6854: devtools: Add security-check.py (master...2015_10_security_checks) https://github.com/bitcoin/bitcoin/pull/6854
1902015-10-22T08:25:30 *** Thireus has quit IRC
1912015-10-22T08:26:09 <wumpus> phantomcircuit: yeah https://github.com/bitcoin/bitcoin/issues/5610
1922015-10-22T08:26:16 <wumpus> no clue on solving it though
1932015-10-22T08:26:43 <wumpus> for some reason most people with a clue about windows internals seem to be blackhats, and they like things broken :p
1942015-10-22T08:27:13 *** Thireus has joined #bitcoin-core-dev
1952015-10-22T08:27:39 <phantomcircuit> wumpus, it's using windows memory mapping
1962015-10-22T08:27:39 <gmaxwell> phantomcircuit: reports went up astronomically after 0.10 and some users reported it worked fine for them before them and fails every time since.
1972015-10-22T08:27:44 <phantomcircuit> im going to assume it's that
1982015-10-22T08:28:00 <gmaxwell> phantomcircuit: oh it's writing via mmap too?
1992015-10-22T08:28:19 <phantomcircuit> i just checked and there's a Win32Map something or other
2002015-10-22T08:28:23 <phantomcircuit> i assume it's using that
2012015-10-22T08:29:15 <gmaxwell> https://litteration.files.wordpress.com/2013/05/science-dog.jpg
2022015-10-22T08:29:27 <wumpus> it has been a while since we updated leveldb, maybe some things got fixed since?
2032015-10-22T08:30:11 <wumpus> heh
2042015-10-22T08:30:36 <phantomcircuit> gmaxwell, that's about right
2052015-10-22T08:31:42 <phantomcircuit> it's windows... how do you even begin to debug this
2062015-10-22T08:32:38 <wumpus> that's a dark art
2072015-10-22T08:33:38 <wumpus> wouldn't be surprised if it requieres at least some, possibly expensive, proprietary software applications
2082015-10-22T08:35:25 <wumpus> debugging-by-mutating-the-code may be the most promising approach, extract a bare bones program that uses leveldb and crashes, then change it (adding flushes everywhere) until it manages to not corrupt anymore
2092015-10-22T08:36:09 <gmaxwell> it's 100% reproducable it seems, which helps.
2102015-10-22T08:36:11 <wumpus> this is the approach that works for fixing problems with underdocumented hardware, it should work for windows too...
2112015-10-22T08:36:53 <gmaxwell> the fact that it went way up when we turned up checking suggests to me that it's something like a truncated record at the end that was otherwise silently recoverable.
2122015-10-22T08:37:01 <gmaxwell> e.g. something like the osx bug.
2132015-10-22T08:38:10 <wumpus> yes good point - can you send me a database that is corrupted by this problem?
2142015-10-22T08:39:36 <gmaxwell> phantomcircuit: you would be you, as I don't have a repro.
2152015-10-22T08:40:59 <phantomcircuit> heh
2162015-10-22T08:41:16 <phantomcircuit> gmaxwell, i suspect you're right that it's similar to the OSX bug
2172015-10-22T08:41:27 <phantomcircuit> i also suspect it would be easier to replace leveldb with lmdb
2182015-10-22T08:41:40 <wumpus> phantomcircuit: can you send me a database that is corrupted by this problem?
2192015-10-22T08:41:40 <gmaxwell> mmap only, goodbye 32 bit hosts.
2202015-10-22T08:41:44 <gmaxwell> Among other limitations. :(
2212015-10-22T08:41:45 <Arnavion> It's funny because if you ask Windows devs they'll tell you the dev tooling on Windows is far better than on Linux
2222015-10-22T08:42:10 <phantomcircuit> wumpus, i mindlessly clicked the reindex button
2232015-10-22T08:42:11 <gmaxwell> Arnavion: there is some aspect of "tools you know are always better than ones you don't"
2242015-10-22T08:42:17 <phantomcircuit> i can certainly try to do so though
2252015-10-22T08:42:24 <wumpus> Arnavion: it may be more user friendly on windows, but it's harder to get insight about what is happening on a system
2262015-10-22T08:42:25 <Arnavion> There's also the aspect of GUI tools that don't suck
2272015-10-22T08:42:27 <Arnavion> but I'm biased
2282015-10-22T08:42:46 <gmaxwell> Arnavion: you mean GUI tools inherently sucking? :P
2292015-10-22T08:43:05 <Arnavion> Oh, but that's what WinDbg is for
2302015-10-22T08:43:28 <Arnavion> You can do everything with a keyboard if you wish
2312015-10-22T08:43:54 <wumpus> anyhow this is not constructive
2322015-10-22T08:44:13 <wumpus> if you feel helpful, please demonstrate your skills by solving the leveldb corruption issue Arnavion :)
2332015-10-22T08:44:59 <Arnavion> I have no skills
2342015-10-22T08:45:21 <Arnavion> but I can take a look
2352015-10-22T08:45:27 <wumpus> demonstrate the great capability of windows debugging tools, then
2362015-10-22T08:46:49 <wumpus> phantomcircuit: but you can reproduce it! just start a new datadir, crash the thing, and send it to me
2372015-10-22T08:51:51 <wumpus> lmdb is not the panacea some people think - but if you feel like patching bitcoind to use it, that'd be pretty neat as we could run some benchmarks and comparisons
2382015-10-22T08:59:07 <phantomcircuit> wumpus, the main thing that's missing from leveldb is... sequence numbers in the journal
2392015-10-22T08:59:21 <phantomcircuit> if you miss a full record of writes it doesn't know
2402015-10-22T09:00:28 <wumpus> still, we need to debug this issue, as it is clearly windows specific, so not fundamental to leveldb
2412015-10-22T09:00:54 <wumpus> if this was broken on all platforms I'd agree that looking for a replacement would be advisable...
2422015-10-22T09:01:43 <phantomcircuit> wumpus, it *is* broken on all platforms
2432015-10-22T09:01:52 <phantomcircuit> the failure rate is simply much lower
2442015-10-22T09:02:24 <gmaxwell> one step at a time.
2452015-10-22T09:02:32 <gmaxwell> lets making the bleeding stop on windows first.
2462015-10-22T09:03:52 <wumpus> that's a nihilistic way of looking at it phantomcircuit, I like that, yes, it's all broken, some things with a very low failure rate... but it doesn't help solving the issue :)
2472015-10-22T09:04:35 *** rubensayshi has joined #bitcoin-core-dev
2482015-10-22T09:05:44 <wumpus> I'm sure lmdb is also broken in some subtle ways
2492015-10-22T09:06:17 <wumpus> anyhow, if you're not going to send me a corrupted datbase I'm going to start working on something else
2502015-10-22T09:07:50 <gmaxwell> I assume he has to wait for the reindex to complete to reproduce. :P
2512015-10-22T09:08:07 <wumpus> wha?
2522015-10-22T09:08:25 <wumpus> I assumed it would also happen when crashing during reindex?
2532015-10-22T09:08:37 <Luke-Jr> wumpus: what is Windows specfic?
2542015-10-22T09:09:50 <Luke-Jr> [08:13:52] <phantomcircuit> can confirm bitcoin core requires reindex on power failure under windows <-- I can confirm it requires reindex on power failure under Linux..
2552015-10-22T09:09:53 <wumpus> in that case, sorry, no hurry implied
2562015-10-22T09:10:49 <Luke-Jr> wumpus: if you in fact want such a corrupt bitcoin dir, I can probably get one for oyu
2572015-10-22T09:10:54 * Luke-Jr ponders where he put his USB Armory
2582015-10-22T09:11:19 <Luke-Jr> oh, it's plugged in
2592015-10-22T09:11:53 <wumpus> Luke-Jr: great!
2602015-10-22T09:12:14 <phantomcircuit> Luke-Jr, ive never had that issue on linux
2612015-10-22T09:12:17 <gmaxwell> it's interesting that the usb armory corrupts, perhaps not the same issue as windows though (though perhaps also worth fixing)
2622015-10-22T09:14:41 <wumpus> indeed, may be a different issue
2632015-10-22T09:22:26 <Luke-Jr> FWIW, removing power immediately after IBD started did NOT reproduce it
2642015-10-22T09:22:40 <Luke-Jr> anything in debug.log to look for to know when it does the first flush to disk?
2652015-10-22T09:23:27 <wumpus> set a low dbcache to force lots of flushes?
2662015-10-22T09:23:55 <wumpus> AFAIK it doesn't flush unless necessary until the initial block download is complete
2672015-10-22T09:25:04 *** BashCo has joined #bitcoin-core-dev
2682015-10-22T09:27:52 <Luke-Jr> 'often' is too slow
2692015-10-22T09:28:01 <Luke-Jr> 2015-10-22 09:27:59 UpdateTip: new best=000000003f7e074587fa1684ac863519fea3c64040b05ddd04948a13f7b19b42 height=415 log2_work=40.700462 tx=423 date=2009-01-14 05:56:05 progress=0.000002 cache=419
2702015-10-22T09:33:59 <Arnavion> From just lazy-mode browsing the code, Win32MapFile::Sync() flushes but Win32MapFile::Flush() is a no-op, whereas the equivalent functions for posix both do things
2712015-10-22T09:34:26 <Arnavion> I did not see how they are called from common code to see if that matters
2722015-10-22T09:37:56 <wumpus> it all depends on whether this map file is used for writing
2732015-10-22T09:38:17 <wumpus> AFAIK on other OSes, writing uses normal file system commands, whereas reading can use mmap
2742015-10-22T09:38:41 <wumpus> if t his is different on windows that would explain something
2752015-10-22T09:39:18 <Arnavion> Windows has a dedicated FlushViewOfFile function to flush a mapped region
2762015-10-22T09:39:53 <Arnavion> That is called by Sync(), but not by Flush()
2772015-10-22T09:58:45 <Luke-Jr> hmm
2782015-10-22T09:58:57 <Luke-Jr> it seems I can't reproduce during IBD or something
2792015-10-22T10:07:25 <Luke-Jr> and a fully synced state is huge :/
2802015-10-22T10:07:48 *** jgarzik has quit IRC
2812015-10-22T10:08:36 <midnightmagic> Luke-Jr: you're reproducing on a Windows OS?
2822015-10-22T10:08:44 <Luke-Jr> midnightmagic: no
2832015-10-22T10:18:03 *** jgarzik has joined #bitcoin-core-dev
2842015-10-22T10:18:03 *** jgarzik has joined #bitcoin-core-dev
2852015-10-22T10:25:35 * Luke-Jr ponders
2862015-10-22T10:26:31 <Luke-Jr> wumpus: well, the last chainstate I had before today is 922 MB and demanded a reindex.. not sure if that's good enough, because I can't upload 44 GB :/
2872015-10-22T10:27:12 <wumpus> that's kind of huge - do you perhaps know what ldb file the corruption is in?
2882015-10-22T10:27:38 <Luke-Jr> no, I have no idea how leveldb works :/
2892015-10-22T10:28:08 <Luke-Jr> I could probably just give you access to the device?
2902015-10-22T10:39:52 <Luke-Jr> wumpus: see PM for login info
2912015-10-22T10:39:55 <wumpus> ok
2922015-10-22T10:40:16 <Luke-Jr> setup auth for your github ssh key; let me know if there's a better one
2932015-10-22T10:40:48 <Luke-Jr> wumpus: the leveldb in question is under ~/.bitcoin.bak
2942015-10-22T10:41:34 <Luke-Jr> perhaps relevant: tar: /home/usbarmory/.bitcoin.bak/chainstate/1463292.ldb: File shrank by 2020259 bytes; padding with zeros
2952015-10-22T10:42:39 <wumpus> uh...
2962015-10-22T10:42:56 <wumpus> yes, that doesn't sound good. Were you tarring while bitcoin running?
2972015-10-22T10:43:33 <Luke-Jr> not sure if bitcoind is running, but if it is, it's not running on *this* database
2982015-10-22T10:43:59 * Luke-Jr wonders what it's doing so much to hang SSH
2992015-10-22T10:44:31 <wumpus> ok, it would not be strange if it is happening while running tar on the datadir that bitcoin was running in, those files are deleted and recreated all the time
3002015-10-22T10:44:46 <Luke-Jr> right
3012015-10-22T10:45:32 <sipa> wumpus: leveldb seems unmaintained
3022015-10-22T10:47:26 <wumpus> bleh
3032015-10-22T10:52:50 <wumpus> I'm happy we at least never decided to use it for the wallet
3042015-10-22T10:53:39 <sipa> maybe we should switch to sqlite... at least that's very well tested and maintained
3052015-10-22T10:54:33 <wumpus> it is, but it's not very fast for key/value storage, and I don't think it handles such large databases very well
3062015-10-22T10:55:40 <btcdrak> sipa: sqlite is very slow
3072015-10-22T10:56:08 <wumpus> if you need more advanced query features it is very nice though
3082015-10-22T10:57:09 <btcdrak> it would be so much easier and flexible if we were using SQL of some kind...
3092015-10-22T10:57:23 <wumpus> what would be easier?
3102015-10-22T10:57:45 <sipa> btcdrak: leveldb's own benchmark says that sqlite is faster for random reads
3112015-10-22T10:57:55 <btcdrak> sipa: interesting
3122015-10-22T10:57:59 <wumpus> for wallet metadata sql would be reasonably useful
3132015-10-22T10:58:02 <sipa> writes are much slower, but we do very few writes
3142015-10-22T10:58:13 <btcdrak> wumpus: well you get a lot of stuff for free with a SQL database stack.
3152015-10-22T10:58:24 <sipa> such as?
3162015-10-22T10:58:43 <Luke-Jr> # dmesg
3172015-10-22T10:58:45 <Luke-Jr> Segmentation fault
3182015-10-22T10:58:46 <Luke-Jr> -.-
3192015-10-22T10:59:07 <wumpus> during initial sync we do quite a lot of writes (every coin that is touched is written back), after that, most are reads
3202015-10-22T10:59:21 <btcdrak> sipa: indexes for a start. the ability to query the dataset externally from bitcoind etc
3212015-10-22T10:59:51 <wumpus> using database indexes would mean using a very verbose format
3222015-10-22T10:59:54 <sipa> btcdrak: we're not storing any data in a way that's useful to index
3232015-10-22T11:00:07 <sipa> we'd lose massive performance by expanding the data
3242015-10-22T11:00:28 <wumpus> that kind of defeats the purpose - bitcoind's database should optimized for its own purposes as running a node, not external ones
3252015-10-22T11:00:39 <btcdrak> I had wondered about leveldb's ongoing support. it really looks like an orphaned project at this time.
3262015-10-22T11:01:23 <wumpus> sipa: right
3272015-10-22T11:01:45 <jgarzik> btcdrak, the expanded dataset in SQL is something like 40GB at least
3282015-10-22T11:01:58 <jgarzik> not including the block data itself
3292015-10-22T11:02:38 <jgarzik> sipa, worse comes to worse, we can do our own key/value store...
3302015-10-22T11:02:45 <wumpus> also bitcoin's databases are not an external interface, don't query them directly unless it's to write troubleshooting/recovery tools
3312015-10-22T11:03:24 <jgarzik> there are actually a few optimizations you can make if you assume your keys are hashes already
3322015-10-22T11:03:48 <jgarzik> not that seeks matter these days, but some seeks can be eliminated
3332015-10-22T11:04:09 <wumpus> reinventing the wheel should be the last recourse
3342015-10-22T11:04:36 <Luke-Jr> wumpus: whatever just happened appears to have fried the device, so I guess it'll be a while :/
3352015-10-22T11:04:40 <jgarzik> <grin> https://github.com/jgarzik/pgdb https://github.com/jgarzik/pagedb
3362015-10-22T11:04:48 <wumpus> Luke-Jr: I hope it wasn't me logging in! :p
3372015-10-22T11:05:07 <Luke-Jr> heh, I don't see how that could do it
3382015-10-22T11:05:14 <sipa> leveldb does have a benchmark about batch write performance
3392015-10-22T11:05:23 <sipa> where sqlite is much slower
3402015-10-22T11:05:24 <wumpus> I don't see either, I didn't even get a prompt
3412015-10-22T11:05:40 <wumpus> I've disconnected now
3422015-10-22T11:05:46 <sipa> though sqlite does win in synchronous write speed, which we also do frequently
3432015-10-22T11:06:34 <Luke-Jr> wumpus: makes me wonder if it's just a bad microSD card, which could make the failures invalid
3442015-10-22T11:06:49 <wumpus> would be interesting to patch sqlite into bitcoind and compare and do benchmarks
3452015-10-22T11:06:52 <sipa> the only reason i'm suggesting sqlite is because it is *very* well tested afaik
3462015-10-22T11:07:00 * wumpus says that for the second time today about a different database
3472015-10-22T11:07:20 <wumpus> sipa: also on windows?
3482015-10-22T11:07:42 <wumpus> leveldb is also very well tested, it is used by many companies in production... just mostly on linux/unix servers
3492015-10-22T11:08:05 <sipa> yes
3502015-10-22T11:10:09 <wumpus> do you know, how does sqlite do at caching?
3512015-10-22T11:10:20 <btcdrak> sipa: wumpus: maintenance is probably more of a concern than anything else. I do remember wondering about leveldb's longevity. sqlite is way too popular to disappear.
3522015-10-22T11:10:36 <jgarzik> sqlite leans a lot of OS caching, temporary files etc.
3532015-10-22T11:10:58 <wumpus> jgarzik: it can't do a much worse job than leveldb at that at least
3542015-10-22T11:11:36 <btcdrak> if we're just doing key/value what about all those database that do that sort of thing, like those nosql stacks?
3552015-10-22T11:12:02 *** PRab has joined #bitcoin-core-dev
3562015-10-22T11:12:38 <jgarzik> I'm a big fan of sqlite, and used it in my DNS server project. I don't know that sqlite has batch update - does begin/commit suffice given our workflow? They are -mostly- equivalent but not 100% equiv
3572015-10-22T11:12:42 <btcdrak> still. sqlite would be a better choice. But here's the thing, if we use sqlite then you're opening up the door for just about any database backend. specially if we made an abstraction layer
3582015-10-22T11:12:44 <wumpus> "The maximum size of a database file is 2147483646 pages. At the maximum page size of 65536 bytes, this translates into a maximum database size of approximately 1.4e+14 bytes " ok, apparently I misremembered
3592015-10-22T11:13:15 <wumpus> btcdrak: there aren't that much embeddable key/value stores
3602015-10-22T11:13:26 <btcdrak> wumpus: true.
3612015-10-22T11:13:39 <btcdrak> and sqlite wins in terms of support/maintenance.
3622015-10-22T11:13:42 <jgarzik> I certainly wouldn't suggest expanding the bitcoind db in SQL, but using it as a dumb blob datastore shouldn't be a big issue
3632015-10-22T11:13:43 <wumpus> and most of them are likely one-off projects, even worsely maintained than leveldb
3642015-10-22T11:13:44 <wumpus> right
3652015-10-22T11:13:51 <wumpus> jgarzik: +1
3662015-10-22T11:13:59 *** PRab_ has quit IRC
3672015-10-22T11:13:59 <btcdrak> +!
3682015-10-22T11:14:04 <btcdrak> +1
3692015-10-22T11:14:41 <Luke-Jr> reminder: UTXO db is consensus-critical.
3702015-10-22T11:15:06 <sipa> very aware
3712015-10-22T11:15:12 <wumpus> sqlite is extremely easy to embed in a project, it's just one C file
3722015-10-22T11:15:27 <jgarzik> kv datastores off the top of my head: leveldb, gdbm, [n]dbm, tokyo cabinet, kyoto cabinet, berkeley db
3732015-10-22T11:15:35 *** Thireus has quit IRC
3742015-10-22T11:15:47 <jgarzik> wumpus, er huh?
3752015-10-22T11:15:52 <Luke-Jr> wumpus: surely one *big* C file. all of SQL can't be trivial.
3762015-10-22T11:15:53 *** Thireus has joined #bitcoin-core-dev
3772015-10-22T11:15:58 <jgarzik> wumpus, sqlite is quite bigger than one C file
3782015-10-22T11:16:05 <wumpus> hm it used to be at least
3792015-10-22T11:16:55 <jgarzik> berkeley db is well maintained :) :)
3802015-10-22T11:17:51 <Luke-Jr> sqlite is like 140kLOC
3812015-10-22T11:18:20 <jgarzik> for sqlite, if you pre-compile the SQL queries, it goes pretty fast
3822015-10-22T11:18:42 <sipa> of course we'd precompile them...
3832015-10-22T11:19:15 <sipa> i din't think git subtreeing sqlite is reasonable
3842015-10-22T11:20:07 <wumpus> oh this is why I thought it was one C file, yes it's a very large one with everything pasted together: https://www.sqlite.org/amalgamation.html
3852015-10-22T11:20:29 <wumpus> sipa: license-wise?
3862015-10-22T11:20:34 <sipa> size wise
3872015-10-22T11:21:16 <jgarzik> sqlite is public domain, one of the few
3882015-10-22T11:22:07 <sipa> oh, that amalgation is pretty awesome
3892015-10-22T11:22:33 <sipa> we can compile it with various flags for extensions disabled
3902015-10-22T11:22:49 <wumpus> apparantly leveldb is 26kLOC in total
3912015-10-22T11:22:53 <wumpus> sipa: yep
3922015-10-22T11:24:01 <jgarzik> imo LOC and size are secondary factors
3932015-10-22T11:24:20 <jgarzik> primary is on going maintenance, reliability, ...
3942015-10-22T11:24:46 <wumpus> LOC would be important if there is the chance we end up having to maintain/troubleshoot it ourself
3952015-10-22T11:24:55 <wumpus> (like now for leveldb)
3962015-10-22T11:25:00 <jgarzik> indeed
3972015-10-22T11:25:58 <jgarzik> I think pgdb will likely be 10k loc when finished
3982015-10-22T11:27:39 <wumpus> "SQlite as a key-value database" https://www.sqlite.org/cvstrac/wiki?p=KeyValueDatabase
3992015-10-22T11:28:15 <wumpus> looks very easy to do, the only question is indeed jgarzik's whether the TRANSACTION/COMMIT is equivalent to batching as we use it now
4002015-10-22T11:29:06 <jgarzik> another sql caveat - based on experience - you have to be careful that your 'strings' are not translated in any way by the engine via unicode etc.
4012015-10-22T11:29:17 <Luke-Jr> jgarzik: nobody is going to maintain consensus-critical behaviour
4022015-10-22T11:29:20 <wumpus> sql injections? *ducks*
4032015-10-22T11:29:23 <jgarzik> I used sqlite to store binary DNS records
4042015-10-22T11:29:27 <sipa> sqlite is fully transactional
4052015-10-22T11:29:28 <Luke-Jr> no matter what we use, we will need to maintain it
4062015-10-22T11:30:05 <Luke-Jr> even if that just means reviewing each upstream release
4072015-10-22T11:30:20 <sipa> or just reviewing their test practices
4082015-10-22T11:30:32 <jgarzik> sqlite is very well hammered
4092015-10-22T11:30:32 <wumpus> (no, sqlite has proper parametrized queries, and from my experience its BLOBs are binary clean)
4102015-10-22T11:30:41 <jgarzik> few maintain software to consensus critical standards though
4112015-10-22T11:31:40 <sipa> agree, but a database interface clearly should... it's perfectly specified in both dirextion: it should find every record that exist, and not find any reorc that doesn't exist
4122015-10-22T11:31:48 <jgarzik> another bit - make sure there are no limitations on rows-modified-at-once
4132015-10-22T11:31:50 <sipa> so anything that is nkt consensus compatible is a bug
4142015-10-22T11:31:56 <jgarzik> otherwise we re-introduce the BDB fork issue (locks)
4152015-10-22T11:32:20 <Luke-Jr> sipa: fixing bugs is a consensus compatibility bug!
4162015-10-22T11:32:49 <sipa> jgarzik: the "bug" in the bdb case was that we didn't chefk tue bdb return value, and regarded failure to write as a block validation failure
4172015-10-22T11:33:05 <sipa> Luke-Jr: fully agree there
4182015-10-22T11:33:08 <Luke-Jr> anyhow, I need to go to sleep, but I think we will regret it if we import sqlite to consensus-critical code.
4192015-10-22T11:33:27 <jgarzik> sipa, well there was an issue where we were hitting BDB max lock
4202015-10-22T11:33:54 <jgarzik> sipa, the point still stands - sql engines do somethings have max-rows-updated-in-one-transaction type limits
4212015-10-22T11:33:55 <sipa> jgarzik: sure, but that wouldn't have been a consensus failure threat if we didn't treat out-of-locks as invalid-block
4222015-10-22T11:34:10 <jgarzik> *sometimes
4232015-10-22T11:34:39 <sipa> jgarzik: it was us who turned an administrative restriction into a consensus problem
4242015-10-22T11:34:48 <jgarzik> so, sql transaction limits is another one for the eval list
4252015-10-22T11:35:09 <sipa> if we would just have gone "oops! can't write! quitting", like we do for out of disk, bdb would not have caused forks
4262015-10-22T11:35:16 <wumpus> right, if it would treat database errors as a fatal error instead of rejection, it'd wouldn't have been as bad
4272015-10-22T11:35:17 <jgarzik> nod
4282015-10-22T11:35:20 <sipa> it would have been a bad DoS attack, though
4292015-10-22T11:35:27 <jgarzik> yep
4302015-10-22T11:35:35 <sipa> but not a fork
4312015-10-22T11:35:39 <wumpus> but fairly easy to resolve
4322015-10-22T11:35:50 <wumpus> right
4332015-10-22T11:36:28 <sipa> of course, you are absolutely right we need to be aware of such limits in sqlite or whatever we're incestigating
4342015-10-22T11:36:33 <sipa> eh, investigating
4352015-10-22T11:36:38 <jgarzik> lol
4362015-10-22T11:36:52 <sipa> the keys are like right nezt to each other
4372015-10-22T11:37:40 <sipa> also, sqlite databases are single files, right?
4382015-10-22T11:38:26 <jgarzik> yes*
4392015-10-22T11:38:42 <jgarzik> * some temporary files such as journals are created along the way, and must exist for recovery post-crash
4402015-10-22T11:38:57 <sipa> ok, similar to bdb
4412015-10-22T11:39:10 <sipa> that's fine for application database stuff
4422015-10-22T11:39:15 <sipa> maybe not wantes for wallets
4432015-10-22T11:39:19 <sipa> *wanted
4442015-10-22T11:40:52 <jgarzik> sqlite files do want periodic maintenance via 'VACUUM'
4452015-10-22T11:41:28 <sipa> we can do that in our flushtodisk function
4462015-10-22T11:41:30 <jgarzik> they get fragmented, old records cluttered, performance degrades a bit over time
4472015-10-22T11:41:46 <jgarzik> it is a very, very heavyweight operation
4482015-10-22T11:42:16 <jgarzik> info: https://sqlite.org/lang_vacuum.html
4492015-10-22T11:42:53 <jgarzik> auto_vacuum handles large deletes, but not fragments over time
4502015-10-22T11:43:15 <jgarzik> could just run it at startup and then auto_vacuum
4512015-10-22T11:43:28 <sipa> ok
4522015-10-22T11:56:54 <btcdrak> I've not seen everyone get so excited about something for a long time.
4532015-10-22T11:58:54 *** Thireus has quit IRC
4542015-10-22T12:01:41 <wumpus> "The author disclaims copyright to this source code. In place ofa legal notice, here is a blessing: May you do good and not evil. May you find forgiveness for yourself and forgive others. May you share freely, never taking more than you give." I love sqlite's appraoch to licensing
4552015-10-22T12:11:20 <btcdrak> wumpus: if only there were more like that
4562015-10-22T13:06:08 <jgarzik> At https://www.sqlite.org/cvstrac/wiki?p=KeyValueDatabase sqlite seems across the board worse... except for large numbers of records (our use case)
4572015-10-22T13:06:48 <jgarzik> I'm happy to create an sqlite branch for bitcoind, if nobody else is working on it
4582015-10-22T13:07:50 *** pigeons has quit IRC
4592015-10-22T13:09:03 *** pigeons has joined #bitcoin-core-dev
4602015-10-22T13:09:26 *** pigeons is now known as Guest89666
4612015-10-22T13:10:55 <btcdrak> jgarzik: go for it
4622015-10-22T13:13:47 <sipa> how mature is sqlite4?
4632015-10-22T13:14:05 <jgarzik> good question
4642015-10-22T13:14:13 <jgarzik> vast majority in field is 3
4652015-10-22T13:14:33 <sipa> its design seems much closer to leveldb
4662015-10-22T13:14:47 <sipa> as it is... a key/value store internally
4672015-10-22T13:15:02 <sipa> i wonder if there is an api to access that key/value store interface
4682015-10-22T13:15:10 <sipa> i have read before that you can
4692015-10-22T13:15:22 <wumpus> I was starting on it, but will happily leave it to you jgarzik
4702015-10-22T13:15:53 <sipa> The default built-in storage engine is a log-structured merge database. It is very fast, faster than LevelDB, supports nested transactions, and stores all content in a single disk file.
4712015-10-22T13:16:01 <sipa> ^ from the sqlite4 design
4722015-10-22T13:16:26 <wumpus> where do you seea nything about sqlite4? at least the download page only has 3
4732015-10-22T13:16:54 <sipa> ok, sqlite4 is far from released
4742015-10-22T13:17:02 <wumpus> oh, "trunk" :-)
4752015-10-22T13:17:06 <sipa> not an option at this point
4762015-10-22T13:17:25 <wumpus> sounds great though
4772015-10-22T13:17:59 <sipa> alsoz sqlite3 does not promise backward write compatibility... so it is like bdb, and we need to guarantee only increases in version
4782015-10-22T13:18:26 <wumpus> it does promise backward read compatibility?
4792015-10-22T13:18:29 <sipa> yes
4802015-10-22T13:19:07 <wumpus> well then it's not too bad, if it detects a downgrade, it could export|import the database
4812015-10-22T13:19:39 <sipa> wait
4822015-10-22T13:19:43 <wumpus> berkeleydb doesn't even guarantee that much
4832015-10-22T13:19:56 <sipa> databases created by 3.3 can't be read by earlier 3.x versions
4842015-10-22T13:20:27 <sipa> lol, and this caused so much problems they reverted to the old format by default in 3.3.6
4852015-10-22T13:21:24 <jgarzik> note - single file db implies large file support req
4862015-10-22T13:22:11 <sipa> is that not available on every platform these days?
4872015-10-22T13:22:13 <wumpus> is that unreasonable these days?
4882015-10-22T13:22:50 <jgarzik> every platform, but some filesystems may be e.g. fat32 for weird reasons
4892015-10-22T13:22:57 <wumpus> ... and has been for 10 years or so
4902015-10-22T13:23:02 <jgarzik> I think it's reasonable
4912015-10-22T13:23:03 <sipa> oh: a database created by version X will always be read/write compatible with version X, even if modified by later versions
4922015-10-22T13:23:06 <jgarzik> just noting
4932015-10-22T13:23:23 <wumpus> fat32 is pretty much dead
4942015-10-22T13:23:27 <sipa> so it's not an auto-update to whatever the format in the later version is, like bdb
4952015-10-22T13:23:27 <jgarzik> sipa, yeah, kernel filesystem rules
4962015-10-22T13:23:33 <wumpus> it also doesn't support bigger disks
4972015-10-22T13:23:39 <jgarzik> wumpus, sadly very much alive on USB sticks
4982015-10-22T13:24:04 <sipa> put bitcoind'd chainstate on a USB stick: you are eaten by a grue
4992015-10-22T13:24:20 <wumpus> *small* USB sticks
5002015-10-22T13:24:40 <sipa> also: for now the chainstate still fits in 2 GB...
5012015-10-22T13:24:44 <sipa> ... for now
5022015-10-22T13:25:30 <jgarzik> as I said... just making a note because it's something of which people should be aware. continuing to hack on it :)
5032015-10-22T13:25:45 <sipa> awesome, thanks
5042015-10-22T13:25:50 <wumpus> great
5052015-10-22T13:27:53 <wumpus> 32GB is pretty much the limit for FAT32 devices, it's possible to have larger volumes, but it becomes really ineffiecient and at least windows doesn't allow formatting them
5062015-10-22T14:41:11 *** lightningbot has joined #bitcoin-core-dev
5072015-10-22T14:44:22 *** Greg__ has joined #bitcoin-core-dev
5082015-10-22T14:53:03 *** malte has joined #bitcoin-core-dev
5092015-10-22T14:54:15 <GitHub122> [bitcoin] MarcoFalke closed pull request #6866: [trivial] fix white space in rpc help messages (master...MarcoFalke-2015-rpcWhitespace) https://github.com/bitcoin/bitcoin/pull/6866
5102015-10-22T14:54:32 <morcos> sipa: i'm trying to think about how to preserve the state of the mempool while generating block templates
5112015-10-22T14:59:14 *** lightningbot` has joined #bitcoin-core-dev
5122015-10-22T15:00:42 *** cfields has joined #bitcoin-core-dev
5132015-10-22T15:00:58 *** kanzure_ has joined #bitcoin-core-dev
5142015-10-22T15:01:14 *** Eliel_ has joined #bitcoin-core-dev
5152015-10-22T15:02:43 <sipa> morcos: how about being able to mark mempool txn as locked?
5162015-10-22T15:03:03 <sipa> individual transactions, that is
5172015-10-22T15:03:20 <morcos> sipa: how do you mean? the block template generation code needs to mark all of them as locked?
5182015-10-22T15:03:49 <morcos> or do you mean if you need to delete something, such as with RBF, then you mark it as deleted and go ahead and add the new one?
5192015-10-22T15:04:14 *** __Greg has joined #bitcoin-core-dev
5202015-10-22T15:04:19 <sipa> morcos: while the template generator runs, it locks transactions one by one as they are added to the template
5212015-10-22T15:04:40 *** teward- has joined #bitcoin-core-dev
5222015-10-22T15:04:49 *** banana_lotus has joined #bitcoin-core-dev
5232015-10-22T15:04:54 <sipa> so the next transactions it fetches are guaranteed to be not conflicting with the ones it already has
5242015-10-22T15:05:30 <sipa> you want a new incoming block to override that though, and cancel the generator to make it start over
5252015-10-22T15:05:47 <morcos> yes agreed with that last part, i'll tackle that later
5262015-10-22T15:06:16 *** guruvan- has joined #bitcoin-core-dev
5272015-10-22T15:06:29 <morcos> but i'm not sure how to just lock txs one by one, you'd have to lock the mempool.cs each time you went to look up a tx to decide to add it to the mempool
5282015-10-22T15:06:42 <morcos> i'm starting off by trying to modify the existing CNB to hold locks a lot less
5292015-10-22T15:06:46 *** lightningbot has quit IRC
5302015-10-22T15:06:46 *** PRab has quit IRC
5312015-10-22T15:06:46 *** Greg__ has quit IRC
5322015-10-22T15:06:47 *** molly has quit IRC
5332015-10-22T15:06:47 *** teward has quit IRC
5342015-10-22T15:06:47 *** kanzure has quit IRC
5352015-10-22T15:06:47 *** guruvan has quit IRC
5362015-10-22T15:06:47 *** BananaLotus has quit IRC
5372015-10-22T15:06:48 *** cfields_ has quit IRC
5382015-10-22T15:06:48 *** Luke-Jr has quit IRC
5392015-10-22T15:06:48 *** murr4y has quit IRC
5402015-10-22T15:06:49 *** Eliel has quit IRC
5412015-10-22T15:06:53 *** teward- is now known as teward
5422015-10-22T15:06:53 *** banana_lotus is now known as BananaLotus
5432015-10-22T15:06:53 *** guruvan- is now known as guruvan
5442015-10-22T15:07:10 <morcos> so i'm imagning it basically copies ptrs to all the txs in the mempool and the score it needs for them
5452015-10-22T15:07:31 <morcos> then it doesn't need to hold mempool.cs any more as long as the ptr does not become invalid
5462015-10-22T15:07:53 <morcos> which would only happne if the txn is deleted
5472015-10-22T15:08:05 <sipa> i guess transactions would get a refcount
5482015-10-22T15:08:21 <sipa> and the refcount value itself would be protected by mempool.cs
5492015-10-22T15:08:23 <morcos> but i think that has to be for all of the txs as you're running the logic of figuring out which ones need to be in the template
5502015-10-22T15:08:53 <morcos> yeah, but i was imagining it was a singular refcount
5512015-10-22T15:08:55 <sipa> hmm, does the mempool not hold some index sorted by template inclusion preference?
5522015-10-22T15:08:58 <morcos> instead of per tx
5532015-10-22T15:09:44 <morcos> yeah so thats where my approach might not be the best approach b/c the new template generation code will have such an index
5542015-10-22T15:09:49 <sipa> another idea is to remove the storage if txn from the mempool entirely
5552015-10-22T15:10:05 <morcos> but even still you need to iterate far past the size of a block
5562015-10-22T15:10:14 <sipa> you just get a transaction store manager, which you give transactions, and it returns a refcounted pointer
5572015-10-22T15:10:21 <sipa> the mempool stores the pointers
5582015-10-22T15:10:34 <sipa> i guess that's just smart pointers
5592015-10-22T15:11:24 <sipa> so the template generator would grab pointers to the top N (some multiple of a block), increase their refcount, release memlool.cs
5602015-10-22T15:11:40 <sipa> then goes off to build a block, verify if wanted, ...
5612015-10-22T15:11:58 <sipa> and then release the pointers
5622015-10-22T15:12:27 <sipa> the mempool can change during that time... you just know that the set you grabbed at that point while holding the lock is internally consistent
5632015-10-22T15:12:42 <sipa> and consistent with the block is was building on
5642015-10-22T15:12:44 <morcos> yes thats what i'm going for
5652015-10-22T15:12:59 <morcos> i guess it depends on how big the set is compared to the whole mempool
5662015-10-22T15:13:16 <morcos> so my idea was that rather than marking individual txs
5672015-10-22T15:13:18 *** PRab has joined #bitcoin-core-dev
5682015-10-22T15:13:25 <morcos> you just tell the mempool, hey, referring to things, don't delete
5692015-10-22T15:13:42 <morcos> and then occasionally that gets freed, and anything marked for deletion can happen
5702015-10-22T15:13:47 <sipa> hmm
5712015-10-22T15:13:48 <sipa> not sure
5722015-10-22T15:13:52 <morcos> since deletes are rarer and smaller than adds
5732015-10-22T15:13:56 <sipa> sounds more complicated
5742015-10-22T15:14:59 <morcos> ok, maybe i'll explore both
5752015-10-22T15:15:09 <morcos> next question and this might be related to CSV
5762015-10-22T15:15:22 <morcos> really annoying that CheckInputs needs cs_main
5772015-10-22T15:15:34 <morcos> doesn't seem like it should have to?
5782015-10-22T15:16:04 <sipa> so i think it's relative easy this way: turn the storage of txn in the mempool into smart pointers, and you can very cheaply and efficiently ask the mempool for the set of all its transactions
5792015-10-22T15:16:05 <morcos> you're storing the hashblock that its valid at, and that can't change height
5802015-10-22T15:16:32 <sipa> it shouldn't be needed, indeed
5812015-10-22T15:17:06 <morcos> sipa: not that easy, you need stuff from the mempool entries, thats where fees are stored
5822015-10-22T15:17:24 <morcos> so you still have to iterate to figure out which pointers you need and copy the meta information
5832015-10-22T15:17:45 <sipa> ok, so that fee information needs to be inside the smart pointed-to objects
5842015-10-22T15:18:38 <morcos> so all you've done now is taken the multi-index approach and say ehh forget that, lets just have multiple indexes referring to this same set of pointers
5852015-10-22T15:19:16 <sipa> i may be missing something
5862015-10-22T15:19:44 <morcos> i think the only complication from my idea is damn RBF... otherwise you just don't call any eviction code except periodically... and there is no problem running it only periodically
5872015-10-22T15:19:58 <morcos> and you don't even have to track anything
5882015-10-22T15:20:11 *** CodeShark__ has joined #bitcoin-core-dev
5892015-10-22T15:20:17 <sipa> what i'm suggesting is a way to make cheap snapshots of subsets of the mempool at a given time
5902015-10-22T15:20:44 <sipa> those snapshots themselves don't have an index, but can be ordered by one at the time they are created
5912015-10-22T15:20:52 <sipa> isn't that enough?
5922015-10-22T15:21:54 <morcos> yeah maybe, i think i'm just leary of the double indirection now where the multi-index is a multi-index of ptrs, but maybe thats something stupid to worry about
5932015-10-22T15:25:29 <sipa> the transactions inside the mempool already have a dozen or so indirections
5942015-10-22T15:26:29 <sipa> ok, more like 6
5952015-10-22T15:26:35 <sipa> but still!
5962015-10-22T15:27:36 <morcos> so something like a boost::shared_ptr
5972015-10-22T15:28:18 <sipa> yes, that specifically :)
5982015-10-22T15:28:27 <sipa> sorry, i thought it was called smart_ptr :)
5992015-10-22T15:29:03 <sipa> you probably want to wrap CTransaction inside something that also contains its (direct) fee and perhaps other immutable statistics
6002015-10-22T15:29:26 <sipa> and then have sharee_ptrs to those inside the mempool entries
6012015-10-22T15:29:43 <sipa> heh, we could even serialize the transactions inside to reduce memory usage...
6022015-10-22T15:31:47 <morcos> yes, so then it'll get a bit more complicated with ancestor package tracking when you have mutable state like ancestor fee rate that you want to copy to your template generation code, but that was going to be difficult period
6032015-10-22T15:32:22 <sipa> why do you need it in the template generator code?
6042015-10-22T15:32:55 <morcos> thats the sort the logic uses
6052015-10-22T15:33:09 <morcos> yikes
6062015-10-22T15:33:18 <sipa> sort the list of copied pointers according to the sorting criterion you want while creating it (so while holding mempool.cs)
6072015-10-22T15:33:30 <sipa> then release mempool.cs and you can forget it
6082015-10-22T15:33:45 <morcos> ok, but see thats the complicated part that takes a long time
6092015-10-22T15:33:54 <sipa> ah, ok
6102015-10-22T15:34:11 <morcos> the tx thats sorted at the top , really links to a lot of other txs , whose ptrs you have to grab to
6112015-10-22T15:34:20 <sipa> hmm, right
6122015-10-22T15:34:22 <morcos> but then how do you solve the problem of once you include that in your template
6132015-10-22T15:34:27 <morcos> everything elses sort changes
6142015-10-22T15:34:52 <sipa> i wouldn't bother with changing sorts
6152015-10-22T15:36:05 <morcos> well that'll be the magic of the algorithm, it'll probably have to be some heuristic, but its very easy to have expensive chain A-B-C and so C is sorted first, but then cheap C2 is still sorted quite high.
6162015-10-22T15:36:37 <morcos> uh not exactly but close
6172015-10-22T15:36:53 <morcos> but yes, i agree we don't have to be perfect
6182015-10-22T15:37:20 <morcos> ok you've given me some ideas... let me see what i come up with
6192015-10-22T15:40:15 <sipa> great
6202015-10-22T15:40:28 <sipa> feel free to completely disregard my ideas :)
6212015-10-22T15:48:08 <jgarzik> RE locking transactions one-by-one - that is the typical parallelism solution
6222015-10-22T15:48:45 <jgarzik> just need a gatekeeper (core struct lock) for insert/delete/move
6232015-10-22T15:53:27 <GitHub181> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/a09297010e17...2cd020d054d6
6242015-10-22T15:53:27 <GitHub181> bitcoin/master 3cb56f3 Daniel Cousens: *: alias -h for --help
6252015-10-22T15:53:28 <GitHub181> bitcoin/master 2cd020d Wladimir J. van der Laan: Merge pull request #6846...
6262015-10-22T15:53:32 <GitHub119> [bitcoin] laanwj closed pull request #6846: [Trivial] bitcoind: alias -h for -help (master...aliash) https://github.com/bitcoin/bitcoin/pull/6846
6272015-10-22T16:02:42 <GitHub145> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/2cd020d054d6...f2c869aef2e7
6282015-10-22T16:02:42 <GitHub145> bitcoin/master c6824f8 J Ross Nicoll: Add DERSIG transaction test cases...
6292015-10-22T16:02:43 <GitHub145> bitcoin/master f2c869a Wladimir J. van der Laan: Merge pull request #6848...
6302015-10-22T16:02:46 <GitHub77> [bitcoin] laanwj closed pull request #6848: Add DERSIG transaction test cases (master...bip66-tests) https://github.com/bitcoin/bitcoin/pull/6848
6312015-10-22T16:15:38 *** BashCo has quit IRC
6322015-10-22T16:18:01 <gavinandresen> FYI: git HEAD master isn't working for me on OSX. Running make check gets me:
6332015-10-22T16:18:04 <gavinandresen> libc++abi.dylib: terminating with uncaught exception of type boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::lock_error> >: boost: mutex lock failed in pthread_mutex_lock: Invalid argument
6342015-10-22T16:20:09 <gavinandresen> I'm very confused, because git bisect blames a commit that works....
6352015-10-22T16:20:51 <cfields> gavinandresen: sounds like your boost may be built against the wrong stdlib? checking master on osx here now.
6362015-10-22T16:21:25 <gavinandresen> cfields: I thought that too, did a brew uninstall / brew install of boost
6372015-10-22T16:25:22 <btcdrak> morcos: sorry was afk. My other question was how does mempool evictions play into the mix? I assume one doesnt want to evict a txn that you're trying to mine.
6382015-10-22T16:26:21 <morcos> btcdrak: yes, thats what my concern was. the only way a pointer to a tx becomes invalid now is if the tx is deleted. A) throught being mined, in which case you want to abort block template generation anyway or B) through eviction or RBF
6392015-10-22T16:27:27 <morcos> for B1) eviction, i thought it would just be easy to just skip eviction if the template_tx_lock is held, and just make sure eviction gets to run every so often.
6402015-10-22T16:28:06 <morcos> but B2) RBF, doesn't have an ideal solution, i wanted to just flag those txs, but maybe sipas idea with shared_ptrs is cleaner
6412015-10-22T16:34:54 <cfields> gavinandresen: ok here. What compiler/boost/osx version?
6422015-10-22T16:34:56 *** BashCo has joined #bitcoin-core-dev
6432015-10-22T16:45:42 <gavinandresen> cfields: Apple LLVM version 7.0.0 (clang-700.0.72), boost /usr/local/Cellar/boost/1.58.0, osx 10.10.5 ....
6442015-10-22T16:46:15 <gavinandresen> ... but it is very possible my machine is in some weird state, I upgraded my XCode yesterday.
6452015-10-22T16:47:23 <btcdrak> gavinandresen: first rule of fight club - "never upgrade XCode"
6462015-10-22T16:56:35 <jgarzik> heh
6472015-10-22T16:56:39 <jgarzik> just did last night...
6482015-10-22T16:59:21 <cfields> gmaxwell / jgarzik: ping. I must be going crazy. From what I can tell experimentally (and backed by man pages), on Linux, our incoming connections are being treated as blocking.
6492015-10-22T16:59:49 <jgarzik> cfields, post-accept(2) you set sockets to non-blocking
6502015-10-22T16:59:56 <cfields> because O_NONBLOCK isn't inherited by accept
6512015-10-22T17:00:01 <jgarzik> nod
6522015-10-22T17:00:05 <cfields> jgarzik: should, sure
6532015-10-22T17:00:08 <cfields> but we don't
6542015-10-22T17:00:13 <jgarzik> oops
6552015-10-22T17:00:44 <jgarzik> seems surprising, seems like it would have been well noticed before now
6562015-10-22T17:01:03 <sipa> that would imply that the "opportunistic write" always succeeds
6572015-10-22T17:01:09 <cfields> for the most part it should be ok because reads wait for select(), but i'm surprised it hasn't caused issues
6582015-10-22T17:01:13 <sipa> that should be trivial to test?
6592015-10-22T17:01:15 *** rubensayshi has quit IRC
6602015-10-22T17:01:45 <jgarzik> no wait - send/recv have per-call NB flags
6612015-10-22T17:02:53 <cfields> jgarzik: ah right
6622015-10-22T17:03:20 <cfields> jgarzik: still though, i've read about select() acting funny on linux with blocking sockets
6632015-10-22T17:04:00 <jgarzik> like haha, that's a laugh funny?
6642015-10-22T17:05:31 * jgarzik attempts to implement db iteration without brute force 'select * from table' into RAM ;p
6652015-10-22T17:06:10 <cfields> "Under Linux, select() may report a socket file descriptor as "ready for reading", while nevertheless a subsequent read blocks... Thus it may be safer to use O_NONBLOCK on sockets that should not block."
6662015-10-22T17:06:44 <jgarzik> recv() will fail in that case
6672015-10-22T17:06:50 <cfields> Either way, forcing them to nonblock after accept can't hurt, right?
6682015-10-22T17:06:57 <cfields> hmm
6692015-10-22T17:07:19 <sipa> jgarzik: i guess you can get some sort of object/state which you can ask for "more results" ?
6702015-10-22T17:07:21 <cfields> it wouldn't just block?
6712015-10-22T17:08:02 <GitHub132> [bitcoin] MarcoFalke opened pull request #6870: [trivial] Misc cleanup and translations (master...MarcoFalke-2015-trivial3) https://github.com/bitcoin/bitcoin/pull/6870
6722015-10-22T17:08:10 <cfields> jgarzik: er, the DONTWAIT. nm.
6732015-10-22T17:12:50 <jgarzik> sipa, you can 'select *' and iterate in one direction (forwards; next-result), whereas the CDBIterator wants Next, Prev and Seek operations
6742015-10-22T17:13:05 * jgarzik is studying the callers/users of CDBIterator now
6752015-10-22T17:13:27 <sipa> jgarzik: i don't think we use prev
6762015-10-22T17:15:47 <jgarzik> so it seems
6772015-10-22T17:16:44 <sipa> and seek would be equivalent to just starting the search
6782015-10-22T17:16:51 <cfields> gavinandresen: that's no good. i'm confused about that error, though
6792015-10-22T17:17:19 <jgarzik> sipa, well seek-start, seek-end and seek-key are all different
6802015-10-22T17:17:27 <sipa> jgarzik: i think the abstraction can be changed so we have a few 'entry types' (which would translate to separate sqlite tables)
6812015-10-22T17:17:42 <sipa> jgarzik: and only have iterators for all-data-within-one-entry-type
6822015-10-22T17:17:54 <jgarzik> sure
6832015-10-22T17:18:04 <sipa> which would be compatible with both leveldb and sqlite
6842015-10-22T17:21:00 <jgarzik> ah a few of these db ops are like Prev(), unused. job easier++
6852015-10-22T17:21:07 *** murr4y has joined #bitcoin-core-dev
6862015-10-22T17:24:35 <jgarzik> hrm
6872015-10-22T17:24:41 <jgarzik> sipa, lex order?
6882015-10-22T17:25:43 <jgarzik> seems so
6892015-10-22T17:43:09 <sipa> some things need lex order, yes
6902015-10-22T18:11:51 *** Greg__ has joined #bitcoin-core-dev
6912015-10-22T18:12:12 *** __Greg has quit IRC
6922015-10-22T18:28:10 <gavinandresen> FYI: my build-failing-on-OSX problem went away with a 'git clean -dxf' then re-autogen/configure/build....
6932015-10-22T18:35:35 *** belcher has joined #bitcoin-core-dev
6942015-10-22T18:36:31 * jonasschnelli did accidentally sent untracked files to nirvana serval times with `git clean -dxf`...
6952015-10-22T18:36:59 * sipa too
6962015-10-22T18:37:05 <jonasschnelli> heh
6972015-10-22T18:37:07 *** Thireus has quit IRC
6982015-10-22T18:39:04 *** d_t has joined #bitcoin-core-dev
6992015-10-22T18:41:36 *** Thireus has joined #bitcoin-core-dev
7002015-10-22T18:51:17 *** jl2012 has joined #bitcoin-core-dev
7012015-10-22T18:58:44 <gmaxwell> I haven't tried it on bitcoin core yet, but the latest functionality in rr (replay debugger) looks pretty great: http://robert.ocallahan.org/2015/10/rr-40-released-with-reverse-execution.html
7022015-10-22T19:02:58 <jcorgan> did i hear rumblings about switching to sqlite?
7032015-10-22T19:03:10 <maaku> i wish there was a 'git stage --clean' command
7042015-10-22T19:03:23 <btcdrak> meeting over at #bitcoin-dev now
7052015-10-22T19:03:33 <jcorgan> isn't it git checkout . ?
7062015-10-22T19:03:41 <btcdrak> jcorgan: yes, jgarzik is working on it now
7072015-10-22T19:04:01 <jcorgan> oh sweet. then we can use sqlcipher and get full db ondisk encryption
7082015-10-22T19:04:43 <jcorgan> it's a drop-in replacement/fork for sqlite that operates at the db page level, like dmcrypte for block devices
7092015-10-22T19:05:13 <sipa> why do you need encryption for public data? :)
7102015-10-22T19:05:44 <maaku> sipa: i presume he's talking about wallet?
7112015-10-22T19:05:53 <sipa> i wouldn't use sqlite for the wallet
7122015-10-22T19:05:58 <gmaxwell> jcorgan: you saw that discussion, but I expect it won't happen (At least with sqllite3) as-- if I'm not mistaken-- btcd tried it and found the performance unacceptable. (but we should anyways, both to have a comparison point, and because sqllite4 might be faster even if 3 is too slow)
7132015-10-22T19:06:26 <jcorgan> encrypt ALL THE THINGS
7142015-10-22T19:07:36 <gavinandresen> RE: switching to sqlite: this CACM article was very interesting: http://cacm.acm.org/magazines/2015/10/192379-crash-consistency/abstract
7152015-10-22T19:07:53 <gavinandresen> ... bashes on leveldb a bit, praises sqlite ....
7162015-10-22T19:08:09 * maaku is very confused as to what the benefit of switching to sqlite would be for this particular use case
7172015-10-22T19:08:43 <jgarzik> maaku: it's maintained and reliable
7182015-10-22T19:09:10 <gavinandresen> maaku: assuming sqlite is better about crash consistency (and from all I've read, it is), would cut down on the "core sucks, because my PC crashed and then eleven days to reindex..."
7192015-10-22T19:10:21 <sipa> yes, that's the only reason
7202015-10-22T19:10:26 <sipa> sqlite is known to be rock solid
7212015-10-22T19:10:37 <sipa> however, performance would need to be acceptable
7222015-10-22T19:14:35 <maaku> sipa: well, also interop and the ease of using a different sql backend
7232015-10-22T19:16:57 <jcorgan> i would definitely advocate making a sql abstraction layer with pluggable back ends, with sqlite as the default
7242015-10-22T19:17:26 <CodeShark__> I'm using ODB in my stack for that
7252015-10-22T19:17:48 <sipa> jcorgan: to be clear, i think SQL is a downside
7262015-10-22T19:18:06 <sipa> we're not looking for a database, and turning things into a database is against what we're trying to achieve
7272015-10-22T19:18:20 <jgarzik> nod
7282015-10-22T19:18:27 <jcorgan> my secret plan revealed, again
7292015-10-22T19:18:29 <sipa> our database is not designed to be accessible by multiple processes
7302015-10-22T19:18:39 *** CodeShark has quit IRC
7312015-10-22T19:18:41 <sipa> nor do we need any fancy indexes or queries
7322015-10-22T19:18:42 *** CodeShark__ has quit IRC
7332015-10-22T19:18:49 <gavinandresen> if there's a key-val store as well-supported and rock solid as sqlite..... that's the right answer.
7342015-10-22T19:18:51 <jgarzik> I still like the custom db approach, but I'm highly biased
7352015-10-22T19:18:52 *** CodeShark has joined #bitcoin-core-dev
7362015-10-22T19:19:03 <sipa> if sqlite offered a mechanism to just use its low-level key-value store, i would
7372015-10-22T19:19:15 <sipa> (because it's a database layer built on top of a key-value store)
7382015-10-22T19:19:46 <jgarzik> program in sqlite VM assembler and you can...
7392015-10-22T19:19:48 * jgarzik runs
7402015-10-22T19:19:50 <maaku> imho the answer would be to switch to a better maintained descendent of leveldb (there are many)
7412015-10-22T19:20:14 <maaku> also minimizes risk of fork
7422015-10-22T19:20:18 <gmaxwell> Its unhelpful to have this conversation in parallel with the meeting.
7432015-10-22T19:21:48 *** bsm1175321 has joined #bitcoin-core-dev
7442015-10-22T19:24:22 *** luke-jr_ is now known as Luke-Jr
7452015-10-22T19:26:36 <kanzure_> upstream maintainers, apparently
7462015-10-22T19:26:41 <kanzure_> oops, bad scrollback
7472015-10-22T19:27:53 *** skyraider has joined #bitcoin-core-dev
7482015-10-22T19:30:24 *** crescendo has joined #bitcoin-core-dev
7492015-10-22T19:32:36 <bsm1175321> Don't want to interject in the meeting, but I re-ran the SQLite vs. libdb benchmarks sipa mentioned (because they were 6 years old).
7502015-10-22T19:32:40 <bsm1175321> Results here: https://gist.github.com/mcelrath/6952eab246a7c705a0fb
7512015-10-22T19:35:47 <kanzure_> huh? why should that not be mentioned?
7522015-10-22T19:36:06 <bsm1175321> Because the topic has moved on, and the conclusion was to continue research.
7532015-10-22T19:39:22 *** kanzure_ is now known as kanzure
7542015-10-22T19:44:38 <bsm1175321> Above link updated with 500k records, transactional DB. (Took a few minutes to run)
7552015-10-22T19:47:48 *** MarcoFalke has joined #bitcoin-core-dev
7562015-10-22T20:03:28 <BlueMatt> IIRC gmaxwell was gonna ask the crash consistency guys for their test harness?
7572015-10-22T20:04:04 <BlueMatt> so we could test other dbs
7582015-10-22T20:04:09 <BlueMatt> or maybe I'm crazy
7592015-10-22T20:04:14 <BlueMatt> someone should do that, though
7602015-10-22T20:04:18 <BlueMatt> before we decide
7612015-10-22T20:06:27 <sipa> maaku: sqlite is better at large writes than leveldb
7622015-10-22T20:06:27 <maaku> btcdrak: sqlite4 maybe matches leveldb ... because it's basically using leveldb's datastructure under the hood
7632015-10-22T20:06:36 <sipa> as leveldb writes everything twice tovdisk
7642015-10-22T20:08:20 <btcdrak> gmaxwell: this was the discussion earlier today about sqlite: https://botbot.me/freenode/bitcoin-core-dev/2015-10-22/?msg=52492157&page=2
7652015-10-22T20:08:45 <gmaxwell> btcdrak: yes I read it. I am confident from expirence that its incorrect, but I still think jeff's work is very useful.
7662015-10-22T20:09:33 <btcdrak> gmaxwell: yeah, I was also surprised, but fingers crossed something impressive comes out of jeff's research.
7672015-10-22T20:09:37 <gmaxwell> (in part because sqllite4 may be faster)
7682015-10-22T20:09:55 <btcdrak> sqlite4 is still in development though isnt it?
7692015-10-22T20:10:03 <gmaxwell> sure, not useful yet.
7702015-10-22T20:10:20 <btcdrak> i wonder how far on they are and if they have a release schedule yet
7712015-10-22T20:10:24 <Luke-Jr> am I missing something? why is this being treated as a non-consensus thing?
7722015-10-22T20:10:47 <sipa> it obviously is consensus critical
7732015-10-22T20:10:54 <sipa> it's not a consensus change though
7742015-10-22T20:11:02 <Luke-Jr> we don't and can't know that.
7752015-10-22T20:11:10 <sipa> recompiling with a different libc is also consensus critical
7762015-10-22T20:11:23 *** jtimon has joined #bitcoin-core-dev
7772015-10-22T20:11:23 <Luke-Jr> replacing 26kLOC to 140kLOC is impossible to be sure on
7782015-10-22T20:11:24 <BlueMatt> Luke-Jr: we should be confident that it is not before anything moves forward
7792015-10-22T20:11:27 <sipa> or running on a new linix kernel
7802015-10-22T20:12:08 <Luke-Jr> BlueMatt: I don't think we can.
7812015-10-22T20:12:13 <helo> i really appreciate the meetings, great idea (btcdrak?)
7822015-10-22T20:12:23 <btcdrak> helo
7832015-10-22T20:12:31 <sipa> we are making certain assumptions from other software we are using
7842015-10-22T20:12:32 <BlueMatt> Luke-Jr: then we're stuck with a shitty db that corrupts randomly and isnt self-consistent (ie not consensus-compatible)
7852015-10-22T20:12:36 <Luke-Jr> best I can see happening, is knowing it's unlikely to have any accidentally-triggered failures
7862015-10-22T20:12:38 <gmaxwell> Luke-Jr: it's not being treated as anything right now, it's just a test.
7872015-10-22T20:13:00 <sipa> that includes compilers, libraries, and operation systems with _well specified_ behaviour
7882015-10-22T20:13:03 <gmaxwell> Luke-Jr: by that basis we don't know that leveldb is self-consistent, or consistent between windows / unix, etc.
7892015-10-22T20:13:04 <maaku> Other than Windows corruption errors, is this solving any real problems right now?
7902015-10-22T20:13:07 <btcdrak> Luke-Jr: research is a good thing.
7912015-10-22T20:13:14 *** tombtc has joined #bitcoin-core-dev
7922015-10-22T20:13:26 <gmaxwell> maaku: we have durability problems on all platforms, though the windows ones are very severe.
7932015-10-22T20:13:30 <Luke-Jr> btcdrak: yes, if this is just research fine
7942015-10-22T20:14:01 <gmaxwell> The performance of leveldb is also problematic already, (well reported for very large databases), enough that our fat caching layer is very very criticial for performance, and thats causing us other problems.
7952015-10-22T20:14:31 <gmaxwell> maaku: durability problems in this database are incompatible with the survival of a network where puning is widely used.
7962015-10-22T20:14:55 <sipa> puny network
7972015-10-22T20:15:02 <gmaxwell> hah
7982015-10-22T20:15:05 <gmaxwell> pruning*
7992015-10-22T20:15:10 <btcdrak> LOL
8002015-10-22T20:16:18 <wumpus> in general leveldb works fine; greatest concern is that it seems to be no longer maintained, so any bugs (like corruption on crashes in windows) is unlikely to get solved
8012015-10-22T20:16:28 <maaku> i thought gmaxwell was declaring sipa a risk to bitcoin there
8022015-10-22T20:17:36 <gmaxwell> maaku: well that much is obvious.
8032015-10-22T20:17:37 <sipa> we couls try to find another win env for leveldb
8042015-10-22T20:17:37 <sipa> there may be bugs in that
8052015-10-22T20:18:42 <jonasschnelli> i know this sounds hard: but did we never consider to stop supporting windows? Would this be so wrong?
8062015-10-22T20:18:45 <wumpus> I'm sure it is possible to fix the windows issue, but it's not really nice to inherit a dependency and suddenly have to maintain it
8072015-10-22T20:19:31 <Luke-Jr> jonasschnelli: I think it would be problematic to do so right now.
8082015-10-22T20:19:40 <Luke-Jr> jonasschnelli: as bad as it is, many people still use it
8092015-10-22T20:19:44 <gmaxwell> jonasschnelli: I think it would be a bad idea, its a very widely used platform and dropping it would be a major move against the ability of people to independnantly run, audit, etc. the bitcoin system.
8102015-10-22T20:19:48 <wumpus> jonasschnelli: well I've been close to declaring that. it wouldn't be pretty but we have zero active developers for windows.
8112015-10-22T20:20:00 <gmaxwell> The security problems of windows stink, but hardware wallets are a tidy tool to address that.
8122015-10-22T20:20:22 <wumpus> jonasschnelli: then again, the software is working for a lot of people also on windows
8132015-10-22T20:20:24 <jonasschnelli> not even speaking for the wallet. Just the node/core itself.
8142015-10-22T20:20:37 <gmaxwell> we are in a state right now where it is is almost constructively non-supported by our failure to do a a great job about it.
8152015-10-22T20:20:38 <maaku> well to be clear, gmaxwell's concerns about a pruned network wouldn't apply if the durability issues were wholly isolated to the windows platform
8162015-10-22T20:20:39 <Luke-Jr> jonasschnelli: then people will just stop running a node :/
8172015-10-22T20:21:01 <Luke-Jr> before we drop Windows support, we would ideally want the alternative to be "run a node on Linux", not "stop running a node altogether"
8182015-10-22T20:21:01 <wumpus> gmaxwell: I disagree
8192015-10-22T20:21:04 <gmaxwell> maaku: they are not wholly isolated to windows, though perhaps its safe enough elsewhere that my concern is less of an issue.
8202015-10-22T20:21:05 <jonasschnelli> wumpus: Agree. It's running. But i think we should not move away form leveldb because of windows...
8212015-10-22T20:21:13 <wumpus> gmaxwell: many people are running it on windows, and not complaining at all
8222015-10-22T20:21:30 <wumpus> gmaxwell: no need to blow this issue out of proportion
8232015-10-22T20:21:34 <gmaxwell> wumpus: and many people have stopped running it too.
8242015-10-22T20:21:39 <wumpus> gmaxwell: obviously
8252015-10-22T20:22:00 <gmaxwell> I believe more people have stopped running bitcoin core on windows than currently run it.
8262015-10-22T20:22:13 <gmaxwell> (stopped due to corruption issues of varrious forms)
8272015-10-22T20:22:36 <wumpus> well this is an open source project - if we don't have developers supporting windows, we can't support windows
8282015-10-22T20:22:41 <jonasschnelli> are most corruption issues caused by leveldbs layer?
8292015-10-22T20:22:43 <gmaxwell> (not just leveldb but also AV)
8302015-10-22T20:22:51 <sipa> jonasschnelli: we don't know
8312015-10-22T20:23:19 <gmaxwell> We know there is leveldb corruption on unclean shutdown, and we know about antivirus mediated corruption. But there may be more things we don't know about.
8322015-10-22T20:23:30 <wumpus> if the corruption issues are as bad as you say they are, and no one is fixing them, then we should indeed drop windows support
8332015-10-22T20:23:35 <Luke-Jr> maybe if we stop supporting Windows "officially", but leave it open, others will step up to do it
8342015-10-22T20:23:35 <wumpus> but I don't believe you
8352015-10-22T20:23:45 <gmaxwell> wumpus: or we should get more windows developers.
8362015-10-22T20:23:51 <wumpus> gmaxwell: well, good luck with that
8372015-10-22T20:23:54 <gmaxwell> Which can be done.
8382015-10-22T20:24:09 <gmaxwell> wumpus: I mean I can _hire_ people for this, but it hasn't been on my radar.
8392015-10-22T20:24:13 <wumpus> in any case, this is not constructive *goes back to sleep*
8402015-10-22T20:24:19 <jonasschnelli> we could ship it together with a vmbox ubuntu. *duck*
8412015-10-22T20:24:25 * Luke-Jr wonders if Diapolo is good enough at C++ that he could transition to more than just GUI if he gets funding
8422015-10-22T20:24:39 * jonasschnelli looks strange at Luke-Jr
8432015-10-22T20:24:45 <gmaxwell> wumpus: :-/ I think you're misunderstanding my perspective, sorry I've communicated poorly.
8442015-10-22T20:24:53 <Luke-Jr> jonasschnelli: got a better suggestion? :P
8452015-10-22T20:25:02 <Luke-Jr> I don't know many Windows developers..
8462015-10-22T20:25:05 <jonasschnelli> Luke-Jr: hah. No. Not really. :)
8472015-10-22T20:25:48 <jonasschnelli> serious: what would be the overhead to run bitcoin-qt/core in a vm on windows?
8482015-10-22T20:25:49 <Luke-Jr> maybe the people who did those libconsensus bindings?
8492015-10-22T20:26:03 <wumpus> gmaxwell: it's not impossible to just solve the windows crash corruption issue
8502015-10-22T20:26:04 <jonasschnelli> 10% CPU loss, 512MB ram overhead?
8512015-10-22T20:26:09 <Luke-Jr> jonasschnelli: I would be surprised if a VM kill reproduced the problem
8522015-10-22T20:26:17 <wumpus> gmaxwell: I even proposed to help with it today if someone can send me a corrupted database
8532015-10-22T20:26:29 <gmaxwell> wumpus: yup. did anyone do that?
8542015-10-22T20:27:36 <wumpus> gmaxwell: no
8552015-10-22T20:27:37 <jonasschnelli> Luke-Jr: the idea would be to bundle bitcoin-qt/core on windows together with vbox and a tiny distro. This would eliminate some ugly platform dependents.
8562015-10-22T20:27:37 <Luke-Jr> gmaxwell: I tried to, but couldn't reproduce it with IBD, and shortly thereafter my USB Armory died entirely. So it seems likely my problem was a bad microSD
8572015-10-22T20:27:58 <Luke-Jr> jonasschnelli: that sounds like a terrible UX
8582015-10-22T20:28:09 <jonasschnelli> Luke-Jr: the UX could still be native (once it's decoupled)
8592015-10-22T20:28:09 <wumpus> Luke-Jr: yes, your problem is likely adifferent one. You're the only one reporting it on linu
8602015-10-22T20:28:57 <wumpus> jonasschnelli: if you go that far to make a bundle you may as well fix the windows version :-) (it would be about as much development work)
8612015-10-22T20:29:24 <gmaxwell> jonasschnelli: doesn't jive really well with the resource costs of running a node.... VM with the constantly expanding storage is no fun. :P plus overheads. :P
8622015-10-22T20:29:44 <Luke-Jr> does UML support Windows? ;)
8632015-10-22T20:29:47 <gmaxwell> Fixing the windows shouldn't be a big deal once we get enough repros and data, similar to OSX.
8642015-10-22T20:29:54 <gmaxwell> sounded like it might be the same issue.
8652015-10-22T20:30:06 <wumpus> yeah, let's just solve the database problem instead of trying to work around it
8662015-10-22T20:30:29 <wumpus> even if we have to put a flush after every line in leveldb ...
8672015-10-22T20:30:34 <gmaxwell> I considered the exploration of other options to be orthorgonal with windows being on fire FWIW.
8682015-10-22T20:30:34 <jonasschnelli> gmaxwell: nowadays VM overhead is tiny... it's not perfect. But better a solution that works with a lost of 10-20% in overhead than unsolved corrupted databases.
8692015-10-22T20:30:58 <wumpus> gmaxwell: sure, I still like the idea of exploring other databases
8702015-10-22T20:31:13 <wumpus> but yes, orthogonally
8712015-10-22T20:31:20 <sipa> i really like the idea of just being able to use a maintained and tested database
8722015-10-22T20:31:23 <gmaxwell> (also the attempt may turn up bugs elsewhere in our codebase)
8732015-10-22T20:31:37 <wumpus> don't we all
8742015-10-22T20:31:43 <sipa> and not a hack we had to pull together
8752015-10-22T20:31:45 * wumpus thought leveldb was that
8762015-10-22T20:31:46 *** paveljanik has quit IRC
8772015-10-22T20:31:51 <gavinandresen> a bug bounty worked for the last leveldb corruption issue we had (if i recall correctly). I'm still holding some bitcoin in the core dev expenses fund
8782015-10-22T20:31:56 <sipa> wumpus: leveldb windows certainly isn't
8792015-10-22T20:32:02 <Luke-Jr> jonasschnelli: if someone really wants to investigate VM stuff, I'd suggest instead a thin OS that runs *Windows* in the VM, and provides a hardwarewallet-like interface on top
8802015-10-22T20:32:04 <jonasschnelli> is there an approach to test sqlites performance which represents our db-style?
8812015-10-22T20:32:19 <sipa> wumpus: we have local modifications to the win env
8822015-10-22T20:32:30 <wumpus> jonasschnelli: best would be to just try it
8832015-10-22T20:32:41 <gmaxwell> jonasschnelli: yes, testing bitcoin core using <alternative database> :P it shouldn't be that much work; presumably jeff will report back on that soon
8842015-10-22T20:33:17 <jonasschnelli> sqlite could initially be slower. But it's better maintained and much more portable.
8852015-10-22T20:33:39 <sipa> jonasschnelli: there is slower and slower
8862015-10-22T20:33:50 <jonasschnelli> Indeed
8872015-10-22T20:33:53 <sipa> if it's slower for things our caches compensates for, who cares
8882015-10-22T20:33:54 <gmaxwell> jonasschnelli: speed is a security consideration for us currently, if you were talking about 5% or something I'd agree. but thats now what I expect.
8892015-10-22T20:34:09 <gmaxwell> s/now/not/
8902015-10-22T20:34:09 <sipa> if it's slower to the point that it affects block propagation, it's a no go
8912015-10-22T20:34:33 <gmaxwell> s/currently/sadly/
8922015-10-22T20:34:35 * wumpus doubts it will make much of a difference
8932015-10-22T20:34:39 <jonasschnelli> Quote from sqlite4: The default built-in storage engine is a log-structured merge database. It is very fast, faster than LevelDB, supports nested transactions, and stores all content in a single disk file. Future versions of SQLite4 might also include a built-in B-Tree storage engine.
8942015-10-22T20:34:57 <jonasschnelli> promising. :)
8952015-10-22T20:35:07 <wumpus> there are so many factors influencing performance of bitcoind, that a slightly slower database won't be a big issue
8962015-10-22T20:35:14 <sipa> yes, sqlite4 sounds awesome _once_ it has had the same amount of testing and battle hardending as sqlite3
8972015-10-22T20:35:20 <wumpus> jonasschnelli: yes that was pasted before today :)
8982015-10-22T20:35:50 <jonasschnelli> ha.. okay. I'd love to play around with it. But my stack of things to work down is just to big right now.
8992015-10-22T20:36:13 <gmaxwell> we might also want to maintain a patch for sqllite4 and use it to try to get the sqllite devs to use us as a test harness. :P
9002015-10-22T20:36:43 <jonasschnelli> sipa: sqlite is widely used, also in almost every browser (local dbs) and smartphone (android and iOS IIRC). They very likely to update once to sqlite4...
9012015-10-22T20:36:45 *** Greg__ has left #bitcoin-core-dev
9022015-10-22T20:38:18 <sipa> but quote from one of the sqlite devs a few months ago "sqlite4 is a dev toy"
9032015-10-22T20:38:53 <jonasschnelli> sipa: that sounds perfect for our bitcoin <1.0 version. :)
9042015-10-22T20:39:18 <sipa> jonasschnelli: if only we didn't have this pesky economy thing that relies on bitcoin
9052015-10-22T20:39:47 <jonasschnelli> sipa: na... that are just some fiat exchange bubbles.. :)
9062015-10-22T20:39:56 <gmaxwell> Lets fix that first then. :P
9072015-10-22T20:40:02 <wumpus> trying it out doesn't hurt, no one is talking about releasing with it...
9082015-10-22T20:40:12 <sipa> wumpus: exactly
9092015-10-22T20:40:14 <sipa> dev toy :p
9102015-10-22T20:40:28 <jonasschnelli> wumpus: +1 ... and a such pr would at least take 1/2 year to get in.
9112015-10-22T20:40:39 <gmaxwell> well as I said, might even be possible to get the sqllite developers to test using us. We're a pretty cool load generator (esp with signatures off)
9122015-10-22T20:41:08 <sipa> gmaxwell: we're boring
9132015-10-22T20:41:20 <sipa> single-threaded access, only bulk writes and small random reads
9142015-10-22T20:48:33 *** tombtc has quit IRC
9152015-10-22T21:10:51 <jcorgan> < Luke-Jr> jonasschnelli: if someone really wants to investigate VM stuff, I'd suggest instead a thin OS that runs *Windows* in the VM
9162015-10-22T21:11:06 <jcorgan> *cough* Qubes OS *cough*
9172015-10-22T21:11:24 <Luke-Jr> jcorgan: Qubes OS is not very thin, and doesn't do GPU passthrough
9182015-10-22T21:12:07 <jcorgan> didn't think about GPU
9192015-10-22T21:12:18 <jcorgan> but thinness is poorly defined
9202015-10-22T21:12:27 <btcdrak> sipa: are the tests in #6816 (versionbits) enough? I was thinking it would be good to have some rpc-tests there as well, generating blocks and running through a couple of scenarios.
9212015-10-22T21:13:11 <btcdrak> or is that overkill?
9222015-10-22T21:13:34 *** jl2012 has quit IRC
9232015-10-22T21:21:12 <CodeShark> btcdrak: I think we should probably do some regtests with the integration, not necessarily over RPC
9242015-10-22T21:21:24 <CodeShark> well...
9252015-10-22T21:21:31 <CodeShark> other than just generating blocks
9262015-10-22T21:21:43 <CodeShark> I guess just calling generate and getblockchaininfo
9272015-10-22T21:26:17 <btcdrak> CodeShark: I mean tests like https://github.com/bitcoin/bitcoin/blob/master/qa/rpc-tests/bipdersig.py etc
9282015-10-22T21:27:29 <jgarzik> It builds: https://github.com/jgarzik/bitcoin/tree/2015_sqlite
9292015-10-22T21:28:07 <btcdrak> jgarzik: That was fast
9302015-10-22T21:28:28 <CodeShark> jgarzik: nicely done :)
9312015-10-22T21:28:40 <sipa> yes, but does it run linyx?
9322015-10-22T21:30:07 <CodeShark> btcdrak: the -blockversion thing will probably not be a good idea anymore using versionbits
9332015-10-22T21:30:35 <CodeShark> it will be better to provide a list of BIPs you don't want to set the bit for
9342015-10-22T21:30:40 <jgarzik> a few bits can be forwarded upstream immediately, like Seek* and Prev() removal
9352015-10-22T21:31:52 <btcdrak> CodeShark: well obviously the bipdersig.py tests are for an ISM sf.
9362015-10-22T21:32:36 <btcdrak> CodeShark: I'm just saying a set of tests like that which generate blocks and simulate a sf rollout but using versionbits protocol.
9372015-10-22T21:32:43 <CodeShark> I suppose we can just use VERSION_HIGH_BITS and VERSION_HIGH_BITS | 0x1 instead of versions 2 and 3 :)
9382015-10-22T21:33:08 <Luke-Jr> CodeShark: does versionbits sanitise the encoding btw? ;)
9392015-10-22T21:33:18 <Luke-Jr> ie, define the first 32-bits to be a big endian number
9402015-10-22T21:33:36 <maaku> Luke-Jr: it's little endian
9412015-10-22T21:33:44 <Luke-Jr> would be nice to fix that at the same time
9422015-10-22T21:33:53 <CodeShark> encoding? versionbits doesn't deal with serialization stuff
9432015-10-22T21:33:58 <btcdrak> CodeShark: it makes the most conclusive tests that it works, and provides some protection against regressions.
9442015-10-22T21:34:11 <maaku> Luke-Jr: why? nothing else is big endian
9452015-10-22T21:34:14 <Luke-Jr> CodeShark: anything consensus-relevant must deal with encoding
9462015-10-22T21:34:27 <maaku> Luke-Jr: he means versionbits doesn't touch serialization
9472015-10-22T21:34:33 <CodeShark> Luke-Jr: versionbits deals with block header data that has already been deserialized
9482015-10-22T21:34:38 <Luke-Jr> maaku: big endian is standard for protocols; and hashes at least are
9492015-10-22T21:34:54 <sipa> Luke-Jr: please
9502015-10-22T21:35:05 <sipa> we're not changing the serialization of block headers
9512015-10-22T21:36:46 <jgarzik> big endian is dead
9522015-10-22T21:37:14 <sipa> no endianness flamewar please
9532015-10-22T21:37:23 <CodeShark> in any case, the serialization format is entirely a separate issue...versionbits will play no part in that either way
9542015-10-22T21:39:01 <Luke-Jr> so the bits are numbered 7 6 5 4 3 2 1 15 14 13 12 11 10 9 8 23 22 21 20 19 18 17 16 31 30 29 28 27 26 25 24 ?
9552015-10-22T21:39:12 <sipa> irrelevant
9562015-10-22T21:39:15 <Luke-Jr> â¦
9572015-10-22T21:39:57 <CodeShark> versionbits deals with ints, uint32_t, etc...
9582015-10-22T21:39:59 <Luke-Jr> it's relevant because if I set the wrong BIP, it won't work.
9592015-10-22T21:40:00 <sipa> the block header serialization defines nVersion as a 32-bit little-endian signed integer
9602015-10-22T21:40:02 *** aj has joined #bitcoin-core-dev
9612015-10-22T21:40:08 <Luke-Jr> wrong bit*
9622015-10-22T21:40:10 <CodeShark> it doesn't care what the underlying representation is for the particular architecture
9632015-10-22T21:40:17 <sipa> versionbits changes the semantics of the nVersion integer
9642015-10-22T21:40:49 <CodeShark> by the time versionbits comes into play, the version field has already been decoded into an int
9652015-10-22T21:41:30 <Luke-Jr> CodeShark: that is an implementation detail. the whole point of version bits is that it is no longer an int, but a bit array
9662015-10-22T21:41:45 <CodeShark> to test a bit, you do (nVersion >> bit) & 0x1
9672015-10-22T21:41:50 <sipa> Luke-Jr: it's still an int; an int in which particular bits are set
9682015-10-22T21:41:54 * Luke-Jr sighs.
9692015-10-22T21:42:02 <sipa> now please stop this silly discussion, we're not redefining how integers work
9702015-10-22T21:42:34 <gmaxwell> but, base-3!!!
9712015-10-22T21:42:57 <Luke-Jr> fine, not worth the time to argue about this stupid design decision.
9722015-10-22T21:43:46 *** sipa has left #bitcoin-core-dev
9732015-10-22T21:45:23 <CodeShark> base-3 would allow us to do yea/nay/abstain :p
9742015-10-22T21:46:34 *** MarcoFalke has quit IRC
9752015-10-22T21:46:52 *** MarcoFalke has joined #bitcoin-core-dev
9762015-10-22T21:52:17 <Luke-Jr> the annoying part will be having to answer questions in a few years why it's misordered such. like I had to do with the getwork data for years
9772015-10-22T21:55:12 <phantomcircuit> jgarzik, you have benchmarks for sqlite?
9782015-10-22T21:55:49 <phantomcircuit> my experience using it in the past has been that it's slow as hell for inserts or any kind of concurrent access
9792015-10-22T21:56:41 <phantomcircuit> probably time would be better spent on building an interface spec for databases
9802015-10-22T21:58:35 <CodeShark> for insertions we could just use a raw file - the blockchain is a linear structure anyhow. then all we would need to do in sqlite is keep track of file positions ;)
9812015-10-22T22:03:05 <phantomcircuit> CodeShark, utxo only
9822015-10-22T22:04:03 <CodeShark> yeah, so we maintain a utxo index
9832015-10-22T22:04:11 <CodeShark> or are you talking about a pruned node?
9842015-10-22T22:04:19 <phantomcircuit> the index would be roughly the same size as the utxo data :P
9852015-10-22T22:04:30 <CodeShark> lol
9862015-10-22T22:05:11 <CodeShark> ok, for a utxo database perhaps it doesn't make so much sense
9872015-10-22T22:05:25 <phantomcircuit> yeah the actual data is really small
9882015-10-22T22:06:47 <gmaxwell> CodeShark: txindex works like you suggest there.
9892015-10-22T22:07:55 <gmaxwell> but for utxo set we get benefits from reducing the working set. saving 20 bytes of scriptpubkey/value in exchange for another random seek to the middle of some huge block file where 2/3rd of the data is not utxo relevant, basically defeates disk caching entirely. :)
9902015-10-22T22:11:59 <GitHub160> [bitcoin] petertodd opened pull request #6871: Full-RBF with opt-out (master...2015-10-rbf-with-opt-out) https://github.com/bitcoin/bitcoin/pull/6871
9912015-10-22T22:22:13 *** skyraider has quit IRC
9922015-10-22T22:33:19 <bsm1175321> G
9932015-10-22T22:51:11 *** bsm1175321 has quit IRC
9942015-10-22T22:51:23 *** bsm1175321 has joined #bitcoin-core-dev
9952015-10-22T22:53:29 <GitHub51> [bitcoin] TheBlueMatt opened pull request #6872: Remove UTXO cache entries when the tx they were added for is removed/does not enter mempool (master...limitucache) https://github.com/bitcoin/bitcoin/pull/6872
9962015-10-22T22:53:52 *** sipa has joined #bitcoin-core-dev
9972015-10-22T23:00:43 *** MarcoFalke has quit IRC
9982015-10-22T23:05:01 *** dcousens has joined #bitcoin-core-dev
9992015-10-22T23:05:19 <dcousens> petertodd: I hope you don't mind my questions :)
10002015-10-22T23:06:19 <phantomcircuit> BlueMatt, the last commit on #6872 seems to evict from the cache things which other transactions in the mempool want cached
10012015-10-22T23:06:36 <phantomcircuit> ie AcceptToMempool fails because it's a double spend of something with a higher feerate
10022015-10-22T23:08:02 <petertodd> dcousens: no worries!
10032015-10-22T23:08:10 <phantomcircuit> scratch that
10042015-10-22T23:08:13 <phantomcircuit> i missed the check
10052015-10-22T23:08:14 <petertodd> dcousens: I'm doing three things at once so the responses are probably sounding a bit terse :)
10062015-10-22T23:08:20 <BlueMatt> phantomcircuit: ahh, ok, good, i was about to say
10072015-10-22T23:08:33 <dcousens> petertodd: all good, I read everything in good spirit, just hope it comes across the same :)
10082015-10-22T23:18:16 *** molly has joined #bitcoin-core-dev
10092015-10-22T23:21:19 *** moli has quit IRC
10102015-10-22T23:40:57 *** sipa has quit IRC
10112015-10-22T23:41:05 *** dcousens has quit IRC
10122015-10-22T23:42:08 *** sipa has joined #bitcoin-core-dev
10132015-10-22T23:56:57 *** sipa has quit IRC
10142015-10-22T23:56:57 *** sipa has joined #bitcoin-core-dev