1 2015-10-22T00:12:54  <Luke-Jr> re sigop-limit flooding, can we just merge acceptnonstdtxn (mainnet option, off by default) finally? this would not be a problem if people were using code with it merged..
   2 2015-10-22T00:15:32  <Luke-Jr> https://github.com/bitcoin/bitcoin/pull/559
   3 2015-10-22T00:16:36  *** PaulCapestany has quit IRC
   4 2015-10-22T00:19:02  <gmaxwell> Luke-Jr: explain your logic?
   5 2015-10-22T00:19:32  <Luke-Jr> gmaxwell: the top commit limits (in policy) tx sigop count based on its size
   6 2015-10-22T00:19:55  <Luke-Jr> so the spam would need to be larger for it to use 15 sigops
   7 2015-10-22T00:20:21  <Luke-Jr> thus hitting the block size limit approximately at the same time as the sigop limit
   8 2015-10-22T00:20:49  <gmaxwell> I ~think~ the suggestion I made of max(size, sigops*1e6/20000) is more general than the limit thing.
   9 2015-10-22T00:20:54  <Luke-Jr> (and triggering higher fees, of course)
  10 2015-10-22T00:21:28  <Luke-Jr> gmaxwell: yes, my point is that the bug wouldn't exist in the first place had this already been merged
  11 2015-10-22T00:21:42  *** jl2012 has quit IRC
  12 2015-10-22T00:23:13  *** PaulCapestany has joined #bitcoin-core-dev
  13 2015-10-22T00:54:19  *** Ylbam has quit IRC
  14 2015-10-22T00:54:27  *** PaulCapestany has quit IRC
  15 2015-10-22T00:55:38  *** PaulCapestany has joined #bitcoin-core-dev
  16 2015-10-22T01:10:07  <BlueMatt> morcos: you asked about tx cache? https://github.com/TheBlueMatt/bitcoin/tree/limitucache
  17 2015-10-22T01:10:27  <BlueMatt> actually, I'll just pr
  18 2015-10-22T01:11:47  <GitHub110> [bitcoin] TheBlueMatt opened pull request #6868: Limit txn cache growth (master...limitucache) https://github.com/bitcoin/bitcoin/pull/6868
  19 2015-10-22T01:13:17  <gmaxwell> BlueMatt: for your don't fork from here-- I suggest https://github.com/gmaxwell/secp256k1
  20 2015-10-22T01:13:50  <BlueMatt> gmaxwell: yes, but that takes effort....
  21 2015-10-22T01:14:44  <gmaxwell> well if someone does it to the repo first you can check out theirs and force push it into yours. :P
  22 2015-10-22T01:14:51  <gmaxwell> I guess I should do bitcoin core too.
  23 2015-10-22T01:16:48  <GitHub26> [bitcoin] TheBlueMatt closed pull request #6868: Limit txn cache growth (master...limitucache) https://github.com/bitcoin/bitcoin/pull/6868
  24 2015-10-22T01:46:28  <morcos> BlueMatt: interesting idea.. i'm not sure what i think of it yet...  you don't thinks also potentially inefficient
  25 2015-10-22T01:48:26  <morcos> did you see the conversation wumpus and I were having earlier
  26 2015-10-22T01:48:52  <morcos> one thing we could do in ATMP, is not pull into the pcoinstip cache coins for txs which aren't actually accepted into the memory pool
  27 2015-10-22T01:56:03  <sipa> yeah, we should do that
  28 2015-10-22T01:56:35  <sipa> i see sometimes huge increases in utxo cache size after a transaction
  29 2015-10-22T01:58:24  <sipa> however, with bith limited utxo cache and mempool, memory usage actually remains *very* low
  30 2015-10-22T01:58:50  <sipa> have a node with 100 MB utxo and 200 MB mempool, and total memory is staying below 570 M
  31 2015-10-22T01:59:08  *** amiller has quit IRC
  32 2015-10-22T02:01:59  <morcos> just don't run GetBlockTemplate! :)
  33 2015-10-22T02:02:17  <morcos> sipa: why does CreateNewBlock need to hold cs_main the whole time
  34 2015-10-22T02:02:35  <sipa> no idea
  35 2015-10-22T02:02:50  <sipa> also no idea why it actually needs to verify the result
  36 2015-10-22T02:03:03  <morcos> it seems like after the first round of calling HaveCoins on all the txins of all the txs.. you now have a view with all of of your coins in it
  37 2015-10-22T02:03:16  <morcos> and you could release cs_main
  38 2015-10-22T02:03:27  <sipa> the mempool could change then, though?
  39 2015-10-22T02:03:35  <morcos> i took a first stab at a background miner, but realized its useless if its holding cs_main
  40 2015-10-22T02:04:39  <morcos> well... what do you mean change?  i think we don't want to lose txs that we might be trying to mine
  41 2015-10-22T02:04:54  <morcos> but maybe you can handle that by not removing things
  42 2015-10-22T02:05:29  <morcos> a smaller lock that just guards actually removing txs
  43 2015-10-22T02:06:13  <morcos> why would it not need to verify the result, you mean because it shouldn't have been able to create something invalid
  44 2015-10-22T02:06:20  <sipa> yes
  45 2015-10-22T02:06:21  <morcos> i do think thats a nice double check
  46 2015-10-22T02:06:33  <sipa> i don't think it's worth the cost
  47 2015-10-22T02:06:41  <sipa> given that it hasn't failed in years
  48 2015-10-22T02:06:54  <morcos> well it will be if we're chaning the mining algorithm!
  49 2015-10-22T02:06:59  <morcos> changing
  50 2015-10-22T02:06:59  <sipa> maybe a spot check would be useful
  51 2015-10-22T02:07:08  <sipa> run it once every 1000 calls
  52 2015-10-22T02:07:13  <jgarzik> since it's in the background the cost is mitigated
  53 2015-10-22T02:07:25  <morcos> holds cs_main
  54 2015-10-22T02:07:31  <sipa> block validation will not be background
  55 2015-10-22T02:07:44  <sipa> it needs the utxo set
  56 2015-10-22T02:08:08  <sipa> though i guess you are right that of it actually does make a full copy of all inlits, it can release the lock afterwards
  57 2015-10-22T02:08:10  <jgarzik> yes it needs utxo set but doesn't need cs_main long term
  58 2015-10-22T02:08:21  <sipa> well it needs the utxo lock, whatever that is
  59 2015-10-22T02:08:28  <morcos> yeah i think we could be smart about it
  60 2015-10-22T02:08:43  <sipa> i think it shouldn't need the utxo set or copies at all
  61 2015-10-22T02:08:49  <sipa> the mempool should be consistent
  62 2015-10-22T02:09:01  <sipa> do a spot check occasionally to make sure the code isn't broken
  63 2015-10-22T02:09:10  <morcos> right now it already pulls the potentially used utxos into caches TWICE.  so you could use one for creation and the other for validation without needing to grab cs_main again
  64 2015-10-22T02:09:17  <sipa> yup
  65 2015-10-22T02:09:26  <sipa> it should be 0, imho
  66 2015-10-22T02:09:45  <morcos> oh
  67 2015-10-22T02:09:52  <morcos> thats what you mean by mempool should be consistent
  68 2015-10-22T02:09:55  *** MarcoFalke has quit IRC
  69 2015-10-22T02:09:55  <sipa> the mempool is known to not double spend
  70 2015-10-22T02:09:56  <morcos> yikes!
  71 2015-10-22T02:10:03  <sipa> not contain invalid transactions
  72 2015-10-22T02:10:04  <morcos> but thats also been broken
  73 2015-10-22T02:10:10  <sipa> so we should fix it
  74 2015-10-22T02:10:32  <morcos> but what i'm saying is that breaks often enough that you wouldn't want it to cause you to mine an invalid block?
  75 2015-10-22T02:10:42  <morcos> at least one check should still exist
  76 2015-10-22T02:10:43  <sipa> that's like saying "we're not sure about the utxo cache, let's run checkblockchain on every update"
  77 2015-10-22T02:10:49  <sipa> sure the check should exist
  78 2015-10-22T02:11:00  <sipa> but it should not be slowing down GBT by seconds!
  79 2015-10-22T02:11:21  <morcos> ok ok
  80 2015-10-22T02:11:29  <sipa> every single call
  81 2015-10-22T02:12:04  <morcos> its just scary when you have a giant block of code protected by cs_main..  trying to reason about what in there really cares about it
  82 2015-10-22T02:12:28  <sipa> well you don't want the best block or the mempool to change underneath it
  83 2015-10-22T02:12:57  <sipa> or you will be pulling in transactions that may conflict with each other
  84 2015-10-22T02:13:00  <Luke-Jr> sipa: GBT verifies the result because it's worth it. No miner wants to find out after the fact they mined an invalid block.
  85 2015-10-22T02:13:01  <morcos> if the best block changes i've got other problems..  i don't want to be calculating this block anyway
  86 2015-10-22T02:13:17  <sipa> Luke-Jr: the code should not produce invalid blocks, period
  87 2015-10-22T02:13:30  <Luke-Jr> also, spot checking just leads to people /expecting/ a lower time than they can rely on
  88 2015-10-22T02:13:38  <sipa> Luke-Jr: if you run the check once every 1000 calls you will equally well detect broken code
  89 2015-10-22T02:13:48  <Luke-Jr> not equally well, no.
  90 2015-10-22T02:13:58  <sipa> if the code is broken, it will fail badly
  91 2015-10-22T02:14:01  <Luke-Jr> and it will mislead people into thinking it's faster than it's necessarily
  92 2015-10-22T02:14:22  <Luke-Jr> an edge case will not necessarily fail badly.
  93 2015-10-22T02:14:40  <sipa> so run with -checkmempool on then
  94 2015-10-22T02:14:50  <Luke-Jr> without the verification, there is no consensus code involved in producing blocks here..
  95 2015-10-22T02:14:54  <Luke-Jr> valid transactions != valid block
  96 2015-10-22T02:14:54  <sipa> then you will also not get an invalid memlool state
  97 2015-10-22T02:15:13  <sipa> just get that check out of the mining path
  98 2015-10-22T02:15:14  *** amiller has joined #bitcoin-core-dev
  99 2015-10-22T02:15:27  <sipa> i wonder how much gbt delay has encourage validationless mining...
 100 2015-10-22T02:15:55  <Luke-Jr> removing the check *is* validationless.
 101 2015-10-22T02:16:00  <morcos> I'm with Luke-Jr that I think consensus code should happen at least once in the path
 102 2015-10-22T02:16:07  <sipa> it is not if the code is not broken
 103 2015-10-22T02:16:08  <morcos> but it doesn't have to happen twice as it does now
 104 2015-10-22T02:16:19  <Luke-Jr> morcos: wait, twice?
 105 2015-10-22T02:16:31  <sipa> once for building, once for checking
 106 2015-10-22T02:16:37  <morcos> CheckInputs on the transactions and TestBlockValidity
 107 2015-10-22T02:16:59  *** amiller has quit IRC
 108 2015-10-22T02:17:17  <Luke-Jr> CheckInputs doesn't check the entire block, but maybe it can be removed
 109 2015-10-22T02:17:40  <sipa> it does check the whole block eventually, as it's called for all transactions that end up in it
 110 2015-10-22T02:18:02  <Luke-Jr> sipa: no, because all the transactions can be valid without the block itself being valid
 111 2015-10-22T02:18:05  <morcos> but i think we're talking about different things here
 112 2015-10-22T02:18:15  <sipa> Luke-Jr: i know that
 113 2015-10-22T02:18:28  <morcos> i'm concerned with having the block-template thread hold cs_main too much...
 114 2015-10-22T02:18:36  <morcos> CheckInputs doesn't need cs_main once you have a view
 115 2015-10-22T02:18:50  <sipa> but checkinputs is called for the equivalent amount of work of validating the whole block
 116 2015-10-22T02:18:51  <morcos> sipa your concern seems to be latency in GBT
 117 2015-10-22T02:18:58  <morcos> but thats solvable in other ways
 118 2015-10-22T02:18:59  <sipa> yes
 119 2015-10-22T02:19:02  <sipa> ok
 120 2015-10-22T02:19:40  <morcos> as long as you think its ok to have brief periods of mining on an empty block or something
 121 2015-10-22T02:19:50  <Luke-Jr> GBT isn't supposed to be time-critical anyway (but I already plan to improve it)
 122 2015-10-22T02:20:01  *** belcher has quit IRC
 123 2015-10-22T02:20:09  <sipa> well it can't take seconds...
 124 2015-10-22T02:20:44  <Luke-Jr> maybe ideally it wouldn't, but it shouldn't hurt much
 125 2015-10-22T02:20:55  <morcos> forget about how long it takes, what you care about is how long after a new block comes in that you have a header you can mine that builds off it right?
 126 2015-10-22T02:21:08  <Luke-Jr> also afaik right now it only takes seconds when the miner is neglecting to keep a clean mempool
 127 2015-10-22T02:21:12  <morcos> it doesn't matter if it then takes seconds to generate a new one with txs right?
 128 2015-10-22T02:21:31  <Luke-Jr> since it needs to go over every tx in the mempool
 129 2015-10-22T02:21:37  <Luke-Jr> not because of the test afterward
 130 2015-10-22T02:21:39  <sipa> morcos: as long as fees are negligablez yes
 131 2015-10-22T02:21:40  <morcos> although seconds is really absurdly slow and we should be able to get it under that even for that case
 132 2015-10-22T02:21:57  <morcos> Luke-Jr: yeah thats one of the problems, is that now it loops all the txs twice
 133 2015-10-22T02:22:01  <morcos> once putting them all in vecPriority
 134 2015-10-22T02:22:15  <morcos> and then trying to go through vecPriority and put them all in a block
 135 2015-10-22T02:22:18  <Luke-Jr> yeah
 136 2015-10-22T02:22:19  <morcos> with a sorted mempool
 137 2015-10-22T02:22:30  <morcos> both become unnecessary (the 2nd was already unnecessary)
 138 2015-10-22T02:22:34  <Luke-Jr> sorted mempool is essentially what I'm working on
 139 2015-10-22T02:22:43  <Luke-Jr> although I need to rebase on top of the packages stuff I expect
 140 2015-10-22T02:22:55  <morcos> sdaftuar is very close to finishing the ancestor package tracking
 141 2015-10-22T02:23:16  <morcos> if you want to build a more general sorting priority score, thats probably what you should build off of
 142 2015-10-22T02:23:26  <Luke-Jr> not priority score
 143 2015-10-22T02:23:36  <Luke-Jr> sorting based on the user-defined policy ;)
 144 2015-10-22T02:23:42  <morcos> yeah thats what i meant
 145 2015-10-22T02:24:04  <morcos> don't you have to somehow turn their policy into a score?
 146 2015-10-22T02:24:09  <Luke-Jr> no
 147 2015-10-22T02:24:18  <morcos> how can you sort without a score?
 148 2015-10-22T02:24:25  <Luke-Jr> the policy is implemented mostly by a method that compares two transactions
 149 2015-10-22T02:25:12  *** amiller has joined #bitcoin-core-dev
 150 2015-10-22T02:25:24  <morcos> and if  A < B and B < C then A < C right?
 151 2015-10-22T02:27:05  <Luke-Jr> right
 152 2015-10-22T02:27:16  <Luke-Jr> or at least, that's assumed by the mempool
 153 2015-10-22T02:27:34  <Luke-Jr> it might be fun to some day make blocks with a policy that just returns rand() :P
 154 2015-10-22T02:27:37  <morcos> ok well the point i was going to make was that if you could look at it as a score
 155 2015-10-22T02:27:47  <morcos> and A has child A2 and B has child B2
 156 2015-10-22T02:28:04  <sipa> you're defining a total ordering, that is equivalent to a score function
 157 2015-10-22T02:28:09  <Luke-Jr> yes, you could look at it that way; my point was that actually using a score is overcomplex
 158 2015-10-22T02:28:28  <morcos> then if you use suhas's ancestor package tracking with your score instead of fee you'll get CPFP in terms of your policy
 159 2015-10-22T02:31:02  <morcos> sipa: what are your thoughts on maintaining ancestor package tracking on nodes that aren't mining
 160 2015-10-22T02:31:46  <morcos> its not totally useless and it hopefully shouldn't be too expensive.  but certainly adds more mememory/cpu for something they are basically not using
 161 2015-10-22T02:32:47  <morcos> the one thing i think it could be used for is fee estimation?  but thats only a maybe, i haven't really figured out how or if that would be worth it.  but estimating what miners are going to do seems like potentially a valuable tool for fee estimation.
 162 2015-10-22T02:45:24  *** CodeShark has quit IRC
 163 2015-10-22T02:45:43  *** CodeShark has joined #bitcoin-core-dev
 164 2015-10-22T02:48:04  *** CodeShark has quit IRC
 165 2015-10-22T02:48:50  *** CodeShark has joined #bitcoin-core-dev
 166 2015-10-22T03:01:16  *** Arnavion has quit IRC
 167 2015-10-22T03:01:40  *** Arnavion has joined #bitcoin-core-dev
 168 2015-10-22T03:39:25  <gmaxwell> cfields_: good catch, apparently it's not inhereted reliably and there is a bunch of broken software out there.
 169 2015-10-22T03:50:02  <gmaxwell> (by reliably I mean it's system dependant)
 170 2015-10-22T04:59:59  *** harding has quit IRC
 171 2015-10-22T05:54:15  *** Thireus has joined #bitcoin-core-dev
 172 2015-10-22T06:21:58  *** Thireus has quit IRC
 173 2015-10-22T06:22:10  *** Ylbam has joined #bitcoin-core-dev
 174 2015-10-22T06:35:39  *** fkhan has quit IRC
 175 2015-10-22T06:41:06  *** Thireus has joined #bitcoin-core-dev
 176 2015-10-22T06:47:25  *** fkhan has joined #bitcoin-core-dev
 177 2015-10-22T07:38:12  *** Thireus has quit IRC
 178 2015-10-22T07:40:32  *** ParadoxSpiral_ has joined #bitcoin-core-dev
 179 2015-10-22T07:43:54  *** ParadoxSpiral has quit IRC
 180 2015-10-22T08:11:48  *** Thireus has joined #bitcoin-core-dev
 181 2015-10-22T08:13:53  <phantomcircuit> can confirm bitcoin core requires reindex on power failure under windows
 182 2015-10-22T08:14:05  <phantomcircuit> probably leveldb env driver is stupid
 183 2015-10-22T08:15:16  <gmaxwell> phantomcircuit: fixy fixy?  unfortunately I think it's unmaintained.
 184 2015-10-22T08:15:35  <gmaxwell> as chrome uses some chrome specific abstractions for IO. :(
 185 2015-10-22T08:19:37  <phantomcircuit> gmaxwell, i can take a look at it, probably it should just be replaced with the current posix version which is just fopen/fread/fwrite basically
 186 2015-10-22T08:24:07  <GitHub16> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/0fbfc5106cd9...a09297010e17
 187 2015-10-22T08:24:07  <GitHub16> bitcoin/master 579b863 Wladimir J. van der Laan: devtools: Add security-check.py...
 188 2015-10-22T08:24:08  <GitHub16> bitcoin/master a092970 Wladimir J. van der Laan: Merge pull request #6854...
 189 2015-10-22T08:24:18  <GitHub129> [bitcoin] laanwj closed pull request #6854: devtools: Add security-check.py (master...2015_10_security_checks) https://github.com/bitcoin/bitcoin/pull/6854
 190 2015-10-22T08:25:30  *** Thireus has quit IRC
 191 2015-10-22T08:26:09  <wumpus> phantomcircuit: yeah https://github.com/bitcoin/bitcoin/issues/5610
 192 2015-10-22T08:26:16  <wumpus> no clue on solving it though
 193 2015-10-22T08:26:43  <wumpus> for some reason most people with a clue about windows internals seem to be blackhats, and they like things broken :p
 194 2015-10-22T08:27:13  *** Thireus has joined #bitcoin-core-dev
 195 2015-10-22T08:27:39  <phantomcircuit> wumpus, it's using windows memory mapping
 196 2015-10-22T08:27:39  <gmaxwell> phantomcircuit: reports went up astronomically after 0.10 and some users reported it worked fine for them before them and fails every time since.
 197 2015-10-22T08:27:44  <phantomcircuit> im going to assume it's that
 198 2015-10-22T08:28:00  <gmaxwell> phantomcircuit: oh it's writing via mmap too?
 199 2015-10-22T08:28:19  <phantomcircuit> i just checked and there's a Win32Map something or other
 200 2015-10-22T08:28:23  <phantomcircuit> i assume it's using that
 201 2015-10-22T08:29:15  <gmaxwell> https://litteration.files.wordpress.com/2013/05/science-dog.jpg
 202 2015-10-22T08:29:27  <wumpus> it has been a while since we updated leveldb, maybe some things got fixed since?
 203 2015-10-22T08:30:11  <wumpus> heh
 204 2015-10-22T08:30:36  <phantomcircuit> gmaxwell, that's about right
 205 2015-10-22T08:31:42  <phantomcircuit> it's windows... how do you even begin to debug this
 206 2015-10-22T08:32:38  <wumpus> that's a dark art
 207 2015-10-22T08:33:38  <wumpus> wouldn't be surprised if it requieres at least some, possibly expensive, proprietary software applications
 208 2015-10-22T08:35:25  <wumpus> debugging-by-mutating-the-code may be the most promising approach, extract a bare bones program that uses leveldb and crashes, then change it (adding flushes everywhere) until it manages to not corrupt anymore
 209 2015-10-22T08:36:09  <gmaxwell> it's 100% reproducable it seems, which helps.
 210 2015-10-22T08:36:11  <wumpus> this is the approach that works for fixing problems with underdocumented hardware, it should work for windows too...
 211 2015-10-22T08:36:53  <gmaxwell> the fact that it went way up when we turned up checking suggests to me that it's something like a truncated record at the end that was otherwise silently recoverable.
 212 2015-10-22T08:37:01  <gmaxwell> e.g. something like the osx bug.
 213 2015-10-22T08:38:10  <wumpus> yes good point - can you send me a database that is corrupted by this problem?
 214 2015-10-22T08:39:36  <gmaxwell> phantomcircuit: you would be you, as I don't have a repro.
 215 2015-10-22T08:40:59  <phantomcircuit> heh
 216 2015-10-22T08:41:16  <phantomcircuit> gmaxwell, i suspect you're right that it's similar to the OSX bug
 217 2015-10-22T08:41:27  <phantomcircuit> i also suspect it would be easier to replace leveldb with lmdb
 218 2015-10-22T08:41:40  <wumpus> phantomcircuit:  can you send me a database that is corrupted by this problem?
 219 2015-10-22T08:41:40  <gmaxwell> mmap only, goodbye 32 bit hosts.
 220 2015-10-22T08:41:44  <gmaxwell> Among other limitations. :(
 221 2015-10-22T08:41:45  <Arnavion> It's funny because if you ask Windows devs they'll tell you the dev tooling on Windows is far better than on Linux
 222 2015-10-22T08:42:10  <phantomcircuit> wumpus, i mindlessly clicked the reindex button
 223 2015-10-22T08:42:11  <gmaxwell> Arnavion: there is some aspect of "tools you know are always better than ones you don't"
 224 2015-10-22T08:42:17  <phantomcircuit> i can certainly try to do so though
 225 2015-10-22T08:42:24  <wumpus> Arnavion: it may be more user friendly on windows, but it's harder to get insight about what is happening on a system
 226 2015-10-22T08:42:25  <Arnavion> There's also the aspect of GUI tools that don't suck
 227 2015-10-22T08:42:27  <Arnavion> but I'm biased
 228 2015-10-22T08:42:46  <gmaxwell> Arnavion: you mean GUI tools inherently sucking? :P
 229 2015-10-22T08:43:05  <Arnavion> Oh, but that's what WinDbg is for
 230 2015-10-22T08:43:28  <Arnavion> You can do everything with a keyboard if you wish
 231 2015-10-22T08:43:54  <wumpus> anyhow this is not constructive
 232 2015-10-22T08:44:13  <wumpus> if you feel helpful, please demonstrate your skills by solving the leveldb corruption issue Arnavion :)
 233 2015-10-22T08:44:59  <Arnavion> I have no skills
 234 2015-10-22T08:45:21  <Arnavion> but I can take a look
 235 2015-10-22T08:45:27  <wumpus> demonstrate the great capability of windows debugging tools, then
 236 2015-10-22T08:46:49  <wumpus> phantomcircuit: but you can reproduce it! just start a new datadir, crash the thing, and send it to me
 237 2015-10-22T08:51:51  <wumpus> lmdb is not the panacea some people think - but if you feel like patching bitcoind to use it, that'd be pretty neat as we could run some benchmarks and comparisons
 238 2015-10-22T08:59:07  <phantomcircuit> wumpus, the main thing that's missing from leveldb is... sequence numbers in the journal
 239 2015-10-22T08:59:21  <phantomcircuit> if you miss a full record of writes it doesn't know
 240 2015-10-22T09:00:28  <wumpus> still, we need to debug this issue, as it is clearly windows specific, so not fundamental to leveldb
 241 2015-10-22T09:00:54  <wumpus> if this was broken on all platforms I'd agree that looking for a replacement would be advisable...
 242 2015-10-22T09:01:43  <phantomcircuit> wumpus, it *is* broken on all platforms
 243 2015-10-22T09:01:52  <phantomcircuit> the failure rate is simply much lower
 244 2015-10-22T09:02:24  <gmaxwell> one step at a time.
 245 2015-10-22T09:02:32  <gmaxwell> lets making the bleeding stop on windows first.
 246 2015-10-22T09:03:52  <wumpus> that's a nihilistic way of looking at it phantomcircuit, I like that, yes, it's all broken, some things with a very low failure rate... but it doesn't help solving the issue :)
 247 2015-10-22T09:04:35  *** rubensayshi has joined #bitcoin-core-dev
 248 2015-10-22T09:05:44  <wumpus> I'm sure lmdb is also broken in some subtle ways
 249 2015-10-22T09:06:17  <wumpus> anyhow, if you're not going to send me a corrupted datbase I'm going to start working on something else
 250 2015-10-22T09:07:50  <gmaxwell> I assume he has to wait for the reindex to complete to reproduce. :P
 251 2015-10-22T09:08:07  <wumpus> wha?
 252 2015-10-22T09:08:25  <wumpus> I assumed it would also happen when crashing during reindex?
 253 2015-10-22T09:08:37  <Luke-Jr> wumpus: what is Windows specfic?
 254 2015-10-22T09:09:50  <Luke-Jr> [08:13:52] <phantomcircuit> can confirm bitcoin core requires reindex on power failure under windows <-- I can confirm it requires reindex on power failure under Linux..
 255 2015-10-22T09:09:53  <wumpus> in that case, sorry, no hurry implied
 256 2015-10-22T09:10:49  <Luke-Jr> wumpus: if you in fact want such a corrupt bitcoin dir, I can probably get one for oyu
 257 2015-10-22T09:10:54  * Luke-Jr ponders where he put his USB Armory
 258 2015-10-22T09:11:19  <Luke-Jr> oh, it's plugged in
 259 2015-10-22T09:11:53  <wumpus> Luke-Jr: great!
 260 2015-10-22T09:12:14  <phantomcircuit> Luke-Jr, ive never had that issue on linux
 261 2015-10-22T09:12:17  <gmaxwell> it's interesting that the usb armory corrupts, perhaps not the same issue as windows though (though perhaps also worth fixing)
 262 2015-10-22T09:14:41  <wumpus> indeed, may be a different issue
 263 2015-10-22T09:22:26  <Luke-Jr> FWIW, removing power immediately after IBD started did NOT reproduce it
 264 2015-10-22T09:22:40  <Luke-Jr> anything in debug.log to look for to know when it does the first flush to disk?
 265 2015-10-22T09:23:27  <wumpus> set a low dbcache to force lots of flushes?
 266 2015-10-22T09:23:55  <wumpus> AFAIK it doesn't flush unless necessary until the initial block download is complete
 267 2015-10-22T09:25:04  *** BashCo has joined #bitcoin-core-dev
 268 2015-10-22T09:27:52  <Luke-Jr> 'often' is too slow
 269 2015-10-22T09:28:01  <Luke-Jr> 2015-10-22 09:27:59 UpdateTip: new best=000000003f7e074587fa1684ac863519fea3c64040b05ddd04948a13f7b19b42  height=415  log2_work=40.700462  tx=423  date=2009-01-14 05:56:05 progress=0.000002  cache=419
 270 2015-10-22T09:33:59  <Arnavion> From just lazy-mode browsing the code, Win32MapFile::Sync() flushes but Win32MapFile::Flush() is a no-op, whereas the equivalent functions for posix both do things
 271 2015-10-22T09:34:26  <Arnavion> I did not see how they are called from common code to see if that matters
 272 2015-10-22T09:37:56  <wumpus> it all depends on whether this map file is used for writing
 273 2015-10-22T09:38:17  <wumpus> AFAIK on other OSes, writing uses normal file system commands, whereas reading can use mmap
 274 2015-10-22T09:38:41  <wumpus> if t his is different on windows that would explain something
 275 2015-10-22T09:39:18  <Arnavion> Windows has a dedicated FlushViewOfFile function to flush a mapped region
 276 2015-10-22T09:39:53  <Arnavion> That is called by Sync(), but not by Flush()
 277 2015-10-22T09:58:45  <Luke-Jr> hmm
 278 2015-10-22T09:58:57  <Luke-Jr> it seems I can't reproduce during IBD or something
 279 2015-10-22T10:07:25  <Luke-Jr> and a fully synced state is huge :/
 280 2015-10-22T10:07:48  *** jgarzik has quit IRC
 281 2015-10-22T10:08:36  <midnightmagic> Luke-Jr: you're reproducing on a Windows OS?
 282 2015-10-22T10:08:44  <Luke-Jr> midnightmagic: no
 283 2015-10-22T10:18:03  *** jgarzik has joined #bitcoin-core-dev
 284 2015-10-22T10:18:03  *** jgarzik has joined #bitcoin-core-dev
 285 2015-10-22T10:25:35  * Luke-Jr ponders
 286 2015-10-22T10:26:31  <Luke-Jr> wumpus: well, the last chainstate I had before today is 922 MB and demanded a reindex.. not sure if that's good enough, because I can't upload 44 GB :/
 287 2015-10-22T10:27:12  <wumpus> that's kind of huge - do you perhaps know what ldb file the corruption is in?
 288 2015-10-22T10:27:38  <Luke-Jr> no, I have no idea how leveldb works :/
 289 2015-10-22T10:28:08  <Luke-Jr> I could probably just give you access to the device?
 290 2015-10-22T10:39:52  <Luke-Jr> wumpus: see PM for login info
 291 2015-10-22T10:39:55  <wumpus> ok
 292 2015-10-22T10:40:16  <Luke-Jr> setup auth for your github ssh key; let me know if there's a better one
 293 2015-10-22T10:40:48  <Luke-Jr> wumpus: the leveldb in question is under ~/.bitcoin.bak
 294 2015-10-22T10:41:34  <Luke-Jr> perhaps relevant: tar: /home/usbarmory/.bitcoin.bak/chainstate/1463292.ldb: File shrank by 2020259 bytes; padding with zeros
 295 2015-10-22T10:42:39  <wumpus> uh...
 296 2015-10-22T10:42:56  <wumpus> yes, that doesn't sound good. Were you tarring while bitcoin running?
 297 2015-10-22T10:43:33  <Luke-Jr> not sure if bitcoind is running, but if it is, it's not running on *this* database
 298 2015-10-22T10:43:59  * Luke-Jr wonders what it's doing so much to hang SSH
 299 2015-10-22T10:44:31  <wumpus> ok, it would not be strange if it is happening while running tar on the datadir that bitcoin was running in, those files are deleted and recreated all the time
 300 2015-10-22T10:44:46  <Luke-Jr> right
 301 2015-10-22T10:45:32  <sipa> wumpus: leveldb seems unmaintained
 302 2015-10-22T10:47:26  <wumpus> bleh
 303 2015-10-22T10:52:50  <wumpus> I'm happy we at least never decided to use it for the wallet
 304 2015-10-22T10:53:39  <sipa> maybe we should switch to sqlite... at least that's very well tested and maintained
 305 2015-10-22T10:54:33  <wumpus> it is, but it's not very fast for key/value storage, and I don't think it handles such large databases very well
 306 2015-10-22T10:55:40  <btcdrak> sipa: sqlite is very slow
 307 2015-10-22T10:56:08  <wumpus> if you need more advanced query features it is very nice though
 308 2015-10-22T10:57:09  <btcdrak> it would be so much easier and flexible if we were using SQL of some kind...
 309 2015-10-22T10:57:23  <wumpus> what would be easier?
 310 2015-10-22T10:57:45  <sipa> btcdrak: leveldb's own benchmark says that sqlite is faster for random reads
 311 2015-10-22T10:57:55  <btcdrak> sipa: interesting
 312 2015-10-22T10:57:59  <wumpus> for wallet metadata sql would be reasonably useful
 313 2015-10-22T10:58:02  <sipa> writes are much slower, but we do very few writes
 314 2015-10-22T10:58:13  <btcdrak> wumpus: well you get a lot of stuff for free with a SQL database stack.
 315 2015-10-22T10:58:24  <sipa> such as?
 316 2015-10-22T10:58:43  <Luke-Jr> # dmesg
 317 2015-10-22T10:58:45  <Luke-Jr> Segmentation fault
 318 2015-10-22T10:58:46  <Luke-Jr> -.-
 319 2015-10-22T10:59:07  <wumpus> during initial sync we do quite a lot of writes (every coin that is touched is written back), after that, most are reads
 320 2015-10-22T10:59:21  <btcdrak> sipa: indexes for a start. the ability to query the dataset externally from bitcoind etc
 321 2015-10-22T10:59:51  <wumpus> using database indexes would mean using a very verbose format
 322 2015-10-22T10:59:54  <sipa> btcdrak: we're not storing any data in a way that's useful to index
 323 2015-10-22T11:00:07  <sipa> we'd lose massive performance by expanding the data
 324 2015-10-22T11:00:28  <wumpus> that kind of defeats the purpose - bitcoind's database should optimized for its own purposes as running a node, not external ones
 325 2015-10-22T11:00:39  <btcdrak> I had wondered about leveldb's ongoing support. it really looks like an orphaned project at this time.
 326 2015-10-22T11:01:23  <wumpus> sipa: right
 327 2015-10-22T11:01:45  <jgarzik> btcdrak, the expanded dataset in SQL is something like 40GB at least
 328 2015-10-22T11:01:58  <jgarzik> not including the block data itself
 329 2015-10-22T11:02:38  <jgarzik> sipa, worse comes to worse, we can do our own key/value store...
 330 2015-10-22T11:02:45  <wumpus> also bitcoin's databases are not an external interface, don't query them directly unless it's to write troubleshooting/recovery tools
 331 2015-10-22T11:03:24  <jgarzik> there are actually a few optimizations you can make if you assume your keys are hashes already
 332 2015-10-22T11:03:48  <jgarzik> not that seeks matter these days, but some seeks can be eliminated
 333 2015-10-22T11:04:09  <wumpus> reinventing the wheel should be  the last recourse
 334 2015-10-22T11:04:36  <Luke-Jr> wumpus: whatever just happened appears to have fried the device, so I guess it'll be a while :/
 335 2015-10-22T11:04:40  <jgarzik> <grin> https://github.com/jgarzik/pgdb https://github.com/jgarzik/pagedb
 336 2015-10-22T11:04:48  <wumpus> Luke-Jr: I hope it wasn't me logging in! :p
 337 2015-10-22T11:05:07  <Luke-Jr> heh, I don't see how that could do it
 338 2015-10-22T11:05:14  <sipa> leveldb does have a benchmark about batch write performance
 339 2015-10-22T11:05:23  <sipa> where sqlite is much slower
 340 2015-10-22T11:05:24  <wumpus> I don't see either, I didn't even get a prompt
 341 2015-10-22T11:05:40  <wumpus> I've disconnected now
 342 2015-10-22T11:05:46  <sipa> though sqlite does win in synchronous write speed, which we also do frequently
 343 2015-10-22T11:06:34  <Luke-Jr> wumpus: makes me wonder if it's just a bad microSD card, which could make the failures invalid
 344 2015-10-22T11:06:49  <wumpus> would be interesting to patch sqlite into bitcoind and compare and do benchmarks
 345 2015-10-22T11:06:52  <sipa> the only reason i'm suggesting sqlite is because it is *very* well tested afaik
 346 2015-10-22T11:07:00  * wumpus says that for the second time today about a different database
 347 2015-10-22T11:07:20  <wumpus> sipa: also on windows?
 348 2015-10-22T11:07:42  <wumpus> leveldb is also very well tested, it is used by many companies in production... just mostly on linux/unix servers
 349 2015-10-22T11:08:05  <sipa> yes
 350 2015-10-22T11:10:09  <wumpus> do you know, how does sqlite do at caching?
 351 2015-10-22T11:10:20  <btcdrak> sipa: wumpus: maintenance is probably more of a concern than anything else. I do remember wondering about leveldb's longevity. sqlite is way too popular to disappear.
 352 2015-10-22T11:10:36  <jgarzik> sqlite leans a lot of OS caching, temporary files etc.
 353 2015-10-22T11:10:58  <wumpus> jgarzik: it can't do a much worse job than leveldb at that at least
 354 2015-10-22T11:11:36  <btcdrak> if we're just doing key/value what about all those database that do that sort of thing, like those nosql stacks?
 355 2015-10-22T11:12:02  *** PRab has joined #bitcoin-core-dev
 356 2015-10-22T11:12:38  <jgarzik> I'm a big fan of sqlite, and used it in my DNS server project.   I don't know that sqlite has batch update - does begin/commit suffice given our workflow?  They are -mostly- equivalent but not 100% equiv
 357 2015-10-22T11:12:42  <btcdrak> still. sqlite would be a better choice. But here's the thing, if we use sqlite then you're opening up the door for just about any database backend. specially if we made an abstraction layer
 358 2015-10-22T11:12:44  <wumpus> "The maximum size of a database file is 2147483646 pages. At the maximum page size of 65536 bytes, this translates into a maximum database size of approximately 1.4e+14 bytes "   ok, apparently I misremembered
 359 2015-10-22T11:13:15  <wumpus> btcdrak: there aren't that much embeddable key/value stores
 360 2015-10-22T11:13:26  <btcdrak> wumpus: true.
 361 2015-10-22T11:13:39  <btcdrak> and sqlite wins in terms of support/maintenance.
 362 2015-10-22T11:13:42  <jgarzik> I certainly wouldn't suggest expanding the bitcoind db in SQL, but using it as a dumb blob datastore shouldn't be a big issue
 363 2015-10-22T11:13:43  <wumpus> and most of them are likely one-off projects, even worsely maintained than leveldb
 364 2015-10-22T11:13:44  <wumpus> right
 365 2015-10-22T11:13:51  <wumpus> jgarzik: +1
 366 2015-10-22T11:13:59  *** PRab_ has quit IRC
 367 2015-10-22T11:13:59  <btcdrak> +!
 368 2015-10-22T11:14:04  <btcdrak> +1
 369 2015-10-22T11:14:41  <Luke-Jr> reminder: UTXO db is consensus-critical.
 370 2015-10-22T11:15:06  <sipa> very aware
 371 2015-10-22T11:15:12  <wumpus> sqlite is extremely easy to embed in a project, it's just one C file
 372 2015-10-22T11:15:27  <jgarzik> kv datastores off the top of my head: leveldb, gdbm, [n]dbm, tokyo cabinet, kyoto cabinet, berkeley db
 373 2015-10-22T11:15:35  *** Thireus has quit IRC
 374 2015-10-22T11:15:47  <jgarzik> wumpus, er huh?
 375 2015-10-22T11:15:52  <Luke-Jr> wumpus: surely one *big* C file. all of SQL can't be trivial.
 376 2015-10-22T11:15:53  *** Thireus has joined #bitcoin-core-dev
 377 2015-10-22T11:15:58  <jgarzik> wumpus, sqlite is quite bigger than one C file
 378 2015-10-22T11:16:05  <wumpus> hm it used to be at least
 379 2015-10-22T11:16:55  <jgarzik> berkeley db is well maintained :) :)
 380 2015-10-22T11:17:51  <Luke-Jr> sqlite is like 140kLOC
 381 2015-10-22T11:18:20  <jgarzik> for sqlite, if you pre-compile the SQL queries, it goes pretty fast
 382 2015-10-22T11:18:42  <sipa> of course we'd precompile them...
 383 2015-10-22T11:19:15  <sipa> i din't think git subtreeing sqlite is reasonable
 384 2015-10-22T11:20:07  <wumpus> oh this is why I thought it was one C file, yes it's a very large one with everything pasted together: https://www.sqlite.org/amalgamation.html
 385 2015-10-22T11:20:29  <wumpus> sipa: license-wise?
 386 2015-10-22T11:20:34  <sipa> size wise
 387 2015-10-22T11:21:16  <jgarzik> sqlite is public domain, one of the few
 388 2015-10-22T11:22:07  <sipa> oh, that amalgation is pretty awesome
 389 2015-10-22T11:22:33  <sipa> we can compile it with various flags for extensions disabled
 390 2015-10-22T11:22:49  <wumpus> apparantly leveldb is 26kLOC in total
 391 2015-10-22T11:22:53  <wumpus> sipa: yep
 392 2015-10-22T11:24:01  <jgarzik> imo LOC and size are secondary factors
 393 2015-10-22T11:24:20  <jgarzik> primary is on going maintenance, reliability, ...
 394 2015-10-22T11:24:46  <wumpus> LOC would be important if there is the chance we end up having to maintain/troubleshoot it ourself
 395 2015-10-22T11:24:55  <wumpus> (like now for leveldb)
 396 2015-10-22T11:25:00  <jgarzik> indeed
 397 2015-10-22T11:25:58  <jgarzik> I think pgdb will likely be 10k loc when finished
 398 2015-10-22T11:27:39  <wumpus> "SQlite as a  key-value database" https://www.sqlite.org/cvstrac/wiki?p=KeyValueDatabase
 399 2015-10-22T11:28:15  <wumpus> looks very easy to do, the only question is indeed jgarzik's whether the TRANSACTION/COMMIT is equivalent to batching as we use it now
 400 2015-10-22T11:29:06  <jgarzik> another sql caveat - based on experience - you have to be careful that your 'strings' are not translated in any way by the engine via unicode etc.
 401 2015-10-22T11:29:17  <Luke-Jr> jgarzik: nobody is going to maintain consensus-critical behaviour
 402 2015-10-22T11:29:20  <wumpus> sql injections? *ducks*
 403 2015-10-22T11:29:23  <jgarzik> I used sqlite to store binary DNS records
 404 2015-10-22T11:29:27  <sipa> sqlite is fully transactional
 405 2015-10-22T11:29:28  <Luke-Jr> no matter what we use, we will need to maintain it
 406 2015-10-22T11:30:05  <Luke-Jr> even if that just means reviewing each upstream release
 407 2015-10-22T11:30:20  <sipa> or just reviewing their test practices
 408 2015-10-22T11:30:32  <jgarzik> sqlite is very well hammered
 409 2015-10-22T11:30:32  <wumpus> (no, sqlite has proper parametrized queries, and from my experience its BLOBs are binary clean)
 410 2015-10-22T11:30:41  <jgarzik> few maintain software to consensus critical standards though
 411 2015-10-22T11:31:40  <sipa> agree, but a database interface clearly should... it's perfectly specified in both dirextion: it should find every record that exist, and not find any reorc that doesn't exist
 412 2015-10-22T11:31:48  <jgarzik> another bit - make sure there are no limitations on rows-modified-at-once
 413 2015-10-22T11:31:50  <sipa> so anything that is nkt consensus compatible is a bug
 414 2015-10-22T11:31:56  <jgarzik> otherwise we re-introduce the BDB fork issue (locks)
 415 2015-10-22T11:32:20  <Luke-Jr> sipa: fixing bugs is a consensus compatibility bug!
 416 2015-10-22T11:32:49  <sipa> jgarzik: the "bug" in the bdb case was that we didn't chefk tue bdb return value, and regarded failure to write as a block validation failure
 417 2015-10-22T11:33:05  <sipa> Luke-Jr: fully agree there
 418 2015-10-22T11:33:08  <Luke-Jr> anyhow, I need to go to sleep, but I think we will regret it if we import sqlite to consensus-critical code.
 419 2015-10-22T11:33:27  <jgarzik> sipa, well there was an issue where we were hitting BDB max lock
 420 2015-10-22T11:33:54  <jgarzik> sipa, the point still stands - sql engines do somethings have max-rows-updated-in-one-transaction type limits
 421 2015-10-22T11:33:55  <sipa> jgarzik: sure, but that wouldn't have been a consensus failure threat if we didn't treat out-of-locks as invalid-block
 422 2015-10-22T11:34:10  <jgarzik> *sometimes
 423 2015-10-22T11:34:39  <sipa> jgarzik: it was us who turned an administrative restriction into a consensus problem
 424 2015-10-22T11:34:48  <jgarzik> so, sql transaction limits is another one for the eval list
 425 2015-10-22T11:35:09  <sipa> if we would just have gone "oops! can't write! quitting", like we do for out of disk, bdb would not have caused forks
 426 2015-10-22T11:35:16  <wumpus> right, if it would treat database errors as a fatal error instead of rejection, it'd wouldn't have been as bad
 427 2015-10-22T11:35:17  <jgarzik> nod
 428 2015-10-22T11:35:20  <sipa> it would have been a bad DoS attack, though
 429 2015-10-22T11:35:27  <jgarzik> yep
 430 2015-10-22T11:35:35  <sipa> but not a fork
 431 2015-10-22T11:35:39  <wumpus> but fairly easy to resolve
 432 2015-10-22T11:35:50  <wumpus> right
 433 2015-10-22T11:36:28  <sipa> of course, you are absolutely right we need to be aware of such limits in sqlite or whatever we're incestigating
 434 2015-10-22T11:36:33  <sipa> eh, investigating
 435 2015-10-22T11:36:38  <jgarzik> lol
 436 2015-10-22T11:36:52  <sipa> the keys are like right nezt to each other
 437 2015-10-22T11:37:40  <sipa> also, sqlite databases are single files, right?
 438 2015-10-22T11:38:26  <jgarzik> yes*
 439 2015-10-22T11:38:42  <jgarzik> * some temporary files such as journals are created along the way, and must exist for recovery post-crash
 440 2015-10-22T11:38:57  <sipa> ok, similar to bdb
 441 2015-10-22T11:39:10  <sipa> that's fine for application database stuff
 442 2015-10-22T11:39:15  <sipa> maybe not wantes for wallets
 443 2015-10-22T11:39:19  <sipa> *wanted
 444 2015-10-22T11:40:52  <jgarzik> sqlite files do want periodic maintenance via 'VACUUM'
 445 2015-10-22T11:41:28  <sipa> we can do that in our flushtodisk function
 446 2015-10-22T11:41:30  <jgarzik> they get fragmented, old records cluttered, performance degrades a bit over time
 447 2015-10-22T11:41:46  <jgarzik> it is a very, very heavyweight operation
 448 2015-10-22T11:42:16  <jgarzik> info: https://sqlite.org/lang_vacuum.html
 449 2015-10-22T11:42:53  <jgarzik> auto_vacuum handles large deletes, but not fragments over time
 450 2015-10-22T11:43:15  <jgarzik> could just run it at startup and then auto_vacuum
 451 2015-10-22T11:43:28  <sipa> ok
 452 2015-10-22T11:56:54  <btcdrak> I've not seen everyone get so excited about something for a long time.
 453 2015-10-22T11:58:54  *** Thireus has quit IRC
 454 2015-10-22T12:01:41  <wumpus> "The author disclaims copyright to this source code.  In place ofa legal notice, here is a blessing:   May you do good and not evil.  May you find forgiveness for yourself and forgive others.  May you share freely, never taking more than you give." I love sqlite's appraoch to licensing
 455 2015-10-22T12:11:20  <btcdrak> wumpus: if only there were more like that
 456 2015-10-22T13:06:08  <jgarzik> At https://www.sqlite.org/cvstrac/wiki?p=KeyValueDatabase sqlite seems across the board worse... except for large numbers of records (our use case)
 457 2015-10-22T13:06:48  <jgarzik> I'm happy to create an sqlite branch for bitcoind, if nobody else is working on it
 458 2015-10-22T13:07:50  *** pigeons has quit IRC
 459 2015-10-22T13:09:03  *** pigeons has joined #bitcoin-core-dev
 460 2015-10-22T13:09:26  *** pigeons is now known as Guest89666
 461 2015-10-22T13:10:55  <btcdrak> jgarzik: go for it
 462 2015-10-22T13:13:47  <sipa> how mature is sqlite4?
 463 2015-10-22T13:14:05  <jgarzik> good question
 464 2015-10-22T13:14:13  <jgarzik> vast majority in field is 3
 465 2015-10-22T13:14:33  <sipa> its design seems much closer to leveldb
 466 2015-10-22T13:14:47  <sipa> as it is... a key/value store internally
 467 2015-10-22T13:15:02  <sipa> i wonder if there is an api to access that key/value store interface
 468 2015-10-22T13:15:10  <sipa> i have read before that you can
 469 2015-10-22T13:15:22  <wumpus> I was starting on it, but will happily leave it to you jgarzik
 470 2015-10-22T13:15:53  <sipa> The default built-in storage engine is a log-structured merge database. It is very fast, faster than LevelDB, supports nested transactions, and stores all content in a single disk file.
 471 2015-10-22T13:16:01  <sipa> ^ from the sqlite4 design
 472 2015-10-22T13:16:26  <wumpus> where do you seea nything about sqlite4? at least the download page only has 3
 473 2015-10-22T13:16:54  <sipa> ok, sqlite4 is far from released
 474 2015-10-22T13:17:02  <wumpus> oh, "trunk" :-)
 475 2015-10-22T13:17:06  <sipa> not an option at this point
 476 2015-10-22T13:17:25  <wumpus> sounds great though
 477 2015-10-22T13:17:59  <sipa> alsoz sqlite3 does not promise backward write compatibility... so it is like bdb, and we need to guarantee only increases in version
 478 2015-10-22T13:18:26  <wumpus> it does promise backward read compatibility?
 479 2015-10-22T13:18:29  <sipa> yes
 480 2015-10-22T13:19:07  <wumpus> well then it's not too bad, if it detects a downgrade, it could export|import the database
 481 2015-10-22T13:19:39  <sipa> wait
 482 2015-10-22T13:19:43  <wumpus> berkeleydb doesn't even guarantee that much
 483 2015-10-22T13:19:56  <sipa> databases created by 3.3 can't be read by earlier 3.x versions
 484 2015-10-22T13:20:27  <sipa> lol, and this caused so much problems they reverted to the old format by default in 3.3.6
 485 2015-10-22T13:21:24  <jgarzik> note - single file db implies large file support req
 486 2015-10-22T13:22:11  <sipa> is that not available on every platform these days?
 487 2015-10-22T13:22:13  <wumpus> is that unreasonable these days?
 488 2015-10-22T13:22:50  <jgarzik> every platform, but some filesystems may be e.g. fat32 for weird reasons
 489 2015-10-22T13:22:57  <wumpus> ... and has been for 10 years or so
 490 2015-10-22T13:23:02  <jgarzik> I think it's reasonable
 491 2015-10-22T13:23:03  <sipa> oh: a database created by version X will always be read/write compatible with version X, even if modified by later versions
 492 2015-10-22T13:23:06  <jgarzik> just noting
 493 2015-10-22T13:23:23  <wumpus> fat32 is pretty much dead
 494 2015-10-22T13:23:27  <sipa> so it's not an auto-update to whatever the format in the later version is, like bdb
 495 2015-10-22T13:23:27  <jgarzik> sipa, yeah, kernel filesystem rules
 496 2015-10-22T13:23:33  <wumpus> it also doesn't support bigger disks
 497 2015-10-22T13:23:39  <jgarzik> wumpus, sadly very much alive on USB sticks
 498 2015-10-22T13:24:04  <sipa> put bitcoind'd chainstate on a USB stick: you are eaten by a grue
 499 2015-10-22T13:24:20  <wumpus> *small* USB sticks
 500 2015-10-22T13:24:40  <sipa> also: for now the chainstate still fits in 2 GB...
 501 2015-10-22T13:24:44  <sipa> ... for now
 502 2015-10-22T13:25:30  <jgarzik> as I said... just making a note because it's something of which people should be aware.  continuing to hack on it :)
 503 2015-10-22T13:25:45  <sipa> awesome, thanks
 504 2015-10-22T13:25:50  <wumpus> great
 505 2015-10-22T13:27:53  <wumpus> 32GB is pretty much the limit for FAT32 devices, it's possible to have larger volumes, but it becomes really ineffiecient and at least windows doesn't allow formatting them
 506 2015-10-22T14:41:11  *** lightningbot has joined #bitcoin-core-dev
 507 2015-10-22T14:44:22  *** Greg__ has joined #bitcoin-core-dev
 508 2015-10-22T14:53:03  *** malte has joined #bitcoin-core-dev
 509 2015-10-22T14:54:15  <GitHub122> [bitcoin] MarcoFalke closed pull request #6866: [trivial] fix white space in rpc help messages (master...MarcoFalke-2015-rpcWhitespace) https://github.com/bitcoin/bitcoin/pull/6866
 510 2015-10-22T14:54:32  <morcos> sipa: i'm trying to think about how to preserve the state of the mempool while generating block templates
 511 2015-10-22T14:59:14  *** lightningbot` has joined #bitcoin-core-dev
 512 2015-10-22T15:00:42  *** cfields has joined #bitcoin-core-dev
 513 2015-10-22T15:00:58  *** kanzure_ has joined #bitcoin-core-dev
 514 2015-10-22T15:01:14  *** Eliel_ has joined #bitcoin-core-dev
 515 2015-10-22T15:02:43  <sipa> morcos: how about being able to mark mempool txn as locked?
 516 2015-10-22T15:03:03  <sipa> individual transactions, that is
 517 2015-10-22T15:03:20  <morcos> sipa: how do you mean?  the block template generation code needs to mark all of them as locked?
 518 2015-10-22T15:03:49  <morcos> or do you mean if you need to delete something, such as with RBF, then you mark it as deleted and go ahead and add the new one?
 519 2015-10-22T15:04:14  *** __Greg has joined #bitcoin-core-dev
 520 2015-10-22T15:04:19  <sipa> morcos: while the template generator runs, it locks transactions one by one as they are added to the template
 521 2015-10-22T15:04:40  *** teward- has joined #bitcoin-core-dev
 522 2015-10-22T15:04:49  *** banana_lotus has joined #bitcoin-core-dev
 523 2015-10-22T15:04:54  <sipa> so the next transactions it fetches are guaranteed to be not conflicting with the ones it already has
 524 2015-10-22T15:05:30  <sipa> you want a new incoming block to override that though, and cancel the generator to make it start over
 525 2015-10-22T15:05:47  <morcos> yes agreed with that last part, i'll tackle that later
 526 2015-10-22T15:06:16  *** guruvan- has joined #bitcoin-core-dev
 527 2015-10-22T15:06:29  <morcos> but i'm not sure how to just lock txs one by one, you'd have to lock the mempool.cs each time you went to look up a tx to decide to add it to the mempool
 528 2015-10-22T15:06:42  <morcos> i'm starting off by trying to modify the existing CNB to hold locks a lot less
 529 2015-10-22T15:06:46  *** lightningbot has quit IRC
 530 2015-10-22T15:06:46  *** PRab has quit IRC
 531 2015-10-22T15:06:46  *** Greg__ has quit IRC
 532 2015-10-22T15:06:47  *** molly has quit IRC
 533 2015-10-22T15:06:47  *** teward has quit IRC
 534 2015-10-22T15:06:47  *** kanzure has quit IRC
 535 2015-10-22T15:06:47  *** guruvan has quit IRC
 536 2015-10-22T15:06:47  *** BananaLotus has quit IRC
 537 2015-10-22T15:06:48  *** cfields_ has quit IRC
 538 2015-10-22T15:06:48  *** Luke-Jr has quit IRC
 539 2015-10-22T15:06:48  *** murr4y has quit IRC
 540 2015-10-22T15:06:49  *** Eliel has quit IRC
 541 2015-10-22T15:06:53  *** teward- is now known as teward
 542 2015-10-22T15:06:53  *** banana_lotus is now known as BananaLotus
 543 2015-10-22T15:06:53  *** guruvan- is now known as guruvan
 544 2015-10-22T15:07:10  <morcos> so i'm imagning it basically copies ptrs to all the txs in the mempool and the score it needs for them
 545 2015-10-22T15:07:31  <morcos> then it doesn't need to hold mempool.cs any more as long as the ptr does not become invalid
 546 2015-10-22T15:07:53  <morcos> which would only happne if the txn is deleted
 547 2015-10-22T15:08:05  <sipa> i guess transactions would get a refcount
 548 2015-10-22T15:08:21  <sipa> and the refcount value itself would be protected by mempool.cs
 549 2015-10-22T15:08:23  <morcos> but i think that has to be for all of the txs as you're running the logic of figuring out which ones need to be in the template
 550 2015-10-22T15:08:53  <morcos> yeah, but i was imagining it was a singular refcount
 551 2015-10-22T15:08:55  <sipa> hmm, does the mempool not hold some index sorted by template inclusion preference?
 552 2015-10-22T15:08:58  <morcos> instead of per tx
 553 2015-10-22T15:09:44  <morcos> yeah so thats where my approach might not be the best approach b/c the new template generation code will have such an index
 554 2015-10-22T15:09:49  <sipa> another idea is to remove the storage if txn from the mempool entirely
 555 2015-10-22T15:10:05  <morcos> but even still you need to iterate far past the size of a block
 556 2015-10-22T15:10:14  <sipa> you just get a transaction store manager, which you give transactions, and it returns a refcounted pointer
 557 2015-10-22T15:10:21  <sipa> the mempool stores the pointers
 558 2015-10-22T15:10:34  <sipa> i guess that's just smart pointers
 559 2015-10-22T15:11:24  <sipa> so the template generator would grab pointers to the top N (some multiple of a block), increase their refcount, release memlool.cs
 560 2015-10-22T15:11:40  <sipa> then goes off to build a block, verify if wanted, ...
 561 2015-10-22T15:11:58  <sipa> and then release the pointers
 562 2015-10-22T15:12:27  <sipa> the mempool can change during that time... you just know that the set you grabbed at that point while holding the lock is internally consistent
 563 2015-10-22T15:12:42  <sipa> and consistent with the block is was building on
 564 2015-10-22T15:12:44  <morcos> yes thats what i'm going for
 565 2015-10-22T15:12:59  <morcos> i guess it depends on how big the set is compared to the whole mempool
 566 2015-10-22T15:13:16  <morcos> so my idea was that rather than marking individual txs
 567 2015-10-22T15:13:18  *** PRab has joined #bitcoin-core-dev
 568 2015-10-22T15:13:25  <morcos> you just tell the mempool, hey, referring to things, don't delete
 569 2015-10-22T15:13:42  <morcos> and then occasionally that gets freed, and anything marked for deletion can happen
 570 2015-10-22T15:13:47  <sipa> hmm
 571 2015-10-22T15:13:48  <sipa> not sure
 572 2015-10-22T15:13:52  <morcos> since deletes are rarer and smaller than adds
 573 2015-10-22T15:13:56  <sipa> sounds more complicated
 574 2015-10-22T15:14:59  <morcos> ok, maybe i'll explore both
 575 2015-10-22T15:15:09  <morcos> next question and this might be related to CSV
 576 2015-10-22T15:15:22  <morcos> really annoying that CheckInputs needs cs_main
 577 2015-10-22T15:15:34  <morcos> doesn't seem like it should have to?
 578 2015-10-22T15:16:04  <sipa> so i think it's relative easy this way: turn the storage of txn in the mempool into smart pointers, and you can very cheaply and efficiently ask the mempool for the set of all its transactions
 579 2015-10-22T15:16:05  <morcos> you're storing the hashblock that its valid at, and that can't change height
 580 2015-10-22T15:16:32  <sipa> it shouldn't be needed, indeed
 581 2015-10-22T15:17:06  <morcos> sipa: not that easy, you need stuff from the mempool entries, thats where fees are stored
 582 2015-10-22T15:17:24  <morcos> so you still have to iterate to figure out which pointers you need and copy the meta information
 583 2015-10-22T15:17:45  <sipa> ok, so that fee information needs to be inside the smart pointed-to objects
 584 2015-10-22T15:18:38  <morcos> so all you've done now is taken the multi-index approach and say ehh forget that, lets just have multiple indexes referring to this same set of pointers
 585 2015-10-22T15:19:16  <sipa> i may be missing something
 586 2015-10-22T15:19:44  <morcos> i think the only complication from my idea is damn RBF... otherwise you just don't call any eviction code except periodically...  and there is no problem running it only periodically
 587 2015-10-22T15:19:58  <morcos> and you don't even have to track anything
 588 2015-10-22T15:20:11  *** CodeShark__ has joined #bitcoin-core-dev
 589 2015-10-22T15:20:17  <sipa> what i'm suggesting is a way to make cheap snapshots of subsets of the mempool at a given time
 590 2015-10-22T15:20:44  <sipa> those snapshots themselves don't have an index, but can be ordered by one at the time they are created
 591 2015-10-22T15:20:52  <sipa> isn't that enough?
 592 2015-10-22T15:21:54  <morcos> yeah maybe, i think i'm just leary of the double indirection now where the multi-index is a multi-index of ptrs, but maybe thats something stupid to worry about
 593 2015-10-22T15:25:29  <sipa> the transactions inside the mempool already have a dozen or so indirections
 594 2015-10-22T15:26:29  <sipa> ok, more like 6
 595 2015-10-22T15:26:35  <sipa> but still!
 596 2015-10-22T15:27:36  <morcos> so something like a boost::shared_ptr
 597 2015-10-22T15:28:18  <sipa> yes, that specifically :)
 598 2015-10-22T15:28:27  <sipa> sorry, i thought it was called smart_ptr :)
 599 2015-10-22T15:29:03  <sipa> you probably want to wrap CTransaction inside something that also contains its (direct) fee and perhaps other immutable statistics
 600 2015-10-22T15:29:26  <sipa> and then have sharee_ptrs to those inside the mempool entries
 601 2015-10-22T15:29:43  <sipa> heh, we could even serialize the transactions inside to reduce memory usage...
 602 2015-10-22T15:31:47  <morcos> yes, so then it'll get a bit more complicated with ancestor package tracking when you have mutable state like ancestor fee rate that you want to copy to your template generation code, but that was going to be difficult period
 603 2015-10-22T15:32:22  <sipa> why do you need it in the template generator code?
 604 2015-10-22T15:32:55  <morcos> thats the sort the logic uses
 605 2015-10-22T15:33:09  <morcos> yikes
 606 2015-10-22T15:33:18  <sipa> sort the list of copied pointers according to the sorting criterion you want while creating it (so while holding mempool.cs)
 607 2015-10-22T15:33:30  <sipa> then release mempool.cs and you can forget it
 608 2015-10-22T15:33:45  <morcos> ok, but see thats the complicated part that takes a long time
 609 2015-10-22T15:33:54  <sipa> ah, ok
 610 2015-10-22T15:34:11  <morcos> the tx thats sorted at the top , really links to a lot of other txs , whose ptrs you have to grab to
 611 2015-10-22T15:34:20  <sipa> hmm, right
 612 2015-10-22T15:34:22  <morcos> but then how do you solve the problem of once you include that in your template
 613 2015-10-22T15:34:27  <morcos> everything elses sort changes
 614 2015-10-22T15:34:52  <sipa> i wouldn't bother with changing sorts
 615 2015-10-22T15:36:05  <morcos> well that'll be the magic of the algorithm, it'll probably have to be some heuristic, but its very easy to have expensive chain A-B-C and so C is sorted first, but then cheap C2 is still sorted quite high.
 616 2015-10-22T15:36:37  <morcos> uh not exactly but close
 617 2015-10-22T15:36:53  <morcos> but yes, i agree we don't have to be perfect
 618 2015-10-22T15:37:20  <morcos> ok you've given me some ideas... let me see what i come up with
 619 2015-10-22T15:40:15  <sipa> great
 620 2015-10-22T15:40:28  <sipa> feel free to completely disregard my ideas :)
 621 2015-10-22T15:48:08  <jgarzik> RE locking transactions one-by-one - that is the typical parallelism solution
 622 2015-10-22T15:48:45  <jgarzik> just need a gatekeeper (core struct lock) for insert/delete/move
 623 2015-10-22T15:53:27  <GitHub181> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/a09297010e17...2cd020d054d6
 624 2015-10-22T15:53:27  <GitHub181> bitcoin/master 3cb56f3 Daniel Cousens: *: alias -h for --help
 625 2015-10-22T15:53:28  <GitHub181> bitcoin/master 2cd020d Wladimir J. van der Laan: Merge pull request #6846...
 626 2015-10-22T15:53:32  <GitHub119> [bitcoin] laanwj closed pull request #6846: [Trivial] bitcoind: alias -h for -help (master...aliash) https://github.com/bitcoin/bitcoin/pull/6846
 627 2015-10-22T16:02:42  <GitHub145> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/2cd020d054d6...f2c869aef2e7
 628 2015-10-22T16:02:42  <GitHub145> bitcoin/master c6824f8 J Ross Nicoll: Add DERSIG transaction test cases...
 629 2015-10-22T16:02:43  <GitHub145> bitcoin/master f2c869a Wladimir J. van der Laan: Merge pull request #6848...
 630 2015-10-22T16:02:46  <GitHub77> [bitcoin] laanwj closed pull request #6848: Add DERSIG transaction test cases (master...bip66-tests) https://github.com/bitcoin/bitcoin/pull/6848
 631 2015-10-22T16:15:38  *** BashCo has quit IRC
 632 2015-10-22T16:18:01  <gavinandresen> FYI: git HEAD master isn't working for me on OSX.  Running make check gets me:
 633 2015-10-22T16:18:04  <gavinandresen> libc++abi.dylib: terminating with uncaught exception of type boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::lock_error> >: boost: mutex lock failed in pthread_mutex_lock: Invalid argument
 634 2015-10-22T16:20:09  <gavinandresen> I'm very confused, because git bisect blames a commit that works....
 635 2015-10-22T16:20:51  <cfields> gavinandresen: sounds like your boost may be built against the wrong stdlib? checking master on osx here now.
 636 2015-10-22T16:21:25  <gavinandresen> cfields: I thought that too, did a brew uninstall / brew install of boost
 637 2015-10-22T16:25:22  <btcdrak> morcos: sorry was afk. My other question was how does mempool evictions play into the mix? I assume one doesnt want to evict a txn that you're trying to mine.
 638 2015-10-22T16:26:21  <morcos> btcdrak: yes, thats what my concern was.  the only way a pointer to a tx becomes invalid now is if the tx is deleted.  A) throught being mined, in which case you want to abort block template generation anyway or B) through eviction or RBF
 639 2015-10-22T16:27:27  <morcos> for B1) eviction, i thought it would just be easy to just skip eviction if the template_tx_lock is held, and just make sure eviction gets to run every so often.
 640 2015-10-22T16:28:06  <morcos> but B2) RBF, doesn't have an ideal solution,  i wanted to just flag those txs, but maybe sipas idea with shared_ptrs is cleaner
 641 2015-10-22T16:34:54  <cfields> gavinandresen: ok here. What compiler/boost/osx version?
 642 2015-10-22T16:34:56  *** BashCo has joined #bitcoin-core-dev
 643 2015-10-22T16:45:42  <gavinandresen> cfields: Apple LLVM version 7.0.0 (clang-700.0.72), boost /usr/local/Cellar/boost/1.58.0, osx 10.10.5  ....
 644 2015-10-22T16:46:15  <gavinandresen> ... but it is very possible my machine is in some weird state, I upgraded my XCode yesterday.
 645 2015-10-22T16:47:23  <btcdrak> gavinandresen: first rule of fight club - "never upgrade XCode"
 646 2015-10-22T16:56:35  <jgarzik> heh
 647 2015-10-22T16:56:39  <jgarzik> just did last night...
 648 2015-10-22T16:59:21  <cfields> gmaxwell / jgarzik: ping. I must be going crazy. From what I can tell experimentally (and backed by man pages), on Linux, our incoming connections are being treated as blocking.
 649 2015-10-22T16:59:49  <jgarzik> cfields, post-accept(2) you set sockets to non-blocking
 650 2015-10-22T16:59:56  <cfields> because O_NONBLOCK isn't inherited by accept
 651 2015-10-22T17:00:01  <jgarzik> nod
 652 2015-10-22T17:00:05  <cfields> jgarzik: should, sure
 653 2015-10-22T17:00:08  <cfields> but we don't
 654 2015-10-22T17:00:13  <jgarzik> oops
 655 2015-10-22T17:00:44  <jgarzik> seems surprising, seems like it would have been well noticed before now
 656 2015-10-22T17:01:03  <sipa> that would imply that the "opportunistic write" always succeeds
 657 2015-10-22T17:01:09  <cfields> for the most part it should be ok because reads wait for select(), but i'm surprised it hasn't caused issues
 658 2015-10-22T17:01:13  <sipa> that should be trivial to test?
 659 2015-10-22T17:01:15  *** rubensayshi has quit IRC
 660 2015-10-22T17:01:45  <jgarzik> no wait - send/recv have per-call NB flags
 661 2015-10-22T17:02:53  <cfields> jgarzik: ah right
 662 2015-10-22T17:03:20  <cfields> jgarzik: still though, i've read about select() acting funny on linux with blocking sockets
 663 2015-10-22T17:04:00  <jgarzik> like haha, that's a laugh funny?
 664 2015-10-22T17:05:31  * jgarzik attempts to implement db iteration without brute force 'select * from table' into RAM ;p
 665 2015-10-22T17:06:10  <cfields> "Under Linux, select() may report a socket file descriptor as "ready for reading", while nevertheless a subsequent read blocks... Thus it may be safer to use O_NONBLOCK on sockets that should not block."
 666 2015-10-22T17:06:44  <jgarzik> recv() will fail in that case
 667 2015-10-22T17:06:50  <cfields> Either way, forcing them to nonblock after accept can't hurt, right?
 668 2015-10-22T17:06:57  <cfields> hmm
 669 2015-10-22T17:07:19  <sipa> jgarzik: i guess you can get some sort of object/state which you can ask for "more results" ?
 670 2015-10-22T17:07:21  <cfields> it wouldn't just block?
 671 2015-10-22T17:08:02  <GitHub132> [bitcoin] MarcoFalke opened pull request #6870: [trivial] Misc cleanup and translations (master...MarcoFalke-2015-trivial3) https://github.com/bitcoin/bitcoin/pull/6870
 672 2015-10-22T17:08:10  <cfields> jgarzik: er, the DONTWAIT. nm.
 673 2015-10-22T17:12:50  <jgarzik> sipa, you can 'select *' and iterate in one direction (forwards; next-result), whereas the CDBIterator wants Next, Prev and Seek operations
 674 2015-10-22T17:13:05  * jgarzik is studying the callers/users of CDBIterator now
 675 2015-10-22T17:13:27  <sipa> jgarzik: i don't think we use prev
 676 2015-10-22T17:15:47  <jgarzik> so it seems
 677 2015-10-22T17:16:44  <sipa> and seek would be equivalent to just starting the search
 678 2015-10-22T17:16:51  <cfields> gavinandresen: that's no good. i'm confused about that error, though
 679 2015-10-22T17:17:19  <jgarzik> sipa, well seek-start, seek-end and seek-key are all different
 680 2015-10-22T17:17:27  <sipa> jgarzik: i think the abstraction can be changed so we have a few 'entry types' (which would translate to separate sqlite tables)
 681 2015-10-22T17:17:42  <sipa> jgarzik: and only have iterators for all-data-within-one-entry-type
 682 2015-10-22T17:17:54  <jgarzik> sure
 683 2015-10-22T17:18:04  <sipa> which would be compatible with both leveldb and sqlite
 684 2015-10-22T17:21:00  <jgarzik> ah a few of these db ops are like Prev(), unused.  job easier++
 685 2015-10-22T17:21:07  *** murr4y has joined #bitcoin-core-dev
 686 2015-10-22T17:24:35  <jgarzik> hrm
 687 2015-10-22T17:24:41  <jgarzik> sipa, lex order?
 688 2015-10-22T17:25:43  <jgarzik> seems so
 689 2015-10-22T17:43:09  <sipa> some things need lex order, yes
 690 2015-10-22T18:11:51  *** Greg__ has joined #bitcoin-core-dev
 691 2015-10-22T18:12:12  *** __Greg has quit IRC
 692 2015-10-22T18:28:10  <gavinandresen> FYI: my build-failing-on-OSX problem went away with a 'git clean -dxf' then re-autogen/configure/build....
 693 2015-10-22T18:35:35  *** belcher has joined #bitcoin-core-dev
 694 2015-10-22T18:36:31  * jonasschnelli did accidentally sent untracked files to nirvana serval times with `git clean -dxf`...
 695 2015-10-22T18:36:59  * sipa too
 696 2015-10-22T18:37:05  <jonasschnelli> heh
 697 2015-10-22T18:37:07  *** Thireus has quit IRC
 698 2015-10-22T18:39:04  *** d_t has joined #bitcoin-core-dev
 699 2015-10-22T18:41:36  *** Thireus has joined #bitcoin-core-dev
 700 2015-10-22T18:51:17  *** jl2012 has joined #bitcoin-core-dev
 701 2015-10-22T18:58:44  <gmaxwell> I haven't tried it on bitcoin core yet, but the latest functionality in rr (replay debugger) looks pretty great: http://robert.ocallahan.org/2015/10/rr-40-released-with-reverse-execution.html
 702 2015-10-22T19:02:58  <jcorgan> did i hear rumblings about switching to sqlite?
 703 2015-10-22T19:03:10  <maaku> i wish there was a 'git stage --clean' command
 704 2015-10-22T19:03:23  <btcdrak> meeting over at #bitcoin-dev now
 705 2015-10-22T19:03:33  <jcorgan> isn't it git checkout . ?
 706 2015-10-22T19:03:41  <btcdrak> jcorgan: yes, jgarzik is working on it now
 707 2015-10-22T19:04:01  <jcorgan> oh sweet.  then we can use sqlcipher and get full db ondisk encryption
 708 2015-10-22T19:04:43  <jcorgan> it's a drop-in replacement/fork for sqlite that operates at the db page level, like dmcrypte for block devices
 709 2015-10-22T19:05:13  <sipa> why do you need encryption for public data? :)
 710 2015-10-22T19:05:44  <maaku> sipa: i presume he's talking about wallet?
 711 2015-10-22T19:05:53  <sipa> i wouldn't use sqlite for the wallet
 712 2015-10-22T19:05:58  <gmaxwell> jcorgan: you saw that discussion, but I expect it won't happen (At least with sqllite3) as-- if I'm not mistaken-- btcd tried it and found the performance unacceptable.  (but we should anyways, both to have a comparison point, and because sqllite4 might be faster even if 3 is too slow)
 713 2015-10-22T19:06:26  <jcorgan> encrypt ALL THE THINGS
 714 2015-10-22T19:07:36  <gavinandresen> RE: switching to sqlite: this CACM article was very interesting:  http://cacm.acm.org/magazines/2015/10/192379-crash-consistency/abstract
 715 2015-10-22T19:07:53  <gavinandresen> ... bashes on leveldb a bit, praises sqlite ....
 716 2015-10-22T19:08:09  * maaku is very confused as to what the benefit of switching to sqlite would be for this particular use case
 717 2015-10-22T19:08:43  <jgarzik> maaku: it's maintained and reliable
 718 2015-10-22T19:09:10  <gavinandresen> maaku: assuming sqlite is better about crash consistency (and from all I've read, it is), would cut down on the "core sucks, because my PC crashed and then eleven days to reindex..."
 719 2015-10-22T19:10:21  <sipa> yes, that's the only reason
 720 2015-10-22T19:10:26  <sipa> sqlite is known to be rock solid
 721 2015-10-22T19:10:37  <sipa> however, performance would need to be acceptable
 722 2015-10-22T19:14:35  <maaku> sipa: well, also interop and the ease of using a different sql backend
 723 2015-10-22T19:16:57  <jcorgan> i would definitely advocate making a sql abstraction layer with pluggable back ends, with sqlite as the default
 724 2015-10-22T19:17:26  <CodeShark__> I'm using ODB in my stack for that
 725 2015-10-22T19:17:48  <sipa> jcorgan: to be clear, i think SQL is a downside
 726 2015-10-22T19:18:06  <sipa> we're not looking for a database, and turning things into a database is against what we're trying to achieve
 727 2015-10-22T19:18:20  <jgarzik> nod
 728 2015-10-22T19:18:27  <jcorgan> my secret plan revealed, again
 729 2015-10-22T19:18:29  <sipa> our database is not designed to be accessible by multiple processes
 730 2015-10-22T19:18:39  *** CodeShark has quit IRC
 731 2015-10-22T19:18:41  <sipa> nor do we need any fancy indexes or queries
 732 2015-10-22T19:18:42  *** CodeShark__ has quit IRC
 733 2015-10-22T19:18:49  <gavinandresen> if there's a key-val store as well-supported and rock solid as sqlite..... that's the right answer.
 734 2015-10-22T19:18:51  <jgarzik> I still like the custom db approach, but I'm highly biased
 735 2015-10-22T19:18:52  *** CodeShark has joined #bitcoin-core-dev
 736 2015-10-22T19:19:03  <sipa> if sqlite offered a mechanism to just use its low-level key-value store, i would
 737 2015-10-22T19:19:15  <sipa> (because it's a database layer built on top of a key-value store)
 738 2015-10-22T19:19:46  <jgarzik> program in sqlite VM assembler and you can...
 739 2015-10-22T19:19:48  * jgarzik runs
 740 2015-10-22T19:19:50  <maaku> imho the answer would be to switch to a better maintained descendent of leveldb (there are many)
 741 2015-10-22T19:20:14  <maaku> also minimizes risk of fork
 742 2015-10-22T19:20:18  <gmaxwell> Its unhelpful to have this conversation in parallel with the meeting.
 743 2015-10-22T19:21:48  *** bsm1175321 has joined #bitcoin-core-dev
 744 2015-10-22T19:24:22  *** luke-jr_ is now known as Luke-Jr
 745 2015-10-22T19:26:36  <kanzure_> upstream maintainers, apparently
 746 2015-10-22T19:26:41  <kanzure_> oops, bad scrollback
 747 2015-10-22T19:27:53  *** skyraider has joined #bitcoin-core-dev
 748 2015-10-22T19:30:24  *** crescendo has joined #bitcoin-core-dev
 749 2015-10-22T19:32:36  <bsm1175321> Don't want to interject in the meeting, but I re-ran the SQLite vs. libdb benchmarks sipa mentioned (because they were 6 years old).
 750 2015-10-22T19:32:40  <bsm1175321> Results here: https://gist.github.com/mcelrath/6952eab246a7c705a0fb
 751 2015-10-22T19:35:47  <kanzure_> huh? why should that not be mentioned?
 752 2015-10-22T19:36:06  <bsm1175321> Because the topic has moved on, and the conclusion was to continue research.
 753 2015-10-22T19:39:22  *** kanzure_ is now known as kanzure
 754 2015-10-22T19:44:38  <bsm1175321> Above link updated with 500k records, transactional DB.  (Took a few minutes to run)
 755 2015-10-22T19:47:48  *** MarcoFalke has joined #bitcoin-core-dev
 756 2015-10-22T20:03:28  <BlueMatt> IIRC gmaxwell was gonna ask the crash consistency guys for their test harness?
 757 2015-10-22T20:04:04  <BlueMatt> so we could test other dbs
 758 2015-10-22T20:04:09  <BlueMatt> or maybe I'm crazy
 759 2015-10-22T20:04:14  <BlueMatt> someone should do that, though
 760 2015-10-22T20:04:18  <BlueMatt> before we decide
 761 2015-10-22T20:06:27  <sipa> maaku: sqlite is better at large writes than leveldb
 762 2015-10-22T20:06:27  <maaku> btcdrak: sqlite4 maybe matches leveldb ... because it's basically using leveldb's datastructure under the hood
 763 2015-10-22T20:06:36  <sipa> as leveldb writes everything twice tovdisk
 764 2015-10-22T20:08:20  <btcdrak> gmaxwell: this was the discussion earlier today about sqlite: https://botbot.me/freenode/bitcoin-core-dev/2015-10-22/?msg=52492157&page=2
 765 2015-10-22T20:08:45  <gmaxwell> btcdrak: yes I read it. I am confident from expirence that its incorrect, but I still think jeff's work is very useful.
 766 2015-10-22T20:09:33  <btcdrak> gmaxwell: yeah, I was also surprised, but fingers crossed something impressive comes out of jeff's research.
 767 2015-10-22T20:09:37  <gmaxwell> (in part because sqllite4 may be faster)
 768 2015-10-22T20:09:55  <btcdrak> sqlite4 is still in development though isnt it?
 769 2015-10-22T20:10:03  <gmaxwell> sure, not useful yet.
 770 2015-10-22T20:10:20  <btcdrak> i wonder how far on they are and if they have a release schedule yet
 771 2015-10-22T20:10:24  <Luke-Jr> am I missing something? why is this being treated as a non-consensus thing?
 772 2015-10-22T20:10:47  <sipa> it obviously is consensus critical
 773 2015-10-22T20:10:54  <sipa> it's not a consensus change though
 774 2015-10-22T20:11:02  <Luke-Jr> we don't and can't know that.
 775 2015-10-22T20:11:10  <sipa> recompiling with a different libc is also consensus critical
 776 2015-10-22T20:11:23  *** jtimon has joined #bitcoin-core-dev
 777 2015-10-22T20:11:23  <Luke-Jr> replacing 26kLOC to 140kLOC is impossible to be sure on
 778 2015-10-22T20:11:24  <BlueMatt> Luke-Jr: we should be confident that it is not before anything moves forward
 779 2015-10-22T20:11:27  <sipa> or running on a new linix kernel
 780 2015-10-22T20:12:08  <Luke-Jr> BlueMatt: I don't think we can.
 781 2015-10-22T20:12:13  <helo> i really appreciate the meetings, great idea (btcdrak?)
 782 2015-10-22T20:12:23  <btcdrak> helo
 783 2015-10-22T20:12:31  <sipa> we are making certain assumptions from other software we are using
 784 2015-10-22T20:12:32  <BlueMatt> Luke-Jr: then we're stuck with a shitty db that corrupts randomly and isnt self-consistent (ie not consensus-compatible)
 785 2015-10-22T20:12:36  <Luke-Jr> best I can see happening, is knowing it's unlikely to have any accidentally-triggered failures
 786 2015-10-22T20:12:38  <gmaxwell> Luke-Jr: it's not being treated as anything right now, it's just a test.
 787 2015-10-22T20:13:00  <sipa> that includes compilers, libraries, and operation systems with _well specified_ behaviour
 788 2015-10-22T20:13:03  <gmaxwell> Luke-Jr: by that basis we don't know that leveldb is self-consistent, or consistent between windows / unix, etc.
 789 2015-10-22T20:13:04  <maaku> Other than Windows corruption errors, is this solving any real problems right now?
 790 2015-10-22T20:13:07  <btcdrak> Luke-Jr: research is a good thing.
 791 2015-10-22T20:13:14  *** tombtc has joined #bitcoin-core-dev
 792 2015-10-22T20:13:26  <gmaxwell> maaku: we have durability problems on all platforms, though the windows ones are very severe.
 793 2015-10-22T20:13:30  <Luke-Jr> btcdrak: yes, if this is just research fine
 794 2015-10-22T20:14:01  <gmaxwell> The performance of leveldb is also problematic already, (well reported for very large databases), enough that our fat caching layer is very very criticial for performance, and thats causing us other problems.
 795 2015-10-22T20:14:31  <gmaxwell> maaku: durability problems in this database are incompatible with the survival of a network where puning is widely used.
 796 2015-10-22T20:14:55  <sipa> puny network
 797 2015-10-22T20:15:02  <gmaxwell> hah
 798 2015-10-22T20:15:05  <gmaxwell> pruning*
 799 2015-10-22T20:15:10  <btcdrak> LOL
 800 2015-10-22T20:16:18  <wumpus> in general leveldb works fine; greatest concern is that it seems to be no longer maintained, so any bugs (like corruption on crashes in windows) is unlikely to get solved
 801 2015-10-22T20:16:28  <maaku> i thought gmaxwell was declaring sipa a risk to bitcoin there
 802 2015-10-22T20:17:36  <gmaxwell> maaku: well that much is obvious.
 803 2015-10-22T20:17:37  <sipa> we couls try to find another win env for leveldb
 804 2015-10-22T20:17:37  <sipa> there may be bugs in that
 805 2015-10-22T20:18:42  <jonasschnelli> i know this sounds hard: but did we never consider to stop supporting windows? Would this be so wrong?
 806 2015-10-22T20:18:45  <wumpus> I'm sure it is possible to fix the windows issue, but it's not really nice to inherit a dependency and suddenly have to maintain it
 807 2015-10-22T20:19:31  <Luke-Jr> jonasschnelli: I think it would be problematic to do so right now.
 808 2015-10-22T20:19:40  <Luke-Jr> jonasschnelli: as bad as it is, many people still use it
 809 2015-10-22T20:19:44  <gmaxwell> jonasschnelli: I think it would be a bad idea, its a very widely used platform and dropping it would be a major move against the ability of people to independnantly run, audit, etc. the bitcoin system.
 810 2015-10-22T20:19:48  <wumpus> jonasschnelli: well I've been close to declaring that. it wouldn't be pretty but we have zero active developers for windows.
 811 2015-10-22T20:20:00  <gmaxwell> The security problems of windows stink, but hardware wallets are a tidy tool to address that.
 812 2015-10-22T20:20:22  <wumpus> jonasschnelli: then again, the software is working for a lot of people also on windows
 813 2015-10-22T20:20:24  <jonasschnelli> not even speaking for the wallet. Just the node/core itself.
 814 2015-10-22T20:20:37  <gmaxwell> we are in a state right now where it is is almost constructively non-supported by our failure to do a a great job about it.
 815 2015-10-22T20:20:38  <maaku> well to be clear, gmaxwell's concerns about a pruned network wouldn't apply if the durability issues were wholly isolated to the windows platform
 816 2015-10-22T20:20:39  <Luke-Jr> jonasschnelli: then people will just stop running a node :/
 817 2015-10-22T20:21:01  <Luke-Jr> before we drop Windows support, we would ideally want the alternative to be "run a node on Linux", not "stop running a node altogether"
 818 2015-10-22T20:21:01  <wumpus> gmaxwell: I disagree
 819 2015-10-22T20:21:04  <gmaxwell> maaku: they are not wholly isolated to windows, though perhaps its safe enough elsewhere that my concern is less of an issue.
 820 2015-10-22T20:21:05  <jonasschnelli> wumpus: Agree. It's running. But i think we should not move away form leveldb because of windows...
 821 2015-10-22T20:21:13  <wumpus> gmaxwell: many people are running it on windows, and not complaining at all
 822 2015-10-22T20:21:30  <wumpus> gmaxwell: no need to blow this issue out of proportion
 823 2015-10-22T20:21:34  <gmaxwell> wumpus: and many people have stopped running it too.
 824 2015-10-22T20:21:39  <wumpus> gmaxwell: obviously
 825 2015-10-22T20:22:00  <gmaxwell> I believe more people have stopped running bitcoin core on windows than currently run it.
 826 2015-10-22T20:22:13  <gmaxwell> (stopped due to corruption issues of varrious forms)
 827 2015-10-22T20:22:36  <wumpus> well this is an open source project - if we don't have developers supporting windows, we can't support windows
 828 2015-10-22T20:22:41  <jonasschnelli> are most corruption issues caused by leveldbs layer?
 829 2015-10-22T20:22:43  <gmaxwell> (not just leveldb but also AV)
 830 2015-10-22T20:22:51  <sipa> jonasschnelli: we don't know
 831 2015-10-22T20:23:19  <gmaxwell> We know there is leveldb corruption on unclean shutdown, and we know about antivirus mediated corruption. But there may be more things we don't know about.
 832 2015-10-22T20:23:30  <wumpus> if the corruption issues are as bad as you say they are, and no one is fixing them,  then we should indeed drop windows support
 833 2015-10-22T20:23:35  <Luke-Jr> maybe if we stop supporting Windows "officially", but leave it open, others will step up to do it
 834 2015-10-22T20:23:35  <wumpus> but I don't believe you
 835 2015-10-22T20:23:45  <gmaxwell> wumpus: or we should get more windows developers.
 836 2015-10-22T20:23:51  <wumpus> gmaxwell: well, good luck with that
 837 2015-10-22T20:23:54  <gmaxwell> Which can be done.
 838 2015-10-22T20:24:09  <gmaxwell> wumpus: I mean I can _hire_ people for this, but it hasn't been on my radar.
 839 2015-10-22T20:24:13  <wumpus> in any case, this is not constructive *goes back to sleep*
 840 2015-10-22T20:24:19  <jonasschnelli> we could ship it together with a vmbox ubuntu. *duck*
 841 2015-10-22T20:24:25  * Luke-Jr wonders if Diapolo is good enough at C++ that he could transition to more than just GUI if he gets funding
 842 2015-10-22T20:24:39  * jonasschnelli looks strange at Luke-Jr
 843 2015-10-22T20:24:45  <gmaxwell> wumpus: :-/ I think you're misunderstanding my perspective, sorry I've communicated poorly.
 844 2015-10-22T20:24:53  <Luke-Jr> jonasschnelli: got a better suggestion? :P
 845 2015-10-22T20:25:02  <Luke-Jr> I don't know many Windows developers..
 846 2015-10-22T20:25:05  <jonasschnelli> Luke-Jr: hah. No. Not really. :)
 847 2015-10-22T20:25:48  <jonasschnelli> serious: what would be the overhead to run bitcoin-qt/core in a vm on windows?
 848 2015-10-22T20:25:49  <Luke-Jr> maybe the people who did those libconsensus bindings?
 849 2015-10-22T20:26:03  <wumpus> gmaxwell: it's not impossible to just solve the windows crash corruption issue
 850 2015-10-22T20:26:04  <jonasschnelli> 10% CPU loss, 512MB ram overhead?
 851 2015-10-22T20:26:09  <Luke-Jr> jonasschnelli: I would be surprised if a VM kill reproduced the problem
 852 2015-10-22T20:26:17  <wumpus> gmaxwell: I even proposed to help with it today if someone can send me a corrupted database
 853 2015-10-22T20:26:29  <gmaxwell> wumpus: yup. did anyone do that?
 854 2015-10-22T20:27:36  <wumpus> gmaxwell: no
 855 2015-10-22T20:27:37  <jonasschnelli> Luke-Jr: the idea would be to bundle bitcoin-qt/core on windows together with vbox and a tiny distro. This would eliminate some ugly platform dependents.
 856 2015-10-22T20:27:37  <Luke-Jr> gmaxwell: I tried to, but couldn't reproduce it with IBD, and shortly thereafter my USB Armory died entirely. So it seems likely my problem was a bad microSD
 857 2015-10-22T20:27:58  <Luke-Jr> jonasschnelli: that sounds like a terrible UX
 858 2015-10-22T20:28:09  <jonasschnelli> Luke-Jr: the UX could still be native (once it's decoupled)
 859 2015-10-22T20:28:09  <wumpus> Luke-Jr: yes, your problem is likely adifferent one. You're the only one reporting it on linu
 860 2015-10-22T20:28:57  <wumpus> jonasschnelli: if you go that far to make a bundle you may as well fix the windows version :-) (it would be about as much development work)
 861 2015-10-22T20:29:24  <gmaxwell> jonasschnelli: doesn't jive really well with the resource costs of running a node.... VM with the constantly expanding storage is no fun. :P plus overheads. :P
 862 2015-10-22T20:29:44  <Luke-Jr> does UML support Windows? ;)
 863 2015-10-22T20:29:47  <gmaxwell> Fixing the windows shouldn't be a big deal once we get enough repros and data, similar to OSX.
 864 2015-10-22T20:29:54  <gmaxwell> sounded like it might be the same issue.
 865 2015-10-22T20:30:06  <wumpus> yeah, let's just solve the database problem instead of trying to work around it
 866 2015-10-22T20:30:29  <wumpus> even if we have to put a flush after every line in leveldb ...
 867 2015-10-22T20:30:34  <gmaxwell> I considered the exploration of other options to be orthorgonal with windows being on fire FWIW.
 868 2015-10-22T20:30:34  <jonasschnelli> gmaxwell: nowadays VM overhead is tiny... it's not perfect. But better a solution that works with a lost of 10-20% in overhead than unsolved corrupted databases.
 869 2015-10-22T20:30:58  <wumpus> gmaxwell: sure, I still like the idea of exploring other databases
 870 2015-10-22T20:31:13  <wumpus> but yes, orthogonally
 871 2015-10-22T20:31:20  <sipa> i really like the idea of just being able to use a maintained and tested database
 872 2015-10-22T20:31:23  <gmaxwell> (also the attempt may turn up bugs elsewhere in our codebase)
 873 2015-10-22T20:31:37  <wumpus> don't we all
 874 2015-10-22T20:31:43  <sipa> and not a hack we had to pull together
 875 2015-10-22T20:31:45  * wumpus thought leveldb was that
 876 2015-10-22T20:31:46  *** paveljanik has quit IRC
 877 2015-10-22T20:31:51  <gavinandresen> a bug bounty worked for the last leveldb corruption issue we had (if i recall correctly). I'm still holding some bitcoin in the core dev expenses fund
 878 2015-10-22T20:31:56  <sipa> wumpus: leveldb windows certainly isn't
 879 2015-10-22T20:32:02  <Luke-Jr> jonasschnelli: if someone really wants to investigate VM stuff, I'd suggest instead a thin OS that runs *Windows* in the VM, and provides a hardwarewallet-like interface on top
 880 2015-10-22T20:32:04  <jonasschnelli> is there an approach to test sqlites performance which represents our db-style?
 881 2015-10-22T20:32:19  <sipa> wumpus: we have local modifications to the win env
 882 2015-10-22T20:32:30  <wumpus> jonasschnelli: best would be to just try it
 883 2015-10-22T20:32:41  <gmaxwell> jonasschnelli: yes, testing bitcoin core using <alternative database> :P  it shouldn't be that much work; presumably jeff will report back on that soon
 884 2015-10-22T20:33:17  <jonasschnelli> sqlite could initially be slower. But it's better maintained and much more portable.
 885 2015-10-22T20:33:39  <sipa> jonasschnelli: there is slower and slower
 886 2015-10-22T20:33:50  <jonasschnelli> Indeed
 887 2015-10-22T20:33:53  <sipa> if it's slower for things our caches compensates for, who cares
 888 2015-10-22T20:33:54  <gmaxwell> jonasschnelli: speed is a security consideration for us currently, if you were talking about 5% or something I'd agree. but thats now what I expect.
 889 2015-10-22T20:34:09  <gmaxwell> s/now/not/
 890 2015-10-22T20:34:09  <sipa> if it's slower to the point that it affects block propagation, it's a no go
 891 2015-10-22T20:34:33  <gmaxwell> s/currently/sadly/
 892 2015-10-22T20:34:35  * wumpus doubts it will make much of a difference
 893 2015-10-22T20:34:39  <jonasschnelli> Quote from sqlite4: The default built-in storage engine is a log-structured merge database. It is very fast, faster than LevelDB, supports nested transactions, and stores all content in a single disk file. Future versions of SQLite4 might also include a built-in B-Tree storage engine.
 894 2015-10-22T20:34:57  <jonasschnelli> promising. :)
 895 2015-10-22T20:35:07  <wumpus> there are so many factors influencing performance of bitcoind, that a slightly slower database won't be a big issue
 896 2015-10-22T20:35:14  <sipa> yes, sqlite4 sounds awesome _once_ it has had the same amount of testing and battle hardending as sqlite3
 897 2015-10-22T20:35:20  <wumpus> jonasschnelli: yes that was pasted before today :)
 898 2015-10-22T20:35:50  <jonasschnelli> ha.. okay. I'd love to play around with it. But my stack of things to work down is just to big right now.
 899 2015-10-22T20:36:13  <gmaxwell> we might also want to maintain a patch for sqllite4 and use it to try to get the sqllite devs to use us as a test harness. :P
 900 2015-10-22T20:36:43  <jonasschnelli> sipa: sqlite is widely used, also in almost every browser (local dbs) and smartphone (android and iOS IIRC). They very likely to update once to sqlite4...
 901 2015-10-22T20:36:45  *** Greg__ has left #bitcoin-core-dev
 902 2015-10-22T20:38:18  <sipa> but quote from one of the sqlite devs a few months ago "sqlite4 is a dev toy"
 903 2015-10-22T20:38:53  <jonasschnelli> sipa: that sounds perfect for our bitcoin <1.0 version. :)
 904 2015-10-22T20:39:18  <sipa> jonasschnelli: if only we didn't have this pesky economy thing that relies on bitcoin
 905 2015-10-22T20:39:47  <jonasschnelli> sipa: na... that are just some fiat exchange bubbles.. :)
 906 2015-10-22T20:39:56  <gmaxwell> Lets fix that first then. :P
 907 2015-10-22T20:40:02  <wumpus> trying it out doesn't hurt, no one is talking about releasing with it...
 908 2015-10-22T20:40:12  <sipa> wumpus: exactly
 909 2015-10-22T20:40:14  <sipa> dev toy :p
 910 2015-10-22T20:40:28  <jonasschnelli> wumpus: +1 ... and a such pr would at least take 1/2 year to get in.
 911 2015-10-22T20:40:39  <gmaxwell> well as I said, might even be possible to get the sqllite developers to test using us. We're a pretty cool load generator (esp with signatures off)
 912 2015-10-22T20:41:08  <sipa> gmaxwell: we're boring
 913 2015-10-22T20:41:20  <sipa> single-threaded access, only bulk writes and small random reads
 914 2015-10-22T20:48:33  *** tombtc has quit IRC
 915 2015-10-22T21:10:51  <jcorgan> < Luke-Jr> jonasschnelli: if someone really wants to investigate VM stuff, I'd suggest instead a thin OS that runs *Windows* in the VM
 916 2015-10-22T21:11:06  <jcorgan> *cough* Qubes OS *cough*
 917 2015-10-22T21:11:24  <Luke-Jr> jcorgan: Qubes OS is not very thin, and doesn't do GPU passthrough
 918 2015-10-22T21:12:07  <jcorgan> didn't think about GPU
 919 2015-10-22T21:12:18  <jcorgan> but thinness is poorly defined
 920 2015-10-22T21:12:27  <btcdrak> sipa: are the tests in #6816 (versionbits) enough? I was thinking it would be good to have some rpc-tests there as well, generating blocks and running through a couple of scenarios.
 921 2015-10-22T21:13:11  <btcdrak> or is that overkill?
 922 2015-10-22T21:13:34  *** jl2012 has quit IRC
 923 2015-10-22T21:21:12  <CodeShark> btcdrak: I think we should probably do some regtests with the integration, not necessarily over RPC
 924 2015-10-22T21:21:24  <CodeShark> well...
 925 2015-10-22T21:21:31  <CodeShark> other than just generating blocks
 926 2015-10-22T21:21:43  <CodeShark> I guess just calling generate and getblockchaininfo
 927 2015-10-22T21:26:17  <btcdrak> CodeShark: I mean tests like https://github.com/bitcoin/bitcoin/blob/master/qa/rpc-tests/bipdersig.py etc
 928 2015-10-22T21:27:29  <jgarzik> It builds: https://github.com/jgarzik/bitcoin/tree/2015_sqlite
 929 2015-10-22T21:28:07  <btcdrak> jgarzik: That was fast
 930 2015-10-22T21:28:28  <CodeShark> jgarzik: nicely done :)
 931 2015-10-22T21:28:40  <sipa> yes, but does it run linyx?
 932 2015-10-22T21:30:07  <CodeShark> btcdrak: the -blockversion thing will probably not be a good idea anymore using versionbits
 933 2015-10-22T21:30:35  <CodeShark> it will be better to provide a list of BIPs you don't want to set the bit for
 934 2015-10-22T21:30:40  <jgarzik> a few bits can be forwarded upstream immediately, like Seek* and Prev() removal
 935 2015-10-22T21:31:52  <btcdrak> CodeShark: well obviously the bipdersig.py tests are for an ISM sf.
 936 2015-10-22T21:32:36  <btcdrak> CodeShark: I'm just saying a set of tests like that which generate blocks and simulate a sf rollout but using versionbits protocol.
 937 2015-10-22T21:32:43  <CodeShark> I suppose we can just use VERSION_HIGH_BITS and VERSION_HIGH_BITS | 0x1 instead of versions 2 and 3 :)
 938 2015-10-22T21:33:08  <Luke-Jr> CodeShark: does versionbits sanitise the encoding btw? ;)
 939 2015-10-22T21:33:18  <Luke-Jr> ie, define the first 32-bits to be a big endian number
 940 2015-10-22T21:33:36  <maaku> Luke-Jr: it's little endian
 941 2015-10-22T21:33:44  <Luke-Jr> would be nice to fix that at the same time
 942 2015-10-22T21:33:53  <CodeShark> encoding? versionbits doesn't deal with serialization stuff
 943 2015-10-22T21:33:58  <btcdrak> CodeShark: it makes the most conclusive tests that it works, and provides some protection against regressions.
 944 2015-10-22T21:34:11  <maaku> Luke-Jr: why? nothing else is big endian
 945 2015-10-22T21:34:14  <Luke-Jr> CodeShark: anything consensus-relevant must deal with encoding
 946 2015-10-22T21:34:27  <maaku> Luke-Jr: he means versionbits doesn't touch serialization
 947 2015-10-22T21:34:33  <CodeShark> Luke-Jr: versionbits deals with block header data that has already been deserialized
 948 2015-10-22T21:34:38  <Luke-Jr> maaku: big endian is standard for protocols; and hashes at least are
 949 2015-10-22T21:34:54  <sipa> Luke-Jr: please
 950 2015-10-22T21:35:05  <sipa> we're not changing the serialization of block headers
 951 2015-10-22T21:36:46  <jgarzik> big endian is dead
 952 2015-10-22T21:37:14  <sipa> no endianness flamewar please
 953 2015-10-22T21:37:23  <CodeShark> in any case, the serialization format is entirely a separate issue...versionbits will play no part in that either way
 954 2015-10-22T21:39:01  <Luke-Jr> so the bits are numbered 7 6 5 4 3 2 1 15 14 13 12 11 10 9 8 23 22 21 20 19 18 17 16 31 30 29 28 27 26 25 24 ?
 955 2015-10-22T21:39:12  <sipa> irrelevant
 956 2015-10-22T21:39:15  <Luke-Jr> …
 957 2015-10-22T21:39:57  <CodeShark> versionbits deals with ints, uint32_t, etc...
 958 2015-10-22T21:39:59  <Luke-Jr> it's relevant because if I set the wrong BIP, it won't work.
 959 2015-10-22T21:40:00  <sipa> the block header serialization defines nVersion as a 32-bit little-endian signed integer
 960 2015-10-22T21:40:02  *** aj has joined #bitcoin-core-dev
 961 2015-10-22T21:40:08  <Luke-Jr> wrong bit*
 962 2015-10-22T21:40:10  <CodeShark> it doesn't care what the underlying representation is for the particular architecture
 963 2015-10-22T21:40:17  <sipa> versionbits changes the semantics of the nVersion integer
 964 2015-10-22T21:40:49  <CodeShark> by the time versionbits comes into play, the version field has already been decoded into an int
 965 2015-10-22T21:41:30  <Luke-Jr> CodeShark: that is an implementation detail. the whole point of version bits is that it is no longer an int, but a bit array
 966 2015-10-22T21:41:45  <CodeShark> to test a bit, you do (nVersion >> bit) & 0x1
 967 2015-10-22T21:41:50  <sipa> Luke-Jr: it's still an int; an int in which particular bits are set
 968 2015-10-22T21:41:54  * Luke-Jr sighs.
 969 2015-10-22T21:42:02  <sipa> now please stop this silly discussion, we're not redefining how integers work
 970 2015-10-22T21:42:34  <gmaxwell> but, base-3!!!
 971 2015-10-22T21:42:57  <Luke-Jr> fine, not worth the time to argue about this stupid design decision.
 972 2015-10-22T21:43:46  *** sipa has left #bitcoin-core-dev
 973 2015-10-22T21:45:23  <CodeShark> base-3 would allow us to do yea/nay/abstain :p
 974 2015-10-22T21:46:34  *** MarcoFalke has quit IRC
 975 2015-10-22T21:46:52  *** MarcoFalke has joined #bitcoin-core-dev
 976 2015-10-22T21:52:17  <Luke-Jr> the annoying part will be having to answer questions in a few years why it's misordered such. like I had to do with the getwork data for years
 977 2015-10-22T21:55:12  <phantomcircuit> jgarzik, you have benchmarks for sqlite?
 978 2015-10-22T21:55:49  <phantomcircuit> my experience using it in the past has been that it's slow as hell for inserts or any kind of concurrent access
 979 2015-10-22T21:56:41  <phantomcircuit> probably time would be better spent on building an interface spec for databases
 980 2015-10-22T21:58:35  <CodeShark> for insertions we could just use a raw file - the blockchain is a linear structure anyhow. then all we would need to do in sqlite is keep track of file positions ;)
 981 2015-10-22T22:03:05  <phantomcircuit> CodeShark, utxo only
 982 2015-10-22T22:04:03  <CodeShark> yeah, so we maintain a utxo index
 983 2015-10-22T22:04:11  <CodeShark> or are you talking about a pruned node?
 984 2015-10-22T22:04:19  <phantomcircuit> the index would be roughly the same size as the utxo data :P
 985 2015-10-22T22:04:30  <CodeShark> lol
 986 2015-10-22T22:05:11  <CodeShark> ok, for a utxo database perhaps it doesn't make so much sense
 987 2015-10-22T22:05:25  <phantomcircuit> yeah the actual data is really small
 988 2015-10-22T22:06:47  <gmaxwell> CodeShark: txindex works like you suggest there.
 989 2015-10-22T22:07:55  <gmaxwell> but for utxo set we get benefits from reducing the working set. saving 20 bytes of scriptpubkey/value in exchange for another random seek to the middle of some huge block file where 2/3rd of the data is not utxo relevant, basically defeates disk caching entirely. :)
 990 2015-10-22T22:11:59  <GitHub160> [bitcoin] petertodd opened pull request #6871: Full-RBF with opt-out (master...2015-10-rbf-with-opt-out) https://github.com/bitcoin/bitcoin/pull/6871
 991 2015-10-22T22:22:13  *** skyraider has quit IRC
 992 2015-10-22T22:33:19  <bsm1175321> G
 993 2015-10-22T22:51:11  *** bsm1175321 has quit IRC
 994 2015-10-22T22:51:23  *** bsm1175321 has joined #bitcoin-core-dev
 995 2015-10-22T22:53:29  <GitHub51> [bitcoin] TheBlueMatt opened pull request #6872: Remove UTXO cache entries when the tx they were added for is removed/does not enter mempool (master...limitucache) https://github.com/bitcoin/bitcoin/pull/6872
 996 2015-10-22T22:53:52  *** sipa has joined #bitcoin-core-dev
 997 2015-10-22T23:00:43  *** MarcoFalke has quit IRC
 998 2015-10-22T23:05:01  *** dcousens has joined #bitcoin-core-dev
 999 2015-10-22T23:05:19  <dcousens> petertodd: I hope you don't mind my questions :)
1000 2015-10-22T23:06:19  <phantomcircuit> BlueMatt, the last commit on #6872 seems to evict from the cache things which other transactions in the mempool want cached
1001 2015-10-22T23:06:36  <phantomcircuit> ie AcceptToMempool fails because it's a double spend of something with a higher feerate
1002 2015-10-22T23:08:02  <petertodd> dcousens: no worries!
1003 2015-10-22T23:08:10  <phantomcircuit> scratch that
1004 2015-10-22T23:08:13  <phantomcircuit> i missed the check
1005 2015-10-22T23:08:14  <petertodd> dcousens: I'm doing three things at once so the responses are probably sounding a bit terse :)
1006 2015-10-22T23:08:20  <BlueMatt> phantomcircuit: ahh, ok, good, i was about to say
1007 2015-10-22T23:08:33  <dcousens> petertodd: all good, I read everything in good spirit, just hope it comes across the same :)
1008 2015-10-22T23:18:16  *** molly has joined #bitcoin-core-dev
1009 2015-10-22T23:21:19  *** moli has quit IRC
1010 2015-10-22T23:40:57  *** sipa has quit IRC
1011 2015-10-22T23:41:05  *** dcousens has quit IRC
1012 2015-10-22T23:42:08  *** sipa has joined #bitcoin-core-dev
1013 2015-10-22T23:56:57  *** sipa has quit IRC
1014 2015-10-22T23:56:57  *** sipa has joined #bitcoin-core-dev