1 2015-11-21T00:06:52 <gmaxwell> lol, luke's #6851 reduces the time to add 53k tx to the wallet from 843 seconds to 35 seconds. 2 2015-11-21T00:32:48 *** ParadoxSpiral has quit IRC 3 2015-11-21T00:36:50 *** tripleslash_x has quit IRC 4 2015-11-21T00:37:09 <GitHub179> [bitcoin] pstratem closed pull request #6966: [WIP] Wallet: Cache CWalletDB pointer in CWallet to improve performance (master...wallet_speedup) https://github.com/bitcoin/bitcoin/pull/6966 5 2015-11-21T00:38:20 *** tripleslash has joined #bitcoin-core-dev 6 2015-11-21T00:51:27 <gmaxwell> Luke-Jr: around? 7 2015-11-21T00:51:31 <Luke-Jr> gmaxwell: ? 8 2015-11-21T00:54:31 <phantomcircuit> gmaxwell, comment changed and merged petertodd's fRelayTxs rpc addition 9 2015-11-21T00:55:10 *** randy-waterhouse has joined #bitcoin-core-dev 10 2015-11-21T01:05:09 *** belcher has joined #bitcoin-core-dev 11 2015-11-21T01:15:29 *** jtimon_ has quit IRC 12 2015-11-21T01:28:26 *** CodeShark has joined #bitcoin-core-dev 13 2015-11-21T02:18:54 *** gavinandresen has joined #bitcoin-core-dev 14 2015-11-21T02:18:57 *** morcos_ has joined #bitcoin-core-dev 15 2015-11-21T02:18:59 *** nanotube has quit IRC 16 2015-11-21T02:19:00 *** tripleslash has quit IRC 17 2015-11-21T02:19:02 *** Anduck_ has joined #bitcoin-core-dev 18 2015-11-21T02:19:04 *** morcos has quit IRC 19 2015-11-21T02:19:05 *** gavinand1esen has quit IRC 20 2015-11-21T02:19:06 *** Anduck has quit IRC 21 2015-11-21T02:19:57 *** tripleslash has joined #bitcoin-core-dev 22 2015-11-21T02:22:21 *** skyraider has quit IRC 23 2015-11-21T02:28:07 *** nanotube has joined #bitcoin-core-dev 24 2015-11-21T02:34:28 *** Ylbam has quit IRC 25 2015-11-21T02:35:15 *** zookolaptop has quit IRC 26 2015-11-21T02:38:59 <GitHub132> [bitcoin] Har01d opened pull request #7072: [RPC] Add transaction size to JSON output (master...master) https://github.com/bitcoin/bitcoin/pull/7072 27 2015-11-21T02:53:07 <midnightmagic> gmaxwell: http://0bin.net/paste/vKUjDmnyTyWAkxP2#gkQ4nb+AGhl0QZKpjE1W0oaQqM0ElXzZMseXDp+fwmY corruption.. 28 2015-11-21T02:53:38 <GitHub123> [bitcoin] gmaxwell pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/776848acefa8...616d61b20d56 29 2015-11-21T02:53:39 <GitHub123> bitcoin/master 3e7c891 Luke Dashjr: Optimisation: Store transaction list order in memory rather than compute it every need... 30 2015-11-21T02:53:39 <GitHub123> bitcoin/master 616d61b Gregory Maxwell: Merge pull request #6851... 31 2015-11-21T02:53:40 <GitHub42> [bitcoin] gmaxwell closed pull request #6851: Optimisation: Store transaction list order in memory rather than compute it every need (master...opti_txorder) https://github.com/bitcoin/bitcoin/pull/6851 32 2015-11-21T02:54:16 <gmaxwell> midnightmagic: interesting, so that had been running without interruption and just crashed while syncing? 33 2015-11-21T02:55:19 <midnightmagic> gmaxwell: I did a -reindex and -connect to an internal node. 34 2015-11-21T02:55:49 <midnightmagic> bitcoind -connect=x.y.z.a -listen -daemon -nodnsseed -nodns -reindex 35 2015-11-21T02:56:12 <gmaxwell> I guess I need to fix the ppc host to see if I can repro. 36 2015-11-21T02:56:24 <midnightmagic> Come to think of it.. I'm not sure this machine has ever reliably sync'd to head. 37 2015-11-21T02:56:56 <gmaxwell> midnightmagic: I think before you might have been failing when it was fine for me. 38 2015-11-21T02:57:22 <midnightmagic> gmaxwell: Yes. At one point it seemed to be coincident with a hard-disconnect of the network in the middle of a block transfer. 39 2015-11-21T02:58:42 <midnightmagic> Perhaps this machine is just flakey. 40 2015-11-21T03:09:40 <midnightmagic> no other evidence of a machine problem. everything else has been running for 300+ days 41 2015-11-21T03:10:43 <midnightmagic> drat. another one. 42 2015-11-21T03:10:46 <midnightmagic> http://0bin.net/paste/Jf7jsnULbX4jHJIU#TlLnA8uBw6uxDrYJicVgeOgtgfN37ouH51otjBRT2jC 43 2015-11-21T03:17:10 <midnightmagic> ooo tor stream isolation 44 2015-11-21T03:23:16 <gmaxwell> woot, dual G5 is back up. 45 2015-11-21T03:29:02 *** belcher has quit IRC 46 2015-11-21T03:35:54 *** go1111111 has quit IRC 47 2015-11-21T03:49:14 *** go1111111 has joined #bitcoin-core-dev 48 2015-11-21T03:54:32 <midnightmagic> ooo fascinating. 49 2015-11-21T03:55:36 *** challisto has joined #bitcoin-core-dev 50 2015-11-21T03:55:37 *** challisto has joined #bitcoin-core-dev 51 2015-11-21T03:55:45 <midnightmagic> http://0bin.net/paste/mK4mOOxCYGHmdN7a#6N5+i3ta5ZuVpyVBPX3Uk5NVna+MZUvDvCz5EGc7cTx 52 2015-11-21T04:13:45 <btcdrak> that's weird 53 2015-11-21T04:17:35 * midnightmagic shrugs and reindexes the main node. 54 2015-11-21T04:21:52 *** MarcoFalke has quit IRC 55 2015-11-21T04:24:13 <gmaxwell> in any case, my 2xG5 host seems to be catching up fine with git master.. taking like 10 seconds for some blocks. :) 56 2015-11-21T05:10:59 *** arowser has quit IRC 57 2015-11-21T05:56:39 *** tulip has joined #bitcoin-core-dev 58 2015-11-21T06:16:35 *** guest234234 has quit IRC 59 2015-11-21T07:39:26 <midnightmagic> what magic incantation do you do to make it go fast 60 2015-11-21T07:46:47 *** ParadoxSpiral has joined #bitcoin-core-dev 61 2015-11-21T07:50:07 *** lecusemble has quit IRC 62 2015-11-21T07:53:45 *** d_t has quit IRC 63 2015-11-21T07:56:52 *** lecusemble has joined #bitcoin-core-dev 64 2015-11-21T08:24:43 *** moli has quit IRC 65 2015-11-21T08:35:12 *** guest234234 has joined #bitcoin-core-dev 66 2015-11-21T09:11:05 *** Ylbam has joined #bitcoin-core-dev 67 2015-11-21T10:03:46 *** dgenr8 has quit IRC 68 2015-11-21T10:04:13 *** dgenr8 has joined #bitcoin-core-dev 69 2015-11-21T10:07:24 *** randy-waterhouse has quit IRC 70 2015-11-21T10:08:06 *** dcousens has joined #bitcoin-core-dev 71 2015-11-21T10:39:20 *** dcousens has quit IRC 72 2015-11-21T11:02:03 *** jtimon has joined #bitcoin-core-dev 73 2015-11-21T11:35:19 *** PaulCapestany has quit IRC 74 2015-11-21T11:37:01 *** PaulCapestany has joined #bitcoin-core-dev 75 2015-11-21T12:03:27 *** guest234234 has quit IRC 76 2015-11-21T12:35:30 *** belcher has joined #bitcoin-core-dev 77 2015-11-21T12:57:55 *** guest234234 has joined #bitcoin-core-dev 78 2015-11-21T13:05:03 *** moli has joined #bitcoin-core-dev 79 2015-11-21T13:35:32 *** tulip has quit IRC 80 2015-11-21T13:44:18 <jtimon> morcos the linked commit doesn't touch TrimToSize, that's the next one 81 2015-11-21T13:46:05 *** davec has quit IRC 82 2015-11-21T13:46:35 <jtimon> ACK on policy estimator called directly instead of through the mempool (but I've been asked not to worked not to work on policy refactors for now) and that one is certainly disruptive 83 2015-11-21T13:47:40 <jtimon> setting the attribute by calling TrimToSize seems weird, what's wrong with doing it in the constructor (or in a setter if necessary)? 84 2015-11-21T13:48:16 <jtimon> morcos_: the estimator is currently not asking questions to the mempool and it doesn't need to, please don't couple that again 85 2015-11-21T13:55:28 <jtimon> you want the mempool decoupled from policy/fees and I agree, and have done it already in a "private" outdated branch 86 2015-11-21T13:55:54 <jtimon> but to do so, it is not necessary to couple policy/fees to the mempool instead, they can be completely independent 87 2015-11-21T13:56:49 <jtimon> whenever we think is the moment, I can completely decouple them 88 2015-11-21T14:05:41 *** jtimon has quit IRC 89 2015-11-21T14:29:56 *** Guyver2 has joined #bitcoin-core-dev 90 2015-11-21T14:41:42 *** Anduck_ is now known as Anduck 91 2015-11-21T15:02:32 *** morcos_ is now known as morcos 92 2015-11-21T15:04:02 <morcos> jtimon: the question is where shoould the logic live in smartly estimate fees. in the policyestimator. that logic requires asking question of the mempool. otherwise you're going to have to put that logic repeated in several different places in wallet and gui. 93 2015-11-21T15:04:21 <morcos> s/live in/live to/ 94 2015-11-21T15:06:08 <morcos> jtimon: the reason I like setting the mempool size in TrimToSize is you can easily imagine logic where the size is non-constant. So after you have trimmed it the mempool knows what size it currently is (the logic as to what size to trim it lives outside mempool) 95 2015-11-21T15:06:31 <morcos> for instance we discussed using less size for the mempool during IBD so you could use more of your allocated memory for the dbcache 96 2015-11-21T15:06:39 <morcos> or you can imagine other scenarios 97 2015-11-21T15:24:49 *** zookolaptop has joined #bitcoin-core-dev 98 2015-11-21T15:26:15 *** guest234234 has quit IRC 99 2015-11-21T15:35:21 *** trippysalmon has joined #bitcoin-core-dev 100 2015-11-21T15:44:53 *** trippysalmon has quit IRC 101 2015-11-21T15:57:10 *** davec has joined #bitcoin-core-dev 102 2015-11-21T16:09:20 *** JackH has joined #bitcoin-core-dev 103 2015-11-21T16:25:35 *** zmanian_ has quit IRC 104 2015-11-21T16:32:35 *** zookolaptop has quit IRC 105 2015-11-21T16:33:30 *** davec has quit IRC 106 2015-11-21T16:35:43 *** MarcoFalke has joined #bitcoin-core-dev 107 2015-11-21T16:49:19 <gmaxwell> 1h 11m to rescan a wallet with 11.7m transactions now. 108 2015-11-21T16:51:19 <sipa> how long for a wallet with 0? 109 2015-11-21T16:53:22 <gmaxwell> sipa: less than 10 minutes. 110 2015-11-21T16:53:46 <gmaxwell> listunspent on that wallet is taking 2m10s right now (to return 15870 coins) 111 2015-11-21T16:54:50 *** zmanian_ has joined #bitcoin-core-dev 112 2015-11-21T16:55:08 <gmaxwell> Wallet.dat is 11G which isn't bad. 113 2015-11-21T16:58:37 <gmaxwell> bleh, and getinfo takes 1m8s. 114 2015-11-21T16:59:42 <gmaxwell> Luke-Jr: It might be useful to add an index of which transactions have unspent coins to make listunspent fast. But I think with the way balance calculations work getinfo is going to remain slow. :( 115 2015-11-21T17:01:04 <Luke-Jr> I have never once had a reason to use listunspent.. 116 2015-11-21T17:01:24 <gmaxwell> Luke-Jr: it also means selectcoins will be superslow. 117 2015-11-21T17:04:28 *** Ylbam has quit IRC 118 2015-11-21T17:07:36 <gmaxwell> 4m 15s for a getbalance "*" 0 true, I dunno why getinfo is faster. 119 2015-11-21T17:14:11 *** Ylbam has joined #bitcoin-core-dev 120 2015-11-21T17:34:01 *** davec has joined #bitcoin-core-dev 121 2015-11-21T17:45:46 *** teward has quit IRC 122 2015-11-21T17:52:33 *** teward has joined #bitcoin-core-dev 123 2015-11-21T18:14:55 <gmaxwell> ::sigh:: we really need a remove feature for the wallet, but the remove needs to keep track of what was removed so rescan doesn't read... and we can't remove things with spendable txouts because they're not seperate. 124 2015-11-21T18:18:43 <Luke-Jr> jonas is still rewriting it, right? 125 2015-11-21T18:22:48 <gmaxwell> We've fucked over the project for years with that kind of thinking; we can't stop improving the wallet because of a grand rewrite. 126 2015-11-21T18:24:30 <Luke-Jr> sure, it's just one of those things that if I were to personally try to do it, I would end up probably rewriting the wallet myself :p 127 2015-11-21T18:24:53 <sipa> gmaxwell: what do you mean by remove? 128 2015-11-21T18:25:10 <sipa> remove keys? 129 2015-11-21T18:25:22 <gmaxwell> sipa: no, no reason to remove keys. Remove transactions. 130 2015-11-21T18:26:45 <gmaxwell> sipa: right now large parties using bitcoin core have to periodically rotate out wallets to keep things managable. Things are much better now because of varrious fat trimming. (Including the addtowallet fix we just merged from luke) 131 2015-11-21T18:27:09 <sipa> so more like mark transactions as archived 132 2015-11-21T18:27:13 <sipa> ? 133 2015-11-21T18:27:37 <sipa> so they are no longer considered for balance computations etc 134 2015-11-21T18:27:38 <gmaxwell> Yea, in particular, get them out of the linear scans used to getbalance/listunspent/etc. 135 2015-11-21T18:28:25 <gmaxwell> But not do so in a way that causes coins to fall out of balance and listunspent, because that will cause funds loss when you think a wallet is empty and it's not. 136 2015-11-21T18:29:13 <gmaxwell> so that could be refusing to archive when there are unspent outputs, but thats kind of obnoxious, since it would make visible wallet internal behavior that the caller shouldn't really care about. 137 2015-11-21T18:29:23 <gmaxwell> (Though, otoh, it might encourage sweeping the utxo set. :) ) 138 2015-11-21T18:30:16 <gmaxwell> Also, 'the must be spent completely' rule wouldn't be reorg robust. 139 2015-11-21T18:30:28 <gmaxwell> And, of course, an archived transaction shouldn't be added back by rescanning. 140 2015-11-21T18:31:21 <gmaxwell> Some parties (e.g. bitstamp about a year ago) also complained about the size of the wallet files when they had long histories because it complicated backups; but I think the key exports patch that somewhat. 141 2015-11-21T18:32:41 <sipa> seems easy enough to build a second map inside that does not contain transactions whose outputs have been spent for ages 142 2015-11-21T18:32:53 <sipa> andnuse that for balance calculations etc 143 2015-11-21T18:34:48 <gmaxwell> I think accounts mess this up somewhat; or at least make it not a transparent implementation detail. 144 2015-11-21T18:38:34 <sipa> uh, right 145 2015-11-21T18:38:55 <gmaxwell> thats why I was talking about remove/archive. :-/ 146 2015-11-21T18:43:41 *** CodeShark has quit IRC 147 2015-11-21T19:24:20 *** zarathustra has quit IRC 148 2015-11-21T20:19:31 *** belcher has quit IRC 149 2015-11-21T20:23:23 *** d_t has joined #bitcoin-core-dev 150 2015-11-21T20:34:58 *** ParadoxSpiral has quit IRC 151 2015-11-21T20:37:01 *** ParadoxSpiral has joined #bitcoin-core-dev 152 2015-11-21T20:46:11 *** AtashiCon has quit IRC 153 2015-11-21T20:50:56 *** AtashiCon has joined #bitcoin-core-dev 154 2015-11-21T21:31:50 *** helo has quit IRC 155 2015-11-21T21:32:53 *** helo has joined #bitcoin-core-dev 156 2015-11-21T22:01:35 <GitHub9> [bitcoin] gmaxwell pushed 5 new commits to master: https://github.com/bitcoin/bitcoin/compare/616d61b20d56...31de2414c65d 157 2015-11-21T22:01:36 <GitHub9> bitcoin/master 748321e Peter Todd: Add mediantime field to getblockchaininfo RPC call... 158 2015-11-21T22:01:36 <GitHub9> bitcoin/master c277a63 Peter Todd: Clarify nLockTime-by-time comment in CheckFinalTx() 159 2015-11-21T22:01:37 <GitHub9> bitcoin/master 7259769 Peter Todd: Document new mediantime field in getblockchaininfo 160 2015-11-21T22:01:42 <GitHub66> [bitcoin] gmaxwell closed pull request #7011: Add mediantime to getblockchaininfo (master...2015-11-add-mediantime-to-getblockchaininfo) https://github.com/bitcoin/bitcoin/pull/7011 161 2015-11-21T22:23:19 *** Guyver2 has quit IRC 162 2015-11-21T22:56:44 *** CodeShark has joined #bitcoin-core-dev 163 2015-11-21T23:34:42 <gmaxwell> phantomcircuit: were you going to fix the trickle logic? it really is broken. 164 2015-11-21T23:35:08 <sipa> yes, please! 165 2015-11-21T23:36:05 <gmaxwell> I'm testing the blocks only mode with relaynetwork client as a whitelisted peer (and I cut out the forced broadcasting for whitelisted peers)... and I'm watching it instantly inv any tx that comes in to all it's neighbors. 166 2015-11-21T23:44:17 *** jtimon has joined #bitcoin-core-dev 167 2015-11-21T23:52:09 *** MarcoFalke has quit IRC 168 2015-11-21T23:55:19 <gmaxwell> of course, there is some node connected to me which is sending me every transaction without an INV.