1 2015-11-21T00:06:52  <gmaxwell> lol, luke's #6851 reduces the time to add 53k tx to the wallet from 843 seconds to 35 seconds.
  2 2015-11-21T00:32:48  *** ParadoxSpiral has quit IRC
  3 2015-11-21T00:36:50  *** tripleslash_x has quit IRC
  4 2015-11-21T00:37:09  <GitHub179> [bitcoin] pstratem closed pull request #6966: [WIP] Wallet: Cache CWalletDB pointer in CWallet to improve performance (master...wallet_speedup) https://github.com/bitcoin/bitcoin/pull/6966
  5 2015-11-21T00:38:20  *** tripleslash has joined #bitcoin-core-dev
  6 2015-11-21T00:51:27  <gmaxwell> Luke-Jr: around?
  7 2015-11-21T00:51:31  <Luke-Jr> gmaxwell: ?
  8 2015-11-21T00:54:31  <phantomcircuit> gmaxwell, comment changed and merged petertodd's fRelayTxs rpc addition
  9 2015-11-21T00:55:10  *** randy-waterhouse has joined #bitcoin-core-dev
 10 2015-11-21T01:05:09  *** belcher has joined #bitcoin-core-dev
 11 2015-11-21T01:15:29  *** jtimon_ has quit IRC
 12 2015-11-21T01:28:26  *** CodeShark has joined #bitcoin-core-dev
 13 2015-11-21T02:18:54  *** gavinandresen has joined #bitcoin-core-dev
 14 2015-11-21T02:18:57  *** morcos_ has joined #bitcoin-core-dev
 15 2015-11-21T02:18:59  *** nanotube has quit IRC
 16 2015-11-21T02:19:00  *** tripleslash has quit IRC
 17 2015-11-21T02:19:02  *** Anduck_ has joined #bitcoin-core-dev
 18 2015-11-21T02:19:04  *** morcos has quit IRC
 19 2015-11-21T02:19:05  *** gavinand1esen has quit IRC
 20 2015-11-21T02:19:06  *** Anduck has quit IRC
 21 2015-11-21T02:19:57  *** tripleslash has joined #bitcoin-core-dev
 22 2015-11-21T02:22:21  *** skyraider has quit IRC
 23 2015-11-21T02:28:07  *** nanotube has joined #bitcoin-core-dev
 24 2015-11-21T02:34:28  *** Ylbam has quit IRC
 25 2015-11-21T02:35:15  *** zookolaptop has quit IRC
 26 2015-11-21T02:38:59  <GitHub132> [bitcoin] Har01d opened pull request #7072: [RPC] Add transaction size to JSON output (master...master) https://github.com/bitcoin/bitcoin/pull/7072
 27 2015-11-21T02:53:07  <midnightmagic> gmaxwell: http://0bin.net/paste/vKUjDmnyTyWAkxP2#gkQ4nb+AGhl0QZKpjE1W0oaQqM0ElXzZMseXDp+fwmY  corruption..
 28 2015-11-21T02:53:38  <GitHub123> [bitcoin] gmaxwell pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/776848acefa8...616d61b20d56
 29 2015-11-21T02:53:39  <GitHub123> bitcoin/master 3e7c891 Luke Dashjr: Optimisation: Store transaction list order in memory rather than compute it every need...
 30 2015-11-21T02:53:39  <GitHub123> bitcoin/master 616d61b Gregory Maxwell: Merge pull request #6851...
 31 2015-11-21T02:53:40  <GitHub42> [bitcoin] gmaxwell closed pull request #6851: Optimisation: Store transaction list order in memory rather than compute it every need (master...opti_txorder) https://github.com/bitcoin/bitcoin/pull/6851
 32 2015-11-21T02:54:16  <gmaxwell> midnightmagic: interesting, so that had been running without interruption and just crashed while syncing?
 33 2015-11-21T02:55:19  <midnightmagic> gmaxwell: I did a -reindex and -connect to an internal node.
 34 2015-11-21T02:55:49  <midnightmagic> bitcoind -connect=x.y.z.a -listen -daemon -nodnsseed -nodns -reindex
 35 2015-11-21T02:56:12  <gmaxwell> I guess I need to fix the ppc host to see if I can repro.
 36 2015-11-21T02:56:24  <midnightmagic> Come to think of it.. I'm not sure this machine has ever reliably sync'd to head.
 37 2015-11-21T02:56:56  <gmaxwell> midnightmagic: I think before you might have been failing when it was fine for me.
 38 2015-11-21T02:57:22  <midnightmagic> gmaxwell: Yes. At one point it seemed to be coincident with a hard-disconnect of the network in the middle of a block transfer.
 39 2015-11-21T02:58:42  <midnightmagic> Perhaps this machine is just flakey.
 40 2015-11-21T03:09:40  <midnightmagic> no other evidence of a machine problem. everything else has been running for 300+ days
 41 2015-11-21T03:10:43  <midnightmagic> drat. another one.
 42 2015-11-21T03:10:46  <midnightmagic> http://0bin.net/paste/Jf7jsnULbX4jHJIU#TlLnA8uBw6uxDrYJicVgeOgtgfN37ouH51otjBRT2jC
 43 2015-11-21T03:17:10  <midnightmagic> ooo tor stream isolation
 44 2015-11-21T03:23:16  <gmaxwell> woot, dual G5 is back up.
 45 2015-11-21T03:29:02  *** belcher has quit IRC
 46 2015-11-21T03:35:54  *** go1111111 has quit IRC
 47 2015-11-21T03:49:14  *** go1111111 has joined #bitcoin-core-dev
 48 2015-11-21T03:54:32  <midnightmagic> ooo fascinating.
 49 2015-11-21T03:55:36  *** challisto has joined #bitcoin-core-dev
 50 2015-11-21T03:55:37  *** challisto has joined #bitcoin-core-dev
 51 2015-11-21T03:55:45  <midnightmagic> http://0bin.net/paste/mK4mOOxCYGHmdN7a#6N5+i3ta5ZuVpyVBPX3Uk5NVna+MZUvDvCz5EGc7cTx
 52 2015-11-21T04:13:45  <btcdrak> that's weird
 53 2015-11-21T04:17:35  * midnightmagic shrugs and reindexes the main node.
 54 2015-11-21T04:21:52  *** MarcoFalke has quit IRC
 55 2015-11-21T04:24:13  <gmaxwell> in any case, my 2xG5 host seems to be catching up fine with git master.. taking like 10 seconds for some blocks. :)
 56 2015-11-21T05:10:59  *** arowser has quit IRC
 57 2015-11-21T05:56:39  *** tulip has joined #bitcoin-core-dev
 58 2015-11-21T06:16:35  *** guest234234 has quit IRC
 59 2015-11-21T07:39:26  <midnightmagic> what magic incantation do you do to make it go fast
 60 2015-11-21T07:46:47  *** ParadoxSpiral has joined #bitcoin-core-dev
 61 2015-11-21T07:50:07  *** lecusemble has quit IRC
 62 2015-11-21T07:53:45  *** d_t has quit IRC
 63 2015-11-21T07:56:52  *** lecusemble has joined #bitcoin-core-dev
 64 2015-11-21T08:24:43  *** moli has quit IRC
 65 2015-11-21T08:35:12  *** guest234234 has joined #bitcoin-core-dev
 66 2015-11-21T09:11:05  *** Ylbam has joined #bitcoin-core-dev
 67 2015-11-21T10:03:46  *** dgenr8 has quit IRC
 68 2015-11-21T10:04:13  *** dgenr8 has joined #bitcoin-core-dev
 69 2015-11-21T10:07:24  *** randy-waterhouse has quit IRC
 70 2015-11-21T10:08:06  *** dcousens has joined #bitcoin-core-dev
 71 2015-11-21T10:39:20  *** dcousens has quit IRC
 72 2015-11-21T11:02:03  *** jtimon has joined #bitcoin-core-dev
 73 2015-11-21T11:35:19  *** PaulCapestany has quit IRC
 74 2015-11-21T11:37:01  *** PaulCapestany has joined #bitcoin-core-dev
 75 2015-11-21T12:03:27  *** guest234234 has quit IRC
 76 2015-11-21T12:35:30  *** belcher has joined #bitcoin-core-dev
 77 2015-11-21T12:57:55  *** guest234234 has joined #bitcoin-core-dev
 78 2015-11-21T13:05:03  *** moli has joined #bitcoin-core-dev
 79 2015-11-21T13:35:32  *** tulip has quit IRC
 80 2015-11-21T13:44:18  <jtimon> morcos the linked commit doesn't touch TrimToSize, that's the next one
 81 2015-11-21T13:46:05  *** davec has quit IRC
 82 2015-11-21T13:46:35  <jtimon> ACK on policy estimator called directly instead of through the mempool (but I've been asked not to worked not to work on policy refactors for now) and that one is certainly disruptive
 83 2015-11-21T13:47:40  <jtimon> setting the attribute by calling TrimToSize seems weird, what's wrong with doing it in the constructor (or in a setter if necessary)?
 84 2015-11-21T13:48:16  <jtimon> morcos_: the estimator is currently not asking questions to the mempool and it doesn't need to, please don't couple that again
 85 2015-11-21T13:55:28  <jtimon> you want the mempool decoupled from policy/fees and I agree, and have done it already in a "private" outdated branch
 86 2015-11-21T13:55:54  <jtimon> but to do so, it is not necessary to couple policy/fees to the mempool instead, they can be completely independent
 87 2015-11-21T13:56:49  <jtimon> whenever we think is the moment, I can completely decouple them
 88 2015-11-21T14:05:41  *** jtimon has quit IRC
 89 2015-11-21T14:29:56  *** Guyver2 has joined #bitcoin-core-dev
 90 2015-11-21T14:41:42  *** Anduck_ is now known as Anduck
 91 2015-11-21T15:02:32  *** morcos_ is now known as morcos
 92 2015-11-21T15:04:02  <morcos> jtimon: the question is where shoould the logic live in smartly estimate fees.  in the policyestimator.  that logic requires asking question of the mempool. otherwise you're going to have to put that logic repeated in several different places in wallet and gui.
 93 2015-11-21T15:04:21  <morcos> s/live in/live to/
 94 2015-11-21T15:06:08  <morcos> jtimon: the reason I like setting the mempool size in TrimToSize is you can easily imagine logic where the size is non-constant.  So after you have trimmed it the mempool knows what size it currently is  (the logic as to what size to trim it lives outside mempool)
 95 2015-11-21T15:06:31  <morcos> for instance we discussed using less size for the mempool during IBD so you could use more of your allocated memory for the dbcache
 96 2015-11-21T15:06:39  <morcos> or you can imagine other scenarios
 97 2015-11-21T15:24:49  *** zookolaptop has joined #bitcoin-core-dev
 98 2015-11-21T15:26:15  *** guest234234 has quit IRC
 99 2015-11-21T15:35:21  *** trippysalmon has joined #bitcoin-core-dev
100 2015-11-21T15:44:53  *** trippysalmon has quit IRC
101 2015-11-21T15:57:10  *** davec has joined #bitcoin-core-dev
102 2015-11-21T16:09:20  *** JackH has joined #bitcoin-core-dev
103 2015-11-21T16:25:35  *** zmanian_ has quit IRC
104 2015-11-21T16:32:35  *** zookolaptop has quit IRC
105 2015-11-21T16:33:30  *** davec has quit IRC
106 2015-11-21T16:35:43  *** MarcoFalke has joined #bitcoin-core-dev
107 2015-11-21T16:49:19  <gmaxwell> 1h 11m to rescan a wallet with 11.7m transactions now.
108 2015-11-21T16:51:19  <sipa> how long for a wallet with 0?
109 2015-11-21T16:53:22  <gmaxwell> sipa: less than 10 minutes.
110 2015-11-21T16:53:46  <gmaxwell> listunspent on that wallet is taking 2m10s right now (to return 15870 coins)
111 2015-11-21T16:54:50  *** zmanian_ has joined #bitcoin-core-dev
112 2015-11-21T16:55:08  <gmaxwell> Wallet.dat is 11G which isn't bad.
113 2015-11-21T16:58:37  <gmaxwell> bleh, and getinfo takes 1m8s.
114 2015-11-21T16:59:42  <gmaxwell> Luke-Jr: It might be useful to add an index of which transactions have unspent coins to make listunspent fast.  But I think with the way balance calculations work getinfo is going to remain slow. :(
115 2015-11-21T17:01:04  <Luke-Jr> I have never once had a reason to use listunspent..
116 2015-11-21T17:01:24  <gmaxwell> Luke-Jr: it also means selectcoins will be superslow.
117 2015-11-21T17:04:28  *** Ylbam has quit IRC
118 2015-11-21T17:07:36  <gmaxwell> 4m 15s for a getbalance "*" 0 true, I dunno why getinfo is faster.
119 2015-11-21T17:14:11  *** Ylbam has joined #bitcoin-core-dev
120 2015-11-21T17:34:01  *** davec has joined #bitcoin-core-dev
121 2015-11-21T17:45:46  *** teward has quit IRC
122 2015-11-21T17:52:33  *** teward has joined #bitcoin-core-dev
123 2015-11-21T18:14:55  <gmaxwell> ::sigh:: we really need a remove feature for the wallet, but the remove needs to keep track of what was removed so rescan doesn't read... and we can't remove things with spendable txouts because they're not seperate.
124 2015-11-21T18:18:43  <Luke-Jr> jonas is still rewriting it, right?
125 2015-11-21T18:22:48  <gmaxwell> We've fucked over the project for years with that kind of thinking; we can't stop improving the wallet because of a grand rewrite.
126 2015-11-21T18:24:30  <Luke-Jr> sure, it's just one of those things that if I were to personally try to do it, I would end up probably rewriting the wallet myself :p
127 2015-11-21T18:24:53  <sipa> gmaxwell: what do you mean by remove?
128 2015-11-21T18:25:10  <sipa> remove keys?
129 2015-11-21T18:25:22  <gmaxwell> sipa: no, no reason to remove keys. Remove transactions.
130 2015-11-21T18:26:45  <gmaxwell> sipa: right now large parties using bitcoin core have to periodically rotate out wallets to keep things managable. Things are much better now because of varrious fat trimming. (Including the addtowallet fix we just merged from luke)
131 2015-11-21T18:27:09  <sipa> so more like mark transactions as archived
132 2015-11-21T18:27:13  <sipa> ?
133 2015-11-21T18:27:37  <sipa> so they are no longer considered for balance computations etc
134 2015-11-21T18:27:38  <gmaxwell> Yea, in particular, get them out of the linear scans used to getbalance/listunspent/etc.
135 2015-11-21T18:28:25  <gmaxwell> But not do so in a way that causes coins to fall out of balance and listunspent, because that will cause funds loss when you think a wallet is empty and it's not.
136 2015-11-21T18:29:13  <gmaxwell> so that could be refusing to archive when there are unspent outputs, but thats kind of obnoxious, since it would make visible wallet internal behavior that the caller shouldn't really care about.
137 2015-11-21T18:29:23  <gmaxwell> (Though, otoh, it might encourage sweeping the utxo set. :) )
138 2015-11-21T18:30:16  <gmaxwell> Also, 'the must be spent completely' rule wouldn't be reorg robust.
139 2015-11-21T18:30:28  <gmaxwell> And, of course, an archived transaction shouldn't be added back by rescanning.
140 2015-11-21T18:31:21  <gmaxwell> Some parties (e.g. bitstamp about a year ago) also complained about the size of the wallet files when they had long histories because it complicated backups; but I think the key exports patch that somewhat.
141 2015-11-21T18:32:41  <sipa> seems easy enough to build a second map inside that does not contain transactions whose outputs have been spent for ages
142 2015-11-21T18:32:53  <sipa> andnuse that for balance calculations etc
143 2015-11-21T18:34:48  <gmaxwell> I think accounts mess this up somewhat; or at least make it not a transparent implementation detail.
144 2015-11-21T18:38:34  <sipa> uh, right
145 2015-11-21T18:38:55  <gmaxwell> thats why I was talking about remove/archive. :-/
146 2015-11-21T18:43:41  *** CodeShark has quit IRC
147 2015-11-21T19:24:20  *** zarathustra has quit IRC
148 2015-11-21T20:19:31  *** belcher has quit IRC
149 2015-11-21T20:23:23  *** d_t has joined #bitcoin-core-dev
150 2015-11-21T20:34:58  *** ParadoxSpiral has quit IRC
151 2015-11-21T20:37:01  *** ParadoxSpiral has joined #bitcoin-core-dev
152 2015-11-21T20:46:11  *** AtashiCon has quit IRC
153 2015-11-21T20:50:56  *** AtashiCon has joined #bitcoin-core-dev
154 2015-11-21T21:31:50  *** helo has quit IRC
155 2015-11-21T21:32:53  *** helo has joined #bitcoin-core-dev
156 2015-11-21T22:01:35  <GitHub9> [bitcoin] gmaxwell pushed 5 new commits to master: https://github.com/bitcoin/bitcoin/compare/616d61b20d56...31de2414c65d
157 2015-11-21T22:01:36  <GitHub9> bitcoin/master 748321e Peter Todd: Add mediantime field to getblockchaininfo RPC call...
158 2015-11-21T22:01:36  <GitHub9> bitcoin/master c277a63 Peter Todd: Clarify nLockTime-by-time comment in CheckFinalTx()
159 2015-11-21T22:01:37  <GitHub9> bitcoin/master 7259769 Peter Todd: Document new mediantime field in getblockchaininfo
160 2015-11-21T22:01:42  <GitHub66> [bitcoin] gmaxwell closed pull request #7011: Add mediantime to getblockchaininfo (master...2015-11-add-mediantime-to-getblockchaininfo) https://github.com/bitcoin/bitcoin/pull/7011
161 2015-11-21T22:23:19  *** Guyver2 has quit IRC
162 2015-11-21T22:56:44  *** CodeShark has joined #bitcoin-core-dev
163 2015-11-21T23:34:42  <gmaxwell> phantomcircuit: were you going to fix the trickle logic? it really is broken.
164 2015-11-21T23:35:08  <sipa> yes, please!
165 2015-11-21T23:36:05  <gmaxwell> I'm testing the blocks only mode with relaynetwork client as a whitelisted peer (and I cut out the forced broadcasting for whitelisted peers)... and I'm watching it instantly inv any tx that comes in to all it's neighbors.
166 2015-11-21T23:44:17  *** jtimon has joined #bitcoin-core-dev
167 2015-11-21T23:52:09  *** MarcoFalke has quit IRC
168 2015-11-21T23:55:19  <gmaxwell> of course, there is some node connected to me which is sending me every transaction without an INV.