12015-11-21T00:06:52 <gmaxwell> lol, luke's #6851 reduces the time to add 53k tx to the wallet from 843 seconds to 35 seconds.
22015-11-21T00:32:48 *** ParadoxSpiral has quit IRC
32015-11-21T00:36:50 *** tripleslash_x has quit IRC
42015-11-21T00:37:09 <GitHub179> [bitcoin] pstratem closed pull request #6966: [WIP] Wallet: Cache CWalletDB pointer in CWallet to improve performance (master...wallet_speedup) https://github.com/bitcoin/bitcoin/pull/6966
52015-11-21T00:38:20 *** tripleslash has joined #bitcoin-core-dev
62015-11-21T00:51:27 <gmaxwell> Luke-Jr: around?
72015-11-21T00:51:31 <Luke-Jr> gmaxwell: ?
82015-11-21T00:54:31 <phantomcircuit> gmaxwell, comment changed and merged petertodd's fRelayTxs rpc addition
92015-11-21T00:55:10 *** randy-waterhouse has joined #bitcoin-core-dev
102015-11-21T01:05:09 *** belcher has joined #bitcoin-core-dev
112015-11-21T01:15:29 *** jtimon_ has quit IRC
122015-11-21T01:28:26 *** CodeShark has joined #bitcoin-core-dev
132015-11-21T02:18:54 *** gavinandresen has joined #bitcoin-core-dev
142015-11-21T02:18:57 *** morcos_ has joined #bitcoin-core-dev
152015-11-21T02:18:59 *** nanotube has quit IRC
162015-11-21T02:19:00 *** tripleslash has quit IRC
172015-11-21T02:19:02 *** Anduck_ has joined #bitcoin-core-dev
182015-11-21T02:19:04 *** morcos has quit IRC
192015-11-21T02:19:05 *** gavinand1esen has quit IRC
202015-11-21T02:19:06 *** Anduck has quit IRC
212015-11-21T02:19:57 *** tripleslash has joined #bitcoin-core-dev
222015-11-21T02:22:21 *** skyraider has quit IRC
232015-11-21T02:28:07 *** nanotube has joined #bitcoin-core-dev
242015-11-21T02:34:28 *** Ylbam has quit IRC
252015-11-21T02:35:15 *** zookolaptop has quit IRC
262015-11-21T02:38:59 <GitHub132> [bitcoin] Har01d opened pull request #7072: [RPC] Add transaction size to JSON output (master...master) https://github.com/bitcoin/bitcoin/pull/7072
272015-11-21T02:53:07 <midnightmagic> gmaxwell: http://0bin.net/paste/vKUjDmnyTyWAkxP2#gkQ4nb+AGhl0QZKpjE1W0oaQqM0ElXzZMseXDp+fwmY corruption..
282015-11-21T02:53:38 <GitHub123> [bitcoin] gmaxwell pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/776848acefa8...616d61b20d56
292015-11-21T02:53:39 <GitHub123> bitcoin/master 3e7c891 Luke Dashjr: Optimisation: Store transaction list order in memory rather than compute it every need...
302015-11-21T02:53:39 <GitHub123> bitcoin/master 616d61b Gregory Maxwell: Merge pull request #6851...
312015-11-21T02:53:40 <GitHub42> [bitcoin] gmaxwell closed pull request #6851: Optimisation: Store transaction list order in memory rather than compute it every need (master...opti_txorder) https://github.com/bitcoin/bitcoin/pull/6851
322015-11-21T02:54:16 <gmaxwell> midnightmagic: interesting, so that had been running without interruption and just crashed while syncing?
332015-11-21T02:55:19 <midnightmagic> gmaxwell: I did a -reindex and -connect to an internal node.
342015-11-21T02:55:49 <midnightmagic> bitcoind -connect=x.y.z.a -listen -daemon -nodnsseed -nodns -reindex
352015-11-21T02:56:12 <gmaxwell> I guess I need to fix the ppc host to see if I can repro.
362015-11-21T02:56:24 <midnightmagic> Come to think of it.. I'm not sure this machine has ever reliably sync'd to head.
372015-11-21T02:56:56 <gmaxwell> midnightmagic: I think before you might have been failing when it was fine for me.
382015-11-21T02:57:22 <midnightmagic> gmaxwell: Yes. At one point it seemed to be coincident with a hard-disconnect of the network in the middle of a block transfer.
392015-11-21T02:58:42 <midnightmagic> Perhaps this machine is just flakey.
402015-11-21T03:09:40 <midnightmagic> no other evidence of a machine problem. everything else has been running for 300+ days
412015-11-21T03:10:43 <midnightmagic> drat. another one.
422015-11-21T03:10:46 <midnightmagic> http://0bin.net/paste/Jf7jsnULbX4jHJIU#TlLnA8uBw6uxDrYJicVgeOgtgfN37ouH51otjBRT2jC
432015-11-21T03:17:10 <midnightmagic> ooo tor stream isolation
442015-11-21T03:23:16 <gmaxwell> woot, dual G5 is back up.
452015-11-21T03:29:02 *** belcher has quit IRC
462015-11-21T03:35:54 *** go1111111 has quit IRC
472015-11-21T03:49:14 *** go1111111 has joined #bitcoin-core-dev
482015-11-21T03:54:32 <midnightmagic> ooo fascinating.
492015-11-21T03:55:36 *** challisto has joined #bitcoin-core-dev
502015-11-21T03:55:37 *** challisto has joined #bitcoin-core-dev
512015-11-21T03:55:45 <midnightmagic> http://0bin.net/paste/mK4mOOxCYGHmdN7a#6N5+i3ta5ZuVpyVBPX3Uk5NVna+MZUvDvCz5EGc7cTx
522015-11-21T04:13:45 <btcdrak> that's weird
532015-11-21T04:17:35 * midnightmagic shrugs and reindexes the main node.
542015-11-21T04:21:52 *** MarcoFalke has quit IRC
552015-11-21T04:24:13 <gmaxwell> in any case, my 2xG5 host seems to be catching up fine with git master.. taking like 10 seconds for some blocks. :)
562015-11-21T05:10:59 *** arowser has quit IRC
572015-11-21T05:56:39 *** tulip has joined #bitcoin-core-dev
582015-11-21T06:16:35 *** guest234234 has quit IRC
592015-11-21T07:39:26 <midnightmagic> what magic incantation do you do to make it go fast
602015-11-21T07:46:47 *** ParadoxSpiral has joined #bitcoin-core-dev
612015-11-21T07:50:07 *** lecusemble has quit IRC
622015-11-21T07:53:45 *** d_t has quit IRC
632015-11-21T07:56:52 *** lecusemble has joined #bitcoin-core-dev
642015-11-21T08:24:43 *** moli has quit IRC
652015-11-21T08:35:12 *** guest234234 has joined #bitcoin-core-dev
662015-11-21T09:11:05 *** Ylbam has joined #bitcoin-core-dev
672015-11-21T10:03:46 *** dgenr8 has quit IRC
682015-11-21T10:04:13 *** dgenr8 has joined #bitcoin-core-dev
692015-11-21T10:07:24 *** randy-waterhouse has quit IRC
702015-11-21T10:08:06 *** dcousens has joined #bitcoin-core-dev
712015-11-21T10:39:20 *** dcousens has quit IRC
722015-11-21T11:02:03 *** jtimon has joined #bitcoin-core-dev
732015-11-21T11:35:19 *** PaulCapestany has quit IRC
742015-11-21T11:37:01 *** PaulCapestany has joined #bitcoin-core-dev
752015-11-21T12:03:27 *** guest234234 has quit IRC
762015-11-21T12:35:30 *** belcher has joined #bitcoin-core-dev
772015-11-21T12:57:55 *** guest234234 has joined #bitcoin-core-dev
782015-11-21T13:05:03 *** moli has joined #bitcoin-core-dev
792015-11-21T13:35:32 *** tulip has quit IRC
802015-11-21T13:44:18 <jtimon> morcos the linked commit doesn't touch TrimToSize, that's the next one
812015-11-21T13:46:05 *** davec has quit IRC
822015-11-21T13:46:35 <jtimon> ACK on policy estimator called directly instead of through the mempool (but I've been asked not to worked not to work on policy refactors for now) and that one is certainly disruptive
832015-11-21T13:47:40 <jtimon> setting the attribute by calling TrimToSize seems weird, what's wrong with doing it in the constructor (or in a setter if necessary)?
842015-11-21T13:48:16 <jtimon> morcos_: the estimator is currently not asking questions to the mempool and it doesn't need to, please don't couple that again
852015-11-21T13:55:28 <jtimon> you want the mempool decoupled from policy/fees and I agree, and have done it already in a "private" outdated branch
862015-11-21T13:55:54 <jtimon> but to do so, it is not necessary to couple policy/fees to the mempool instead, they can be completely independent
872015-11-21T13:56:49 <jtimon> whenever we think is the moment, I can completely decouple them
882015-11-21T14:05:41 *** jtimon has quit IRC
892015-11-21T14:29:56 *** Guyver2 has joined #bitcoin-core-dev
902015-11-21T14:41:42 *** Anduck_ is now known as Anduck
912015-11-21T15:02:32 *** morcos_ is now known as morcos
922015-11-21T15:04:02 <morcos> jtimon: the question is where shoould the logic live in smartly estimate fees. in the policyestimator. that logic requires asking question of the mempool. otherwise you're going to have to put that logic repeated in several different places in wallet and gui.
932015-11-21T15:04:21 <morcos> s/live in/live to/
942015-11-21T15:06:08 <morcos> jtimon: the reason I like setting the mempool size in TrimToSize is you can easily imagine logic where the size is non-constant. So after you have trimmed it the mempool knows what size it currently is (the logic as to what size to trim it lives outside mempool)
952015-11-21T15:06:31 <morcos> for instance we discussed using less size for the mempool during IBD so you could use more of your allocated memory for the dbcache
962015-11-21T15:06:39 <morcos> or you can imagine other scenarios
972015-11-21T15:24:49 *** zookolaptop has joined #bitcoin-core-dev
982015-11-21T15:26:15 *** guest234234 has quit IRC
992015-11-21T15:35:21 *** trippysalmon has joined #bitcoin-core-dev
1002015-11-21T15:44:53 *** trippysalmon has quit IRC
1012015-11-21T15:57:10 *** davec has joined #bitcoin-core-dev
1022015-11-21T16:09:20 *** JackH has joined #bitcoin-core-dev
1032015-11-21T16:25:35 *** zmanian_ has quit IRC
1042015-11-21T16:32:35 *** zookolaptop has quit IRC
1052015-11-21T16:33:30 *** davec has quit IRC
1062015-11-21T16:35:43 *** MarcoFalke has joined #bitcoin-core-dev
1072015-11-21T16:49:19 <gmaxwell> 1h 11m to rescan a wallet with 11.7m transactions now.
1082015-11-21T16:51:19 <sipa> how long for a wallet with 0?
1092015-11-21T16:53:22 <gmaxwell> sipa: less than 10 minutes.
1102015-11-21T16:53:46 <gmaxwell> listunspent on that wallet is taking 2m10s right now (to return 15870 coins)
1112015-11-21T16:54:50 *** zmanian_ has joined #bitcoin-core-dev
1122015-11-21T16:55:08 <gmaxwell> Wallet.dat is 11G which isn't bad.
1132015-11-21T16:58:37 <gmaxwell> bleh, and getinfo takes 1m8s.
1142015-11-21T16:59:42 <gmaxwell> Luke-Jr: It might be useful to add an index of which transactions have unspent coins to make listunspent fast. But I think with the way balance calculations work getinfo is going to remain slow. :(
1152015-11-21T17:01:04 <Luke-Jr> I have never once had a reason to use listunspent..
1162015-11-21T17:01:24 <gmaxwell> Luke-Jr: it also means selectcoins will be superslow.
1172015-11-21T17:04:28 *** Ylbam has quit IRC
1182015-11-21T17:07:36 <gmaxwell> 4m 15s for a getbalance "*" 0 true, I dunno why getinfo is faster.
1192015-11-21T17:14:11 *** Ylbam has joined #bitcoin-core-dev
1202015-11-21T17:34:01 *** davec has joined #bitcoin-core-dev
1212015-11-21T17:45:46 *** teward has quit IRC
1222015-11-21T17:52:33 *** teward has joined #bitcoin-core-dev
1232015-11-21T18:14:55 <gmaxwell> ::sigh:: we really need a remove feature for the wallet, but the remove needs to keep track of what was removed so rescan doesn't read... and we can't remove things with spendable txouts because they're not seperate.
1242015-11-21T18:18:43 <Luke-Jr> jonas is still rewriting it, right?
1252015-11-21T18:22:48 <gmaxwell> We've fucked over the project for years with that kind of thinking; we can't stop improving the wallet because of a grand rewrite.
1262015-11-21T18:24:30 <Luke-Jr> sure, it's just one of those things that if I were to personally try to do it, I would end up probably rewriting the wallet myself :p
1272015-11-21T18:24:53 <sipa> gmaxwell: what do you mean by remove?
1282015-11-21T18:25:10 <sipa> remove keys?
1292015-11-21T18:25:22 <gmaxwell> sipa: no, no reason to remove keys. Remove transactions.
1302015-11-21T18:26:45 <gmaxwell> sipa: right now large parties using bitcoin core have to periodically rotate out wallets to keep things managable. Things are much better now because of varrious fat trimming. (Including the addtowallet fix we just merged from luke)
1312015-11-21T18:27:09 <sipa> so more like mark transactions as archived
1322015-11-21T18:27:13 <sipa> ?
1332015-11-21T18:27:37 <sipa> so they are no longer considered for balance computations etc
1342015-11-21T18:27:38 <gmaxwell> Yea, in particular, get them out of the linear scans used to getbalance/listunspent/etc.
1352015-11-21T18:28:25 <gmaxwell> But not do so in a way that causes coins to fall out of balance and listunspent, because that will cause funds loss when you think a wallet is empty and it's not.
1362015-11-21T18:29:13 <gmaxwell> so that could be refusing to archive when there are unspent outputs, but thats kind of obnoxious, since it would make visible wallet internal behavior that the caller shouldn't really care about.
1372015-11-21T18:29:23 <gmaxwell> (Though, otoh, it might encourage sweeping the utxo set. :) )
1382015-11-21T18:30:16 <gmaxwell> Also, 'the must be spent completely' rule wouldn't be reorg robust.
1392015-11-21T18:30:28 <gmaxwell> And, of course, an archived transaction shouldn't be added back by rescanning.
1402015-11-21T18:31:21 <gmaxwell> Some parties (e.g. bitstamp about a year ago) also complained about the size of the wallet files when they had long histories because it complicated backups; but I think the key exports patch that somewhat.
1412015-11-21T18:32:41 <sipa> seems easy enough to build a second map inside that does not contain transactions whose outputs have been spent for ages
1422015-11-21T18:32:53 <sipa> andnuse that for balance calculations etc
1432015-11-21T18:34:48 <gmaxwell> I think accounts mess this up somewhat; or at least make it not a transparent implementation detail.
1442015-11-21T18:38:34 <sipa> uh, right
1452015-11-21T18:38:55 <gmaxwell> thats why I was talking about remove/archive. :-/
1462015-11-21T18:43:41 *** CodeShark has quit IRC
1472015-11-21T19:24:20 *** zarathustra has quit IRC
1482015-11-21T20:19:31 *** belcher has quit IRC
1492015-11-21T20:23:23 *** d_t has joined #bitcoin-core-dev
1502015-11-21T20:34:58 *** ParadoxSpiral has quit IRC
1512015-11-21T20:37:01 *** ParadoxSpiral has joined #bitcoin-core-dev
1522015-11-21T20:46:11 *** AtashiCon has quit IRC
1532015-11-21T20:50:56 *** AtashiCon has joined #bitcoin-core-dev
1542015-11-21T21:31:50 *** helo has quit IRC
1552015-11-21T21:32:53 *** helo has joined #bitcoin-core-dev
1562015-11-21T22:01:35 <GitHub9> [bitcoin] gmaxwell pushed 5 new commits to master: https://github.com/bitcoin/bitcoin/compare/616d61b20d56...31de2414c65d
1572015-11-21T22:01:36 <GitHub9> bitcoin/master 748321e Peter Todd: Add mediantime field to getblockchaininfo RPC call...
1582015-11-21T22:01:36 <GitHub9> bitcoin/master c277a63 Peter Todd: Clarify nLockTime-by-time comment in CheckFinalTx()
1592015-11-21T22:01:37 <GitHub9> bitcoin/master 7259769 Peter Todd: Document new mediantime field in getblockchaininfo
1602015-11-21T22:01:42 <GitHub66> [bitcoin] gmaxwell closed pull request #7011: Add mediantime to getblockchaininfo (master...2015-11-add-mediantime-to-getblockchaininfo) https://github.com/bitcoin/bitcoin/pull/7011
1612015-11-21T22:23:19 *** Guyver2 has quit IRC
1622015-11-21T22:56:44 *** CodeShark has joined #bitcoin-core-dev
1632015-11-21T23:34:42 <gmaxwell> phantomcircuit: were you going to fix the trickle logic? it really is broken.
1642015-11-21T23:35:08 <sipa> yes, please!
1652015-11-21T23:36:05 <gmaxwell> I'm testing the blocks only mode with relaynetwork client as a whitelisted peer (and I cut out the forced broadcasting for whitelisted peers)... and I'm watching it instantly inv any tx that comes in to all it's neighbors.
1662015-11-21T23:44:17 *** jtimon has joined #bitcoin-core-dev
1672015-11-21T23:52:09 *** MarcoFalke has quit IRC
1682015-11-21T23:55:19 <gmaxwell> of course, there is some node connected to me which is sending me every transaction without an INV.