19:02:40 #startmeeting 19:02:40 Meeting started Thu Jun 23 19:02:40 2016 UTC. The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:02:40 Useful Commands: #action #agreed #help #info #idea #link #topic. 19:03:05 any proposed topics? 19:04:01 I think https://github.com/bitcoin/bitcoin/issues/8245 needs discussion, whether reindex/verification slowed down 19:04:04 sdaftuar: sipa: phantomcircuit: jonasschnelli: MarcoFalke: jtimon: phantomcircuit: instagibbs: petertodd: morcos: 19:04:08 ack topic 19:04:11 ack. 19:04:41 I missed a couple of meetings and I have some questions about merging segwit and 0.13 feature freeze, but maybe that can wait for after the meeting 19:04:46 #topic perceived validation slowdowns 19:04:58 i noticed very slow chainstate writes (close to a minute, even though the chainstate is only 53 MB) 19:05:03 I was just going to go try reindexes with 0.12.1 and master on two hosts, to see if its a regression. Even if it is not, it's absurdly slow (and I assume for sync too, but that should be validated) and maybe we should consider cranking dbcache and making a release note for loe memory hosts. 19:05:13 though this may be due to my laptop's disk setup (zfs on encrypted lvm volume) 19:05:27 so, some people are noticiing slow validation, I wonder what changed there 19:05:29 (two identical hosts) 19:05:49 from gmaxwell's numbers, it does not look like his time is dominated by database flushes, however 19:05:57 I think I am often guilty of only testing things with a non-default dbcache. 19:06:03 likewise 19:06:12 I previously _expected_ reindex to be slow due to the signature checking. 19:06:19 I usually run with default dbcache 19:06:30 maybe a solution would be to change the default dbcache for something more common among testers? 19:06:46 my laptop runs with default dbcache and I have been noticing verification is slow for a while, but I have not reindexed for quite a long time. 19:06:48 jtimon: we still need to figure out what is behind this... but independently, yes 19:06:50 yes, we should definitely crank up the dbcache no matter what 19:06:53 but I like to know what happened 19:07:08 sipa: yeah, of course, I meant independently 19:07:30 should at least try to bisec tthis, I know it's frustrating as everything takes so long :) 19:08:07 oh, this is with txindex enabled 19:08:14 Well I will test against 0.12.1 and see if there is a regression. oh crap. testing against 0.12.1 will be messed up by signature checking, I guess I will test against patched 0.12.1 that skips signature checking before block 295000? does that sound okay? 19:08:18 gmaxwell didn't you have some suspicions that CB would affect initial sync? could that also affect reindex? 19:08:24 not in my case 19:08:27 wumpus: gmaxwell's test was before cb was merged 19:08:57 my test is without cb merged, also the CB concern was just related to general sync logic, not processing. 19:08:58 are there recent changes that people suspect may have caused the regression ? 19:09:07 wouldbe good to test against #7917, as many people benchmarked that 19:09:18 and it was perceived fast at the time 19:09:40 jtimon: not really 19:09:42 I can refine further if it looks like a regression. I think it's likely when I tested 7917 it was with a high dbcache and the result was that things got much faster. 19:10:13 At least the logs I'm seeing suggest that the slow performace is leveldb performance being slow, so it would be completely hidden by a sufficiently large dbcache. 19:10:30 (maybe not leveldb itself being slow, e.g. making extranious accesses to it) 19:10:47 it may be the windows machine I'm testing on is just crappy, I also had a strange issue with ldb files: https://github.com/bitcoin/bitcoin/issues/8250 .. possible that the disk is just very slow due to other processes interfering 19:10:56 gmaxwell: note that txindex may actually influence this; the txindex entries are written continuously to the db, and not cached inside the application or batched together with the rist 19:11:03 So actions are. determine if regression, if so where, ... seperately, consider a dbcache increase for the next release. 19:11:08 I doubt it is leveldb itself as there haven't been changed to that for ages 19:11:17 sipa: yes this might be txindex related. I can test that too. 19:11:23 yes, txindex is *slow* 19:11:24 is something slowing down validation that wasn't before? 19:11:29 CodeShark: maybe 19:11:32 lots of extra I/O. I also made that mistake once 19:11:40 wumpus: we have changed (reduced) the amount of interaction with leveldb during validation. 19:11:51 gmaxwell: sure, it may be the level above leveldb 19:12:00 yes, it may be 19:12:06 I just don't suspect the databse itself this time 19:12:34 but yes, the only path I see to leveldb itself would just be that it now has more data in it than ever before, and perhaps it crossed some performance cliff. but otherwise, nah. 19:12:36 unless some compiler flag changed things shockingly, say, the c++11 switch, but I doubt it 19:12:53 I think txindex is a good lead, my laptop is the only host I regularly use with it, and I've been noticing poor validation performance for a while. 19:13:07 i briefly suspected the IsInitialBlockDownload change, but apart from using an atomic, its semantics should be unchanged 19:13:19 especialy if you see the slowdown in the flush; txindex writes a lot of data to the block index databse 19:13:42 wumpus: but the txindex writes don't happen during the flush 19:13:45 so maybe false alarm, the sync is slow, news at 11 19:13:59 wumpus: though they may accumulate somewhere in leveldb until a flush is issued 19:14:01 can't we add tracers around calls to narrow it down? 19:14:08 wumpus: well, the "lets increase the dbcache" is still a useful response to this catching attention again. 19:14:31 action points: benchmark in same conditions without txindex, and with larger dbcache? 19:14:55 yes we should increate the dbcache, and probably change how it is allocated 19:15:09 (e.g. a relatively large part now goes to leveldb caches, that's a waste 19:15:16 (as my laptop is about to be on day three of reindex) 19:15:51 wumpus: the reasoning was that leveldb cache is serialized, so it has a much large impact per byte than the internal cache 19:15:55 leveldb's own caches are completely ineffective compared to bitcoind's application level cache 19:15:59 but it has the deserialization overhead 19:16:17 but that's mostly a guess without substansive benchmarking 19:16:18 sipa: that makes a lot of sense in theory, but leveldb just sucks at caching :) 19:16:43 I can benchmark a bunch of mixes of cache sizes and see how they pan out. 19:16:53 wumpus: it may also be due to our access pattern... the things that get written to leveldb haven't been touched for a while; maybe they won't be touched any time soon as a result either 19:17:04 the current values are probably ok, I just mean: if we scale dbcache we probably don't want to scale those caches with it 19:17:10 not sure I'm following but maybe we want separate options for the "internal" and "external" caches? 19:17:27 jtimon: for testing that'd be awesome 19:17:42 jtimon: for testing that makes sense, but as a product it should work well with the fewest number of tunable possible 19:17:46 In principle we shouldn't add knobs as a punt for highly technical settings that even we haven't figured out, the users will do no better at it. :P (as hidden options for testing or whatnot, sure) 19:17:58 jynx 19:17:59 On phone but present 19:18:21 there's some options that can only be accessed if --debug, right? 19:18:25 gmaxwell: you'd be surprised :-) 19:18:35 some users are very persistent in trying to find settings that are fastest for them 19:18:48 I'd venture to say it's probably not the majority 19:18:51 but yes, it should be a hidden option (--help-debug) 19:18:52 oh, no they may not be showed in the help but they're still accesible I think 19:19:00 CodeShark: sure, but don't underestimate peole 19:19:08 wumpus: maybe such options should be marked with ("If you find a setting for this value that improves performance, please let us know") 19:19:15 sipa: +1 :D 19:19:18 -funroll-loops -O20 19:19:34 -femit-broken-code 19:20:05 -fskip-computing-even-bits 19:20:13 ok, any other topics? 19:20:15 relatedly, i think -qt can by default use more ram 19:20:37 also, 100 mb is kinda ridiculous with the mempool already being 300 mb 19:20:42 yes, probably, although qt itself also uses more ram 19:21:02 yes, agreed 19:21:11 other topic: merge segwit yay or nay 19:21:16 let's reduce the mempool to 100mb and increase dbcache to 300mb 19:21:17 okay, I have my action items on this. I will benchmark a bunch of configurations. 0.12.1 vs master; and master w/wo txindex, w/wo default dbcache.... and also try different cache splits. I may ask for suggestions on the interesting parameter space when I go to do that. If there is a 0.12.1 vs master regression I'll find out where. 19:21:38 sipa: re: segwit, has it been rebased? 19:21:45 #topic segwit 19:21:46 petertodd: 12 times by now 19:21:50 lol 19:21:53 see 8149 19:21:55 poor sipa 19:21:55 how can we feature freeze without merging segwit? 19:22:03 sipa is getting carpal tunnel syndrome from rebasing 19:22:05 We can do some checking to see what mempool size should be based on current traffic, in principle I'd agree shifting memory from one to the other. 19:22:16 jtimon: softforks / consensus changes don't count as software features 19:22:28 jtimon: that's also why they're allowed to be introduced in minor versions 19:22:29 sipa: I mean, is the current pull-req rebased since compact blocks got merged? 19:22:32 also it's not even activated in any case. it's pure code updates. 19:22:35 petertodd: yes 19:22:49 wumpus: so it won't be possible to include any feature post segwit merge in 0.13 ? 19:22:52 current is #8149, yes? 19:23:00 petertodd: and i've been running the rebase on both testnet (where it syncs segwit blocks) and on mainnet (where it uses compact blocks) 19:23:05 jtimon: right, 0.13 is closed feature-wise 19:23:52 I haven't done CB+segwit testing yet, but I'm due to bring up a new testnet node, so I can do that. 19:23:55 features will be merged again on master after a 0.13 branch is created, which will be around the rc1 release (planned july 6 I think) 19:24:02 wumpus: that is a no, that is unconvenient and I wasn't expecting it, but thanks 19:24:34 #link see the release schedule here: https://github.com/bitcoin/bitcoin/issues/8250 19:24:56 sipa: which PR should we be testing against? 19:24:56 yeah, should have asked "what about segwit?" back then 19:25:06 CodeShark: 8149 and 7190 are the exact same code 19:25:10 so whatever you like 19:25:44 I had completed review up to the CB rebase, which I have not looked at yet. 19:26:03 (I mean I haven't looked at segwit's rebase for CB) 19:26:24 I would have preferred that segwit would have been merged before feature freeze to have time to do something post-segwit for 0.13, but mainly I just wanted to undesrtand the situation 19:26:45 git show -w c4e3c755d7e41aaabe74c84af7e4bf00a62c96fb 19:26:54 and then search for cmpctblk and blocktxn 19:27:01 to see the changes related to that 19:27:05 oh I'm late 19:27:21 jtimon: we all would have preferred other timings, but we have to cope with how things actually are :) 19:27:46 I planned to rebase/rewrite some of the consensus encapsulation code after segwit, I guess the plan doesn't change anyway 19:28:02 well you can still do that, it just won't make 0.13 19:28:37 wumpus: well, yeah, I could have helped more with reviewing and testing segwit 19:28:41 sipa: thanks, will review once I start up the test panel for the prior topic. :) 19:28:48 wumpus: understood 19:29:07 jtimon: you can still do that, even after merge :) 19:29:22 yes, post merge review and testing is super important too. 19:29:44 yes, absolutely 19:30:22 In any case I am in favor of the merge. (and don't expect my remaining review to turn up any reason not to) 19:30:43 i'm running the cb+segwit rebase on bitcoin.sipa.be since a few days, to see if there was an impact on memory usage - i see none 19:30:48 Obviously there may still need to be some fixes and nits. But thats what we have time for. 19:30:54 (compared to running just cb before) 19:30:56 anyone against merging segwit? 19:31:03 (I mean right now, not in general) 19:32:00 *crickets* 19:32:11 i have no objections :) 19:32:14 that's clear then 19:32:17 I'd like to see it merged too 19:32:19 yes, I understood 19:32:27 sipa: yeah, it just won't make it to 0.13 as wumpus explained 19:32:30 the sooner the better (as long as it doesn't break current behavior) 19:32:39 #action Merge segwit 19:32:45 o/ 19:33:10 at this point it will amplify testing, since we've done a much of the specialized testing. Testing for incidental effects by a broader audience would be good. 19:33:26 Good. 19:34:00 Would it be awful to suggest that we put out 'testnet binaries' right away off master to also get more people testing with that code in use? 19:34:19 gmaxwell: fine by me 19:34:21 i believe jonasschnelli builds nightly binaries 19:34:22 sure, why not 19:34:47 I prefer doing that outside the 'official' release cycle, but I don't mind running gitian for it 19:34:51 I think that the prior segnet testing didn't include anyone that was likely to be confused by status changes in the UI or whatnot-- too technical an audience since you had to build it. :) 19:35:15 yes, I don't want something part of the release cycle. Just binaries to give to power users to give it a spin. 19:35:27 testnet-only 19:35:51 testnet as in not segnet, right? 19:35:52 yea, pre-RC testnet only, we could one line patch to change the default network, so it'll be easier to use. 19:36:07 CodeShark: segnet has been removed a few weeks ago from the patch 19:36:17 Testnet is segwit now. Segnet is dead long live segnet. 19:36:21 gmaxwell: maybe better to just make it fail unless -testnet is enabled, in case someone does run it in production 19:36:26 no, not segnet. Regtest could be allowed. But mainnet disabled and testnet as default 19:36:32 is segwit live on testnet?!?! 19:36:36 yea. exactly. 19:36:38 CodeShark: yes, since months 19:37:01 petertodd: what is the worst that could happen if you accidentally run testnet? 19:37:14 wumpus: hard to know! 19:37:39 it's just pre-RC master, lots of people do run that in production. I wouldn't worry too much, discretion of whomever makes the testnet-as-default patch? 19:37:44 wumpus: having to add a -testnet flag isn't a big deal; and failing hard if you don't shouldn't have any consequences 19:37:46 at least the default ports etc will be different, default data dir is different, etc 19:38:13 petertodd: apart from GUI users I guess 19:38:19 petertodd: I hope the user doesn't have to set any flags, if they have to set flags far fewer people will try it. 19:38:25 i believe this is a nit not worth minutes of discussion time 19:38:25 wumpus: yes, but imagine someone has an automated system setup which calls bitcoin-cli... 19:38:41 in any case, can be discussed later or not. 19:38:42 :) 19:38:46 any more topics? 19:40:06 are we merging the bip9 activation parameters for testnet? 19:40:27 hmm - I don't see the activation parameters for segwit on testnet 19:40:39 how can it be live on testnet? 19:40:54 CodeShark: people running custom code I assume 19:40:57 CodeShark: in the segwit branch 19:41:02 oh 19:41:03 not in master 19:41:20 CodeShark: it was activated months ago 19:41:44 yes, but I'm still confused as to the release here 19:41:59 shouldn't testnet only run stuff that's been merged? 19:42:11 bip 9 activation can still be set. 19:42:13 no, we set the parameters 19:42:26 CodeShark: we wanted testing in a mixed enviroment with most nodes not upgraded. 19:42:26 and that allowed a miner to activate it 19:42:29 CodeShark: it was a very useful testing exercise 19:42:32 CodeShark: since thats realistic. 19:42:48 and segnet couldn't easily provide that. 19:43:00 sure, I'm all for the additional testing - I'm just concerned about breaking the master builds for testnet 19:43:10 CodeShark: it's a softfork, hurrah. 19:43:18 (although, there are so few testnet segwit nodes running right now that it really does not work without -addnode) 19:43:19 answer my own question: yes, the testnet activation is part of #7910 : https://github.com/bitcoin/bitcoin/pull/8149/files#diff-64cbe1ad5465e13bc59ee8bb6f3de2e7R191 19:43:39 gmaxwell: yes, but it will still break miners. however, if no issues arose as a result I'm fine with it 19:43:58 CodeShark: testnet miners are clearly already running it 19:44:05 yes, indeed 19:44:09 so how could merging break anything for them? 19:44:19 nvm, all's good 19:44:24 CodeShark: no issues appear to have arisen. (also testnet reorgs a lot in any case, so a little instability would have been an issue) 19:44:37 er, wouldn't have been 19:45:02 it is the first time a softfork is activated on testnet before it is in master thought, right? 19:45:02 in any case, so that would be an action to go with the testnet only builds: get more testnet nodes running upgraded so that it works without addnode. 19:45:14 Are the testnet seeds running the code that can give filtered responses? 19:45:16 indeed, and we can test the dns filtering 19:45:23 gmaxwell: jonasschnelli's is 19:45:27 not sure about petertodd's 19:45:37 oh, yes 19:45:49 https://github.com/bitcoin/bitcoin/pull/8204 19:46:49 petertodd: are you sure that's running the filtering code? 19:46:57 gmaxwell: the "testnet only" builds are just from master after merging segwit, right? 19:47:02 jtimon: yes 19:47:02 sipa: should be 19:47:03 good okay, so an action right after merge will be to get some more testnet nodes running it spun up, and cooridnate a pre-rc testnet-default/only binary. 19:47:10 sipa: I'll recheck 19:47:14 x9.seed.tbtc.petertodd.org gives many results, more than i'd expect 19:47:32 sipa: note that it is running with extra args to support rbf 19:47:40 jtimon: yes, likely with a trivial patch to change it to default to testnet (or require it). (and maybe a renamed binary) 19:47:42 sipa: so maybe there's a bug there that you haven't run into yet 19:48:16 petertodd: can i see the code for that? 19:48:24 gmaxwell: why the changes? aren't this power users? 19:48:33 after the meeting, please 19:48:44 sipa: it's with the command line arg; which incidentlaly was broken when I tried it (see my pullreq) 19:49:02 any other topics for this meeting? 19:49:31 doesn't seem so 19:49:58 #endmeeting