19:01:40 #startmeeting 19:01:40 Meeting started Thu Jul 13 19:01:40 2017 UTC. The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:40 Useful Commands: #action #agreed #help #info #idea #link #topic. 19:01:40 Hi 19:01:41 i'm here for 30 mins 19:01:44 hi 19:01:51 I'm here until I pass out <.< 19:01:59 hi 19:02:03 proposed topics? 19:02:03 hi 19:02:19 possible topic: 0.15 feature freeze 19:02:31 it's soon, july 16th 19:02:39 hi 19:02:43 review begging as first topic ? 19:02:48 PSA: https://github.com/bitcoin/bitcoin/issues/9961, feature freeze is sunday (which is an awful day, it'll probably be monday in practice) 19:03:19 #topic high priority for review 19:03:25 #link https://github.com/bitcoin/bitcoin/projects/8 19:03:32 everything marked as 0.15? 19:03:36 :p 19:03:50 we can delay the feature freeze to July 16th, 2018 to avoid a Sunday 19:04:00 hi. 19:04:11 #10711 can be removed from high-priority as its not for 0.15 19:04:13 https://github.com/bitcoin/bitcoin/issues/10711 | [tests] Introduce TestNode by jnewbery · Pull Request #10711 · bitcoin/bitcoin · GitHub 19:04:31 agree with that jnewbery? 19:04:38 Please add #10706 to high priority i guess, since the PR's it was depending on were merged 19:04:40 https://github.com/bitcoin/bitcoin/issues/10706 | Improve wallet fee logic and fix GUI bugs by morcos · Pull Request #10706 · bitcoin/bitcoin · GitHub 19:04:45 i think i asked him, but he stepped away 19:04:56 I'm removing the 0.15 milestone from #10240 (will def. not make it) 19:04:59 https://github.com/bitcoin/bitcoin/issues/10240 | Add HD wallet auto-restore functionality by jonasschnelli · Pull Request #10240 · bitcoin/bitcoin · GitHub 19:05:04 jonasschnelli: no 19:05:07 since it seems #8498 cannot be priority for some reason that scapes me, what about #10757 from me ? 19:05:09 https://github.com/bitcoin/bitcoin/issues/8498 | Near-Bugfix: Optimization: Minimize the number of times it is checked that no money... by jtimon · Pull Request #8498 · bitcoin/bitcoin · GitHub 19:05:11 https://github.com/bitcoin/bitcoin/issues/10757 | RPC: Introduce getblockstats to plot things by jtimon · Pull Request #10757 · bitcoin/bitcoin · GitHub 19:05:15 sipa: you want that in? 19:05:21 jonasschnelli: if we remove that from 0.15, we must revert the hd split 19:05:28 I tend to agree with achow101 - it's better to use the 0.15 tag now for high priority for review 19:05:36 sipa, ? 19:05:41 proposed topic then: Hd split / hd restore 19:05:46 #9566 can be removed from 0.15 19:05:48 https://github.com/bitcoin/bitcoin/issues/9566 | threading: use std::chrono for timestamps by theuni · Pull Request #9566 · bitcoin/bitcoin · GitHub 19:05:51 wumpus: i think its still helpful to distinguish between hope for 0.15 and really need 19:06:00 https://github.com/bitcoin/bitcoin/milestone/25 19:06:20 I think if we do not fix the restore we need to disable HD by default. The current situation can pretty easily lead to funds loss. 19:06:31 but yes i also agree we need to clean up the 0.15 milestone list 19:06:41 cfields: bumped to 0.16 19:06:50 thanks 19:06:58 gmaxwell: that would be very confusing to users, since older versions have HD 19:07:00 wumpus: yess please remove 10711 19:07:11 I can work on the HD restrore. But It's pretty complex with pruning / encrypted wallets... the PR is already large and will get bigger... 19:07:21 If there is enough review power, we can try for 0.15 19:07:28 E.g. just pick up a walled you'd previously saved, rescan won't move the keypool forward, and you'll end up missing transactions (then discarding wallets with money), and handing out addresses to people you already gave to other people and misattributing payments. 19:07:28 I can have it overhauled by tuesday 19:07:39 jnewbery: done 19:07:51 gmaxwell: Yeah.. is also true for all other wallets with gap limits of 5 (most do) 19:07:59 We should def. do better 19:08:00 can just the wallet-format-touching parts of HD restore be prioritised? eg, move out the actual restoring logic? 19:08:22 jonasschnelli: this is not true for anything that automatically tops up the keypool 19:08:28 What about just provide HD restore for non-pruning (to reduce the size)? 19:08:43 mhm, only #10652 in project 8... 19:08:45 https://github.com/bitcoin/bitcoin/issues/10652 | Small step towards demangling cs_main from CNodeState by TheBlueMatt · Pull Request #10652 · bitcoin/bitcoin · GitHub 19:09:06 sipa: most HD wallets in the wilde stop topping the window futher up if a gap of >5< keys where found 19:09:15 jonasschnelli: yes 19:09:19 jonasschnelli: to that extent that thats true at least in other cases those wallets behaviors are well documented and the interfaces is built around them, they're also used almost exclusively for personal use, rather than industrial use... (and it's not completely true because if there isn't a long gap they do handling it right and we do not) 19:09:20 jonasschnelli: but we don't top up at all 19:09:39 sipa: Yes. Not saying that is better. :) 19:09:47 jonasschnelli: and hd split makes it worse, because it risks reusing a key that was previously used as change as a payment address 19:09:50 I just wanted to re-state the HD restore in general is a broken thing 19:09:53 making you miss it as incoming payment 19:09:59 So what should we do? 19:10:02 fix it 19:10:09 #10240 is a bug fix 19:10:11 https://github.com/bitcoin/bitcoin/issues/10240 | Add HD wallet auto-restore functionality by jonasschnelli · Pull Request #10240 · bitcoin/bitcoin · GitHub 19:10:14 Agree 19:10:19 Okay. Then we have more time.. :) 19:10:54 Right now this is responsible for several serious bugs in our behavior, which regressed vs the past, and will predictable result in funds loss through several different vectors. I don't see an easy workaround to prevent exposure-- I thought perhaps refusing to load a wallet if the tip doesn't match the chain tip, but thats too cumbersome and disruptive. 19:10:57 Since we have great reviewers, .. I'm convinced we get it in 19:11:09 10240 (when ready) is an example of something that should also be on high-priority... it's going to take some review time and its important to get in (in addition to 0.15 milestone) 19:11:42 Okay. I though it not going to make it for 0.15 thats why I moved focus away.. but I see the issue now better 19:11:50 *thought 19:11:55 jonas: anything I can do to help for 10240? Would you like me to rebase it? 19:12:02 ok, will add 10240 19:12:12 jnewbery: Please take over if you can 19:12:15 jonasschnelli: rescanning beyond your pruning depth should already be an issue? what do we do in that case? 19:12:21 achow101: what about the signrawtransaction splitting stuff, is that still aimed for 0.15? 19:12:22 sure. I'll take it 19:12:29 sipa: the PR halts validation 19:12:35 jnewbery: You are now my personal hero for the day. 19:12:45 jonasschnelli: no, i mean right now 19:12:53 what do we do if we try to rescan beyond the prune depth 19:12:55 morcos: I'd like it to be. and the validateaddress stuff as that is related to #7965 19:12:56 https://github.com/bitcoin/bitcoin/issues/7965 | Remaining instances of ENABLE_WALLET in `libbitcoin_server.a` · Issue #7965 · bitcoin/bitcoin · GitHub 19:13:06 sipa: I don't know: :/ 19:13:17 I guess you get an expection 19:13:45 so, i think pruning is not relevant for 10240 19:13:56 the rescan calls just say no if you try that. 19:14:01 it's a problem right now if you rescan beyong the pruning depth, and it remains so 19:14:27 A large part of 10240 is about haling the full node in pruning... dropping that would reduce the review workload 19:15:21 jonasschnelli: i think it should stop regardless of pruning 19:15:21 So. Drop the pruning option from 10240? 19:15:31 it's crazy that your wallet would go out of sync with your node 19:15:39 that's a totally unsupported state right now 19:15:45 From the PR on encrypted wallets: 19:15:46 Same as above, but, If we hit the gap limit with an encrypted wallet, we can't topup the keypool. In that case, we just pause the sync (not the node, only the wallet). 19:15:54 maybe that can be enabled later, once the wallet is more independent from the node 19:16:19 but i think 10240 should just stop sync entirely if your wallet is encrypted and the keypool runs out 19:16:21 Anyone familiar enough with constraint solving to help me out with this model? https://gist.github.com/RHavar/0710144c713033d42f8f443a99fefbb7 19:16:25 sipa: well, if you use a backup wallet you have the same state 19:16:28 rhavar: not now, meeting 19:16:36 rhavar, ask again in 45 min :P 19:16:40 jonasschnelli: at startup; not anymore after rescan 19:16:51 during normal operation the wallet is always in sync with the node 19:16:53 Yes. Thats true 19:16:59 * luke-jr wonders if a halted node will rewind based on headers 19:17:11 All that because of hardened derivation! 19:17:27 it's also easy to avoid; using 10000 keys in the keypool 19:17:36 (indeed, which I also keep recommending) 19:17:58 You don't avoid it, you just make the timespan for the possible impact smaller 19:18:07 okay 19:18:12 And 10000 is just inefficient 19:18:41 well, i think all of that isn't the priority now 19:18:58 What about only allowing non-hardened derivation for encrypted wallets and disable all pkey export calls? 19:18:59 for 0.15, we need to have automatic marking of seen keys 19:19:10 jonasschnelli: yes, i like that, but not 0.15 19:19:34 Okay. jnewbery will focus on 10240 (he will rebase and overhaul I guess) 19:19:48 jonasschnelli: we have a program that requires >1GB ram, runs best with >8GB ram, that does hours of processing just to start up-- I don't think worrying about 320k of key material is a major concern. 19:19:57 awesome; let's discuss further on the 10240 PR 19:20:06 (also 1000 works too, it 10k is really too much.) 19:20:13 gmaxwell: should not be a concern. But it's still an inefficient fix for the problem we have 19:20:26 sipa: ack. Thanks jnewbery 19:20:53 Inefficient compared to what? Inefficient to taking away private key export? In efficient compared to even one moment of one users time? 19:21:04 no problem. Topic suggestion: #10650 19:21:07 https://github.com/bitcoin/bitcoin/issues/10650 | Multiwallet: add RPC endpoint support by jonasschnelli · Pull Request #10650 · bitcoin/bitcoin · GitHub 19:21:25 gmaxwell: Inefficient compared to support pub key derivation for encrypted keys or to topup the keypool on the fly 19:21:53 jonasschnelli: are you actively trying to sabotage the project? 19:22:07 gmaxwell: please 19:22:13 ? 19:22:22 jonasschnelli: there are good reasons to support hardened and unhardered derivation both; adding a feature is not a substitute for fixing a problem we have 19:22:24 gmaxwell: was that a joke or a serious question? 19:22:46 jonasschnelli: using non-hardened derivation implies you'll need to have a big keypool; it comes with that design choice 19:23:20 Sorry to be rude, but I am just gobsmacked about aruging that setting the keypool to be big "on the order of tens or hundreds of kilobytes" is opposed compared to this long saw about public derrivation; which we aren't doing for the wallet at least now. 19:23:46 So it seems to me like that you're intentionally in broken directions because you disagree with another decision. 19:24:29 er intentionally pushing in 19:24:47 I though avoiding keypool with non-hardened derivation may be seen as a benefit for some of the users.. but it seems that i'm wrong. But at least it's not intentional sabotage 19:24:50 let's move on from this at least duing the meeting, i think we all agree that the keypool can be bigger than 200 regardless of otehr chnages we make 19:25:11 Do we? it keeps getting argued against. 19:25:22 thats why i ended it by saying we all agree. :) 19:25:25 well, having non-hardened derivation with disabled key export is a perfectly fine _feature_ - but it's not usable for everyone (some people need key export), and for those users, we'll need to be able to deal with hardened derivation 19:25:35 so let's do that 19:25:36 next topic 19:25:39 ack 19:25:40 NicolasDorier: how does #9728 interact with rescan ? 19:25:40 Yes. Agree 19:25:42 https://github.com/bitcoin/bitcoin/issues/9728 | Can create Watch Only HD wallet with -hdwatchonly by NicolasDorier · Pull Request #9728 · bitcoin/bitcoin · GitHub 19:25:57 jtimon, it doesn't do anything special for now 19:26:12 same as hardened 19:26:39 other topics? 19:26:52 jnewbery suggested 10650 19:27:24 I think we're almost there with 10650. Only major sticking point is not having a default wallet when there are multiple wallets loaded 19:27:25 instagibbs: I see, so it kind of depends on #10240 ? 19:27:28 https://github.com/bitcoin/bitcoin/issues/10240 | Add HD wallet auto-restore functionality by jonasschnelli · Pull Request #10240 · bitcoin/bitcoin · GitHub 19:27:46 jtimon, we can chat offline about that after meeting 19:27:53 #topic Multiwallet: add RPC endpoint support 19:27:59 sure 19:28:12 I just pushed the overhaule of 10650 that fixes the points reported by ryanofsky jnewbery and morcos 19:28:16 *overhaul 19:28:20 jonasschnelli: cool 19:28:28 #10650 19:28:31 https://github.com/bitcoin/bitcoin/issues/10650 | Multiwallet: add RPC endpoint support by jonasschnelli · Pull Request #10650 · bitcoin/bitcoin · GitHub 19:28:45 great! Will review 19:28:49 thanks jonas 19:29:06 nice 19:29:08 yes, excellent. woo! 19:29:21 I very much dislike passing wallet by name. That just makes the GUI side ugly 19:29:42 luke-jr: you mean the selecting walltes by name? 19:29:43 luke-jr: as opposed to what? (sorry, i'm not up to date) 19:29:54 sipa: as opposed to passing a CWallet* on the JSONRPCRequest 19:30:09 that seems like something that's easy to change later 19:30:09 jonasschnelli: the GUI would have to go CWallet* -> string -> CWallet* 19:30:13 and hope it matches the right one up 19:30:22 sipa: I suppose, yes 19:30:35 yes, indeed, can we avoid long discussions about small details that don't matter for correctness? 19:30:42 we really want this in before the feature freeze 19:30:47 so let's be pracical about it 19:30:48 <3 19:30:57 Yes. WalletID or similar can be done later. 19:31:32 yep 19:32:12 One thing that is a bit cumbersome is that you have to remove the -wallet argument from bitcoin-cli when calling a non wallet command 19:32:35 The endpoint node/wallet split is not very practical from the -cli use perspective 19:32:41 well it makes some sense 19:32:45 hmm, bitcoin-cli reads bitcoin.conf, doesn't it? how does that interact? :/ 19:32:59 then make the cli command handle that internally? 19:33:17 also, we can take some clunkyness with this expiremental feature in 0.15. 19:33:21 gmaxwell: Yes. I though of that. 19:33:23 e.g. fix cli later. 19:33:32 Yes. Sure. 19:33:46 If you want to use multiwallet now, you need to add/remove -wallet when fiddling with -cli 19:33:50 IMO a clean separation between wallet and non wallet commands is good 19:33:51 fixing cli could mean changing the -wallet= to something else 19:33:54 even if it seems cumbersome in the beginning 19:34:04 jonasschnelli: how do yuo remove it, if it's in bitcoin.conf? 19:34:08 luke-jr, yeah, something that means "use wallet" not "load wallet" 19:34:13 Yes. Its good. 19:34:14 luke-jr: ? I can't follow 19:34:16 bitcoin-cli calls with -wallet that don't need it could just ignore the extra argument and spit a warning somehwere or something? 19:34:23 I mean it'd be easy to put every non-wallet command on wallet endpoints as well, but that's something that is awfullly hard to change later 19:34:30 jonasschnelli: typical multiwallet use case has wallet=abc.db \n wallet=def.db in bitcoin.conf 19:34:36 jonasschnelli: there are a bunch of things that the wallet - even after separation - will need access to (like fee estimates, mempool, ...)... i think it's fine if those remain inside the v1/wallet API (and also accessible as node commands) 19:34:37 wait.. that's actually a good point! 19:34:42 jonasschnelli: bitcoin-cli will get these options too. 19:34:55 Yes.. haven't tested that. :/ 19:35:13 instagibbs's -usewallet or similar seems like a good solution 19:35:16 it likely takes the first wallet arg and uses that 19:35:28 Yes. I think -usewallet is better 19:35:40 so i think i may agree with having pretty much everything available through the wallet endpoint 19:35:43 luke-jr, something like that, if it's not complicating something else 19:35:48 (but not getinfo) *ducks* 19:36:22 It's an expiremental feature, API isn't table. The endpoint can be leaky for now. 19:36:26 it's also easier to collapse args later than to split them, if we end up regretting it 19:36:31 I guess we can leave it for now we just need to mark the /v1 *EXPERIMENTAL* in the release notes 19:36:36 yes please 19:36:42 ack 19:36:45 s/table/stable/ 19:37:03 And I think the alternative is to not have it at all, which isn't preferable. 19:37:05 should do that anyway 19:37:14 absolutely 19:37:41 as long as it doesn't cause regressions for single-wallet mode 19:38:13 that would be unacceptable - but everything new is experimental 19:38:55 agree 19:39:26 I guess user feedback will also help us how to extend this further 19:39:48 indeed 19:40:25 But once 10650 is in, we have finally usable multiwallet in Core! That's a big step. 19:40:30 jonasschnelli: it is! 19:40:44 And it wasn't the first try. 19:41:19 hehe multiwallet slipped so many releases it's a shame 19:41:23 yeah, is a nice feature to anounce in 0.15, even if as experimental 19:42:17 just to repeat some other happy results briefly: my reindex-chainstate to 450k with infinity dbcache runs about 40% faster on master than on 0.14.2 19:42:40 sipa: what do you mean by "infinity dbcache"? 19:42:41 wow 19:42:42 which reminds me, someone should really write a release notes section about all the wonderful perf improvements in 0.15 19:42:59 petertodd: dbcache sufficient for the entire utxo sets 19:43:02 wumpus: I've already been talking to drak about some blog posts and whatnot on it. 19:43:06 everything from crc instruction support in leveldb to the new and better database formats, to faster validation, etc 19:43:21 sipa: ah, cool! 19:43:27 wumpus: don't forget tx validation caching; that's massive for performance at the tip 19:43:31 ^ 19:43:44 sipa: +1 19:43:46 (just somewhat harder to benchmark and give cool numbers for) 19:44:30 petertodd: around 6-8GB in 0.15, in practice (and more in 0.14.2, due to the blowup at flushing time) 19:44:47 sipa: that's still pretty small fortunately :) 19:45:07 Was 2GB not that long ago. 19:45:37 gmaxwell: be interesting to know why UTXO growth has stopped temporarily... 19:46:27 if you know, post it here: https://bitcoin.stackexchange.com/questions/56513/why-has-utxo-set-stopped-growing-since-2017-06-03 19:46:29 (total size has been shrinking for a month now) 19:47:13 could easily be someone ran out of money for an attack, and is spending the coins again 19:47:57 #link https://bitcoin.stackexchange.com/questions/56513/why-has-utxo-set-stopped-growing-since-2017-06-03 19:49:06 any other topics? 19:49:08 many more possible explanations... 19:49:25 maybe a reminder that Tokyo Core is in a few weeks 19:50:21 2 weeks specifically 19:50:54 https://coredev.tech/tokyo.html lots of unconfirmed invites still 19:51:26 kind of a hike for those not already in area for other reasons 19:51:29 oh, missed that one, I guess I can confirm now I'm not attending 19:51:30 I'm not coming to tokyo core, but will be in the SF one in september 19:51:41 wumpus: great! 19:51:52 SF is pretty full booked.. 19:52:13 Not saying we running out of space, but >20 confirmed 19:52:33 sipa: you're coming too? great 19:52:54 wumpus: yes, of course 19:53:08 (i live nearby) 19:53:10 do we have a location for the sf meetup? 19:54:06 achow101: https://coredev.tech/nextmeeting.html 19:54:08 yes, but it's still possible it will change 19:55:09 depending on just how large the room needs to be I guess :-) 19:55:28 will we finally activate segwit there? *ducks* 19:55:40 petertodd: the day it stopped growing is the day alphabay shut down 19:55:52 instagibbs: it'll already be active by then 19:56:01 luke-jr: hopefully 19:56:03 rhavar: interesting! 19:56:13 rhavar, ? what date did it go down 19:56:22 the 3rd i think 19:56:46 been holding/dropping since 3rd of June though 19:56:58 might be contributing factor still.. 19:57:10 sorry, is meeting done 19:57:11 oh, sorry -- I'm a month off 19:57:12 3 minutes 19:57:24 alphabay shut down the 3rd of july 19:57:33 and that question was 3rd of june 19:57:51 my SWAG: I think internalizing customer transaction costs did the equiv of fees for exchange trading in China 19:58:07 seems time to end the meeting, speculation about the reason for the stop of utxo growth is interesting but not a bitcoin core meeting topic 19:58:08 #endmeeting