19:01:14 #startmeeting 19:01:14 Meeting started Thu Sep 5 19:01:14 2019 UTC. The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:01:14 Useful Commands: #action #agreed #help #info #idea #link #topic. 19:01:15 meeting? 19:01:18 hi 19:01:21 hi 19:01:29 hi 19:01:31 hola 19:01:32 hoy 19:01:48 hi 19:02:03 hi 19:02:11 hi 19:02:16 there are three proposed topics in https://gist.github.com/moneyball/071d608fdae217c2a6d7c35955881d8a 19:02:32 - proposed by jnewbery: review/merge #16704 or #16713 to remove worrying "unknown new rules activated (versionbit 1)" warning 19:02:34 - proposed by fjahr: Rolling UTXO set hash 19:02:34 https://github.com/bitcoin/bitcoin/issues/16704 | Dont warn about activated buried BIP 9 deployments by achow101 · Pull Request #16704 · bitcoin/bitcoin · GitHub 19:02:36 https://github.com/bitcoin/bitcoin/issues/16713 | Ignore old versionbit activations to avoid unknown softforks warning by jnewbery · Pull Request #16713 · bitcoin/bitcoin · GitHub 19:02:36 - proposed by achow101: avoid loading every wallet transaction into memory 19:02:43 hi 19:02:48 hi 19:02:51 hi 19:03:03 #bitcoin-core-dev Meeting: wumpus sipa gmaxwell jonasschnelli morcos luke-jr sdaftuar jtimon cfields petertodd kanzure bluematt instagibbs phantomcircuit codeshark michagogo marcofalke paveljanik NicolasDorier jl2012 achow101 meshcollider jnewbery maaku fanquake promag provoostenator aj Chris_Stewart_5 dongcarl gwillen jamesob ken281221 ryanofsky gleb moneyball kvaciral 19:03:42 but let's start with the usual 19:03:47 #topic High priority for review 19:04:04 7 blockers, 7 things chasing concept ACK https://github.com/bitcoin/bitcoin/projects/8 19:04:09 hi 19:04:18 I think its already there, but https://github.com/bitcoin/bitcoin/pull/16421 is close to landing and I still really want it for 19 19:04:20 note that the feature freeze for 0.19 is in 10 days 19:04:23 (and its a small diff!) 19:04:32 so we likely want to prioritize features that are close to ready now 19:04:54 right 19:05:08 would be nice if that makes it in 19:05:32 #16702 is done code-wise (I think), but perhaps it's suboptimal to add it to high prio at this point when we're close to the feature freeze. 19:05:35 https://github.com/bitcoin/bitcoin/issues/16702 | p2p: supplying and using asmap to improve IP bucketing in addrman by naumenkogs · Pull Request #16702 · bitcoin/bitcoin · GitHub 19:05:47 also #15759 19:05:51 https://github.com/bitcoin/bitcoin/issues/15759 | p2p: Add 2 outbound block-relay-only connections by sdaftuar · Pull Request #15759 · bitcoin/bitcoin · GitHub 19:06:22 gleb: yes, we might want to keep that for 0.20 19:07:31 also requires too much review to still make it to 0.19 anyway 19:07:38 still, great to hear you're making progress! 19:08:18 yeah, that sounds like too close to make it 19:08:35 Not super critical, and it's relatively new so not much time for review, but would help me to get #16766 as my work on OP_SECURETHEBAG wallet support depends on it 19:08:37 https://github.com/bitcoin/bitcoin/issues/16766 | wallet: Make IsTrusted scan parents recursively by JeremyRubin · Pull Request #16766 · bitcoin/bitcoin · GitHub 19:08:38 i'll review 15759 again 19:08:46 16702 is probably a bit far review-wise, though it would be nice, but 15759 is also really close and should god 19:08:47 And I think it is a bug 19:09:57 bug fixes can go after the feature freeze 19:10:11 yes 19:10:32 wasn't sure as it's a substantial behavior change for wallet, but fine :) 19:10:45 they can go any time ("scan parents recursively" sounds scary to me though, performance wise :) 19:11:04 it's cached to avoid exponential blowup 19:11:10 but otherwise, yeah 19:11:15 also depends on how new the bug is. I *think* it's ancient behavior that "normally" never hits 19:11:16 phew 19:11:17 anyways 19:11:46 instagibbs: agree 19:12:16 instagibbs: I think that is correct 19:12:29 with fancier wallet setups we may actually hit it :) 19:12:46 anyhow I've added it to high priority for review as requested, people can choose for themselves what to review 19:13:26 #topic remove worrying "unknown new rules activated (versionbit 1)" warning (jnewbery) 19:13:30 yes please 19:13:54 so I guess the disussion is: #16704 or #16713 19:13:55 https://github.com/bitcoin/bitcoin/issues/16704 | Dont warn about activated buried BIP 9 deployments by achow101 · Pull Request #16704 · bitcoin/bitcoin · GitHub 19:13:56 https://github.com/bitcoin/bitcoin/issues/16713 | Ignore old versionbit activations to avoid unknown softforks warning by jnewbery · Pull Request #16713 · bitcoin/bitcoin · GitHub 19:14:24 16713 is the simpler, easier to review fix 19:14:52 But I think it would be better to eventually get rid of these deployment parameters and have a more permanent solution that lets us reuse bits in the future 19:15:19 what disadvantages does it have compared to the other one? 19:15:25 bits can be reused as long as they don't overlap in time 19:15:27 16713 is updated as of a few hours to be even simpler; i think it lets us reuse bits fine? 19:15:42 e.g. why are there two open at all? 19:16:00 MarcoFalke: yup 19:16:10 I already closed the one by achow101 *hides* 19:16:26 MarcoFalke: smooth 19:17:24 ok, that concludes the discussion then I guess :) 19:17:26 my understanding of how consensus.vDeployments worked was that you couldn't define two forks with the same bit since they'd have to occupy the same index position and that's not possible 19:17:40 oh 19:17:42 but 16713 has changed since I last reviewed it and it looks very different 19:17:43 sorry 19:18:06 oh, maybe you are right when it comes to the implementation. Though, BIP9 does allow it 19:18:21 BIP9 definitely allows it, that was an important part of the design 19:18:25 achow101: vDeployments just matches the enum, the actual bits used are independant, and just have to not overlap per their corresponding timestamps 19:18:51 i haven't looked at the code in a while, but it was certainly intended to permit reuse of bits 19:19:26 the deployments array index is independent from the bip9 bit position 19:19:38 yep 19:19:57 achow101: see VersionBitsConditionChecker::Mask in versionbits.cpp, it pulls out the .bit field from BIP9Deployment struct 19:21:08 i'm probably wrong 19:21:24 It was confusing the first time I read it, I came to same wrong conclusion 19:22:10 achow101: there's also a unit test for overlapping bit usage on mainnet in src/test/versionbits_tests.cpp, "Verify that the deployment windows of [...]" 19:22:26 if even experienced developers get confused by it, some documentation/comments might help in that case 19:24:04 #action please review #16713 so that it can be merged asap 19:24:04 I think it shouldn't matter in practice, hopefully there are less than 27 softforks in the future 19:24:06 https://github.com/bitcoin/bitcoin/issues/16713 | Ignore old versionbit activations to avoid unknown softforks warning by jnewbery · Pull Request #16713 · bitcoin/bitcoin · GitHub 19:24:27 given that 'the future' is unbounded, that's a difficult statement 19:25:07 #topic Rolling UTXO set hash (fjahr) 19:25:40 Did anyone have time to look at the proposal? Any questions? 19:25:54 link? 19:26:01 fjahr, pitch it to us in a few sentences :) 19:26:23 https://gist.github.com/fjahr/fa4892874b090d3a4f4fccc5bafa0210 19:26:24 fjahr: it's not clear to me exactly what you're proposal :) 19:26:32 I have picked up Pieter Wuille's proposal from 2017 to use a rolling hash for the UTXO set hash. It deals with the problem of a long computation time of the UTXO set hash which results in a slow RPC call gettxoutsetinfo (can take several minutes depending on hardware). I investigated three hash functions: MuHash, ECMH and LtHash and started implementing them in Bitcoin Core for comparison. However only MuHash 19:26:32 has a rolling hash implementation so far and my LtHash code is not as optimized as MuHash and ECMH. I am looking for feedback on concept, which choice to make for the hash function and implementation details before filing a PR to Bitcoin Core. 19:27:04 it's a complicated question, as the right design may depend on how we intend to use the construction 19:27:12 fjahr: thanks for picking it up! 19:27:33 I think one use case is assumeutxo 19:28:01 sipa: by construction you mean the hash, right? 19:28:08 fjahr: right 19:28:24 assumeutxo (at least with a from-network-sync approach) will probably need more than just a single hash, as you want to be able to verify chunks etc 19:28:58 fjahr: if i recall correctly, the biggest question i had was how to prioritize computation time vs use-from-cache time 19:29:44 ECMH is slower to compute, but very fast to use from cached values (e.g. if you have precomputed the "diff" ECMH hash a transaction has, applying to a global sum is super fast) 19:29:51 but MuHash is faster is overall computation time 19:31:21 From my benchmarks ECMH was faster overall but I am not sure why MuHash did not perform as you expect in you Mail 19:32:07 where do you see a discrepancy? in the hash-to-group-element operation, or in the multity? 19:32:11 *multiply? 19:32:24 actually i think the discussion of what hash to pick is less important for this meeting 19:32:34 we should probably focus on ways to integrate 19:33:16 ok, and also if there is enough interest for this 19:33:25 it's super cool 19:33:48 Are both constructions one-way? 19:33:55 How does that interplay with reorgs 19:33:58 In terms of integration I chose to implement this as an index, any feedback on this? 19:34:11 Maybe I should read more out of band of here... 19:34:12 fjahr: that really depends what for 19:34:56 jeremyrubin: hashes can also be removed again so no problem with reorgs 19:35:00 i think as an index it's hard to use precomputation 19:35:15 which you really want if you want the rolling part 19:37:27 sipa: what would you suggest in terms of integration? 19:38:04 that really depends on what we want to use it for 19:39:17 sipa: could have it rolling, but in it's own slightly delayed thread like tx indexes, without worrying about precomputation, at least if you don't want to enforce it in consensus 19:39:39 there's 20 minutes to go and still a topic left, let's move on? 19:39:52 fjahr: let's talk more after the meeting 19:40:00 sipa: sure 19:40:13 #topic avoid loading every wallet transaction into memory (achow101) 19:40:18 thanks 19:40:18 btw, the secp ECMH code you have looks great 19:40:37 This is a wallet topic, and probably better for the wallet meeting, but that's next week.. 19:40:50 is there any hurry? :-) 19:41:00 I was thinking about ways to reduce the memory footprint of loaded wallet files 19:41:17 concept ACK anyhow 19:41:24 jup. also ack 19:41:36 achow101: seems hard 19:41:42 I was wondering if anyone who was more familiar with the wallet tracking part of the wallet knew if this was attempted before or would majorly break something? 19:42:00 it always seemed unnecessary to me to keep all transactions, even historical ones with all outputs spent, in memory 19:42:15 There is a PR where all wallettxns where kept in mem 19:42:31 +different DB formst 19:42:34 but yes, it's such a part of the current wallet design, it's definitely not going to be easy 19:42:49 #5686 (very old) 19:42:51 https://github.com/bitcoin/bitcoin/issues/5686 | [Wallet] replace BDB with internal append only (logdb) backend by jonasschnelli · Pull Request #5686 · bitcoin/bitcoin · GitHub 19:42:53 achow101: you mean not _show_ them anymore 19:42:59 or just not load them in memory? 19:43:15 not load them into memory, so it would read from disk when the full tx is needed 19:43:21 right 19:43:26 I plan on just keeping UTXOs and txids in memory 19:43:40 achow101: i wish you good luck :) 19:43:44 heh 19:43:46 outward behavior probably shouldn't change 19:44:05 Why not just loading everything into memory? 19:44:15 I think 1000+ wallets are OOS 19:44:26 * jeremyrubin wonders how big the largest wallet is 19:44:37 OOS? 19:44:41 out of scope 19:44:42 out of scope 19:44:48 there are some heavy users of bitcoin core which have to re-cycle their wallet once in a while because it becomes to big 19:45:00 i don't think memory usage is the problem there 19:45:10 big wallets also take a while to load, although I don't expect this to effect that 19:45:12 just the size of the maps that's being traversed for a multitude of operations 19:45:18 loading time is likely the problem, yes 19:45:31 then again that's directly related 19:45:37 ideally this will also reduce the time it takes to calculate things like balances since not every single transaction will be iterated over 19:45:54 yes, exactly 19:45:56 achow101: i'm very very scared of things like that 19:45:59 Are exchanges and other large-scale wallet-holders recommended to use Core purely for networking, and maintain their own non-Core wallets? 19:46:01 it needs a completely new design i'm afraid 19:46:16 sipa: how so? 19:46:27 everything is transaction oriented in the current wallet 19:46:34 jarthur: some use core, but can't mention any names 19:46:40 new transactions that can others to become conflicted etc 19:47:04 changing that to a UTXO model and keeping it in sync with the list of transactions... sounds very hard 19:47:19 right, but I think those can still be done by just a list of txids, spent prevouts, and utxos 19:47:24 is it a more important goal to reduce the number of txns or the amount of data each one is storing? 19:47:45 but you still need all the dependency tracking between transactions to compute the utxos 19:48:04 you can cache the utxo list; that would probably be a worthwhile performance improvement 19:48:18 that would likely be a better first step 19:48:28 but getting rid of the transaction entirely... i don't see how 19:48:36 also, maybe just loading a neutered transaction without input scripts, because we don't need those 19:48:49 achow101: yeah that can work 19:48:53 achow101: that's what I was getting at :) 19:49:18 achow101: also, don't let me discourage you if you see a good way to implement it :) 19:49:32 i'm happy to be convinced otherwise 19:49:35 there's definitely some cut-off possible, I mean, transactions from years ago can't really become conflicted anymore 19:49:35 achow101: a good first step might be to cache the scriptPubKey of inputs in a WtX 19:50:36 i suspect thay in native descriptor wallets the IsMine check will be a lot faster than it currently is 19:50:38 anyways, that's all. just wanted to get some opinions before diving in 19:50:43 at some point spent transactions deep enough in the chain can be permantly archived 19:51:24 might be non-trivial to come up with criteria but I don't think every transaction needs to be potentially active forever 19:51:37 achow101: good luck ! 19:53:03 #endmeeting