19:00:39 <meshcollider> #startmeeting
19:00:39 <lightningbot> Meeting started Fri Aug 16 19:00:39 2019 UTC.  The chair is meshcollider. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:39 <lightningbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
19:00:46 <meshcollider> #bitcoin-core-dev Wallet Meeting: wumpus sipa gmaxwell jonasschnelli morcos luke-jr sdaftuar jtimon cfields petertodd kanzure bluematt instagibbs phantomcircuit codeshark michagogo marcofalke paveljanik NicolasDorier jl2012 achow101 meshcollider jnewbery maaku fanquake promag provoostenator aj Chris_Stewart_5 dongcarl gwillen jamesob ken281221 ryanofsky gleb moneyball
19:01:03 <kanzure> hi
19:01:04 <sipa> hi
19:01:08 <jnewbery> hi
19:01:52 <achow101> hi
19:02:51 <provoostenator> hi
19:02:53 <ariard> hi
19:02:54 <meshcollider> Firstly apologies from me for the lack of review/merging I've done recently, had an especially busy project at uni which just finished so I'll get a few PRs merged today hopefully
19:02:59 <meshcollider> Any topics?
19:03:16 <achow101> People should test and review #16341
19:03:20 <gribble> https://github.com/bitcoin/bitcoin/issues/16341 | Introduce ScriptPubKeyMan interface and use it for key and script management (aka wallet boxes) by achow101 · Pull Request #16341 · bitcoin/bitcoin · GitHub
19:03:29 <meshcollider> Yes :)
19:05:08 <ariard> I'm trying to rationalize the way we track txn in the wallet, if any ones has opinions on it feel free to review : https://github.com/bitcoin/bitcoin/pull/16624
19:05:33 <ariard> I hope it moves things in the right direction to solve long-standing issues like #7315
19:05:34 <gribble> https://github.com/bitcoin/bitcoin/issues/7315 | Minor wallet issue with conflicted txs · Issue #7315 · bitcoin/bitcoin · GitHub
19:06:21 <achow101> topic: replacing coin selection
19:06:27 <achow101> *topic suggestion
19:07:00 <jnewbery> I think that's a nice change. In general there are a few places where serialization details leak into the wallet implementation unnecessarily. 16624 cleans up some of that.
19:07:50 <meshcollider> Yes it looks nice at first read
19:08:10 <meshcollider> #topic replacing coin selection (achow101)
19:08:18 <achow101> ping instagibbs
19:08:42 <sipa> random selection fallback?
19:08:45 <achow101> current coin selection uses branch and bound which uses effective value, and then falls back to the old algo which does not use effective value if that fals
19:08:58 <achow101> it would be nice to retry random selection
19:09:04 <achow101> as the fallback and get rid of the old thing entirely
19:09:16 <achow101> the previous pr was #13307 and it just kind of stalled out
19:09:19 <gribble> https://github.com/bitcoin/bitcoin/issues/13307 | Replace coin selection fallback strategy with Single Random Draw by achow101 · Pull Request #13307 · bitcoin/bitcoin · GitHub
19:10:16 <sipa> i guess the only reason not to would be because we're not entirely sure that the algorithm works reasonably well on all kinds of utxo sets
19:10:19 <achow101> I think the main issue was that with SRD as the fallback, we weren't consolidating as much as the legacy algo did
19:10:59 <sipa> hmm, i was worrying about the opposite, where it might occasionally pick a crazy high number of inputs despite being unnecessary
19:11:47 <achow101> the simulations I did previously showed that basically for all runs, there were a lot more utxos in the wallet on average
19:12:17 <sipa> simulations only show behavior for the type of wallets you have data for, though
19:12:48 <achow101> right..
19:13:47 <achow101> I think we could mitigate the possiblity of choosing too many utxos by selecting multiple times and choosing the "least waste"
19:13:53 <achow101> kind of like we do for BnB already
19:13:55 <sipa> but when you do things like "run SRD 3 times, and pick the cheapest result", the risk of overly large input set goes down dramatically, but you're also worsening the consolidation further
19:14:30 <meshcollider> Have you got some results that indicate it is somehow better in some aspect then? This seems like the naive approach, why do we want to switch it in?
19:15:22 <achow101> part of the whole consolidation thing I think was that the legacy algo would actually select dust (negative effective value) for inputs whereas in BnB and SRD, we would ignore those entirely
19:15:39 <sipa> i suspect that's an inherent tradeoff
19:15:51 <sipa> anything that does not consume dust will have worse consolidation
19:15:58 <achow101> meshcollider: https://gist.github.com/achow101/242470486265d3f21adab08f65b9102c old simulation results. it generally resulted in better fees
19:16:37 <achow101> also, from murch's original paper, he found that it helped BnB get better exact matches over other methods
19:17:12 <achow101> one of the issues with the legacy selection is that it does not use effective value which makes doing some other things difficult, according to instagibbs
19:17:32 <sipa> does SRD use effective value?
19:17:46 <achow101> yes
19:18:14 <sipa> hmm, remind me how?
19:18:39 <sipa> it just doesn't consider anything with negative effective value?
19:18:50 <achow101> basically
19:19:22 <achow101> effective value just subtracts the fee required for that input from its value so that we don't have to constantly guess what the fees are going to be
19:19:46 <sipa> ah right that too, it lets you predict in one pass whether you have enough
19:20:17 <sipa> i guess you could make the consolidation/dust tradeoff configurable by making it allow negative EV, but limiting how negative it can be (for example as a percentage of the payment value)
19:20:18 <achow101> yes, so we can get rid of this stupid loop in CreateTransaction
19:20:32 <meshcollider> Can't the same effective value check just be added to the legacy code
19:22:07 <achow101> probably
19:22:28 <achow101> but the legacy code is also kind of pointless as it also tries to do an exact match which we are already doing in BnB
19:23:46 <achow101> one of the main things that SRD did in my simulations was that it made BnB have more exact matches which helps with privacy
19:26:03 <meshcollider> This is the thing I don't like about the coin selection, its all super heuristic and the tradeoffs are really hard to identify or compare
19:26:43 <sipa> it is
19:26:47 <meshcollider> If you think this is worth revisiting will you reopen the PR?
19:27:00 <sipa> but it's also an algorithm that has weathered a lot
19:27:01 <achow101> instagibbs said he would take a crack at it
19:28:22 <meshcollider> Ok
19:28:23 <achow101> it would be useful to have other kinds of simulation data rather than just the one or two datasets I got
19:28:39 <sipa> ping murch :)
19:29:02 <achow101> he says no everytime I ask
19:29:13 <achow101> won't give me the secrets to Bitgo's coin selection :p
19:31:19 <sipa> achow101: just an idea, what happens if you simulate something SRD like, but permit individual utxos whose EV is > -0.2 * feerate*size
19:31:33 <sipa> rather than > 0
19:32:27 <sipa> this can be discussed outside of the meeting of course
19:32:51 <achow101> it probably should be. simulations take a long time
19:33:32 <achow101> I would assume something with more mean utxos than the simulation that allows any negative EV utxo but not as bad as those that exclude
19:33:58 <sipa> actually that rule is equivalent to "only utxos that are positive EV at 80% of the feerate"
19:35:05 <achow101> in terms of fees, including negative ev still does better than current
19:35:08 <meshcollider> You had a lot of simulations when we switched the BnB initially, what happened to those datasets?
19:35:51 <achow101> they're all in public gists on github
19:36:14 <achow101> i have the data somewhere too
19:37:17 <meshcollider> Yeah I was wondering if they have more variation in types of utxo set than what you've tested so far
19:37:32 <achow101> those datasets were still only based on two actual datasets, just with different variations of filtering and ordering of the data
19:38:52 <meshcollider> Ah
19:39:30 <meshcollider> Alright so we just see what instagibbs does and discuss further in the PR?
19:39:45 <achow101> all of the different BnB simulations were mostly for different feerates
19:40:17 <achow101> yeah. i guess the question was really whether this was something even worth pursuing again
19:42:50 <sipa> i think so
19:44:29 <meshcollider> If sipa thinks so then I think so too ;)
19:44:37 <meshcollider> Any other last topics?
19:46:04 <meshcollider> #endmeeting