19:00:06 <wumpus> #startmeeting
19:00:06 <lightningbot> Meeting started Thu May 24 19:00:06 2018 UTC.  The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:06 <lightningbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
19:00:13 <promag> hi
19:00:31 <jimpo> hi
19:00:32 <murch1> hello
19:00:41 <jcorgan> lurking
19:00:49 <wumpus> #bitcoin-core-dev Meeting: wumpus sipa gmaxwell jonasschnelli morcos luke-jr btcdrak sdaftuar jtimon cfields petertodd kanzure bluematt instagibbs phantomcircuit codeshark michagogo marcofalke paveljanik NicolasDorier jl2012 achow101 meshcollider jnewbery maaku fanquake promag provoostenator
19:00:57 <achow101> hi
19:01:01 <jonasschnelli> hi
19:01:32 <wumpus> I guess the topic of priority today is 0.16.1
19:01:44 <jamesob> hi
19:01:50 <jnewbery> hello
19:01:56 <wumpus> #topic 0.16.1
19:02:03 <wumpus> there's a few backports left to do
19:02:31 <wumpus> or does #13317 include all of them?
19:02:31 <cfields> hi
19:02:32 <gribble> https://github.com/bitcoin/bitcoin/issues/13317 | [0.16.1] Remainig backports by MarcoFalke · Pull Request #13317 · bitcoin/bitcoin · GitHub
19:02:41 <MarcoFalke> wumpus: I left out the qt one
19:02:53 <MarcoFalke> I think it changes translations. I might do it in a separate pull
19:03:04 <wumpus> I'll add that one to "high priority for review" at least
19:03:07 <wumpus> MarcoFalke: ok, thanks!
19:03:45 <wumpus> so please all review that PR ^^
19:04:25 <wumpus> especially the non-trivial backports
19:04:42 <MarcoFalke> With regard to the other high priority prs in the project: I think we can remove the one from BlueMatt for now
19:04:51 <jtimon> https://github.com/bitcoin/bitcoin/pull/12172 is a small thing, but it is a bugfix, perhaps it makes sense to backport it?
19:04:52 <MarcoFalke> And add next week when the rebase is done
19:05:09 <sipa> #12172
19:05:13 <gribble> https://github.com/bitcoin/bitcoin/issues/12172 | Bugfix: RPC: savemempool: Dont save until LoadMempool() is finished by jtimon · Pull Request #12172 · bitcoin/bitcoin · GitHub
19:05:34 <wumpus> I think we should focus on 0.16.1 now, we'll get around to the other high priority stuff again next week
19:05:36 <MarcoFalke> jtimon: that lead to a ton of issues in our tests
19:05:46 <MarcoFalke> I'd prefer we didn't backport that one
19:06:10 <wumpus> I don't think it's terribly urgent to backport, maybe for 0.16.2
19:06:13 <jamesob> is #12431 worth a backport?
19:06:15 <gribble> https://github.com/bitcoin/bitcoin/issues/12431 | Only call NotifyBlockTip when chainActive changes by jamesob · Pull Request #12431 · bitcoin/bitcoin · GitHub
19:06:26 <MarcoFalke> jamesob: What bug does it fix?
19:07:00 <jamesob> potentially incorrect behavior in waitforblockheight (and anything else relying on rpc/blockchain.cpp:latestblock)
19:07:02 <jtimon> MarcoFalke: ok, no hurry, good to know someone tried it, now I'm curious about the issues in the tests and I'll play with travis and the backport
19:07:20 <MarcoFalke> jamesob: waitforblockheight is a hidden tests-only rpc
19:07:23 <wumpus> potentially how bad?
19:07:25 <wumpus> oh
19:08:22 <jnewbery> jtimon: #12863
19:08:23 <gribble> https://github.com/bitcoin/bitcoin/issues/12863 | mempool_persist.py failing in travis · Issue #12863 · bitcoin/bitcoin · GitHub
19:08:59 <jtimon> jnewbery: thanks
19:09:42 <wumpus> other topics?
19:10:37 <sipa> i want to bring up #13298 briefly
19:10:39 <gribble> https://github.com/bitcoin/bitcoin/issues/13298 | Net: Random delays *per network group* to obfuscate transaction time by naumenkogs · Pull Request #13298 · bitcoin/bitcoin · GitHub
19:10:39 <wumpus> anything else important for 0.16.1 that still needs to be done?
19:10:49 <sipa> (not for 0.16.1, to be clear)
19:11:04 <wumpus> (no problem, that was an orthogonal question)
19:11:08 <wumpus> #topic Random delays *per network group* to obfuscate transaction time
19:11:45 <sipa> i want to bring it up because it's a possibly significant effect on P2P transaction relay
19:11:59 <sipa> and it needs review beyond "does the code work"
19:12:02 <wumpus> I haven't had chance to look at it yet
19:12:17 <sipa> but it's also just local policy and not something that warrants a BIP imho
19:12:37 <wumpus> agree
19:12:55 <sipa> maybe there's not much more to say about that, just hoping to get people to think about it a bit
19:13:33 <wumpus> #action look at PR 13298
19:13:54 <wumpus> it's three days old, so people are excused not having an opinion on it yet :-)
19:14:09 <sipa> end topic :)
19:14:15 <cfields> thanks for bringing it up, I'll have a look as well...
19:14:23 <wumpus> other topics?
19:14:24 <cfields> it'd be nice to have a tool to model that kind of change, though.
19:14:47 <provoostenator> Topic suggestion: GUI prune setting
19:15:23 <wumpus> cfields: makes sense, maybe propose it in the PR
19:15:36 <wumpus> #topic GUI prune setting (provoostenator)
19:15:38 <provoostenator> AKA #13043
19:15:41 <gribble> https://github.com/bitcoin/bitcoin/issues/13043 | [qt] OptionsDialog: add prune setting by Sjors · Pull Request #13043 · bitcoin/bitcoin · GitHub
19:16:23 <jonasschnelli> Looks good,.. maybe orthogonal, but the prune settings should be in the intro...
19:16:26 <provoostenator> There's some confusion around whether using QT settings is appropriate for this, and I see three ways out.
19:16:29 <jonasschnelli> That's where it is probably most valuable
19:16:59 <provoostenator> 1. ignore the problem
19:17:02 <wumpus> I'm ok with most solutions, except writing to bitcoin.conf
19:17:12 <provoostenator> 2. go the writable config file route
19:17:13 <jonasschnelli> what wumpus sais
19:17:14 <kanzure> hi.
19:17:28 <provoostenator> 3. interpret a lack of prune= setting differently
19:18:15 <jonasschnelli> Can't the GUI settings not just override what is already set?
19:18:16 <instagibbs> (3) is interesting
19:18:33 <achow101> there currently are a few settings that are saved to qt settings that are shared between qt and bitcoind
19:18:40 <achow101> but bitcoind can't access
19:18:45 <provoostenator> If we go for (2) then I'd like to nominate #11082 for priority review
19:18:46 <gribble> https://github.com/bitcoin/bitcoin/issues/11082 | Add new bitcoin_rw.conf file that is used for settings modified by this software itself by luke-jr · Pull Request #11082 · bitcoin/bitcoin · GitHub
19:18:56 <jonasschnelli> If prune is set via confi/startip, disallow access in the GUI settings (only display it)
19:18:57 <achow101> so if we follow what has been done previously, then we ignore it
19:19:01 <wumpus> yes an additional writable config file would be fine
19:19:31 <jonasschnelli> Do we want four(!) levels of configuration?
19:19:35 <jimpo> What about augmenting the GUI to help you generate a default config file when none already exists?
19:19:37 <jonasschnelli> And eventually importing conf-files?
19:19:40 <wumpus> there have also been plans to add RPCs to change configuration settings, that'd require a similar thing, just don't write the -conf file it's just as likely to be in an read-only directory
19:20:23 <jonasschnelli> Would the rw_conf file be replacing the QSettings layer?
19:20:31 <wumpus> jonasschnelli: probably
19:20:31 <provoostenator> Right, I'd also like to get rid of QT settings completely and use a read-write (seperate) config file.
19:20:38 <wumpus> jonasschnelli: long term, at least
19:20:39 <jonasschnelli> That would be acceptable
19:20:58 <provoostenator> I wrote a migration away from QTSettings here: #12833
19:21:01 <gribble> https://github.com/bitcoin/bitcoin/issues/12833 | [qt] move QSettings to bitcoin.conf where possible by Sjors · Pull Request #12833 · bitcoin/bitcoin · GitHub
19:21:08 <jonasschnelli> But please not conf<->startup-cli-params<->QTSettings<->level4_rw_conf
19:21:14 <wumpus> nice
19:21:34 <jonasschnelli> Thanks provoostenator for working on that!
19:21:36 <achow101> since this is a problem that effects multiple options, can we just ignore the problem for now and deal with them all together at the same time with a better solution?
19:21:40 <wumpus> yes storing some of the settings in a different place has been problematic
19:21:54 <wumpus> (at least in the QSettings - because bitcoind can't get there)
19:22:00 <instagibbs> users could by and large be migrated over the rw unless they have a need for read-only
19:22:09 <instagibbs> for simplicity
19:22:09 <wumpus> the only thing that would idally be stored in the QSettings would be the data directory
19:22:22 <wumpus> well the bitcoin.conf is for human editing
19:22:28 <provoostenator> This migration would also work if it's done _after_ we add prune stuff to QTSettings.
19:22:35 <wumpus> the _rw is machine writable, all comments will be discarded etc
19:22:41 <instagibbs> ah hm
19:22:45 <jonasschnelli> yes
19:22:48 <provoostenator> But if we can get the proper solution over with, that might be better.
19:23:34 <jonasschnelli> For 12833 I'm also unsure about the term "Limit".
19:23:35 <wumpus> on the other hand, having things be dependent on each other is usually a bad idea, just draws out things
19:23:36 <jonasschnelli> Since this is not true
19:24:25 <provoostenator> jonasschnelli: what "Limit"?
19:25:09 <jonasschnelli> provoostenator: 12833 currently tells user "Limit block storage to: " which is not true... though the tooptip hints towards the right handling.
19:25:28 <wumpus> how is it not true?
19:25:47 <provoostenator> That's #13043
19:25:49 <gribble> https://github.com/bitcoin/bitcoin/issues/13043 | [qt] OptionsDialog: add prune setting by Sjors · Pull Request #13043 · bitcoin/bitcoin · GitHub
19:26:13 <provoostenator> Which I fixed, but screenshot is outdated.
19:26:28 <provoostenator> It's now "Prune &amp;block storage to"
19:26:45 <jonasschnelli> wumpus: AFAIK the 550MB assumption (if one sets 550) is still on 1MB blocks
19:26:56 <jonasschnelli> But 288min blocks & undos are enforced
19:27:31 <wumpus> jonasschnelli: right, that's an issue for the command line help too though, I think?
19:27:36 <jonasschnelli> but maybe we should fix that (if it turns out to be a problem) rathern then changing the word "limit" :)
19:27:42 <provoostenator> I'm not too worried about details of the text and minimum size. It's what we need to do about saving settings that seems to cause things to get stuck.
19:27:49 <jonasschnelli> Indeed
19:28:35 <wumpus> other topics?
19:28:41 <jonasschnelli> I also hoped we could educate the user withing the GUI a bit more about pruning... but can be done later via extending the intro.cpp
19:29:27 <wumpus> right, let's focus on getting the functionality in first. I think educating the user is a good thing, but having the PR depend on all those things being worked out is going to put it way past 0.17 probably.
19:29:54 <jonasschnelli> I have a short topic: sipa raised concerns about the scantxoutset RPC command
19:30:24 <jonasschnelli> (if that is something we want to discuss here)
19:30:45 <wumpus> so the feature freeze for 0.17 is 2018-07-16, less than two months away
19:31:02 <provoostenator> Also note that I don't think Mac shows the intro dialog.
19:31:09 <wumpus> #topic scantxoutset RPC command (jonasschnelli)
19:31:14 <jonasschnelli> provoostenator: it does at least for me
19:31:18 <wumpus> provoostenator: it should!
19:31:24 <jonasschnelli> https://github.com/bitcoin/bitcoin/pull/12196#pullrequestreview-121570579
19:31:40 <provoostenator> jonasschnelli: ok, maybe I need to delete more stuff for it to appear.
19:31:56 <jonasschnelli> Before continuing on #12196 we may want to discuss it it makes sense to have a scantxoutset command
19:31:59 <gribble> https://github.com/bitcoin/bitcoin/issues/12196 | Add scantxoutset RPC method by jonasschnelli · Pull Request #12196 · bitcoin/bitcoin · GitHub
19:32:03 <wumpus> you need to delete -whereever it stores the qsettings on Mac-
19:32:31 <wumpus> on UNIX you can pass the flag -resetguisettings but that's not easy on Mac I think
19:32:42 <jonasschnelli> (works on mac as well)
19:33:00 <wumpus> good :)
19:33:22 <provoostenator> Resetting gui settings didn't do it for me, but will debug some other time.
19:33:24 <achow101> jonasschnelli: what is the command supposed to do?
19:33:26 <jonasschnelli> The scan functionality allows utxo sweeping (rawsweeptransaction) with no block scanning
19:33:49 <jonasschnelli> You can pass in n pubkeys/addresses or even xpubs with a lookup window and it gives you back all unspents
19:33:56 <sipa> yeah i just mentioned that we preferably don't commit to having functionality that's hard to maintain in the future
19:33:58 <jonasschnelli> And eben a rawsweeptransaction to a single address
19:34:17 <sipa> fun.
19:34:18 <instagibbs> err what
19:34:21 <wumpus> services massacre
19:34:23 <cfields> irc unicorns...
19:34:55 <cfields> let's move to slack!
19:34:59 <cfields> (/s)
19:35:06 <wumpus> :-(
19:35:06 * jonasschnelli stabs cfields
19:36:27 <sipa> back to topic?
19:36:29 <sipa> yeah i just mentioned that we preferably don't commit to having functionality that's hard to maintain in the future
19:36:30 <jonasschnelli> Yes. I think sipa's point is a valid point. Do we want to maintain something that may be incompatible with future utxo handling (or new model=
19:36:32 <provoostenator> Such functionality seems quite useful for watch-only stuff
19:36:42 <provoostenator> Without actually having to import things into a wallet.
19:36:53 <sipa> which isn't a problem if it were implemented on top of an optional index
19:37:36 <jimpo> sipa: is the optional index a full scriptPubKey index?
19:37:52 <provoostenator> But without caching of some sort (or an index?) I'm guessing it'd be very slow.
19:37:57 <jimpo> or something less than that?
19:38:10 <jonasschnelli> sipa: what if we allow it for now an mention in the RN it may later require an optional index?
19:38:41 <sipa> yes, perhaps
19:38:44 <jonasschnelli> provoostenator: 30seconds for the whole index with an xpub & 1000 keys lookup window on a SSD/fastCPU machine
19:38:51 <jimpo> Or just have an explicit flag enabling the RPC now, even if it requires no additional index at present?
19:38:54 <jonasschnelli> *whole set
19:39:15 <sipa> yeah, that seems fine
19:39:30 <sipa> i do see the usefulness of scanning the UTXO set
19:39:38 <provoostenator> All the way back to the genesis block?
19:39:48 <jonasschnelli> jimpo: yes. But feels a bit after artificially holding back functions due to possible future work (which may never happen)
19:39:51 <sipa> provoostenator: it scans the UTXO set, not the blockchain
19:39:53 <wumpus> yes, scanning the utxo set would be incredibly useful
19:40:03 <wumpus> I've wanted this functionality since forever really
19:40:18 <provoostenator> Ah OK, it just gets unspends, not transaction history. Well that's still quite useful indeed.
19:40:27 <jonasschnelli> I wanted that in master before the fork-coins happend. :)
19:40:54 <wumpus> even if slow it's so much faster than scanning the entire chain
19:41:01 <wumpus> (without index)
19:41:07 <jonasschnelli> it would also allow importing wallets (no rescan required) if one don't care about the tx history
19:41:39 <wumpus> using importprunedfunds I guess?
19:42:06 <jonasschnelli> Yes..
19:43:02 <jonasschnelli> conclusion? Hide behind an artificial block-setting or risk it will not maintainable over time?
19:43:48 <jimpo> I think it's probably fine to risk it not being maintainable over time without an explicit index
19:44:19 <jonasschnelli> Yes. I would feel okay with that...
19:44:45 <jonasschnelli> Let me mention that risk in the RN and fix the PR in general /topic
19:44:50 <wumpus> so it will just become slower due to the linear scan?
19:44:53 <provoostenator> By "not being maintainable over time" do you mean if the UTXO set gets really large or is it a code maintenance things?
19:45:06 <jonasschnelli> from sipa:
19:45:07 <jonasschnelli> Overall, I'm unsure about this. This is functionality that is more easily provided by software that maintains a UTXO index by script, and is not possible in general if we'd move to a design like UHF (see mailinglist) or other UTXO avoidance techniques. Those are far away of course, and features like this can be made optional (like txindex is) if needed. I'm just generally unconvinced a full node is the best place to put
19:45:07 <jonasschnelli> this.
19:45:10 <wumpus> or is this 'unmaintainable' as in 'will give wumpus more headaches'?
19:45:22 <jonasschnelli> no.. changes of the general model
19:45:39 <sipa> wumpus: in a UHF model, without indexes, implementing a scan of the UTXO set requires going through the blockchain
19:46:03 <sipa> (where we store hashes of UTXOs rather than the UTXOs themselves)
19:46:08 <instagibbs> sipa, searchable index has been tried a few times, and sadly dropped, something like 3 times
19:46:12 <wumpus> sipa: so that's a far future thing right?
19:46:15 <instagibbs> not saying it's not the right way
19:46:28 <sipa> instagibbs: yes, my preference is that's implemented in other software
19:46:51 <jcorgan> i bet there's a dozen private implementations of tx/addr external indexing for xpub related things
19:47:10 <wumpus> I'd really like a way to scan UTXOs, my own appraoch was to stream the UTXO set over HTTP and do it client-side, but that ran into problems with the libevent http server :(
19:47:26 <jonasschnelli> Yes. But there is no fast access to the UTXO set from outside of our code-base IMO
19:47:36 <jcorgan> i do it with zmq notifications and the REST interface
19:47:46 * jonasschnelli curses zmq
19:47:56 <jcorgan> you're welcome :-)
19:47:57 * wumpus still wants to resurrect #7759 some day
19:48:00 <gribble> https://github.com/bitcoin/bitcoin/issues/7759 | [WIP] rest: Stream entire utxo set by laanwj · Pull Request #7759 · bitcoin/bitcoin · GitHub
19:48:18 <achow101> I've taken to modifying gettxoutsetinfo whenever I need to scan the utxo set. takes like 20 minutes though to scan the whole thing
19:48:21 <jonasschnelli> interesting... almost forgotten
19:48:28 <provoostenator> In this future where "we store hashes of UTXOs rather than the UTXOs themselves", wouldn't there simply be an optional "index" with the UTXO, which this RPC method could then move to?
19:48:43 <jonasschnelli> achow101: only if you are in debug mode? right... takes 30secs here
19:48:45 <wumpus> provoostenator: yes... exactly... I say it's a problem/option for then
19:49:12 <provoostenator> We'd have to make it clear that in the future the method would / might require an index.
19:49:26 <achow101> jonasschnelli: I think it was the operations I was doing, or my slow hdd
19:49:28 <wumpus> achow101: on ARM it's pretty slow (but still faster than scanning the whole chain!)
19:49:28 <sipa> it's the same issue as we had with txindex
19:49:41 <sipa> people build solution that assume txindex is always there... then we moved to a UTXO model
19:50:08 <sipa> and txindex became an inefficient optional thing
19:50:08 <wumpus> we can even deprecate the RPC then
19:50:14 <achow101> sipa: then it will be the same situation as getrawtransaction
19:50:49 <jimpo> with #13243, it should become less costly to build future indexes in the background...
19:50:50 <wumpus> I don't think we should reject useful, optional, functionality just because of some future data structure change
19:50:51 <gribble> https://github.com/bitcoin/bitcoin/issues/13243 | Make reusable base class for auxiliary indices by jimpo · Pull Request #13243 · bitcoin/bitcoin · GitHub
19:50:52 <sipa> achow101: my concern is not the incompatibility
19:51:07 <sipa> achow101: my concern is people building an ecosystem that assumes it's always possible and cheap to do
19:51:43 <sipa> but okay, i agree with the points here
19:52:07 <wumpus> sipa: I agree with that in general, but I'm not sure here
19:52:12 <jonasschnelli> jimpo: great work!
19:52:17 <provoostenator> The fact that it takes 30 seconds is helpful in that case. :-)
19:52:17 <jcorgan> if we want to encourage people to treat bitcoind as the "ground truth", instead of baking up their own stuff, giving them easier access to the "database" would help
19:52:49 <sipa> jcorgan: yes... except that the ground truth in the future may not be the UTXO set
19:53:06 <provoostenator> jcorgan: that too, would be nice to be able to get easy to export dumps of useful things, though not should what format.
19:53:08 <jcorgan> sure, but anything can happen in the future
19:53:17 <wumpus> sure, but anything can happen in the future <- that
19:53:21 <jonasschnelli> jimpo: the question is, if we not want to build a base for an external indexing daemon (outside of the Core project)
19:53:45 <wumpus> in any case this is hard to do out-of-process right now!
19:53:48 <wumpus> even harder than indexing
19:53:52 <jcorgan> i'd be happy with that, too, instead of making everyone else recreate it
19:54:05 <jonasschnelli> wumpus: Indeed
19:54:10 <sipa> it's also less a concern to add optional indexes now with jimpo's background index work
19:54:24 <sipa> before, new indexes always required ugly hacks all over the validation code
19:54:30 <wumpus> nice!
19:54:32 <jimpo> My guess is there will be ongoing tension between adding RPC functionality and keeping the node requirements small unless there are more options for users
19:54:33 <jcorgan> yes
19:54:46 <echeveria> it's a really bad idea to add an address index.
19:54:47 <provoostenator> Not too mention that you couldn't turn on/off txindex without reindexing everything.
19:54:52 <instagibbs> jimpo, an rpc call in every garage!
19:54:56 <jcorgan> i used to maintain the addrindex patch set and it got uglier and uglier over time
19:54:58 <echeveria> it means people will willyfully build insane systems.
19:55:02 <jimpo> haha
19:55:05 <sipa> echeveria: yes :(
19:55:06 <jonasschnelli> echeveria: agree
19:55:12 <sipa> but i fear they'll do so anyway
19:55:15 <echeveria> rather than sane, scalable things that don't require an ever growing index.
19:55:41 <jimpo> I tend to agree a better solution is to have a separate indexing service that doesn't do consensus but maintains the full chain state
19:55:42 <jonasschnelli> The question is, is it faster to index everything or to have Core running with 10k wallets
19:55:54 <jcorgan> +1 jimpo
19:55:54 <jimpo> and gets blocks via zmq
19:55:59 <sipa> decent wallet software shouldn't ever need to scan
19:56:04 <sipa> except for recovery
19:56:09 <jcorgan> i want to let bitcoind do the hard stuff (validation)
19:56:27 <jcorgan> but then i want to easily get at all the validated data
19:56:38 <wumpus> maybe it never *needs* to, but there are legit situations in which it's useful as a tool
19:56:43 <sipa> sure!
19:58:10 <jonasschnelli> jimpo: fork Core, ripout the validation stuff and you have an indexing daemon you can connect to your core node over p2p
19:58:31 <sipa> having a way to disable validation would also help with that :)
19:58:40 <sipa> anyway, this is turning into a philosophical discussion
19:58:40 <wumpus> jonasschnelli: why not use your indexing daemon?
19:58:57 <wumpus> yes, I think we're through the meeting
19:59:06 <jonasschnelli> wumpus: maybe. Not sure if we want another p2p library introduced
19:59:21 <wumpus> jonasschnelli: not into bitcoin core, I mean as external thing
19:59:27 <wumpus> #endmeeting