19:00:04 #startmeeting 19:00:04 Meeting started Thu Nov 8 19:00:04 2018 UTC. The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:00:04 Useful Commands: #action #agreed #help #info #idea #link #topic. 19:00:17 hi 19:00:19 #bitcoin-core-dev Meeting: wumpus sipa gmaxwell jonasschnelli morcos luke-jr btcdrak sdaftuar jtimon cfields petertodd kanzure bluematt instagibbs phantomcircuit codeshark michagogo marcofalke paveljanik NicolasDorier jl2012 achow101 meshcollider jnewbery maaku fanquake promag provoostenator 19:00:25 hi 19:00:47 topic proposals? 19:01:12 suggested topic: utility vs rpc? (#14671) 19:01:14 https://github.com/bitcoin/bitcoin/issues/14671 | Utility to replace RPC calls that dont need wallet or chain context · Issue #14671 · bitcoin/bitcoin · GitHub 19:01:23 hi 19:01:29 yes that's a good one, that comes up very often 19:02:02 preferably a separate repo entirely IMO 19:02:03 suggested topic: [RFC] for future of RNG #14623 19:02:04 https://github.com/bitcoin/bitcoin/issues/14623 | [RFC] Built-in RNG design · Issue #14623 · bitcoin/bitcoin · GitHub 19:02:05 hi 19:02:38 #topic high priority for review 19:02:44 hi 19:02:56 https://github.com/bitcoin/bitcoin/projects/8 anything to add or remove? 19:03:14 luke-jr achow101 phantomcircuit sipa ryanofsky currently have open PRs there 19:03:15 Some of the hardware wallet support would be nice to add, e.g. the overhaul PR by sipa. 19:03:31 well, I rebased rwconf (#11082), but now #14532 is in the way :x 19:03:33 https://github.com/bitcoin/bitcoin/issues/11082 | Add new bitcoin_rw.conf file that is used for settings modified by this software itself by luke-jr · Pull Request #11082 · bitcoin/bitcoin · GitHub 19:03:35 https://github.com/bitcoin/bitcoin/issues/14532 | Never bind INADDR_ANY by default, and warn when doing so explicitly by luke-jr · Pull Request #14532 · bitcoin/bitcoin · GitHub 19:03:39 he already has #14477 19:03:41 https://github.com/bitcoin/bitcoin/issues/14477 | Add ability to convert solvability info to descriptor by sipa · Pull Request #14477 · bitcoin/bitcoin · GitHub 19:03:54 luke-jr I'll review that one soonish, as well as rebase my work on top of it. 19:04:01 i think of my PRs, 14646 is probably blocking more 19:04:16 #14646 19:04:18 https://github.com/bitcoin/bitcoin/issues/14646 | Add expansion cache functions to descriptors (unused for now) by sipa · Pull Request #14646 · bitcoin/bitcoin · GitHub 19:04:18 ok, will replace it 19:04:25 sdaftuar found a bug in #14336 on os x (which i think is fixed now), think it needs some more testing there 19:04:28 https://github.com/bitcoin/bitcoin/issues/14336 | net: implement poll by pstratem · Pull Request #14336 · bitcoin/bitcoin · GitHub 19:04:44 wumpus: thanks 19:04:44 hi 19:05:14 hi 19:05:18 #action test PR 14336 on OSX 19:06:05 anything else? 19:06:55 wumpus: 13501 19:06:58 ok, time for next topic I guess 19:07:14 #13501 19:07:16 not exactly a topic, but ##hwi in case people want to contribute to or lurk hww support work 19:07:18 https://github.com/bitcoin/bitcoin/issues/13501 | ~~Correctly~~ terminate HTTP server by promag · Pull Request #13501 · bitcoin/bitcoin · GitHub 19:07:45 #action join ##hwi to contribute to or lurk hardware wallet support work 19:07:48 ops not that 19:07:58 #14670 19:08:00 https://github.com/bitcoin/bitcoin/issues/14670 | http: Fix HTTP server shutdown by promag · Pull Request #14670 · bitcoin/bitcoin · GitHub 19:08:06 Quick topic: EOL date for 0.15 https://github.com/bitcoin-core/bitcoincore.org/pull/630 19:08:19 ok added 19:08:40 #topic RPC or utility? 19:08:48 provoostenator, MarcoFalke ^ 19:08:52 Just to repeat my latest comment there: Many users already have bitcoind running and don't care about performance, so I don't see why we'd want to stop providing utility functions for them. 19:08:59 most recently this came up in #14667 19:09:00 https://github.com/bitcoin/bitcoin/issues/14667 | Add deriveaddress RPC util method by Sjors · Pull Request #14667 · bitcoin/bitcoin · GitHub 19:09:01 If we want to (in addition) provide a high-performance library or executable for those functions, that seems partly orthogonal. And maybe even out of scope for our project? 19:09:09 why not both? 19:09:15 +1 achow101 19:09:17 and #14476 19:09:20 https://github.com/bitcoin/bitcoin/issues/14476 | RPC method encodescript by mrwhythat · Pull Request #14476 · bitcoin/bitcoin · GitHub 19:09:30 achow101: it complicates bitcoind, more code to maintain, more API to support 19:09:35 achow101: Sure, but has a bit of maintanence overhead 19:09:36 are existing utils well-tested and used? 19:09:37 I agree that performance really isn't an issue for at least some users. There's a reason people run NodeJS :-) 19:09:40 bitcoin-qt already exposes RPC functionality without network overhead 19:09:43 more testing as well; API coverage is not free 19:10:04 I don't think performance is the only consideration 19:10:14 wumpus: but what about code that we already have, which can be used to implement a useful function, but would require a lot of work to reimplement elsewhere? 19:10:33 sipa: it removes the flexibility to change that code later I guess 19:10:34 preferably a separate repo entirely IMO 19:10:38 say, signing PSBT with a descriptor 19:10:40 Also, I guess there'd be design tradeoffs that make the library/executable usable by some and not by others, so maybe just too much to worry about for us? 19:11:06 I think there's two seperate questions: 1) do we want a sep. utility 2) do we want to deprecate / stop adding that stuff to the RPC 19:11:24 descriptors are a bitcoin core specific thing (at least for now), it would be trivial amount of code to implement given the existing infrastructure, but a lot of work to do elsewhere 19:11:26 I think having duplicate code that does basically the same thing can be problematic. We've already had issues where bitcoin-tx has fallen out of sync with the equivalent RPC methods. 19:11:30 I would say (1) would be nice in general, even in a seperate repo. I'm a bit reluctant about (2) for reasons Marco pointed to. 19:11:31 I mean obviously the ship to have bitcoind be a minimal consensus node implementation sailed, but I'm not sure we should be tacking on more that *could* be implemented outside it and is independent on any node functionality 19:11:39 2) NACK on stop adding utility to RPC 19:11:51 but I'm sure this is pointless for me to argue 19:11:53 also nack 2 19:11:56 so do whatever you want.. 19:12:08 MarcoFalke, most of the utility like things can trivially be RPC and some utility binary 19:12:35 For 1) it is not even clear if it should be a util binary or a util library 19:12:36 it's quite a nuisance to implement lots of things as another project, if only because of serialization stuff 19:12:41 It would be good to have a _generic_ way to map utility functions to RPC functions, so that it's easy to keep well tested. 19:12:44 MarcoFalke: library + utility 19:12:51 MarcoFalke: see libbase58 for example 19:12:54 Right now every RPC is bespoke. 19:13:10 butting in. this is more of a wallet vs node implementation thing than a RPC vs node thing, right. 19:13:36 esotericnonsense: we're talking about RPCs that don't need a wallet or node/chainstate/blocks 19:13:41 esotericnonsense, there's lots of non-wallet utility rpc's 19:13:57 Neither wallet, nore node, since both have state. We are talking about stateless util functions 19:14:01 sure, i suppose what i'm saying is that they're more wallet-like than node-like (if they deal with transactions, signing, etc) 19:14:11 esotericnonsense: maybe 19:14:16 maybe, but AFAIK they're also enabled with the wallet disabled 19:14:31 always compiled in 19:14:32 decodescript is pretty independent of anything 19:14:40 yes 19:14:43 say, decoding a transactions, has nothing to do with wallet imo 19:15:11 longer term if the wallet and node become more 'split out' then it seems like you're going to have a whole bunch of wallet rpc's build up over time that aren't relevant for the node, it seems unavoidable 19:15:12 I don't think the current ones should be removed, but i'm not sure about adding new ones 19:15:26 We don't talk about keystore or "chainstore" rn 19:15:35 esotericnonsense: i really think you're talking about something else 19:15:36 I would rather that it not be separated into a separate repo. a big reason for me to have utility functions in core is that most of the framework for doing something is already there. in a separate repo, you have to reimplement stuff like serializations, etc. which is a major pain in the ass 19:15:36 (agree that decodescript doesn't really belong in either) 19:15:58 achow101: if only those parts of core were available as a library heh... 19:16:18 i think that bitcoin core, for better or worse, implements certain featuresets - and it makes sense to implement those features completely, even if some are stateless 19:16:33 that doesn't mean we should add arbitrary utilities because they may be useful for something 19:16:39 sometimes seems we're using RPC to compensate for the fact there's no library or API separation 19:16:43 i think provoostenator's point about RPC being non-automated is interesting 19:16:44 yes 19:17:04 i don't really care for decodescript being in our codebase really - it's trivial to do anywhere 19:17:06 bitcoin-cli add 1 2 19:17:16 things that have no state don't need *remote* procedure calls 19:17:18 like effectively a lot of RPC's are just a wrapper over a raw function call, but with the "stable ABI" pushed outwards a bit 19:17:19 but signrawtransaction makes perfect sense, as we offer raw transaction functionality 19:17:28 luke-jr yes, so we can claim that Bitcoin Core is turning complete :-P 19:17:37 luke-jr: bitcoin-cli 1 2 add surely. RPN-RPC. 19:18:22 We already kind of have frozen random util addition, I think? 19:18:46 instagibbs: yes, there has been sort of a policy like that for a while 19:18:47 wumpus, we definitely use the rpc stuff to compensate for there not being a good library 19:18:49 instagibbs: that was the idea! but recently a few new PRs have been opened in that regard which prompted this discussion 19:18:50 instagibbs: I thought so, but I think I've seen more going in since then -.- 19:19:04 and apparently it's still controversial 19:19:10 maybe just saw PRs, perhaps not going in 19:19:15 Lots of folks us a microservices like architecture, wher eyou have bunch of applications on seperate servers talking to eachother. RPC is useful for that, but less so once there are enough libraries that can replace it. 19:19:16 luke-jr, for example? the ones I've suggested have gone nowhere :P 19:19:19 ah ok 19:19:22 fair enough 19:19:29 So I agree with wumpus that it's paritally a chicken egg thing 19:19:36 provoostenator: that RPC is useful is not the discussion, does it need to be in bitcoin core! 19:19:54 we can't add everything that is useful 19:20:00 that's infinite scope creep 19:20:03 Of course 19:20:09 absolutely 19:20:18 for utility stuff that doesn't go in bitcoin core, is there some way to make it available to end-users who are not developers? 19:20:23 e.g. a library doesn't really help them 19:20:33 gwillen: can you give an example? 19:20:34 gwillen: you could make your own RPC server, btcutilityd, or so 19:20:35 unless there's a utility wrapper around it that also ships with core 19:20:37 gwillen, see bitcoin-tx 19:20:46 it could be completely separate from bitcoin core 19:20:50 as it shares not state 19:20:50 gwillen: it's sort of what I've been doing in my projects, but it's not public 19:20:51 gwillen: i think wumpus' argument is not having such things in our codebase entirely 19:20:58 gwillen: libraries should have wrappers for shell scripts to use them, but having trouble seeing a case where a real end user would need such things 19:20:59 not just not exposing it over RPC 19:21:19 Right, so a seperate utility repo could also include an RPC wrapper. 19:21:22 or rather, it's "public" as in it's OSS/free software, but it's not marketed or known or anything (I expect there's a lot of stuff out there like this) 19:21:39 if people really need RPC, let someone do a RPC-to-exec server :/ 19:21:58 I'm just trying to prevent the codebase complicating even more, I mean, I think we should focus on consensus issues in bitcoin core, it's hard enough 19:22:04 +1 19:22:08 wumpus, agreed 19:22:15 so anyways 19:22:15 sipa, suggest changing the GetRandBytes functions to GetWeakRandBytes to make it more obvious, maybe 19:22:16 I don't think RPC is necessary for quick utilities with no state, but I do think commandline is necessary 19:22:20 versus just a library 19:22:21 wumpus: so, just to gauge your opinion here, imagine we didn't have any raw transaction or PSBT functionality included, and it is being added now; would you argue for only having signrawtransactionwithwallet, but say signrawtransactionwithkey needs to be implemented elsewhere, even though it shares 95% of the code? 19:22:45 gwillen: https://github.com/bitcoin/libbase58/blob/master/clitool.c 19:23:30 I thikn that's reasonable actually. Especially considering that the _wallet_ could also be split out at some distant future. 19:23:34 sipa: IMO, ideally we'd abstract that code into a library itself, and use it in Core 19:23:41 sipa: sure, it's always possible to argue some grey area where it seems harmless to add because it's only a small difference, maybe that would be ok, just not complicated things that we don't use otherwise 19:23:42 or rather, use it in "wallet split out of Core" 19:23:44 luke-jr: agree, but that's a separate discussion 19:24:14 The number of RPC methods that would be left if we strip out utilities and the wallet would be nice and small. 19:24:26 wumpus: my point here is that the "feature creep" is in deciding whether to have raw transaction support or not; the fact that part of the necessary RPC calls happen to be stateless isn't 19:24:45 I'm not arguing to remove the current ones or so, just prevent an onslaught of new ones 19:24:58 (this is also orthogonal to whether it should be implemented as RPC or separate utility; i'm just discussing having the functionality somewhere in our codebase) 19:25:17 note also, as someone doing development on the GUI right now, that the GUI might reasonably need utilities that don't touch the wallet 19:25:28 (by strip out I just mean not adding more than we already have, only deprecating if they become a problem) 19:25:38 unless the goal for the GUI also is to be functionally minimal, which I don't think i tis 19:25:51 if only the GUI needs it, it could be a library at the GUI side 19:26:21 gwillen: I think the goal is to split the GUI out too, but in the meantime, I see your point 19:26:52 so maybe it's better to discuss per PR 19:27:28 wumpus: say for example, if somewhere were to propose a set of RPCs just to do script debugging, fully stateless, I would nack that - there is no reason for that to be in core 19:27:46 say, deriveaddress seems small and is only using what is already in the code base: #14667 19:27:48 https://github.com/bitcoin/bitcoin/issues/14667 | Add deriveaddress RPC util method by Sjors · Pull Request #14667 · bitcoin/bitcoin · GitHub 19:28:26 right; i think deriveaddress is a very natural part of "supporting output descriptors" 19:28:44 sipa: script debugging is a libbitcoinconsensus thing 19:28:53 luke-jr: i very strongly disagree :) 19:29:08 sipa: it's inherently part of consensus code 19:29:11 (but that's also another discussion, i think; i was just giving an example) 19:29:16 true 19:29:21 but I guess 'encodescript' (#14476) is more controversial 19:29:24 https://github.com/bitcoin/bitcoin/issues/14476 | RPC method encodescript by mrwhythat · Pull Request #14476 · bitcoin/bitcoin · GitHub 19:29:41 luke-jr: no, debugging does not belong in consensus 19:29:47 yeah i don't think we should have had decodescript 19:31:07 wumpus: I guess I just mean the hooks, which can't really be anywhere else 19:31:07 it's also not difficult to implement on the application side, unlike the cryptography of deriving addresses 19:31:45 so that was on the topic of "whether something belongs in the codebase or not"; another question is, for the things that do, separate utility/library, or RPC, or both? 19:32:27 sipa: if it's being pulled in because it's tied to something that should be, then presumably that other thing is RPC already 19:32:36 luke-jr: that's fair 19:32:57 if it could be a separate utility/library, it wouldn't need to be in the codebase I guess 19:33:09 (but maybe it makes sense to do all 3?) 19:33:19 I think there is not too much cost in doing both (but currently we don't have a utility/library) 19:33:31 MarcoFalke: we do have bitcoin-tx 19:33:43 sipa: ideally I'd prefer having a library, this more general and allows for implementing a RPC service that does the same, the other way is more limiting 19:33:48 luke-jr: that is not useful for general utility functions 19:33:56 MarcoFalke: even if we rename it? 19:34:07 luke-jr: i think tx is too specific 19:34:17 its arguments are all transformations on transactions 19:34:18 good luck renaming it, heh 19:34:23 I guess there's no real need to have just one tool 19:34:28 the advantage of an executable is that it's separate 19:34:41 even for libraries, it makes sense to have different ones for substantially different tasks 19:34:50 it's annoying to interface with in most languages, though :) 19:34:53 perhaps we should change topics, i need to go in 10-15 minutes 19:35:10 also need to spawn a new process for every call 19:35:16 which perf-wise is even worse than RPC 19:35:36 #topic [RFC] for future of RNG #14623 19:35:37 https://github.com/bitcoin/bitcoin/issues/14623 | [RFC] Built-in RNG design · Issue #14623 · bitcoin/bitcoin · GitHub 19:35:50 suggest changing the GetRandBytes functions to GetWeakRandBytes to make it more obvious, maybe 19:35:59 phantomcircuit: good idea 19:36:18 just a quicj topic, i'd like to ask for opinions if we want to go in that direction 19:36:19 otherwise seems good to me 19:36:22 if so, i'll work on it 19:36:43 possibly want to measure how fast various sources of entropy are? 19:36:50 my only concern is if RNG gets too slow for non-HD wallets to work reasonably, but I doubt that's an issue 19:36:50 phantomcircuit: yeah 19:37:08 #action read issue 14623: Built-in RNG design 19:37:10 luke-jr, the openssl rng is already lol slow 19:37:11 (since we don't make them, so I guess new keys are only generated one at a time) 19:37:23 does slowness matter for anywhere we need strong crypto? 19:37:24 This isn't actually going to slow things down is it, long term might even speed it up? 19:37:29 I mean, now that key generation is HD 19:37:41 wumpus: my point is that key generation is not necessarily HD 19:37:44 generating the initial seed isn't perf critical and certainly not high bandwidth 19:37:59 is there anything besides that? 19:38:08 phantomcircuit: actually, GetWeakRandBytes is perhaps a bit confusing too; the data returned would be cryptographic strength, seeded by all available randomness sources - just not reseeded regularly 19:38:33 GetRandomStrengthRandBytes 19:38:36 I guess the master key for the wallet; but that's als oa one-time thing 19:38:38 GetWeaklySeededRandBytes 19:38:40 luke-jr, by far the slowest thing about keygeneration is the ridiculous number of fsync calls from bdb 19:38:51 luke-jr: what phantomcircuit says ^ 19:39:03 generating thousands of keys per second should be trivial 19:39:19 sipa, Strong/Mediocre/Weak 19:39:27 heh 19:40:20 phantomcircuit: i think two levels is sufficient :) 19:40:26 so ok, weak and strong :) 19:40:35 sipa, the distinction is really about things like vm state copies where we hope that /dev/random would know about that right? 19:40:39 was about to ask what the use case is for the middle level :P 19:40:53 If you take GitHub emoji reactions as comments it looks like there's a pretty good approval consensus anyway 19:41:36 phantomcircuit: so even weakrand in my suggestion would invoke high-accuracy timestamp (rdtsc etc), so it'd be highly unlikely that you get the same random numbers in two instances of a VM copy 19:41:36 what if: if you need high speed key generation, switch to HD wallet 19:41:43 if not, the current fallback is fine 19:41:50 it will be slower but work 19:41:58 timezones are difficult. 19:42:12 RNG is really not the bottleneck for key generation 19:42:15 wumpus: it sounds like it's a non-issue either way? 19:42:21 yes 19:42:22 i think it's a non issue 19:43:12 #topic EOL date for 0.15 (meshcollider) 19:43:28 https://github.com/bitcoin-core/bitcoincore.org/pull/630 19:43:33 Yeah just wanted to check this date was sane to put on the website 19:43:34 most nodes are still using 0.15 :x 19:43:52 luke-jr: EOL would be august next year. Seems reasonable 19:43:57 what is the proposed date? 19:43:58 yeah, hopefully it will change by then 19:44:01 2019-08-01 19:44:04 sipa, indeed 19:44:20 seems reasonable 19:44:20 that seems.. far enough away 19:44:21 luke-jr: it's maintenance end now anyway 19:45:04 Ok that's all then, unless anyone disagrees :) 19:45:06 (correction: 0.16 is actually now the most-used branch, barely) 19:46:33 yep, please do any ACKing in the PR 19:46:39 any other topics? 19:47:39 RPC is used by many companies as the only interface to bitcoin at all; not sure i understand the scope of discussion about moving RPC into another utility or project but yeah be careful i guess. 19:48:09 kanzure: only a few utility RPCs, not the whole thing ;) 19:48:16 kanzure: the topic is not of moving any current RPCs away 19:48:24 one possible alternative I can think of would be to give bitcoin-cli a way to talk to multiple backends 19:48:35 then you can easily put utilities in a separate codebase 19:48:36 gwillen: it already can? 19:48:41 without any user issues 19:49:14 luke-jr: I mean something like dispatching based on which command it's given 19:49:15 hm. but isn't the issue the ongoing maintenance of ensuring RPC works (integrating it into the test suite etc) 19:49:19 for a seamless experience 19:49:21 gwillen: ew :< 19:49:29 kanzure: as far as I'm concerned the discussion is about the addition of new pure, utility-only RPCs, what is the crit to add them, or never do it? but removing the current ones is out of the question I think 19:49:34 i'm struggling to understand how seperating it makes it "simpler" if you keep the maintenance burden anyway 19:49:54 something out of scope for bitcoind but exchanges and merchants could really benefit from tracking slightly more state on bitcoind end (like "updates since last sync"- specific to an external application which otherwise has to write their own since-last-sync tracking code etc) 19:50:07 no way... 19:50:11 wumpus: got it, understood. i didn't understand context. my apologies. 19:50:19 wumpus: yeah i said it's out of scope :-) 19:50:31 really, tracking consensus is hard enough, a REALLY hard problem 19:51:18 I think there's too little respect for just how hard that is, even as a single concern, we can't pull in anything that is marginally useful to anyone else also 19:52:10 I have to say that I don't have much use for a cryptocurrency node that isn't either A) a wallet, or B) providing other services with currency network functionality. If separating the concerns out still gives me those things as I need them, no complaints. 19:52:16 it sounds to me like provoostenator's idea of making RPC more 'automated' somehow would be useful; in the sense that you have a ton of functions that could just be exposed over rpc by like, function name or something, if they're stateless, without doing anything 19:52:33 and then it's "someone else's job" to maintain the bitcoin library to RPC shim which ensures RPC backward compatibility 19:53:09 again, you're speaking of 'backward compatibility' 19:53:15 there's no talk of dropping anything here 19:53:34 the discussion was about adding... 19:53:40 wumpus: right, what I mean is that, when you add stuff, it's not a problem a priori, it becomes a problem because over time you're maintaining it 19:53:48 gwillen, please no 19:54:42 there is also the overhead of dealing with a PR, merging, rebasing, basically actually doing the work 19:54:43 going to close the meeting we're out of topics 19:54:45 esotericnonsense: i would phrase that as automatically generated bindings 19:55:01 #endmeeting