19:00:49 <wumpus> #startmeeting
19:00:49 <lightningbot> Meeting started Thu Jan 30 19:00:49 2020 UTC.  The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:00:49 <lightningbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
19:00:51 <provoostenator> hi
19:00:58 <sipa> hi
19:01:01 <fjahr> hi
19:01:02 <emilengler> hi
19:01:04 <sipsorcery> hi
19:01:04 <promag> hi
19:01:06 <hebasto> hi
19:01:06 <gwillen> hi
19:01:08 <nehan_> hi
19:01:12 <wumpus> #bitcoin-core-dev Meeting: wumpus sipa gmaxwell jonasschnelli morcos luke-jr sdaftuar jtimon cfields petertodd kanzure bluematt instagibbs phantomcircuit codeshark michagogo marcofalke paveljanik NicolasDorier jl2012 achow101 meshcollider jnewbery maaku fanquake promag provoostenator aj Chris_Stewart_5 dongcarl gwillen jamesob ken281221 ryanofsky gleb moneyball kvaciral ariard digi_james amiti fjahr
19:01:14 <wumpus> jeremyrubin lightlike emilengler jonatack hebasto jb55
19:01:17 <jonasschnelli> hi
19:01:42 <meshcollider> hi
19:02:02 <jonatack> hi
19:02:16 <wumpus> one pre-proposed topic in https://gist.github.com/moneyball/071d608fdae217c2a6d7c35955881d8a: topic idea collection for physical meeting (kanzure)
19:02:44 <elichai2> Hi
19:02:45 <wumpus> PSA: 0.19.1rc1 was released, please help testing and report any issues you find to the bug tracker
19:04:00 <jeremyrubin> Hiya!
19:04:03 <wumpus> also, the 0.20 feature freeze is in one and a half month (see #17432)
19:04:04 <gribble> https://github.com/bitcoin/bitcoin/issues/17432 | Release schedule for 0.20.0 · Issue #17432 · bitcoin/bitcoin · GitHub
19:04:24 <wumpus> any last minute topic proposals?
19:04:33 <jeremyrubin> #proposedmeetingtopic I'd love to chat about the mempool project and share trajectory
19:04:44 <wumpus> thanks
19:05:24 <wumpus> let's start with the usual then
19:05:30 <wumpus> #topic High priority for review
19:06:23 <achow101> #16528 pls
19:06:27 <gribble> https://github.com/bitcoin/bitcoin/issues/16528 | Native Descriptor Wallets using DescriptorScriptPubKeyMan by achow101 · Pull Request #16528 · bitcoin/bitcoin · GitHub
19:06:36 <wumpus> we've managed to merge a few things this week! the first PR for the asmap-based clustering of peers went in, and step 3 of sipa's serialization improvements
19:07:05 <wumpus> that leaves 7 blockers, 1 bugfix and 6 items chasing concept ACKs
19:07:06 <jeremyrubin> #17925 I think is more or less RTM, and there's a lot of work waiting on it. Not sure it needs to go in high prio since things seem to be moving that way.
19:07:06 <sipa> and the wallet boxes
19:07:08 <gribble> https://github.com/bitcoin/bitcoin/issues/17925 | Improve UpdateTransactionsFromBlock with Epochs by JeremyRubin · Pull Request #17925 · bitcoin/bitcoin · GitHub
19:07:13 <wumpus> sipa: yes!
19:07:43 <fanquake> hi
19:08:54 <wumpus> added #16528 and #17925
19:08:58 <gribble> https://github.com/bitcoin/bitcoin/issues/16528 | Native Descriptor Wallets using DescriptorScriptPubKeyMan by achow101 · Pull Request #16528 · bitcoin/bitcoin · GitHub
19:09:00 <gribble> https://github.com/bitcoin/bitcoin/issues/17925 | Improve UpdateTransactionsFromBlock with Epochs by JeremyRubin · Pull Request #17925 · bitcoin/bitcoin · GitHub
19:09:15 <jeremyrubin> thanks -- will share more details when it's my topics turn
19:09:40 <jeremyrubin> #proposedmeetingtopic nanobench
19:09:50 <wumpus> FWIW kallewoof has the idea review kind of ground to a halt on signet (#16411) and is looking for more reviewers
19:09:54 <gribble> https://github.com/bitcoin/bitcoin/issues/16411 | BIP-325: Signet support by kallewoof · Pull Request #16411 · bitcoin/bitcoin · GitHub
19:11:15 <wumpus> #topic mempool project (jeremyrubin)
19:11:22 <jeremyrubin> Howdy!
19:11:43 <jeremyrubin> So I've been working on a bunch of improvements to the Mempool with a few other contributors
19:11:59 <jeremyrubin> We have a project allocated here https://github.com/bitcoin/bitcoin/projects/14
19:12:09 <jeremyrubin> to triage work on the MemPool
19:12:26 <jnewbery> hi
19:12:53 <jeremyrubin> The general goal is to get these changes reviewed and merged in a more orderly fashion
19:13:27 <jeremyrubin> And prevent things from suffering the tradeoffs of small PRs and big PRs by more clearly presenting what the projects are
19:13:31 <kanzure> hi.
19:13:51 <wumpus> thanks for the explanation,  I found "mempool improvements" is a bit vague for a project name as it doesn't really aim at a specific goal
19:13:57 <jeremyrubin> One of the first projects is to refactor almost every traversal algorithm in the mempool to use Epochs
19:14:48 <jeremyrubin> This should be an enormous performance improvement, but the goal is not to improve performance nescessarily, but rather to permit larger descendants limits
19:15:15 <wumpus> awesome!
19:15:29 <jeremyrubin> Increasing the descendants limits (or making some new policys) is going to be neccessary to make Lightning-y stuff work better (and CTV)
19:15:45 <jeremyrubin> Because currently there are issues with "pinning" caused by descendants limits
19:16:12 <wumpus> you also might want to write this up somewhere else than IRC so it doesn't get lost :)
19:16:26 <hebasto> what is an estimation of future descendants limits?
19:16:29 <wumpus> maybe the project description
19:16:32 <jeremyrubin> None at present
19:16:38 <jeremyrubin> wumpus: will do
19:16:46 <jeremyrubin> In conjunction with/after the epoch mempool improvements, it then becomes possible to make a lot of the mempool algorithms have no "short lived" allocations
19:17:19 <fjahr> jeremyrubin: cool that you are coordinating this but is there an endgame to this or is the plan to keep this open indefinitely? Just curious...
19:17:29 <jeremyrubin> We allocate a ton of memory in mempool traversal, these allocations can basically go to zero in a lot of places by having some preserved scratch space
19:18:12 <jeremyrubin> fjahr: It's sort of indefinite, but I would like to get all the changes through to the point that we solve these higher order goals, but then maybe we won't need more stuff in the mempool
19:18:21 <jeremyrubin> * major changes == stuff
19:18:44 <jeremyrubin> Also as a part of this work we've lumped in amiti's rebroadcasting and sdaftuar's packagerelay work
19:19:20 <jeremyrubin> Both of these should greatly help with making the mempool more rational, and better traversal algorithms help package relay be less DoS-able
19:19:36 <wumpus> well, closing projects is a whole different issue, we still have "libconsensus" and "P2P refactor" open despite not having had work in progress for quite some time
19:19:49 <jeremyrubin> There are also, looking forwards, some more changes inspecting the interface between mempool mining and validation
19:20:27 <jeremyrubin> There's a general notion of figuring out a "streaming createnewblock" algorithm that always immediately returns the best block but does constant gradient descent to find better ones
19:20:30 <bitcoin-git> [13bitcoin] 15jonatack closed pull request #17535: test: add block height test to listsinceblock.py  (06master...06rpc-wallet-blockheight-followups) 02https://github.com/bitcoin/bitcoin/pull/17535
19:21:03 <jeremyrubin> This in general can allow us to have more expensive to traverse mempool graphs (so we can not have restrictions that create pinning)
19:21:22 <jeremyrubin> But when mining we can still quickly return "good" blocks
19:21:58 <jeremyrubin> There are a few other projects being considered, such as Child*ren* pay for parent, Cousin-RBF (instead of Conflict-RBF), and some other optimizations
19:22:27 <jeremyrubin> As a bedrock to some of this, we need to have much better instrumentation and testing of the mempool
19:22:40 <jeremyrubin> There are a lot of edge cases currently not tested anywhere
19:22:45 <jeremyrubin> We should test these!
19:22:55 <jeremyrubin> We also don't have a good asymptotic framework for microbenches\
19:23:03 <jeremyrubin> We should do that (see nanobench)
19:23:20 <jeremyrubin> So to summarize a bit (and then maybe more questions):
19:23:33 <jeremyrubin> 1) There's a lot of exciting work being shaped out & plotted for the mempool
19:23:46 <jeremyrubin> 2) If you're excited about the design space/working on this, please let me know
19:24:11 <jeremyrubin> 3) Key to making this happen is clear communication, but also keeping review motivated for sometimes small PRs as a part of a bigger picture
19:25:06 <jeremyrubin> 4) keeping non-functional PRs (or other mempool work) limited in scope/in consultation with the project, to keep prioritization in focus & not cause a crapload of rebase hell as these projects will have potentially a lot of un-PR'd code
19:25:50 <jeremyrubin> If you disagree with 4 it's fine, ultimately up to the contributors & maintainers, but I'm trying to chart a course that's going to keep these projects moving forward
19:26:42 <jeremyrubin> We could maybe open a sub-channel for this stuff on IRC if people want Yet Another IRC Channel
19:26:48 <jeremyrubin> Any questions?
19:27:27 <jonasschnelli> as for testing, I think kallewoof has some recordings and a test framework (AFAIK)
19:27:31 <wumpus> it's not like a lot of discussion is happening in this channel lately, IMO it's fine (and preferable) to do so here
19:28:16 <wumpus> you can always decide to create another channel if it reall yends up monopolizing the channel (e.g. I guess that's why #bitcoin-builds is separate)
19:28:33 <wumpus> it's up to you of course
19:28:37 <fanquake> What do you mean by non-functional PRs in 4)? Is there a specific set of things that you’d like to have done for 0.20.0?
19:29:13 <jeremyrubin> Things which should have no user observable change
19:29:15 <MarcoFalke> I think for 0.20.0 we should focus on amiti's rebroadcast stuff
19:29:24 <jeremyrubin> E.g., moveonly
19:29:28 <jeremyrubin> renaming variables
19:29:49 <wumpus> MarcoFalke: agree, it wold be really nice to get the mempool privacy in
19:29:54 <amiti> :D
19:30:13 <jeremyrubin> I agree modulo concerns raised by some about needing more understanding of it.
19:30:18 <sipa> jeremyrubin: are there any PRs being merged that are just moveonly/variable rename stuff? that shouldn't be the case
19:30:33 <jeremyrubin> sipa: there are
19:30:47 <MarcoFalke> sipa: I removed a bunch of ::mempool
19:30:54 <MarcoFalke> I think jeremyrubin is referring to that
19:31:07 <jeremyrubin> there are other ones too that have a lot of acks and stuff not yet merged
19:31:24 <jeremyrubin> I guess it would be nice to have sort of the expectation that stuff isn't getting merged for now
19:31:29 <wumpus> please be more specific
19:31:53 <jeremyrubin> sure, didn't want to pick on anyone's PR but will give an example
19:32:04 <jeremyrubin> https://github.com/bitcoin/bitcoin/pull/17786
19:32:19 <wumpus> FWIW I try to mostly focus on "high priority for review" as for what to merge lately, there's just too many PRs
19:32:25 <jnewbery> I think removing ::mempool in order to clarify initialization order is a sensible project
19:32:40 <jonasschnelli> 17786 has a single ack
19:32:44 <wumpus> #17786
19:32:46 <gribble> https://github.com/bitcoin/bitcoin/issues/17786 | refactor: Nuke policy/fees->mempool circular dependencies by hebasto · Pull Request #17786 · bitcoin/bitcoin · GitHub
19:33:01 <wumpus> also: feel free to comment these kind of things *in* the PRs
19:33:23 <wumpus> if something has a lot of ACKs and no one is pseaking against it, it tends to get merged
19:33:26 <jeremyrubin> jnewbery: I agree, but the point is that if we're maintaining a few different complex projects on mempool stuff, and we're trying to slice it into many PRs to make review easier, it just becomes a headache
19:33:33 <MarcoFalke> Yeah, maybe we could postpone 17786 for until after the functional/user-facing mempool changes got in?
19:33:40 <sipa> yeah, if there are obvious things that interfere, it's best to discuss in the PR itself; maybe one of the authors has no problem rebasing on top of the other for example
19:33:47 <sipa> or discuss what can wait
19:33:58 <wumpus> yes, what sipa says
19:34:02 <sipa> but it's hard to discuss a blanket "please don't do things that interfere with my work"
19:34:47 <MarcoFalke> I think we can have a rough sketch in what order things should be merged. It can always be amended as needed
19:34:48 <wumpus> right
19:35:01 <jeremyrubin> Yeah it's fine, this is part of the goal of the mempool project
19:35:06 <sipa> ok
19:35:07 <provoostenator> It's useful to make Draft PR's so that Drahtbot can warn others about overlap.
19:35:08 <jeremyrubin> to triage the priority of these things
19:35:53 <wumpus> yes, makes sense
19:36:10 <wumpus> but please:  comment these things on github on the issue too, otherwise it'll likely be forgotten at some point
19:36:22 <jeremyrubin> fanquake: I'd like to get Amiti's stuff improved. I'd really like to get package relay shipped. And I think it's possible to do enough work on the Epoch Mempool to bump the descendants limit by 2x.
19:36:26 <MarcoFalke> agree with wumpus
19:37:34 <jeremyrubin> amiti and I are discussing how to chop up her work to get the PR complexity down, I think the worry is that the earlier PRs don't do anything "useful", except to make the later work easier (things like new testing harnesses)
19:37:36 <fanquake> jeremyrubin: ok. Re the non-PR’d/WIP changes, are they linked to from the project as well?
19:37:47 <jeremyrubin> fanquake: yes, in general
19:37:49 <jeremyrubin> If there's code
19:37:56 <jeremyrubin> Things that are still design stage not really
19:38:26 <fanquake> Given the amount of basic fuzzing harnesses we’re adding at the moment, I don’t think a mempool test harness would be rejected
19:39:06 <wumpus> more fuzzing harnesses is good
19:39:09 <jeremyrubin> I think it's possible it would be bikeshedded though, which I'd want to avoid
19:39:12 <MarcoFalke> left a comment here: https://github.com/bitcoin/bitcoin/pull/17786#issuecomment-580419545
19:39:29 <fanquake> wumpus yes
19:40:18 <MarcoFalke> jeremyrubin: if you refer to style feedback with "bikeshedding", keep in mind that a valid response to style feedback is a simple "no, I like the current style because I don't see the benefit of switching to something else"
19:40:25 <wumpus> which reminds me, we need to merge the examples in the -qa repository
19:40:33 <MarcoFalke> wumpus: Already did
19:40:34 <fanquake> I think that has been done
19:40:39 <wumpus> MarcoFalke: thanks
19:40:43 <jeremyrubin> MarcoFalke: in this case more of a "we could also Mock out XXXX in this harness too!"
19:40:59 <jeremyrubin> Whereas for the testing harness we need to just introduce YYYY
19:41:10 <MarcoFalke> In general I am not a fan of mocking all the stuff
19:41:26 <MarcoFalke> If it can be tested reasonably without mocking, it should be done without
19:42:04 <jeremyrubin> MarcoFalke: fair, which is sort of what I'm pointing to?
19:42:18 <MarcoFalke> I guees, yes
19:42:20 <jeremyrubin> I.e., do we need to mock this? What if we do x y z instead?
19:42:41 <jeremyrubin> But amiti has already done a lot of hard work in making a mock framework and writing tests against that framework
19:42:42 <sipa> this is a very abstract discussion
19:42:58 <jeremyrubin> oops I went concrete
19:43:00 <amiti> yeah... it seems like we're discussing my proposal to mock the scheduler, which marco and jeremy are both aware of... I think this convo makes more sense for when I open an actual PR
19:43:08 <jeremyrubin> sgtm
19:43:21 <wumpus> we still have two topocs to go and about 15 minutes
19:43:23 <sipa> and the question of mocking vs testing things otherwise seems very much a case by case discussion
19:43:29 <jeremyrubin> yeah
19:43:38 <MarcoFalke> Yes, it is case-by-case
19:43:43 <MarcoFalke> ok, other topics?
19:43:48 <jeremyrubin> we can move on unless there's a new question unanswered, I'll also be online after meeting
19:44:30 <wumpus> #topic propose physical meeting topics (kanzure)
19:44:34 <kanzure> just the topic collection topic.
19:44:50 <jeremyrubin> So not actual topics?
19:44:55 <kanzure> right, so, i'd like suggestions so i can write a document giving an overview of what people would like to hear
19:45:00 <kanzure> for the physical IRL meeting
19:45:11 <kanzure> these are just suggestions and not actually a schedule or anything draconian like that.
19:45:19 <jeremyrubin> Where should we send them? Is there a google form or just email you?
19:45:29 <provoostenator> I think jnewbery has been collecting topics too.
19:45:31 <kanzure> just send em to me.
19:45:43 <jeremyrubin> kanzure: I wouldn't *mind* a schedule ;)
19:45:46 <kanzure> right or him
19:45:52 <fanquake> kanzure will this be public somewhere before the meetup?
19:46:04 <kanzure> well in the past they have been semi-public (link circulated) but not actually public
19:46:12 <kanzure> i think that's up to the group really. do you prefer public or private topic suggestions?
19:46:17 <fanquake> Sure, semi-public
19:46:30 <jeremyrubin> public; I can't remember which ones I already submitted
19:46:54 <jnewbery> yeah, I've also sent out a survey to some people. Thanks to everyone who responded (which is most!)
19:47:02 <kanzure> anyway, for topics, it's not just what you have been working on, but also things that you think the group would benefit from hearing from someone else
19:47:24 <provoostenator> jeremyrubin: downside of doing open source work where everything is logged is that you tend to develop a write-only memory :-)
19:48:15 <wumpus> hehe
19:48:24 * jeremyrubin searches for my memex
19:49:20 <kanzure> that's all i have.
19:49:52 <wumpus> #topic nanobench (jeremyrubin)
19:50:04 <jeremyrubin> So I'm not the maintainer of nanobench
19:50:07 <jeremyrubin> but I really like it
19:50:18 <jeremyrubin> I think if we can do a cursory check it's not actually malware
19:50:29 <jeremyrubin> we should just merge it
19:50:34 <wumpus> general question: why do people want to switch to another benchmarking framework?
19:50:39 <wumpus> what's wrong with the current one?
19:50:51 <jeremyrubin> a few things:
19:50:51 <wumpus> a while ago there was a PR to switch it to boost::test, now yet another dependency
19:50:57 <jeremyrubin> 1) It's slow
19:51:03 <wumpus> is there anything they do that we cannot do?
19:51:08 <MarcoFalke> Yeah, I think we need to take a closer look to see where they differ and what they improve. To make sure there are no regressions
19:51:09 <jeremyrubin> 2) there's no support for testing asymptotics
19:51:10 <wumpus> what makes it so slow? it's very simple
19:51:22 <MarcoFalke> For example, a lot of tools rely on the output format of the current bench framework
19:51:27 <wumpus> it should hardly have any overhead
19:51:33 <jeremyrubin> I think it runs too many trials
19:51:42 <wumpus> then reduce that?
19:51:43 <jonasschnelli> which could be changed,... right?
19:51:44 <jeremyrubin> I beleive nanobench autodetects variance or something
19:52:30 <jeremyrubin> martinus also claims it's more accurate -- less variance than with old benching framework
19:52:49 <wumpus> so does it use a different clock?
19:52:51 <jeremyrubin> It also measures more things
19:52:53 <wumpus> how can accuracy differ?
19:53:26 <wumpus> or does it do CPU/OS-specific cache flushing?
19:53:28 <sipa> i think the current code we have is kinda crap; it started off being kinda general and automatically measuring things, and when it was shown that it introduces inaccuracies, it was changed to needing iterations counts in the code itself
19:54:11 <sipa> wumpus: i think it's mostly due to some tests running far longer than necessary, resulting in getting OS interrupts etc inside of them
19:54:12 <jeremyrubin> w.r.t. output and tooling, nanobench also outputs new information (e.g., instructions, cycles, branches, ips, branch misses)
19:54:31 <jeremyrubin> so we'd fundamentally need some new tools.
19:54:31 <jnewbery> is there anything that needs discussing here that isn't covered in the PR? I think the only action is to review that if you're interested, no? (#18011)
19:54:33 <gribble> https://github.com/bitcoin/bitcoin/issues/18011 | Replace current benchmarking framework with nanobench by martinus · Pull Request #18011 · bitcoin/bitcoin · GitHub
19:54:36 <wumpus> we used to measure cycles, this was removed at some point
19:54:53 <sipa> yeah let's discuss in the PR
19:55:30 <jeremyrubin> Sounds good -- my point in making it a topic was that it's relatively low risk to adopt as nothing really relies heavily on the benching
19:55:36 <wumpus> sounds good to me, I was just curious why everyone wants to replace the benchmark framework (with different things)
19:55:39 <jeremyrubin> and being able to write asymptotic benches is going to be a big help
19:55:44 <wumpus> but if the current one is crap that's clear :)
19:55:46 <jeremyrubin> Because we need that for the mempool work
19:56:04 <jeremyrubin> And I tried to introduce it in a since-closed PR, but it seemed we needed a more thought out approach
19:56:09 <jeremyrubin> and nanobench has that
19:56:18 <jeremyrubin> fin
19:57:07 <wumpus> #endmeeting