19:02:49 <wumpus> #startmeeting
19:02:49 <lightningbot> Meeting started Thu Oct  4 19:02:49 2018 UTC.  The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
19:02:49 <lightningbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
19:02:55 <wumpus> promag: sure
19:02:58 <promag> hi
19:03:00 <promag> thanks!
19:04:12 <phantomcircuit> hi
19:05:05 <jcorgan> i bet many are sleeping or about to
19:05:07 <wumpus> #bitcoin-core-dev Meeting: wumpus sipa gmaxwell jonasschnelli morcos luke-jr btcdrak sdaftuar jtimon cfields petertodd kanzure bluematt instagibbs phantomcircuit codeshark michagogo marcofalke paveljanik NicolasDorier jl2012 achow101 meshcollider jnewbery maaku fanquake promag provoostenator
19:05:18 <wumpus> yes—I expect so too
19:05:25 <jonasschnelli> hi
19:05:43 <wumpus> don't know if we really have things to discuss without the people in Tokyo
19:05:56 * jonasschnelli is sad to miss SB tokyo
19:06:01 * wumpus too
19:06:17 <jcorgan> #MeToo
19:06:22 <jcorgan> sri
19:06:25 <jonasschnelli> heh
19:06:38 <wumpus> lol
19:07:26 <gmaxwell> hi
19:07:27 <wumpus> anything for high-priority for review? or what can be merged? I haven't been able to focus much this week
19:07:44 <phantomcircuit> #14335 and maybe #14336
19:07:46 <gribble> https://github.com/bitcoin/bitcoin/issues/14335 | net: refactor: cleanup ThreadSocketHandler by pstratem · Pull Request #14335 · bitcoin/bitcoin · GitHub
19:07:47 <gribble> https://github.com/bitcoin/bitcoin/issues/14336 | net: implement poll by pstratem · Pull Request #14336 · bitcoin/bitcoin · GitHub
19:08:19 <phantomcircuit> i still cant exactly ping down why some of the commits in 14336 fail tests but the final one doesn't
19:08:28 <phantomcircuit> and they seem like races more than actual failures
19:08:41 <wumpus> do they fail locally for you too?
19:08:43 <wumpus> or only on travis?
19:09:01 <phantomcircuit> wumpus, only on travis
19:09:12 <wumpus> aww
19:09:25 <wumpus> I can try running them in some weird environments
19:10:29 <phantomcircuit> 14335 however passes everywhere
19:10:34 <gmaxwell> you might try inserting sleeps to see if you can trigger something locally.
19:10:38 <wumpus> so if they are races they only happen part of the time, not always
19:10:45 <phantomcircuit> it's just refactoring no logic changes at all now
19:11:02 <wumpus> well if it causes test failures there might be logic changes you don't know about
19:11:07 <phantomcircuit> gmaxwell, i can randomly trigger the rpc_zmq.py test to fail
19:11:14 <wumpus> I don't think a pure refactoring can result in more races
19:11:17 <phantomcircuit> but it's zmq so i would expect it to be very racey
19:11:27 <phantomcircuit> wumpus, yeah that pr doesn't have any issues
19:11:34 <wumpus> no, zmq is not inherently racy
19:11:35 <phantomcircuit> it's just the one implementing poll()
19:11:43 <phantomcircuit> (this is why it's two separate pulls)
19:12:10 <phantomcircuit> wumpus, the test is though, it starts the node and then instantly makes an rpc call for zmq notifications
19:12:14 <wumpus> (maybe our use of it is, I don't now)
19:12:25 <wumpus> ohh
19:12:27 <jcorgan> the test is written in a racy way as described
19:12:27 <phantomcircuit> cant probably make that more robust but im not super familiar with the testing framework
19:13:39 <wumpus> someone needs to take a look at it; we can't merge something that makes any test randomly fail, as it wil keep coming back in later PRs
19:13:51 <wumpus> #action reviews #14336
19:13:53 <gribble> https://github.com/bitcoin/bitcoin/issues/14336 | net: implement poll by pstratem · Pull Request #14336 · bitcoin/bitcoin · GitHub
19:14:34 <gmaxwell> We should just fix that test, but is the zmq test the only one that fails?
19:14:46 <gmaxwell> There is something other tests do to wait until the node is ready.
19:15:00 <phantomcircuit> gmaxwell, feature notifications fails on windows
19:15:01 <wumpus> yes, fixing the test is acceptable too if it is broken
19:15:17 <phantomcircuit> but im having trouble replicating that for a whole bunch of reasons
19:16:15 <phantomcircuit> gmaxwell, they wait on nodes to be synchronized i think, but that doesn't apply to the zmq test
19:16:31 <phantomcircuit> (again im not exactly sure though)
19:16:58 <jcorgan> which file is the zmq test
19:17:03 <wumpus> if there's multiple nodes connected to each other they need to wait to be synchronized
19:17:21 <phantomcircuit> jcorgan, test/functional/rpc_zmq.py
19:17:26 <wumpus> test/functional/rpc_zmq.py
19:17:28 <wumpus> yes
19:17:34 <phantomcircuit> jynx
19:19:02 <wumpus> okay, any other topics?
19:19:08 <jcorgan> it's the assert on L31 that non-deterministically fails?
19:19:37 <gmaxwell> I could give a little update on our work on set recon relay which has been ongoing.
19:19:56 <wumpus> lol that test doesn't even *test* zmq
19:20:03 <wumpus> I don't understand how it can fail
19:20:38 <wumpus> it jst tests some administrative rpc functions related to zmq
19:21:02 <wumpus> gmaxwell: might be better to do that when there's more people
19:21:21 <gmaxwell> OK
19:21:24 <wumpus> but, is up to you
19:21:48 <jcorgan> maybe it's interface_zmq.py
19:22:19 <wumpus> jcorgan: that's the one that really tests the zmq interface, yes
19:22:45 <gmaxwell> wumpus: well, I'd like to-- the people who aren't here will probably hear about it at the events there at. :)
19:22:48 <jcorgan> right, but is that the one randomly failing?
19:23:19 <wumpus> phantomcircuit: I don't understand it, you make P2P changes, and a test completely unrelated to P2P starts failing
19:23:20 <gmaxwell> wumpus: I guess it can fail if the node isn't up by the time it attempts the rpc?
19:23:52 <wumpus> unless your poll somehow interferes with libevent, but I wouldn't understand why
19:23:55 <gmaxwell> like, e.g. poll having a sleep timeout during startup where the select didn't could slow down bringup slightly.
19:24:15 <wumpus> the test framework is smart enough to wait for the RPC interface to actually work, AFAIK
19:24:31 <wumpus> (there's this 'warmup' period that it needs to ignore, for example)
19:24:32 <gmaxwell> if so, then I've got no suggstions. :)
19:24:56 <wumpus> well it mightb be that this test does something ... special
19:25:15 <wumpus> #topic recon relay (gmaxwell)
19:25:42 <gmaxwell> Sipa, gleb, and I have been continuing to work on set recon relay.  Gleb has made a lot of progress with simulations.
19:26:28 <gmaxwell> In his simulated network topology he shows that the best possible (no 'overhead') relay using 8 byte tx identifiers would use about 44x less bandwidth for relay than what the code currently does.
19:27:40 <gmaxwell> (optimal would basically be exach node gets exactly 1 inv per tx and no more) Simulations of our recon work don't quite achieve optimality (since we also want relay to be reasonably fast) but end up only using 2-3x the bandwidth of the theoretically optimal usage, which sounds pretty good compared to 44x. :)
19:28:25 <gmaxwell> Sipa and I (mostly sipa) have been working on optimizing the recon code itself, which is in part important because its performance helps tell us what parameters make sense to propose.
19:28:36 <gmaxwell> E.g. some benchmark results from last night: https://people.xiph.org/~greg/temp/srr2.png
19:28:42 <wumpus> nice !
19:29:33 <gmaxwell> This shows how long it takes to do the computation for reconciling 150 differences (which is a good high watermark from glebs simulations), as a function of how long the short-txids we're using (in bits).
19:30:22 <gmaxwell> The graph shows that some sizes are much faster in the implementation currently, for optimization (and the number theory that enables those optimizations) reasons.
19:30:26 <wumpus> 44x less bandwidth is more than impressive, I didn't know that the invs were such a large part of the traffic
19:31:25 <gmaxwell> Ah! that figure is perfectly efficient invs (one inv per host) vs invs in what we do today.  Invs are a majority of traffic on nodes right now, but those figure ignore all the non-inv traffic.
19:31:25 <wumpus> but that's similar to my surprise how much (at least of outgoing traffic) is 'reject' messages
19:31:50 <gmaxwell> So in terms of actual total traffic impact, maybe halve those numbers for your expectations.
19:32:09 <sipa> ohai
19:32:38 <gmaxwell> Basically gleb's simulator simulates a plausable bitcoin network topology, and then measures how transactions relay around in it with different relaying schemes, so we can try different ideas and measure their impacts.
19:32:55 <gmaxwell> So in any case, lots of progress going on there.
19:32:59 <wumpus> is this simulator available anywhere?
19:33:36 <gmaxwell> Not yet. Sipa and I need to convince gleb that his code is not too awful, I think.
19:34:10 <wumpus> no hurry, though it would be nice to have, if it was available we'd want to link it
19:34:16 <gmaxwell> Absolutely.
19:34:53 <gmaxwell> In any case, right now we're still in a fairly researchy mode, looping over detailed ideas and feeding the results back in to tell us what to try next.
19:35:48 <gmaxwell> Thats all I've got for now, unless sleepwalking sipa has more.
19:36:16 <sipa> not really
19:36:52 <wumpus> you could say so, it's .. 04:36 there?
19:37:26 <wumpus> thanks for the update!
19:38:44 <wumpus> I guess it's time to close the meeting
19:39:04 <wumpus> #endmeeting