20:08:05 #startmeeting 20:08:05 Meeting started Mon Apr 27 20:08:05 2020 UTC. The chair is niftynei. Information about MeetBot at http://wiki.debian.org/MeetBot. 20:08:05 Useful Commands: #action #agreed #help #info #idea #link #topic. 20:08:13 hey yall 20:08:21 * t-bast waves at BlueMatt 20:08:36 hi everyone welcome to the lightning-dev 27 Apr 2020 spec meeting 20:08:49 cdecker: Error: Can't start another meeting, one is in progress. 20:08:57 Oh, missed that one 20:09:06 #link https://github.com/lightningnetwork/lightning-rfc/issues/768 20:09:10 cdecker tried to hijack the chair!!! 20:09:28 wow this is starting off exciting. 20:09:40 Man, we need a visit to IKEA to get more chairs... end the lockdown 20:09:53 first item on the agenda is stuck channels 20:10:02 #topic stuck channels #740 20:10:16 #link https://github.com/lightningnetwork/lightning-rfc/pull/740 20:10:35 After many iterations on the wording, I think we're reaching something somewhat clear on stuck channels 20:10:45 FWIW, this is implemented in our coming release, too. 20:11:05 There's even one ACK from Joost (thanks!) so maybe another impl to chime in and we should be good to go? 20:11:11 it looks like the PR has two requisite acks 20:11:29 c-lightning has an implemenation, has anyone else implemented this yet? 20:11:40 yep it's in eclair too 20:11:52 done slightly differently though 20:12:39 (Whoever commits this please squash it in rather than merge or rebase, since it's 9 commits for one change!) 20:13:19 yep definitely, squash it's going to be 20:13:36 I can commit this if everyone feels it's ready 20:13:37 ok. so it seems we're ready to mergesquash this one? 20:13:47 ack! 20:13:55 ack! 20:14:16 roasbeef do you know if lnd has implemented that? 20:14:37 I know johan and joost were mostly looking at this, so since joost ack-ed it's probably underway or done? 20:15:04 I've also seen that val has linked this for RL 20:15:55 no implementation yet 20:16:04 (i believe val is valwal_ on irc) 20:16:07 yea, we have a pr with that and a few other things that valwal_'s doing 20:16:24 the change looked good to me last I looked, so happy if it gets merged 20:16:40 great 20:16:43 #action t-bast to squash/merge PR #740 for stuck channels 20:16:56 ok, moving on 20:17:08 we have some other heuristics we use, they may be included in this tho, end of the day it's all whack-a-mole really, but we'll prob add this eventually 20:17:32 #topic PR #767 prune channels with old channel_updates 20:17:34 #link https://github.com/lightningnetwork/lightning-rfc/pull/767 20:17:41 roasbeef: :+1: 20:18:56 looks like there's still discussion happening on the PR 20:19:14 I think this one has its logic backward: a channel that is inactive should not receive updates from either endpoints 20:19:17 Who has tried applying bitconner's suggestion to their own node? 20:20:04 Hmm, it'd be good to mark which channels fall under this then probe to see if they're *really* dead... 20:20:20 cdecker: I think he means that one side is dead, but the other keeps updating, but it actually isn't useable at all 20:20:34 At first glance it looked useful to me, but when I applied it the results were bad; it seems to highlight that there are gossip issues in the network, some nodes behaving strangely with their channel_updates 20:20:46 t-bast: resultings being? 20:20:53 Yes, so why would the other side still be sending updates? Keepalive is not necessary since we'll just reannounce 20:21:11 But I do like the idea that you shouldn't update if the other side doesn't. Though how to define this exactly: you would send out one at (say) 13 days, but then you'd suppress the next one I guess. 20:21:14 cdecker: the other side does as it's still active, and will prob just follow the guidfelines to make sure it sends on before the 2 week period 20:21:29 cdecker: yeah, we would reannounce, even if just to say it's disabled. 20:21:44 roasbeef: see my comment on the PR: surprisingly some updates either weren't propagated properly all the way to our node, or weren't emitted in time by the corresponding node (for a channel that's clearly alive) 20:21:48 lnd will just disable that end tho, once the other party is offline for a period 20:22:08 t-bast: interesting, i know CL has some suppression limits on nodes, maybe you're running into that? 20:22:20 roasbeef: yeah, we would too: if it's offline at the time we need to refresh, it'll get a disabled update. 20:22:56 roasbeef: can you clarify the "y'alls continues to send fresh channel_updates" bit ? 20:23:02 My point is that no node should be sending updates for channels that are not routable, rather they should let them be pruned, and then be reannounced if they become usable again, rather than having a constant background radiation of nodes that aren't usable but want to be kept alive 20:23:23 sstone: didn't write it, I think like updating fees n stuff? 20:23:50 cdecker: agreed, but it seems that all three implementations are supposed to do that (at least we all think our impl does that) 20:23:57 Yes, I wonder if c-lightning is suppressing spam too aggressively and causing propagation issues? Though we're too small a mnority to do that in general I think. However, my impression was that we're not seeing as much gossip spam in Modern Times (though let me check...) 20:24:50 rusty: that may explain what I've seen, I failed to receive channel_updates in time from a node that seems to be running lnd (BlueWallet), but I cannot know if that node never emitted the update or if it wasn't propagated all the way to me 20:25:13 Easy, connect directly 20:25:13 And I'm unsure what impl the other end of the channel is running 20:25:40 t-bast: from stuff gathered on our end, the bluewallet node seems to be funky, I think they're running some custom stuff 20:25:52 roasbeef: ahah that may be another explanation 20:25:56 Interesting 20:26:17 I think maybe this issue was partially inspired by some weirdness conner saw when examining their node 20:26:36 I guess let's just wait faor him to comment on the issue, may be able to provide some more insight 20:27:01 allright, sgtm 20:27:28 ack ^^ 20:28:07 2782437 suppressions in ~2 weeks, but you'd have to spam (4/day get ignored) then stop spamming at all for 13 days to get pruned. 20:29:01 rusty: you mean that there are 1782437 channel_update that you received but didn't relay? 20:30:26 t-bast: yeah, but sorting by channel/nodeid now. Suspect it's a handful. 20:30:52 that's a lot, that means ~5 per node per day (assuming ~40k nodes in the network) 20:31:10 ah no it's ~2 since there are 2 ends of a channel (doh) 20:32:10 Anyway, let's keep moving while I wait for my machine to churn the results... 20:32:21 ack 20:33:07 ok so it sounds like we should continue discussion on the PR 20:33:20 sgtm 20:33:30 #action rusty to figure out c-lightning pruning reality 20:33:42 #action discussion for PR 768 to continue on github 20:33:49 ok let's see what's next 20:34:44 #topic issue 761, which is a discussion branched off of 740 20:34:44 #link https://github.com/lightningnetwork/lightning-rfc/issues/761 20:34:52 I added that to the agenda because it's digging an old topic (asynchronous protocol and its pitfalls) but I couldn't find previous discussions around that...can one of the protocol OGs just quickly skim through that issue and provide some links to previous discussions? 20:35:26 I don't know if it's worth spending time during the meeting, but if someone can take an action item to do a quick pass on this it would be great 20:35:46 can check it out, but it is the case that there're some fundamental concurrency isuses that can happen w/ the state machine as is, without the existence of some unadd type thing 20:36:40 roasbeef: great, I know that had been discussed in the past, but searching for it in the ML archives I couldn't find anything... 20:36:44 Yeah, it's the nature of the beast. You can only prevent it by slowing the entire thing down, basically. Though it's less of a problem in practice. 20:38:44 this is a cool discussion to highlight t-bast 20:38:59 seems like we should keep moving along 20:39:13 ack! 20:39:16 if i've understood correctly, roasbeef is going to give it a look-see 20:39:44 #action roasbeef to check thread out 20:40:02 #info "it's the nature of the beast" - rusty 20:40:14 moving on 20:40:29 I think we could print a t-shirt for that "it's the nature of the beast" 20:40:32 Loving the quote, that should become our official motto 20:40:40 xD 20:40:47 we've now reached the crowd favorite "long term updates" segment 20:40:48 kek 20:41:30 blinded paths maybe? 20:41:37 ariard, and THEBLUEMATT would you be ok to start with tx pinning attack? 20:41:42 So say we all "natura bestiorum" xD 20:41:42 or that 20:41:44 Yeah, blinded paths are cooler! 20:41:51 ;) 20:42:17 the r's have spoken, we'll start blind and then roll into pinning then 20:42:18 cdecker: sounds kinda occult lol 20:42:34 would prob make for a bad ass shirt/hoodie 20:42:35 #topic long term updates: blinded paths cc t-bast 20:42:36 yo 20:42:44 *BlueMatt sry 20:42:45 niftynei: sure, though why am I in CAPS 20:42:49 roasbeef: it's likely also monstrously wrong, my latin is a bit rusty :-) 20:42:59 my shift key finger got lazy :{ 20:43:32 #link https://github.com/lightningnetwork/lightning-rfc/pull/765 20:43:36 t-bast do you want to give us an update? 20:43:57 So the gist of route blinding is that it's doing a Sphinx round from the recipient to an introduction point 20:44:19 And using a key derived from the shared_secrets to blind the scids 20:45:04 It requires nodes *after* the introduction point to receive a curve point *outside* of the onion, to be able to derive a tweak to their node_id that lets them decrypt the onion 20:45:24 First of all it needs more eyes on the crypto itself :) 20:45:34 Then there are two things that need to be challenged 20:45:58 The fact that it adds data to the tlv extension of update_add_htlc, making them distinguishable at the link layer (different packet size) 20:46:31 And the fact that an attacker is able to uncover one of the channels by probing the fees/cltv -> maybe they could do worse? 20:47:14 t-bast: it aklso needs a bolt11 extension, though I've been thinking about that. 20:47:45 The added flexibility compared to rendezvous is what gives attackers some probing potential - we need to evaluate how much and how we can mitigate those without building something too complex 20:48:13 rusty: true, it needs changes to Bolt11 as well (but those could be for the best as nobody like the current routing_hints xD) 20:48:21 I guess we couldn't normalize the TLV addition along the entire path, i.e., sending the adjunct point always, even if not needed? 20:48:33 cdecker: yes we definitely could 20:48:52 cdecker: that feels hackish, but it could be the way to go 20:49:17 But we can't blind it normally, since then we'd have the usual 'meet-in-the-middle' issue we had with RV 20:49:19 cdecker: you can send an empty ping, too. (Though technically there are rules against sending pings when you're still waiting for a response) 20:49:27 So it'd be a decoy that is sent along 20:50:11 yes, a dummy decoy would be enough since it's only to thwart link layer attacker who would see encrypted bytes, it's just to make the lengths match 20:50:24 i have a dumb question. how does the route composer ensure that the capacity for that route is ok? 20:50:26 I was just thinking we could make it seamless by always going through the motions, to have constant-time and constant-size 20:50:39 or is the path not meant to be sized/usable? 20:51:28 niftynei: you run the path-finding like you'd do for a normal payment; ideally you'd select a path with the biggest capacity and additional channels to leverage non-strict forwarding 20:51:44 niftynei: but it's true that you can't guarantee that it will always succeed 20:52:06 so every payment failure will need a round-trip convo with the path initiator? 20:52:33 "hey this one didn't work, what else you got" 20:52:54 Yep, that's what I think it means 20:52:57 niftynei: that, or you'd provide multiple paths in the invoice, but both of these potentially also give more data to an attacker to unblind :/ 20:53:02 t-bast: this is an existing probing vector re fees/cltv tho? 20:53:06 or do you send directly a set of blinded-path? 20:53:27 roasbeef: exactly, that's what I'm worried about 20:53:29 But it adds more flexibility to what my RV construction can offer (payment_hash and amounts are not committed to, so you could retry with different fees at least) 20:53:56 ack ok, this fits my understanding of the proposal then :) 20:54:19 Note: there is a way to combat ctlv/fee probes, and that is to put advice inside the enctlv telling the node what minima to use (presumably, >= its existing minima). Then you can force the path to have uniform values. 20:54:20 I think it's showing that when you give some flexibility around fees and cltv, you make this more usable, but you give away a bit of privacy by adding probing, so we need to find the sweet spot 20:54:38 yeh as cdecker said, we can start to pad out update_add_htlc at all times, would need to look into the specifics of why it can't be attached in the onion 20:55:15 how's that failure condition communicated? we assume a new sync connetion between the sender/receiver? 20:55:20 roasbeef: you need it to know the key to decrypt the onion, that's why (at least in the current proposal), because the blinding works by tweaking node_ids 20:55:42 That being said, I think rusty is planning to use these for the offers, i.e., route communication to the real endpoint, not caring about fees, cltvs and amounts, and then we can have iterative approaches to fetch alternative paths in the background 20:56:09 roasbeef: in my mind it should be a single-shot, if we provide interaction between payer/recipient to get other paths I'm afraid the probing would be quite efficient 20:56:48 t-bast: but how do you dissociate a real failure from a simulated-one for probing? 20:56:50 t-bast: doesn't sound realistic in practice tho? like even stuff like a channel update being out of date that you need to fetch or something (I still need to read it fully for broader context too) 20:57:26 rusty: I like the proposal to raise the lower bound for all channels in the path, everyone would benefit from this and it could help with probing - I need to think about this more 20:57:32 roasbeef: yeah, they're no more robust than route hints today. 20:57:37 Well, for a communication substrate this works quite nicely, doesn't it? 20:58:09 roasbeef: I don't know, if you do what rusty proposes (raise fee and cltv compared to current channel_update) it may work quite nicely in practice 20:58:26 And for a communication substrate it's fine indeed 20:58:44 (Ofc you need to do this for every node on the path, even if it's redundant, since otherwie you're flagging which ones need to be told!) 20:58:53 ariard: what do you mean? who distinguishes that failure? 20:59:39 #action t-bast to investigate making fee/cltv uniform in the blinded path, describe that in the PR 20:59:59 FYI we are about at 1hr of meeting time for today 21:00:14 t-bast: like you receive a blinded route in invoice, you do a first probing? you ask for a 2nd invoice with another blinded route...? 21:00:15 already?? :'( 21:01:22 True, I wanted to mention that the path encoded in the blinded path needs to be contiguous and each hop needs to support the extra transport, so this becomes useful only rather late when most nodes have updated 21:01:25 (rusty sneaking in that fee increase he promised at LN-conf in berlin last year) 21:01:29 ariard: the way I see it you shouldn't give two blinded routes to the same payer. But maybe in practice it's simply impossible to avoid, it's a good point I need to think about that a bit more E2E 21:02:13 cdecker: yes it would take time to be adopted, it needs enough nodes to support it 21:02:25 t-bast: I also need to implement dummy path extension, so we can put advice in the spec. 21:02:29 cdecker: once it's implemented, the road to E2E adoption will be quite long 21:03:07 rusty: yes I think that part is quite useful too! in combination with uniform fees/cltv it may also fix ariard's comment 21:03:22 t-bast: but you need sender authentication and we want both-side anonymity? 21:04:19 should we spend 10min on pinning? 21:04:26 ariard: yeah this is why I think preventing this is hard and potentially not doable, but the uniform fees combined with dummy path extension may allow you to give many blinded paths without risk. I need to spend more time on this 21:04:52 niftynei: ack, I think it would be good to discuss that a bit while we're all (?) here 21:04:57 t-bast: yeah let's continue on the PR, replying on your comments 21:05:08 #action discussion to continue on PR 21:05:13 thanks everyone for the feedback, it already gave me a lot to work on! Don't hesitate to put comments on the PR 21:05:38 Need to drop off in 10, but let's quickly fix RBF pinning ;-) 21:05:40 #topic long term updates: mempool tx pinning attack cc ariad & BlueMatt 21:06:13 ariad or BlueMatt do you have a link you could pop into the chat for this? 21:06:37 yes optech has a resume 21:06:39 well there's that recent ML thread 21:06:43 iiuc this was mostly a ML thread? 21:06:55 https://github.com/bitcoinops/bitcoinops.github.io/pull/394/files 21:06:59 mostly the ml thread i think is a good resource 21:07:09 optech has a good summary as well that describes also the responses on the ml 21:07:13 https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-April/017757.html ... 21:07:14 not just the high-level issue 21:07:32 something I pointed out in the thread is that the proposed HTLC changes would require a pretty fundemantel state machine change, which is still an open design question, what's in the current draft PR gets us pretty far along imo and ppl can layer the mempool stuff on top of that till we figure out the state machine changes 21:07:44 #link https://lists.linuxfoundation.org/pipermail/lightning-dev/2020-April/002639.html 21:08:04 favoring incremental updates to fix existing issues and make progress 21:08:07 #link https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2020-April/017757.html 21:08:08 roasbeef: I disagree that its "pretty fundamental" 21:08:29 BlueMatt: it would need a new set before commit_sig, seems big, no? idk how that would even look liek atm 21:08:35 but it does require an additional step between update_add and commitment_signed 21:08:49 new step* 21:09:00 ready_to_sign_commitment -> ok_go_ahead -> commitment_signed 21:09:00 #link ttps://github.com/bitcoinops/bitcoinops.github.io/pull/394/files 21:09:01 :) 21:09:10 #link https://github.com/bitcoinops/bitcoinops.github.io/pull/394/files 21:09:11 like provide_remote_sig_on_local 21:09:14 ok now ensure that works on teh concurrent async setting 21:09:29 just saying it isn't fully figured out yet 21:09:29 no problem, ok_go_ahead ignores any things going the other way 21:09:37 and is "a part of the commitment_signed" 21:09:42 i dont think thats complicated 21:09:51 the devil is always in the details ;) 21:09:59 monitoring mempool isn't a really fix it's easy to pour a local conflict in your mempool while announcing something else to the rest of netwrok 21:10:08 but just pointing out how it sprwals, and ppl can deploy a solution that fixes a lot of issues today as we have 21:10:10 but, anyway, I think if we want to move forward, then we can do anchor without any htlc transaction changes, that seems reasonable and leaves this issue as-is 21:10:15 and identifying mempools of big ln nodes shouldn't be that hard due to tx-relay leaks 21:10:26 ariard: yeh and the other direction is mempool fixes, as jeremey rubin mentioned in the thread 21:10:47 roasbeef: on the long term we agree, but it may takes months/years :( 21:10:49 note that rubin mentioned specifically that he does *not* have any mempool changes to fix this queued up 21:10:55 BlueMatt: yeah i'm on board w/ that, then we can continue to grind on this issue in the background 21:11:01 only that he's slowly working on slowly making discussing such changes an option :/ 21:11:01 can someone summarize the main discussion point rn? 21:11:32 niftynei: essentially you dont learn preimages from txn in the mempool, this may result in you not getting the preimage. 21:11:41 niftyney: what fixes can we come up which is reasonable to implement in a short timeline? 21:11:45 niftynei: move forwaard w/ anchors as is rn that has this issue with htlcs, or try to finish up this other working thing that requires some bigger changes to the statemachien which aren't fully figured out yet 21:11:49 there isnt really any fix that can be done to the mempool policy that fixes it any time soon, and maybe ever. 21:11:58 ty! 21:12:04 down with the mempool! 21:12:15 Maybe this is a dumb question: but why doesn't the RBF rule ask for a pro-rata increase in feerate, rather than a linear diff. Pro-rata (say each replacement needs 10% higher feerate) we'd have a far smaller issue with spamming, and we could lose the absolute fee diff requirement 21:12:16 package relay should make us safe 21:12:27 we can probably do something with anchor that, like, has different htlc formats based on the relative values of the htlc to the feerate 21:12:29 * rusty starts reading thread, confused about why we're talking about changing the wire protocol... 21:12:31 but we'll need to figure that out 21:12:47 cdecker: I think it does, but one issue is the absolute fee increase requirement, if it was just feerate this particular pinning issue wouldn't exist 21:13:23 lockingdown htlcs and requiring CPFP for them will make them costlier which means more dust htlcs 21:13:25 That's my point: an exponential increase in feerate could replace the absolute fee increase requirement, couldn't it? 21:13:55 so to summarize: there's a mempool related info propagation problem that impacts htlc txs? 21:14:03 cdecker: it asks for a pro-rate, bip125 rule4? 21:14:07 * BlueMatt notes that the whole rbf fee-based policy thing is *not* the only issue with rbf-pinning, so its somewhat of missing the point to focus on it. 21:14:07 niftynei: yeh 21:14:19 kk 21:15:04 BlueMatt: that's true, but we've been burned by it a couple of times, so I was just wondering 21:15:25 cdecker: yeah but for performance reasons it's not implemented right now, you just sum up absolute fees of conflicted tx + descendants 21:16:03 niftynei: the anchors PR on rfc solves some but not all of the issues, but a big thing is that yo can actully modify the commitmetn fee finally post broadcast, and also bump htlc fees. this issue can degrade into a bidding war for htlc inclusion basically 21:16:05 cdecker russell o'connor had a bitcoin ML post re: adjusting the feerate floor for RBFs a while ago 21:16:50 I mean if every replacement just required a 10% increase in feerate (and dropped the absolute fee increase requirement) wouldn't we end up in confirmation territory very quickly, and a node could actually decide locally whether the bump will be accepted or not 21:16:57 there's also the 'likely to be rbf'd' flag suggestion if we want to get into mempool policy changes 21:17:07 Oh, I'll go and read that, thanks niftynei ^^ 21:17:42 roasbeef: sadly there is clever way to pin by exploiting ancestor size limit to obstrucate replacement 21:18:13 cdecker https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2018-February/015717.html 21:18:14 BlueMatt: while not directly actionable I do get the feeling that a mempool RBF logic simplification is the only durable fix for these issues, but I'm happy to be corrected on this one :-) 21:18:43 anyway, so it sounds like the next steps are: a) amend anchors to make no changes to htlc signatures, and resolve the other issues (needs tests cases, debate about 1-vs-2 etc) on the ml or issue, then b) work on redoing things to use anchors (maybe based on feerates) in htlc txn 21:18:48 ariard: yeh there's prob a ton of other pinning vectors, but at least we can make a step forwrd to actually allow fee updates, maybe just a summary on all the ways pinning can happen would be helpful, I don't think those limits had stuff like this in mind when they were added (as stuff like this didn't really exist) then 21:18:57 BlueMatt: +1 21:19:00 BlueMatt: sgtm 21:19:22 #action amend anchors to make no changes to htlc signatures, and resolve the other issues (needs tests cases, debate about 1-vs-2 etc) on the ml or issue 21:19:32 #action work on redoing things to use anchors (maybe based on feerates) in htlc txn 21:19:36 cdecker: I tend to agree, but I'm not sure what it *is* - there's a very good reason why it is what it is around DoS issues, and while I agree its not ideal even in its own right, the solution isnt clear. I have lots of ideas, but the resources needed to implement them are....no where near there 21:20:00 ariard: :+1: 21:20:00 yeh would need to be like a big initiative 21:20:28 roasbeef: true, all mempool rules should be documented as in bip125 21:20:35 i'm sorry to say that we are out of time 21:20:37 to make it easier to reason for mempool doing offchain stufff 21:20:41 *people 21:21:00 thanks for the great discussion today everyone and t-bast for the agenda 21:21:05 + 21:21:06 1 21:21:13 and cdecker for the co-chair assist ;) 21:21:27 thanks niftynei! 21:21:27 i'm going to close the meeting but feel free to continue discussion 21:21:36 #endmeeting