12018-08-28T00:03:37 *** Chris_Stewart_5 has quit IRC
22018-08-28T00:05:19 *** Chris_Stewart_5 has joined #bitcoin-core-dev
32018-08-28T00:09:23 *** phantomcircuit has quit IRC
42018-08-28T00:09:36 *** phantomcircuit has joined #bitcoin-core-dev
52018-08-28T00:13:57 *** kallisteiros has quit IRC
62018-08-28T00:14:41 *** drexl has quit IRC
72018-08-28T00:15:15 *** promag has joined #bitcoin-core-dev
82018-08-28T00:22:19 <achow101> github ded :(
92018-08-28T00:22:36 <sipa> aww
102018-08-28T00:23:24 *** promag has quit IRC
112018-08-28T00:24:00 *** promag has joined #bitcoin-core-dev
122018-08-28T00:29:09 *** promag has quit IRC
132018-08-28T00:32:27 <midnightmagic> achow101: huh?
142018-08-28T00:37:26 *** promag has joined #bitcoin-core-dev
152018-08-28T00:40:30 *** Chris_Stewart_5 has quit IRC
162018-08-28T00:41:51 *** promag has quit IRC
172018-08-28T00:50:22 *** BGL has joined #bitcoin-core-dev
182018-08-28T00:55:58 *** wraithm has joined #bitcoin-core-dev
192018-08-28T00:58:15 *** promag has joined #bitcoin-core-dev
202018-08-28T01:03:03 *** promag has quit IRC
212018-08-28T01:15:51 *** elichai2 has quit IRC
222018-08-28T01:23:35 *** promag has joined #bitcoin-core-dev
232018-08-28T01:28:21 *** promag has quit IRC
242018-08-28T01:30:44 *** AaronvanW has quit IRC
252018-08-28T01:53:48 <achow101> I was getting a bunch of errors earlier
262018-08-28T02:18:59 *** plankers has quit IRC
272018-08-28T02:33:58 *** chainhead has joined #bitcoin-core-dev
282018-08-28T03:39:15 *** Krellan has quit IRC
292018-08-28T03:59:09 *** Urgo has quit IRC
302018-08-28T04:16:23 *** vexbuy has quit IRC
312018-08-28T04:20:02 *** d9b4bef9 has quit IRC
322018-08-28T04:21:07 *** d9b4bef9 has joined #bitcoin-core-dev
332018-08-28T04:30:32 *** vexbuy has joined #bitcoin-core-dev
342018-08-28T04:44:30 *** unholymachine has quit IRC
352018-08-28T04:46:16 *** Rootsudo has quit IRC
362018-08-28T04:46:57 *** Rootsudo has joined #bitcoin-core-dev
372018-08-28T04:47:56 *** Rootsudo has joined #bitcoin-core-dev
382018-08-28T04:48:45 *** promag has joined #bitcoin-core-dev
392018-08-28T04:49:51 *** shesek has quit IRC
402018-08-28T04:49:57 *** Victorsueca has quit IRC
412018-08-28T04:51:08 *** Victorsueca has joined #bitcoin-core-dev
422018-08-28T04:53:28 *** promag has quit IRC
432018-08-28T04:55:29 *** Rootsudo has quit IRC
442018-08-28T04:56:07 *** Rootsudo has joined #bitcoin-core-dev
452018-08-28T04:57:47 *** Rootsudo has joined #bitcoin-core-dev
462018-08-28T04:58:26 *** Rootsudo has joined #bitcoin-core-dev
472018-08-28T04:59:20 *** Rootsudo has joined #bitcoin-core-dev
482018-08-28T05:08:58 *** phantomcircuit has quit IRC
492018-08-28T05:10:42 *** phantomcircuit has joined #bitcoin-core-dev
502018-08-28T05:12:04 *** ken2812221_ has joined #bitcoin-core-dev
512018-08-28T05:19:21 *** ken2812221_ has quit IRC
522018-08-28T05:22:48 *** promag has joined #bitcoin-core-dev
532018-08-28T05:27:32 *** promag has quit IRC
542018-08-28T05:48:37 *** vexbuy_ has joined #bitcoin-core-dev
552018-08-28T05:51:56 *** Krellan has joined #bitcoin-core-dev
562018-08-28T05:52:11 *** promag has joined #bitcoin-core-dev
572018-08-28T05:52:12 *** vexbuy has quit IRC
582018-08-28T05:56:51 *** Chris_Stewart_5 has joined #bitcoin-core-dev
592018-08-28T05:57:12 *** promag has quit IRC
602018-08-28T06:01:13 *** vexbuy_ has quit IRC
612018-08-28T06:14:07 *** vexbuy has joined #bitcoin-core-dev
622018-08-28T06:26:07 *** Chris_Stewart_5 has quit IRC
632018-08-28T06:27:25 *** promag has joined #bitcoin-core-dev
642018-08-28T06:32:41 *** promag has quit IRC
652018-08-28T06:37:22 *** pkx1 has joined #bitcoin-core-dev
662018-08-28T06:38:20 *** promag has joined #bitcoin-core-dev
672018-08-28T06:40:29 *** vexbuy has quit IRC
682018-08-28T06:43:26 *** promag has quit IRC
692018-08-28T06:50:08 *** Rootsudo has joined #bitcoin-core-dev
702018-08-28T06:52:38 *** pkx1 has quit IRC
712018-08-28T06:57:21 *** karelb has quit IRC
722018-08-28T06:59:13 *** rex4539 has joined #bitcoin-core-dev
732018-08-28T07:11:38 *** Amuza has joined #bitcoin-core-dev
742018-08-28T07:13:01 *** promag has joined #bitcoin-core-dev
752018-08-28T07:17:36 *** promag has quit IRC
762018-08-28T07:24:11 *** pkx1 has joined #bitcoin-core-dev
772018-08-28T07:25:10 *** promag has joined #bitcoin-core-dev
782018-08-28T07:28:17 *** pkx1 has quit IRC
792018-08-28T07:29:30 *** promag has quit IRC
802018-08-28T07:31:30 *** vexbuy has joined #bitcoin-core-dev
812018-08-28T07:37:02 *** Rootsudo has quit IRC
822018-08-28T07:49:10 *** wxss has joined #bitcoin-core-dev
832018-08-28T07:59:18 *** wxss has quit IRC
842018-08-28T08:12:08 *** wxss has joined #bitcoin-core-dev
852018-08-28T08:13:51 *** SopaXorzTaker has joined #bitcoin-core-dev
862018-08-28T08:29:50 *** promag has joined #bitcoin-core-dev
872018-08-28T08:38:02 *** d9b4bef9 has quit IRC
882018-08-28T08:39:07 *** d9b4bef9 has joined #bitcoin-core-dev
892018-08-28T08:53:58 *** WhatDaMath has quit IRC
902018-08-28T09:07:36 <jonasschnelli> sipa, gmaxwell: I think we should keep 2^31 for the maximal message size, reducing to 3bytes (23bits + rekey bit) seems too small and not worth saving a byte
912018-08-28T09:08:08 <jonasschnelli> BTW: -maxreceivebuffer is also respected with the new message types... (this is a level deeper more on the socket)
922018-08-28T09:09:43 <jonasschnelli> Ah... nm. I was wrong. The buffer check only happens when a complete message has been loaded
932018-08-28T09:09:47 *** AaronvanW has joined #bitcoin-core-dev
942018-08-28T09:09:51 <wumpus> why would you ever need such a big message? in my experience it's usually better to split something up into smaller messages in that case
952018-08-28T09:10:43 <wumpus> (for one, example to be able to continue an interrupted transfer, if it's all-or-nothing that won't work)
962018-08-28T09:11:44 <jonasschnelli> 3 bytes - one rekey bit results in max size of 8â388â608 bytes (hardcoded in the protocol). I agree that we want smaller junks enforced on the application layer... but unsure about the protocol layer
972018-08-28T09:12:27 <wumpus> so if you really need >24MB network messages, that is to me a suggestion something is wrong with the design at a higher level
982018-08-28T09:12:40 <wumpus> right
992018-08-28T09:13:37 <wumpus> if you also want to cram other data into the length word, it changes, though you could reserve the bits instead
1002018-08-28T09:14:08 <jonasschnelli> Yes. Thats also an idea.
1012018-08-28T09:14:47 <jonasschnelli> I also fail to see why someone should not be able to mem DOS a current peer with a incomplete message up to MAX_SIZE (33'554'432)...
1022018-08-28T09:14:59 <jonasschnelli> The -maxreceivebuffer hits only when the message is complete
1032018-08-28T09:15:01 <wumpus> so still leave 24 bits for the message length, a bit for rekey, and leave the rest for future use--it seems a bit wasteful but if your protocol is byte-based you don't get around such things
1042018-08-28T09:15:58 <wumpus> right
1052018-08-28T09:16:54 <wumpus> though as attacker you have to actually send the data, so if so it's a very slow DoS, but yes DoS potential is another good reason to not allow very large messages
1062018-08-28T09:17:08 <jonasschnelli> [22:56:32] <gmaxwell> I think as you have it now thats probably a trivial DOS attack, e.g. I can make your node use 32mb * npeers ram... so 4GB additional usage in the default configuration
1072018-08-28T09:17:17 <jonasschnelli> wumpus: I think that attack is possible today
1082018-08-28T09:17:35 <jonasschnelli> npeers * MAX_SIZE
1092018-08-28T09:17:38 <wumpus> yes
1102018-08-28T09:18:35 <jonasschnelli> The question is, why is MAX_SIZE 0x2000000
1112018-08-28T09:18:59 <jonasschnelli> Ah. That constant is also used at http level
1122018-08-28T09:19:20 <jonasschnelli> and other places
1132018-08-28T09:20:07 <wumpus> for http is pretty arbitrary
1142018-08-28T09:20:28 <wumpus> but less important, as the http interface is meant for internal use, there are probably tons of ways to DoS it
1152018-08-28T09:20:47 <jonasschnelli> I mean as an attacker, you can create a IPv6 peer and create 128 connections to a peer (assume remote peer has no used ports) and send the same incomplete 32MB message to the peer and consume 4GB
1162018-08-28T09:22:48 <wumpus> as buffers are per connection, the only way around that would be to account, on a process level, of the total size of those buffers and go into some kind of DoS mode if too much is consumed in total
1172018-08-28T09:23:22 <wumpus> you *need* a fairly large receive buffer (though maybe not that large) to handle all the current messages, multiplied with 120+ that's always going to be a lot !
1182018-08-28T09:26:07 <jonasschnelli> I think there should be a combined view on those buffers and as you said some sort of DoS prevention mode or a max total buffer size that makes per-connection-queues pause
1192018-08-28T09:26:32 <wumpus> indeed
1202018-08-28T09:27:19 <wumpus> one way to handle going over the total buffer cap would be to simply stop reading until it's processed, another way would be to drop connections with low priority and large buffers
1212018-08-28T09:28:35 <jonasschnelli> Yes.
1222018-08-28T09:28:40 *** plankers has joined #bitcoin-core-dev
1232018-08-28T09:29:36 <wumpus> might be out of scope for what you're doing, *not making it worse* would be nice
1242018-08-28T09:30:17 <wumpus> (re: bitcoin P2P specifically: do we ever get *unexpected* big messages?)
1252018-08-28T09:30:56 <wumpus> my intuition would be that the larger messages are always replies to requests such as getblock, getaddr, etc
1262018-08-28T09:31:02 <jonasschnelli> The buffer attack problem is orthogonal to the v2 protocol I'm implementing
1272018-08-28T09:31:20 <wumpus> if so, someone connecting to you and instantly offering a large message should probably be blasted
1282018-08-28T09:31:23 <wumpus> yes, agree
1292018-08-28T09:32:32 <jonasschnelli> I guess the attack would be, a) connect with a bunch of fake peers, do proper version handshake, send incomplete 32MB message (only header will be parsed) via all connections.
1302018-08-28T09:32:50 <jonasschnelli> The attacked peer could reject messages based on per command max sizes
1312018-08-28T09:33:03 <jonasschnelli> But then, unknown messages.. (unknown commands)
1322018-08-28T09:33:30 <wumpus> well if someone wants to send you an unknown message, out of the blue, larger than N -> bye
1332018-08-28T09:33:54 <wumpus> there's some scope for behavior modeling here, though I think keeping track of total buffer sizes is more predictable and easier to implement
1342018-08-28T09:33:55 <jonasschnelli> Yes. That is implemented, though N is 0x2000000
1352018-08-28T09:33:55 *** Krellan has quit IRC
1362018-08-28T09:34:06 <jonasschnelli> wumpus: yes. agree
1372018-08-28T09:35:06 *** Krellan has joined #bitcoin-core-dev
1382018-08-28T09:37:46 <wumpus> it's even possible that libevent already has this funcionality, as it's not protocol specific
1392018-08-28T09:39:06 *** math_ is now known as math
1402018-08-28T09:39:32 *** math is now known as mthiel
1412018-08-28T09:39:36 <jonasschnelli> good point...
1422018-08-28T09:45:26 <jonasschnelli> Anyone know how I can solve this gcc -fPIC error while compiling test_bitcoin or bench_bitcoin:
1432018-08-28T09:45:27 <jonasschnelli> CXXLD test/test_bitcoin
1442018-08-28T09:45:37 <jonasschnelli> /usr/bin/ld: crypto/libbitcoin_crypto_base.a(crypto_libbitcoin_crypto_base_a-chacha.o): relocation R_X86_64_32 against `.rodata.cst16' can not be used when making a PIE object; recompile with -fPIC
1452018-08-28T09:45:46 <jonasschnelli> https://api.travis-ci.org/v3/job/421481343/log.txt
1462018-08-28T09:45:55 <jonasschnelli> Is it because I added extern C code?
1472018-08-28T09:46:39 <jonasschnelli> I think its clang not gcc, sry
1482018-08-28T09:50:15 <wumpus> "You can assign bufferevents to a rate limiting group if you want to limit their total bandwidth usage." seems like a different thing though
1492018-08-28T09:50:26 <wumpus> rate limiting versus total buffer limiting
1502018-08-28T09:50:52 <wumpus> jonasschnelli: you're probably linking to a static library that wasn't compiled with -fPIC
1512018-08-28T09:51:04 <wumpus> either link to a dynamic library instead or re-compile it with -fPIC
1522018-08-28T09:52:37 <jonasschnelli> wumpus: could it be because test_bitcoin has LDFLAGS -static? It works on master, but when I add my chacha20/poly1305 code to LIBBITCOIN_CRYPTO, it failes
1532018-08-28T09:54:24 <wumpus> on a general level it happens when some objects are built without -fPIC while requesting a position independent library/executable at link time-- '-static' will, on most platforms, generate a position dependent executable so no I don't think that's the reason
1542018-08-28T09:55:05 *** elichai2 has joined #bitcoin-core-dev
1552018-08-28T09:58:01 <wumpus> in this specific case, it looks like crypto_libbitcoin_crypto_base_a-chacha.o is built without -fPIC so the linker cannot include it
1562018-08-28T10:01:53 <jonasschnelli> wumpus: thanks... I'll take a closer look
1572018-08-28T10:08:34 <jonasschnelli> wumpus: I guess i found it, the added code (chacha/poly1305) are .c files and compiled with CC (CFLAGS) and not with CXX
1582018-08-28T10:16:32 <wumpus> ahh the ancient problem
1592018-08-28T10:20:46 <luke-jr> can probably just rename to .cpp?
1602018-08-28T10:23:27 *** Guyver2 has joined #bitcoin-core-dev
1612018-08-28T10:24:03 <jonasschnelli> luke-jr: trying right now...
1622018-08-28T10:25:14 <wumpus> yes, 'porting' it to C++ might be the most straightforward solution, unless you need to do it repeatedly (such as when there's an upstream)
1632018-08-28T10:27:00 <jonasschnelli> I guess porting is fine... but the compile cache is killing me
1642018-08-28T10:28:02 <wumpus> both the name of the file and the compile flags will have changed, why would the cache pick that up?
1652018-08-28T10:46:37 <wumpus> please, don't do this: #14087 we have PRs enough to cope with
1662018-08-28T10:46:39 <gribble> https://github.com/bitcoin/bitcoin/issues/14087 | Remove unnecessary G_TRANSLATION_FUN nullptr assignment by promag · Pull Request #14087 · bitcoin/bitcoin · GitHub
1672018-08-28T10:47:39 <wumpus> the code is correct and clear, no need to fiddle with it
1682018-08-28T10:59:16 *** wxss has quit IRC
1692018-08-28T10:59:33 *** wxss has joined #bitcoin-core-dev
1702018-08-28T11:33:20 *** farmerwampum has joined #bitcoin-core-dev
1712018-08-28T11:52:30 *** promag has quit IRC
1722018-08-28T11:53:03 *** promag has joined #bitcoin-core-dev
1732018-08-28T11:56:08 *** wxss has quit IRC
1742018-08-28T11:57:33 *** promag has quit IRC
1752018-08-28T12:22:32 *** Sinclair6 has joined #bitcoin-core-dev
1762018-08-28T12:45:12 *** SopaXT has joined #bitcoin-core-dev
1772018-08-28T12:48:22 *** SopaXorzTaker has quit IRC
1782018-08-28T12:48:30 *** SopaXT has quit IRC
1792018-08-28T12:48:46 *** SopaXorzTaker has joined #bitcoin-core-dev
1802018-08-28T13:23:26 *** Dizzle has joined #bitcoin-core-dev
1812018-08-28T13:27:13 *** vexbuy has quit IRC
1822018-08-28T13:27:46 *** vexbuy has joined #bitcoin-core-dev
1832018-08-28T13:32:34 *** vexbuy has quit IRC
1842018-08-28T13:34:26 *** vexbuy has joined #bitcoin-core-dev
1852018-08-28T13:35:58 *** vexbuy_ has joined #bitcoin-core-dev
1862018-08-28T13:39:07 *** vexbuy has quit IRC
1872018-08-28T13:58:12 *** itaseski has joined #bitcoin-core-dev
1882018-08-28T13:58:55 <wumpus> ok, apparently the code was not correct and clear, it was assigning nullptr to a std::function, sorry
1892018-08-28T14:26:09 <wumpus> btw: I intend to change my git mail to laanwj@protonmail.com (from laanwj@gmail.com), is this going to give any problems or set off alarms? (I'll keep using the same gpg key FWIW)
1902018-08-28T14:26:25 *** Chris_Stewart_5 has joined #bitcoin-core-dev
1912018-08-28T14:28:25 *** promag has joined #bitcoin-core-dev
1922018-08-28T14:32:41 *** cdecker has left #bitcoin-core-dev
1932018-08-28T14:33:10 *** promag has quit IRC
1942018-08-28T14:38:34 <gmaxwell> 02:07:36 < jonasschnelli> sipa, gmaxwell: I think we should keep 2^31 for the maximal message size, reducing to 3bytes (23bits + rekey bit) seems too small and not worth saving a byte
1952018-08-28T14:38:54 <gmaxwell> jonasschnelli: I agree that the byte is unimportant, but we will never have such big messages.
1962018-08-28T14:39:19 <wumpus> indeed
1972018-08-28T14:39:31 *** Amuza has quit IRC
1982018-08-28T14:39:32 <gmaxwell> Even sending around 2MB messages is pretty terrible. Anything where a message would be that big should be split into smaller messages.
1992018-08-28T14:39:44 *** promag has joined #bitcoin-core-dev
2002018-08-28T14:40:14 <gmaxwell> I would say that we should introduce new messages to do this for blocks already, except that other than in IBD/catchup blocks are almost never sent whole anymore.
2012018-08-28T14:40:18 <wumpus> yep, said the same thing
2022018-08-28T14:44:00 <gmaxwell> E.g. a N-way split block message could send the merkle branches for the remaining parts, so you can verify the merkle root for each chunk as you get it, and not bother buffering something that is never going to be useful.
2032018-08-28T14:45:18 <wumpus> similar to how bittorrent uses merkle trees IIRC
2042018-08-28T14:45:49 *** vexbuy_ has quit IRC
2052018-08-28T14:46:32 <wumpus> but yes, good idea
2062018-08-28T14:50:31 <jonasschnelli> yes. agree.
2072018-08-28T14:53:30 <gmaxwell> Really the message sizes should be driven by being large enough to make overheads low, and large enough to usefully process something incrementally (it's not so useful to send a message where we can do nothing at all with it except wait for more), and otherwise as small as possible to avoid creating large buffering and head of line blocking problems.
2082018-08-28T15:06:54 <sipa> wumpus: add a new uid to your gpg key and upload it somewhere
2092018-08-28T15:08:37 *** michaelsdunn1 has joined #bitcoin-core-dev
2102018-08-28T15:09:22 <wumpus> sipa: I already did that part :)
2112018-08-28T15:13:28 <waxwing> sipa, what's going on here? why would someone be adding both witness and non-witness utxo fields in the same input (bip174)? https://github.com/bitcoin/bitcoin/blob/master/src/script/sign.cpp#L256-L259
2122018-08-28T15:18:23 <achow101> waxwing: it's just a safety check
2132018-08-28T15:19:04 <achow101> waxwing: that could happen in the case that someone adds a non-witness utxo to a witness one because it is p2sh wrapped. Then someone else could, seeing that there was no witness utxo, add a witness one without removing the non-witness one
2142018-08-28T15:19:21 <achow101> that would of course be an implementation error, but it should still be a case that needs to be handled
2152018-08-28T15:21:52 <sipa> waxwing: both are added, and the irrelevant one is later deleted (because it's the signing code that determines which one was needed)
2162018-08-28T15:22:05 <sipa> it will never make it out to the serialization
2172018-08-28T15:22:09 <waxwing> it's technically more lax than what was specc'ed right, which was to reject anything with both. but the example you give is a plausible case for allowing it.
2182018-08-28T15:22:35 <sipa> waxwing: this is just internals
2192018-08-28T15:22:46 <sipa> it's not observable behavior
2202018-08-28T15:23:31 <waxwing> ah ok, thanks
2212018-08-28T15:23:38 <sipa> (scroll down a bit, see line 280-287
2222018-08-28T15:26:18 *** promag has quit IRC
2232018-08-28T15:26:58 *** rex4539 has quit IRC
2242018-08-28T15:30:39 *** kallisteiros has joined #bitcoin-core-dev
2252018-08-28T15:40:31 *** Victorsueca has quit IRC
2262018-08-28T15:41:42 *** Victorsueca has joined #bitcoin-core-dev
2272018-08-28T15:52:13 <gmaxwell> jonasschnelli: re memory usage, see MAX_PROTOCOL_MESSAGE_LENGTH. -- you can't cause 32MB usage today.
2282018-08-28T15:53:25 <gmaxwell> I think if we were to redo things from scratch, we'd end up with a 256KB maximum message size or something like that. The only reason I don't suggest making the encrypted transport limited to that sort of size is because we'd have to introduced split block messages first.
2292018-08-28T15:54:06 <gmaxwell> and as I mentioned before, due to compact blocks I think it isn't particularly important to add split blocks.
2302018-08-28T15:56:28 *** Krellan has quit IRC
2312018-08-28T15:57:37 *** Krellan has joined #bitcoin-core-dev
2322018-08-28T16:01:18 *** Rootsudo has joined #bitcoin-core-dev
2332018-08-28T16:02:09 *** schmidty has joined #bitcoin-core-dev
2342018-08-28T16:10:39 *** Rootsudo has quit IRC
2352018-08-28T16:14:01 *** plankers has quit IRC
2362018-08-28T16:17:23 *** Rootsudo has joined #bitcoin-core-dev
2372018-08-28T16:42:12 *** blackbaba has joined #bitcoin-core-dev
2382018-08-28T16:42:32 *** vexbuy has joined #bitcoin-core-dev
2392018-08-28T16:55:06 *** blackbaba has quit IRC
2402018-08-28T16:57:20 *** blackbaba has joined #bitcoin-core-dev
2412018-08-28T17:00:00 *** blackbaba has quit IRC
2422018-08-28T17:04:11 <ryanofsky> MarcoFalke, any suggestion on how handle "AppVeyor was unable to build non-mergeable pull request" in 9381? there are no conflicts. maybe close and reopen?
2432018-08-28T17:05:16 *** Rootsudo has quit IRC
2442018-08-28T17:06:02 <gmaxwell> I recommend a goat sacrifice.
2452018-08-28T17:14:47 *** grafcaps has joined #bitcoin-core-dev
2462018-08-28T17:18:31 <wumpus> yes, a goat should do in this case
2472018-08-28T17:22:18 <luke-jr> who has one? someone is more rural than me?
2482018-08-28T17:26:55 *** Giszmo has quit IRC
2492018-08-28T17:30:30 <Chris_Stewart_5> any idea what is going on testnet ?
2502018-08-28T17:31:06 <Chris_Stewart_5> or ¯\_(ã)_/¯
2512018-08-28T17:39:32 *** Giszmo has joined #bitcoin-core-dev
2522018-08-28T17:41:22 *** ken2812221 has quit IRC
2532018-08-28T17:48:19 *** owowo has joined #bitcoin-core-dev
2542018-08-28T17:56:18 *** Dizzle has quit IRC
2552018-08-28T17:58:29 *** Dizzle has joined #bitcoin-core-dev
2562018-08-28T18:00:11 *** promag has joined #bitcoin-core-dev
2572018-08-28T18:04:49 *** profmac has joined #bitcoin-core-dev
2582018-08-28T18:05:55 *** Krellan has quit IRC
2592018-08-28T18:07:11 *** Krellan has joined #bitcoin-core-dev
2602018-08-28T18:17:04 *** ExtraCrispy has joined #bitcoin-core-dev
2612018-08-28T18:25:24 *** sipa has quit IRC
2622018-08-28T18:32:43 *** Krellan has quit IRC
2632018-08-28T18:33:10 *** kallisteiros has quit IRC
2642018-08-28T18:38:07 <jonasschnelli> gmaxwell: oh. Right. I missed MAX_PROTOCOL_MESSAGE_LENGTH.
2652018-08-28T18:38:44 <jonasschnelli> That check is also there for the v2 protocol (in my PR)
2662018-08-28T18:39:29 <jonasschnelli> But its in-precise in v2 because one package can contain multiple messages (and this checks a "package")
2672018-08-28T18:39:57 <gmaxwell> We might want to log the command, length, number for each message we send, and see if the ability to batch multiple messages even potentially pays for the extra 3/4 bytes of length.
2682018-08-28T18:40:32 *** vexbuy has quit IRC
2692018-08-28T18:41:40 *** willyko_ has joined #bitcoin-core-dev
2702018-08-28T18:42:04 <gmaxwell> jonasschnelli: what do you mean that its imprecise?
2712018-08-28T18:42:29 <gmaxwell> The package payload should be limited to MAX_PROTOCOL_MESSAGE_LENGTH.
2722018-08-28T18:42:33 <jonasschnelli> gmaxwell: if you combine 10 blocks into a single encryption package it gets rejected
2732018-08-28T18:42:36 *** vexbuy has joined #bitcoin-core-dev
2742018-08-28T18:42:54 <jonasschnelli> gmaxwell: Yes. I agree.
2752018-08-28T18:43:00 <jonasschnelli> I guess its a wording thing,...
2762018-08-28T18:43:17 <jonasschnelli> message is IMO the v2 inner message (the actual command&size&payload)
2772018-08-28T18:43:19 <willyko_> hi all, i'm wondering what's the current gitian build tech stack people are using right now? i haven't been able to gitian build since the bionic upgrade
2782018-08-28T18:43:22 <gmaxwell> Right good. I understand, you're just saying that by applying it as max package length you can't always use a message of that size.
2792018-08-28T18:43:29 <jonasschnelli> Where the outer message is the "package"
2802018-08-28T18:43:36 *** promag has quit IRC
2812018-08-28T18:43:56 <jonasschnelli> If combining messages gets a thing, people would expect that MAX_PROTOCOL_MESSAGE_LENGTH is per inner message
2822018-08-28T18:44:33 <gmaxwell> jonasschnelli: if you would otherwise buffer more than MAX_PROTOCOL_MESSAGE_LENGTH then that just goes into another package.
2832018-08-28T18:44:55 *** murrayn has quit IRC
2842018-08-28T18:45:17 <jonasschnelli> gmaxwell: but you need to read-process the whole package to check the MAC before you can look at the inner messages
2852018-08-28T18:45:47 <gmaxwell> Yes. I am referring to the sender.
2862018-08-28T18:46:01 <jonasschnelli> oh. yes.
2872018-08-28T18:46:38 <jonasschnelli> Yeah. The question is if we want to drop the multi-message thing and save 4 bytes (and some complication is size checkes)
2882018-08-28T18:46:54 <jonasschnelli> I guess originally cfields came up with that idea
2892018-08-28T18:47:06 <jonasschnelli> (the idea to use multi-message envelopes
2902018-08-28T18:47:08 <jonasschnelli> +)
2912018-08-28T18:47:23 <gmaxwell> But I'm wondering if the ability to put multiple messages in a package even makes sense. It adds 4 bytes (maybe we change to 3) overhead to every message, ... but only saves 16 bytes when we bundle. And except for ping no common message is especially small.
2922018-08-28T18:47:45 <gmaxwell> A headers message, we want to minimize latency, so it will almost never be bundled.
2932018-08-28T18:47:59 <gmaxwell> INVs get batched internally.
2942018-08-28T18:48:01 <cfields> jonasschnelli: I believe I argued against :)
2952018-08-28T18:48:24 <cfields> jonasschnelli: iirc my request was that IF we bundle, to throw an index up-front so that we could do optimistic preallocation
2962018-08-28T18:48:50 <gmaxwell> It seemed like sipa was for some reason thinking that we could incrementally process inner messages before buffering the whole thing. But we can't or otherwise that moots the MAC.
2972018-08-28T18:49:03 <jonasschnelli> Yeah...
2982018-08-28T18:49:17 <jonasschnelli> I say drop it (the envelope/inner-message wrapping)
2992018-08-28T18:49:59 <jonasschnelli> And reduce the length to 3 bytes (- rekey bit) resulting in 2^23 (8â388â608) max message size
3002018-08-28T18:51:08 <gmaxwell> Does the inner length as done now include the variable length command?
3012018-08-28T18:51:27 *** profmac has quit IRC
3022018-08-28T18:51:29 <jonasschnelli> no
3032018-08-28T18:51:46 <gmaxwell> It should. So you don't have to read the command in order to process the mac.
3042018-08-28T18:52:37 <gmaxwell> by read I mean decode.
3052018-08-28T18:52:44 <gmaxwell> [3 byte - 1 bit lenth][varlen command][16 byte mac].
3062018-08-28T18:52:55 <jonasschnelli> wouldn't it be -> ( 3bytes_cipher_length_plus_flag || cipher_command || payload || 128bit-MAC )
3072018-08-28T18:53:17 <gmaxwell> yes, well I mean, if you want to actually send payloads... :P
3082018-08-28T18:53:28 <jonasschnelli> You decrypt the length, then read and MAC-check the rest, then decrypt
3092018-08-28T18:53:51 <jonasschnelli> if there is no payload you would figure out during command varint processing and after the MAC
3102018-08-28T18:55:33 <jonasschnelli> AEAD( AAD( K1_ENC( 3bytes_cipher_length_plus_flag ) ) || K2_ENC( cipher_command || payload ) )
3112018-08-28T18:56:13 <gmaxwell> in the above the crypto layer just becomes completely oblivious to commands. it's all just payload to it.
3122018-08-28T18:57:30 <gmaxwell> how is the variable length command implemented? the bip just says varlen which is not very clear.
3132018-08-28T18:58:13 <jonasschnelli> varstr (a varint plus a string),... I think probably always one byte & the string
3142018-08-28T18:58:29 <jonasschnelli> 0x03 'i' 'n' 'v'
3152018-08-28T18:58:50 <gmaxwell> yes, it should be at most one byte. Allowing more to be encoded is just begging for bugs.
3162018-08-28T18:59:12 <jonasschnelli> Yes. We could enforce one bytes length by the specs
3172018-08-28T19:01:04 <gmaxwell> We could also change to command byte where 0-12 mean 0-12 byte explicit comands, and other values are mapped to some existing commands (like ping/inv/header) and the rest are undefined.
3182018-08-28T19:01:39 <gmaxwell> but at least the size should just be made a byte.
3192018-08-28T19:02:12 <gmaxwell> in general we should watch for cases where a parser would need to impose limits to protect itself from bad values, and just make those bad values impossible to encode.
3202018-08-28T19:02:13 <jonasschnelli> Yeah.. would save another byte. :)
3212018-08-28T19:04:03 <gmaxwell> well for ping the packed commands would save 4 bytes. For inv 3 bytes. for headers it would save 7 bytes...
3222018-08-28T19:04:03 *** Orion3k has joined #bitcoin-core-dev
3232018-08-28T19:04:46 *** kallisteiros has joined #bitcoin-core-dev
3242018-08-28T19:05:29 <jonasschnelli> gmaxwell: Aha. I see now. Dropping the str-based command for a number-to-command mapping...
3252018-08-28T19:06:02 <jonasschnelli> But smells a bit after central planing (I guess developers will tend then to address new values to new commands)
3262018-08-28T19:06:02 <gmaxwell> jonasschnelli: str based exists for some messages, its needed for extensibility.
3272018-08-28T19:06:14 <jonasschnelli> Yes. Sure. We can set 0xFF for str or something
3282018-08-28T19:06:43 <jonasschnelli> or as you said, just use 0-12 for the very known ones
3292018-08-28T19:06:50 <gmaxwell> Well I was suggesting that we have values 1-12 be for strings of length 1-12.
3302018-08-28T19:06:53 <gmaxwell> right.
3312018-08-28T19:07:19 <jonasschnelli> Or that. Yes.
3322018-08-28T19:07:30 <gmaxwell> then use values for the few very common small protocol messages (ping, inv, header probably) and the rest just stay strings.
3332018-08-28T19:07:48 <jonasschnelli> Jup. I like that.
3342018-08-28T19:08:04 <gmaxwell> as far as centeral planning goes there isn't a reason that nodes couldn't use different numbers on dinfferent links.
3352018-08-28T19:08:20 <gmaxwell> we wouldn't implement support for that unless needed, but it wouldn't be a problem.
3362018-08-28T19:09:25 <gmaxwell> they 1-byte commands would really only be needed for commands which are frequent and small... So, for example if we introduced a new kind of headers message, we might want to use one for that.
3372018-08-28T19:09:47 <gmaxwell> But I don't see any reason to use a 1-byte command for a block.
3382018-08-28T19:11:56 <jonasschnelli> Your argument is then more for latency then for bandwidth, right?
3392018-08-28T19:13:46 <jonasschnelli> I guess if your peer gets heavy used for IBDing, saving a couple of bytes on the block command has the same effect then on the inv command (in terms of total bandwidth
3402018-08-28T19:14:05 *** promag has joined #bitcoin-core-dev
3412018-08-28T19:14:07 <gmaxwell> But say, for example, we decide we want to have a HEADERS2 message coded as value 27 in bitcoin core. We're going to negoiate its activation with something like SENDHEADERS. Say another implementation BitcoinConnectUnlimitedClassic (BCUX) wants to use 27 for XHEADERS. An implementation could support talking to both kinds of peers just by assigning the meaning of 27 based on the negoation.
3422018-08-28T19:14:30 <gmaxwell> jonasschnelli: not in terms of percentage, however.
3432018-08-28T19:15:51 *** promag has quit IRC
3442018-08-28T19:17:00 <gmaxwell> Over the life of a node, even one that serves IBDing peers, it should service a lot more INV messages than block messages.
3452018-08-28T19:19:28 <jonasschnelli> agree
3462018-08-28T19:20:40 <gmaxwell> we can even specify that any usage of short types other than the ones initially set should be negoigated. So then there is basically no collision problem.
3472018-08-28T19:22:29 *** profmac has joined #bitcoin-core-dev
3482018-08-28T19:22:42 <jonasschnelli> With negotiating you mean some "proprietary" message?
3492018-08-28T19:23:24 <gmaxwell> I mean just that you don't use them unless you know the other side can handle them, based on something else they've sent.
3502018-08-28T19:24:25 <jonasschnelli> Yes. We should mention that...
3512018-08-28T19:24:56 <gmaxwell> So for example, we'll likely introduce a new ADDR message in the future. So your peer can send you a SENDADDR2 command, and then after they get that they know they can send you value=22 messages, per the ADDR2 BIP.
3522018-08-28T19:25:30 <jonasschnelli> Got it. Yes.
3532018-08-28T19:25:57 <jonasschnelli> gmaxwell: since the length is now caped at 2^23, how would decomposing a message distributed over two packages look like?
3542018-08-28T19:26:25 <gmaxwell> Thats specific to the message. It should be decomposed in a way that allows incremental processing.
3552018-08-28T19:26:42 <jonasschnelli> Aha.. PARTIALBLOCK or something
3562018-08-28T19:27:17 <jonasschnelli> Good. Then there is no extra handling on the transport layer.
3572018-08-28T19:27:32 <gmaxwell> So for example, say we wanted to support breaking blocks up, you'd split a block into N messages, the first message might be the header, tx count, and N hashes, where the N hashes are the merkelroots for the N chunks of the block. Then additional messages for each chunk. But the transport layer doesn't care.
3582018-08-28T19:27:44 <gmaxwell> The rationale is that the right splitting is message type dependant.
3592018-08-28T19:28:01 <gmaxwell> For an INV, for example, there is _just no reason_ to ever make a single INV that big, just make another INV.
3602018-08-28T19:28:11 <gmaxwell> so the splitting for INVs is more invs.
3612018-08-28T19:32:10 *** booyah_ has joined #bitcoin-core-dev
3622018-08-28T19:32:31 *** booyah_ is now known as THE_JUDGE
3632018-08-28T19:33:26 *** THE_JUDGE has quit IRC
3642018-08-28T19:35:51 *** adiabat has joined #bitcoin-core-dev
3652018-08-28T19:37:18 *** Giszmo has quit IRC
3662018-08-28T19:39:07 *** SopaXorzTaker has quit IRC
3672018-08-28T19:53:40 *** Giszmo has joined #bitcoin-core-dev
3682018-08-28T19:54:32 *** Chris_Stewart_5 has quit IRC
3692018-08-28T20:01:20 *** promag has joined #bitcoin-core-dev
3702018-08-28T20:15:27 *** Chris_Stewart_5 has joined #bitcoin-core-dev
3712018-08-28T20:42:23 *** adiabat has quit IRC
3722018-08-28T20:42:55 *** elichai2 has quit IRC
3732018-08-28T20:54:41 <promag> #13825 in HP?
3742018-08-28T20:54:42 <gribble> https://github.com/bitcoin/bitcoin/issues/13825 | [wallet] Kill accounts by jnewbery · Pull Request #13825 · bitcoin/bitcoin · GitHub
3752018-08-28T20:58:10 *** promag has quit IRC
3762018-08-28T21:05:19 *** schmidty has quit IRC
3772018-08-28T21:15:28 *** Giszmo has quit IRC
3782018-08-28T21:18:33 <gmaxwell> jonasschnelli: interesting to note: 4(magic) + 12(command) + 4(length) + 4(sha2) = 24 which is more than 3 (length) + 1(type) + 16(mac) = 20 for messages that can use the 1-byte type.
3792018-08-28T21:19:06 *** profmac has quit IRC
3802018-08-28T21:31:28 *** Giszmo has joined #bitcoin-core-dev
3812018-08-28T21:38:51 *** Dizzle has quit IRC
3822018-08-28T21:38:55 *** drexl has joined #bitcoin-core-dev
3832018-08-28T21:47:21 *** adiabat has joined #bitcoin-core-dev
3842018-08-28T22:06:03 *** profmac has joined #bitcoin-core-dev
3852018-08-28T22:09:52 *** michaelsdunn1 has quit IRC
3862018-08-28T22:19:40 <Chris_Stewart_5> luke-jr: I could get my hands on a goat -- don't think i could sacrifice it though.
3872018-08-28T22:19:45 <Chris_Stewart_5> ;)
3882018-08-28T22:29:25 *** grubles_ has joined #bitcoin-core-dev
3892018-08-28T22:30:10 *** grubles has quit IRC
3902018-08-28T22:36:26 *** grubles_ has quit IRC
3912018-08-28T22:44:52 *** shesek has joined #bitcoin-core-dev
3922018-08-28T22:46:58 *** grubles has joined #bitcoin-core-dev
3932018-08-28T22:47:40 *** Krellan has joined #bitcoin-core-dev
3942018-08-28T22:50:32 *** Krellan_ has joined #bitcoin-core-dev
3952018-08-28T22:51:06 *** sipa has joined #bitcoin-core-dev
3962018-08-28T22:52:14 *** Krellan has joined #bitcoin-core-dev
3972018-08-28T22:59:19 *** Guyver2 has quit IRC
3982018-08-28T22:59:24 *** promag has joined #bitcoin-core-dev
3992018-08-28T23:02:12 *** unholymachine has joined #bitcoin-core-dev
4002018-08-28T23:04:46 *** kallisteiros has quit IRC
4012018-08-28T23:11:00 *** lnostdal has quit IRC
4022018-08-28T23:13:35 *** lnostdal has joined #bitcoin-core-dev
4032018-08-28T23:16:36 *** drexl has quit IRC
4042018-08-28T23:20:55 *** grubles has quit IRC
4052018-08-28T23:22:58 *** profmac has quit IRC
4062018-08-28T23:29:52 *** grubles has joined #bitcoin-core-dev
4072018-08-28T23:43:13 *** rhavar_ has joined #bitcoin-core-dev
4082018-08-28T23:47:12 *** willyko_ has quit IRC