1 2018-08-28T00:03:37  *** Chris_Stewart_5 has quit IRC
  2 2018-08-28T00:05:19  *** Chris_Stewart_5 has joined #bitcoin-core-dev
  3 2018-08-28T00:09:23  *** phantomcircuit has quit IRC
  4 2018-08-28T00:09:36  *** phantomcircuit has joined #bitcoin-core-dev
  5 2018-08-28T00:13:57  *** kallisteiros has quit IRC
  6 2018-08-28T00:14:41  *** drexl has quit IRC
  7 2018-08-28T00:15:15  *** promag has joined #bitcoin-core-dev
  8 2018-08-28T00:22:19  <achow101> github ded :(
  9 2018-08-28T00:22:36  <sipa> aww
 10 2018-08-28T00:23:24  *** promag has quit IRC
 11 2018-08-28T00:24:00  *** promag has joined #bitcoin-core-dev
 12 2018-08-28T00:29:09  *** promag has quit IRC
 13 2018-08-28T00:32:27  <midnightmagic> achow101: huh?
 14 2018-08-28T00:37:26  *** promag has joined #bitcoin-core-dev
 15 2018-08-28T00:40:30  *** Chris_Stewart_5 has quit IRC
 16 2018-08-28T00:41:51  *** promag has quit IRC
 17 2018-08-28T00:50:22  *** BGL has joined #bitcoin-core-dev
 18 2018-08-28T00:55:58  *** wraithm has joined #bitcoin-core-dev
 19 2018-08-28T00:58:15  *** promag has joined #bitcoin-core-dev
 20 2018-08-28T01:03:03  *** promag has quit IRC
 21 2018-08-28T01:15:51  *** elichai2 has quit IRC
 22 2018-08-28T01:23:35  *** promag has joined #bitcoin-core-dev
 23 2018-08-28T01:28:21  *** promag has quit IRC
 24 2018-08-28T01:30:44  *** AaronvanW has quit IRC
 25 2018-08-28T01:53:48  <achow101> I was getting a bunch of errors earlier
 26 2018-08-28T02:18:59  *** plankers has quit IRC
 27 2018-08-28T02:33:58  *** chainhead has joined #bitcoin-core-dev
 28 2018-08-28T03:39:15  *** Krellan has quit IRC
 29 2018-08-28T03:59:09  *** Urgo has quit IRC
 30 2018-08-28T04:16:23  *** vexbuy has quit IRC
 31 2018-08-28T04:20:02  *** d9b4bef9 has quit IRC
 32 2018-08-28T04:21:07  *** d9b4bef9 has joined #bitcoin-core-dev
 33 2018-08-28T04:30:32  *** vexbuy has joined #bitcoin-core-dev
 34 2018-08-28T04:44:30  *** unholymachine has quit IRC
 35 2018-08-28T04:46:16  *** Rootsudo has quit IRC
 36 2018-08-28T04:46:57  *** Rootsudo has joined #bitcoin-core-dev
 37 2018-08-28T04:47:56  *** Rootsudo has joined #bitcoin-core-dev
 38 2018-08-28T04:48:45  *** promag has joined #bitcoin-core-dev
 39 2018-08-28T04:49:51  *** shesek has quit IRC
 40 2018-08-28T04:49:57  *** Victorsueca has quit IRC
 41 2018-08-28T04:51:08  *** Victorsueca has joined #bitcoin-core-dev
 42 2018-08-28T04:53:28  *** promag has quit IRC
 43 2018-08-28T04:55:29  *** Rootsudo has quit IRC
 44 2018-08-28T04:56:07  *** Rootsudo has joined #bitcoin-core-dev
 45 2018-08-28T04:57:47  *** Rootsudo has joined #bitcoin-core-dev
 46 2018-08-28T04:58:26  *** Rootsudo has joined #bitcoin-core-dev
 47 2018-08-28T04:59:20  *** Rootsudo has joined #bitcoin-core-dev
 48 2018-08-28T05:08:58  *** phantomcircuit has quit IRC
 49 2018-08-28T05:10:42  *** phantomcircuit has joined #bitcoin-core-dev
 50 2018-08-28T05:12:04  *** ken2812221_ has joined #bitcoin-core-dev
 51 2018-08-28T05:19:21  *** ken2812221_ has quit IRC
 52 2018-08-28T05:22:48  *** promag has joined #bitcoin-core-dev
 53 2018-08-28T05:27:32  *** promag has quit IRC
 54 2018-08-28T05:48:37  *** vexbuy_ has joined #bitcoin-core-dev
 55 2018-08-28T05:51:56  *** Krellan has joined #bitcoin-core-dev
 56 2018-08-28T05:52:11  *** promag has joined #bitcoin-core-dev
 57 2018-08-28T05:52:12  *** vexbuy has quit IRC
 58 2018-08-28T05:56:51  *** Chris_Stewart_5 has joined #bitcoin-core-dev
 59 2018-08-28T05:57:12  *** promag has quit IRC
 60 2018-08-28T06:01:13  *** vexbuy_ has quit IRC
 61 2018-08-28T06:14:07  *** vexbuy has joined #bitcoin-core-dev
 62 2018-08-28T06:26:07  *** Chris_Stewart_5 has quit IRC
 63 2018-08-28T06:27:25  *** promag has joined #bitcoin-core-dev
 64 2018-08-28T06:32:41  *** promag has quit IRC
 65 2018-08-28T06:37:22  *** pkx1 has joined #bitcoin-core-dev
 66 2018-08-28T06:38:20  *** promag has joined #bitcoin-core-dev
 67 2018-08-28T06:40:29  *** vexbuy has quit IRC
 68 2018-08-28T06:43:26  *** promag has quit IRC
 69 2018-08-28T06:50:08  *** Rootsudo has joined #bitcoin-core-dev
 70 2018-08-28T06:52:38  *** pkx1 has quit IRC
 71 2018-08-28T06:57:21  *** karelb has quit IRC
 72 2018-08-28T06:59:13  *** rex4539 has joined #bitcoin-core-dev
 73 2018-08-28T07:11:38  *** Amuza has joined #bitcoin-core-dev
 74 2018-08-28T07:13:01  *** promag has joined #bitcoin-core-dev
 75 2018-08-28T07:17:36  *** promag has quit IRC
 76 2018-08-28T07:24:11  *** pkx1 has joined #bitcoin-core-dev
 77 2018-08-28T07:25:10  *** promag has joined #bitcoin-core-dev
 78 2018-08-28T07:28:17  *** pkx1 has quit IRC
 79 2018-08-28T07:29:30  *** promag has quit IRC
 80 2018-08-28T07:31:30  *** vexbuy has joined #bitcoin-core-dev
 81 2018-08-28T07:37:02  *** Rootsudo has quit IRC
 82 2018-08-28T07:49:10  *** wxss has joined #bitcoin-core-dev
 83 2018-08-28T07:59:18  *** wxss has quit IRC
 84 2018-08-28T08:12:08  *** wxss has joined #bitcoin-core-dev
 85 2018-08-28T08:13:51  *** SopaXorzTaker has joined #bitcoin-core-dev
 86 2018-08-28T08:29:50  *** promag has joined #bitcoin-core-dev
 87 2018-08-28T08:38:02  *** d9b4bef9 has quit IRC
 88 2018-08-28T08:39:07  *** d9b4bef9 has joined #bitcoin-core-dev
 89 2018-08-28T08:53:58  *** WhatDaMath has quit IRC
 90 2018-08-28T09:07:36  <jonasschnelli> sipa, gmaxwell: I think we should keep 2^31 for the maximal message size, reducing to 3bytes (23bits + rekey bit) seems too small and not worth saving a byte
 91 2018-08-28T09:08:08  <jonasschnelli> BTW: -maxreceivebuffer is also respected with the new message types... (this is a level deeper more on the socket)
 92 2018-08-28T09:09:43  <jonasschnelli> Ah... nm. I was wrong. The buffer check only happens when a complete message has been loaded
 93 2018-08-28T09:09:47  *** AaronvanW has joined #bitcoin-core-dev
 94 2018-08-28T09:09:51  <wumpus> why would you ever need such a big message? in my experience it's usually better to split something up into smaller messages in that case
 95 2018-08-28T09:10:43  <wumpus> (for one, example to be able to continue an interrupted transfer, if it's all-or-nothing that won't work)
 96 2018-08-28T09:11:44  <jonasschnelli> 3 bytes - one rekey bit results in max size of 8’388’608 bytes (hardcoded in the protocol). I agree that we want smaller junks enforced on the application layer... but unsure about the protocol layer
 97 2018-08-28T09:12:27  <wumpus> so if you really need >24MB network messages, that is to me a suggestion something is wrong with the design at a higher level
 98 2018-08-28T09:12:40  <wumpus> right
 99 2018-08-28T09:13:37  <wumpus> if you also want to cram other data into the length word, it changes, though you could reserve the bits instead
100 2018-08-28T09:14:08  <jonasschnelli> Yes. Thats also an idea.
101 2018-08-28T09:14:47  <jonasschnelli> I also fail to see why someone should not be able to mem DOS a current peer with a incomplete message up to MAX_SIZE (33'554'432)...
102 2018-08-28T09:14:59  <jonasschnelli> The -maxreceivebuffer hits only when the message is complete
103 2018-08-28T09:15:01  <wumpus> so still leave 24 bits for the message length, a bit for rekey, and leave the rest for future use--it seems a bit wasteful but if your protocol is byte-based you don't get around such things
104 2018-08-28T09:15:58  <wumpus> right
105 2018-08-28T09:16:54  <wumpus> though as attacker you have to actually send the data, so if so it's a very slow DoS, but yes DoS potential is another good reason to not allow very large messages
106 2018-08-28T09:17:08  <jonasschnelli> [22:56:32]  <gmaxwell>	I think as you have it now thats probably a trivial DOS attack, e.g. I can make your node use 32mb * npeers ram... so 4GB additional usage in the default configuration
107 2018-08-28T09:17:17  <jonasschnelli> wumpus: I think that attack is possible today
108 2018-08-28T09:17:35  <jonasschnelli> npeers * MAX_SIZE
109 2018-08-28T09:17:38  <wumpus> yes
110 2018-08-28T09:18:35  <jonasschnelli> The question is, why is MAX_SIZE 0x2000000
111 2018-08-28T09:18:59  <jonasschnelli> Ah. That constant is also used at http level
112 2018-08-28T09:19:20  <jonasschnelli> and other places
113 2018-08-28T09:20:07  <wumpus> for http is pretty arbitrary
114 2018-08-28T09:20:28  <wumpus> but less important, as the http interface is meant for internal use, there are probably tons of ways to DoS it
115 2018-08-28T09:20:47  <jonasschnelli> I mean as an attacker, you can create a IPv6 peer and create 128 connections to a peer (assume remote peer has no used ports) and send the same incomplete 32MB message to the peer and consume 4GB
116 2018-08-28T09:22:48  <wumpus> as buffers are per connection, the only way around that would be to account, on a process level, of the total size of those buffers and go into some kind of DoS mode if too much is consumed in total
117 2018-08-28T09:23:22  <wumpus> you *need* a fairly large receive buffer (though maybe not that large) to handle all the current messages, multiplied with 120+ that's always going to be a lot !
118 2018-08-28T09:26:07  <jonasschnelli> I think there should be a combined view on those buffers and as you said some sort of DoS prevention mode or a max total buffer size that makes per-connection-queues pause
119 2018-08-28T09:26:32  <wumpus> indeed
120 2018-08-28T09:27:19  <wumpus> one way to handle going over the total buffer cap would be to simply stop reading until it's processed, another way would be to drop connections with low priority and large buffers
121 2018-08-28T09:28:35  <jonasschnelli> Yes.
122 2018-08-28T09:28:40  *** plankers has joined #bitcoin-core-dev
123 2018-08-28T09:29:36  <wumpus> might be out of scope for what you're doing, *not making it worse* would be nice
124 2018-08-28T09:30:17  <wumpus> (re: bitcoin P2P specifically: do we ever get *unexpected* big messages?)
125 2018-08-28T09:30:56  <wumpus> my intuition would be that the larger messages are always replies to requests such as getblock, getaddr, etc
126 2018-08-28T09:31:02  <jonasschnelli> The buffer attack problem is orthogonal to the v2 protocol I'm implementing
127 2018-08-28T09:31:20  <wumpus> if so, someone connecting to you and instantly offering a large message should probably be blasted
128 2018-08-28T09:31:23  <wumpus> yes, agree
129 2018-08-28T09:32:32  <jonasschnelli> I guess the attack would be, a) connect with a bunch of fake peers, do proper version handshake, send incomplete 32MB message (only header will be parsed) via all connections.
130 2018-08-28T09:32:50  <jonasschnelli> The attacked peer could reject messages based on per command max sizes
131 2018-08-28T09:33:03  <jonasschnelli> But then, unknown messages.. (unknown commands)
132 2018-08-28T09:33:30  <wumpus> well if someone wants to send you an unknown message, out of the blue, larger than N -> bye
133 2018-08-28T09:33:54  <wumpus> there's some scope for behavior modeling here, though I think keeping track of total buffer sizes is more predictable and easier to implement
134 2018-08-28T09:33:55  <jonasschnelli> Yes. That is implemented, though N is 0x2000000
135 2018-08-28T09:33:55  *** Krellan has quit IRC
136 2018-08-28T09:34:06  <jonasschnelli> wumpus: yes. agree
137 2018-08-28T09:35:06  *** Krellan has joined #bitcoin-core-dev
138 2018-08-28T09:37:46  <wumpus> it's even possible that libevent already has this funcionality, as it's not protocol specific
139 2018-08-28T09:39:06  *** math_ is now known as math
140 2018-08-28T09:39:32  *** math is now known as mthiel
141 2018-08-28T09:39:36  <jonasschnelli> good point...
142 2018-08-28T09:45:26  <jonasschnelli> Anyone know how I can solve this gcc -fPIC error while compiling test_bitcoin or bench_bitcoin:
143 2018-08-28T09:45:27  <jonasschnelli>   CXXLD    test/test_bitcoin
144 2018-08-28T09:45:37  <jonasschnelli>  /usr/bin/ld: crypto/libbitcoin_crypto_base.a(crypto_libbitcoin_crypto_base_a-chacha.o): relocation R_X86_64_32 against `.rodata.cst16' can not be used when making a PIE object; recompile with -fPIC
145 2018-08-28T09:45:46  <jonasschnelli> https://api.travis-ci.org/v3/job/421481343/log.txt
146 2018-08-28T09:45:55  <jonasschnelli> Is it because I added extern C code?
147 2018-08-28T09:46:39  <jonasschnelli> I think its clang not gcc, sry
148 2018-08-28T09:50:15  <wumpus> "You can assign bufferevents to a rate limiting group if you want to limit their total bandwidth usage." seems like a different thing though
149 2018-08-28T09:50:26  <wumpus> rate limiting versus total buffer limiting
150 2018-08-28T09:50:52  <wumpus> jonasschnelli: you're probably linking to a static library that wasn't compiled with -fPIC
151 2018-08-28T09:51:04  <wumpus> either link to a dynamic library instead or re-compile it with -fPIC
152 2018-08-28T09:52:37  <jonasschnelli> wumpus: could it be because test_bitcoin has LDFLAGS -static? It works on master, but when I add my chacha20/poly1305 code to LIBBITCOIN_CRYPTO, it failes
153 2018-08-28T09:54:24  <wumpus> on a general level it happens when some objects are built without -fPIC while requesting a position independent library/executable at link time-- '-static' will, on most platforms, generate a position dependent executable so no I don't think that's the reason
154 2018-08-28T09:55:05  *** elichai2 has joined #bitcoin-core-dev
155 2018-08-28T09:58:01  <wumpus> in this specific case, it looks like crypto_libbitcoin_crypto_base_a-chacha.o is built without -fPIC so the linker cannot include it
156 2018-08-28T10:01:53  <jonasschnelli> wumpus: thanks... I'll take a closer look
157 2018-08-28T10:08:34  <jonasschnelli> wumpus: I guess i found it, the added code (chacha/poly1305) are .c files and compiled with CC (CFLAGS) and not with CXX
158 2018-08-28T10:16:32  <wumpus> ahh the ancient problem
159 2018-08-28T10:20:46  <luke-jr> can probably just rename to .cpp?
160 2018-08-28T10:23:27  *** Guyver2 has joined #bitcoin-core-dev
161 2018-08-28T10:24:03  <jonasschnelli> luke-jr: trying right now...
162 2018-08-28T10:25:14  <wumpus> yes, 'porting' it to C++ might be the most straightforward solution, unless you need to do it repeatedly (such as when there's an upstream)
163 2018-08-28T10:27:00  <jonasschnelli> I guess porting is fine... but the compile cache is killing me
164 2018-08-28T10:28:02  <wumpus> both the name of the file and the compile flags will have changed, why would the cache pick that up?
165 2018-08-28T10:46:37  <wumpus> please, don't do this: #14087  we have PRs enough to cope with
166 2018-08-28T10:46:39  <gribble> https://github.com/bitcoin/bitcoin/issues/14087 | Remove unnecessary G_TRANSLATION_FUN nullptr assignment by promag · Pull Request #14087 · bitcoin/bitcoin · GitHub
167 2018-08-28T10:47:39  <wumpus> the code is correct and clear, no need to fiddle with it
168 2018-08-28T10:59:16  *** wxss has quit IRC
169 2018-08-28T10:59:33  *** wxss has joined #bitcoin-core-dev
170 2018-08-28T11:33:20  *** farmerwampum has joined #bitcoin-core-dev
171 2018-08-28T11:52:30  *** promag has quit IRC
172 2018-08-28T11:53:03  *** promag has joined #bitcoin-core-dev
173 2018-08-28T11:56:08  *** wxss has quit IRC
174 2018-08-28T11:57:33  *** promag has quit IRC
175 2018-08-28T12:22:32  *** Sinclair6 has joined #bitcoin-core-dev
176 2018-08-28T12:45:12  *** SopaXT has joined #bitcoin-core-dev
177 2018-08-28T12:48:22  *** SopaXorzTaker has quit IRC
178 2018-08-28T12:48:30  *** SopaXT has quit IRC
179 2018-08-28T12:48:46  *** SopaXorzTaker has joined #bitcoin-core-dev
180 2018-08-28T13:23:26  *** Dizzle has joined #bitcoin-core-dev
181 2018-08-28T13:27:13  *** vexbuy has quit IRC
182 2018-08-28T13:27:46  *** vexbuy has joined #bitcoin-core-dev
183 2018-08-28T13:32:34  *** vexbuy has quit IRC
184 2018-08-28T13:34:26  *** vexbuy has joined #bitcoin-core-dev
185 2018-08-28T13:35:58  *** vexbuy_ has joined #bitcoin-core-dev
186 2018-08-28T13:39:07  *** vexbuy has quit IRC
187 2018-08-28T13:58:12  *** itaseski has joined #bitcoin-core-dev
188 2018-08-28T13:58:55  <wumpus> ok, apparently the code was not correct and clear, it was assigning nullptr to a std::function, sorry
189 2018-08-28T14:26:09  <wumpus> btw: I intend to change my git mail to laanwj@protonmail.com (from laanwj@gmail.com), is this going to give any problems or set off alarms? (I'll keep using the same gpg key FWIW)
190 2018-08-28T14:26:25  *** Chris_Stewart_5 has joined #bitcoin-core-dev
191 2018-08-28T14:28:25  *** promag has joined #bitcoin-core-dev
192 2018-08-28T14:32:41  *** cdecker has left #bitcoin-core-dev
193 2018-08-28T14:33:10  *** promag has quit IRC
194 2018-08-28T14:38:34  <gmaxwell> 02:07:36 < jonasschnelli> sipa, gmaxwell: I think we should keep 2^31 for the  maximal message size, reducing to 3bytes (23bits +  rekey bit) seems too small and not worth saving a byte
195 2018-08-28T14:38:54  <gmaxwell> jonasschnelli: I agree that the byte is unimportant, but we will never have such big messages.
196 2018-08-28T14:39:19  <wumpus> indeed
197 2018-08-28T14:39:31  *** Amuza has quit IRC
198 2018-08-28T14:39:32  <gmaxwell> Even sending around 2MB messages is pretty terrible. Anything where a message would be that big should be split into smaller messages.
199 2018-08-28T14:39:44  *** promag has joined #bitcoin-core-dev
200 2018-08-28T14:40:14  <gmaxwell> I would say that we should introduce new messages to do this for blocks already, except that other than in IBD/catchup blocks are almost never sent whole anymore.
201 2018-08-28T14:40:18  <wumpus> yep, said the same thing
202 2018-08-28T14:44:00  <gmaxwell> E.g. a N-way split block message could send the merkle branches for the remaining parts, so you can verify the merkle root for each chunk as you get it, and not bother buffering something that is never going to be useful.
203 2018-08-28T14:45:18  <wumpus> similar to how bittorrent uses merkle trees IIRC
204 2018-08-28T14:45:49  *** vexbuy_ has quit IRC
205 2018-08-28T14:46:32  <wumpus> but yes, good idea
206 2018-08-28T14:50:31  <jonasschnelli> yes. agree.
207 2018-08-28T14:53:30  <gmaxwell> Really the message sizes should be driven by being large enough to make overheads low, and large enough to usefully process something incrementally (it's not so useful to send a message where we can do nothing at all with it except wait for more), and otherwise as small as possible to avoid creating large buffering and head of line blocking problems.
208 2018-08-28T15:06:54  <sipa> wumpus: add a new uid to your gpg key and upload it somewhere
209 2018-08-28T15:08:37  *** michaelsdunn1 has joined #bitcoin-core-dev
210 2018-08-28T15:09:22  <wumpus> sipa: I already did that part :)
211 2018-08-28T15:13:28  <waxwing> sipa, what's going on here? why would someone be adding both witness and non-witness utxo fields in the same input (bip174)? https://github.com/bitcoin/bitcoin/blob/master/src/script/sign.cpp#L256-L259
212 2018-08-28T15:18:23  <achow101> waxwing: it's just a safety check
213 2018-08-28T15:19:04  <achow101> waxwing: that could happen in the case that someone adds a non-witness utxo to a witness one because it is p2sh wrapped. Then someone else could, seeing that there was no witness utxo, add a witness one without removing the non-witness one
214 2018-08-28T15:19:21  <achow101> that would of course be an implementation error, but it should still be a case that needs to be handled
215 2018-08-28T15:21:52  <sipa> waxwing: both are added, and the irrelevant one is later deleted (because it's the signing code that determines which one was needed)
216 2018-08-28T15:22:05  <sipa> it will never make it out to the serialization
217 2018-08-28T15:22:09  <waxwing> it's technically more lax than what was specc'ed right, which was to reject anything with both. but the example you give is a plausible case for allowing it.
218 2018-08-28T15:22:35  <sipa> waxwing: this is just internals
219 2018-08-28T15:22:46  <sipa> it's not observable behavior
220 2018-08-28T15:23:31  <waxwing> ah ok, thanks
221 2018-08-28T15:23:38  <sipa> (scroll down a bit, see line 280-287
222 2018-08-28T15:26:18  *** promag has quit IRC
223 2018-08-28T15:26:58  *** rex4539 has quit IRC
224 2018-08-28T15:30:39  *** kallisteiros has joined #bitcoin-core-dev
225 2018-08-28T15:40:31  *** Victorsueca has quit IRC
226 2018-08-28T15:41:42  *** Victorsueca has joined #bitcoin-core-dev
227 2018-08-28T15:52:13  <gmaxwell> jonasschnelli: re memory usage, see MAX_PROTOCOL_MESSAGE_LENGTH. -- you can't cause 32MB usage today.
228 2018-08-28T15:53:25  <gmaxwell> I think if we were to redo things from scratch, we'd end up with a 256KB maximum message size or something like that.  The only reason I don't suggest making the encrypted transport limited to that sort of size is because we'd have to introduced split block messages first.
229 2018-08-28T15:54:06  <gmaxwell> and as I mentioned before, due to compact blocks I think it isn't particularly important to add split blocks.
230 2018-08-28T15:56:28  *** Krellan has quit IRC
231 2018-08-28T15:57:37  *** Krellan has joined #bitcoin-core-dev
232 2018-08-28T16:01:18  *** Rootsudo has joined #bitcoin-core-dev
233 2018-08-28T16:02:09  *** schmidty has joined #bitcoin-core-dev
234 2018-08-28T16:10:39  *** Rootsudo has quit IRC
235 2018-08-28T16:14:01  *** plankers has quit IRC
236 2018-08-28T16:17:23  *** Rootsudo has joined #bitcoin-core-dev
237 2018-08-28T16:42:12  *** blackbaba has joined #bitcoin-core-dev
238 2018-08-28T16:42:32  *** vexbuy has joined #bitcoin-core-dev
239 2018-08-28T16:55:06  *** blackbaba has quit IRC
240 2018-08-28T16:57:20  *** blackbaba has joined #bitcoin-core-dev
241 2018-08-28T17:00:00  *** blackbaba has quit IRC
242 2018-08-28T17:04:11  <ryanofsky> MarcoFalke, any suggestion on how handle "AppVeyor was unable to build non-mergeable pull request" in 9381? there are no conflicts. maybe close and reopen?
243 2018-08-28T17:05:16  *** Rootsudo has quit IRC
244 2018-08-28T17:06:02  <gmaxwell> I recommend a goat sacrifice.
245 2018-08-28T17:14:47  *** grafcaps has joined #bitcoin-core-dev
246 2018-08-28T17:18:31  <wumpus> yes, a goat should do in this case
247 2018-08-28T17:22:18  <luke-jr> who has one? someone is more rural than me?
248 2018-08-28T17:26:55  *** Giszmo has quit IRC
249 2018-08-28T17:30:30  <Chris_Stewart_5> any idea what is going on testnet ?
250 2018-08-28T17:31:06  <Chris_Stewart_5> or ¯\_(ツ)_/¯
251 2018-08-28T17:39:32  *** Giszmo has joined #bitcoin-core-dev
252 2018-08-28T17:41:22  *** ken2812221 has quit IRC
253 2018-08-28T17:48:19  *** owowo has joined #bitcoin-core-dev
254 2018-08-28T17:56:18  *** Dizzle has quit IRC
255 2018-08-28T17:58:29  *** Dizzle has joined #bitcoin-core-dev
256 2018-08-28T18:00:11  *** promag has joined #bitcoin-core-dev
257 2018-08-28T18:04:49  *** profmac has joined #bitcoin-core-dev
258 2018-08-28T18:05:55  *** Krellan has quit IRC
259 2018-08-28T18:07:11  *** Krellan has joined #bitcoin-core-dev
260 2018-08-28T18:17:04  *** ExtraCrispy has joined #bitcoin-core-dev
261 2018-08-28T18:25:24  *** sipa has quit IRC
262 2018-08-28T18:32:43  *** Krellan has quit IRC
263 2018-08-28T18:33:10  *** kallisteiros has quit IRC
264 2018-08-28T18:38:07  <jonasschnelli> gmaxwell: oh. Right. I missed MAX_PROTOCOL_MESSAGE_LENGTH.
265 2018-08-28T18:38:44  <jonasschnelli> That check is also there for the v2 protocol (in my PR)
266 2018-08-28T18:39:29  <jonasschnelli> But its in-precise in v2 because one package can contain multiple messages (and this checks a "package")
267 2018-08-28T18:39:57  <gmaxwell> We might want to log the command, length, number for each message we send, and see if the ability to batch multiple messages even potentially pays for the extra 3/4 bytes of length.
268 2018-08-28T18:40:32  *** vexbuy has quit IRC
269 2018-08-28T18:41:40  *** willyko_ has joined #bitcoin-core-dev
270 2018-08-28T18:42:04  <gmaxwell> jonasschnelli: what do you mean that its imprecise?
271 2018-08-28T18:42:29  <gmaxwell> The package payload should be limited to MAX_PROTOCOL_MESSAGE_LENGTH.
272 2018-08-28T18:42:33  <jonasschnelli> gmaxwell: if you combine 10 blocks into a single encryption package it gets rejected
273 2018-08-28T18:42:36  *** vexbuy has joined #bitcoin-core-dev
274 2018-08-28T18:42:54  <jonasschnelli> gmaxwell: Yes. I agree.
275 2018-08-28T18:43:00  <jonasschnelli> I guess its a wording thing,...
276 2018-08-28T18:43:17  <jonasschnelli> message is IMO the v2 inner message (the actual command&size&payload)
277 2018-08-28T18:43:19  <willyko_> hi all, i'm wondering what's the current gitian build tech stack people are using right now? i haven't been able to gitian build since the bionic upgrade
278 2018-08-28T18:43:22  <gmaxwell> Right good.  I understand, you're just saying that by applying it as max package length you can't always use a message of that size.
279 2018-08-28T18:43:29  <jonasschnelli> Where the outer message is the "package"
280 2018-08-28T18:43:36  *** promag has quit IRC
281 2018-08-28T18:43:56  <jonasschnelli> If combining messages gets a thing, people would expect that MAX_PROTOCOL_MESSAGE_LENGTH is per inner message
282 2018-08-28T18:44:33  <gmaxwell> jonasschnelli: if you would otherwise buffer more than MAX_PROTOCOL_MESSAGE_LENGTH then that just goes into another package.
283 2018-08-28T18:44:55  *** murrayn has quit IRC
284 2018-08-28T18:45:17  <jonasschnelli> gmaxwell: but you need to read-process the whole package to check the MAC before you can look at the inner messages
285 2018-08-28T18:45:47  <gmaxwell> Yes. I am referring to the sender.
286 2018-08-28T18:46:01  <jonasschnelli> oh. yes.
287 2018-08-28T18:46:38  <jonasschnelli> Yeah. The question is if we want to drop the multi-message thing and save 4 bytes (and some complication is size checkes)
288 2018-08-28T18:46:54  <jonasschnelli> I guess originally cfields came up with that idea
289 2018-08-28T18:47:06  <jonasschnelli> (the idea to use multi-message envelopes
290 2018-08-28T18:47:08  <jonasschnelli> +)
291 2018-08-28T18:47:23  <gmaxwell> But I'm wondering if the ability to put multiple messages in a package even makes sense. It adds 4 bytes (maybe we change to 3) overhead to every message, ... but only saves 16 bytes when we bundle.  And except for ping  no common message is especially small.
292 2018-08-28T18:47:45  <gmaxwell> A headers message, we want to minimize latency, so it will almost never be bundled.
293 2018-08-28T18:47:59  <gmaxwell> INVs get batched internally.
294 2018-08-28T18:48:01  <cfields> jonasschnelli: I believe I argued against :)
295 2018-08-28T18:48:24  <cfields> jonasschnelli: iirc my request was that IF we bundle, to throw an index up-front so that we could do optimistic preallocation
296 2018-08-28T18:48:50  <gmaxwell> It seemed like sipa was for some reason thinking that we could incrementally process inner messages before buffering the whole thing. But we can't or otherwise that moots the MAC.
297 2018-08-28T18:49:03  <jonasschnelli> Yeah...
298 2018-08-28T18:49:17  <jonasschnelli> I say drop it (the envelope/inner-message wrapping)
299 2018-08-28T18:49:59  <jonasschnelli> And reduce the length to 3 bytes (- rekey bit) resulting in 2^23 (8’388’608) max message size
300 2018-08-28T18:51:08  <gmaxwell> Does the inner length as done now include the variable length command?
301 2018-08-28T18:51:27  *** profmac has quit IRC
302 2018-08-28T18:51:29  <jonasschnelli> no
303 2018-08-28T18:51:46  <gmaxwell> It should. So you don't have to read the command in order to process the mac.
304 2018-08-28T18:52:37  <gmaxwell> by read I mean decode.
305 2018-08-28T18:52:44  <gmaxwell> [3 byte - 1 bit lenth][varlen command][16 byte mac].
306 2018-08-28T18:52:55  <jonasschnelli> wouldn't it be -> ( 3bytes_cipher_length_plus_flag || cipher_command || payload || 128bit-MAC )
307 2018-08-28T18:53:17  <gmaxwell> yes, well I mean, if you want to actually send payloads... :P
308 2018-08-28T18:53:28  <jonasschnelli> You decrypt the length, then read and MAC-check the rest, then decrypt
309 2018-08-28T18:53:51  <jonasschnelli> if there is no payload you would figure out during command varint processing and after the MAC
310 2018-08-28T18:55:33  <jonasschnelli> AEAD( AAD( K1_ENC( 3bytes_cipher_length_plus_flag ) ) || K2_ENC( cipher_command || payload ) )
311 2018-08-28T18:56:13  <gmaxwell> in the above the crypto layer just becomes completely oblivious to commands.  it's all just payload to it.
312 2018-08-28T18:57:30  <gmaxwell> how is the variable length command implemented?  the bip just says varlen which is not very clear.
313 2018-08-28T18:58:13  <jonasschnelli> varstr (a varint plus a string),... I think probably always one byte & the string
314 2018-08-28T18:58:29  <jonasschnelli> 0x03 'i' 'n' 'v'
315 2018-08-28T18:58:50  <gmaxwell> yes, it should be at most one byte. Allowing more to be encoded is just begging for bugs.
316 2018-08-28T18:59:12  <jonasschnelli> Yes. We could enforce one bytes length by the specs
317 2018-08-28T19:01:04  <gmaxwell> We could also change to command byte where 0-12 mean 0-12 byte explicit comands, and other values are mapped to some existing commands (like ping/inv/header) and the rest are undefined.
318 2018-08-28T19:01:39  <gmaxwell> but at least the size should just be made a byte.
319 2018-08-28T19:02:12  <gmaxwell> in general we should watch for cases where a parser would need to impose limits to protect itself from bad values, and just make those bad values impossible to encode.
320 2018-08-28T19:02:13  <jonasschnelli> Yeah.. would save another byte. :)
321 2018-08-28T19:04:03  <gmaxwell> well for ping the packed commands would save 4 bytes. For inv 3 bytes. for headers it would save 7 bytes...
322 2018-08-28T19:04:03  *** Orion3k has joined #bitcoin-core-dev
323 2018-08-28T19:04:46  *** kallisteiros has joined #bitcoin-core-dev
324 2018-08-28T19:05:29  <jonasschnelli> gmaxwell: Aha. I see now. Dropping the str-based command for a number-to-command mapping...
325 2018-08-28T19:06:02  <jonasschnelli> But smells a bit after central planing (I guess developers will tend then to address new values to new commands)
326 2018-08-28T19:06:02  <gmaxwell> jonasschnelli: str based exists for some messages, its needed for extensibility.
327 2018-08-28T19:06:14  <jonasschnelli> Yes. Sure. We can set 0xFF for str or something
328 2018-08-28T19:06:43  <jonasschnelli> or as you said, just use 0-12 for the very known ones
329 2018-08-28T19:06:50  <gmaxwell> Well I was suggesting that we have values 1-12 be for strings of length 1-12.
330 2018-08-28T19:06:53  <gmaxwell> right.
331 2018-08-28T19:07:19  <jonasschnelli> Or that. Yes.
332 2018-08-28T19:07:30  <gmaxwell> then use values for the few very common small protocol messages (ping, inv, header probably) and the rest just stay strings.
333 2018-08-28T19:07:48  <jonasschnelli> Jup. I like that.
334 2018-08-28T19:08:04  <gmaxwell> as far as centeral planning goes there isn't a reason that nodes couldn't use different numbers on dinfferent links.
335 2018-08-28T19:08:20  <gmaxwell> we wouldn't implement support for that unless needed, but it wouldn't be a problem.
336 2018-08-28T19:09:25  <gmaxwell> they 1-byte commands would really only be needed for commands which are frequent and small...  So, for example if we introduced a new kind of headers message, we might want to use one for that.
337 2018-08-28T19:09:47  <gmaxwell> But I don't see any reason to use a 1-byte command for a block.
338 2018-08-28T19:11:56  <jonasschnelli> Your argument is then more for latency then for bandwidth, right?
339 2018-08-28T19:13:46  <jonasschnelli> I guess if your peer gets heavy used for IBDing, saving a couple of bytes on the block command has the same effect then on the inv command (in terms of total bandwidth
340 2018-08-28T19:14:05  *** promag has joined #bitcoin-core-dev
341 2018-08-28T19:14:07  <gmaxwell> But say, for example, we decide we want to have a HEADERS2 message coded as value 27 in bitcoin core.  We're going to negoiate its activation with something like SENDHEADERS.   Say another implementation BitcoinConnectUnlimitedClassic (BCUX) wants to use 27 for XHEADERS. An implementation could support talking to both kinds of peers just by assigning the meaning of 27 based on the negoation.
342 2018-08-28T19:14:30  <gmaxwell> jonasschnelli: not in terms of percentage, however.
343 2018-08-28T19:15:51  *** promag has quit IRC
344 2018-08-28T19:17:00  <gmaxwell> Over the life of a node, even one that serves IBDing peers, it should service a lot more INV messages than block messages.
345 2018-08-28T19:19:28  <jonasschnelli> agree
346 2018-08-28T19:20:40  <gmaxwell> we can even specify that any usage of short types other than the ones initially set should be negoigated. So then there is basically no collision problem.
347 2018-08-28T19:22:29  *** profmac has joined #bitcoin-core-dev
348 2018-08-28T19:22:42  <jonasschnelli> With negotiating you mean some "proprietary" message?
349 2018-08-28T19:23:24  <gmaxwell> I mean just that you don't use them unless you know the other side can handle them, based on something else they've sent.
350 2018-08-28T19:24:25  <jonasschnelli> Yes. We should mention that...
351 2018-08-28T19:24:56  <gmaxwell> So for example, we'll likely introduce a new ADDR message in the future.  So your peer can send you a SENDADDR2 command, and then after they get that they know they can send you value=22 messages, per the ADDR2 BIP.
352 2018-08-28T19:25:30  <jonasschnelli> Got it. Yes.
353 2018-08-28T19:25:57  <jonasschnelli> gmaxwell: since the length is now caped at 2^23, how would decomposing a message distributed over two packages look like?
354 2018-08-28T19:26:25  <gmaxwell> Thats specific to the message.  It should be decomposed in a way that allows incremental processing.
355 2018-08-28T19:26:42  <jonasschnelli> Aha.. PARTIALBLOCK or something
356 2018-08-28T19:27:17  <jonasschnelli> Good. Then there is no extra handling on the transport layer.
357 2018-08-28T19:27:32  <gmaxwell> So for example, say we wanted to support breaking blocks up,  you'd split a block into N messages, the first message might be the header, tx count, and N hashes, where the N hashes are the merkelroots for the N chunks of the block.  Then additional messages for each chunk.  But the transport layer doesn't care.
358 2018-08-28T19:27:44  <gmaxwell> The rationale is that the right splitting is message type dependant.
359 2018-08-28T19:28:01  <gmaxwell> For an INV, for example, there is _just no reason_ to ever make a single INV that big, just make another INV.
360 2018-08-28T19:28:11  <gmaxwell> so the splitting for INVs is more invs.
361 2018-08-28T19:32:10  *** booyah_ has joined #bitcoin-core-dev
362 2018-08-28T19:32:31  *** booyah_ is now known as THE_JUDGE
363 2018-08-28T19:33:26  *** THE_JUDGE has quit IRC
364 2018-08-28T19:35:51  *** adiabat has joined #bitcoin-core-dev
365 2018-08-28T19:37:18  *** Giszmo has quit IRC
366 2018-08-28T19:39:07  *** SopaXorzTaker has quit IRC
367 2018-08-28T19:53:40  *** Giszmo has joined #bitcoin-core-dev
368 2018-08-28T19:54:32  *** Chris_Stewart_5 has quit IRC
369 2018-08-28T20:01:20  *** promag has joined #bitcoin-core-dev
370 2018-08-28T20:15:27  *** Chris_Stewart_5 has joined #bitcoin-core-dev
371 2018-08-28T20:42:23  *** adiabat has quit IRC
372 2018-08-28T20:42:55  *** elichai2 has quit IRC
373 2018-08-28T20:54:41  <promag> #13825 in HP?
374 2018-08-28T20:54:42  <gribble> https://github.com/bitcoin/bitcoin/issues/13825 | [wallet] Kill accounts by jnewbery · Pull Request #13825 · bitcoin/bitcoin · GitHub
375 2018-08-28T20:58:10  *** promag has quit IRC
376 2018-08-28T21:05:19  *** schmidty has quit IRC
377 2018-08-28T21:15:28  *** Giszmo has quit IRC
378 2018-08-28T21:18:33  <gmaxwell> jonasschnelli: interesting to note: 4(magic) + 12(command) + 4(length) + 4(sha2) = 24  which is more than 3 (length) + 1(type) + 16(mac) = 20  for messages that can use the 1-byte type.
379 2018-08-28T21:19:06  *** profmac has quit IRC
380 2018-08-28T21:31:28  *** Giszmo has joined #bitcoin-core-dev
381 2018-08-28T21:38:51  *** Dizzle has quit IRC
382 2018-08-28T21:38:55  *** drexl has joined #bitcoin-core-dev
383 2018-08-28T21:47:21  *** adiabat has joined #bitcoin-core-dev
384 2018-08-28T22:06:03  *** profmac has joined #bitcoin-core-dev
385 2018-08-28T22:09:52  *** michaelsdunn1 has quit IRC
386 2018-08-28T22:19:40  <Chris_Stewart_5> luke-jr: I could get my hands on a goat -- don't think i could sacrifice it though.
387 2018-08-28T22:19:45  <Chris_Stewart_5> ;)
388 2018-08-28T22:29:25  *** grubles_ has joined #bitcoin-core-dev
389 2018-08-28T22:30:10  *** grubles has quit IRC
390 2018-08-28T22:36:26  *** grubles_ has quit IRC
391 2018-08-28T22:44:52  *** shesek has joined #bitcoin-core-dev
392 2018-08-28T22:46:58  *** grubles has joined #bitcoin-core-dev
393 2018-08-28T22:47:40  *** Krellan has joined #bitcoin-core-dev
394 2018-08-28T22:50:32  *** Krellan_ has joined #bitcoin-core-dev
395 2018-08-28T22:51:06  *** sipa has joined #bitcoin-core-dev
396 2018-08-28T22:52:14  *** Krellan has joined #bitcoin-core-dev
397 2018-08-28T22:59:19  *** Guyver2 has quit IRC
398 2018-08-28T22:59:24  *** promag has joined #bitcoin-core-dev
399 2018-08-28T23:02:12  *** unholymachine has joined #bitcoin-core-dev
400 2018-08-28T23:04:46  *** kallisteiros has quit IRC
401 2018-08-28T23:11:00  *** lnostdal has quit IRC
402 2018-08-28T23:13:35  *** lnostdal has joined #bitcoin-core-dev
403 2018-08-28T23:16:36  *** drexl has quit IRC
404 2018-08-28T23:20:55  *** grubles has quit IRC
405 2018-08-28T23:22:58  *** profmac has quit IRC
406 2018-08-28T23:29:52  *** grubles has joined #bitcoin-core-dev
407 2018-08-28T23:43:13  *** rhavar_ has joined #bitcoin-core-dev
408 2018-08-28T23:47:12  *** willyko_ has quit IRC