1 2018-08-27T00:00:00  *** promag has quit IRC
  2 2018-08-27T00:50:44  *** Chris_Stewart_5 has quit IRC
  3 2018-08-27T00:52:38  *** promag has joined #bitcoin-core-dev
  4 2018-08-27T00:57:31  *** promag has quit IRC
  5 2018-08-27T00:57:47  *** Chris_Stewart_5 has joined #bitcoin-core-dev
  6 2018-08-27T01:15:02  *** d9b4bef9 has quit IRC
  7 2018-08-27T01:16:07  *** d9b4bef9 has joined #bitcoin-core-dev
  8 2018-08-27T01:17:01  *** d9b4bef9 has quit IRC
  9 2018-08-27T01:20:15  *** d9b4bef9 has joined #bitcoin-core-dev
 10 2018-08-27T01:40:27  *** pkx1 has joined #bitcoin-core-dev
 11 2018-08-27T01:43:41  *** pkx1 has quit IRC
 12 2018-08-27T01:53:04  *** pkx1 has joined #bitcoin-core-dev
 13 2018-08-27T02:08:32  *** intcat has quit IRC
 14 2018-08-27T02:10:11  *** intcat has joined #bitcoin-core-dev
 15 2018-08-27T02:14:35  *** AaronvanW has quit IRC
 16 2018-08-27T02:20:46  *** plankers has quit IRC
 17 2018-08-27T02:34:50  *** plankers has joined #bitcoin-core-dev
 18 2018-08-27T02:37:00  *** phantomcircuit has joined #bitcoin-core-dev
 19 2018-08-27T02:41:55  *** pkx1 has quit IRC
 20 2018-08-27T02:42:16  *** pkx1 has joined #bitcoin-core-dev
 21 2018-08-27T02:48:07  *** pkx1 has quit IRC
 22 2018-08-27T02:48:13  *** Jmabsd has joined #bitcoin-core-dev
 23 2018-08-27T02:48:47  <Jmabsd> wait, where in the code again are the logics for checking if the PoW is on-target - i think there are two locations that do this comparison.
 24 2018-08-27T02:53:56  <sipa> pow.cpp, CheckProofOfWork
 25 2018-08-27T02:56:07  <phantomcircuit> sipa, can pnode->GetRefCount() be == instead of <= 0
 26 2018-08-27T02:56:25  <phantomcircuit> you put in an assert that it's >= 0 in 2013 soo
 27 2018-08-27T02:56:57  <sipa> phantomcircuit: i'm probably the wrong person to ask
 28 2018-08-27T03:03:51  *** StopAndDecrypt has joined #bitcoin-core-dev
 29 2018-08-27T03:11:07  *** ken2812221_ has joined #bitcoin-core-dev
 30 2018-08-27T03:28:58  <ossifrage> gmaxwell, it seems adding MADV_RANDOM to the leveldb mmap would help reduce the memory footprint, but it isn't going to help performance if there is any locality or larger reads
 31 2018-08-27T03:29:18  <sipa> there is no locality
 32 2018-08-27T03:29:27  <sipa> they're sorted by txid
 33 2018-08-27T03:30:13  <gmaxwell> ossifrage: I think it's somewhat likely to help performance specifically because there is no locality in reads.
 34 2018-08-27T03:31:15  <gmaxwell> ossifrage: I think the thing we really need to do is start measuring dirty pages, rather than RES.  And I expect MADV_RANDOM will make it faster.
 35 2018-08-27T03:31:19  <ossifrage> gmaxwell, it all depends on the timeframe, if over enough time does it hit contiguous blocks and is that timeframe shorter then the the life of the page
 36 2018-08-27T03:31:39  <ossifrage> pay now vs pay later
 37 2018-08-27T03:36:48  *** vexbuy_ has quit IRC
 38 2018-08-27T03:39:18  <ossifrage> looking at: pmap -x $(pidof bitcoin-qt) | grep .ldb | awk '{if($3 == 0){print}}'       shows that many of the ldb files aren't using any RSS memory (after being up for almost a month)
 39 2018-08-27T03:39:34  <ossifrage> But I tend to OOM the POS box far too often
 40 2018-08-27T03:39:45  <Jmabsd> crypto/sha256.* is *single-iteration* SHA256?  (would be great if this was marked out in the file)
 41 2018-08-27T03:40:37  <sipa> Jmabsd: of course; SHA256 is SHA256, not double SHA256
 42 2018-08-27T03:40:44  <Jmabsd> (y)
 43 2018-08-27T03:41:03  <sipa> double SHA256 is called "Hash" in the sourcecode
 44 2018-08-27T03:41:13  <sipa> and SHA256+RIPEMD160 is called Hash160
 45 2018-08-27T03:41:28  <Jmabsd> (y)
 46 2018-08-27T04:03:00  *** Chris_Stewart_5 has quit IRC
 47 2018-08-27T04:11:16  <Jmabsd> wait, in the Bitcoin PoW 32bit floating point structure "deserializer" at https://github.com/bitcoin/bitcoin/blob/eb7daf4d600eeb631427c018a984a77a34aca66e/src/arith_uint256.cpp#L209
 48 2018-08-27T04:11:35  <Jmabsd> , you first shift-right a 32bit uint by 24 positions - this means the highest 8 bits become the lowest 8 bits, and the remainder of bits will be zero
 49 2018-08-27T04:11:54  <Jmabsd> then you bitwise-and the whole thing with 0x007fffff - but that's useless, isn't it?
 50 2018-08-27T04:12:13  <Jmabsd> here that's the same as bitwise-anding with 0xff, which was already implied by the shift-right ??
 51 2018-08-27T04:13:28  *** thaumavorio has joined #bitcoin-core-dev
 52 2018-08-27T04:14:57  <sipa> Jmabsd: read again
 53 2018-08-27T04:15:59  <Jmabsd> >> is right-shift, line 208 does "int nSize = nCompact >> 24;".
 54 2018-08-27T04:16:10  <Jmabsd> nCompact is uint32_t which is afaik an unsigned 32bit integer.
 55 2018-08-27T04:16:32  <Jmabsd> right-shifting 24 bits means what i suggested above. "int" here doesn't matter because it's at least 32bits and we're decimating the size here.
 56 2018-08-27T04:16:38  <Jmabsd> OH
 57 2018-08-27T04:16:47  <Jmabsd> sipa: ah you're right, the AND is on the original compact value. thx.
 58 2018-08-27T04:22:47  *** kallewoof has joined #bitcoin-core-dev
 59 2018-08-27T04:29:08  *** instagibbs has quit IRC
 60 2018-08-27T04:30:02  *** instagibbs has joined #bitcoin-core-dev
 61 2018-08-27T05:03:13  <luke-jr> FWIW, the most non-mmap files open I've seen is 39
 62 2018-08-27T05:03:48  <luke-jr> seems like it's txindex-related, as that's using 47 mmaps itself
 63 2018-08-27T05:07:44  <luke-jr> err, not txindex, I have that disabled apparently
 64 2018-08-27T05:07:49  <luke-jr> blocks/index :p
 65 2018-08-27T05:08:36  <luke-jr> I imagine it'd be worse with txindex
 66 2018-08-27T05:10:21  <Jmabsd> Wait, just a funny detail on the "showing block hash in reverse hex byte order in order for the PoW zeroes to be represented in the string's leading position":
 67 2018-08-27T05:10:26  <Jmabsd> *actually*, the hex string printer does NOT reverse the hex positions (https://github.com/bitcoin/bitcoin/blob/eb7daf4d600eeb631427c018a984a77a34aca66e/src/utilstrencodings.h#L121 - see here there's no reversing of the order of the bits in the byte) - and the reversing logic applied on the whole per-structure level is reversing byte order only, never bit order (ref. https://github.com/bitcoin/bitcoin/blob/eb7daf4d600eeb631427c018a984a77a34aca66e/sr
 68 2018-08-27T05:10:26  <Jmabsd> c/uint256.cpp#L23)
 69 2018-08-27T05:10:31  <Jmabsd> for this reason, actually, within each 8-multiple of PoW zeroes, actually the serialization will NOT be made in reverse order, and, we can see this already at block #5 here: https://www.blockchain.com/btc/block/000000003031a0e73735690c5a1ff2a4be82553b2a12b776fbd3a215dc8f778d
 70 2018-08-27T05:10:33  <gribble> https://github.com/bitcoin/bitcoin/issues/5 | Make the version number the protocol version and not the client version · Issue #5 · bitcoin/bitcoin · GitHub
 71 2018-08-27T05:10:34  <Jmabsd> *#6
 72 2018-08-27T05:10:36  <Jmabsd> Its hash is "000000003031a0e73735690c5a1ff2a4be82553b2a12b776fbd3a215dc8f778d".
 73 2018-08-27T05:10:37  <gribble> https://github.com/bitcoin/bitcoin/issues/6 | Treat wallet as a generic keystore · Issue #6 · bitcoin/bitcoin · GitHub
 74 2018-08-27T05:10:40  <Jmabsd> Do you see there, that, zeroes at the end of the block hash, which do count as PoW zeroes, are actually represented in the hex string *BEHIND* the digit 3,
 75 2018-08-27T05:10:45  <Jmabsd> yeah so this is not a big deal but the reversing (done for illustrative purposes) is only partial.
 76 2018-08-27T05:11:36  <luke-jr> Jmabsd: pretty sure you are wrong
 77 2018-08-27T05:11:51  <luke-jr> endianness deals with the bytes, not the bits
 78 2018-08-27T05:13:16  <sipa> Jmabsd: you're getting far too worked up about endianness issues
 79 2018-08-27T05:13:50  <sipa> they're conventions, and usually every convention sucks for some reason, but you still need to pick one
 80 2018-08-27T05:13:53  <sipa> deal with it
 81 2018-08-27T05:14:00  <sipa> if you get it wrong, try the other way
 82 2018-08-27T05:14:26  <luke-jr> only downside to big endian AFAIK is that you can't truncate by a simple cast :P
 83 2018-08-27T05:15:37  *** Krellan has quit IRC
 84 2018-08-27T05:16:07  *** Krellan has joined #bitcoin-core-dev
 85 2018-08-27T05:17:44  <sipa> but yes, endianness is about the bytes (or symbols), not the indivudual bits
 86 2018-08-27T05:19:21  *** plankers has quit IRC
 87 2018-08-27T05:20:20  *** sipa has quit IRC
 88 2018-08-27T05:25:02  *** d9b4bef9 has quit IRC
 89 2018-08-27T05:25:38  *** sipa has joined #bitcoin-core-dev
 90 2018-08-27T05:27:46  <Jmabsd> sipa: right, and an example of where PoW-significant zeroes show up in a non-leading position is block #6, https://www.blockchain.com/btc/block/000000003031a0e73735690c5a1ff2a4be82553b2a12b776fbd3a215dc8f778d
 91 2018-08-27T05:27:48  <gribble> https://github.com/bitcoin/bitcoin/issues/6 | Treat wallet as a generic keystore · Issue #6 · bitcoin/bitcoin · GitHub
 92 2018-08-27T05:28:29  <Jmabsd> the 5:th byte in the reverse order there, is 0x30, the 0 there is zeroes however representation is byte-reversed only not bit reversed, and so the last bits in that byte get on the second hex position behind the 3.
 93 2018-08-27T05:28:35  <Jmabsd> this is trivial but anyhow.
 94 2018-08-27T05:28:51  <sipa> Jmabsd: i think it's getting offtopic here
 95 2018-08-27T05:29:11  <sipa> we can discuss the general principle, but at some point you just to try and see
 96 2018-08-27T05:30:14  <Jmabsd> yep.
 97 2018-08-27T05:33:23  *** savil has quit IRC
 98 2018-08-27T05:43:18  *** plankers has joined #bitcoin-core-dev
 99 2018-08-27T05:45:54  *** kallisteiros has joined #bitcoin-core-dev
100 2018-08-27T05:54:39  *** savil has joined #bitcoin-core-dev
101 2018-08-27T06:01:56  *** Jmabsd has quit IRC
102 2018-08-27T06:02:05  *** kallisteiros has quit IRC
103 2018-08-27T06:03:16  *** kallisteiros has joined #bitcoin-core-dev
104 2018-08-27T06:04:42  <ossifrage> luke-jr, the original reason for the leveldb mmap change was that it was using a large number of file descriptors, enough to push incoming connections out of the <1024 fd select() limit. Bumping up the mmap count was sorta a workaround for that problem
105 2018-08-27T06:05:19  *** karelb has quit IRC
106 2018-08-27T06:05:54  *** jl2012 has quit IRC
107 2018-08-27T06:06:07  *** jl2012 has joined #bitcoin-core-dev
108 2018-08-27T06:07:03  *** karelb has joined #bitcoin-core-dev
109 2018-08-27T06:10:52  *** karelb has quit IRC
110 2018-08-27T06:11:38  *** atroxes has quit IRC
111 2018-08-27T06:11:48  *** karelb has joined #bitcoin-core-dev
112 2018-08-27T06:11:53  *** atroxes has joined #bitcoin-core-dev
113 2018-08-27T06:12:51  *** kallisteiros has quit IRC
114 2018-08-27T06:13:11  *** kallisteiros has joined #bitcoin-core-dev
115 2018-08-27T06:22:56  *** kallisteiros has quit IRC
116 2018-08-27T06:23:26  *** rex4539 has quit IRC
117 2018-08-27T06:23:26  *** kallisteiros has joined #bitcoin-core-dev
118 2018-08-27T06:27:51  *** Victorsueca has quit IRC
119 2018-08-27T06:29:15  *** Victorsueca has joined #bitcoin-core-dev
120 2018-08-27T06:32:43  *** kallisteiros has quit IRC
121 2018-08-27T06:35:51  <luke-jr> ossifrage: yes, I understand why now
122 2018-08-27T06:46:41  *** takinbo has joined #bitcoin-core-dev
123 2018-08-27T07:06:44  *** setpill has joined #bitcoin-core-dev
124 2018-08-27T07:21:48  *** Krellan has quit IRC
125 2018-08-27T07:22:29  *** Krellan has joined #bitcoin-core-dev
126 2018-08-27T07:34:44  *** vindard has joined #bitcoin-core-dev
127 2018-08-27T07:40:48  *** vindard has left #bitcoin-core-dev
128 2018-08-27T08:30:09  *** Guyver2 has joined #bitcoin-core-dev
129 2018-08-27T08:40:53  *** AaronvanW has joined #bitcoin-core-dev
130 2018-08-27T08:45:29  *** Zenton has joined #bitcoin-core-dev
131 2018-08-27T08:48:12  *** SopaXorzTaker has joined #bitcoin-core-dev
132 2018-08-27T08:54:25  *** promag has joined #bitcoin-core-dev
133 2018-08-27T08:56:39  *** Guyver2 has quit IRC
134 2018-08-27T09:00:08  *** elichai2 has joined #bitcoin-core-dev
135 2018-08-27T09:05:27  *** promag has quit IRC
136 2018-08-27T09:05:37  *** promag has joined #bitcoin-core-dev
137 2018-08-27T09:11:10  *** Zenton has quit IRC
138 2018-08-27T09:11:10  *** Zenton has joined #bitcoin-core-dev
139 2018-08-27T09:11:30  *** Zenton is now known as vicenteH
140 2018-08-27T09:12:52  *** vicenteH is now known as Zenton
141 2018-08-27T09:14:07  *** d9b4bef9 has joined #bitcoin-core-dev
142 2018-08-27T09:17:01  *** d9b4bef9 has quit IRC
143 2018-08-27T09:18:07  *** d9b4bef9 has joined #bitcoin-core-dev
144 2018-08-27T09:30:08  *** Krellan has quit IRC
145 2018-08-27T09:30:11  *** promag has quit IRC
146 2018-08-27T09:31:24  *** Krellan has joined #bitcoin-core-dev
147 2018-08-27T09:37:20  *** Zenton has quit IRC
148 2018-08-27T09:37:35  *** Zenton has joined #bitcoin-core-dev
149 2018-08-27T09:43:08  *** Victorsueca has quit IRC
150 2018-08-27T09:44:25  *** Victorsueca has joined #bitcoin-core-dev
151 2018-08-27T09:46:44  *** arubi has quit IRC
152 2018-08-27T09:47:07  *** arubi has joined #bitcoin-core-dev
153 2018-08-27T10:08:53  <sdaftuar> luke-jr: thanks for diving into the mmap leveldb issue. i was also confused about how those limits interact and was doing some measurements of mmap usage.
154 2018-08-27T10:09:49  <sdaftuar> with txindex off it looked like i'd never get more than 1000 mmaps, but with it on i'd get up to 2000
155 2018-08-27T10:11:18  <sdaftuar> one thing i was curious about was whether we have a good handle on the maximum number of non-mmap'ed files
156 2018-08-27T10:11:47  *** ken2812221_ has quit IRC
157 2018-08-27T10:13:58  <sdaftuar> to make sure there's not some remotely-triggerable leveldb behavior where we end up opening 1000 fd's and breaking select()
158 2018-08-27T10:19:04  <luke-jr> sdaftuar: looks like we'd be safe doing something like (4096 - 1) / (2 + txindex?1:0) for the file limit
159 2018-08-27T10:19:15  <luke-jr> (4096 only with the patched mmap limit ofc)
160 2018-08-27T10:19:37  *** vexbuy has joined #bitcoin-core-dev
161 2018-08-27T10:19:37  <luke-jr> in theory, we could probably burn fds on top of mmaps, but that might hurt performance more than it's worth
162 2018-08-27T10:19:47  <luke-jr> (since we know 64 * 3 was safe)
163 2018-08-27T10:20:42  *** vexbuy_ has joined #bitcoin-core-dev
164 2018-08-27T10:23:52  *** ken2812221_ has joined #bitcoin-core-dev
165 2018-08-27T10:24:11  *** vexbuy has quit IRC
166 2018-08-27T10:30:13  *** vexbuy has joined #bitcoin-core-dev
167 2018-08-27T10:33:46  *** vexbuy_ has quit IRC
168 2018-08-27T10:34:20  *** vexbuy_ has joined #bitcoin-core-dev
169 2018-08-27T10:38:21  *** vexbuy has quit IRC
170 2018-08-27T10:39:07  *** vexbuy has joined #bitcoin-core-dev
171 2018-08-27T10:41:48  *** vexbuy_ has quit IRC
172 2018-08-27T10:48:22  *** vexbuy_ has joined #bitcoin-core-dev
173 2018-08-27T10:51:59  *** vexbuy has quit IRC
174 2018-08-27T10:53:01  *** vexbuy has joined #bitcoin-core-dev
175 2018-08-27T10:56:18  *** vexbuy_ has quit IRC
176 2018-08-27T11:01:43  *** vexbuy_ has joined #bitcoin-core-dev
177 2018-08-27T11:05:54  *** vexbuy has quit IRC
178 2018-08-27T11:06:04  *** vexbuy has joined #bitcoin-core-dev
179 2018-08-27T11:08:40  *** vexbuy has quit IRC
180 2018-08-27T11:09:15  *** vexbuy_ has quit IRC
181 2018-08-27T11:13:42  *** chainhead has quit IRC
182 2018-08-27T11:13:52  *** plankers has quit IRC
183 2018-08-27T11:15:18  *** ken2812221_ has quit IRC
184 2018-08-27T11:23:45  *** kallisteiros has joined #bitcoin-core-dev
185 2018-08-27T11:25:35  *** zivl has quit IRC
186 2018-08-27T11:26:24  *** zivl has joined #bitcoin-core-dev
187 2018-08-27T11:36:08  *** Krellan has quit IRC
188 2018-08-27T11:37:11  *** Krellan has joined #bitcoin-core-dev
189 2018-08-27T11:38:58  *** drexl has joined #bitcoin-core-dev
190 2018-08-27T12:03:36  *** rafalcpp_ has joined #bitcoin-core-dev
191 2018-08-27T12:04:09  *** rafalcpp has quit IRC
192 2018-08-27T12:08:16  *** vexbuy has joined #bitcoin-core-dev
193 2018-08-27T12:32:06  *** atroxes has quit IRC
194 2018-08-27T12:38:23  *** atroxes has joined #bitcoin-core-dev
195 2018-08-27T12:42:23  *** setpill has quit IRC
196 2018-08-27T12:44:02  *** setpill has joined #bitcoin-core-dev
197 2018-08-27T12:44:26  *** itaseski has joined #bitcoin-core-dev
198 2018-08-27T13:00:53  *** Chris_Stewart_5 has joined #bitcoin-core-dev
199 2018-08-27T13:08:16  *** kallisteiros has quit IRC
200 2018-08-27T13:20:47  *** arubi has quit IRC
201 2018-08-27T13:27:39  *** arubi has joined #bitcoin-core-dev
202 2018-08-27T13:33:17  *** Krellan has quit IRC
203 2018-08-27T13:33:28  *** Krellan has joined #bitcoin-core-dev
204 2018-08-27T13:36:02  *** vexbuy has quit IRC
205 2018-08-27T13:36:40  *** vexbuy has joined #bitcoin-core-dev
206 2018-08-27T13:41:06  *** vexbuy has quit IRC
207 2018-08-27T13:42:00  *** promag has joined #bitcoin-core-dev
208 2018-08-27T13:42:01  *** arubi has quit IRC
209 2018-08-27T13:42:53  *** intcat has quit IRC
210 2018-08-27T13:45:27  *** intcat has joined #bitcoin-core-dev
211 2018-08-27T13:45:28  *** marcinja has joined #bitcoin-core-dev
212 2018-08-27T13:47:35  *** arubi has joined #bitcoin-core-dev
213 2018-08-27T13:53:34  *** Giszmo has joined #bitcoin-core-dev
214 2018-08-27T13:54:04  *** Giszmo has quit IRC
215 2018-08-27T13:57:32  <jnewbery> Review beg for #14023. It's not urgent, but it requires frequent rebase and it should be a very easy, mechanical review.
216 2018-08-27T13:57:34  <gribble> https://github.com/bitcoin/bitcoin/issues/14023 | Remove accounts rpcs by jnewbery · Pull Request #14023 · bitcoin/bitcoin · GitHub
217 2018-08-27T13:58:05  <jnewbery> It'd be nice to get a critical number of people review it once than have lots of re-ACKs as it gets rebased
218 2018-08-27T13:58:10  *** vexbuy has joined #bitcoin-core-dev
219 2018-08-27T13:58:37  <jnewbery> wumpus: not sure if it makes the bar for high priority, but if you think it does, please add it :)
220 2018-08-27T14:01:31  <wumpus> jnewbery: sure, added!
221 2018-08-27T14:05:21  *** phwalkr has joined #bitcoin-core-dev
222 2018-08-27T14:10:16  *** kallisteiros has joined #bitcoin-core-dev
223 2018-08-27T14:11:45  *** atroxes_ has joined #bitcoin-core-dev
224 2018-08-27T14:12:16  *** setpill has quit IRC
225 2018-08-27T14:12:54  *** atroxes has quit IRC
226 2018-08-27T14:12:55  *** atroxes_ is now known as atroxes
227 2018-08-27T14:13:39  *** arubi has quit IRC
228 2018-08-27T14:14:21  *** setpill has joined #bitcoin-core-dev
229 2018-08-27T14:22:55  *** michaelsdunn1 has joined #bitcoin-core-dev
230 2018-08-27T14:25:17  *** arubi has joined #bitcoin-core-dev
231 2018-08-27T14:28:56  *** vexbuy_ has joined #bitcoin-core-dev
232 2018-08-27T14:32:08  *** vexbuy has quit IRC
233 2018-08-27T14:32:25  *** Rootsudo has joined #bitcoin-core-dev
234 2018-08-27T14:40:14  <jnewbery> thanks!
235 2018-08-27T14:57:57  *** a5m0_ has quit IRC
236 2018-08-27T15:00:37  *** vexbuy_ has quit IRC
237 2018-08-27T15:01:12  *** vexbuy has joined #bitcoin-core-dev
238 2018-08-27T15:04:44  *** phwalkr has quit IRC
239 2018-08-27T15:06:35  *** ExtraCrispy has joined #bitcoin-core-dev
240 2018-08-27T15:10:31  *** SopaXorzTaker has quit IRC
241 2018-08-27T15:12:29  *** Guyver2 has joined #bitcoin-core-dev
242 2018-08-27T15:13:52  *** rafalcpp_ is now known as rafalcpp
243 2018-08-27T15:15:51  *** setpill has quit IRC
244 2018-08-27T15:26:36  *** nullptr| has quit IRC
245 2018-08-27T15:36:10  *** jhfrontz has joined #bitcoin-core-dev
246 2018-08-27T15:39:44  *** Victorsueca has quit IRC
247 2018-08-27T15:40:17  *** a5m0 has joined #bitcoin-core-dev
248 2018-08-27T15:40:58  *** Victorsueca has joined #bitcoin-core-dev
249 2018-08-27T15:45:15  *** Dizzle has joined #bitcoin-core-dev
250 2018-08-27T15:48:00  <jamesob> gmaxwell sipa: so are we thinking at this point that RSS is kind of a useless metric because clean ldb pages will just be pushed out under memory pressure?
251 2018-08-27T15:50:50  <sipa> jamesob: possibly
252 2018-08-27T15:50:54  <gmaxwell> jamesob: That is my view.
253 2018-08-27T15:52:03  <jamesob> do we want to have bitcoinperf count dirty pages as an additional metric?
254 2018-08-27T15:55:09  * wumpus waves farewell to account API (finally)
255 2018-08-27T15:55:41  <jnewbery> \o/
256 2018-08-27T15:55:50  <gmaxwell> jamesob: I think we should.
257 2018-08-27T15:55:53  <jnewbery> thanks wumpus!
258 2018-08-27T15:55:59  <sipa> wumpus: it will be bittersweet goodbye
259 2018-08-27T15:56:14  <sipa> i think i've developed somewhat of a stockholm syndrome with it
260 2018-08-27T15:56:27  <instagibbs> sipa, be healed
261 2018-08-27T15:56:43  <wumpus> yes that's what happens :(
262 2018-08-27T15:56:51  <instagibbs> I spent hours trying to write a sensible comment about balances with accounts in the code; failed. No love lost there.
263 2018-08-27T15:57:30  <wumpus> it was the same for me for getinfo
264 2018-08-27T15:58:36  <wumpus> but it just had to happen, after all that time, and I'm happy this particular knot is finally cut
265 2018-08-27T16:00:40  <jamesob> gmaxwell: so basically we want to poll `pmap -x $(pidof bitcoind) | tail -n 1 | tr -s ' ' | cut -d' ' -f 4` for a max value throughout ibd/reindex
266 2018-08-27T16:03:47  <gmaxwell> jamesob: I think that is what we want, yes.
267 2018-08-27T16:08:26  <jnewbery> The PR that was just merged removed the accounts RPCs. There's a bunch of dead code left behind that's removed in #13825. Feel free to review that if you want to help remove all traces of accounts. (non-urgent and I'm happy to keep rebasing)
268 2018-08-27T16:08:28  <gribble> https://github.com/bitcoin/bitcoin/issues/13825 | [wallet] Kill accounts by jnewbery · Pull Request #13825 · bitcoin/bitcoin · GitHub
269 2018-08-27T16:18:37  <gmaxwell> :(
270 2018-08-27T16:18:58  <gmaxwell> 14023 was merged without any actual answer to the locked fund issue.
271 2018-08-27T16:22:17  <sipa> what locked fund issue?
272 2018-08-27T16:22:45  <sipa> ah
273 2018-08-27T16:23:08  <sipa> a reference would be useful
274 2018-08-27T16:23:22  <rockhouse> gmaxwell: Would you say this definition is correct? "CTs don't use homomorphic encryption. They use "additively homomorphic commitments" which are, essentially, cryptographic proofs."
275 2018-08-27T16:25:19  <gmaxwell> sipa: I don't know if there is an issue on github anywhere. But as I pointed out, in the past people have attached accounts to addresses, gotten funds assigned to accounts, then found themselves unable to use send to address to send funds until they MOVEed the funds back.
276 2018-08-27T16:26:29  <gmaxwell> rockhouse: They're additively homorphic commitments, but I wouldn't say that the commitments themselves are "cryptographic proofs", they're just commitments. CT also uses cryptographic proofs.
277 2018-08-27T16:26:58  <rockhouse> thx
278 2018-08-27T16:28:16  <gmaxwell> I'm sorry I didn't see that jnewbery responded to me (weird, I thought I had looked yesturday too!)
279 2018-08-27T16:29:08  <sipa> gmaxwell: that's certainly possible with accounts based RPCs... there were some that did balance checks
280 2018-08-27T16:29:18  <sipa> gmaxwell: but now the whole concept of balances is gone
281 2018-08-27T16:29:33  <gmaxwell> What PR removed the balance checks?
282 2018-08-27T16:30:02  <sipa> accounts are gone.
283 2018-08-27T16:30:14  <sipa> without accounts, there are no more RPCs that do balance checks
284 2018-08-27T16:30:24  <sipa> because the concept of balance doesn't exist anymore
285 2018-08-27T16:30:56  <gmaxwell> Okay, what I'm asking is where were the balance checks removed? I looked at 13825 and I didn't see that.
286 2018-08-27T16:31:21  <sipa> well the account RPCs are gone, no?
287 2018-08-27T16:32:12  <gmaxwell> My concern is that people assigned funds to labels in the past, then load up one of their wallets in new code. So they don't need accounts rpcs to get in that state, if the backend code still checks balances.
288 2018-08-27T16:33:02  <sipa> i think we really need to see a report
289 2018-08-27T16:33:19  <sipa> because the only things that should be doing balance checks is accounts RPCs
290 2018-08-27T16:33:24  <sipa> even in the past
291 2018-08-27T16:34:41  <gmaxwell> There are people in my IRC logs reporting issues sending with sendtoaddress then luke telling them to move and then them saying it worked.
292 2018-08-27T16:38:40  <sipa> that seems like a very serious bug, if true
293 2018-08-27T16:38:46  <sipa> first time i hear about it, though
294 2018-08-27T16:41:12  <gmaxwell> "sendtoaddress only subtracts the amount from the default account,"
295 2018-08-27T16:41:40  <jnewbery> what are you quoting?
296 2018-08-27T16:42:06  <gmaxwell> IRC logs.
297 2018-08-27T16:42:31  <gmaxwell> If that changed (or was never the case) OKAY, but I also thought it was the case, and I don't remember a PR changing it.
298 2018-08-27T16:43:29  <sipa> sendtoaddress subtracts the default account, but doesn't check its balance
299 2018-08-27T16:43:49  <sipa> it doesn't do anything with accounts at all
300 2018-08-27T16:43:56  <sipa> as balances are just defined implicitly
301 2018-08-27T16:49:28  <gmaxwell> it might be the case that the reporters were actually using sendmany, which did check, but which has since had it removed.
302 2018-08-27T16:49:40  <gmaxwell> I found the PR disabling it in sendmany.
303 2018-08-27T16:52:51  <sipa> right
304 2018-08-27T16:54:26  <gmaxwell> and/or people have been uselessly telling people to use move when their real problem was something else.
305 2018-08-27T16:54:37  <gmaxwell> (like needing to wait for unconfirmed funds to confirm)
306 2018-08-27T16:56:20  <jnewbery> yes, sendmany would call getLegacyBalance() and earlier getAccountBalance(). senttoaddress has always used getBalance(), which doesn't check accounts/labels at all
307 2018-08-27T16:56:43  <jnewbery> sendmany now uses getLegacyBalance() without an account argument
308 2018-08-27T16:57:40  <jnewbery> if 14023 was done right, then none of the RPCs should now know anything about account balances at all
309 2018-08-27T17:06:49  *** promag has quit IRC
310 2018-08-27T17:22:22  *** Rootsudo has quit IRC
311 2018-08-27T17:26:03  *** Rootsudo has joined #bitcoin-core-dev
312 2018-08-27T17:26:42  *** kallisteiros has quit IRC
313 2018-08-27T17:27:43  *** kallisteiros has joined #bitcoin-core-dev
314 2018-08-27T17:30:30  *** karelb has quit IRC
315 2018-08-27T17:30:52  *** karelb has joined #bitcoin-core-dev
316 2018-08-27T17:40:01  *** Zenton has quit IRC
317 2018-08-27T17:41:00  <phantomcircuit> so i notice select has a 50ms timeout, that seems kind of short given all it's only purpose is to consume pnode->vSend when the initial attempt failed
318 2018-08-27T17:54:12  *** SopaXorzTaker has joined #bitcoin-core-dev
319 2018-08-27T17:58:02  <gmaxwell> I assume it was made short so that it wouldn't block sending new messages.
320 2018-08-27T18:00:32  <phantomcircuit> gmaxwell, send() is called on the full buffer before anything is added to pnode->vSend
321 2018-08-27T18:01:00  *** Amuza has joined #bitcoin-core-dev
322 2018-08-27T18:03:20  <phantomcircuit> actually wait this is still doing the thing of calling send() for each message
323 2018-08-27T18:06:49  <gmaxwell> oh we didn't fix the thing where the header and payload go out in seperate packets? :(
324 2018-08-27T18:07:55  *** str4d has joined #bitcoin-core-dev
325 2018-08-27T18:08:20  <phantomcircuit> gmaxwell, i think that's fixed i mean sipa changed this in 2013 from the buffer per connection to a list of messages per connection
326 2018-08-27T18:08:24  <phantomcircuit> and i cant remember why
327 2018-08-27T18:08:50  <phantomcircuit> doesn't change the optimistic send logic meaningfully though
328 2018-08-27T18:09:00  <phantomcircuit> so im still wondering about the 50ms timeout
329 2018-08-27T18:09:53  *** str4d_ has joined #bitcoin-core-dev
330 2018-08-27T18:10:43  *** str4d_ has quit IRC
331 2018-08-27T18:10:57  *** hebasto has joined #bitcoin-core-dev
332 2018-08-27T18:11:03  *** str4d_ has joined #bitcoin-core-dev
333 2018-08-27T18:12:08  *** str4d_ has quit IRC
334 2018-08-27T18:12:56  *** str4d has quit IRC
335 2018-08-27T18:13:21  *** str4d has joined #bitcoin-core-dev
336 2018-08-27T18:21:10  *** str4d has quit IRC
337 2018-08-27T18:33:05  *** Dizzle has quit IRC
338 2018-08-27T18:33:59  *** str4d has joined #bitcoin-core-dev
339 2018-08-27T18:40:35  *** Victorsueca has quit IRC
340 2018-08-27T18:41:44  *** Victorsueca has joined #bitcoin-core-dev
341 2018-08-27T18:44:50  <jonasschnelli> gmaxwell: I can't follow your comment here: https://github.com/bitcoin/bitcoin/pull/14049#discussion_r213027446
342 2018-08-27T18:45:28  <jonasschnelli> You mean a 32byte pubkey with missing first byte to test for?
343 2018-08-27T18:53:55  <gmaxwell> BIP151 sends a 32 byte pubkey, not a 33 byte one. Right?  so a test that gives a junk value to the first byte doesn't actually test any case that could be triggered by BIP151. (though it's fine to test that too)  The test I'm going for is an x value not on the curve, since failing to check that is a common implementation bug in ECDH implementations.
344 2018-08-27T18:54:30  *** Krellan has quit IRC
345 2018-08-27T18:57:15  *** Rootsudo has quit IRC
346 2018-08-27T18:57:46  *** Rootsudo has joined #bitcoin-core-dev
347 2018-08-27T18:57:55  *** str4d has quit IRC
348 2018-08-27T19:02:58  *** Rootsudo has quit IRC
349 2018-08-27T19:13:55  *** promag has joined #bitcoin-core-dev
350 2018-08-27T19:14:48  *** harrymm_ has quit IRC
351 2018-08-27T19:15:50  *** harrymm_ has joined #bitcoin-core-dev
352 2018-08-27T19:18:30  *** promag has quit IRC
353 2018-08-27T19:20:01  <gmaxwell> jonasschnelli: make sense now?
354 2018-08-27T19:20:23  <jonasschnelli> Ah. I see! Will fix thanks gmaxwell
355 2018-08-27T19:21:50  *** promag has joined #bitcoin-core-dev
356 2018-08-27T19:25:01  <jonasschnelli> gmaxwell, sipa: what was the conclusion on the inefficiency of the ChaCha20Poly1305@openssh AEAD?
357 2018-08-27T19:25:13  <jonasschnelli> (currently benchmarking)
358 2018-08-27T19:26:17  *** kallisteiros has quit IRC
359 2018-08-27T19:29:51  <jonasschnelli> CHACHA20POLY1305AEAD_BIG, 5, 340, 3.68279, 0.00215035, 0.00219169, 0.00216025
360 2018-08-27T19:29:52  <jonasschnelli> CHACHA20POLY1305AEAD_SMALL, 5, 250000, 1.08673, 8.51516e-07, 8.93585e-07, 8.61119e-07
361 2018-08-27T19:29:52  <jonasschnelli> HASH256_BIG, 5, 340, 3.81384, 0.00222589, 0.00226436, 0.00224086
362 2018-08-27T19:29:52  <jonasschnelli> HASH256_SMALL, 5, 250000, 1.1305, 8.96669e-07, 9.15482e-07, 9.03866e-07
363 2018-08-27T19:30:01  <jonasschnelli> (BIG = 1MB, SMALL: 256bytes)
364 2018-08-27T19:30:24  <jonasschnelli> (and this is SHA256 asm vs non-asm ChaCha)
365 2018-08-27T19:31:11  <jonasschnelli> But I agree that the chacha20 rounds (one for the poly key, one for the length) is not very very efficient...
366 2018-08-27T19:31:28  <jonasschnelli> But compared to dbl-SHA256 is probably faster
367 2018-08-27T19:31:34  <jonasschnelli> (once optimized)
368 2018-08-27T19:33:48  *** Zenton has joined #bitcoin-core-dev
369 2018-08-27T19:39:15  <gmaxwell> jonasschnelli: so there is an obvious optimization to get the length key from bytes 33-36 of the polykey chacha run.  the only apparent argument against doing that is that its less compatible with existing constructs.  it sounds like openssl didn't do it that way so they could just call a function that implemented that TLS style AEAD.
370 2018-08-27T19:39:31  *** WhatDaMath has joined #bitcoin-core-dev
371 2018-08-27T19:40:18  <jonasschnelli> gmaxwell: Yes. That would work,... but would reduce the AAD len to 4 (currently flexible, but not required in our case).
372 2018-08-27T19:40:32  <jonasschnelli> My only worry is that we start cooking our own crypto...
373 2018-08-27T19:45:28  <jonasschnelli> There is still the option to use AES-CGM *duck*
374 2018-08-27T19:46:17  <jonasschnelli> Though non NI implementation will bring back cache-time attacks (or non efficient implementations)
375 2018-08-27T19:46:37  <jonasschnelli> I guess ChaCha20 is preferable if we can't guarantee for AES-NI (which I guess we can't)
376 2018-08-27T19:46:55  <gmaxwell> jonasschnelli: for AES-GCM there are very fast non-NI that are cache time fine, for machines with SIMD.
377 2018-08-27T19:47:04  <gmaxwell> It's only CBC mode that really stinks without NI.
378 2018-08-27T19:48:18  <jonasschnelli> Okay. Yeah. Unsure then if switching "back" (we once discussed AES-CGM vs ChaChaPoly back in 2015) would make sense then.
379 2018-08-27T19:49:03  <gmaxwell> basically chacha20 is the fastest on ARM by a wide margin.  Basically tied with non-NI on x86 if you use SIMD, and somewhat worse than NI GCM.  Our reasoning, which I assume is the same as others using chacha-- is that it makes sense to optimize for the least powerful computers.
380 2018-08-27T19:49:47  <gmaxwell> I think thats still true, though the 3x runs of chacha for small messages shifts the equation further in favor of AES simply because most of our messages are pretty small.
381 2018-08-27T19:50:33  <jonasschnelli> Yes. Indeed. I think optimizing for an AAD-len of 4 and reducing thus to 2x could make sense then
382 2018-08-27T19:50:39  <gmaxwell> jonasschnelli: I'm not sure why it would reduce the AAD len?  poly1305 uses 32 bytes of key regardless of the auth token size.
383 2018-08-27T19:51:41  <jonasschnelli> The 2nd rund is currently to derive the 2nd ChaCha20 key for the AAD encryption (the 4 byte length in our case).
384 2018-08-27T19:52:18  *** promag has quit IRC
385 2018-08-27T19:52:22  <gmaxwell> oh I see what you're saying the length of the lenghts.  Well there are 64 bytes available.
386 2018-08-27T19:52:40  <gmaxwell> 32 get used for poly1305's key, leaving 32 that get throw away right now.
387 2018-08-27T19:52:57  <jonasschnelli> Yes. We can set the max AAD size to 32
388 2018-08-27T19:53:51  <jonasschnelli> But I'm not a cryptographer and I'm not sure if this would require some analysis first (isn't it basically a new AEAD)?
389 2018-08-27T19:55:40  <jonasschnelli> Where I'm unsure: currently, there are two ChaCha20 contexts (two keys), one context is for the actual stream _AND_ the poly1305 key, the 2nd context for the AAD (the payload length)... and now
390 2018-08-27T19:56:10  <jonasschnelli> We would drop the second context,... and take the AAD key from the main context
391 2018-08-27T19:56:46  <jonasschnelli> Reducing back to 32bytes of required key material...
392 2018-08-27T19:57:23  <gmaxwell> At least the discussion I linked to before said that they designed it that way so they could use an unmodified implementation of the TLS AEAD to implement the openssl one. (this seems dumb to me, but its the rational they gave)
393 2018-08-27T19:57:41  <jonasschnelli> Which sounds _not_ after a problem... but again, IF we do that, probably good if a cryptograph-expert takes a closer look at this
394 2018-08-27T19:57:58  <gmaxwell> the defense against the length oracle just depends on you not being able to shift the input.
395 2018-08-27T19:58:16  <jonasschnelli> I see...
396 2018-08-27T19:58:59  <jonasschnelli> Yes. Indeed. Reducing the performance in our system due to legacy stuff from the AEAD-origin system seems indeed inefficient
397 2018-08-27T19:59:37  <gmaxwell> also, does the openssl usage actually send the full 128 bit poly1305 tag?  I had thought it used a 96 bit tag.
398 2018-08-27T19:59:40  <jonasschnelli> To bad this is not an optimisation we can do later. :/
399 2018-08-27T19:59:51  <jonasschnelli> gmaxwell: you mean openssh?
400 2018-08-27T20:00:11  <gmaxwell> well thats why I'm looking at it now. What got me going down this path was checking to see if anything would get in the way of getting the most out of AVX2/SSE2 in the future.
401 2018-08-27T20:00:13  <gmaxwell> jonasschnelli: ye.
402 2018-08-27T20:00:15  <gmaxwell> yes.
403 2018-08-27T20:01:12  <jonasschnelli> The openssh function produces a 128bit tag,... but not if they wired it over (probably)
404 2018-08-27T20:03:09  *** farmerwampum has quit IRC
405 2018-08-27T20:03:46  <gmaxwell> There is another change we might want to consider, but I wanted to research if anyone else has proposed it-- that we should chain the unused part of the tag from each message to the AAD (additional authenticated data) in the next message, so we get the full tag's worth of protection, but then noticed we were using the whole tag output. When I thought it was truncated.
406 2018-08-27T20:05:37  <jonasschnelli> gmaxwell: with the goal of saving 4 bytes per message, right?
407 2018-08-27T20:05:55  <jonasschnelli> (in case we trunc down to 96bits)
408 2018-08-27T20:06:22  *** plankers has joined #bitcoin-core-dev
409 2018-08-27T20:08:37  <jonasschnelli> gmaxwell: where I'm also unsure – in case we would not reduce the AEAD to use a single 256bit key: the length is pretty much known (known plaintext), right now, in the AEAD@openssh, compromising the second, AAD key, is eventually not too critical
410 2018-08-27T20:08:54  <jonasschnelli> *in case we would reduce
411 2018-08-27T20:13:05  <gmaxwell> There is nothing wrong with the lengths being known. 99% of the protocol is known. It's very obvious from traffic analysis when pings are sent, for example. And the sequences of messages we send at the start are the same each time.
412 2018-08-27T20:14:32  <jonasschnelli> Yes. That true,...
413 2018-08-27T20:15:09  <gmaxwell> jonasschnelli: yes, making the tag shorter reduces overhead.  Right now for short messages like a tx inv, almost half the bandwidth is used by overhead (the 4 byte length and the 16 byte tag).  If I was imagining it and no one else is truncating the tags, then I guess we shouldn't just to avoid the security analysis.
414 2018-08-27T20:15:58  <jonasschnelli> I think its def. worth to see if we can.
415 2018-08-27T20:17:48  <jonasschnelli> Compared to the v1 protocol (I agree that comparing is not the best metric), the encryption-overhead is not too bad. -4byte magic, -4bytes SHA-chksm,-~8bytes cmd, +16bytes MAC + 4byte length
416 2018-08-27T20:18:01  *** SopaXorzTaker has quit IRC
417 2018-08-27T20:20:26  *** kallisteiros has joined #bitcoin-core-dev
418 2018-08-27T20:23:30  <gmaxwell> still ends up being about twice as much. Which is not unacceptable.  It seems I was dreaming about the 96-bit tag, I don't see anyone using poly1305 with truncated tags.
419 2018-08-27T20:35:19  <jonasschnelli> twice as much? v1 versus v2? v1 inv: 24(hdr)+36(msg) == 60 bytes    < versus >. v2: 4(outter-len)+4(cmd)+4(inner-len)+36(msg)+16MAC == 64
420 2018-08-27T20:35:32  <jonasschnelli> So an inv is 4 bytes longer
421 2018-08-27T20:36:26  <jonasschnelli> Also,... multiple messages can be combined into a single encrypted package
422 2018-08-27T20:36:51  <jonasschnelli> reducing the "weight" of the AAD (payload length) and the MAC tag
423 2018-08-27T20:38:23  *** Krellan has joined #bitcoin-core-dev
424 2018-08-27T20:38:39  <gmaxwell> oh I forgot we could do that, well that basically kills any overhead concerns I had.
425 2018-08-27T20:39:31  *** Krellan has quit IRC
426 2018-08-27T20:40:28  <gmaxwell> on your message above, an INV is 4+12+1+36+4 on the wire today.
427 2018-08-27T20:41:03  <gmaxwell> The twice comment was 4+4 vs 4+16 (-command savings) ~= 2x more overhead data.
428 2018-08-27T20:41:43  <jonasschnelli> didn't you miss the message size in your calc?
429 2018-08-27T20:42:01  *** jnewbery has quit IRC
430 2018-08-27T20:42:05  *** Krellan has joined #bitcoin-core-dev
431 2018-08-27T20:42:43  <jonasschnelli> gmaxwell: AFAIK a v1 header is 24 bytes (4+12+!4!+4)
432 2018-08-27T20:42:49  <sipa> yup
433 2018-08-27T20:43:43  <gmaxwell> magic, command, (something), checksum, whats something?
434 2018-08-27T20:44:06  <sipa> length
435 2018-08-27T20:44:09  <jonasschnelli> (something) == 4 bytes
436 2018-08-27T20:44:12  <jonasschnelli> (length)
437 2018-08-27T20:44:24  <jonasschnelli> (non variable)
438 2018-08-27T20:45:18  <gmaxwell> and in the new protocol we made the lengths implicit to the command type?
439 2018-08-27T20:45:57  <jonasschnelli> 10x v1 invs result in 600 bytes where a 10x v2 inv (packed) could result in 388 bytes
440 2018-08-27T20:46:16  <jonasschnelli> gmaxwell: the v2 protocol uses varstr for the command rather then fixed 12 bytes
441 2018-08-27T20:46:47  <jonasschnelli> but that reduction above comes at the cost of combining 10invs (time lag)
442 2018-08-27T20:46:49  <gmaxwell> I know that the command is variable in v2. But do we have a second length then now?
443 2018-08-27T20:47:00  <gmaxwell> jonasschnelli: if you're going to combine 10 invs, you'd combine them in the inv message. :)
444 2018-08-27T20:47:00  <jonasschnelli> inner length, outer length
445 2018-08-27T20:47:13  <jonasschnelli> gmaxwell: oh. yes. :/
446 2018-08-27T20:47:32  <jonasschnelli> outer length is the encrypted payload length that can combine multiple messages
447 2018-08-27T20:47:48  <jonasschnelli> (where the inner length is the message payload length)
448 2018-08-27T20:48:14  <gmaxwell> right so it's not like we saved coding that length.
449 2018-08-27T20:49:11  <jonasschnelli> you mean by using varlen for the inner message size?
450 2018-08-27T20:49:14  <jonasschnelli> *varint
451 2018-08-27T20:49:17  <gmaxwell> No.
452 2018-08-27T20:49:27  <gmaxwell> the 16 byte tag  replaces magic and the checksum but the additional length at the encryption layer is just additional.
453 2018-08-27T20:49:47  <jonasschnelli> yes
454 2018-08-27T20:50:09  <jonasschnelli> Its somehow required since you first need to check the MAC before processing the inner messages
455 2018-08-27T20:51:18  <gmaxwell> Sure. from the above I thought you were saying that it was the only length we have. generally. But sadly we can't use implicit lengths because we need to be able to handle unknown messages.
456 2018-08-27T20:51:41  <jonasschnelli> yeah
457 2018-08-27T20:52:21  <gmaxwell> Whats the maximum size of an encrypted message? the outer length. Just the same as the maximum message size?
458 2018-08-27T20:52:30  <jonasschnelli> 2^31
459 2018-08-27T20:52:44  <jonasschnelli> (bit 32 is used for the rekey)
460 2018-08-27T20:52:44  <gmaxwell> I sure hope not, else it'll be awfully easy to exhaust your ram. :P
461 2018-08-27T20:53:18  *** Krellan has quit IRC
462 2018-08-27T20:53:25  <gmaxwell> I know what the length can encode, but there needs to be a smaller limit in the implementation or otherwise there is a trivial memory exhaustion DOS.
463 2018-08-27T20:53:31  <jonasschnelli> gmaxwell: of course there is the MAX_SIZE check before decrypting the payload
464 2018-08-27T20:54:11  <jonasschnelli> https://github.com/bitcoin/bitcoin/pull/14032/files#diff-35820b8a93cd261189cd3ea9a4e611e1R45
465 2018-08-27T20:54:14  <gmaxwell> So, if it's the same MAX_SIZE as messages, does this end up effectively reducing MAX_SIZE by a few bytes?
466 2018-08-27T20:55:02  <jonasschnelli> gmaxwell: Yes. I guess that would require fixing once we combine multiple messages (or expect it will be combined).
467 2018-08-27T20:55:11  <jonasschnelli> So,... I guess it needs fixing now
468 2018-08-27T20:55:43  <jonasschnelli> It's indeed missing bytes,... especially if we assume combined messages into a single enc package
469 2018-08-27T20:55:52  <jonasschnelli> (will fix)
470 2018-08-27T20:56:07  <gmaxwell> well yes/no, so max_size is much larger than any message we allow today.
471 2018-08-27T20:56:28  <gmaxwell> so it's not actually a limitation. But I think it actually needs to be set to the largest message plus a few bytes for the overhead.
472 2018-08-27T20:56:33  <gmaxwell> I think as you have it now thats probably a trivial DOS attack, e.g. I can make your node use 32mb * npeers ram... so 4GB additional usage in the default configuration.
473 2018-08-27T20:57:13  <jonasschnelli> hmm...
474 2018-08-27T20:57:24  <sipa> use 3 byte lengths :)
475 2018-08-27T20:58:02  <gmaxwell> 3 byte is kinda limiting, I really wanted to suggest that previously. Realistically if we ever had messages that needed to be bigger we'd have to break them up for processing reasons.
476 2018-08-27T20:58:02  <jonasschnelli> How does the current network code prevents that (there is the same MAX_SIZE check)?
477 2018-08-27T20:58:27  *** Amuza has quit IRC
478 2018-08-27T20:58:27  <sipa> the serialization code will only reallocate in blobs of 5 MB as the data is read
479 2018-08-27T20:58:35  <gmaxwell> jonasschnelli: there are size limits on the particular messages.
480 2018-08-27T20:58:51  <jonasschnelli> ah... we don't need to auth before process there, right
481 2018-08-27T20:59:00  <sipa> it never allocates more than 5 MB in addition to what it has already read from the stream
482 2018-08-27T20:59:49  <gmaxwell> Like if we were to support (say) 40MB blocks pratically we should have messages that split the blocks into chunks, because transfering and allocating 40MB at a time would be kind of absurd.
483 2018-08-27T21:00:01  <gmaxwell> so 3byte lengths might actually be reasonable.
484 2018-08-27T21:00:32  <jonasschnelli> ack
485 2018-08-27T21:00:59  <jonasschnelli> But due to the rekey approach, it would be 2^23
486 2018-08-27T21:02:25  <jonasschnelli> But I guess we don't want more then that per encryption package that needs auth (doesn't mean we don't want larger messages)
487 2018-08-27T21:03:11  * jonasschnelli needs to go afk
488 2018-08-27T21:09:16  *** plankers has quit IRC
489 2018-08-27T21:24:30  *** plankers has joined #bitcoin-core-dev
490 2018-08-27T21:25:46  *** Krellan has joined #bitcoin-core-dev
491 2018-08-27T21:28:29  *** Rootsudo has joined #bitcoin-core-dev
492 2018-08-27T21:44:29  *** rex4539 has joined #bitcoin-core-dev
493 2018-08-27T21:46:42  *** ExtraCrispy has quit IRC
494 2018-08-27T22:00:59  *** contrapumpkin has joined #bitcoin-core-dev
495 2018-08-27T22:01:57  *** copumpkin has quit IRC
496 2018-08-27T22:19:00  *** Giszmo has joined #bitcoin-core-dev
497 2018-08-27T22:28:18  *** shesek has joined #bitcoin-core-dev
498 2018-08-27T22:36:25  *** hebasto has quit IRC
499 2018-08-27T22:41:08  *** Chris_Stewart_5 has quit IRC
500 2018-08-27T22:43:36  *** Chris_Stewart_5 has joined #bitcoin-core-dev
501 2018-08-27T22:45:16  *** plankers has quit IRC
502 2018-08-27T22:58:37  *** rex4539 has quit IRC
503 2018-08-27T23:05:42  *** opdenkamp has quit IRC
504 2018-08-27T23:06:45  *** michaelsdunn1 has quit IRC
505 2018-08-27T23:07:03  *** michaelsdunn1 has joined #bitcoin-core-dev
506 2018-08-27T23:07:09  *** michaelsdunn1 has quit IRC
507 2018-08-27T23:08:53  *** plankers has joined #bitcoin-core-dev
508 2018-08-27T23:16:03  *** Guyver2 has quit IRC
509 2018-08-27T23:19:45  *** opdenkamp has joined #bitcoin-core-dev
510 2018-08-27T23:20:02  *** d9b4bef9 has quit IRC
511 2018-08-27T23:21:07  *** d9b4bef9 has joined #bitcoin-core-dev
512 2018-08-27T23:24:32  *** GoldenBear has quit IRC
513 2018-08-27T23:24:32  *** comboy has quit IRC
514 2018-08-27T23:24:36  *** opdenkamp has quit IRC
515 2018-08-27T23:26:23  *** GoldenBear has joined #bitcoin-core-dev
516 2018-08-27T23:27:22  *** BGL has quit IRC
517 2018-08-27T23:29:18  *** opdenkamp has joined #bitcoin-core-dev
518 2018-08-27T23:43:08  *** Emcy has quit IRC
519 2018-08-27T23:44:19  *** Emcy has joined #bitcoin-core-dev
520 2018-08-27T23:53:26  *** itaseski has quit IRC