1 2018-08-25T00:00:16  <gmaxwell> right. well there is also no need to discard bytes from K_2 but it does that too. the performance hit is especially gratitious for lengths.
  2 2018-08-25T00:02:00  *** goatpig has joined #bitcoin-core-dev
  3 2018-08-25T00:03:57  *** intcat has quit IRC
  4 2018-08-25T00:04:33  *** AaronvanW has quit IRC
  5 2018-08-25T00:04:44  <gmaxwell> sipa: oh geesh, for every message we do _another_ chacha20 run to derrive the poly key.
  6 2018-08-25T00:05:30  <gmaxwell> so encrypting a single small message requires 3 runs of the chacha20 function, one to encrypt the length, one to establsh the poly1305 key, and one to encrypt the payload.
  7 2018-08-25T00:06:02  <gmaxwell> this seems pants on head stupid.
  8 2018-08-25T00:06:28  *** intcat has joined #bitcoin-core-dev
  9 2018-08-25T00:07:42  <gmaxwell> The polykey needs to be per packet for poly1305's requirements, so I suppose it's only throwing out 32 bytes of chacha output.
 10 2018-08-25T00:14:09  <gmaxwell> https://www.ietf.org/mail-archive/web/secsh/current/msg01224.html  seems other people have suggested combining the poly1305 chacha block with the length encryption.
 11 2018-08-25T00:14:46  <gmaxwell> and it seems that they didn't so that you could just use a RFC5116 implementation of it.
 12 2018-08-25T00:14:49  * gmaxwell cries
 13 2018-08-25T00:16:32  *** plankers has joined #bitcoin-core-dev
 14 2018-08-25T00:24:32  <gmaxwell> So encrypting a 12 byte message will run on the order of 109 cycles/byte... which means that for small messages a straighforward implementation of AES-GCM would likely be faster, even on hardware without AES instructions.
 15 2018-08-25T00:24:46  <gmaxwell> (109 cycles/byte for the chacha20 part alone)
 16 2018-08-25T00:28:01  *** d9b4bef9 has quit IRC
 17 2018-08-25T00:29:08  *** d9b4bef9 has joined #bitcoin-core-dev
 18 2018-08-25T00:30:02  *** d9b4bef9 has quit IRC
 19 2018-08-25T00:30:39  *** IGHOR has quit IRC
 20 2018-08-25T00:31:09  *** d9b4bef9 has joined #bitcoin-core-dev
 21 2018-08-25T00:31:29  <sipa> gmaxwell: but what is the average message length for us?
 22 2018-08-25T00:32:09  *** IGHOR has joined #bitcoin-core-dev
 23 2018-08-25T00:34:20  <sipa> it seems we don't keep stats on message counts
 24 2018-08-25T00:42:24  *** promag has quit IRC
 25 2018-08-25T00:59:24  *** meshcollider_ has quit IRC
 26 2018-08-25T01:03:10  *** profmac has joined #bitcoin-core-dev
 27 2018-08-25T01:09:43  *** promag has joined #bitcoin-core-dev
 28 2018-08-25T01:14:54  *** promag has quit IRC
 29 2018-08-25T01:24:08  <cfields> luke-jr: hmm?
 30 2018-08-25T01:24:44  <cfields> luke-jr: if qt's copy is missing the files that need patching, what's to patch?
 31 2018-08-25T01:36:21  *** Chris_Stewart_5 has quit IRC
 32 2018-08-25T01:47:35  *** Chris_Stewart_5 has joined #bitcoin-core-dev
 33 2018-08-25T01:55:28  <luke-jr> cfields: libpng has optimisations for ARM and POWER in separate files missing in Qt, but Qt's copy of the normal files still tries to link them
 34 2018-08-25T01:57:38  <cfields> luke-jr: armhf/aarch64 build fine, what's different about power?
 35 2018-08-25T01:58:03  <luke-jr> cfields: I don't know how ARM works
 36 2018-08-25T01:58:05  <cfields> (not arguing, just trying to understand)
 37 2018-08-25T01:59:54  <cfields> luke-jr: anyway, breaking out libpng is fine with me. IIRC I didn't do that because it requires zlib, as does qt, so that would've meant 2 copies of zlib. But we've since broken zlib out anyway I believe.
 38 2018-08-25T02:00:31  <luke-jr> yeah
 39 2018-08-25T02:01:03  <cfields> luke-jr: while you're at it, feel free to flip -qt-jpeg to -disable-jpeg too
 40 2018-08-25T02:01:16  <cfields> something like those options, anyway
 41 2018-08-25T02:01:46  <cfields> I think we've had no need for jpegs for a long time
 42 2018-08-25T02:04:56  <luke-jr> did we ever? O.o
 43 2018-08-25T02:05:32  <cfields> pretty sure we had some at some point
 44 2018-08-25T02:10:58  <midnightmagic> ‰/w 39
 45 2018-08-25T02:11:46  <cfields> luke-jr: https://github.com/bitcoin/bitcoin/commit/f9124587ccea723dbd743e3877a7071fbb6c5732
 46 2018-08-25T02:12:20  <cfields> 0.8, heh
 47 2018-08-25T02:16:47  <gmaxwell> sipa: the most common message by far is transaction inv.
 48 2018-08-25T02:18:53  <gmaxwell> sipa: it's just so weird that it uses 3 chacha runs, the poly1305 run has 32 bytes totally unused.
 49 2018-08-25T02:28:35  *** Krellan has quit IRC
 50 2018-08-25T02:31:42  *** davex__ has quit IRC
 51 2018-08-25T02:32:53  *** pierre_rochard has joined #bitcoin-core-dev
 52 2018-08-25T02:41:19  *** jimpo has joined #bitcoin-core-dev
 53 2018-08-25T02:42:46  *** promag has joined #bitcoin-core-dev
 54 2018-08-25T02:48:24  *** promag has quit IRC
 55 2018-08-25T03:32:57  *** Chris_Stewart_5 has quit IRC
 56 2018-08-25T03:55:47  *** Chris_Stewart_5 has joined #bitcoin-core-dev
 57 2018-08-25T03:55:57  *** Rootsudo has joined #bitcoin-core-dev
 58 2018-08-25T04:01:27  *** Jmabsd has joined #bitcoin-core-dev
 59 2018-08-25T04:01:47  <Jmabsd> wait, so Bitcoin has the tendency to print (256 & 160bit) hashes in *reverse* order, right - block hashes, transaction hashes and merkle root hashes.
 60 2018-08-25T04:02:12  <Jmabsd> What about pubkey hashes (20B), pubkeys (32B) and signatures (64B) -   are those printed in normal or reverse byte order?    so, I have a P2SH pubkey script, say.  in there is a 20B hash of my redeemscript, right.  when I use Bitcoin Core's script disassembly function, will it print that hash in byte or normal order?   i mean there is an outer extent to what Core prints in reverse order - for instance, binary transaction dumps (in hex) are in
 61 2018-08-25T04:02:14  <Jmabsd> *normal* order, not reverse.
 62 2018-08-25T04:06:46  <sipa> Jmabsd: that's just printing the bytes one by one
 63 2018-08-25T04:07:08  <sipa> it's only when a hash is interested as a number the printing gets reversed
 64 2018-08-25T04:07:46  <sipa> because the bytes are interpreted as little-endian number, but then printed in big endian for human consumption (humans want to see numbers in big endian)
 65 2018-08-25T04:07:58  <sipa> but a script is a number
 66 2018-08-25T04:08:05  <luke-jr> isn't*
 67 2018-08-25T04:08:11  <Jmabsd> gotcha.
 68 2018-08-25T04:08:45  <sipa> *indeed, isn't
 69 2018-08-25T04:11:16  <Jmabsd> aha. so let's see - if you print a hex dump of a signature (71/72/73B), that's not a hash and hence printed in normal order
 70 2018-08-25T04:11:37  <Jmabsd> a P2SH hash, for instance when printing the disassembly of a P2SH pubkey script - will the 20B hash there be printed in reverse ordeR?
 71 2018-08-25T04:12:06  <Jmabsd> also if a pubkey (32B) is printed out, could that ever be in reverse order?
 72 2018-08-25T04:13:02  <luke-jr> why don't you just try it and see? -.-
 73 2018-08-25T04:14:33  <sipa> Jmabsd: pubkeys are not 32 bytes, and they're not hashes
 74 2018-08-25T04:15:02  *** vexbuy has quit IRC
 75 2018-08-25T04:15:10  <Jmabsd> sipa: so the hex printer for other byte structures are never printed in reverse orders.
 76 2018-08-25T04:15:17  <sipa> indeed
 77 2018-08-25T04:15:32  <sipa> only for things that are internally treated as numbers
 78 2018-08-25T04:15:33  <Jmabsd> but.. a P2SH 20B hash, that's a hash right. for printing purposes, is it considered a hash or a byte blob?
 79 2018-08-25T04:15:37  *** vexbuy has joined #bitcoin-core-dev
 80 2018-08-25T04:15:39  <sipa> nope!
 81 2018-08-25T04:15:48  <sipa> because the printer cannot know it is a hash
 82 2018-08-25T04:16:02  <sipa> you'd need to execute the script to know it is treated as such
 83 2018-08-25T04:16:12  <sipa> the script opcode is just "put some bytes on the stack"
 84 2018-08-25T04:17:48  <sipa> so, not reversed there
 85 2018-08-25T04:20:18  *** vexbuy has quit IRC
 86 2018-08-25T04:20:34  *** Jmabsd has quit IRC
 87 2018-08-25T04:21:20  *** Jmabsd has joined #bitcoin-core-dev
 88 2018-08-25T04:21:25  <Jmabsd> (sorry disconnect)
 89 2018-08-25T04:21:30  <Jmabsd> last, > interesting. except for the HD wallet root seed (160b=20B), there is no instance ever where a 20B hash e.g. in P2SH pubkeyscript, is printed in reverse order.
 90 2018-08-25T04:21:35  <Jmabsd> > sipa, right and when getting a disassembly printout in Bitcoin Core and related tools, those 20B:s are printed in normal order
 91 2018-08-25T04:28:18  *** vexbuy has joined #bitcoin-core-dev
 92 2018-08-25T04:32:02  *** d9b4bef9 has quit IRC
 93 2018-08-25T04:33:10  *** d9b4bef9 has joined #bitcoin-core-dev
 94 2018-08-25T04:40:18  *** Chris_Stewart_5 has quit IRC
 95 2018-08-25T05:36:53  *** promag has joined #bitcoin-core-dev
 96 2018-08-25T05:41:58  *** promag has quit IRC
 97 2018-08-25T05:49:07  *** vexbuy has quit IRC
 98 2018-08-25T06:07:11  *** vexbuy has joined #bitcoin-core-dev
 99 2018-08-25T06:16:30  <Jmabsd> the proper way to phrase Core's reversing policy is something like, "any hash that is not part of another binary blob or produced as script data, is hex-serialized in reverse byte order."
100 2018-08-25T06:17:07  <Jmabsd> i'd hope any hash values introduced in the future will not be reversed though.
101 2018-08-25T06:26:19  <sipa> i don't see why not
102 2018-08-25T06:26:32  <sipa> we've always treated hash outputs as numbers and printed them as such
103 2018-08-25T06:26:56  <sipa> if byte swapping is the hardest problem to deal with, i'm not very worried :)
104 2018-08-25T06:43:42  *** Jmabsd has quit IRC
105 2018-08-25T06:53:32  *** vexbuy has quit IRC
106 2018-08-25T06:54:08  *** vexbuy has joined #bitcoin-core-dev
107 2018-08-25T06:58:39  *** vexbuy has quit IRC
108 2018-08-25T07:00:33  *** sipa has quit IRC
109 2018-08-25T07:00:53  *** sipa has joined #bitcoin-core-dev
110 2018-08-25T07:18:38  *** Lauda has joined #bitcoin-core-dev
111 2018-08-25T07:29:26  *** vexbuy has joined #bitcoin-core-dev
112 2018-08-25T07:37:20  *** murrayn has quit IRC
113 2018-08-25T07:39:00  *** promag has joined #bitcoin-core-dev
114 2018-08-25T07:39:24  *** wxss has joined #bitcoin-core-dev
115 2018-08-25T07:43:36  *** promag has quit IRC
116 2018-08-25T07:53:27  *** Rootsudo has quit IRC
117 2018-08-25T07:54:53  *** Rootsudo has joined #bitcoin-core-dev
118 2018-08-25T07:55:39  *** Rootsudo has joined #bitcoin-core-dev
119 2018-08-25T07:57:11  *** Rootsudo has joined #bitcoin-core-dev
120 2018-08-25T08:04:50  *** Krellan has joined #bitcoin-core-dev
121 2018-08-25T08:05:43  *** vexbuy has quit IRC
122 2018-08-25T08:07:20  *** vexbuy has joined #bitcoin-core-dev
123 2018-08-25T08:08:55  *** SopaXorzTaker has joined #bitcoin-core-dev
124 2018-08-25T08:11:27  *** plankers has quit IRC
125 2018-08-25T08:12:26  *** vexbuy has quit IRC
126 2018-08-25T08:34:02  *** d9b4bef9 has quit IRC
127 2018-08-25T08:35:08  *** d9b4bef9 has joined #bitcoin-core-dev
128 2018-08-25T08:40:50  *** SopaXorzTaker has quit IRC
129 2018-08-25T08:42:10  *** vexbuy has joined #bitcoin-core-dev
130 2018-08-25T08:42:29  *** SopaXorzTaker has joined #bitcoin-core-dev
131 2018-08-25T09:08:08  *** csknk has joined #bitcoin-core-dev
132 2018-08-25T09:22:01  *** promag has joined #bitcoin-core-dev
133 2018-08-25T09:22:04  *** promag has quit IRC
134 2018-08-25T09:37:03  *** Krellan has quit IRC
135 2018-08-25T10:03:25  *** AaronvanW has joined #bitcoin-core-dev
136 2018-08-25T10:03:46  *** AaronvanW has quit IRC
137 2018-08-25T10:06:22  *** AaronvanW has joined #bitcoin-core-dev
138 2018-08-25T10:22:42  *** profmac has quit IRC
139 2018-08-25T10:30:03  *** promag has joined #bitcoin-core-dev
140 2018-08-25T10:35:30  *** promag has quit IRC
141 2018-08-25T10:45:24  *** cdecker has quit IRC
142 2018-08-25T11:27:14  *** vexbuy has quit IRC
143 2018-08-25T11:34:54  *** goatpig has quit IRC
144 2018-08-25T11:40:17  *** vexbuy has joined #bitcoin-core-dev
145 2018-08-25T11:42:05  *** promag has joined #bitcoin-core-dev
146 2018-08-25T11:42:32  *** IGHOR has quit IRC
147 2018-08-25T11:43:45  *** IGHOR has joined #bitcoin-core-dev
148 2018-08-25T11:46:19  *** vexbuy has quit IRC
149 2018-08-25T11:46:32  *** promag has quit IRC
150 2018-08-25T12:04:56  *** promag has joined #bitcoin-core-dev
151 2018-08-25T12:11:57  *** vexbuy has joined #bitcoin-core-dev
152 2018-08-25T12:36:02  *** d9b4bef9 has quit IRC
153 2018-08-25T12:37:09  *** d9b4bef9 has joined #bitcoin-core-dev
154 2018-08-25T12:41:51  *** as1nc has joined #bitcoin-core-dev
155 2018-08-25T12:41:56  *** vexbuy has quit IRC
156 2018-08-25T12:42:33  *** vexbuy has joined #bitcoin-core-dev
157 2018-08-25T12:46:48  *** vexbuy has quit IRC
158 2018-08-25T12:47:37  *** promag has quit IRC
159 2018-08-25T12:48:47  *** vexbuy has joined #bitcoin-core-dev
160 2018-08-25T12:56:43  *** esotericnonsense has quit IRC
161 2018-08-25T13:25:10  *** Chris_Stewart_5 has joined #bitcoin-core-dev
162 2018-08-25T13:39:13  *** promag has joined #bitcoin-core-dev
163 2018-08-25T13:40:50  *** Chris_Stewart_5 has quit IRC
164 2018-08-25T13:43:29  *** Victorsueca has quit IRC
165 2018-08-25T13:44:12  *** promag has quit IRC
166 2018-08-25T13:44:42  *** Victorsueca has joined #bitcoin-core-dev
167 2018-08-25T13:55:41  *** Chris_Stewart_5 has joined #bitcoin-core-dev
168 2018-08-25T13:56:21  *** AaronvanW has quit IRC
169 2018-08-25T13:58:42  *** owowo has quit IRC
170 2018-08-25T14:06:42  *** AaronvanW has joined #bitcoin-core-dev
171 2018-08-25T14:08:42  *** Chris_Stewart_5 has quit IRC
172 2018-08-25T14:11:37  *** AaronvanW has quit IRC
173 2018-08-25T14:13:03  *** Chris_Stewart_5 has joined #bitcoin-core-dev
174 2018-08-25T14:23:36  *** owowo has joined #bitcoin-core-dev
175 2018-08-25T14:30:20  *** promag has joined #bitcoin-core-dev
176 2018-08-25T14:43:43  *** promag has quit IRC
177 2018-08-25T15:03:44  *** vexbuy has quit IRC
178 2018-08-25T15:04:22  *** vexbuy has joined #bitcoin-core-dev
179 2018-08-25T15:04:46  *** bsm117532 has joined #bitcoin-core-dev
180 2018-08-25T15:07:43  *** AaronvanW has joined #bitcoin-core-dev
181 2018-08-25T15:08:41  *** Chris_Stewart_5 has quit IRC
182 2018-08-25T15:12:00  *** AaronvanW has quit IRC
183 2018-08-25T15:17:08  *** vexbuy_ has joined #bitcoin-core-dev
184 2018-08-25T15:20:51  *** vexbuy has quit IRC
185 2018-08-25T15:22:20  *** ossifrage has quit IRC
186 2018-08-25T15:23:39  *** ossifrage has joined #bitcoin-core-dev
187 2018-08-25T15:40:47  *** Chris_Stewart_5 has joined #bitcoin-core-dev
188 2018-08-25T16:16:47  *** drexl has joined #bitcoin-core-dev
189 2018-08-25T16:33:11  *** Zenton has quit IRC
190 2018-08-25T16:38:01  *** d9b4bef9 has quit IRC
191 2018-08-25T16:39:08  *** d9b4bef9 has joined #bitcoin-core-dev
192 2018-08-25T16:41:48  *** Krellan has joined #bitcoin-core-dev
193 2018-08-25T16:49:21  *** Krellan has quit IRC
194 2018-08-25T16:55:03  *** CubicEarth has quit IRC
195 2018-08-25T17:02:32  *** AaronvanW has joined #bitcoin-core-dev
196 2018-08-25T17:13:44  *** Krellan has joined #bitcoin-core-dev
197 2018-08-25T17:15:02  *** spinza has quit IRC
198 2018-08-25T17:26:03  *** spinza has joined #bitcoin-core-dev
199 2018-08-25T17:35:58  *** Taifta has joined #bitcoin-core-dev
200 2018-08-25T17:45:31  *** jhfrontz has quit IRC
201 2018-08-25T17:48:24  *** plankers has joined #bitcoin-core-dev
202 2018-08-25T17:49:32  *** jhfrontz has joined #bitcoin-core-dev
203 2018-08-25T18:08:14  *** Chris_Stewart_5 has quit IRC
204 2018-08-25T18:20:02  *** drexl has quit IRC
205 2018-08-25T18:29:31  *** csknk has quit IRC
206 2018-08-25T18:51:57  *** SopaXorzTaker has quit IRC
207 2018-08-25T18:58:45  *** davex__ has joined #bitcoin-core-dev
208 2018-08-25T19:03:20  *** hebasto has joined #bitcoin-core-dev
209 2018-08-25T19:07:42  *** SopaXorzTaker has joined #bitcoin-core-dev
210 2018-08-25T19:10:42  *** ken2812221 has quit IRC
211 2018-08-25T19:21:02  *** d9b4bef9 has quit IRC
212 2018-08-25T19:22:08  *** d9b4bef9 has joined #bitcoin-core-dev
213 2018-08-25T19:32:08  *** SopaXorzTaker has quit IRC
214 2018-08-25T19:32:53  *** wumpus has joined #bitcoin-core-dev
215 2018-08-25T20:39:07  *** vexbuy_ has quit IRC
216 2018-08-25T20:39:22  *** vexbuy has joined #bitcoin-core-dev
217 2018-08-25T20:39:36  <jamesob> re: memory usage increase: preliminary bisections are in and MarcoFalke and I are betting it's the leveldb changes. https://i.imgur.com/8aXRzwe.png
218 2018-08-25T20:41:33  *** intcat has quit IRC
219 2018-08-25T20:42:31  <gmaxwell> jamesob: wait. how are we measuring memory usage in that benchmark?
220 2018-08-25T20:42:52  <jamesob> gmaxwell: time -f %M (ie resident set size)
221 2018-08-25T20:49:08  <jamesob> https://github.com/chaincodelabs/bitcoinperf/blob/master/runner/main.py#L659-L668
222 2018-08-25T20:53:40  <sipa> so every mmap causes 150 kB RSS increase? :s
223 2018-08-25T20:54:31  <gmaxwell> no because we're not actually using the new maximum.
224 2018-08-25T20:54:48  <gmaxwell> (I mean not using all of)
225 2018-08-25T20:54:49  <sipa> aha.
226 2018-08-25T20:54:57  <gmaxwell> So even MOAR.
227 2018-08-25T20:55:00  <gmaxwell> which is suspect
228 2018-08-25T20:55:09  <sipa> mmap by default will prefetch mapped pages into memory
229 2018-08-25T20:55:19  *** plankers has quit IRC
230 2018-08-25T20:55:24  <sipa> by default, 31 4kB blocks
231 2018-08-25T20:55:54  <gmaxwell> oh interesting. but those pages are clean they'll just get evicted, they really shouldn't be counted bt RSS :(
232 2018-08-25T20:57:37  <gmaxwell> so MADV_RANDOM
233 2018-08-25T20:57:38  <gmaxwell> ...
234 2018-08-25T20:57:48  <sipa> madvice(MADV_DONTNEED) will disable the reading entirely
235 2018-08-25T20:58:08  <sipa> which may be useful to diagnose the issue
236 2018-08-25T20:58:44  <jamesob> (for those following along at home: https://github.com/bitcoin/bitcoin/pull/13925/files)
237 2018-08-25T20:58:46  *** AaronvanW has quit IRC
238 2018-08-25T20:59:20  *** AaronvanW has joined #bitcoin-core-dev
239 2018-08-25T20:59:26  *** Chris_Stewart_5 has joined #bitcoin-core-dev
240 2018-08-25T20:59:58  <gmaxwell> Also setting the maximum maps really low, like..2  might be interesting.
241 2018-08-25T21:02:21  <gmaxwell> but if this is the problem, MADV_RANDOM is probably the fix to the extent that its an actual problem at all.
242 2018-08-25T21:02:42  <gmaxwell> Though we should do a reindex benchmark to make sure MADV_RANDOM doesn't hurt performance.
243 2018-08-25T21:03:29  <wumpus> PSA: if after the latest merge you get a linker error "/usr/local/include/boost/smart_ptr/shared_ptr.hpp:728: undefined reference to `translationInterface", you need to do a 'make clean' and re-do the make and it will work
244 2018-08-25T21:03:53  *** AaronvanW has quit IRC
245 2018-08-25T21:04:25  <luke-jr> wumpus: is there a reason the gitian linux yml has g++-riscv64-linux-gnu as a dep? seems to pull in GCC 7 when we're using GCC 8 now?
246 2018-08-25T21:04:51  <luke-jr> wumpus: if `make clean` ever fixes something, that means there's a bug in the build system :/
247 2018-08-25T21:06:05  <sipa> gmaxwell, jamesob: LMDB uses MADV_RANDOM it seems
248 2018-08-25T21:06:22  <sipa> (though its design is different, i don't know their access patterns)
249 2018-08-25T21:06:36  <wumpus> luke-jr: yes, it must be missing some changes in dependency detection between source and header files (another one is if you change something in univalue, it won't detect it)
250 2018-08-25T21:06:55  <sipa> jamesob: try this:
251 2018-08-25T21:06:56  <sipa>          void* base = mmap(NULL, size, PROT_READ, MAP_SHARED, fd, 0);
252 2018-08-25T21:06:56  <sipa>          if (base != MAP_FAILED) {
253 2018-08-25T21:06:56  <sipa> +          madvise(base, size, MADV_DONTNEED);
254 2018-08-25T21:07:07  <sipa> in src/leveldb/util/env_posix.cc
255 2018-08-25T21:07:32  <sipa> (and/or MADV_RANDOM)
256 2018-08-25T21:07:37  <wumpus> luke-jr: it pulls in both gcc 7 and 8, I think that's necessary due to some strangeness with the packages (some symlink will only exist when g++-riscv64-linux-gnu is also installed)
257 2018-08-25T21:07:41  <luke-jr> DONTNEED sounds wrong?
258 2018-08-25T21:07:51  <luke-jr> wumpus: ah, weird
259 2018-08-25T21:08:08  <wumpus> luke-jr: you might be able to get around it, but I noticed and tried as well and ran into a dead end
260 2018-08-25T21:08:20  <sipa> luke-jr: to diagnoze
261 2018-08-25T21:08:48  <sipa> luke-jr: it would be interesting to see what the effect on RSS is with DONTNEED, to have an idea to what extent our memory usage is due to mmap caching
262 2018-08-25T21:09:30  <jamesob> sipa: giving it a shot now
263 2018-08-25T21:09:52  <wumpus> luke-jr: at least it's not MADV_HWPOISON!
264 2018-08-25T21:10:09  *** BashCo has joined #bitcoin-core-dev
265 2018-08-25T21:10:21  *** intcat has joined #bitcoin-core-dev
266 2018-08-25T21:10:40  *** plankers has joined #bitcoin-core-dev
267 2018-08-25T21:10:47  <luke-jr> wumpus: lol
268 2018-08-25T21:11:06  <gmaxwell> It's plausable to me that MADV_RANDOM will help performance or at least be neutral.
269 2018-08-25T21:11:39  <wumpus> yes, to me too
270 2018-08-25T21:11:48  <wumpus> our access pattern is more or less random
271 2018-08-25T21:12:15  <gmaxwell> I don't recall now, though I know I researched this before... does leveldb's bisection interpolate assuming keys are uniform and that their values are uniformly sized or does it plain bisect?
272 2018-08-25T21:12:51  <wumpus> (except in the rare times it's iterating over the whole utxo set in order, like when computing statistics)
273 2018-08-25T21:13:15  <gmaxwell> esp in the case of plain bisection, prefetching is a bad behavior.
274 2018-08-25T21:13:22  <wumpus> I don't know
275 2018-08-25T21:13:44  *** vexbuy has quit IRC
276 2018-08-25T21:14:19  *** vexbuy has joined #bitcoin-core-dev
277 2018-08-25T21:15:44  <sipa> gmaxwell: there is an index at the end of each ldb file
278 2018-08-25T21:15:59  <sipa> i assume it bisects in that index in a naive way, but i'm not sure
279 2018-08-25T21:21:05  <jamesob> sipa: bench is running; we'll know how your change works in about six hours
280 2018-08-25T21:22:06  <sipa> jamesob: awesome
281 2018-08-25T21:22:15  <luke-jr> hm, I think I will regret relatime when I try to prune gitian caches
282 2018-08-25T21:53:25  *** meshcollider_ has joined #bitcoin-core-dev
283 2018-08-25T21:54:31  *** Amuza has joined #bitcoin-core-dev
284 2018-08-25T21:55:36  *** vexbuy_ has joined #bitcoin-core-dev
285 2018-08-25T21:56:07  *** vexbuy has quit IRC
286 2018-08-25T22:02:32  *** jhfrontz has quit IRC
287 2018-08-25T22:08:18  *** AaronvanW has joined #bitcoin-core-dev
288 2018-08-25T22:09:15  *** BashCo has quit IRC
289 2018-08-25T22:11:10  *** BashCo has joined #bitcoin-core-dev
290 2018-08-25T22:17:18  *** plankers has quit IRC
291 2018-08-25T22:29:41  *** BashCo has quit IRC
292 2018-08-25T22:32:12  *** plankers has joined #bitcoin-core-dev
293 2018-08-25T22:32:38  *** BashCo has joined #bitcoin-core-dev
294 2018-08-25T22:33:39  *** Amuza has quit IRC
295 2018-08-25T22:48:21  *** esotericnonsense has joined #bitcoin-core-dev
296 2018-08-25T22:57:51  *** spinza has quit IRC
297 2018-08-25T23:07:35  *** sipa has quit IRC
298 2018-08-25T23:11:03  *** sipa has joined #bitcoin-core-dev
299 2018-08-25T23:18:44  *** dcousens has quit IRC
300 2018-08-25T23:23:02  *** d9b4bef9 has quit IRC
301 2018-08-25T23:24:08  *** d9b4bef9 has joined #bitcoin-core-dev
302 2018-08-25T23:24:58  *** spinza has joined #bitcoin-core-dev
303 2018-08-25T23:31:53  *** Chris_Stewart_5 has quit IRC
304 2018-08-25T23:40:48  *** Krellan has quit IRC
305 2018-08-25T23:41:28  *** Krellan has joined #bitcoin-core-dev