1 2018-06-09T00:03:25  *** Krellan has joined #bitcoin-core-dev
  2 2018-06-09T00:13:46  *** Krellan_ has joined #bitcoin-core-dev
  3 2018-06-09T00:14:21  *** Krellan__ has joined #bitcoin-core-dev
  4 2018-06-09T00:14:34  *** Krellan has quit IRC
  5 2018-06-09T00:15:14  *** Chris_Stewart_5 has joined #bitcoin-core-dev
  6 2018-06-09T00:18:44  *** Krellan_ has quit IRC
  7 2018-06-09T00:48:55  *** Chris_Stewart_5 has quit IRC
  8 2018-06-09T01:03:50  *** fanquake has joined #bitcoin-core-dev
  9 2018-06-09T01:19:12  *** nmnkgl has quit IRC
 10 2018-06-09T01:19:51  *** nmnkgl has joined #bitcoin-core-dev
 11 2018-06-09T01:24:22  *** nmnkgl has quit IRC
 12 2018-06-09T01:26:26  *** Chris_Stewart_5 has joined #bitcoin-core-dev
 13 2018-06-09T01:29:45  <fanquake> Empact Am alive and well :p
 14 2018-06-09T01:35:27  *** Chris_Stewart_5 has quit IRC
 15 2018-06-09T01:40:54  *** jhfrontz has quit IRC
 16 2018-06-09T02:02:35  *** Krellan__ has quit IRC
 17 2018-06-09T02:08:34  *** drexl has quit IRC
 18 2018-06-09T02:17:23  *** nmnkgl has joined #bitcoin-core-dev
 19 2018-06-09T02:19:27  *** AaronvanW has quit IRC
 20 2018-06-09T02:31:04  *** snickerfritz has joined #bitcoin-core-dev
 21 2018-06-09T02:47:50  *** fanquake has quit IRC
 22 2018-06-09T02:54:58  *** pipi has joined #bitcoin-core-dev
 23 2018-06-09T02:57:12  *** pipi has quit IRC
 24 2018-06-09T03:24:42  *** nmnkgl has quit IRC
 25 2018-06-09T03:25:17  *** nmnkgl has joined #bitcoin-core-dev
 26 2018-06-09T03:29:51  *** nmnkgl has quit IRC
 27 2018-06-09T03:41:46  *** CubicEarths has quit IRC
 28 2018-06-09T03:52:55  *** CubicEarths has joined #bitcoin-core-dev
 29 2018-06-09T04:03:57  *** CubicEarths has quit IRC
 30 2018-06-09T04:30:37  *** Deadhandd has quit IRC
 31 2018-06-09T04:33:49  *** Deadhand has joined #bitcoin-core-dev
 32 2018-06-09T04:51:45  *** CubicEarths has joined #bitcoin-core-dev
 33 2018-06-09T05:34:02  *** d9b4bef9 has quit IRC
 34 2018-06-09T05:35:00  *** harrymm has quit IRC
 35 2018-06-09T05:35:07  *** d9b4bef9 has joined #bitcoin-core-dev
 36 2018-06-09T05:37:26  *** harrymm has joined #bitcoin-core-dev
 37 2018-06-09T05:37:51  <murrayn> Is there a reason -O2 is specifically enabled in configure with --enable-debug? Should this not be -Og?
 38 2018-06-09T06:10:11  *** bitconner has quit IRC
 39 2018-06-09T06:19:33  *** Randolf has joined #bitcoin-core-dev
 40 2018-06-09T06:49:57  *** Tralfaz has quit IRC
 41 2018-06-09T07:15:00  *** Guyver2 has joined #bitcoin-core-dev
 42 2018-06-09T07:21:36  *** sud_ has joined #bitcoin-core-dev
 43 2018-06-09T07:23:06  *** sud_ has quit IRC
 44 2018-06-09T07:23:42  *** CubicEarths has quit IRC
 45 2018-06-09T07:54:39  *** CubicEarths has joined #bitcoin-core-dev
 46 2018-06-09T08:07:26  *** bitconner has joined #bitcoin-core-dev
 47 2018-06-09T08:10:52  *** CubicEarths has quit IRC
 48 2018-06-09T08:11:22  *** Guyver2 has quit IRC
 49 2018-06-09T08:40:03  *** SopaXorzTaker has joined #bitcoin-core-dev
 50 2018-06-09T08:55:34  *** Madars has quit IRC
 51 2018-06-09T08:57:57  *** bitconner has quit IRC
 52 2018-06-09T09:01:05  *** bitconner has joined #bitcoin-core-dev
 53 2018-06-09T09:13:36  *** Madars has joined #bitcoin-core-dev
 54 2018-06-09T09:39:22  *** Madars has quit IRC
 55 2018-06-09T09:45:20  *** SopaXorzTaker has quit IRC
 56 2018-06-09T09:45:48  *** SopaXorzTaker has joined #bitcoin-core-dev
 57 2018-06-09T10:00:24  *** echonaut has quit IRC
 58 2018-06-09T10:00:42  *** echonaut has joined #bitcoin-core-dev
 59 2018-06-09T10:18:52  *** Madars has joined #bitcoin-core-dev
 60 2018-06-09T10:22:41  *** grafcaps has joined #bitcoin-core-dev
 61 2018-06-09T10:27:22  *** grafcaps has quit IRC
 62 2018-06-09T10:42:22  *** Madars has quit IRC
 63 2018-06-09T10:59:50  *** SopaXorzTaker has quit IRC
 64 2018-06-09T11:00:09  *** SopaXorzTaker has joined #bitcoin-core-dev
 65 2018-06-09T11:01:56  *** AaronvanW has joined #bitcoin-core-dev
 66 2018-06-09T11:03:52  *** Aaronvan_ has joined #bitcoin-core-dev
 67 2018-06-09T11:06:57  *** AaronvanW has quit IRC
 68 2018-06-09T11:08:01  *** d9b4bef9 has quit IRC
 69 2018-06-09T11:08:51  *** SopaXorzTaker has quit IRC
 70 2018-06-09T11:09:08  *** d9b4bef9 has joined #bitcoin-core-dev
 71 2018-06-09T11:09:47  *** promag has joined #bitcoin-core-dev
 72 2018-06-09T11:17:22  *** Madars has joined #bitcoin-core-dev
 73 2018-06-09T11:18:32  *** promag has quit IRC
 74 2018-06-09T11:19:01  *** promag has joined #bitcoin-core-dev
 75 2018-06-09T11:19:55  *** SopaXorzTaker has joined #bitcoin-core-dev
 76 2018-06-09T11:19:59  *** promag has quit IRC
 77 2018-06-09T11:20:22  *** promag has joined #bitcoin-core-dev
 78 2018-06-09T11:20:26  *** promag has quit IRC
 79 2018-06-09T11:20:47  *** jhfrontz has joined #bitcoin-core-dev
 80 2018-06-09T11:42:32  *** Madars has quit IRC
 81 2018-06-09T11:57:52  *** Victorsueca has quit IRC
 82 2018-06-09T11:59:14  *** Victorsueca has joined #bitcoin-core-dev
 83 2018-06-09T12:05:44  *** Aaronvan_ has quit IRC
 84 2018-06-09T12:06:20  *** AaronvanW has joined #bitcoin-core-dev
 85 2018-06-09T12:10:55  *** AaronvanW has quit IRC
 86 2018-06-09T12:22:29  *** Madars has joined #bitcoin-core-dev
 87 2018-06-09T12:39:49  *** games_ has quit IRC
 88 2018-06-09T12:41:10  *** drexl has joined #bitcoin-core-dev
 89 2018-06-09T12:43:14  *** AaronvanW has joined #bitcoin-core-dev
 90 2018-06-09T12:53:07  *** Madars has quit IRC
 91 2018-06-09T13:16:30  *** ula has joined #bitcoin-core-dev
 92 2018-06-09T13:20:33  *** bitconner has quit IRC
 93 2018-06-09T13:32:33  *** SopaXorzTaker has quit IRC
 94 2018-06-09T13:42:04  *** jcohen has joined #bitcoin-core-dev
 95 2018-06-09T13:43:52  *** Madars has joined #bitcoin-core-dev
 96 2018-06-09T13:54:09  *** jcohen has quit IRC
 97 2018-06-09T13:58:05  *** ken2812221 has joined #bitcoin-core-dev
 98 2018-06-09T14:07:50  *** Guyver2 has joined #bitcoin-core-dev
 99 2018-06-09T14:13:34  *** justanotheruser has quit IRC
100 2018-06-09T14:15:31  *** jcohen has joined #bitcoin-core-dev
101 2018-06-09T14:19:52  *** nekotribal has quit IRC
102 2018-06-09T14:33:12  *** jcohen has quit IRC
103 2018-06-09T14:56:15  *** jcohen has joined #bitcoin-core-dev
104 2018-06-09T15:05:01  *** nekotribal has joined #bitcoin-core-dev
105 2018-06-09T15:09:12  *** SopaXorzTaker has joined #bitcoin-core-dev
106 2018-06-09T15:12:08  *** AaronvanW has quit IRC
107 2018-06-09T15:19:46  *** jcohen has quit IRC
108 2018-06-09T15:24:09  *** SopaXorzTaker has quit IRC
109 2018-06-09T15:32:44  *** justanotheruser has joined #bitcoin-core-dev
110 2018-06-09T15:37:37  *** andytoshi has quit IRC
111 2018-06-09T15:38:26  *** Chris_Stewart_5 has joined #bitcoin-core-dev
112 2018-06-09T15:49:58  *** SopaXorzTaker has joined #bitcoin-core-dev
113 2018-06-09T15:58:30  *** justanotheruser has quit IRC
114 2018-06-09T16:10:08  *** grafcaps has joined #bitcoin-core-dev
115 2018-06-09T16:14:38  *** jcohen has joined #bitcoin-core-dev
116 2018-06-09T16:21:39  *** Randolf has quit IRC
117 2018-06-09T16:25:55  *** AaronvanW has joined #bitcoin-core-dev
118 2018-06-09T16:34:09  <gmaxwell> sipa: linked on your SHANI pr is an implementation where someone else noticed the throuput/latency relationship that I noticed.. they also do a 4way and it's faster (by a small amount) than 2-way.
119 2018-06-09T16:34:27  *** Chris_Stewart_5 has quit IRC
120 2018-06-09T16:34:42  <gmaxwell> they get 18% speedup for 2way over 1way, and 21% for 4-way over two-way.
121 2018-06-09T16:35:15  <gmaxwell> I'm not sure if that difference is even worth it, though perhaps throughput might increase for later cpus.
122 2018-06-09T16:37:23  <sipa> interesting, i'll try that too
123 2018-06-09T16:37:47  <gmaxwell> Their implementation might be interesting to look at to see if they had some smarter way of dealing with register pressure.
124 2018-06-09T16:38:07  <sipa> another remarkable thing i noticed: the speedup of 64-specialized shani over variable length shani was close to 2x
125 2018-06-09T16:38:19  <sipa> far higher than the ratio observed elsewhere
126 2018-06-09T16:38:33  <sipa> gmaxwell: from what i can see it's just interleaving
127 2018-06-09T16:39:17  <gmaxwell> (presumably register churn is why their attempt at 8-way was slower 2/4 way)
128 2018-06-09T16:40:26  <gmaxwell> sipa: The 64-specialized saves expander work, which I guess isn't as fast with shani?  or maybe it's just that shani is faster so calling overhead (which the specialized reduces) matters more?
129 2018-06-09T16:41:12  <provoostenator> Memory management is a pain. I have a device with 1 GB RAM, trying to squeeze as much as possible out of it during IBD. Without swap, if I set it slight too high, it crashes when dbcache gets too large. With swap, it starts using the swap, which presumably defeats the purpose. Is there any way to _have_ swap but prevent dbcache from using it?
130 2018-06-09T16:42:05  <gmaxwell> provoostenator: I doubt swapping is actually defeating the purpose, at least if it isn't doing it heavily.
131 2018-06-09T16:42:25  <gmaxwell> The data that gets swapped is infrequently used stuff first...
132 2018-06-09T16:45:15  <sipa> gmaxwell: SHANI has special instructions both for expansion and transform
133 2018-06-09T16:45:19  <provoostenator> It indeed didn't seem very slow, so maybe it's not too bad in practice then. 450 MB dbcache (with maxmempool=5) seems about the max without swap.
134 2018-06-09T16:51:22  *** jcohen has quit IRC
135 2018-06-09T16:52:59  *** jcohen has joined #bitcoin-core-dev
136 2018-06-09T16:57:10  *** laurentmt has joined #bitcoin-core-dev
137 2018-06-09T17:01:10  *** Sinclair6 has joined #bitcoin-core-dev
138 2018-06-09T17:01:52  *** berndj has quit IRC
139 2018-06-09T17:04:37  *** berndj has joined #bitcoin-core-dev
140 2018-06-09T17:09:43  <sipa> gmaxwell: 4-way seems a bit slower here, but that may be due to less than perfectly interleaved code being emitted
141 2018-06-09T17:11:41  *** laurentmt1 has joined #bitcoin-core-dev
142 2018-06-09T17:12:50  *** laurentmt has quit IRC
143 2018-06-09T17:12:50  *** laurentmt1 is now known as laurentmt
144 2018-06-09T17:16:51  *** laurentmt has quit IRC
145 2018-06-09T17:18:43  *** laurentmt has joined #bitcoin-core-dev
146 2018-06-09T17:33:10  *** ctrlbreak has joined #bitcoin-core-dev
147 2018-06-09T17:36:48  *** Guyver2 has quit IRC
148 2018-06-09T17:37:02  *** Guyver2 has joined #bitcoin-core-dev
149 2018-06-09T17:38:23  *** laurentmt has quit IRC
150 2018-06-09T17:49:28  *** grafcaps has quit IRC
151 2018-06-09T17:51:40  *** grafcaps has joined #bitcoin-core-dev
152 2018-06-09T17:51:46  *** ToURniQue has joined #bitcoin-core-dev
153 2018-06-09T17:59:23  *** jimmysong_ has joined #bitcoin-core-dev
154 2018-06-09T18:03:12  *** TheV01d_ has joined #bitcoin-core-dev
155 2018-06-09T18:03:39  *** kanzure_ has joined #bitcoin-core-dev
156 2018-06-09T18:04:04  *** TheV01d has quit IRC
157 2018-06-09T18:04:04  *** ula has quit IRC
158 2018-06-09T18:04:05  *** kanzure has quit IRC
159 2018-06-09T18:04:05  *** belcher has quit IRC
160 2018-06-09T18:04:06  *** jimmysong has quit IRC
161 2018-06-09T18:04:13  *** TheV01d_ is now known as TheV01d
162 2018-06-09T18:04:26  *** belcher has joined #bitcoin-core-dev
163 2018-06-09T18:06:00  *** stepa[m] has quit IRC
164 2018-06-09T18:06:27  *** thaumavorio has quit IRC
165 2018-06-09T18:06:42  *** kewde[m] has quit IRC
166 2018-06-09T18:06:52  *** joshb[m] has quit IRC
167 2018-06-09T18:07:14  *** squarfed[m] has quit IRC
168 2018-06-09T18:07:17  *** ajtowns[m] has quit IRC
169 2018-06-09T18:07:17  *** herzmeister[m] has quit IRC
170 2018-06-09T18:08:54  *** thaumavorio has joined #bitcoin-core-dev
171 2018-06-09T18:15:40  *** jcohen has quit IRC
172 2018-06-09T18:17:33  *** SopaXorzTaker has quit IRC
173 2018-06-09T18:18:13  *** AaronvanW has quit IRC
174 2018-06-09T18:18:50  *** AaronvanW has joined #bitcoin-core-dev
175 2018-06-09T18:20:00  *** SopaXorzTaker has joined #bitcoin-core-dev
176 2018-06-09T18:23:29  *** AaronvanW has quit IRC
177 2018-06-09T18:28:54  *** AaronvanW has joined #bitcoin-core-dev
178 2018-06-09T18:32:13  *** justanotheruser has joined #bitcoin-core-dev
179 2018-06-09T18:33:26  *** AaronvanW has quit IRC
180 2018-06-09T18:34:07  *** intcat has quit IRC
181 2018-06-09T18:36:28  *** intcat has joined #bitcoin-core-dev
182 2018-06-09T18:51:43  *** SopaXorzTaker has quit IRC
183 2018-06-09T19:05:02  *** TheRec_ has joined #bitcoin-core-dev
184 2018-06-09T19:05:02  *** TheRec_ has joined #bitcoin-core-dev
185 2018-06-09T19:05:50  *** TheRec has quit IRC
186 2018-06-09T19:29:40  *** AaronvanW has joined #bitcoin-core-dev
187 2018-06-09T19:34:32  *** AaronvanW has quit IRC
188 2018-06-09T19:42:18  *** Chris_Stewart_5 has joined #bitcoin-core-dev
189 2018-06-09T20:04:40  *** justanotheruser has quit IRC
190 2018-06-09T20:13:57  *** GoldenBear has quit IRC
191 2018-06-09T20:20:12  *** justanotheruser has joined #bitcoin-core-dev
192 2018-06-09T20:21:59  *** Guyver2 has quit IRC
193 2018-06-09T20:28:53  *** joshb[m] has joined #bitcoin-core-dev
194 2018-06-09T20:43:07  *** justanotheruser has quit IRC
195 2018-06-09T20:44:48  *** Victorsueca has quit IRC
196 2018-06-09T20:45:20  *** ajtowns[m] has joined #bitcoin-core-dev
197 2018-06-09T20:45:20  *** herzmeister[m] has joined #bitcoin-core-dev
198 2018-06-09T20:45:20  *** kewde[m] has joined #bitcoin-core-dev
199 2018-06-09T20:45:21  *** squarfed[m] has joined #bitcoin-core-dev
200 2018-06-09T20:45:21  *** stepa[m] has joined #bitcoin-core-dev
201 2018-06-09T20:46:02  *** Victorsueca has joined #bitcoin-core-dev
202 2018-06-09T20:59:38  *** justanotheruser has joined #bitcoin-core-dev
203 2018-06-09T21:04:53  <provoostenator> I have a new theory as to why my aggresive pruning IBD branch is _slower_ than master. Namely that dirty CCoinsCacheEntry read/write doesn't perform well for very large cache sizes. See See also https://github.com/bitcoin/bitcoin/pull/12404#issuecomment-395998702
204 2018-06-09T21:05:04  <provoostenator> (theory, still have to measure this)
205 2018-06-09T21:27:29  <phantomcircuit> provoostenator, aggressive pruning?
206 2018-06-09T21:27:34  *** kanzure_ is now known as kanzure
207 2018-06-09T21:27:46  <sipa> phantomcircuit: #12404
208 2018-06-09T21:27:48  <gribble> https://github.com/bitcoin/bitcoin/issues/12404 | Prune more aggressively during IBD by Sjors · Pull Request #12404 · bitcoin/bitcoin · GitHub
209 2018-06-09T21:28:52  <phantomcircuit> oh
210 2018-06-09T21:30:28  *** AaronvanW has joined #bitcoin-core-dev
211 2018-06-09T21:34:38  <phantomcircuit> sipa, does flushing the cache still remove everything?
212 2018-06-09T21:35:00  <sipa> yes
213 2018-06-09T21:36:58  *** AaronvanW has quit IRC
214 2018-06-09T21:37:38  <phantomcircuit> sipa, and there's no way to flush "upto block x" right?
215 2018-06-09T21:38:59  <sipa> phantomcircuit: indeed, because there may have been entries created before x, but spent after x, which wouldn't be present on disk
216 2018-06-09T21:39:38  <sipa> it is possible with the non-atomic flushing since 0.15 (which writes to disk a range of blocks rather than a single up-to-x point)
217 2018-06-09T21:39:48  <sipa> though it's pretty complicated to reason about
218 2018-06-09T21:59:25  <phantomcircuit> sipa, so to enable that you'd need to keep around entries that are a record of an entry being deleted?
219 2018-06-09T22:01:37  <sipa> phantomcircuit: you actually don't
220 2018-06-09T22:02:27  <sipa> you just need to accurately keep track of (a) the block up to which you've flushed everything and (b) the block up to which effects may be present on disk, and at startup replay the blocks' UTXO effects between those 2
221 2018-06-09T22:02:34  <sipa> that's already implemented even
222 2018-06-09T22:02:53  <sipa> however, once you introduce partial flushing during reorgs which may overlap etc... it becomes far more complicated
223 2018-06-09T22:03:34  <phantomcircuit> yeah wasn't thinking about reorgs
224 2018-06-09T22:04:27  <sipa> all of this is doable, and i think i know all the algorithms necessary to implement it
225 2018-06-09T22:05:05  <sipa> with the goal of being able to have a background process that just periodically (and asynchronously) flushes the oldest dirty UTXO entries (and wipes the oldest non-dirty ones)
226 2018-06-09T22:05:27  <sipa> but it's a pretty big amount of work without knowing if it'll actually speed things up :)
227 2018-06-09T22:06:17  <phantomcircuit> sipa, i had a patch which did this, but broke consensus across reorgs
228 2018-06-09T22:06:22  <phantomcircuit> it was a substantial speed up
229 2018-06-09T22:06:33  <phantomcircuit> but that was a while ago, so possibly it wouldn't be as large anymore?
230 2018-06-09T22:06:54  <sipa> since per-txout in 0.15 performance profiles of such things may have shifted drastically
231 2018-06-09T22:07:00  <sipa> it could be less or more of a speedup now :)
232 2018-06-09T22:11:37  <phantomcircuit> yeah
233 2018-06-09T22:11:47  <phantomcircuit> iirc it was really simple to do
234 2018-06-09T22:38:23  *** jhfrontz has quit IRC
235 2018-06-09T22:38:48  *** Chris_Stewart_5 has quit IRC
236 2018-06-09T22:52:26  *** jhfrontz has joined #bitcoin-core-dev
237 2018-06-09T22:56:15  *** andytoshi has joined #bitcoin-core-dev
238 2018-06-09T22:59:41  *** andytoshi has joined #bitcoin-core-dev
239 2018-06-09T22:59:42  *** andytoshi has joined #bitcoin-core-dev
240 2018-06-09T23:03:20  *** andytoshi has joined #bitcoin-core-dev
241 2018-06-09T23:23:00  *** Giszmo has joined #bitcoin-core-dev
242 2018-06-09T23:37:44  *** Chris_Stewart_5 has joined #bitcoin-core-dev
243 2018-06-09T23:39:59  <phantomcircuit> sipa, the FRESH flag looks a bit confusing
244 2018-06-09T23:41:04  <phantomcircuit> the idea is that if an entry is added and spent before a flush it's effectively a noop ?
245 2018-06-09T23:41:22  <sipa> it just means "this entry does not exist in the parent cache, so if it is spent, we can just forget about it"
246 2018-06-09T23:41:54  <sipa> phantomcircuit: it's *the* major performance gain our cache gives
247 2018-06-09T23:41:55  <phantomcircuit> ok i get that
248 2018-06-09T23:42:15  <phantomcircuit> yeah
249 2018-06-09T23:43:36  <sipa> because it avoids entries ever hitting disk at all
250 2018-06-09T23:45:07  *** ToURniQue has quit IRC
251 2018-06-09T23:47:26  <phantomcircuit> sipa, yup i definitely get it