1 2015-10-28T00:01:19  *** zooko has quit IRC
  2 2015-10-28T00:07:08  *** Thireus1 has quit IRC
  3 2015-10-28T00:07:23  *** Thireus has joined #bitcoin-core-dev
  4 2015-10-28T00:09:49  *** zooko has joined #bitcoin-core-dev
  5 2015-10-28T00:12:39  *** Thireus1 has joined #bitcoin-core-dev
  6 2015-10-28T00:13:59  *** Thireus2 has joined #bitcoin-core-dev
  7 2015-10-28T00:15:10  *** Thireus has quit IRC
  8 2015-10-28T00:17:57  *** Thireus1 has quit IRC
  9 2015-10-28T00:21:01  <GitHub30> [bitcoin] sipa opened pull request #6895: Update to my new key (master...newkey) https://github.com/bitcoin/bitcoin/pull/6895
 10 2015-10-28T00:36:07  *** Apocalyptic has quit IRC
 11 2015-10-28T00:42:41  *** Apocalyptic has joined #bitcoin-core-dev
 12 2015-10-28T00:55:23  *** zooko has quit IRC
 13 2015-10-28T01:07:31  <GitHub19> [bitcoin] gmaxwell pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/8f3b3cdee497...d0badb916e51
 14 2015-10-28T01:07:31  <GitHub19> bitcoin/master 298e040 Pieter Wuille: Fix chainstate serialized_size computation
 15 2015-10-28T01:07:32  <GitHub19> bitcoin/master d0badb9 Gregory Maxwell: Merge pull request #6865...
 16 2015-10-28T01:07:37  <GitHub22> [bitcoin] gmaxwell closed pull request #6865: Fix chainstate serialized_size computation (master...fixchainsize) https://github.com/bitcoin/bitcoin/pull/6865
 17 2015-10-28T01:20:28  *** treehug88 has quit IRC
 18 2015-10-28T01:22:14  <GitHub43> [bitcoin] sipa pushed 4 new commits to master: https://github.com/bitcoin/bitcoin/compare/d0badb916e51...93521a4f56ce
 19 2015-10-28T01:22:15  <GitHub43> bitcoin/master 27252b7 Matt Corallo: Fix pre-push-hook regexes
 20 2015-10-28T01:22:15  <GitHub43> bitcoin/master 1d94b72 Matt Corallo: Whitelist commits signed with Pieter's now-revoked key
 21 2015-10-28T01:22:16  <GitHub43> bitcoin/master 6e800c2 Matt Corallo: Add Pieter's new PGP key to verify-commits/trusted-keys
 22 2015-10-28T01:22:24  <GitHub113> [bitcoin] sipa closed pull request #6875: Fix pre-push-hook regexes (master...verify-commits-fixes) https://github.com/bitcoin/bitcoin/pull/6875
 23 2015-10-28T01:24:03  <GitHub157> [bitcoin] sipa pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/93521a4f56ce...8756c986420c
 24 2015-10-28T01:24:03  <GitHub157> bitcoin/master 4252cd0 Pieter Wuille: Update to my new key
 25 2015-10-28T01:24:04  <GitHub157> bitcoin/master 8756c98 Pieter Wuille: Merge pull request #6895...
 26 2015-10-28T01:24:13  <GitHub179> [bitcoin] sipa closed pull request #6895: Update to my new key (master...newkey) https://github.com/bitcoin/bitcoin/pull/6895
 27 2015-10-28T01:25:32  <GitHub93> [bitcoin] sipa pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/8756c986420c...e06c14fb59ee
 28 2015-10-28T01:25:32  <GitHub93> bitcoin/master ab1f560 Pieter Wuille: Support -checkmempool=N, which runs checks on average once every N transactions
 29 2015-10-28T01:25:33  <GitHub93> bitcoin/master e06c14f Pieter Wuille: Merge pull request #6776...
 30 2015-10-28T01:25:37  <GitHub73> [bitcoin] sipa closed pull request #6776: Support -checkmempool=N, which runs checks once every N transactions (master...fraccheck) https://github.com/bitcoin/bitcoin/pull/6776
 31 2015-10-28T01:31:04  <GitHub67> [bitcoin] sipa pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/e06c14fb59ee...4764f5db9d2c
 32 2015-10-28T01:31:04  <GitHub67> bitcoin/master 214de7e Philip Kaufmann: [Trivial] ensure minimal header conventions...
 33 2015-10-28T01:31:05  <GitHub67> bitcoin/master 4764f5d Pieter Wuille: Merge pull request #6892...
 34 2015-10-28T01:31:12  <GitHub110> [bitcoin] sipa closed pull request #6892: [Trivial] ensure minimal header conventions (master...headers-new) https://github.com/bitcoin/bitcoin/pull/6892
 35 2015-10-28T01:34:35  <GitHub1> [bitcoin] sipa pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/4764f5db9d2c...8daffe227bc6
 36 2015-10-28T01:34:35  <GitHub1> bitcoin/master ad5aae1 Philip Kaufmann: constify missing catch cases...
 37 2015-10-28T01:34:36  <GitHub1> bitcoin/master 8daffe2 Pieter Wuille: Merge pull request #6891...
 38 2015-10-28T01:34:44  <GitHub130> [bitcoin] sipa closed pull request #6891: constify missing catch cases (master...const-ex) https://github.com/bitcoin/bitcoin/pull/6891
 39 2015-10-28T01:46:53  *** d_t has quit IRC
 40 2015-10-28T01:53:39  *** jtimon has quit IRC
 41 2015-10-28T02:01:51  <gmaxwell> Oh I think I know one reason we see corruption reports on windows.
 42 2015-10-28T02:02:34  <gmaxwell> Stopnode often takes an awful long time, for example on my laptop running defaults it just took 12 seconds.  In windows, on shutdown, tasks that don't stop pretty much immediately get killed.
 43 2015-10-28T02:02:54  <gmaxwell> thats probably a maximally bad case for leveldb as it'll be in the middle of flushing out a bunch of cached updates.
 44 2015-10-28T02:10:09  <GitHub4> [bitcoin] sipa opened pull request #6896: Make -checkmempool=1 not fail through int32 overflow (master...fixchainsize) https://github.com/bitcoin/bitcoin/pull/6896
 45 2015-10-28T02:32:51  *** belcher has quit IRC
 46 2015-10-28T02:35:12  <tripleslash> gmaxwell: its specifically HungAppTimeout and is defaulted to 5 seconds.
 47 2015-10-28T02:38:02  <morcos> test_bitcoin hangs for me a decent fraction of the time.  It looks like it gets stuck at line 109 in scheduler_tests.cpp
 48 2015-10-28T02:56:22  <morcos> ah, i see it is an issue, #6540
 49 2015-10-28T03:12:09  *** fkhan has quit IRC
 50 2015-10-28T03:44:21  *** Ylbam has quit IRC
 51 2015-10-28T04:02:12  *** baldur has quit IRC
 52 2015-10-28T04:02:28  *** baldur has joined #bitcoin-core-dev
 53 2015-10-28T04:03:44  *** blur3d has joined #bitcoin-core-dev
 54 2015-10-28T05:12:10  *** blur3d has quit IRC
 55 2015-10-28T05:24:42  *** gribble has quit IRC
 56 2015-10-28T05:27:20  *** gribble has joined #bitcoin-core-dev
 57 2015-10-28T05:45:39  *** zooko has joined #bitcoin-core-dev
 58 2015-10-28T06:27:19  *** ParadoxSpiral has joined #bitcoin-core-dev
 59 2015-10-28T06:36:29  *** gavinandresen has quit IRC
 60 2015-10-28T06:47:26  *** ParadoxSpiral has quit IRC
 61 2015-10-28T07:08:59  *** deepcore has joined #bitcoin-core-dev
 62 2015-10-28T07:11:14  *** fkhan has joined #bitcoin-core-dev
 63 2015-10-28T07:25:03  *** crescendo has quit IRC
 64 2015-10-28T07:29:23  *** deepcore has quit IRC
 65 2015-10-28T07:40:31  *** Ylbam has joined #bitcoin-core-dev
 66 2015-10-28T08:13:08  <wumpus> gmaxwell: that will likely contribute to it, although most of the time corruption seems to happen on crashes / power failures, when there is no time to flush at all
 67 2015-10-28T08:27:46  <wumpus> but if that is the case too then the flush+sync on windows is not essentially not working at all
 68 2015-10-28T08:27:59  <wumpus> -first not
 69 2015-10-28T08:40:38  <gmaxwell> someone was commenting that we were writing via mmap on windows and that the sync we were using there didn't work on maps; which sounds like the mac problem. I didn't verify these claims at all.
 70 2015-10-28T08:58:35  *** rubensayshi has joined #bitcoin-core-dev
 71 2015-10-28T09:01:31  *** BashCo has quit IRC
 72 2015-10-28T09:11:19  *** aaaaok has joined #bitcoin-core-dev
 73 2015-10-28T09:11:36  <wumpus> didn't check that either
 74 2015-10-28T09:21:23  *** BashCo has joined #bitcoin-core-dev
 75 2015-10-28T09:26:11  *** BashCo has quit IRC
 76 2015-10-28T09:53:00  *** jtimon has joined #bitcoin-core-dev
 77 2015-10-28T10:03:48  <dcousens> wumpus: aye, 1 OOM and my chain was broke
 78 2015-10-28T10:22:48  *** aaaaok has quit IRC
 79 2015-10-28T10:47:03  *** evoskuil has quit IRC
 80 2015-10-28T11:57:57  *** fanquake has joined #bitcoin-core-dev
 81 2015-10-28T12:05:05  *** fanquake has quit IRC
 82 2015-10-28T12:05:28  *** fanquake has joined #bitcoin-core-dev
 83 2015-10-28T12:20:22  *** fanquake has left #bitcoin-core-dev
 84 2015-10-28T12:48:53  *** paveljanik has joined #bitcoin-core-dev
 85 2015-10-28T12:48:53  *** paveljanik has joined #bitcoin-core-dev
 86 2015-10-28T12:57:57  *** treehug88 has joined #bitcoin-core-dev
 87 2015-10-28T13:02:39  *** randy-waterhouse has joined #bitcoin-core-dev
 88 2015-10-28T13:03:44  *** randy-waterhouse has quit IRC
 89 2015-10-28T13:09:19  *** molly has joined #bitcoin-core-dev
 90 2015-10-28T13:12:19  *** moli has quit IRC
 91 2015-10-28T13:14:02  *** dcousens has quit IRC
 92 2015-10-28T13:56:06  *** zooko has quit IRC
 93 2015-10-28T13:56:59  *** gavinandresen has joined #bitcoin-core-dev
 94 2015-10-28T13:57:31  *** zooko has joined #bitcoin-core-dev
 95 2015-10-28T14:02:59  *** davec has quit IRC
 96 2015-10-28T14:19:04  *** davec has joined #bitcoin-core-dev
 97 2015-10-28T14:22:00  *** moli has joined #bitcoin-core-dev
 98 2015-10-28T14:25:19  *** molly has quit IRC
 99 2015-10-28T14:55:27  <jcorgan> cfields: did you ever make progress on #6681?
100 2015-10-28T14:58:40  <jcorgan> i guess that would be #6819 now
101 2015-10-28T15:10:14  *** MarcoFalke has joined #bitcoin-core-dev
102 2015-10-28T15:16:32  <cfields> jcorgan: no, i stopped there. I just wanted to get it building so that someone who knows zmq could make it actually work
103 2015-10-28T15:22:27  <jcorgan> got it
104 2015-10-28T15:32:18  *** ParadoxSpiral has joined #bitcoin-core-dev
105 2015-10-28T15:38:58  *** bsm1175321 has quit IRC
106 2015-10-28T15:47:06  *** MarcoFalke has quit IRC
107 2015-10-28T16:00:06  *** MarcoFalke has joined #bitcoin-core-dev
108 2015-10-28T16:02:57  *** bsm1175321 has joined #bitcoin-core-dev
109 2015-10-28T16:03:05  *** jl2012 has quit IRC
110 2015-10-28T16:03:17  *** jl2012 has joined #bitcoin-core-dev
111 2015-10-28T16:46:49  *** jl2012_ has joined #bitcoin-core-dev
112 2015-10-28T16:47:39  *** jl2012 has quit IRC
113 2015-10-28T16:57:43  *** deepcore has joined #bitcoin-core-dev
114 2015-10-28T17:07:40  *** d_t has joined #bitcoin-core-dev
115 2015-10-28T17:28:16  *** molly has joined #bitcoin-core-dev
116 2015-10-28T17:31:19  *** moli has quit IRC
117 2015-10-28T17:47:48  <morcos> gmaxwell: For this fast template generation on new block code.  Are you envisioning you switch to a new empty template after a new most work header?  Even if you haven't validated the block you're connecting yet?  And then set some sort of timeout with which you'll switch back to not being willing to build off a headers only chain?
118 2015-10-28T17:48:51  <morcos> Once you've connected a new block, I'd say there is no reason not to wait the extra few ms to generate a block template with txs in it.  Which can then be validated after its already been served to you.
119 2015-10-28T17:49:02  <gmaxwell> No I was not. This is less safe than many people think it is, and I think not needed if the other details are handled correctly.
120 2015-10-28T17:49:47  <morcos> Well thats where all the delay is right?  Receiving and connecting the new best block.
121 2015-10-28T17:49:49  <sipa> how about building a new template, switching to working on it immediately, and then starting a validation for it
122 2015-10-28T17:50:32  <morcos> sipa: yes thats what i'm suggesting.  but thats after you've connected the best block.  if we're still requiring to wait for that, don't we think people will choose to short circuit it less safely on their own
123 2015-10-28T17:50:34  <gmaxwell> morcos: No, CNB latency is tens of times slower than validating normally.
124 2015-10-28T17:51:45  <morcos> gmaxwell: ok, i'll give you multiples, but probably less than 10x unless your mempool is really really big.  and thats just compared to validating, what about waiting to receive?
125 2015-10-28T17:52:16  <morcos> i haven't looked at the time delay from receiving most work header to finishing connecting the block, any idea what that is typically?
126 2015-10-28T17:52:19  <morcos> i bet its long
127 2015-10-28T17:52:20  <gmaxwell> When the relay network is working normally about 80% of blocks are transmitted in a single packet.
128 2015-10-28T17:52:30  <morcos> ah yes, forgot about relay network
129 2015-10-28T17:52:42  <morcos> thats why i ask questions
130 2015-10-28T17:53:30  <sipa> it still makes sense to have numbers for the time between receive inv and CNB building a template on top
131 2015-10-28T17:53:39  <gmaxwell> Most of the delays in mining right now appear to be from outside of bitcoin core, actually.
132 2015-10-28T17:54:00  <morcos> sipa in a relay node connected case or regular node or both
133 2015-10-28T17:54:20  <sipa> morcos: in "reality"
134 2015-10-28T17:54:22  <sipa> :)
135 2015-10-28T17:55:08  <morcos> ok, well i'm almost ready to push a WIP branch.  it still doesn't do it in a thread, but the gain is really rather limited at this point, and i'll save that i think for a second pull
136 2015-10-28T17:55:21  <morcos> but the question i want to resolve is what to do when TestBlockValidity fails
137 2015-10-28T17:55:32  <morcos> right now it throws an error
138 2015-10-28T17:55:34  <gmaxwell> hm. we've done something recently that slowed down connectblock (or the network behavior changes have)
139 2015-10-28T17:55:51  <gmaxwell> oh dear
140 2015-10-28T17:55:53  <morcos> connectblock has always been slow since i've measured it
141 2015-10-28T17:56:05  <gmaxwell> debug.log:2015-10-28 16:11:38 - Connect block: 7256.55ms [7475.14s]
142 2015-10-28T17:56:39  <sipa> how much is fetching inputs vs verifying inputs?
143 2015-10-28T17:56:47  <gmaxwell> wtf.
144 2015-10-28T17:57:07  <gmaxwell> since my node here's last update to master connect block's time has increased monotonically for every block.
145 2015-10-28T17:58:02  <sipa> gmaxwell: coincache being thrown out my mempool bloat?
146 2015-10-28T17:59:48  <morcos> gmaxwell, what block hash was that?
147 2015-10-28T18:01:41  <morcos> gmaxwell: are you still running mempool.check?  it runs inside that timer
148 2015-10-28T18:01:43  <gmaxwell> morcos: before this latest rounds of attack this node was taking <100ms to connect block.
149 2015-10-28T18:01:58  <gmaxwell> morcos: must be the mempool checks then.
150 2015-10-28T18:02:29  <gmaxwell> morcos: it's all of them.
151 2015-10-28T18:02:34  <sipa> gmaxwell: turning on mempool checks are a certain way to blow away your cache every block
152 2015-10-28T18:02:48  <morcos> the blocks around that time for me took 500ms and then 1ms (only coinbase)
153 2015-10-28T18:04:41  <gmaxwell> in any case, debug logs ran this thing out of space a couple hours ago, so I've restarted it. I'll run without mempool checks to get some good timings.
154 2015-10-28T18:05:02  <gmaxwell> Numers I posted from shortly before the MTL event were about 80ms.
155 2015-10-28T18:06:52  <morcos> 80ms?? hmm...
156 2015-10-28T18:07:47  <sipa> so with a 200 MB mempool i seem to need a 700-1200 MB coincache
157 2015-10-28T18:07:52  <sipa> with matt's latest patch
158 2015-10-28T18:09:22  <sipa> i wonder if we should (as a short term hack) treat some factor of the size-of-pulled-in-memory of a tx as its txsize
159 2015-10-28T18:11:38  <morcos> gmaxwell:  i looked at some old numbers of mine (couple of months ago) and they were like 500ms on average (during fairly busy time, mid July)
160 2015-10-28T18:11:48  <gmaxwell> morcos: Interesting!
161 2015-10-28T18:12:31  <morcos> sipa: so what do you mean "need"?
162 2015-10-28T18:13:02  <morcos> a 200MB mempool isn't really the right measure right
163 2015-10-28T18:13:06  *** d_t has quit IRC
164 2015-10-28T18:13:52  <sipa> morcos: how do you mean?
165 2015-10-28T18:13:53  <morcos> because as you go from 0 - 200MB you'll pull in a certain amount of txin's.  but as you keep running, youre mempool stays at 200MB, but you'll i think tend to pull in more txin's?   or does matt's patch actually remove txin's no longer requested when a tx gets mined?
166 2015-10-28T18:14:51  <sipa> when a tx gets mined it goes into the cache with dirty bit on
167 2015-10-28T18:15:03  <sipa> so it can't be removed from the cache anymore
168 2015-10-28T18:15:06  <sipa> until a flush
169 2015-10-28T18:15:23  <sipa> but just a 700 MB coincache sounds pretty painful already
170 2015-10-28T18:16:48  <morcos> yes i think 700MB is pretty painful, but we need to think a bit about how to be smarter about it
171 2015-10-28T18:17:49  <sipa> per-txout cache...
172 2015-10-28T18:18:41  <morcos> yeah if it was pertxout, then you'd solve that problem
173 2015-10-28T18:18:55  <morcos> every so often you don't flush your cache, you just write out the dirty entries
174 2015-10-28T18:19:20  <morcos> we could still do that now, except we don't know if that tx should still be in cache b/c of other mempool txs
175 2015-10-28T18:19:46  <sipa> well an lru or random eviction of the utxo set could work too
176 2015-10-28T18:19:55  <sipa> the cache would just become less effective
177 2015-10-28T18:23:30  *** rubensayshi has quit IRC
178 2015-10-28T18:25:30  <morcos> so when we flush the cache, we have to write everything correct?  so its consistent.  i think if we could jsut get smarter about what we wiped from the cache at that point, then maybe we could jsut flush a lot more often, or do you think that would be bad.
179 2015-10-28T18:26:09  <sipa> yes, a flush shouldn't wipe everything
180 2015-10-28T18:26:44  <morcos> tell me if this would be too cumbersome.  i think it might be pretty fast, how long does a flush take?
181 2015-10-28T18:27:50  <morcos> after you write everything, you quickly scan the top 10MB of txs in the mempool, and insert all of their txin.prevout.hash's into a set, and then you iterate throught the cachemap erasing anything thats not one of those
182 2015-10-28T18:30:57  *** MarcoFalke has quit IRC
183 2015-10-28T18:33:35  <morcos> so actually flushing takes over a second right?  i think you might be able to do something like i suggested on the order of 10's of ms, but not sure
184 2015-10-28T18:33:52  *** MarcoFalke has joined #bitcoin-core-dev
185 2015-10-28T18:35:31  *** MarcoFalke has quit IRC
186 2015-10-28T18:36:11  <morcos> i don't know if it's bad (or maybe its even good) to flush more regularly.  but if we did something like that, we wouldn't even need to worry about matt's patch.  we could just "flush" every time the cache was getting too big.
187 2015-10-28T18:37:40  *** MarcoFalke has joined #bitcoin-core-dev
188 2015-10-28T18:38:41  *** MarcoFalke has quit IRC
189 2015-10-28T18:38:57  *** MarcoFalke has joined #bitcoin-core-dev
190 2015-10-28T18:39:01  *** treehug88 has quit IRC
191 2015-10-28T18:41:06  *** d_t has joined #bitcoin-core-dev
192 2015-10-28T18:43:56  *** PaulCapestany has quit IRC
193 2015-10-28T18:45:42  *** PaulCapestany has joined #bitcoin-core-dev
194 2015-10-28T18:51:58  <morcos> re: TestBlockValidity failing.  I think I'm going to log an error and return NULL.  Seems better than throwing an error.  I'd like to reuse FormatStateMessage (in main.cpp), should I move it to a different file, or just declare it in main.h?
195 2015-10-28T18:54:32  *** MarcoFalke has quit IRC
196 2015-10-28T19:34:29  *** hustler has joined #bitcoin-core-dev
197 2015-10-28T19:38:35  *** hustler has left #bitcoin-core-dev
198 2015-10-28T19:56:19  *** zxzzt has quit IRC
199 2015-10-28T19:56:56  *** sdaftuar has quit IRC
200 2015-10-28T19:57:01  *** morcos has quit IRC
201 2015-10-28T19:58:01  *** zxzzt has joined #bitcoin-core-dev
202 2015-10-28T19:58:26  *** sdaftuar has joined #bitcoin-core-dev
203 2015-10-28T19:58:58  *** morcos has joined #bitcoin-core-dev
204 2015-10-28T20:09:28  *** zooko has quit IRC
205 2015-10-28T20:11:31  *** zooko has joined #bitcoin-core-dev
206 2015-10-28T20:11:39  *** belcher has joined #bitcoin-core-dev
207 2015-10-28T20:45:54  *** CodeShark has joined #bitcoin-core-dev
208 2015-10-28T20:47:41  *** d_t has quit IRC
209 2015-10-28T20:54:57  <Luke-Jr> morcos: ⁇? if the block is invalid you do NOT want to use the template ever
210 2015-10-28T20:55:21  <Luke-Jr> morcos: pretty sure you would entirely break proposals too
211 2015-10-28T20:55:42  <morcos> Luke-Jr: I know you don't want to use it, the question is how to handle that case.
212 2015-10-28T20:56:01  <morcos> It means code is broken somewhere, so human intervention is going to be required at some point
213 2015-10-28T20:56:53  <morcos> The existing code would have thrown an error.  I chose to return NULL and log the error (which will cause getblocktemplate to throw a now misnamed JSONRPC error)
214 2015-10-28T20:57:23  <morcos> Another option would be to try to return a template with no tx's instead (since the likely bug is mempool consistency was broken)
215 2015-10-28T20:57:53  <morcos> I'm about to PR as a WIP... , would be great if you want to take a look
216 2015-10-28T20:58:22  <gmaxwell> morcos: I vaguly recall something that if createnewblock can fail there is a crash elsewher.e
217 2015-10-28T20:58:25  <GitHub181> [bitcoin] morcos opened pull request #6898: [WIP] Rewrite CreateNewBlock (master...fasterCNB) https://github.com/bitcoin/bitcoin/pull/6898
218 2015-10-28T20:58:37  <gmaxwell> or maybe I also fixed that in anticipation of people making it possible to fail again.
219 2015-10-28T20:59:30  <morcos> It took a long time to make it produce the exact same blocks as the old code, but helped work out a couple of bugs.
220 2015-10-28T20:59:41  <Luke-Jr> IMO human intervention is preferable to silently mining empty blocks
221 2015-10-28T21:00:00  <Luke-Jr> and debug.log is not non-silent
222 2015-10-28T21:00:13  <gmaxwell> Most miners will never see anything in debug.log.
223 2015-10-28T21:00:16  <morcos> I really hate the ugliness around hacking in the ability to still do priority space in the blocks.
224 2015-10-28T21:00:54  <Luke-Jr> morcos: if that requires ugliness, the old code was better ?/
225 2015-10-28T21:00:59  <Luke-Jr> :\
226 2015-10-28T21:01:17  <morcos> well, I'd call the old code ugly as well.
227 2015-10-28T21:01:22  <gmaxwell> Luke-Jr: the old code is very ugly.
228 2015-10-28T21:01:47  <Luke-Jr> slow != ugly
229 2015-10-28T21:01:47  <morcos> Also I didn't clean up the whole thing, I just am putting this out there for proof of concept.
230 2015-10-28T21:02:32  <Luke-Jr> but I cant see the new code to compare yet..
231 2015-10-28T21:03:45  <morcos> Luke-Jr: the big problem is priority is very difficult to calculate
232 2015-10-28T21:04:27  <Luke-Jr> ? no
233 2015-10-28T21:04:33  <morcos> If there is a consensus that its an important metric to keep, then I think we should #6357 which would make it much faster to calculate the correct priority in the mining code
234 2015-10-28T21:04:49  <morcos> however it'll still be impossible to keep a sort based on it (i think)
235 2015-10-28T21:05:04  <morcos> it changes!
236 2015-10-28T21:05:09  <Luke-Jr> you need to lookup the inputs anyway
237 2015-10-28T21:05:15  <morcos> not anymore
238 2015-10-28T21:06:18  <morcos> but even if you do, the biggest problem i see with the old code, is that you have to look up the inputs for ALL the txs in your mempool
239 2015-10-28T21:06:24  <morcos> not just the ones you're putting in a block
240 2015-10-28T21:06:40  <Luke-Jr> hmm
241 2015-10-28T21:06:45  <morcos> and if you're going to do a priority portion of the block, you have to keep doing that
242 2015-10-28T21:07:06  <Luke-Jr> resorting once per block seems reasonable imo?
243 2015-10-28T21:07:34  <morcos> although maybe the dynamic priority calculation would fix that, i haven't looked at it in a while..   and maybe it could be made even easier now with the concept of mempool children
244 2015-10-28T21:08:31  <morcos> Luke-Jr: you're right, that would be the way to improve this code if priority isn't going away
245 2015-10-28T21:09:16  <Luke-Jr> morcos: I still plan to redo all this btw :p
246 2015-10-28T21:09:27  <morcos> keep an index (either part of the multi-index or separate) of all the priorities sorted, and only update it once per block
247 2015-10-28T21:09:31  <morcos> yeah me too! :)
248 2015-10-28T21:09:41  <Luke-Jr> so the mempool is a list of block templates
249 2015-10-28T21:10:05  <morcos> but i was hoping we might be able to get something simple'ish done for 0.12, which would make GBT run a lot faster.
250 2015-10-28T21:10:11  <gmaxwell> morcos: the input look up problem exists for fees too. :(
251 2015-10-28T21:10:24  <morcos> gmaxwell: they are already stored in CTxMemPoolEntries
252 2015-10-28T21:10:32  <morcos> and now i addded sigops to that too
253 2015-10-28T21:10:41  <gmaxwell> Oh I see right fees can be cached but priority cannot.
254 2015-10-28T21:10:57  <Luke-Jr> right
255 2015-10-28T21:11:49  <Luke-Jr> priority is our best metric right now I think, so I wouldnt want to lose ikt even temporarily
256 2015-10-28T21:12:39  <morcos> best metric for what?  why do you think its better than fees?
257 2015-10-28T21:12:54  <gmaxwell> Luke-Jr: I don't really agree there. Priority works fine for you and I, I don't think he serves most users all that well.
258 2015-10-28T21:13:11  <Luke-Jr> morcos: spammers are happy to pay fees
259 2015-10-28T21:13:34  <morcos> Luke-Jr: yeah i actually agree its a pretty good anti-spam mechanism
260 2015-10-28T21:13:38  <morcos> but thats not how we use it now!
261 2015-10-28T21:14:01  <Luke-Jr> we do both now
262 2015-10-28T21:14:37  <Luke-Jr> gmaxwell: imo thats why it isnt *exclusively* priority
263 2015-10-28T21:14:55  <gmaxwell> Luke-Jr: point was spam goes through currently.
264 2015-10-28T21:14:58  <morcos> Luke-Jr: there is also the problem of incentives
265 2015-10-28T21:15:21  <Luke-Jr> gmaxwell: not via priority…?
266 2015-10-28T21:15:29  <morcos> how do you make miners prefer priority vs fee
267 2015-10-28T21:15:51  <Luke-Jr> morcos: you dont
268 2015-10-28T21:16:23  <phantomcircuit> wumpus, i haven't forgotten about getting you a copy of a corrupted datadir btw
269 2015-10-28T21:16:24  <Luke-Jr> longterm fees are the only realistic metric
270 2015-10-28T21:16:28  <morcos> Luke-Jr: so you view priority as like an HOV lane.  at least some txs will sneak past even if the spam is causing congestion on most of the block
271 2015-10-28T21:16:39  <Luke-Jr> but for now we want to try to keep fees low
272 2015-10-28T21:16:40  <gmaxwell> morcos: and it does have that effect.
273 2015-10-28T21:17:00  <Luke-Jr> morcos: basically
274 2015-10-28T21:17:21  <Luke-Jr> also a nice fallback
275 2015-10-28T21:17:24  *** paveljanik has quit IRC
276 2015-10-28T21:17:26  <morcos> gmaxwell: if we care about preserving that, why don't we just redefine priority, to be your priority at the time the tx was accepted
277 2015-10-28T21:17:50  <morcos> then it can be cached and its easy to reason about and who cares if different nodes/miners calculate it differently
278 2015-10-28T21:17:50  <Luke-Jr> as long as we have priority, every tx can get confirmed *eventually*
279 2015-10-28T21:17:56  <gmaxwell> morcos: we could expect then it'll turn into dead weight in the mempool.
280 2015-10-28T21:18:16  <morcos> oh sorry, thats not what i mean
281 2015-10-28T21:18:17  <morcos> t
282 2015-10-28T21:18:35  <morcos> i meant the priority only depends on your inputs that were confirmed at the time you were accpeted
283 2015-10-28T21:18:51  <morcos> so its still a bit complicated, but way less than currently
284 2015-10-28T21:19:12  <phantomcircuit> a better question is, why would miners use priority?
285 2015-10-28T21:19:13  <gmaxwell> I think that would be fine.  Then it only needs to update by some ratio of the size, I guess.
286 2015-10-28T21:19:16  <Luke-Jr> that may work good enough as a temporary thing
287 2015-10-28T21:19:53  <Luke-Jr> phantomcircuit: already had that minidiscussion scroll up
288 2015-10-28T21:21:39  <morcos> gmaxwell: but yeah, i mean even easier, lets just make it not age once its in your mempool.  if you guessed wrong and it doesn't get confirmed soonish, then you resubmit after it expires (currently 72 hours)
289 2015-10-28T21:22:47  <Luke-Jr> morcos: maybe have a post-block resort for new confirmed inputs optionally enabled in conf file
290 2015-10-28T21:22:49  <Luke-Jr> just in case it turns out bad
291 2015-10-28T21:22:53  <Luke-Jr> re solrt I mean
292 2015-10-28T21:23:15  <Luke-Jr> re sort sorry
293 2015-10-28T21:23:34  <gmaxwell> morcos: thats a little obnoxious, in that if it doesn't make it in the first block then immediately anything new added has an advantage. Maybe it okay?
294 2015-10-28T21:25:03  <morcos> gmaxwell: yeah i think its a tradeoff.  the annoying thing however is i'm not sure you can combine it into one score very easily and still have it serve the purpose you want it to serve
295 2015-10-28T21:25:59  <Luke-Jr> afk
296 2015-10-28T21:26:07  <gmaxwell> I supposed if we cared more we could have a background task that just goes and recosts them from time to time.,
297 2015-10-28T21:26:50  <gmaxwell> presumably we're doing some kind of linear scan for the expiration? (I haven't kept up with the latest changes)
298 2015-10-28T21:33:50  <morcos> gmaxwell: expiration?  oh no, there is an entry time index as well.
299 2015-10-28T21:45:16  *** evoskuil has joined #bitcoin-core-dev
300 2015-10-28T21:48:20  *** zooko` has joined #bitcoin-core-dev
301 2015-10-28T21:49:25  *** zooko has quit IRC
302 2015-10-28T21:51:42  *** zooko` is now known as zooko
303 2015-10-28T21:55:00  <phantomcircuit> gmaxwell, im seeing ~200ms on average for Connect total on my server for the last two months
304 2015-10-28T22:00:06  <gmaxwell> morcos: so... perhaps another way to handle priority is to maintain a seperate very small mempool for it.
305 2015-10-28T22:00:35  <gmaxwell> so then the cost of having to update and resort the 'whole mempool' is not very lage.
306 2015-10-28T22:00:38  <gmaxwell> er large*
307 2015-10-28T22:04:35  *** ParadoxSpiral has quit IRC
308 2015-10-28T22:08:19  *** zooko has quit IRC
309 2015-10-28T22:12:14  *** belcher has quit IRC
310 2015-10-28T22:14:18  *** d_t has joined #bitcoin-core-dev
311 2015-10-28T22:16:23  <phantomcircuit> gmaxwell, why not just mark transaction sin the mempool as dirty when there's a new block and update the priority in a background thread?
312 2015-10-28T22:17:42  *** belcher has joined #bitcoin-core-dev
313 2015-10-28T22:40:42  <sipa> I'm probably wasting my time, but I'm writing a more (space) efficient std::vector drop-in replacement
314 2015-10-28T22:41:02  <sipa> a vector has an average overhead of 40 bytes currently...
315 2015-10-28T22:42:08  <gmaxwell> sipa: probably; ... maybe a more useful thing to do is to find things using vectors for small amounts of data and make them not use vectors. :P
316 2015-10-28T22:42:55  <gmaxwell> sipa: question;  I can't figure out why we would need to cache failing signatures at all:  rationale: constructing a new novel failing signature is free.
317 2015-10-28T22:42:56  <sipa> gmaxwell: the idea is to use a union to either store elements directly, or store the size + pointer of a dynamic array
318 2015-10-28T22:43:17  <sipa> gmaxwell: we cache failing transactions, not signatures
319 2015-10-28T22:43:32  <sipa> to prevent downloading and verifying them over and over again
320 2015-10-28T22:44:32  <gmaxwell> that was just a brainfart, for some reason I thought the sigcache type included a validity flag.
321 2015-10-28T22:44:43  <gmaxwell> of course it doesn't.
322 2015-10-28T22:45:41  <gmaxwell> sipa: I've wondered before why the normal std::vector didn't special case small numbers of objects by making one of the pointers null and storing the objects internally.
323 2015-10-28T22:47:28  <gmaxwell> sipa: even if you superoptimize the vector, there is still malloc overhead and fragmentation and the pointer to the vector object in its parent object which can all be eliminated in cases where the vector can be avoided.
324 2015-10-28T22:48:33  <sipa> gmaxwell: well for example almost every CScript (which is a std::vector<unsigned char>) in the utxo cache and mempool stores at least 25 bytes
325 2015-10-28T22:49:14  <gmaxwell> Every vector<char> is probably suspect to begin with. :P
326 2015-10-28T22:49:23  <sipa> i can make a vector that is exactly 32 bytes (4 bytes size + 28 bytes data, or 4 bytes size + 4 bytes alloced size + pointer to actual data)
327 2015-10-28T22:50:06  <gmaxwell> It might be interesting to instrument std::vector so we can get a report of vectors in the codebase that have a constant number of objects in them.
328 2015-10-28T22:50:23  <sipa> so it's 32B for up to 28 bytes of data, or 32B + N for N > 28 bytes of data
329 2015-10-28T22:51:17  <sipa> instead of vector's 40B + N for everything
330 2015-10-28T22:52:50  <gmaxwell> sipa: or you could make the parent object able to store up to 32 bytes internally, and only create a vector if there is more than that. (I suppose it could use a union with the char[32] to store the pointer to the vector when there is one).
331 2015-10-28T22:53:00  <sipa> gmaxwell: that's what i'm doing
332 2015-10-28T22:53:50  <sipa> when i say "vector" i mean parent type; i think you're interpreting that word as "malloced data"
333 2015-10-28T22:54:05  <gmaxwell> Got it.
334 2015-10-28T22:55:00  <gmaxwell> why 28? why not 31?  you don't need a 4 byte length for only 31 bytes of range.
335 2015-10-28T22:56:13  <sipa> it complicates things a bit if you can't share the size for one with the other
336 2015-10-28T22:56:35  <sipa> but yes, indeed
337 2015-10-28T22:58:30  <gmaxwell> ah I see, actually that also lets you avoid a flag.. since you can switch the behavior on size.
338 2015-10-28T22:59:23  <sipa> i thought about that first, but that's unfortunately not the case
339 2015-10-28T22:59:39  <sipa> as vector guarantees that a resize that shrinks doesn't invalidate pointers
340 2015-10-28T23:00:01  <sipa> so dropping from over the threshold to below the threshold cannot switch to in-object storage
341 2015-10-28T23:01:15  <sipa> so i use 1 bit of the 4-byte size to determine what type of storage to use
342 2015-10-28T23:02:08  <gmaxwell> what I was thinking about when I said changing the datastructure was just flattening. e.g. one dynamically sized txout object that contains everything in it directly, and no pointers at all. It's not like we ever modify any of this except setting flags.
343 2015-10-28T23:02:26  <gmaxwell> so having any pointers in it under any condition, is just waste and overhead.
344 2015-10-28T23:02:56  <gmaxwell> but it means that there needs to be smart accessors for it.
345 2015-10-28T23:49:46  <GitHub107> [bitcoin] mcelrath opened pull request #6899: Warnings clean with C++11 (master...cpp11) https://github.com/bitcoin/bitcoin/pull/6899