1 2016-07-29T00:04:16  *** nets1n has joined #bitcoin-core-dev
  2 2016-07-29T00:04:38  *** nets1n has joined #bitcoin-core-dev
  3 2016-07-29T00:17:32  <cfields> NicolasDorier: hmm, I hadn't seen your hash cache stuff yet
  4 2016-07-29T00:18:01  <cfields> I had been working on something very similar for the purpose of making the sigcache lock free
  5 2016-07-29T00:18:09  <cfields> (after seeing morcos's numbers and discussing with jeremyrubin)
  6 2016-07-29T00:19:07  <NicolasDorier> cfields: why is it so important to be lock free ?
  7 2016-07-29T00:19:10  <cfields> I was going to wait to discuss with Jeremy since it was his idea, but I suppose I should push it up now for discussion. It'd be a shame if we collided with eachother and missed out on the possible speedup
  8 2016-07-29T00:19:44  <cfields> NicolasDorier: it's not, but morcos reported very significant lock contention with 16 cores. and it's not actually necessary to lock it
  9 2016-07-29T00:20:34  <NicolasDorier> for the cached hash make, the lock is tacken very briefly for a lookup in a map
 10 2016-07-29T00:20:40  <cfields> NicolasDorier: sec, I'll push up my (not-fully-vetted) work
 11 2016-07-29T00:20:41  <NicolasDorier> cached hash map
 12 2016-07-29T00:20:47  <NicolasDorier> ok
 13 2016-07-29T00:25:49  <cfields> NicolasDorier: https://github.com/theuni/bitcoin/tree/sigcache-speedup
 14 2016-07-29T00:26:13  <cfields> NicolasDorier: much of it is quite similar to your work :)
 15 2016-07-29T00:27:38  <cfields> NicolasDorier: if i can convince morcos to bench that and there's no real improvement, I'll drop it. Otherwise, we might want to merge approaches a bit
 16 2016-07-29T00:28:16  <sipa> alternatively, for just this, we can just comoute the 3 cached hashes before doing any validation
 17 2016-07-29T00:28:21  <NicolasDorier> oh cool I'll take a look, in my PR I have a benchmark code if you need
 18 2016-07-29T00:28:24  <sipa> and then not have any locking at all
 19 2016-07-29T00:28:26  <NicolasDorier> we can compare easily
 20 2016-07-29T00:29:06  *** wumpus has quit IRC
 21 2016-07-29T00:30:55  <cfields> that also has the advantage of making the sigcache usable by libbitcoinconsensus in a way that's thread-friendly, but I'm not sure how necessary/desirable that is
 22 2016-07-29T00:31:41  <NicolasDorier> cfields: are we talkign about the same thing ? I was talking about midhash caches of segwit, not sig cache in fact
 23 2016-07-29T00:31:48  <cfields> NicolasDorier: that's great. Sorry for not poking at it further before chiming in, making dinner atm. I'll review yours more carefully tomorrow
 24 2016-07-29T00:31:50  <NicolasDorier> looking at your PR you are on the sig cache
 25 2016-07-29T00:32:35  <cfields> NicolasDorier: yes, we're talking about different things. But I think we may want to approach them the same way.
 26 2016-07-29T00:33:02  <NicolasDorier> ah yes indeed I'll check that.
 27 2016-07-29T00:34:26  <cfields> like I said, I just took a quick glimpse and wanted to push it up for discussion before it was too late. I'll have a good look at the hash cache tomorrow :)
 28 2016-07-29T00:35:10  <cfields> bbl
 29 2016-07-29T00:35:49  *** Ylbam has quit IRC
 30 2016-07-29T00:36:25  <NicolasDorier> cool thanks !
 31 2016-07-29T00:39:30  *** wumpus has joined #bitcoin-core-dev
 32 2016-07-29T00:44:14  *** nets1n has quit IRC
 33 2016-07-29T00:45:12  *** Giszmo has quit IRC
 34 2016-07-29T00:47:06  <morcos> NicolasDorier: cfields: jeremyrubin: I'm happy to bench anything.
 35 2016-07-29T00:47:59  <morcos> What I've been working with so far is just a very small change to sigcache that does the erases at the end of block validation (since thats the only thing we need the lock for) and i'm using a boost lock free queue for that
 36 2016-07-29T00:48:15  <morcos> of course it would be easy'ish to make a custom version
 37 2016-07-29T00:49:09  <morcos> jeremy has been hard at work at removing the locking from checkqueue.  i know at least that the time to finish connecting all the transactions (and populating the checkqueue) is much slower due to lock contention..  so its a guess that possibly there is a lot of improvement in the actual validation too
 38 2016-07-29T00:49:24  <morcos> cfields, if your code is ready i'm happy to try that out tomorrow
 39 2016-07-29T00:50:51  <morcos> i've combined this with an approach to keep an always hot CCoinsViewCache on top of pcoinstip instead of creating a new view in ConnectBlock.  This both saves a lot of copying if things are already in pcoinstip and serves to keep things loaded from disk.
 40 2016-07-29T00:51:39  <morcos> I do this by running CreateNewBlock once every 30 seconds.  and then pulling into that hot cache the inputs that would be needed to validate that block
 41 2016-07-29T00:53:34  <morcos> I think this makes it unnecessary to ever have a dbcache bigger than about 300M (although you need up to about 150M additional for the hot cache, but some of that memory you would have been using anyway in block validation)
 42 2016-07-29T00:55:10  <CodeShark> there seems to be a phishing attack going on where you get an email that seems to come from xapo and you can get 10 bitcoins immediately if you download and install a wallet
 43 2016-07-29T00:55:21  <CodeShark> where by wallet, I mean malware
 44 2016-07-29T00:55:40  <morcos> cfields: here is what i have been using: https://github.com/morcos/bitcoin/commit/3e8a10aa78a8a84a44eca9350178ecf7c308c21c
 45 2016-07-29T00:57:56  <NicolasDorier> morcos: is there some kind of free lock map so I can get rid of lock in my hash cache map ?
 46 2016-07-29T00:58:22  <NicolasDorier> ah also, those cache objects might find their way in libconsensus, we'll be able to reference boost ?
 47 2016-07-29T00:58:48  <morcos> NicolasDorier: i don't think boost is the right end decision, and i think lock free coding is beyond the level i feel very comfortable doing myself
 48 2016-07-29T00:58:59  <morcos> so thats why i'm offering to test other peoples implementations
 49 2016-07-29T00:59:11  <morcos> but my goal here is to optimize current bitcoin core performance
 50 2016-07-29T00:59:20  <morcos> how we make that work with libconsensus is a bit of a separate issue
 51 2016-07-29T01:00:12  <morcos> without seeing a design document or us all agreeing on a roadmap for libconsensus, i don't feel able to help with that.  i've been saying that for a long time now.  i think its a worthy goal, but approaching piecemeal like this is frustrating.
 52 2016-07-29T01:04:07  *** Chris_Stewart_5 has quit IRC
 53 2016-07-29T01:04:19  <cfields> morcos: mine's a little different....
 54 2016-07-29T01:05:05  <cfields> morcos: jeremy made the observation that we only ever work with the sigcache from one thread at a time. ignoring the fact that _that_ thread spawns threads that trample on eachother
 55 2016-07-29T01:05:48  <cfields> so the idea is for each of the workers to return what they would have cached, then batch cache/uncache after each block
 56 2016-07-29T01:06:35  <morcos> cfields: thats what that commit i showed you does
 57 2016-07-29T01:06:37  <cfields> rather, the workers return a list of cache hits/misses. the caller can then add the misses and optionally purge the hits (as in the case of connecting a real block)
 58 2016-07-29T01:06:45  <morcos> you dont' actually cache anything in validation
 59 2016-07-29T01:06:48  <morcos> its only uncaching
 60 2016-07-29T01:07:00  <morcos> you dont' add misses in validation
 61 2016-07-29T01:07:08  <morcos> only in ATMP, which isn't multithreaded
 62 2016-07-29T01:07:19  <morcos> although good to make that work so we can make ATMP multi threaded
 63 2016-07-29T01:07:28  *** Chris_Stewart_5 has joined #bitcoin-core-dev
 64 2016-07-29T01:07:30  <morcos> anyway, is that branch you have something thats ready to be tested
 65 2016-07-29T01:07:34  <morcos> thats easy enough for me to do
 66 2016-07-29T01:08:47  <morcos> got to go, will be online tomorrow
 67 2016-07-29T01:09:09  <cfields> morcos: hmm, ok, I'll reread when I'm not preoccupied. I just saw it setting from get()
 68 2016-07-29T01:09:37  <cfields> morcos: that'd be great, thanks. I'll read yours more carefully along with NicolasDorier's
 69 2016-07-29T01:10:30  <NicolasDorier> cfields: wait tomorrow for reviewing my PR, I'm changing some stuff I got from sipa's feedback
 70 2016-07-29T01:15:30  <cfields> NicolasDorier: roger
 71 2016-07-29T01:16:08  <cfields> morcos: ah, heh, the diff threw me off
 72 2016-07-29T01:27:27  *** Alopex has quit IRC
 73 2016-07-29T01:28:32  *** Alopex has joined #bitcoin-core-dev
 74 2016-07-29T01:42:18  *** TomMc has quit IRC
 75 2016-07-29T01:56:20  *** nets1n has joined #bitcoin-core-dev
 76 2016-07-29T01:57:41  <GitHub165> [bitcoin] NicolasDorier opened pull request #8422: Cache hashes (master...cachedhashes2) https://github.com/bitcoin/bitcoin/pull/8422
 77 2016-07-29T01:58:36  <GitHub146> [bitcoin] NicolasDorier closed pull request #8259: Hash Cache (master...cachedhashes) https://github.com/bitcoin/bitcoin/pull/8259
 78 2016-07-29T01:58:47  <NicolasDorier> cfields: when you feel like reviewing cached hashes : https://github.com/bitcoin/bitcoin/pull/8422
 79 2016-07-29T02:00:56  *** a87ry5 has joined #bitcoin-core-dev
 80 2016-07-29T02:01:04  *** nets1n has quit IRC
 81 2016-07-29T02:02:46  *** fengling has quit IRC
 82 2016-07-29T02:08:12  *** nets1n has joined #bitcoin-core-dev
 83 2016-07-29T02:09:01  *** Alopex has quit IRC
 84 2016-07-29T02:10:06  *** Alopex has joined #bitcoin-core-dev
 85 2016-07-29T02:12:27  *** a87ry5 has quit IRC
 86 2016-07-29T02:13:30  *** fengling has joined #bitcoin-core-dev
 87 2016-07-29T02:26:01  *** pmienk has quit IRC
 88 2016-07-29T02:29:14  *** nets1n has quit IRC
 89 2016-07-29T02:34:33  *** Chris_Stewart_5 has quit IRC
 90 2016-07-29T02:38:06  *** pmienk has joined #bitcoin-core-dev
 91 2016-07-29T02:42:01  *** Alopex has quit IRC
 92 2016-07-29T02:43:06  *** Alopex has joined #bitcoin-core-dev
 93 2016-07-29T02:52:04  *** supasonic has quit IRC
 94 2016-07-29T03:00:38  <GitHub111> [bitcoin] fanquake opened pull request #8423: [depends] expat 2.2.0, ccache 3.2.7 (master...expat-ccache-jul) https://github.com/bitcoin/bitcoin/pull/8423
 95 2016-07-29T03:01:15  *** fanquake has joined #bitcoin-core-dev
 96 2016-07-29T03:01:40  <fanquake> morning
 97 2016-07-29T03:18:01  *** Alopex has quit IRC
 98 2016-07-29T03:19:07  *** Alopex has joined #bitcoin-core-dev
 99 2016-07-29T03:21:29  *** molz has joined #bitcoin-core-dev
100 2016-07-29T03:23:14  *** moli has quit IRC
101 2016-07-29T03:24:02  *** supasonic has joined #bitcoin-core-dev
102 2016-07-29T03:38:01  *** Alopex has quit IRC
103 2016-07-29T03:38:42  *** fanquake has left #bitcoin-core-dev
104 2016-07-29T03:39:06  *** Alopex has joined #bitcoin-core-dev
105 2016-07-29T03:44:41  *** goatpig has quit IRC
106 2016-07-29T03:49:06  *** Alopex has quit IRC
107 2016-07-29T03:50:12  *** Alopex has joined #bitcoin-core-dev
108 2016-07-29T03:59:59  *** d_t has joined #bitcoin-core-dev
109 2016-07-29T04:00:36  *** d_t has joined #bitcoin-core-dev
110 2016-07-29T04:02:57  *** supasonic has quit IRC
111 2016-07-29T04:07:07  *** Alopex has quit IRC
112 2016-07-29T04:08:12  *** Alopex has joined #bitcoin-core-dev
113 2016-07-29T04:23:21  *** Alopex has quit IRC
114 2016-07-29T04:24:26  *** Alopex has joined #bitcoin-core-dev
115 2016-07-29T04:24:31  *** anu0 has joined #bitcoin-core-dev
116 2016-07-29T04:28:05  *** anu1 has quit IRC
117 2016-07-29T04:41:43  *** justanotheruser has quit IRC
118 2016-07-29T04:42:11  *** justanotheruser has joined #bitcoin-core-dev
119 2016-07-29T04:43:06  *** fengling has quit IRC
120 2016-07-29T04:47:02  *** Alopex has quit IRC
121 2016-07-29T04:48:07  *** Alopex has joined #bitcoin-core-dev
122 2016-07-29T04:55:53  *** d_t has quit IRC
123 2016-07-29T05:09:46  *** fengling has joined #bitcoin-core-dev
124 2016-07-29T05:13:07  <luke-jr> jtimon: the issue is entirely giving bad advice to miners. regardless of what they might or might not configure in the end, we should not give bad advice
125 2016-07-29T05:13:37  <luke-jr> (btw, I see no reason the performance "hit" of using maxsize would be greater than the performance hit of using 0.13 over 0.12..)
126 2016-07-29T05:14:26  *** fengling has quit IRC
127 2016-07-29T05:22:24  *** kadoban has quit IRC
128 2016-07-29T05:25:21  *** Alopex has quit IRC
129 2016-07-29T05:26:26  *** Alopex has joined #bitcoin-core-dev
130 2016-07-29T05:40:06  *** fengling has joined #bitcoin-core-dev
131 2016-07-29T05:51:40  *** LeMiner2 has quit IRC
132 2016-07-29T05:52:20  *** LeMiner2 has joined #bitcoin-core-dev
133 2016-07-29T06:06:27  <GitHub151> [bitcoin] PrinceofOrange opened pull request #8424: merge (0.8...master) https://github.com/bitcoin/bitcoin/pull/8424
134 2016-07-29T06:08:33  <GitHub87> [bitcoin] laanwj closed pull request #8424: merge (0.8...master) https://github.com/bitcoin/bitcoin/pull/8424
135 2016-07-29T06:09:59  <luke-jr> O.o
136 2016-07-29T06:12:53  <wumpus> yes that's the only possible response - this seems deliberate spammyness, not merely an accident
137 2016-07-29T06:21:37  <GitHub182> [bitcoin] laanwj pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/ad087638ee48...842bf8d2c5af
138 2016-07-29T06:21:38  <GitHub182> bitcoin/master 1de2a46 Suhas Daftuar: Ignore GETBLOCKTXN requests for unknown blocks...
139 2016-07-29T06:21:38  <GitHub182> bitcoin/master 1d06e49 Suhas Daftuar: Ignore CMPCTBLOCK messages for pruned blocks...
140 2016-07-29T06:21:39  <GitHub182> bitcoin/master 842bf8d Wladimir J. van der Laan: Merge #8408: Prevent fingerprinting, disk-DoS with compact blocks...
141 2016-07-29T06:21:47  <GitHub16> [bitcoin] laanwj closed pull request #8408: Prevent fingerprinting, disk-DoS with compact blocks (master...cb-misbehaving) https://github.com/bitcoin/bitcoin/pull/8408
142 2016-07-29T06:28:39  <GitHub2> [bitcoin] laanwj pushed 1 new commit to 0.13: https://github.com/bitcoin/bitcoin/commit/b7e201181bcc0f6328e0a499803f1dbb2c2dbd28
143 2016-07-29T06:28:39  <GitHub2> bitcoin/0.13 b7e2011 Suhas Daftuar: Prevent fingerprinting, disk-DoS with compact blocks...
144 2016-07-29T06:39:07  *** Alopex has quit IRC
145 2016-07-29T06:40:12  *** Alopex has joined #bitcoin-core-dev
146 2016-07-29T06:41:22  *** molly has joined #bitcoin-core-dev
147 2016-07-29T06:44:10  *** molz has quit IRC
148 2016-07-29T07:02:29  *** BashCo has quit IRC
149 2016-07-29T07:23:08  *** BashCo has joined #bitcoin-core-dev
150 2016-07-29T07:24:33  *** cdecker has joined #bitcoin-core-dev
151 2016-07-29T07:39:00  *** cdecker has quit IRC
152 2016-07-29T07:39:06  *** Alopex has quit IRC
153 2016-07-29T07:40:12  *** Alopex has joined #bitcoin-core-dev
154 2016-07-29T07:40:24  *** cdecker has joined #bitcoin-core-dev
155 2016-07-29T07:40:41  *** cdecker has quit IRC
156 2016-07-29T07:44:40  *** molly has quit IRC
157 2016-07-29T07:50:47  <GitHub65> [bitcoin] laanwj pushed 1 new commit to 0.13: https://github.com/bitcoin/bitcoin/commit/b06808c58eb7a997c42b55cba63688aec448a0ea
158 2016-07-29T07:50:47  <GitHub65> bitcoin/0.13 b06808c Wladimir J. van der Laan: doc: Release notes update for rc2
159 2016-07-29T07:56:48  <GitHub72> [bitcoin] laanwj pushed 1 new commit to 0.13: https://github.com/bitcoin/bitcoin/commit/ced6c940da35fcf33160d1c7f2f54a99dc7eedb1
160 2016-07-29T07:56:48  <GitHub72> bitcoin/0.13 ced6c94 Wladimir J. van der Laan: qt: Translations update pre-rc2
161 2016-07-29T07:59:32  *** Guyver2 has joined #bitcoin-core-dev
162 2016-07-29T08:24:46  *** Giszmo has joined #bitcoin-core-dev
163 2016-07-29T08:35:36  <wumpus>  * [new tag]         v0.13.0rc2 -> v0.13.0rc2
164 2016-07-29T08:54:13  *** Ylbam has joined #bitcoin-core-dev
165 2016-07-29T09:07:06  *** fanquake has joined #bitcoin-core-dev
166 2016-07-29T09:17:43  *** Guyver2_ has joined #bitcoin-core-dev
167 2016-07-29T09:18:04  *** Guyver2 has quit IRC
168 2016-07-29T09:18:37  *** Guyver2_ is now known as Guyver2
169 2016-07-29T09:20:54  *** Guyver2 has quit IRC
170 2016-07-29T09:23:19  *** arubi has quit IRC
171 2016-07-29T09:24:01  *** arubi has joined #bitcoin-core-dev
172 2016-07-29T09:29:25  <GitHub55> [bitcoin] laanwj pushed 4 new commits to master: https://github.com/bitcoin/bitcoin/compare/842bf8d2c5af...b77bb95b3cb4
173 2016-07-29T09:29:26  <GitHub55> bitcoin/master 755aa05 Cory Fields: httpserver: use a future rather than relying on boost's try_join_for
174 2016-07-29T09:29:26  <GitHub55> bitcoin/master d3773ca Cory Fields: httpserver: explicitly detach worker threads...
175 2016-07-29T09:29:27  <GitHub55> bitcoin/master 7e87033 Cory Fields: httpserver: replace boost threads with std...
176 2016-07-29T09:29:37  <GitHub56> [bitcoin] laanwj closed pull request #8421: httpserver: drop boost (#8023 dependency) (master...http-thread) https://github.com/bitcoin/bitcoin/pull/8421
177 2016-07-29T09:49:16  *** arubi_ has joined #bitcoin-core-dev
178 2016-07-29T09:53:08  *** arubi has quit IRC
179 2016-07-29T10:03:39  *** laurentmt has joined #bitcoin-core-dev
180 2016-07-29T10:04:25  *** laurentmt has quit IRC
181 2016-07-29T10:11:52  *** anu0 has quit IRC
182 2016-07-29T10:12:29  *** mkarrer has quit IRC
183 2016-07-29T10:31:03  *** anu0 has joined #bitcoin-core-dev
184 2016-07-29T10:33:10  <GitHub12> [bitcoin] laanwj pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/b77bb95b3cb4...7a2d40272717
185 2016-07-29T10:33:11  <GitHub12> bitcoin/master 695041e Wladimir J. van der Laan: util: Update tinyformat...
186 2016-07-29T10:33:12  <GitHub12> bitcoin/master a5072a7 Wladimir J. van der Laan: util: Remove zero-argument versions of LogPrint and error...
187 2016-07-29T10:33:12  <GitHub12> bitcoin/master 7a2d402 Wladimir J. van der Laan: Merge #8274: util: Update tinyformat...
188 2016-07-29T10:33:16  <GitHub181> [bitcoin] laanwj closed pull request #8274: util: Update tinyformat (master...2016_06_update_tinyformat) https://github.com/bitcoin/bitcoin/pull/8274
189 2016-07-29T10:35:12  *** owowo has quit IRC
190 2016-07-29T10:42:58  *** molly has joined #bitcoin-core-dev
191 2016-07-29T10:51:06  *** fengling has quit IRC
192 2016-07-29T10:52:46  <fanquake> Looks like sigs match @wumpus
193 2016-07-29T10:53:38  <fanquake> Have you noticed quite a lot of output during the Windows build now?
194 2016-07-29T10:53:46  <fanquake> Along the lines of warning: conflicts with previous declaration ‘void boost::signals2
195 2016-07-29T11:03:50  <wumpus> good that sigs match!
196 2016-07-29T11:04:07  <wumpus> no, haven't noticed
197 2016-07-29T11:04:35  <fanquake> I'll get the logs and open an issue.
198 2016-07-29T11:04:40  <wumpus> thanks
199 2016-07-29T11:13:19  *** owowo has joined #bitcoin-core-dev
200 2016-07-29T11:23:28  *** BashCo_ has joined #bitcoin-core-dev
201 2016-07-29T11:26:01  *** fanquake has quit IRC
202 2016-07-29T11:26:26  *** BashCo has quit IRC
203 2016-07-29T11:53:32  *** slackircbridge has quit IRC
204 2016-07-29T11:54:07  *** slackircbridge has joined #bitcoin-core-dev
205 2016-07-29T11:55:18  *** mkarrer has joined #bitcoin-core-dev
206 2016-07-29T12:14:19  *** Chris_Stewart_5 has joined #bitcoin-core-dev
207 2016-07-29T12:24:51  *** Chris_Stewart_5 has quit IRC
208 2016-07-29T12:41:04  <morcos> luke-jr: i was just discussing with sdaftuar yesterday, i'm not sure there is much of a perf hit, it just hasn't been benched yet.  i may try to do that today.  why does 0.13 have worse performance than 0.12, i don't think thats true.
209 2016-07-29T12:50:56  <michagogo> The "native_biplist" has a confusing name
210 2016-07-29T12:51:14  <morcos> luke-jr: ugh, sorry. i keep getting confused.  i guess there are two ways it could get slower, and my anecdotal evidence was only that the extra serialization wasn't that bad.  but the repeated failing to find a final tx is blockmaxsize is your constraining factor seems like it could be slow, especially on a big mempool
211 2016-07-29T12:51:26  <michagogo> My first thought was, wtf? Why is a list of BIPs its own package?
212 2016-07-29T12:51:52  <michagogo> Then I clicked it and saw it was downloading from PyPI, and that was even more confusing... until I looked up the package description
213 2016-07-29T12:52:48  <morcos> anyway, too much annoyance to go test all these posibilities..  i think its a clearly correct recommendation that until segwit activates using only blockmaxweight is the desired configuration.  miners will be upgrading anyway for segwit, so there is a chance to argue something different for that
214 2016-07-29T12:52:56  <morcos> lets concentrate on getting 0.13 right.
215 2016-07-29T12:56:52  *** TomMc has joined #bitcoin-core-dev
216 2016-07-29T12:58:39  *** TomMc has quit IRC
217 2016-07-29T13:00:33  *** cryptapus_afk is now known as cryptapus
218 2016-07-29T13:04:49  <sdaftuar> cfields: i'm looking at your crash from yesterday. seems odd! can you give some more color on what the environment was: mainnet/testnet/regtest? fresh node or already synced?  what commit were you running?
219 2016-07-29T13:08:34  *** [b__b] has quit IRC
220 2016-07-29T13:08:59  *** [b__b] has joined #bitcoin-core-dev
221 2016-07-29T13:44:23  <instagibbs> is it expected behavior for a reindex to not show blocks being processed for while a while?
222 2016-07-29T13:44:38  <instagibbs> in getinfo at least
223 2016-07-29T13:44:54  <instagibbs> I seem to recall different behavior
224 2016-07-29T13:57:06  *** TomMc has joined #bitcoin-core-dev
225 2016-07-29T13:59:12  *** instagibbs has quit IRC
226 2016-07-29T14:14:05  <cfields> sdaftuar: uhmm, from my http-thread branch, which was merged today
227 2016-07-29T14:14:19  <cfields> sdaftuar: so current master should be basically the same thing
228 2016-07-29T14:15:26  <cfields> checking my backlog to see how i was running
229 2016-07-29T14:17:08  <cfields> sdaftuar: ah, a funky config, forgot I was running that way. probably significant...
230 2016-07-29T14:17:15  <cfields> gdb --args ./bitcoind -testnet -printtoconsole -reindex-chainstate -debug=http
231 2016-07-29T14:17:51  <cfields> the -reindex-chainstate was accidental, left in there from testing something before.
232 2016-07-29T14:24:21  <morcos> cfields: i have some results.  here are the average times per block for  Connect transactions/Verify txins/Connect block   (meausres are total time so far as reported in bench)
233 2016-07-29T14:24:31  <morcos> in ms
234 2016-07-29T14:24:46  <morcos> master: 26 / 80 / 111
235 2016-07-29T14:25:11  <morcos> your sigcache: 39 / 46 / 82
236 2016-07-29T14:25:45  <morcos> batched deletes using boost lock free (commit i showed you): 22 / 32 / 63
237 2016-07-29T14:26:53  <cfields> morcos: hmm. that's surprising. that's 16 cores?
238 2016-07-29T14:27:11  <morcos> yeah, 16 cores with 8GB dbcache
239 2016-07-29T14:27:40  <morcos> for yours and the batched deletes test i also included your copy-move improvements and my hot tipcache, but those don't make much difference on master
240 2016-07-29T14:28:04  <cfields> ok
241 2016-07-29T14:28:30  <TomMc> In regards to signature hashtypes, if the hashtype is SIGHASH_ALL, is the last byte given a value of 0x01?
242 2016-07-29T14:28:31  <cfields> i'm still working on the prevector btw, i've almost got that rewritten and specialized for the unsigned char case
243 2016-07-29T14:29:00  <morcos> i'll dig into it more, so see if i can understand why your code slowed down Connect transactions
244 2016-07-29T14:29:00  <cfields> i wonder what throws the connect time off...
245 2016-07-29T14:30:01  <cfields> morcos: thanks a bunch for testing. that's very interesting. I'll look into the connect as well.
246 2016-07-29T14:30:06  *** Chris_Stewart_5 has joined #bitcoin-core-dev
247 2016-07-29T14:31:37  <cfields> morcos: one more interesting test, if you don't mind
248 2016-07-29T14:31:59  <morcos> cfields: the connect time can be reduced to as low as 10ms if you eliminate the locking contention in checkqueue
249 2016-07-29T14:32:14  <cfields> morcos: run with -par=1 as a baseline. I was surprised to see that it's actually faster on some of my machines that way
250 2016-07-29T14:33:06  <morcos> :) i know!  how i found this stuff in the first place was running with blocks only mode was actually faster than running with tx, implying using the sigcache was slower than verifying the sigs!
251 2016-07-29T14:33:16  <cfields> morcos: right. I'm looking forward to pairing those two, I should think they would improve eachother
252 2016-07-29T14:33:33  <cfields> heh, right
253 2016-07-29T14:33:52  <morcos> will try par=1 as well
254 2016-07-29T14:34:29  <cfields> morcos: wait. current master actually _is_ faster with -par=1 on 16 cores?!
255 2016-07-29T14:34:56  <morcos> oh, no i haven't tried that, i was just saying i know in that its not surprising to me that it could be in some configs
256 2016-07-29T14:35:22  <cfields> oh, whew :)
257 2016-07-29T14:35:38  <cfields> looks to me like it breaks even around ~3 as-is
258 2016-07-29T14:36:31  <cfields> but I've tested so many configs now that I can't keep the numbers straight, so that could be wrong
259 2016-07-29T14:41:16  <cfields> morcos: ah, looks like it's the overhead of creating a copy of the to-test CScriptCheck that slows down the connect. that could be much improved
260 2016-07-29T14:41:44  <morcos> oh, that was my guess at first but then i didn't see where that happened b/c i thought it was just shared pointers
261 2016-07-29T14:42:15  <sipa> cfields: before libsecp256k1-based verification, i benchmarked where the break even point was on a large-number-of-cores machine, and set the max value for -par based on that
262 2016-07-29T14:42:25  <morcos> the method of batching deletes is extremely effective
263 2016-07-29T14:42:45  <cfields> morcos: yea, those should be cheap, but I'm assuming it's the constant inserts() that suck
264 2016-07-29T14:42:46  <sipa> when we switched to libsecp validation, the break even point probably dropped a lot
265 2016-07-29T14:43:02  <sipa> cfields: copy? it should be using swap
266 2016-07-29T14:43:03  <cfields> sipa: ah yes, that makes sense then.
267 2016-07-29T14:43:13  <morcos> sipa: depends on whether you are taking advantage of the sigcache a lot or not though
268 2016-07-29T14:43:14  <cfields> sipa: they're shared_ptrs
269 2016-07-29T14:43:51  <sipa> cfields: really? when was that changed?
270 2016-07-29T14:44:01  <morcos> :)
271 2016-07-29T14:44:03  <morcos> its not in master
272 2016-07-29T14:44:06  <cfields> sipa: in the branch we're discussing
273 2016-07-29T14:44:40  <cfields> sipa: https://github.com/bitcoin/bitcoin/compare/master...theuni:sigcache-speedup
274 2016-07-29T14:44:52  <morcos> back in a few
275 2016-07-29T14:45:36  <sipa> ah, ok
276 2016-07-29T14:45:45  <cfields> morcos: that whole vector business (including checkqueue) could be much improved with a list i think. Then we could splice all over the place instead.
277 2016-07-29T14:45:47  <sipa> i thought you were comparing with master
278 2016-07-29T14:46:19  <cfields> sipa: no. discussing different ways to reduce the sigcache lock contention
279 2016-07-29T14:46:26  <sipa> ok ok
280 2016-07-29T14:46:34  <sipa> looking forward to the results
281 2016-07-29T14:48:28  *** d_t has joined #bitcoin-core-dev
282 2016-07-29T14:48:33  <michagogo> cfields: are you by any chance in the process of building rc2?
283 2016-07-29T14:48:36  <morcos> i can't remember if i mentioned this on channel before, but i think we made a mistake in the design of BIP 68
284 2016-07-29T14:48:39  <michagogo> Looks like we have 4 sets of sigs already
285 2016-07-29T14:48:47  <morcos> we should have designed that as a per input check
286 2016-07-29T14:49:17  <cfields> morcos: another route would be to maintain a single list rather than a vector of per-txin hits/misses. But I was afraid that would slow down one of the checks. I'll whip that up for another data point
287 2016-07-29T14:49:22  <morcos> that would have done away with that whole issue of calculating prevheights
288 2016-07-29T14:49:32  <cfields> michagogo: yes, builds in progress
289 2016-07-29T14:49:47  <morcos> cfields: keep in mind you never need both hits and misses
290 2016-07-29T14:49:49  <michagogo> (i.e. are detached sigs imminent? If not, I'll just wait until Saturday night, but if they are I can leave the VM running)
291 2016-07-29T14:49:57  <michagogo> Cool, thanks
292 2016-07-29T14:49:57  <morcos> i made a couple comments on your commit
293 2016-07-29T14:51:22  <morcos> and i actually haven't looked at how this interacts with the segwit style cachehashes that NicolasDorier is working on
294 2016-07-29T14:51:28  <cfields> morcos: well you at least need each txin's misses shared with eachother. for the 2dup..checksig..2drop..checksig case, no?
295 2016-07-29T14:53:01  <morcos> cfields: hmm..  explain a bit more.  you mean if you are checking the same sig multiple times you dont' want to have to redo the actual verfication?
296 2016-07-29T14:53:22  <morcos> we don't solve that problem now do we?
297 2016-07-29T14:53:29  <cfields> morcos: right
298 2016-07-29T14:53:40  <cfields> morcos: i believe so, as they're all cached on-the-fly
299 2016-07-29T14:53:51  <morcos> they aren't cached in ConnectBlock
300 2016-07-29T14:53:59  <morcos> they are only cached in ATMP
301 2016-07-29T14:54:16  <sipa> morcos: sorry, what is 'they' ?
302 2016-07-29T14:54:30  <morcos> signature verification results
303 2016-07-29T14:54:40  <morcos> whats in the existing sigcache
304 2016-07-29T14:54:52  <sipa> ah, yes
305 2016-07-29T14:56:50  <cfields> morcos: hmm, yes, you're right
306 2016-07-29T14:58:00  <sipa> we store results in atmp, delete results in connectblock
307 2016-07-29T14:58:33  <sipa> if you duplicate sigs i guess that indeed will hurt
308 2016-07-29T14:58:42  *** instagibbs has joined #bitcoin-core-dev
309 2016-07-29T15:02:28  <morcos> cfields: but you were talking about sharing a txin's misses with other txin checks()'s ?  thats not necessary right?  it's only within a given txin that you'll have a duplicated signature?
310 2016-07-29T15:03:17  <morcos> seems like if thats a worthwhile performance improvement thats easy enough to add separately from the sigcache..  kind of a separate question
311 2016-07-29T15:04:01  <sipa> cfields, morcos: with nicolasdorier's cache, we could add some sort of accumulator to the cache, which lists all the signatures that were valudated (or even sigcache iterators to them), and then delete them all at once after the whole block is validated
312 2016-07-29T15:04:25  <cfields> morcos: right. atm, they're per-txin. But since there's the overhead of creating a vector/list/whatever for each, I was (thinking outloud) that it may end up quicker to do per-block instead
313 2016-07-29T15:04:44  <cfields> sipa: that's exactly what this branch is doing :)
314 2016-07-29T15:04:55  <cfields> sipa: and morcos's as well
315 2016-07-29T15:04:55  <sipa> great
316 2016-07-29T15:05:01  * sipa shuts up
317 2016-07-29T15:05:31  <morcos> sipa: but right that it should probably be combined with NicolasDorier's cache too, b/c you'll want to batch the deleting of the hashes as well
318 2016-07-29T15:05:36  <cfields> sipa: no, it's helpful. The issue is that that means returning a list of hits/misses for each txin, which has a considerable overhead
319 2016-07-29T15:06:12  <morcos> cfields: yes, but a lock free list is relatively straight forward right... we don't have to use the boost version
320 2016-07-29T15:06:58  <morcos> but can accomplish the same thing ourselves..   so instead of returning the list, we just append to the shared lists of hits and misses
321 2016-07-29T15:07:53  *** aalex has joined #bitcoin-core-dev
322 2016-07-29T15:08:16  <cfields> morcos: sure, but i _really_ like the idea of moving the cache out so that it can be passed in without worry of threading semantics. that lets others (libbitcoinconsensus) use it as well
323 2016-07-29T15:08:41  <cfields> but that's not the strongest argument, so I'd be fine with yours as well
324 2016-07-29T15:10:36  <cfields> morcos: passing it in also means that ATMP can be threaded as well (as you mentioned yesterday). I suppose you could do the same with your approach by adding a similar batch-to-add structure
325 2016-07-29T15:10:46  <morcos> right
326 2016-07-29T15:12:09  <sipa> cfields: one way would be to have a simple preallocated vector in each validation thread
327 2016-07-29T15:12:20  <sipa> and afterwards, all results are combined
328 2016-07-29T15:13:44  <cfields> sipa: i started that way, but was hesitant to re-intruduce contention in the re-combining. Did you have something in mind to avoid that?
329 2016-07-29T15:14:25  <sipa> cfields: i meant recombining only after all txn are validated
330 2016-07-29T15:15:41  <morcos> sipa: yep, that was another approach jeremy and i discussed
331 2016-07-29T15:15:52  <cfields> sipa: ah, so the threads would set a pointer to their thread_local vector at creation, then master could iterate when finished
332 2016-07-29T15:16:07  <morcos> cfields: i'm not sure i understand why your branch is safe to remove all the locks...
333 2016-07-29T15:16:14  <cfields> yes, that would be much better
334 2016-07-29T15:16:54  <cfields> morcos: a const cache is passed in, it's not changing during validation
335 2016-07-29T15:16:59  <morcos> ah, i guess its protected by cs_main?
336 2016-07-29T15:17:16  <morcos> yes, but how did you know another thread isn't modifying it while you are reading from it
337 2016-07-29T15:17:36  <cfields> morcos: right, it needs to be globally protected. atm nothing else touches it. so i suppose it's cs_main :)
338 2016-07-29T15:17:56  <morcos> right a rpc call to submit a tx or something
339 2016-07-29T15:19:22  <cfields> yes
340 2016-07-29T15:19:54  <cfields> ok, I'll try the per-thread suggestion. should be easy to whip up i think
341 2016-07-29T15:32:13  *** I_DID_LSD_ON_A_P has quit IRC
342 2016-07-29T15:32:29  *** gribble has quit IRC
343 2016-07-29T15:39:58  <GitHub38> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/7a2d40272717...bbcb8fd88433
344 2016-07-29T15:39:58  <GitHub38> bitcoin/master 54af51d Jonas Schnelli: [QA] Add walletdump RPC test (including HD- & encryption-tests)
345 2016-07-29T15:39:59  <GitHub38> bitcoin/master bbcb8fd Wladimir J. van der Laan: Merge #8417: [QA] Add walletdump RPC test (including HD- & encryption-tests)...
346 2016-07-29T15:40:10  <GitHub139> [bitcoin] laanwj closed pull request #8417: [QA] Add walletdump RPC test (including HD- & encryption-tests) (master...2016/07/dump_test) https://github.com/bitcoin/bitcoin/pull/8417
347 2016-07-29T15:40:10  *** gribble has joined #bitcoin-core-dev
348 2016-07-29T15:42:05  <luke-jr> morcos: CPFP requires updating stuff as parents get included in the block, which probably isnt *that* expensive, but a heck of a lot more than merely adding numbers (the overhead of maxsize); AFAIK it doesn't do any extra serialization, just adding sizes up
349 2016-07-29T15:42:22  *** zooko has joined #bitcoin-core-dev
350 2016-07-29T15:42:53  <morcos> luke-jr: i believe suhas's bench marks for CPFP actually showed it neglibly faster than the pre CPFP code.  this is b/c you can add a whole package of txs at a time i think.
351 2016-07-29T15:43:52  <GitHub141> [bitcoin] laanwj opened pull request #8427: net: Ignore `notfound` P2P messages (master...2016_07_notfound) https://github.com/bitcoin/bitcoin/pull/8427
352 2016-07-29T15:44:42  <GitHub72> [bitcoin] laanwj closed pull request #8403: Process "notfound" messages, and safeguard against unreasonably long … (master...ProcessNotfound) https://github.com/bitcoin/bitcoin/pull/8403
353 2016-07-29T15:47:21  <luke-jr> morcos: benchmarks with what tx sets? I suspect it would vary significantly depending on the input
354 2016-07-29T15:47:50  <morcos> historical simulation over a default mempool with default policy i think
355 2016-07-29T15:52:38  <luke-jr> what is a default mempool? anyhow, I guess the slower path is unlikely to be used historically
356 2016-07-29T15:52:51  <morcos> 300M
357 2016-07-29T15:53:07  <luke-jr> probably not too likely to be used more with CPFP either
358 2016-07-29T15:53:29  <luke-jr> morcos: oh, but we don't have historical mempool contents
359 2016-07-29T15:54:03  *** Chris_Stewart_5 has quit IRC
360 2016-07-29T15:57:28  <morcos> luke-jr: i'm not sure what you mean.  sdaftuar wrote code to save p2p messages to disk and be able to replay them back off disk to a node.  we've been using that for years to test and bench different changes.
361 2016-07-29T15:58:25  <luke-jr> morcos: oh, I was not aware; so you have years of data like that? :o
362 2016-07-29T15:58:40  <morcos> yep
363 2016-07-29T16:00:28  *** zooko has quit IRC
364 2016-07-29T16:03:10  <morcos> its not a perfect system by any means, but its good for benching and it was very useful for evaluating different fee estimation algorithms
365 2016-07-29T16:07:24  *** Chris_Stewart_5 has joined #bitcoin-core-dev
366 2016-07-29T16:08:58  <Chris_Stewart_5> Where do I add a new unit test file to have it included in test_bitcoin? I looked in test_bitcoin.cpp and didn't find an obvious spot, unless i'm blind
367 2016-07-29T16:09:23  *** zooko has joined #bitcoin-core-dev
368 2016-07-29T16:09:46  <sipa> Chris_Stewart_5: just add it to the makefile
369 2016-07-29T16:10:00  <sipa> there is no explicit list of all tests
370 2016-07-29T16:11:08  <Chris_Stewart_5> Hmm, the README seems a litle misleading then
371 2016-07-29T16:14:49  <luke-jr> then clarify it too while you're at it :p
372 2016-07-29T16:18:35  <Chris_Stewart_5> but it is so much more fun to complain! :-)
373 2016-07-29T16:31:04  *** Chris_Stewart_5 has quit IRC
374 2016-07-29T16:34:21  *** Sosumi has quit IRC
375 2016-07-29T16:36:14  *** Chris_Stewart_5 has joined #bitcoin-core-dev
376 2016-07-29T16:43:05  <cfields> gitian signers: v0.13.0rc2 sigs are pushed
377 2016-07-29T16:43:07  <cfields> michagogo: ping ^^
378 2016-07-29T16:47:53  <sdaftuar> cfields: i can reproduce your segfault.  kind of an interesting case really, though i believe unrelated to #8096.
379 2016-07-29T16:48:29  <sdaftuar> cfields: looks like the issue is that if you ctrl-c on startup, while running with -reindex-chainstate, chainActive will not be initialized
380 2016-07-29T16:48:43  <sdaftuar> because ActivateBestChain returns immediately on ShutdownRequested()
381 2016-07-29T16:48:52  <sdaftuar> that violates an assumption in RewindBlockIndex
382 2016-07-29T16:49:40  *** netzin has joined #bitcoin-core-dev
383 2016-07-29T16:53:37  <GitHub108> [bitcoin] Christewart opened pull request #8428: Update README.md (master...imporve_test_readme) https://github.com/bitcoin/bitcoin/pull/8428
384 2016-07-29T16:53:47  <Chris_Stewart_5> luke-jr: ^ :-)
385 2016-07-29T17:03:11  *** d_t has quit IRC
386 2016-07-29T17:14:17  <cfields> sdaftuar: damn. Yea, completely forgot I was running with reindex-chainstate.
387 2016-07-29T17:14:27  <cfields> sdaftuar: thanks for reproducing
388 2016-07-29T17:24:02  *** YOU-JI has joined #bitcoin-core-dev
389 2016-07-29T17:28:39  *** Chris_Stewart_5 has quit IRC
390 2016-07-29T17:28:53  *** BashCo_ has quit IRC
391 2016-07-29T17:34:13  *** arubi_ is now known as arubi
392 2016-07-29T17:44:35  *** Chris_Stewart_5 has joined #bitcoin-core-dev
393 2016-07-29T17:52:44  *** jgarzik has joined #bitcoin-core-dev
394 2016-07-29T17:59:30  *** adiabat has joined #bitcoin-core-dev
395 2016-07-29T18:02:26  *** BashCo has joined #bitcoin-core-dev
396 2016-07-29T18:03:34  *** netzin has quit IRC
397 2016-07-29T18:11:43  *** netzin has joined #bitcoin-core-dev
398 2016-07-29T18:12:31  *** netzin has joined #bitcoin-core-dev
399 2016-07-29T18:28:53  *** anu1 has joined #bitcoin-core-dev
400 2016-07-29T18:32:07  *** anu0 has quit IRC
401 2016-07-29T18:59:12  *** d_t has joined #bitcoin-core-dev
402 2016-07-29T19:36:35  *** Guyver2 has joined #bitcoin-core-dev
403 2016-07-29T19:42:09  *** jtimon has quit IRC
404 2016-07-29T19:54:33  *** TomMc has quit IRC
405 2016-07-29T19:58:04  *** TomMc has joined #bitcoin-core-dev
406 2016-07-29T20:01:22  *** laurentmt has joined #bitcoin-core-dev
407 2016-07-29T20:08:21  *** zooko has quit IRC
408 2016-07-29T20:15:29  *** ovovo has joined #bitcoin-core-dev
409 2016-07-29T20:16:32  *** owowo has quit IRC
410 2016-07-29T20:17:01  *** jannes has quit IRC
411 2016-07-29T20:19:36  *** netzin has quit IRC
412 2016-07-29T20:34:39  *** belcher has joined #bitcoin-core-dev
413 2016-07-29T20:35:49  *** laurentmt has quit IRC
414 2016-07-29T20:54:34  *** dasaj has joined #bitcoin-core-dev
415 2016-07-29T21:18:23  *** YOU-JI has quit IRC
416 2016-07-29T21:37:19  *** arubi has quit IRC
417 2016-07-29T21:38:25  *** arubi has joined #bitcoin-core-dev
418 2016-07-29T21:43:13  *** Chris_Stewart_5 has quit IRC
419 2016-07-29T21:57:10  *** spudowiar has joined #bitcoin-core-dev
420 2016-07-29T22:00:29  *** whphhg_ has joined #bitcoin-core-dev
421 2016-07-29T22:01:33  *** whphhg has quit IRC
422 2016-07-29T22:02:17  *** whphhg_ is now known as whphhg
423 2016-07-29T22:39:11  *** cryptapus is now known as cryptapus_afk
424 2016-07-29T23:11:01  *** Alopex has quit IRC
425 2016-07-29T23:12:06  *** Alopex has joined #bitcoin-core-dev
426 2016-07-29T23:20:36  *** d_t has quit IRC
427 2016-07-29T23:23:35  *** Guyver2 has quit IRC
428 2016-07-29T23:25:43  *** TomMc has quit IRC
429 2016-07-29T23:43:33  *** netzin has joined #bitcoin-core-dev