1 2015-10-30T00:05:34  *** BashCo_ has joined #bitcoin-core-dev
  2 2015-10-30T00:07:23  <gmaxwell> sipa: ^ are you setup to produce windows binaries right now?
  3 2015-10-30T00:07:48  <warren> gmaxwell: here?
  4 2015-10-30T00:08:02  <gmaxwell> right.
  5 2015-10-30T00:08:09  <warren> gmaxwell: sorry the split wasn't entirely clear to me.
  6 2015-10-30T00:08:19  *** BashCo has quit IRC
  7 2015-10-30T00:08:30  <gmaxwell> warren: this channel should be for bitcoin core exclusive discussion.
  8 2015-10-30T00:08:55  <gmaxwell> no one else gives a hoot at how our rpc auth works. :)
  9 2015-10-30T00:09:41  <Luke-Jr> Fedora does apparently
 10 2015-10-30T00:09:42  * Luke-Jr hides
 11 2015-10-30T00:09:55  <sipa> gmaxwell: do i get 30 minutes to find a bug? :)
 12 2015-10-30T00:10:23  <gmaxwell> sipa: hahah
 13 2015-10-30T00:10:55  <gmaxwell> sipa: I don't want it merged; there is a user that manged to get pubkey "" in his wallet on bct, and I want to give him a binary to see if it rejects it.
 14 2015-10-30T00:11:48  <gmaxwell> warren: as far as selinux policy, I think it's unfortunate that you cannot confine bitcoin core to have no access outside of its own directory.
 15 2015-10-30T00:12:00  <gmaxwell> though maybe that should be a boolean.
 16 2015-10-30T00:12:10  <gmaxwell> since its only needed for the walletimport/backup stuff.
 17 2015-10-30T00:13:22  <warren> Would it be bad if the system-level systemd service were named something like bitcoinsys.service, with a selinux context bitcoinsys_t, and a bitcoin-cli wrapper named something like /usr/bin/bitcoinsys_cli that only adds the --datadir /path/to/system/datadir/
 18 2015-10-30T00:13:58  <Luke-Jr> and what if the user wants bitcoind run per-user? :P
 19 2015-10-30T00:16:06  <gmaxwell> warren: what happens if a second datadir is specified to bitcoin-cli?
 20 2015-10-30T00:16:57  <gmaxwell> warren: it's all ugly, bitcoin core is currently not intended for system level operation.
 21 2015-10-30T00:17:25  <warren> gmaxwell: hence why i'm against shipping a .service file at all
 22 2015-10-30T00:20:49  <gmaxwell> in any case that doesn't sound too horrible but the wrapper would need to not prevent you from overriding the datadir, and it needs to be tested with e.g. super long inputs.
 23 2015-10-30T00:21:30  <gmaxwell> better perhaps would be a compile time setting to adjust the default datadir. Do we have one of those?
 24 2015-10-30T00:22:45  *** testing-tape has quit IRC
 25 2015-10-30T00:23:27  <sipa> not afaik
 26 2015-10-30T00:23:36  <sipa> gmaxwell: bug found, will build
 27 2015-10-30T00:23:49  <gmaxwell> lol okay.
 28 2015-10-30T00:24:02  <Luke-Jr> warren: doesn't systemd support per-user services?
 29 2015-10-30T00:24:31  <sipa> Luke-Jr: systemd would be a great operating system, if it only contained a service manager
 30 2015-10-30T00:24:47  <sipa> s/if it only/if only it/
 31 2015-10-30T00:24:55  <Luke-Jr> don't get me wrong, I hate systemd. but I thought this was one of the few benefits to it.
 32 2015-10-30T00:26:26  *** pavel_ has joined #bitcoin-core-dev
 33 2015-10-30T00:26:56  <gmaxwell> Seperate subject; https://github.com/bitcoin/bitcoin/pull/6902  I feel that changes like this make the software more opaque.  Now to understand whats being done there you must read two places instead of one.  If you only look at policy.h you will likely erroniously believe it applies to p2sh somehow.
 34 2015-10-30T00:27:33  <warren> Luke-Jr: yes it's possible
 35 2015-10-30T00:28:25  <gmaxwell> It's of a class of change where I purposfully try to keep my opinion to myself to avoid amplifying bikeshedding, because it fundimentally does not matter.
 36 2015-10-30T00:28:34  *** paveljanik has quit IRC
 37 2015-10-30T00:32:04  <sipa> gmaxwell: 32-bit or 64-bit?
 38 2015-10-30T00:33:39  <sipa> gmaxwell: will take a while to build dependencies
 39 2015-10-30T00:34:27  <gmaxwell> user say "or" so 64-bit I guess.
 40 2015-10-30T00:50:15  <GitHub198> [bitcoin] nathanimpact opened pull request #6909: 0.8 (master...0.8) https://github.com/bitcoin/bitcoin/pull/6909
 41 2015-10-30T00:50:46  <GitHub180> [bitcoin] gmaxwell closed pull request #6909: 0.8 (master...0.8) https://github.com/bitcoin/bitcoin/pull/6909
 42 2015-10-30T00:50:57  <GitHub190> [bitcoin] nathanimpact opened pull request #6910: 0.9 (master...0.9) https://github.com/bitcoin/bitcoin/pull/6910
 43 2015-10-30T00:51:17  <GitHub40> [bitcoin] nathanimpact opened pull request #6911: 0.10 (master...0.10) https://github.com/bitcoin/bitcoin/pull/6911
 44 2015-10-30T00:51:27  <GitHub112> [bitcoin] gmaxwell closed pull request #6910: 0.9 (master...0.9) https://github.com/bitcoin/bitcoin/pull/6910
 45 2015-10-30T00:51:42  <GitHub69> [bitcoin] nathanimpact opened pull request #6912: 0.11 (master...0.11) https://github.com/bitcoin/bitcoin/pull/6912
 46 2015-10-30T00:51:48  <gmaxwell> I wish github would let you block these.
 47 2015-10-30T00:51:54  <GitHub93> [bitcoin] gmaxwell closed pull request #6911: 0.10 (master...0.10) https://github.com/bitcoin/bitcoin/pull/6911
 48 2015-10-30T00:52:12  <GitHub198> [bitcoin] gmaxwell closed pull request #6912: 0.11 (master...0.11) https://github.com/bitcoin/bitcoin/pull/6912
 49 2015-10-30T00:52:54  <GitHub45> [bitcoin] nathanimpact opened pull request #6913: 0.11 (master...0.11) https://github.com/bitcoin/bitcoin/pull/6913
 50 2015-10-30T00:54:03  <belcher> you can block them(!) on the settings panel somewhere in github you can configure these to happen or not
 51 2015-10-30T00:54:30  <gmaxwell> belcher: what specifically does it block?
 52 2015-10-30T00:54:44  <belcher> turns off or on these bots
 53 2015-10-30T00:55:06  <sipa> belcher: we want to block the PRs, not the reporting here :)
 54 2015-10-30T00:55:23  <belcher> i see, im not sure thats possible
 55 2015-10-30T00:55:56  <belcher> if you know they're coming you could set the channel mode +n (i think) which stops them being able to message without joining
 56 2015-10-30T00:56:59  <petertodd> gmaxwell: ACK re your thoughts on #6902 - this isn't a constant that you can change after all!
 57 2015-10-30T00:57:01  <gmaxwell> the bots we want, just not the pointless PRs to merge entire (sometimes altcoin) branches into master.
 58 2015-10-30T00:59:44  <sipa> gmaxwell: building qt failed, sorry
 59 2015-10-30T00:59:49  <petertodd> gmaxwell: er, scrap that, no policy, not script, dumb of me...
 60 2015-10-30T00:59:51  <sipa> not going to spend time investigating
 61 2015-10-30T01:20:43  *** testing-tape has joined #bitcoin-core-dev
 62 2015-10-30T01:25:50  *** testing-tape has quit IRC
 63 2015-10-30T01:29:23  <GitHub139> [bitcoin] sipa opened pull request #6914: [WIP] Add pre-allocated vector type and use it for CScript (master...prevector) https://github.com/bitcoin/bitcoin/pull/6914
 64 2015-10-30T01:31:02  <gmaxwell> sipa: why doesn't it bounds check?
 65 2015-10-30T01:31:28  <sipa> gmaxwell: because i'm lazy
 66 2015-10-30T01:32:10  <gmaxwell> sipa: 13% faster with script checking enabled?!
 67 2015-10-30T01:32:18  <sipa> gmaxwell: disabled
 68 2015-10-30T01:32:20  <gmaxwell> oh disabled good.
 69 2015-10-30T01:32:28  <gmaxwell> okay. Yea, missed that on first read.
 70 2015-10-30T01:33:03  <GitHub7> [bitcoin] sdaftuar opened pull request #6915: [Mempool] Improve removal of invalid transactions after reorgs (master...fix-reorg-handling) https://github.com/bitcoin/bitcoin/pull/6915
 71 2015-10-30T01:34:26  <sipa> the only place where we actually use ::at() is in CScript::IsPayToScriptHash(), and it's prefixed with a size check there
 72 2015-10-30T01:35:41  <gmaxwell> maybe just get rid of the at and make it not compile if its used? Rationale: providing something thats almost vector but less safe in some subtle way will be forgotten. :)
 73 2015-10-30T01:37:12  <gmaxwell> sipa: did you fix the scriptcheck on reindex generally or just patch it out? if the former can you PR the fix?
 74 2015-10-30T01:37:36  <sipa> i just patched it out
 75 2015-10-30T01:37:47  <gmaxwell> (that particular misbehavior is a beautiful interaction with the always need to reindex issue in windows I bet. :-/ )
 76 2015-10-30T01:57:28  <sipa> gmaxwell: building 6906 (+6900, since windows building is broken on modern mingw without 6900)
 77 2015-10-30T01:58:06  <CodeShark> are you guys trying to reproduce the windows db corruption issues? :)
 78 2015-10-30T01:58:39  <gmaxwell> not at the moment.
 79 2015-10-30T01:58:45  <gmaxwell> CodeShark: warren was talking about that earlier.
 80 2015-10-30T01:59:02  <gmaxwell> I'm trying to debug a wallet that ended up with a corrupted pubkey in it.
 81 2015-10-30T01:59:19  <gmaxwell> which is in possession of an end user who is helpful but needed a binary.
 82 2015-10-30T01:59:38  <CodeShark> ah
 83 2015-10-30T02:01:23  *** daniel___ has quit IRC
 84 2015-10-30T02:01:48  *** daniel___ has joined #bitcoin-core-dev
 85 2015-10-30T02:03:06  <gmaxwell> As an aside, same user is running a 0.8GB wallet with hundreds of thousands of transactions, on windows.
 86 2015-10-30T02:04:18  <CodeShark> hah
 87 2015-10-30T02:04:21  *** Ylbam has quit IRC
 88 2015-10-30T02:04:56  <CodeShark> doesn't sound very pleasant on multiple fronts :p
 89 2015-10-30T02:19:22  <dcousens> sipa: around?
 90 2015-10-30T02:19:55  <sipa> yes
 91 2015-10-30T02:20:32  <dcousens> sipa: 23% less memory, sounds like its simply a matter of the allocator being more exact instead of allocating more generously etc
 92 2015-10-30T02:20:47  <sipa> ?
 93 2015-10-30T02:21:15  <dcousens> sipa: conceptually, how does https://github.com/bitcoin/bitcoin/pull/6914 save memory, except that in some cases it is allocated on the stack?
 94 2015-10-30T02:21:33  <sipa> dcousens: well it uses 32-bit instead of 64-bit sizes
 95 2015-10-30T02:21:37  <dcousens> (and ignoring the 4/8 byte pointer)
 96 2015-10-30T02:21:40  <sipa> so it always saves 4 to 8 bytes
 97 2015-10-30T02:21:57  <sipa> apart from that, the only way it saves is by not allocating on the heap
 98 2015-10-30T02:22:05  <sipa> which has a 16-32 byte overhead
 99 2015-10-30T02:22:27  <dcousens> and the hunch is that takes up 23% of available memory?
100 2015-10-30T02:22:33  <sipa> so the old structure is on average 40 + N bytes storage
101 2015-10-30T02:23:21  <sipa> sorry, 48 + N
102 2015-10-30T02:23:57  <sipa> the new one is 32 for N<=28, and 56 + N for N > 28
103 2015-10-30T02:24:37  <sipa> since almost all CScripts in memory are P2PKH or P2SH, which are less than 28 bytes, this is an improvement
104 2015-10-30T02:25:58  <gmaxwell> sipa: aren't the N objects seperately heap allocated?
105 2015-10-30T02:27:06  <dcousens> sipa: any reason not to just use union a std::array and std::vector?
106 2015-10-30T02:27:55  <sipa> gmaxwell: no
107 2015-10-30T02:28:11  <sipa> dcousens: that would be much larger
108 2015-10-30T02:28:31  <sipa> the trick here is to share the size variable between the two approaches
109 2015-10-30T02:28:44  <sipa> which you can't do if you use the standard std::vector
110 2015-10-30T02:30:39  <gmaxwell> sipa: hm for some reason I thought it was, so that the removal of objects could be supported without invalidating pointers.
111 2015-10-30T02:30:51  <dcousens> sipa: could the same memory savings be achieved by just abstracting a CScript specialised for smaller sizes?
112 2015-10-30T02:31:01  <dcousens> Eh, that'd be as bad nvm
113 2015-10-30T02:31:19  <sipa> dcousens: perhaps i failed to highlihght the largest advantage: reducing load on the heap
114 2015-10-30T02:31:42  <sipa> which results in worse memory locality and memory fragmentation :)
115 2015-10-30T02:31:53  <sipa> gmaxwell: you're thinking about a linked list...
116 2015-10-30T02:31:58  <dcousens> indeed, probably where the 13% speedup came from
117 2015-10-30T02:32:13  <sipa> dcousens: and simpler copying
118 2015-10-30T02:32:23  <sipa> gmaxwell: a vector can't delete elements without invalidating iterators
119 2015-10-30T02:33:46  *** belcher has quit IRC
120 2015-10-30T02:35:30  <dcousens> sipa: is push_back used at all?
121 2015-10-30T02:35:51  <sipa> no clue
122 2015-10-30T02:36:05  <dcousens> just wondering, maybe drop the capacity functionality if we don't need it
123 2015-10-30T02:36:15  <dcousens> I feel like we' wouldn't be re-allocating these vectors very often
124 2015-10-30T02:36:28  <sipa> well i wanted something API compatible :)
125 2015-10-30T02:36:44  <sipa> so it's not a huge burden to go review whether it can be used or not
126 2015-10-30T02:37:04  <sipa> if you just want a compact vector, there are other changes you can make
127 2015-10-30T02:37:08  <dcousens> sipa: I guess if you're looking for public compatibility
128 2015-10-30T02:37:42  <dcousens> sipa: IMHO for a change like this, we should start with the smallest subset of changes, optimize on that, then introduce public API compatibility
129 2015-10-30T02:38:01  <sipa> the subset of changes is as small as it can be
130 2015-10-30T02:38:35  <sipa> the PR contains a few not-strictly necessary changes to just break the assumption that CScript is a vector
131 2015-10-30T02:42:27  <dcousens> ooi, why free vs new?
132 2015-10-30T02:42:46  <sipa> because i need realloc
133 2015-10-30T02:43:08  <sipa> and this works at a lower level: it's allocating memory, not objects
134 2015-10-30T02:43:12  <sipa> new is for allocating objects
135 2015-10-30T02:43:33  <dcousens> ? you have a clear type though
136 2015-10-30T02:43:51  <sipa> which?
137 2015-10-30T02:43:57  <dcousens> the realloc might not be necessary if you don't need the push_back/change_capacity etc
138 2015-10-30T02:44:00  <dcousens> T
139 2015-10-30T02:44:06  <sipa> no, that would call T's constructor
140 2015-10-30T02:44:29  <sipa> the reserved but unused memory doesn't contain constructed T objects
141 2015-10-30T02:44:36  <dcousens> True, but isn't T in this case always a char*?
142 2015-10-30T02:44:47  <sipa> i don't plan to just use this for char
143 2015-10-30T02:45:54  <dcousens> hmmm, well, code looks good, tentativee ACK simply cause I'm still not sure its the best way, but, it is a solution, and it exists :) wd
144 2015-10-30T02:49:02  <gmaxwell> sipa: did the prior stl instrumentation you did before allow you to easily measure vector overhead in a useful way to go find the biggest offenders?
145 2015-10-30T02:49:42  <dcousens> if we broke the API compatibility, I feel like we could just use managed pointers [at the CSCript level] quite easily with a pooled stack allocator for small scripts, but, if this might be easier if you have other future plans for other areas
146 2015-10-30T02:50:37  <sipa> dcousens: that would indeed by an improvement; CBlock could have its own pool for example, and CTransaction could couple a pool too
147 2015-10-30T02:50:45  <sipa> dcousens: but it's a much more invasive change to do well
148 2015-10-30T02:50:50  <dcousens> indeed
149 2015-10-30T02:50:51  <sipa> gmaxwell: define offender?
150 2015-10-30T02:51:48  <dcousens> sipa: I guess the benefit of this change, with API compatibility, is it doesn't hinder that path in any (except maybe making it look less attractive haha)
151 2015-10-30T02:51:55  <gmaxwell> I do not see why we'd want _any_ overhead of having constant allocation indirection for data where 99.999% of the time its a fixed size. even having the extra pointer is overhead we don't need.
152 2015-10-30T02:53:01  <dcousens> gmaxwell: as mentioned above, to do that (I assume you mean using fixed size CScripts), it'd be a more invasive change
153 2015-10-30T02:53:12  <gmaxwell> sipa: places where many vectors with a total amount of data them that is almost always small with respect to 40 bytes.
154 2015-10-30T02:53:17  <sipa> gmaxwell: vectors have 40 byte overhead on average (including the malloc overhead) per vector
155 2015-10-30T02:53:57  <dcousens> gmaxwell: or by extra pointer did you mean cap?
156 2015-10-30T02:54:29  <gmaxwell> dcousens: what sipa is doing now is almost a fixed size cscript, no-- only when its an exceptional cscript does it fail over to using an allocation.
157 2015-10-30T02:55:22  <sipa> i'm using 28 bytes as cutoff; there are many scripts larger (nearly every scriptSig)
158 2015-10-30T02:55:40  <sipa> but in places where storage matters, more than 28 is a minority
159 2015-10-30T02:56:40  <sipa> i used 36 earlier, and that was still a net benefit, but a smaller one
160 2015-10-30T02:56:47  <gmaxwell> ah, I did forget this was used for ScriptSig too.
161 2015-10-30T02:58:17  <phantomcircuit> it's kind of amusing how we can optimize for the current set of data and have that not be a huge issue... since it will never change
162 2015-10-30T02:58:48  <sipa> if we'd switch to a P3SH with a 256-bit hash, we may need to change it :)
163 2015-10-30T02:59:11  <sipa> we could use this for valtype inside the script evaluator too, set to 72 bytes (as we rarely have larger script elements), or to 520 (as we don't support larger at all)
164 2015-10-30T02:59:12  <phantomcircuit> heh
165 2015-10-30T02:59:41  <phantomcircuit> sipa, sure but like if block_height < 350000 then vectortype = x else vectortype = y
166 2015-10-30T03:00:01  <gmaxwell> obviously it needs to be autoadaptive. :P
167 2015-10-30T03:00:21  <phantomcircuit> :)
168 2015-10-30T03:02:01  <gmaxwell> Another obvious thing to try is to breakpoint at the validation of block X, then breakpoint at malloc/new  and catch every place it hits the heap allocator during verification of that block.
169 2015-10-30T03:10:21  <dcousens> ignoring memory, I wonder if you just fixed size it to 520 bytes
170 2015-10-30T03:10:47  <gmaxwell> dcousens: it can be larger than 520 bytes.
171 2015-10-30T03:11:21  <phantomcircuit> gmaxwell, how?
172 2015-10-30T03:11:28  <gmaxwell> 0_o
173 2015-10-30T03:11:32  <sipa> CScript can be larger than 520 bytes
174 2015-10-30T03:11:38  <sipa> script stack elements can;t
175 2015-10-30T03:11:44  <gmaxwell> phantomcircuit: this is the full script pubkey in a transaction or a full scriptsig in a transaction.
176 2015-10-30T03:12:06  <gmaxwell> scriptsigs larger than that are probably not even all that uncommon.
177 2015-10-30T03:12:51  *** zooko has joined #bitcoin-core-dev
178 2015-10-30T03:12:52  <phantomcircuit> ohh
179 2015-10-30T03:13:12  <phantomcircuit> i thought you were just optimizing for the push data
180 2015-10-30T03:48:29  <warren> gmaxwell: a safe wrapper that passes all parameters to some other program is simple like: http://pkgs.fedoraproject.org/cgit/couchdb.git/tree/couchdb.temporary.sh
181 2015-10-30T03:48:46  <warren> https://fedoraproject.org/wiki/BITCOIN
182 2015-10-30T03:50:03  <warren> I wrote down what I think would be good for the two different ways in which people want to use bitcoind, as a user and as a system service.  Both would need separate SELinux contexts, and a wrapper like http://pkgs.fedoraproject.org/cgit/couchdb.git/tree/couchdb.temporary.sh  to launch the system service under the different context.
183 2015-10-30T04:12:03  *** evoskuil has quit IRC
184 2015-10-30T04:14:19  *** zooko has quit IRC
185 2015-10-30T04:23:09  *** evoskuil has joined #bitcoin-core-dev
186 2015-10-30T04:43:16  *** NLNico has joined #bitcoin-core-dev
187 2015-10-30T05:18:14  *** PaulCapestany has quit IRC
188 2015-10-30T05:19:59  *** PaulCapestany has joined #bitcoin-core-dev
189 2015-10-30T05:35:30  *** PaulCape_ has joined #bitcoin-core-dev
190 2015-10-30T05:37:46  *** PaulCapestany has quit IRC
191 2015-10-30T06:00:55  *** PaulCapestany has joined #bitcoin-core-dev
192 2015-10-30T06:03:01  *** PaulCape_ has quit IRC
193 2015-10-30T06:17:11  *** droark has quit IRC
194 2015-10-30T06:26:49  *** challisto has joined #bitcoin-core-dev
195 2015-10-30T06:26:56  *** PaulCape_ has joined #bitcoin-core-dev
196 2015-10-30T06:29:35  *** PaulCapestany has quit IRC
197 2015-10-30T07:01:30  *** PaulCape_ has quit IRC
198 2015-10-30T07:01:56  *** PaulCapestany has joined #bitcoin-core-dev
199 2015-10-30T07:13:13  *** danielsocials has joined #bitcoin-core-dev
200 2015-10-30T07:21:39  *** Luke-Jr has quit IRC
201 2015-10-30T07:24:55  *** Luke-Jr has joined #bitcoin-core-dev
202 2015-10-30T07:27:21  *** PaulCape_ has joined #bitcoin-core-dev
203 2015-10-30T07:29:35  *** PaulCapestany has quit IRC
204 2015-10-30T07:32:10  *** danielsocials has quit IRC
205 2015-10-30T07:33:46  *** danielsocials has joined #bitcoin-core-dev
206 2015-10-30T07:38:41  *** danielsocials has quit IRC
207 2015-10-30T07:48:43  *** Ylbam has joined #bitcoin-core-dev
208 2015-10-30T08:02:27  *** danielsocials has joined #bitcoin-core-dev
209 2015-10-30T08:15:16  <jonasschnelli> ping https://github.com/bitcoin/bitcoin/pull/6780
210 2015-10-30T08:15:34  <jonasschnelli> should get in before it gets outdated
211 2015-10-30T08:16:50  *** danielsocials has quit IRC
212 2015-10-30T08:40:54  <GitHub158> [bitcoin] laanwj closed pull request #6913: 0.11 (master...0.11) https://github.com/bitcoin/bitcoin/pull/6913
213 2015-10-30T08:41:56  *** pavel_ has quit IRC
214 2015-10-30T08:42:14  *** paveljanik has joined #bitcoin-core-dev
215 2015-10-30T08:44:42  *** BashCo_ has quit IRC
216 2015-10-30T08:55:57  *** rubensayshi has joined #bitcoin-core-dev
217 2015-10-30T09:03:42  *** BashCo has joined #bitcoin-core-dev
218 2015-10-30T09:50:20  *** PRab_ has joined #bitcoin-core-dev
219 2015-10-30T09:51:01  *** PRab has quit IRC
220 2015-10-30T09:51:02  * wumpus has an old crappy laptop with windows now, let the carnage begin...
221 2015-10-30T09:51:06  *** PRab_ is now known as PRab
222 2015-10-30T09:56:43  <wumpus> I'm sure it comes with rootkits and malware pre-installed to simulate a true, out in the wild windows environment
223 2015-10-30T09:57:45  <phantomcircuit> :P
224 2015-10-30T09:59:57  *** CodeShark has quit IRC
225 2015-10-30T10:15:59  <dcousens> wumpus: what would you say an average time for re-index is?
226 2015-10-30T10:16:07  <dcousens> (ballpark, obv. very dependent on hardware)
227 2015-10-30T10:19:16  *** dcousens has quit IRC
228 2015-10-30T10:31:54  *** NLNico has quit IRC
229 2015-10-30T11:27:34  *** NLNico has joined #bitcoin-core-dev
230 2015-10-30T11:28:43  *** NLNico has quit IRC
231 2015-10-30T11:45:56  *** tulip has joined #bitcoin-core-dev
232 2015-10-30T12:04:45  *** evoskuil has quit IRC
233 2015-10-30T13:12:31  <morcos> sipa: i've been trying to track down some odd slowness in ConnectBlock and I found something I think interesting
234 2015-10-30T13:13:28  <morcos> When we do UpdateCoins and we need to add the outputs for the new coins we are now creating.  We call ModifyCoins(new tx hash) which will end up doing a database read.
235 2015-10-30T13:14:00  <morcos> so even if our cache is completely up to date, we have to read the database anyway
236 2015-10-30T13:14:27  <morcos> There is already an existince check for BIP30 earlier
237 2015-10-30T13:14:44  <morcos> I'm wondering if we should have some sort of negative cache too
238 2015-10-30T13:16:39  <morcos> Also do we need to persist the BIP30 check?  Is it still possible to violate that or now that the old duplicate coinbases are sufficiently buried, its not necessary
239 2015-10-30T13:16:54  <morcos> Anwyay, I have to go afk.  I'll look into it more later.
240 2015-10-30T13:59:07  *** ParadoxSpiral has joined #bitcoin-core-dev
241 2015-10-30T14:00:25  *** testing-tape has joined #bitcoin-core-dev
242 2015-10-30T14:03:27  *** testing-tape has quit IRC
243 2015-10-30T14:08:06  *** Guest75176 has quit IRC
244 2015-10-30T14:08:28  *** pigeons has joined #bitcoin-core-dev
245 2015-10-30T14:08:52  *** pigeons is now known as Guest35844
246 2015-10-30T14:31:20  <morcos> sipa: still afk, but yep, thats a cause of SIGNIFICANT slowness in ConnectBlock.  I think we'd have to change/remove the BIP30 check to really eliminate it, which might be the harder part, bc I think adjusting ModifyCoins to know its a new tx and not lookup seems easy and safe?
247 2015-10-30T14:41:49  *** goregrind has quit IRC
248 2015-10-30T14:45:10  *** goregrind has joined #bitcoin-core-dev
249 2015-10-30T14:55:04  *** [b__b] has quit IRC
250 2015-10-30T14:55:25  *** [b__b] has joined #bitcoin-core-dev
251 2015-10-30T15:04:17  <wumpus> cfields: what would be the best way to include the windows determinism postprocessing in the gitian build (in https://github.com/bitcoin/bitcoin/pull/6900) - call it from the descriptor or 'make'?
252 2015-10-30T15:05:19  <cfields> wumpus: since that's really only useful for gitian (and a specific version of binutils), i'd say in the descriptor is fine
253 2015-10-30T15:05:29  <wumpus> yeah, thought as much
254 2015-10-30T15:05:39  *** zooko has joined #bitcoin-core-dev
255 2015-10-30T15:06:00  <wumpus> do wonder how to make it output the linker map though
256 2015-10-30T15:06:04  <cfields> wumpus: i haven't taken a good look yet, but i'm surprised that you can't find the necessary offsets with objdump/nm/etc. I suppose you tried all those first?
257 2015-10-30T15:06:27  <wumpus> objdump and friends all report virtual addresses
258 2015-10-30T15:06:42  *** zooko` has joined #bitcoin-core-dev
259 2015-10-30T15:06:48  *** zooko has quit IRC
260 2015-10-30T15:06:52  *** zooko` has quit IRC
261 2015-10-30T15:07:10  <cfields> mm, ok
262 2015-10-30T15:07:38  *** zooko has joined #bitcoin-core-dev
263 2015-10-30T15:07:51  <wumpus> and within a section they're no help at all, for finding the offset, it's not like there's a pointer to this variable (the other option may be to parse DWARF information, but hm...)
264 2015-10-30T15:08:32  <wumpus> that may not even work, I don't think gdb has an offset to it, it reports it as GS_ExceptionPointers+8/16
265 2015-10-30T15:08:33  <cfields> wumpus: another fleeting thought.. in the mailing-list link you posted, some of my binutils changes are listed in the same changelog...
266 2015-10-30T15:08:43  <cfields> you completely sure that this is the only hack necessary?
267 2015-10-30T15:09:27  <cfields> iirc it was the one that removed timestamps from windres
268 2015-10-30T15:09:42  <wumpus> well I found no other differences between multiple builds
269 2015-10-30T15:09:51  <wumpus> remember, we still use faketime
270 2015-10-30T15:10:20  <cfields> ah, right
271 2015-10-30T15:10:56  <wumpus> the tor folks build binutils themselves
272 2015-10-30T15:11:36  <wumpus> but anyhow, the current script works, so I'll just integrate that for now
273 2015-10-30T15:11:46  <cfields> sounds good
274 2015-10-30T15:13:22  <cfields> wumpus: one last quick thought, another option might be using LD_PRELOAD to wrap malloc (or whatever is leaving unsanitized data)
275 2015-10-30T15:17:22  <wumpus> hehe.
276 2015-10-30T15:18:01  <wumpus> wonder if  mallopt(M_PERTURB,...) can be set from the environment
277 2015-10-30T15:18:45  <wumpus> I think so! it should do exactly this
278 2015-10-30T15:19:25  <cfields> hah! neat. I was just doing a quick malloc->calloc wrapper. That looks smarter :)
279 2015-10-30T15:22:10  <wumpus>  'export MALLOC_PERTURB_=23'  seems to work
280 2015-10-30T15:22:21  <wumpus> (at least outside of gitian, let's try inside)
281 2015-10-30T15:22:51  <wumpus> that was an awesome idea
282 2015-10-30T15:23:31  * wumpus resents having written a PE parser :-)
283 2015-10-30T15:24:03  <cfields> should probably wrap the linker in a script and export it there, rather than globally. It's hard to imagine anything build-side dependent on that, but i'd hate to see something like the debian-openssl thing crop up again :)
284 2015-10-30T15:24:52  <wumpus> also reduces the performance hit
285 2015-10-30T15:25:28  <cfields> yep
286 2015-10-30T15:26:37  <cfields> wumpus: nice find on MALLOC_PERTURB_. that makes this very simple :)
287 2015-10-30T15:27:14  <wumpus> but the linker is invoked as 'g++' isn't it?
288 2015-10-30T15:27:27  <cfields> g++ invokes ld
289 2015-10-30T15:27:31  *** zooko has quit IRC
290 2015-10-30T15:28:18  <wumpus> oh yes it's even worse for mingw, it invokes collect2, which is a wrapper for ld. I wonder if it's possible to interpose somehwere there
291 2015-10-30T15:28:49  <wumpus> (well obviously, by replacing the executable... but I wonder if it uses path somewhree)
292 2015-10-30T15:29:58  <cfields> uhmm, you can specify your own linker i believe
293 2015-10-30T15:30:57  * Luke-Jr imagines each of these is used for a reason
294 2015-10-30T15:31:27  <wumpus> Luke-Jr: 'collect2' is MS's linker, which for mingw is emulated by ld
295 2015-10-30T15:32:27  <Luke-Jr> oh, it's just emulating it?
296 2015-10-30T15:33:01  <wumpus> yes - it could really use MS's linker, if you have it available
297 2015-10-30T15:33:10  <Luke-Jr> interesting
298 2015-10-30T15:37:12  <cfields> wumpus: yup, looks like it checks for 'real-ld'
299 2015-10-30T15:38:31  <cfields> and ${prefix}-real-ld
300 2015-10-30T15:40:22  <wumpus> I haven't been able to trick it yet into running a different linker outside the $prefix (created all sorts of variants of collect2 and ld in my PATH)
301 2015-10-30T15:41:11  <wumpus> one remaning option would be to override gcc's specs, but bleh
302 2015-10-30T15:46:52  <wumpus> seems it only looks for 'linker' outside its hardcoded path  /usr/lib/gcc/x86_64-w64-mingw32/4.8 if it cannot find it there
303 2015-10-30T15:47:07  *** zooko has joined #bitcoin-core-dev
304 2015-10-30T15:48:10  <wumpus> what works is: 'gcc -dumpspecs > specsfile' 'sed -i s/collect2/mycollect2/g specsfile' 'gcc -specs=specsfile'
305 2015-10-30T15:48:28  <wumpus> then put mycollect2 somewhere in the path...
306 2015-10-30T15:48:36  <wumpus> it's kind of stupid though
307 2015-10-30T15:49:06  <wumpus> hmm overrding GCC_EXEC_PREFIX may work too
308 2015-10-30T15:49:22  <wumpus> but that means copying the other files
309 2015-10-30T15:49:49  <wumpus> oh there is COMPILER_PATH  too, where it will look for subprograms if it cannot find in EXEC_PREFIX
310 2015-10-30T15:49:58  <wumpus> great, this may yet be solvable
311 2015-10-30T15:50:40  <cfields> wumpus: heh, was just testing that one
312 2015-10-30T15:50:42  <cfields> no luck
313 2015-10-30T15:51:20  * wumpus considers just defining the malloc perturb globally and be done with it...
314 2015-10-30T15:51:21  <cfields> wait, that worked :)
315 2015-10-30T15:51:38  <cfields> export COMPILER_PATH=`pwd`
316 2015-10-30T15:51:44  <cfields> then wrap ./collect2
317 2015-10-30T15:52:06  <wumpus> according to the docs it only looks there if it cannot find it in the EXEC_PREFIX, but will try
318 2015-10-30T15:52:38  <wumpus> yep, seems to work!
319 2015-10-30T15:58:42  <wumpus> cfields: something like this? https://gist.github.com/laanwj/10b0981d9d75510a124f
320 2015-10-30T15:58:50  <cfields> wumpus: http://pastebin.com/W350ZGK5
321 2015-10-30T15:58:51  <cfields> hah!
322 2015-10-30T15:59:34  <cfields> looks great
323 2015-10-30T15:59:48  <wumpus> ah yes, $CC -print-prog-name=collect is nice
324 2015-10-30T16:03:02  <wumpus> updated to use that https://gist.github.com/laanwj/10b0981d9d75510a124f  (now request the path of the collect2 tool while creating the wrapper script instead of hardcoding it in the descriptor)
325 2015-10-30T16:04:36  <cfields> wumpus: need to unset COMPILER_PATH or you get an infinite loop
326 2015-10-30T16:04:53  <wumpus> COMPILER_PATH is not set at that point yet :)
327 2015-10-30T16:05:05  <wumpus> it's set in the per-host build loop later on
328 2015-10-30T16:06:09  <cfields> oh heh, you're finding the path outside of the script. nm :)
329 2015-10-30T16:06:11  *** zooko has quit IRC
330 2015-10-30T16:06:30  <wumpus> yes I look up the path upfront
331 2015-10-30T16:06:57  <cfields> well that's nice and simple if it works
332 2015-10-30T16:09:13  <cfields> wumpus: just to be on the paranoid side, should probably compare before/after to make sure that only the expected bytes changed.
333 2015-10-30T16:10:49  <wumpus> well if more bytes change then we should definitely skip trusty, agreed
334 2015-10-30T16:12:26  <cfields> (i just mean eyeballing it once, ofc. not suggesting scripting that part)
335 2015-10-30T16:16:09  <wumpus> leaking heap data into output is really ugly, can be e.g. a privacy issue, I'm happy it was solved two years ago ;)
336 2015-10-30T16:16:37  <wumpus> MS office had some legendary fuckups in this regard
337 2015-10-30T16:17:57  <cfields> wumpus: assuming mingw is self-hosted,  we can check the bins for leaks and see if the dev had anything interesting in mem :)
338 2015-10-30T16:19:41  <wumpus> hehe
339 2015-10-30T16:20:20  <wumpus> well in this case it seems it leaks some internal address; could be used to bypass ASLR, if it was a longer running process
340 2015-10-30T16:20:23  *** dixson3 has joined #bitcoin-core-dev
341 2015-10-30T16:30:09  <sipa> morcos: we should get rid of the bip30 check; it's superseeded by bip34
342 2015-10-30T16:31:34  <cfields> wumpus: for the qt fix, can that ifdef just be c/p to the header as well?
343 2015-10-30T16:33:26  <wumpus> cfields: yes I suppose the same principle can be used to use either PIDLIST_ABSOLUTE or ITEMIDLIST
344 2015-10-30T16:34:04  *** Guyver2 has joined #bitcoin-core-dev
345 2015-10-30T16:34:17  <wumpus> (just copy pasting won't work though because the typedef won't help_
346 2015-10-30T16:34:34  *** dgenr8 has quit IRC
347 2015-10-30T16:35:28  <cfields> wumpus: ok. mind PRing that separate? or I can? That needs fixing regardless of when we switch gitian
348 2015-10-30T16:35:52  <wumpus> no problem if you do :)
349 2015-10-30T16:35:53  <cfields> i'm happy to do it, not trying to con you into that one :)
350 2015-10-30T16:35:56  <cfields> sure, will do
351 2015-10-30T16:36:27  *** dgenr8 has joined #bitcoin-core-dev
352 2015-10-30T16:36:41  * wumpus just crashed a windows PC in the middle of initial sync, came back fine after reboot...
353 2015-10-30T16:37:20  *** dixson3 has quit IRC
354 2015-10-30T16:37:46  *** dixson3 has joined #bitcoin-core-dev
355 2015-10-30T16:38:44  <wumpus> "NotMyFault: Use this executable and driver to crash your system in several different ways" lovely
356 2015-10-30T16:42:45  *** BashCo has quit IRC
357 2015-10-30T16:44:56  *** challisto has quit IRC
358 2015-10-30T16:48:52  <wumpus> the only thing hard to reproduce with a laptop would be a direct power failure
359 2015-10-30T16:51:27  <sipa> remove the battery :)
360 2015-10-30T16:53:05  <wumpus> I'm not sure that's possible with this one without disassembling it
361 2015-10-30T16:54:02  <wumpus> and not sure bitcoin core should be resilient to motherboard-bends-while-application-is-running crashes :-)
362 2015-10-30T17:04:55  *** goregrind has quit IRC
363 2015-10-30T17:05:13  *** BashCo has joined #bitcoin-core-dev
364 2015-10-30T17:10:33  *** zooko has joined #bitcoin-core-dev
365 2015-10-30T17:16:39  *** Thireus2 has quit IRC
366 2015-10-30T17:30:05  <wumpus> cfields: it worked! windows trusty build is now deterministic http://s7.postimg.org/bhrf0eui3/Naamloos.png
367 2015-10-30T17:30:48  <cfields> woohoo :)
368 2015-10-30T17:30:52  <cfields> but... wtf am i looking at?
369 2015-10-30T17:31:23  <mcelrath> What's that usha256?
370 2015-10-30T17:31:41  <wumpus> my hex-as-unicode-blockelements hack
371 2015-10-30T17:32:34  <wumpus> (somewhat easier to compare patterns than letters/numbers)
372 2015-10-30T17:33:25  <cfields> ah, neat
373 2015-10-30T17:34:13  <wumpus> "Fout bij openen blokkendatabase" ah, this time I crashed windows and the block database got corrupted
374 2015-10-30T17:35:53  <wumpus> usha256: https://gist.github.com/laanwj/3a142c48046ef862bf91
375 2015-10-30T17:36:49  <helo> corrupted your error messages too! ;)
376 2015-10-30T17:37:44  <zooko> wtf-8
377 2015-10-30T17:38:41  <wumpus> lol zooko
378 2015-10-30T17:38:48  <zooko> Seriously, what is this for ?
379 2015-10-30T17:40:21  <zooko> Wow.
380 2015-10-30T17:40:42  <zooko> Intriguing.
381 2015-10-30T17:41:13  <mcelrath> I can't decide whether that's easier to differentiate or not. Better let the computer decide: sha256sum a b | awk '{print $1}' | uniq
382 2015-10-30T17:43:13  <mcelrath> Also diff -q
383 2015-10-30T17:48:43  *** zooko has quit IRC
384 2015-10-30T17:52:06  <wumpus> mcelrath: what I don't like about that is that in the succesful case it provides no output, so I can never be sure whether it crashed (and hence produced no output) or succeeded
385 2015-10-30T17:52:29  <mcelrath> What provides no output?
386 2015-10-30T17:52:36  <wumpus> diff, when files match
387 2015-10-30T17:53:06  <sipa> wumpus: how many such block elements are there?
388 2015-10-30T17:54:21  <wumpus> sipa: I used the 16 2x2 ones to get 16 bits - but there are also e.g. 1/8th horizontal/vertical full ones, see https://en.wikipedia.org/wiki/Block_Elements
389 2015-10-30T17:54:28  <wumpus> to get 4 bits*
390 2015-10-30T17:55:24  <wumpus> and if you combine with reverse-video you can get even more, e.g. "right one eights block"
391 2015-10-30T17:56:25  <wumpus> most console applications that use thsese, use them for progress bars/gauges etc
392 2015-10-30T17:57:31  *** Thireus has joined #bitcoin-core-dev
393 2015-10-30T17:59:03  <wumpus> you could also do a kind of Chernof faces visualization using emojis, esp. on mac consoles which render those silly things in color
394 2015-10-30T18:07:45  *** PaulCape_ has quit IRC
395 2015-10-30T18:08:14  *** PaulCapestany has joined #bitcoin-core-dev
396 2015-10-30T18:12:48  <wumpus> "2015-10-30 17:30:05 Corruption: error in middle of record" that's the error with the windows chainstate db after crash
397 2015-10-30T18:14:10  *** jl2012_ has quit IRC
398 2015-10-30T18:14:25  *** jl2012 has joined #bitcoin-core-dev
399 2015-10-30T18:15:41  <wumpus> this happens while reading the log, so it is a partial record error
400 2015-10-30T18:15:53  <wumpus> partial write*
401 2015-10-30T18:22:06  *** rubensayshi has quit IRC
402 2015-10-30T18:37:08  <wumpus> https://github.com/bitcoin/bitcoin/blob/master/src/leveldb/util/env_win.cc#L587  shouldn't the FlushFuleBuffers and FlushViewOfFile be the other way around?
403 2015-10-30T18:37:22  <wumpus> the former syncs changes in the file to disk, the latter changes in the memory map to the file
404 2015-10-30T18:37:34  <wumpus> I may be misunderstanding something of course...
405 2015-10-30T18:39:32  <sipa> wumpus: that code may well be wrong
406 2015-10-30T18:40:08  * wumpus wants to implement a WritableFile with just the win32 base file API and none of this fancy mapping stuff
407 2015-10-30T18:40:50  <wumpus> like PosixWritableFile but for windows... oh wait, maybe I can even just copy PosixWritableFile and rely on mingw to do the rest for me
408 2015-10-30T18:56:37  *** CodeShark has joined #bitcoin-core-dev
409 2015-10-30T19:18:25  *** davec has quit IRC
410 2015-10-30T19:19:27  *** davec has joined #bitcoin-core-dev
411 2015-10-30T19:24:39  *** zooko has joined #bitcoin-core-dev
412 2015-10-30T19:25:34  <GitHub92> [bitcoin] sipa opened pull request #6916: Remove BIP30 enforcement, as it is impossible to trigger since BIP34 (master...nobip30) https://github.com/bitcoin/bitcoin/pull/6916
413 2015-10-30T19:26:05  <sipa> wumpus: we should do that, I think
414 2015-10-30T19:55:41  <GitHub118> [bitcoin] sipa closed pull request #6916: Remove BIP30 enforcement, as it is impossible to trigger since BIP34 (master...nobip30) https://github.com/bitcoin/bitcoin/pull/6916
415 2015-10-30T19:56:31  <gmaxwell> :)
416 2015-10-30T19:58:15  <sipa> gmaxwell: i'm trying to think whether enforcing it for all blocks (except those two) that don't have BIP34 is safe
417 2015-10-30T19:59:01  <sipa> as BIP34 is only sufficient when there are no actual duplicate coinbases
418 2015-10-30T19:59:04  <sipa> but there are
419 2015-10-30T19:59:33  <gmaxwell> sipa: if we ship a wad of the 32 least significant bits of the block hashes up to the height bip 34 activates; and then require that, that would be sufficient.
420 2015-10-30T20:01:49  <sipa> that's not very different from a checkpoint...
421 2015-10-30T20:02:36  <morcos> sipa: ok so i have an even stranger one for you
422 2015-10-30T20:03:19  <morcos> when TestBlockValidity ends, between then and when control returns to CreateNewBlock, consistently takes 10-15 ms
423 2015-10-30T20:03:38  <morcos> if you recently flushed your cache, i think this number can be 20-25 ms
424 2015-10-30T20:04:00  <gmaxwell> sipa: checkpoints don't fix this, since the corruption can slip in between them.
425 2015-10-30T20:04:06  <gmaxwell> is it spending 10-15ms in free? :)
426 2015-10-30T20:04:10  <morcos> but i have no idea what if anything should be happening then, the CCoinsViewCache destructor runs, but what else
427 2015-10-30T20:04:35  <morcos> gmaxwell: is that possible?
428 2015-10-30T20:04:58  <sipa> and that destructor doesn't do anything...
429 2015-10-30T20:05:19  <sipa> except deleting cacheCoins
430 2015-10-30T20:05:37  <sipa> morcos: you could try to do a cacheCoins.clean() in the CCoinsViewCache destructor, and time that?
431 2015-10-30T20:05:54  <morcos> ok
432 2015-10-30T20:06:13  <sipa> or add a .Clean() method that dispatches to cacheCoins.clear()
433 2015-10-30T20:06:14  <gmaxwell> morcos: it's possible e.g. freeing a kazillion and one tiny objects. I'd done some profiling in the past that suggested to me we were losing a lot of time in the allocator.
434 2015-10-30T20:06:54  <sipa> #6914 should help there too
435 2015-10-30T20:12:09  <morcos> yep, clearing the cache takes 10+ ms
436 2015-10-30T20:13:01  <sipa> try with #6914 :)
437 2015-10-30T20:13:23  <sipa> actually, it can at most give a 2x speedup
438 2015-10-30T20:13:29  <morcos> now the confusing thing is it sometimes takes twice as long if pcoinsTip (the base cache, had recently been semi-flushed)
439 2015-10-30T20:13:50  <morcos> i wonder if thats a memory thrashing thing,or it has something to do with the flags on those coins
440 2015-10-30T20:24:44  <wumpus> replaced the leveldb win32 writable file, let's see how it goes https://github.com/laanwj/bitcoin/commit/86050deaabe7ca0a19db1c5a65e3174c9cb14511
441 2015-10-30T20:25:38  <gmaxwell> \O/ always good to see a fix that removes a lot of code.
442 2015-10-30T20:26:48  <wumpus> it is! it's extremely simple now, the only thing I'm not sure about is Sync() versus Flush(), they both do a sync to disk now
443 2015-10-30T20:27:05  <wumpus> that's probably overkill but it cannot hurt either
444 2015-10-30T20:31:52  <wumpus> going to let it sync a bit and then cause another IRQL_NOT_LESS error and see if it avoids ending up in broken state
445 2015-10-30T20:32:17  <sipa> great!
446 2015-10-30T20:32:41  <sipa> we don't need particularly good write performance, i guess
447 2015-10-30T20:32:51  <sipa> as we mostly do batch writes
448 2015-10-30T20:33:24  <morcos> gmaxwell: so the easy simple thing re: BIP 30 would to just be to disable it for blocks after a certain height right?
449 2015-10-30T20:34:01  *** evoskuil has joined #bitcoin-core-dev
450 2015-10-30T20:34:23  <morcos> eh, never mind
451 2015-10-30T20:35:05  <wumpus> agreed
452 2015-10-30T20:35:19  <morcos> we could disable it for blocks that have a certain block as their ancestor
453 2015-10-30T20:35:19  *** zooko has quit IRC
454 2015-10-30T20:35:42  <morcos> such block being after BIP 34 and the pre existing duplicate coinbases diverged and were buried
455 2015-10-30T20:35:53  <sipa> morcos: fundamentally, the bip30 check can be disabled for any transaction that depends on a bip34 coinbase
456 2015-10-30T20:36:30  <gmaxwell> sipa: the existing BIP30 violations overwrote without proliferation.
457 2015-10-30T20:36:34  <sipa> but there have been duplicated coinbases in the past, so there can be transactions that spend such a duplicate, which can become duplicated as well
458 2015-10-30T20:36:39  <gmaxwell> meaning there can be no duplicate children.
459 2015-10-30T20:36:49  <sipa> gmaxwell: if you know the chain
460 2015-10-30T20:36:57  <sipa> you can be fed an alternative chain in which that's not the case
461 2015-10-30T20:37:17  <sipa> no?
462 2015-10-30T20:37:18  <morcos> right so if we only disable if you have an ancestor block which sufficietnly buries those, then you're good on the main chain
463 2015-10-30T20:37:23  <gmaxwell> Right but we could enforce BIP30 for all chains until the point where BIP34 activates.
464 2015-10-30T20:37:28  <morcos> andif you're fed an alternate chain you won't disable the check
465 2015-10-30T20:37:35  <gmaxwell> oh I see if it proliferates its not enough.
466 2015-10-30T20:37:43  <sipa> gmaxwell: BIP34 or not is not enough
467 2015-10-30T20:38:07  <sipa> even post BIP34, a transaction that spends a previous duplicate coinbase is possible (in an alternative chain)
468 2015-10-30T20:38:10  <sipa> so a checkpoint would work
469 2015-10-30T20:38:20  <gmaxwell> yea, so we could disable BIP30 when BIP34 acitivates if and only if the the ancestor chain is the real one.
470 2015-10-30T20:38:34  <morcos> yes, thats what i was trying to say
471 2015-10-30T20:39:03  <morcos> thats the best kind of checkpoint it doesn't require you to actually be on the real chain, it just saves you some work if you are
472 2015-10-30T20:39:22  <sipa> indeed
473 2015-10-30T20:41:05  <gmaxwell> the constant ancestor tests are slightly annoying but much better than running bip30 everywhere.
474 2015-10-30T20:44:57  <morcos> so erasing these caches is absurdly expensive.  sometimes its 40ms+
475 2015-10-30T20:45:13  <morcos> but why is it higher shortly after the underlying cache has been erased, that i don't understand
476 2015-10-30T20:45:13  <sipa> how many entries?
477 2015-10-30T20:45:30  <morcos> the 10ms case was on the order of 5000 entries
478 2015-10-30T20:46:00  <sipa> maybe we need a special throw-away-cache which uses an append-only pooled allocation
479 2015-10-30T20:46:29  <sipa> ... though again, a per-txout cache would be better for that
480 2015-10-30T20:46:36  <morcos> but the number of entries is irrelevant, its the number of txins in those entries i guess right?
481 2015-10-30T20:46:55  <sipa> txouts
482 2015-10-30T20:47:08  <morcos> yeah,
483 2015-10-30T20:58:03  <gmaxwell> what are the heap allocated objects in it? the entries themselves. scripts inside them (largely fixed by pieters alt-vector)? what else?
484 2015-10-30T20:58:58  *** jtimon has quit IRC
485 2015-10-30T20:59:40  <sipa> gmaxwell: one allocation in the hashmap per txid; for each txid there is one allocation for a vector of txouts; for each txout there is one allocation for the script (partially fixed by prevector)
486 2015-10-30T21:00:30  <gmaxwell> interesting that we always load all the txouts but are still taking per-txout allocation overhead.
487 2015-10-30T21:02:26  <gmaxwell> so in any case if transactions are tending to gather from many distinct txids, then you potentially have a huge blowup in allocations; it's e.g. two heap allocations for each txout, ... so if you gather from a 100 txout transaction there is 200 heap allocations for each txin that does that.
488 2015-10-30T21:03:21  <gmaxwell> (well, 201) and pieter's vector thingy would reduce that to 101 heap allocations.
489 2015-10-30T21:04:37  <gmaxwell> They're probably destroyed in the same order they're created, which is probably a fairly expensive order for malloc too.
490 2015-10-30T21:05:57  <gmaxwell> as an aside, after thinking about the heap behavior; I would guess a non-trivial amount of the 'remove on replace' memory savings is ficticious.
491 2015-10-30T21:06:05  <gmaxwell> Due to fragmentation.
492 2015-10-30T21:08:21  <morcos> "remove on replace"?
493 2015-10-30T21:09:30  <gmaxwell> attempts to remove cache entries for transactions evicted from the mempool.  Remove on reject is probably effective, since it's removing things it just added.
494 2015-10-30T21:10:02  <morcos> ah yes, well the whole thing i'm trying to accomplish here is making that unnecessary
495 2015-10-30T21:10:21  <morcos> if flushing the cache more often is doable (b/c you don't flush out the hot entries you're about to use)
496 2015-10-30T21:10:28  <gmaxwell> sipa: so an append only arena that we can dump as a single operation would ... be a thing that could be done, _but_ it leaves us in a world were the only management thing we can do is flush() which would be unfortunate.
497 2015-10-30T21:10:47  <morcos> then instead of worrying about what you're adding, just check far more often that you're not too big.  most of the time you wont' have new stuff to write to the DB anyway
498 2015-10-30T21:11:03  <sipa> gmaxwell: it would be awesome for blocks, not for the mempool
499 2015-10-30T21:11:30  <morcos> but for some reason, the cache clearing of the child cache is slower if you've done one of these controlled flushes on the parent cache (or a real full flush) right before you create the child cache
500 2015-10-30T21:11:35  <morcos> that makes no sense to me
501 2015-10-30T21:13:03  <gmaxwell> sipa: hm. even for blocks, say you process block 100 and read in transactions X,Y,Z. then you flush. Then block 101 needs transactions Y and Z again. I'm not sure how often that happens, but I would be that its way way more common than chance.
502 2015-10-30T21:15:24  *** ParadoxSpiral has quit IRC
503 2015-10-30T21:15:59  <sipa> gmaxwell: i mean for the scripts in a block
504 2015-10-30T21:16:07  <sipa> gmaxwell: the scriptSigs
505 2015-10-30T21:16:20  <sipa> not talking about UTXO set or mempool
506 2015-10-30T21:20:12  <gmaxwell> ah, indeed. you even have a nice figure on the size of the arena to allocate: just allocate the size of the block, and then it never needs to be realloced.
507 2015-10-30T21:21:18  <sipa> even for a single transaction it's probably worth it
508 2015-10-30T21:21:21  <sipa> as they're immutable
509 2015-10-30T21:21:48  <sipa> hell, a whole transaction could be a single malloced block
510 2015-10-30T21:22:00  <sipa> with some internal offsets/pointers in it
511 2015-10-30T21:24:10  <gmaxwell> sipa: thats really what I'd prefer to see. I don't think having one allocation per transaction is unreasonable; but these nested allocations are goofy.
512 2015-10-30T21:27:41  <gmaxwell> sipa: similar to that: AFAIK utxo cache entries cannot change in size.
513 2015-10-30T21:28:10  <sipa> individual txouts no; the per-tx ones can
514 2015-10-30T21:30:09  <gmaxwell> sipa: why? the only operation is that they can have entries spent, but that can be a flag that doesn't require resizing the object.
515 2015-10-30T21:31:45  <gmaxwell> (and the operation flushes the dirty part could resize the object when it does so, I suppose, as it will already have exclusive access to it, no?)
516 2015-10-30T21:32:45  <sipa> gmaxwell: you want to remove the allocated scripts for spent entries, no?
517 2015-10-30T21:33:09  <sipa> gmaxwell: otherwise a full in-memory sync from scratch needs memory for every UTXO every created, rather than a working set
518 2015-10-30T21:33:18  *** randy-waterhouse has joined #bitcoin-core-dev
519 2015-10-30T21:34:52  <gmaxwell> sipa: not quite, if it's spent completely you just drop the object. So how much partially spent would there be?  Thats also an unusual case. In any case, it could trigger a rebuild at any time you have exclusive acccess. basically batching the malloc overhead.
520 2015-10-30T21:35:28  <sipa> you always have exclusive access
521 2015-10-30T21:36:02  <gmaxwell> e.g. when you spend an entry you set a dity flag and increment a wasted bytes counter. Then if the wastes bytes are over X when you spend from it, you rebuild in place and realloc.
522 2015-10-30T21:36:06  <wumpus> partial spent (of transaction outputs) are quite common
523 2015-10-30T21:36:26  <wumpus> there are lots of transactions with many outputs
524 2015-10-30T21:36:31  <gmaxwell> And if it's spent completely, you just free it.
525 2015-10-30T21:37:44  <gmaxwell> but say a block spends the 100 outputs in order in a block, it makes no sense to resize the object 100 times and then free it.
526 2015-10-30T21:38:05  <sipa> vectors never reallocate on shrink
527 2015-10-30T21:38:14  <sipa> only on grow
528 2015-10-30T21:39:43  *** evoskuil has quit IRC
529 2015-10-30T21:39:52  <gmaxwell> sipa: With a vector we instead have 201 mallocs and 201 frees in that case. (and probably a few reallocs along the way)
530 2015-10-30T21:40:28  *** zooko has joined #bitcoin-core-dev
531 2015-10-30T21:41:02  *** paveljanik has quit IRC
532 2015-10-30T21:41:57  <gmaxwell> where as if the cache entry is a single fixed size object, which never grows, and shrinks only in a deferred way when the amount of memory saved by shrinking would be non-trivial, it's 1 malloc, 1 free, and a few reallocs for shrinking along the way... and always in chunks large enough where its meaningful (e.g. where fragmentation doesn't prevent the freed memory from being used anyways).
533 2015-10-30T21:43:29  <GitHub21> [bitcoin] laanwj opened pull request #6917: leveldb: Win32WritableFile without memory mapping (master...2015_10_leveldb_win_nomap) https://github.com/bitcoin/bitcoin/pull/6917
534 2015-10-30T21:44:19  <wumpus> seems to have worked...
535 2015-10-30T21:44:42  <gmaxwell> \O/
536 2015-10-30T21:46:17  <gmaxwell> too bad the chainstate obfscuation is a bit invasive; it would be nice to have a 0.11.2 that fixes all known corruption issues for windows users.
537 2015-10-30T21:47:41  <sipa> i think it's backportable
538 2015-10-30T21:48:05  <wumpus> it's not too bad
539 2015-10-30T21:49:18  <gmaxwell> (it wasn't that interesting to consider when the leveldb ones remained)
540 2015-10-30T21:52:41  <gmaxwell> warren: if you want to do a bounty, instead of fixing windows (which wumpus just did, and we should send him flowers)-- perhaps the bounty we should ask for is this:  A NBD server that sakes an initial disk image file (compatible with loopback mount) and logs all writes to it, instead of actually writing them, and then applies the writes as a patch in memory (doesn't matter if uses lots of ram).
541 2015-10-30T21:52:48  <gmaxwell>   Then it lets you restart it with a image + old log + N and it will truncate the last N writes off the old log.  This would let you construct a test harness where we start a node against an existing chain, sync one block, and shut down. Then attempt to restart with one write less. and so on, for every single write back to the first one, and make sure the node can restart at every point.
542 2015-10-30T21:54:57  *** Guyver2 has quit IRC
543 2015-10-30T21:58:00  <wumpus> yes, that could be useful
544 2015-10-30T22:03:26  <gmaxwell> without that kind of test harness I dunno how _anyone_ manages to write database software that actually works (well: answer, according to that usenix paper, they mostly just fail. :) )
545 2015-10-30T22:04:24  <warren> gmaxwell: you might have convinced me that I shouldn't be spending personal income on bounties anymore, I'd be happy to specify it though =)
546 2015-10-30T22:04:52  <warren> kanzure: ping
547 2015-10-30T22:05:04  <kanzure> pong
548 2015-10-30T22:05:34  <kanzure> for which thing have i been summoned?
549 2015-10-30T22:07:12  <wumpus> there are a few debugging and reverse engineering tools that can rewind execution in memory, but I know of no file systems that can rewind/replay operations explicitly in fine granularity. You'd say that with journalling file systems it should be possible by truncating/rewinding the journal, but never been able to find reports of anyone doing that...
550 2015-10-30T22:07:23  <gmaxwell> kanzure: ever seen people who own both a parrot and a dog, ... and the parrot will sometimes call the dog just to see it come?
551 2015-10-30T22:07:39  <kanzure> there was a linux tool for replay but requires kernel module, does replay and rewind of execution state, very bizarre alien thing
552 2015-10-30T22:07:59  <gmaxwell> wumpus: I wasted a bunch of time searching and could find nothing.  But it seems to me that this is probably only a few days work on top of a simplistic NBD server.
553 2015-10-30T22:08:00  <kanzure> http://velvetpulse.com/2012/11/27/scribe-the-deterministic-transparent-record-replay-engine/
554 2015-10-30T22:08:04  <kanzure> and maybe http://rr-project.org/ but more the last link
555 2015-10-30T22:08:20  <wumpus> yes, for execution state for example pandare can do that
556 2015-10-30T22:08:22  <kanzure> this could possibly be extended to windows vm things but w/e
557 2015-10-30T22:08:37  <gmaxwell> RR is awesome, and can do this for execution state, but we need it for disk state seperate from execution.
558 2015-10-30T22:08:41  <kanzure> can't find pandare, please link?
559 2015-10-30T22:09:08  <wumpus> but that's not really what we want here - we want to restart the application at every point of disk writing
560 2015-10-30T22:09:25  <wumpus> kanzure: https://github.com/moyix/panda
561 2015-10-30T22:09:34  <gmaxwell> I expect most of the work related to my NBD suggestion will be discovering that all filesystems are broken, and being unable to get results against bitcoin until the file systems are fixed. :(
562 2015-10-30T22:09:50  <kanzure> yeah probably just write your own fusefs plugin or something
563 2015-10-30T22:10:06  <kanzure> and ramfs with snapshotting or something
564 2015-10-30T22:10:10  * kanzure handwaves
565 2015-10-30T22:10:27  <gmaxwell> NBD is bascially the perfect interface for this. It is a remote block device whos primitives are "read this block" and "write this block"
566 2015-10-30T22:11:40  <kanzure> "network block device"
567 2015-10-30T22:11:46  <kanzure> i hope this isn't samba
568 2015-10-30T22:11:52  <gmaxwell> so the code becomes  if read  check the log replay hashtable, no hits? go to the read only backing device. if hit return that.  if write, append log, and add to the hashtable.   On restart, replay old log into hashtable up to the cutoff point.
569 2015-10-30T22:12:35  <wumpus> samba is not a network block device, but a network filesystem :)
570 2015-10-30T22:12:45  <kanzure> (samba is many things)
571 2015-10-30T22:14:51  <gmaxwell> the test harness is, mount the thing, sync a block. shut down. unmount. restart daemon with -1, mount, start daemon sync next block and record success, unmount, go back to the original oldlog, start with -2 ... and so on. for each offset. If there are a million writes made while syncing that first block, you restart a million times.   Once that starts failing, you change the behavior so that it
572 2015-10-30T22:14:57  <gmaxwell> turns the last write to line-noise... and try it again.
573 2015-10-30T22:16:02  <gmaxwell> and then you know that, absent reordering at lower levels, there is no point where unclean termination can leave the state irrecoverable in the log-under-test.
574 2015-10-30T22:16:52  <gmaxwell> obviously this could model reordering too, if barriers get communicated across NBD (I dunno if they do)... but the state space becomes huge fast, so I think you could only try random cases, and not an exhaustive test.
575 2015-10-30T22:18:25  <kanzure> warren: random pings in the future are much appreciated, thank you
576 2015-10-30T22:19:57  <gmaxwell> wumpus: please post test binaries with the leveldb fix. 99.9% of windows users won't test from source.
577 2015-10-30T22:20:01  <wumpus> the randomized brute force approach would be to crash, restart, crash, restart a VM running bitcoind in a loop, then a zillion times, trying to get it into a state where it won't recover
578 2015-10-30T22:20:34  <GitHub91> [bitcoin] sipa opened pull request #6918: Make sigcache faster, more efficient, larger (master...smallsigcache) https://github.com/bitcoin/bitcoin/pull/6918
579 2015-10-30T22:20:43  <wumpus> gmaxwell: maybe jonasschnelli would like to post executables; I don't have the infrastrcuture for that at the moment
580 2015-10-30T22:20:48  <gmaxwell> sipa: "faster, stronger, better!"
581 2015-10-30T22:21:26  <kanzure> you could selectively break parts of the vm maybe
582 2015-10-30T22:21:34  <kanzure> e.g. increase unreliability of different system calls
583 2015-10-30T22:23:36  <gmaxwell> sipa: hitrate could be improved further if you store the height at the time an entry is added the random eviction picks N entries and drops the one with the lowest height.
584 2015-10-30T22:24:27  <gmaxwell> sipa: even with it 10x larger, hitrate for a transactions in the current mempool will not be 100% and thats nuts.
585 2015-10-30T22:25:53  *** randy-waterhouse has quit IRC
586 2015-10-30T22:26:13  <sipa> gmaxwell: around 99.3% hitrate on average for the last 8000 entries
587 2015-10-30T22:28:19  <gmaxwell> so lets say the block has 2000 signatures drawn from the last 8000, and a 99.3% hitrate. So 14 misses.  If a verify takes 420 microseconds (as it does for openssl), then thats 6 ms added to block processing.
588 2015-10-30T22:28:49  <sipa> the last 1000 have 99.9% hitrate
589 2015-10-30T22:29:10  <sipa> one way to do it would be to use a unordered_map to int64_t, which just increases sequentially
590 2015-10-30T22:29:23  <sipa> and when removing, randomly find 4 entries and sort them by that number
591 2015-10-30T22:29:33  <gmaxwell> or 1%-6% slowdown (depending on how much we speed up the rest of block processing).
592 2015-10-30T22:29:52  <sipa> maintaining a fully ordered index adds a lot of memory
593 2015-10-30T22:29:58  <gmaxwell> I was basically suggesting that but using height, because I think we actually want the oldest of the ones at the current height usually.
594 2015-10-30T22:30:46  <sipa> otherwise we need to pass down height through several layers of code that are currently height-oblivious
595 2015-10-30T22:30:48  <gmaxwell> so map to int32_t, with the height at insertion time. and then pull four entries and reject the lowest height.
596 2015-10-30T22:31:11  <gmaxwell> oh well meh, true. though block height advance is the magical event that should cause entries to become useless. Not age.
597 2015-10-30T22:31:39  <sipa> we could make it wipe entries on block verify
598 2015-10-30T22:31:56  <sipa> that's trivial
599 2015-10-30T22:33:09  <gmaxwell> sounds even better. I think with that we could avoid increasing its memory usage, and stay at 10mb.
600 2015-10-30T22:34:15  <gmaxwell> sipa: if we can do that, then we could also have that same event trigger a counter that a block has been accepted, no?
601 2015-10-30T22:34:35  <sipa> gmaxwell: it's a boolean passed down saying whether we want to store or not
602 2015-10-30T22:34:43  <sipa> per entry
603 2015-10-30T22:34:49  <phantomcircuit> er
604 2015-10-30T22:34:51  <phantomcircuit>     bool Encrypt(const CKeyingMaterial& vchPlaintext, std::vector<unsigned char> &vchCiphertext);
605 2015-10-30T22:35:03  <phantomcircuit> actually nvm
606 2015-10-30T22:35:10  <phantomcircuit> i had them reversed in my brain
607 2015-10-30T22:36:23  <gmaxwell> sipa: the eviction is probably enough, though it still has a bias towards the more recent inserts rather than a bias towards preserving the oldest inserts which are since the prior block.
608 2015-10-30T22:36:48  <sipa> gmaxwell: meh
609 2015-10-30T22:37:41  <gmaxwell> I don't think it matters much, but evict on verify will make that sighash single trick for spending dustspam slow to verify.
610 2015-10-30T22:41:48  <gmaxwell> sipa: what is the average hit rate for a transactions 6000 to 8000 ago?
611 2015-10-30T22:42:46  <phantomcircuit> gmaxwell, to verify that im reading this right, each time an encrypted wallet is unlocked the first key in mapCryptedKeys is checked against it's pubkey and the first time the wallet is unlocked all of the keys are checked against their pubkey, correct?
612 2015-10-30T22:44:48  <gmaxwell> phantomcircuit: thats my recollection of what it does, yes.
613 2015-10-30T22:45:21  <gmaxwell> it's a slow and not very good test-- the problem, of course, is you get no verification at all if the wallet is never unlocked.
614 2015-10-30T22:45:36  *** [b__b] has quit IRC
615 2015-10-30T22:45:57  *** [b__b] has joined #bitcoin-core-dev
616 2015-10-30T22:46:16  <phantomcircuit> gmaxwell, right
617 2015-10-30T22:46:36  <phantomcircuit> how about make a list for eviction and evict only when a block is completely verified?
618 2015-10-30T22:47:16  <gmaxwell> phantomcircuit: complicated and screws up the layering.
619 2015-10-30T22:47:40  <gmaxwell> phantomcircuit: I'd rather have a hash basked check that stays with the key, and gets checked on load and any time its used.
620 2015-10-30T22:48:06  <phantomcircuit> gmaxwell, no i meant for the sigcache thing
621 2015-10-30T22:48:26  <gmaxwell> phantomcircuit: I know, mutiple threads of discussion.
622 2015-10-30T22:48:27  <sipa> phantomcircuit: complicated and screws up the layering :)
623 2015-10-30T22:48:43  <sipa> phantomcircuit: the sigcache code doesn't know about a 'block' concept at all
624 2015-10-30T22:49:43  <gmaxwell> sipa: I don't see why the sigcache couldn't just have a counter in it, and every time you verify a block you take a lock on the sigcache and increment that counter.  And the current counter value rides with the entries in an ordered map, and does the biased eviction I suggested.
625 2015-10-30T22:50:03  <sipa> gmaxwell: perfectly doable
626 2015-10-30T22:50:05  <gmaxwell> that won't require passing anything height around, it's not hight anymore then, it's just 'sigcache epoch'
627 2015-10-30T22:50:38  <sipa> going to benchmark without it to see how large the sigcache actually gets
628 2015-10-30T22:51:45  <gmaxwell> well it will get to the maximum size, since there is no eviction except on full. :)
629 2015-10-30T22:52:00  <sipa> with the evict-when-seen-in-block rule
630 2015-10-30T22:54:50  <phantomcircuit> gmaxwell, heh
631 2015-10-30T22:54:56  <gmaxwell> k, that does have that patalogical performance I mentioned. sadly I don't see a nice way to fix that, except perhaps having a one entry FIFO for the last evicted item.
632 2015-10-30T22:55:34  <sipa> gmaxwell: hmm, what pathological performance?
633 2015-10-30T22:56:59  <gmaxwell> sipa: There is, for example, a block with a 1MB tx of SIGHASH_SINGLE bug using transactions that have all the same signature.   If it's cached its almost instant to verify ... otherwise....
634 2015-10-30T22:57:15  <sipa> oh!
635 2015-10-30T22:57:22  <gmaxwell> similar OP_DUP2 CHECKSIG OP_DUP2 CHECKSIG ... (or checkmultisig with the same pubkey).
636 2015-10-30T22:58:26  *** belcher has joined #bitcoin-core-dev
637 2015-10-30T23:03:35  <phantomcircuit> we still end up calling SignatureHash even with the sigcache right?
638 2015-10-30T23:04:31  <sipa> yes
639 2015-10-30T23:05:54  <phantomcircuit> any reason CryptedKeyMap is in keystore.h instead of src/wallet/crypter.h ? (it's not used anywhere outside of src/wallet/crypter.{h,cpp})
640 2015-10-30T23:06:03  <sipa> nope
641 2015-10-30T23:06:18  <sipa> CKeyStore itself should move to script/sign.h
642 2015-10-30T23:06:28  <sipa> CBasicKeyStore can stay
643 2015-10-30T23:06:36  <sipa> and CCryptoKeyStore can indeed move to wallet/crypter
644 2015-10-30T23:12:45  <phantomcircuit> gmaxwell, also yes that was my basic plan for adding hashing for ckey entries
645 2015-10-30T23:22:06  *** zooko has quit IRC
646 2015-10-30T23:22:29  *** danielsocials has joined #bitcoin-core-dev
647 2015-10-30T23:22:36  *** zooko has joined #bitcoin-core-dev
648 2015-10-30T23:34:47  <GitHub83> [bitcoin] laanwj closed pull request #6893: forward-declare univalue in rpcclient + remove include in rpcserver.cpp (master...univalue) https://github.com/bitcoin/bitcoin/pull/6893
649 2015-10-30T23:37:30  *** zooko has quit IRC
650 2015-10-30T23:38:58  <GitHub43> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/725539ea0376...d482c0a7b246
651 2015-10-30T23:38:58  <GitHub43> bitcoin/master e9e6163 Pieter Wuille: Make -checkmempool=1 not fail through int32 overflow
652 2015-10-30T23:38:59  <GitHub43> bitcoin/master d482c0a Wladimir J. van der Laan: Merge pull request #6896...
653 2015-10-30T23:39:03  <GitHub192> [bitcoin] laanwj closed pull request #6896: Make -checkmempool=1 not fail through int32 overflow (master...fixchainsize) https://github.com/bitcoin/bitcoin/pull/6896
654 2015-10-30T23:40:05  <GitHub164> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/d482c0a7b246...48b5b84ee511
655 2015-10-30T23:40:05  <GitHub164> bitcoin/master 30d9662 Gregory Maxwell: Reject invalid pubkeys when reading ckey items from the wallet....
656 2015-10-30T23:40:06  <GitHub164> bitcoin/master 48b5b84 Wladimir J. van der Laan: Merge pull request #6906...
657 2015-10-30T23:40:15  <GitHub146> [bitcoin] laanwj closed pull request #6906: Reject invalid pubkeys when reading ckey items from the wallet. (master...ckey_pubkey_check) https://github.com/bitcoin/bitcoin/pull/6906
658 2015-10-30T23:46:29  *** dcousens has joined #bitcoin-core-dev
659 2015-10-30T23:47:11  *** danielsocials has quit IRC
660 2015-10-30T23:48:39  *** evoskuil has joined #bitcoin-core-dev
661 2015-10-30T23:49:26  <dcousens> sipa: don't mind my nits
662 2015-10-30T23:49:40  <sipa> dcousens: you're welcome to ask questions :)
663 2015-10-30T23:50:54  <dcousens> PR is super clean, the only semantics that weren't clear to me was the logic around `store`
664 2015-10-30T23:52:02  <dcousens> What does it represent? A data 'store', or, whether to allow the sigCache to be used?
665 2015-10-30T23:52:28  <sipa> store==true is set for mempool transactions, store==false is set for block validation
666 2015-10-30T23:52:59  <sipa> the original intent was that we wouldn't store validations done during block validation, only query the cache
667 2015-10-30T23:54:16  <dcousens> So the current usage is whether or not it should use a cache at all
668 2015-10-30T23:54:25  <sipa> no
669 2015-10-30T23:54:41  <sipa> with store==false, the cache is still queried; the result of validation is just not stored
670 2015-10-30T23:54:53  <dcousens> But the cache would be empty no?
671 2015-10-30T23:56:17  <dcousens> The set only gets inserted into if `store == true`
672 2015-10-30T23:56:24  <sipa> it gets filled by mempool validations
673 2015-10-30T23:56:30  <sipa> and queried for block validations
674 2015-10-30T23:56:33  *** PaulCape_ has joined #bitcoin-core-dev
675 2015-10-30T23:56:37  <dcousens> AH
676 2015-10-30T23:56:41  <dcousens> bloody inline statics
677 2015-10-30T23:56:44  <dcousens> sorry, missed the static
678 2015-10-30T23:56:55  <dcousens> That makes sense now
679 2015-10-30T23:57:30  <dcousens> Though signatureCache was a member of CachingTransactionSignatureChecker
680 2015-10-30T23:57:37  <dcousens> Thought*
681 2015-10-30T23:59:35  *** PaulCapestany has quit IRC
682 2015-10-30T23:59:36  <dcousens> I'm too used to just never having globals haha, anyway, cheers for answering questions :)