1 2016-07-13T00:06:48  *** Cory has joined #bitcoin-core-dev
  2 2016-07-13T00:15:45  *** belcher has quit IRC
  3 2016-07-13T00:23:28  *** Chris_Stewart_5 has quit IRC
  4 2016-07-13T00:30:12  *** Chris_Stewart_5 has joined #bitcoin-core-dev
  5 2016-07-13T00:36:47  *** Cory has quit IRC
  6 2016-07-13T00:39:49  *** fengling has joined #bitcoin-core-dev
  7 2016-07-13T00:40:54  *** luke-jr has quit IRC
  8 2016-07-13T00:44:33  *** Chris_Stewart_5 has quit IRC
  9 2016-07-13T00:53:01  <phantomcircuit> sipa, i want to (optinally) save the mempool and sigcache
 10 2016-07-13T00:53:04  <phantomcircuit> on shutdown
 11 2016-07-13T00:53:08  <phantomcircuit> where's the right place to put that?
 12 2016-07-13T00:54:19  <gmaxwell> don't save the sigcache. the reload of the mempool from disk will repopulate it.
 13 2016-07-13T00:54:29  <gmaxwell> the stuff that doesn't get repopulated shouldn't be there in any case.
 14 2016-07-13T00:58:23  *** justanotheruser has quit IRC
 15 2016-07-13T00:59:09  *** justanotheruser has joined #bitcoin-core-dev
 16 2016-07-13T01:03:13  *** spudowiar has quit IRC
 17 2016-07-13T01:09:46  *** Chris_Stewart_5 has joined #bitcoin-core-dev
 18 2016-07-13T01:12:32  <phantomcircuit> gmaxwell, yes but repopulating the mempool will take forever without saving the sigcache
 19 2016-07-13T01:15:53  <gmaxwell> phantomcircuit: 300mb of validation isn't ~that~ big a deal.
 20 2016-07-13T01:16:08  <gmaxwell> at least so long as it reloads the mempool in the background.
 21 2016-07-13T01:16:55  <gmaxwell> okay, perhaps worse than I'm thinking due to the fact that the signature validation isn't parallel
 22 2016-07-13T01:18:17  <gmaxwell> but even if the 300mbytes is alll txins, thats 145 bytes per signature, or about 2 million signatures, which even on a single moderately fast core should only be about 121 cpu seconds.
 23 2016-07-13T01:19:10  <gmaxwell> well whatever, 30% slower since I'm assuming libsecp256k1 with performance features we're not using yet.
 24 2016-07-13T01:24:34  <phantomcircuit> gmaxwell, it's quite a bit more than 300MiB on my system :)
 25 2016-07-13T01:25:24  <gmaxwell> phantomcircuit: having a mempool much larger than typical shouldn't increase your hitrate much and if you're mining may cause your to mine poorly propagated crap.
 26 2016-07-13T01:27:09  <phantomcircuit> it wont improve the hitrate for sigcache or compact blocks very much
 27 2016-07-13T01:27:22  <phantomcircuit> i dont see how it can result in you mining poorly propagated things though?
 28 2016-07-13T01:30:03  <gmaxwell> say someone advertises a txn with a very low feerate, it doesn't enter or quickly falls out of most nodes mempools.
 29 2016-07-13T01:30:05  <Chris_Stewart_5> If I add a different version of boost to my /usr/include I end up getting errors with core not being able to find it, is there some where I need to reference a newer version?
 30 2016-07-13T01:30:08  <Chris_Stewart_5> cfields:
 31 2016-07-13T01:30:58  <gmaxwell> phantomcircuit: but not yours... later, the network runs low on transactions in the pool or CPFP ranks that straggler up. now you mine it. And it's a surprise to everyone.
 32 2016-07-13T01:35:46  *** Ylbam has quit IRC
 33 2016-07-13T01:47:58  *** TomMc has quit IRC
 34 2016-07-13T01:57:55  <petertodd> bsm117532: for consensus critical applications, you're going to need to use the raw iterator, and even worse, follow the byte-level tests in the segwit codebase exactly
 35 2016-07-13T02:05:24  *** Chris_Stewart_5 has quit IRC
 36 2016-07-13T02:22:27  *** TomMc has joined #bitcoin-core-dev
 37 2016-07-13T02:23:39  *** go1111111 has quit IRC
 38 2016-07-13T02:25:21  *** DigiByteDev has joined #bitcoin-core-dev
 39 2016-07-13T02:33:18  *** DigiByteDev has joined #bitcoin-core-dev
 40 2016-07-13T02:34:32  *** DigiByteDev has joined #bitcoin-core-dev
 41 2016-07-13T02:39:41  *** Cory has joined #bitcoin-core-dev
 42 2016-07-13T03:04:12  <phantomcircuit> gmaxwell, true
 43 2016-07-13T03:04:30  <phantomcircuit> i think that is generally not easily solved though
 44 2016-07-13T03:04:45  <phantomcircuit> indeed unless you have the exact same limit as the entire rest of the network
 45 2016-07-13T03:04:48  <phantomcircuit> you're screwed there
 46 2016-07-13T03:04:56  *** TomMc has quit IRC
 47 2016-07-13T03:05:03  <phantomcircuit> otoh my brain is on fire so maybe im wrong
 48 2016-07-13T03:09:41  *** hsmiths has quit IRC
 49 2016-07-13T03:18:20  *** hsmiths has joined #bitcoin-core-dev
 50 2016-07-13T03:22:52  *** cryptapus_ has joined #bitcoin-core-dev
 51 2016-07-13T03:31:40  *** cryptapus_ has quit IRC
 52 2016-07-13T03:42:22  *** GreenIsMyPepper has quit IRC
 53 2016-07-13T03:51:31  *** cryptapus_ has joined #bitcoin-core-dev
 54 2016-07-13T03:51:31  *** cryptapus_ has joined #bitcoin-core-dev
 55 2016-07-13T03:55:46  *** GreenIsMyPepper has joined #bitcoin-core-dev
 56 2016-07-13T04:06:38  *** GreenIsMyPepper has quit IRC
 57 2016-07-13T04:11:47  *** GreenIsMyPepper has joined #bitcoin-core-dev
 58 2016-07-13T04:17:33  *** cryptapus_ has quit IRC
 59 2016-07-13T04:20:47  *** cryptapus_ has joined #bitcoin-core-dev
 60 2016-07-13T04:34:48  *** GreenIsMyPepper has quit IRC
 61 2016-07-13T04:40:18  *** GreenIsMyPepper has joined #bitcoin-core-dev
 62 2016-07-13T05:11:34  *** cryptapus_ has quit IRC
 63 2016-07-13T05:12:03  *** jtimon has joined #bitcoin-core-dev
 64 2016-07-13T05:25:51  *** frankenmint has joined #bitcoin-core-dev
 65 2016-07-13T05:30:16  *** GreenIsMyPepper has quit IRC
 66 2016-07-13T05:43:22  *** GreenIsMyPepper has joined #bitcoin-core-dev
 67 2016-07-13T05:44:11  *** frankenmint has quit IRC
 68 2016-07-13T05:49:51  *** fengling_ has joined #bitcoin-core-dev
 69 2016-07-13T05:50:06  *** fengling has quit IRC
 70 2016-07-13T05:56:16  *** GreenIsMyPepper has quit IRC
 71 2016-07-13T06:07:56  *** GreenIsMyPepper has joined #bitcoin-core-dev
 72 2016-07-13T06:08:15  *** morcos has quit IRC
 73 2016-07-13T06:08:16  *** zxzzt has quit IRC
 74 2016-07-13T06:09:02  *** zxzzt has joined #bitcoin-core-dev
 75 2016-07-13T06:09:07  *** morcos has joined #bitcoin-core-dev
 76 2016-07-13T06:10:10  *** DigiByteDev has quit IRC
 77 2016-07-13T06:11:48  *** go1111111 has joined #bitcoin-core-dev
 78 2016-07-13T06:12:18  *** GreenIsMyPepper has quit IRC
 79 2016-07-13T06:13:55  *** GreenIsMyPepper has joined #bitcoin-core-dev
 80 2016-07-13T06:39:23  *** Michail1 has quit IRC
 81 2016-07-13T06:40:37  *** davidlj95 has joined #bitcoin-core-dev
 82 2016-07-13T06:47:13  *** assder has joined #bitcoin-core-dev
 83 2016-07-13T06:56:10  *** bustd_soket has quit IRC
 84 2016-07-13T07:12:04  *** chris2000 has joined #bitcoin-core-dev
 85 2016-07-13T07:15:08  *** Ylbam has joined #bitcoin-core-dev
 86 2016-07-13T07:19:50  *** chris2000 has quit IRC
 87 2016-07-13T07:19:52  <GitHub99> [bitcoin] maiiz opened pull request #8336: Issues #8334 (master...issues-8334) https://github.com/bitcoin/bitcoin/pull/8336
 88 2016-07-13T07:20:37  *** bustd_soket has joined #bitcoin-core-dev
 89 2016-07-13T07:25:02  *** murch has joined #bitcoin-core-dev
 90 2016-07-13T07:25:38  <murch> gmaxwell: https://github.com/bitcoin/bitcoin/issues/7664#issuecomment-232230899 <-- I think you meant "is a large multiple" instead of "isn't"
 91 2016-07-13T07:28:57  *** Cheeseo has quit IRC
 92 2016-07-13T07:29:49  <jtimon> so NicolasDorier and I are discussing libconsensus
 93 2016-07-13T07:29:52  <gmaxwell> murch: ya, thanks.
 94 2016-07-13T07:30:57  <jtimon> do we want locks inside libconsensus? my understanding was no, due to complexity reasons. NicolasDorier thinks it's fine with std C++11 but it wasn't fine with boost
 95 2016-07-13T07:31:41  <jtimon> well, personally I think libconsensus should move towards plain C, but that's another discussion it seems I lost already
 96 2016-07-13T07:32:36  <sipa> jtimon: i was talking about this with NicolasDorier earlier
 97 2016-07-13T07:32:54  *** paveljanik has quit IRC
 98 2016-07-13T07:33:03  <sipa> jtimon: i expected that the point of contention would be the idea of it managing its own state
 99 2016-07-13T07:33:26  <sipa> (which would imply using locks for that state)
100 2016-07-13T07:33:36  <NicolasDorier> ah yes you wanted to keep the libconsensus "stateless"
101 2016-07-13T07:34:18  <sipa> i don't know whether it should be stateless - having state, especially for caches, would hugely simplify things
102 2016-07-13T07:34:44  <sipa> but i think that should be the first thing to discuss
103 2016-07-13T07:34:55  <sipa> as it using locks or not follows naturally
104 2016-07-13T07:36:36  <phantomcircuit> sipa: that pr doesn't seem to have improved removeForBlock times by much
105 2016-07-13T07:36:58  <gmaxwell> phantomcircuit: that means it improved them at all?
106 2016-07-13T07:37:33  <phantomcircuit> im gonna need to do the mempool save/restor thing to know for sure unfortunately
107 2016-07-13T07:37:54  <phantomcircuit> if it did it certainly wasn't spectacular
108 2016-07-13T07:38:21  <NicolasDorier> I'm working right now on a verify_block for consensus lib. (like jtimon doing) I think the question of state won't come immediately to me as I first need to finish that. Once I done my verify_blocks, I'll just make several proposal for handling the hash cache of segwit
109 2016-07-13T07:38:52  <NicolasDorier> I have several idea, will just try to code them see if they make sense at all :p
110 2016-07-13T07:40:09  <sipa> NicolasDorier: how does your verifyblock get access to the chainstate and the block index?
111 2016-07-13T07:40:23  *** KwukDuck has joined #bitcoin-core-dev
112 2016-07-13T07:40:50  <KwukDuck> My Bitcoin Core client keeps crashing since a few days... http://pastebin.com/fiPXrzZY
113 2016-07-13T07:40:53  <KwukDuck> any ideas?
114 2016-07-13T07:41:18  <jtimon> sipa: well, I think the interface for the caches can be really simple, and we can provide the implementations we have for those who don't want to implement their own, so I disagree on "hugely simplify things"
115 2016-07-13T07:41:35  <NicolasDorier> sipa: By cheating, I delegate to the client the calculation of the flags and contextual informations (height, bestblock, mtp)
116 2016-07-13T07:41:56  <NicolasDorier> such that all the consensus code does not depends on CBlockIndex at all
117 2016-07-13T07:42:03  <NicolasDorier> but just to 2 types
118 2016-07-13T07:42:04  <phantomcircuit> KwukDuck, hardware error
119 2016-07-13T07:42:13  <NicolasDorier> CConsensusContextInfo and CConsensusFlags
120 2016-07-13T07:42:35  <jtimon> but yeah, my plan has always been to have a C++ version of verifyBlock first, well, I mean, first VerifyHeader, verifyTx...but never got past the point of simply moving the consensus functions out of main...
121 2016-07-13T07:42:37  <NicolasDorier> both can be calculated by "main" with pIndex trivially. Then passed to consensus methods
122 2016-07-13T07:42:39  <sipa> jtimon: yes, but that means the api changes whenever a cache is added/changed
123 2016-07-13T07:43:13  <jtimon> yes, if you add a cache the api changes, not sure what you mean by changing the cache
124 2016-07-13T07:43:30  <sipa> if it ends up working differently somehow
125 2016-07-13T07:43:34  <jtimon> just as if the storage changes, the api for the storage would change as well
126 2016-07-13T07:44:12  <NicolasDorier> I started working on https://github.com/NicolasDorier/bitcoin/commits/blkconsensus but still not done
127 2016-07-13T07:44:13  <jtimon> assuming people want libconsensus to be storageless
128 2016-07-13T07:44:40  <sipa> jtimon: yeah, i understand the use case
129 2016-07-13T07:45:29  <sipa> it's just a lot harder to introduce abstractions for block indexes and chainstate, especially without degrading performance
130 2016-07-13T07:45:48  <jtimon> I was working on https://github.com/jtimon/bitcoin/commits/jt but I broke the branch when backported bip9 and never cared to fix it (but review is still very welcomed since I plan to rewrite/cherry pick most of the stuff from there)
131 2016-07-13T07:46:34  <jtimon> my interfaces: https://github.com/jtimon/bitcoin/commit/c97fa96c2f9e8989b9f1a9dd41688e07bcc163b9#diff-c2a099d775bac1dccc5f146a3cda81b8R82 https://github.com/jtimon/bitcoin/commit/34780bd9af20ddf60dd309eff90b9caf01b63f64
132 2016-07-13T07:47:53  <NicolasDorier> my main problem with interface is that the callback will probably cross language boundary (C++ calling C#) and it is a performance nightmare (but maybe it has gone better now). I'm not against callback but this has to be coarse grained.
133 2016-07-13T07:48:28  <NicolasDorier> ho I finish my PR I'll see when I hit there
134 2016-07-13T07:49:13  <KwukDuck> phantomcircuit: Hardware error? Like what..? I think my hardware is fine? Experiencing no other issues with the system, even tried running it on a different disk.
135 2016-07-13T07:49:56  <jtimon> at least for bitcoin core as a caller (assuming one day bitcoin core directly calls libconsensus instead of its internals) I don't see how performance would be affected very negatively, I worry more about serializing and deserialzing the parameters (like the tx in verify) for example, since people have complained (that's the reason why libbitcoin copies the code instead of using libconsensus' API directly)
136 2016-07-13T07:50:32  <gmaxwell> KwukDuck: that kind of issue is usually caused by hardware error, lose or bad disk cables, or bad memory (run memtest86?)-- sometimes antivirus software can screw with the files bitcoin is using.
137 2016-07-13T07:50:57  <jtimon> NicolasDorier: as said privately, I think people should use the function pointer to call their DB directly, not calling their external language there, but you said that may be complicated too
138 2016-07-13T07:51:00  <gmaxwell> or running out of disk space potentially.
139 2016-07-13T07:51:54  <NicolasDorier> jtimon: yes, it really depends on the backend technology. Some might have poor support of C lib. :(
140 2016-07-13T07:52:00  <KwukDuck> gmaxwell: I considered a bad disk that's why i ran it on another one. Not sure about the memory though, but i'm not experiencing any other issues so... I guess i'll be running memtest later. Thanks :)
141 2016-07-13T07:52:25  <NicolasDorier> also I know I'm not pro in C so it would take me 10 times  more than you to make use of a C lib even if it existed
142 2016-07-13T07:53:04  <jtimon> on another topic, should utilmoneystr be part of libconsensus only for this line or should we just write the message in satoshis? https://github.com/bitcoin/bitcoin/pull/8329/files#diff-ca81084f62961a188f5c1e86a5ff1d7cR238
143 2016-07-13T07:53:25  <jtimon> maaku things yes it should be part of libconsensus, I'm not very convinced
144 2016-07-13T07:53:51  *** kadoban has quit IRC
145 2016-07-13T07:55:00  <jtimon> if we're sure about it belonging in libconsensus, I should squash https://github.com/bitcoin/bitcoin/pull/8329/commits/8af7c64f6908ea4526efde9896d046c431016e00 in #8328
146 2016-07-13T07:55:53  <jtimon> NicolasDorier: thoughts ?
147 2016-07-13T07:57:00  <NicolasDorier> checking that a sec
148 2016-07-13T07:57:43  *** laurentmt has joined #bitcoin-core-dev
149 2016-07-13T07:58:34  <NicolasDorier> jtimon: well, I think the smaller the diff the better, so I am more inclined to keep it for now, but not really strongly neither
150 2016-07-13T08:01:47  <jtimon> well, if I keep it and move it to the consensus dir, that actually makes the diff bigger
151 2016-07-13T08:02:30  *** laurentmt has quit IRC
152 2016-07-13T08:16:22  *** DigiByteDev has joined #bitcoin-core-dev
153 2016-07-13T08:20:59  *** DigiByteDev has quit IRC
154 2016-07-13T08:34:42  *** AaronvanW has quit IRC
155 2016-07-13T08:37:21  *** AaronvanW has joined #bitcoin-core-dev
156 2016-07-13T08:37:21  *** AaronvanW has joined #bitcoin-core-dev
157 2016-07-13T08:41:34  *** Guyver2 has joined #bitcoin-core-dev
158 2016-07-13T08:53:07  *** chris200_ has joined #bitcoin-core-dev
159 2016-07-13T09:43:01  *** laurentmt has joined #bitcoin-core-dev
160 2016-07-13T09:55:46  *** fengling_ has quit IRC
161 2016-07-13T10:04:38  *** ArthurNumbanumba has joined #bitcoin-core-dev
162 2016-07-13T10:22:49  *** YOU-JI has joined #bitcoin-core-dev
163 2016-07-13T10:33:40  <jonasschnelli> sipa, gmaxwell: mind doing a review of https://github.com/bitcoin/bitcoin/pull/8323? I'd like and I think its important to have this in 0.13
164 2016-07-13T10:41:12  *** fengling_ has joined #bitcoin-core-dev
165 2016-07-13T10:45:46  *** fengling_ has quit IRC
166 2016-07-13T10:55:01  *** xinxi_ has joined #bitcoin-core-dev
167 2016-07-13T10:57:06  *** xinxi has quit IRC
168 2016-07-13T11:17:58  <jonasschnelli> sipa, gmaxwell: maybe you find time to quickly review the BIP151 switch from HMAC_SHA512 "KDF"  to HKDF: https://github.com/bitcoin/bips/compare/master...jonasschnelli:2016/07/bip151_hkdf?expand=1
169 2016-07-13T11:30:48  *** assder has quit IRC
170 2016-07-13T11:31:39  *** harrymm has quit IRC
171 2016-07-13T11:34:08  *** davidlj95 has quit IRC
172 2016-07-13T11:39:00  *** arubi_ has joined #bitcoin-core-dev
173 2016-07-13T11:42:28  *** fengling_ has joined #bitcoin-core-dev
174 2016-07-13T11:43:04  *** arubi has quit IRC
175 2016-07-13T11:43:17  *** afk11 has quit IRC
176 2016-07-13T11:46:40  *** harrymm has joined #bitcoin-core-dev
177 2016-07-13T11:47:26  *** fengling_ has quit IRC
178 2016-07-13T11:57:10  *** cryptapus_ has joined #bitcoin-core-dev
179 2016-07-13T11:57:10  *** cryptapus_ has joined #bitcoin-core-dev
180 2016-07-13T11:57:11  *** cryptapus_ is now known as cryptapus_afk
181 2016-07-13T11:57:28  *** cryptapus has quit IRC
182 2016-07-13T12:06:37  *** arubi__ has joined #bitcoin-core-dev
183 2016-07-13T12:10:30  *** arubi_ has quit IRC
184 2016-07-13T12:20:10  *** Chris_Stewart_5 has joined #bitcoin-core-dev
185 2016-07-13T12:26:51  *** Lysander1 has quit IRC
186 2016-07-13T12:40:41  *** YOU-JI has quit IRC
187 2016-07-13T12:43:59  *** fengling_ has joined #bitcoin-core-dev
188 2016-07-13T12:48:46  *** fengling_ has quit IRC
189 2016-07-13T12:48:54  *** Chris_Stewart_5 has quit IRC
190 2016-07-13T13:07:35  *** PaulCapestany has quit IRC
191 2016-07-13T13:09:02  *** PaulCapestany has joined #bitcoin-core-dev
192 2016-07-13T13:09:12  *** Chris_Stewart_5 has joined #bitcoin-core-dev
193 2016-07-13T13:13:00  *** laurentmt has quit IRC
194 2016-07-13T13:15:34  *** go1111111 has quit IRC
195 2016-07-13T13:21:21  <GitHub12> [bitcoin] jtimon closed pull request #8328: Consensus: Rename: Move consensus code to the consensus directory (master...0.12.99-consensus-rename) https://github.com/bitcoin/bitcoin/pull/8328
196 2016-07-13T13:21:56  *** arubi__ is now known as arubi
197 2016-07-13T13:26:10  <GitHub147> [bitcoin] jtimon reopened pull request #8328: Consensus: Rename: Move consensus code to the consensus directory (master...0.12.99-consensus-rename) https://github.com/bitcoin/bitcoin/pull/8328
198 2016-07-13T13:27:07  *** Cheeseo has joined #bitcoin-core-dev
199 2016-07-13T13:28:38  *** go1111111 has joined #bitcoin-core-dev
200 2016-07-13T13:28:49  <jtimon> kanzure: NicolasDorier: https://github.com/bitcoin/bitcoin/pull/8337
201 2016-07-13T13:32:46  <jtimon> and I will continue in https://github.com/jtimon/bitcoin/tree/0.12.99-consensus but I won't open more PRs for now
202 2016-07-13T13:39:06  *** laurentmt has joined #bitcoin-core-dev
203 2016-07-13T13:41:04  *** laurentmt has quit IRC
204 2016-07-13T13:42:04  *** Chris_Stewart_5 has quit IRC
205 2016-07-13T13:44:23  *** Chris_Stewart_5 has joined #bitcoin-core-dev
206 2016-07-13T13:46:47  *** fengling_ has joined #bitcoin-core-dev
207 2016-07-13T13:49:21  *** Chris_Stewart_5 has quit IRC
208 2016-07-13T13:50:26  *** fengling_ has quit IRC
209 2016-07-13T13:54:12  *** TomMc has joined #bitcoin-core-dev
210 2016-07-13T14:04:35  *** Chris_Stewart_5 has joined #bitcoin-core-dev
211 2016-07-13T14:20:57  *** arubi_ has joined #bitcoin-core-dev
212 2016-07-13T14:23:52  *** kinlo has quit IRC
213 2016-07-13T14:24:43  *** kinlo has joined #bitcoin-core-dev
214 2016-07-13T14:25:04  *** arubi has quit IRC
215 2016-07-13T14:27:06  *** bsm117532 is now known as bsm2319171311753
216 2016-07-13T14:27:27  *** bsm2319171311753 is now known as bsm117532
217 2016-07-13T14:30:27  *** bsm117532 is now known as bsm2357
218 2016-07-13T14:47:01  *** fengling_ has joined #bitcoin-core-dev
219 2016-07-13T14:50:46  *** laurentmt has joined #bitcoin-core-dev
220 2016-07-13T14:51:02  *** laurentmt has quit IRC
221 2016-07-13T14:51:46  *** fengling_ has quit IRC
222 2016-07-13T15:09:07  *** arubi__ has joined #bitcoin-core-dev
223 2016-07-13T15:13:51  *** arubi_ has quit IRC
224 2016-07-13T15:21:28  *** molz has joined #bitcoin-core-dev
225 2016-07-13T15:23:06  *** moli has quit IRC
226 2016-07-13T15:28:50  *** arubi__ is now known as arubi
227 2016-07-13T15:38:51  *** Chris_Stewart_5 has quit IRC
228 2016-07-13T15:48:32  *** fengling_ has joined #bitcoin-core-dev
229 2016-07-13T15:53:26  *** fengling_ has quit IRC
230 2016-07-13T16:24:17  *** LeMiner2 has joined #bitcoin-core-dev
231 2016-07-13T16:25:52  *** LeMiner has quit IRC
232 2016-07-13T16:25:52  *** LeMiner2 is now known as LeMiner
233 2016-07-13T16:38:09  <jtimon> NicolasDorier: it just occurred to me that you can always implement my function pointers interface by pre-filling memory with whatever data you were counting on passing as a parameter. Therefore it will be trivial to offer the all-data-upfront interface you want while using my more generic API inside. The costs will be to reading the pointer to a function, the function and executed it, but the function should be merely a map from
234 2016-07-13T16:38:09  <jtimon>  your parameters to a pre-filled position in memory. Does that sound accetable to you? no C# inside
235 2016-07-13T16:38:58  <jtimon> we can offer both interfaces with trivial development costs I think, let's see
236 2016-07-13T16:39:51  *** owowo has quit IRC
237 2016-07-13T16:42:01  *** spudowiar has joined #bitcoin-core-dev
238 2016-07-13T16:55:05  *** owowo has joined #bitcoin-core-dev
239 2016-07-13T17:08:41  *** laurentmt has joined #bitcoin-core-dev
240 2016-07-13T17:08:58  *** laurentmt has quit IRC
241 2016-07-13T17:12:33  *** spudowiar has quit IRC
242 2016-07-13T17:32:36  <jtimon> just force pushed 0.12.99-consensus
243 2016-07-13T17:33:09  <jtimon> but I will force push again soon
244 2016-07-13T17:43:08  *** Chris_Stewart_5 has joined #bitcoin-core-dev
245 2016-07-13T17:44:11  *** spudowiar has joined #bitcoin-core-dev
246 2016-07-13T17:51:44  *** fifth_ has joined #bitcoin-core-dev
247 2016-07-13T18:07:26  *** fifth_ has quit IRC
248 2016-07-13T18:08:10  *** kadoban has joined #bitcoin-core-dev
249 2016-07-13T18:15:31  *** jgarzik has left #bitcoin-core-dev
250 2016-07-13T18:15:31  *** jgarzik has quit IRC
251 2016-07-13T18:30:13  <bsm2357> I expect a common error for people using python-bitcoinlib will be to call SignatureHash with the scriptPubKey of the corresponding segwit input, rather than an actual script.  This is easy to detect, and insert the corresponding script for calculating the sighash, but causes the SignatureHash function to depart from the behavior of the one in core.
252 2016-07-13T18:30:34  <bsm2357> What are your opinions?  Is detecting this and inserting the correct script desirable or not?
253 2016-07-13T18:33:49  <bsm2357> Hmmm.  Taking this question to #bitcoin-dev.
254 2016-07-13T18:34:39  <sipa> bsm2357: perhaps also good to discuss on the repository itself in an issue
255 2016-07-13T18:35:45  *** Chris_Stewart_5 has quit IRC
256 2016-07-13T18:35:58  <bsm2357> Of course.  I'll push my latest there soon with a working SignatureHash.  Just looking for some quick feedback from people who might use it, before I go write code.
257 2016-07-13T18:41:49  *** frankenmint has joined #bitcoin-core-dev
258 2016-07-13T18:49:00  *** Chris_Stewart_5 has joined #bitcoin-core-dev
259 2016-07-13T18:53:55  *** fengling_ has joined #bitcoin-core-dev
260 2016-07-13T18:58:06  *** fengling_ has quit IRC
261 2016-07-13T19:03:36  <Chris_Stewart_5> Is it unusual for nodes to give back one ip address back for a getaddr message?
262 2016-07-13T19:05:13  <sipa> we only respond to one getaddr per connection
263 2016-07-13T19:05:14  <sipa> after that, only normal addr relay
264 2016-07-13T19:06:58  <Chris_Stewart_5> Ok, but it it unusual to respond with only one ip address? The behavior I am seeing is connecting to a node, I send a getaddr message, then I only get ONE ip address back in the addr message
265 2016-07-13T19:07:20  <Chris_Stewart_5> and that ip address is the node's ip address. I guess I was under the impression that it would send back the nodes that it is connected to
266 2016-07-13T19:07:31  <sipa> it's not responding at all
267 2016-07-13T19:07:42  <sipa> the addr you see is likely just an independently relayed addr
268 2016-07-13T19:08:04  <sipa> hell no, nodes try to hide who they are connected to
269 2016-07-13T19:08:05  <xinxi_> jtimon: are you still there?
270 2016-07-13T19:09:41  <Chris_Stewart_5> so how do you connect with more than one node on the network then? It seems that you could only connect to dns seeds then.
271 2016-07-13T19:10:02  <sipa> i don't understand the question
272 2016-07-13T19:10:16  <sipa> normally, you get 1000 IP addresses back in response to getaddr
273 2016-07-13T19:10:26  <sipa> unless the peer does not know about many, of course
274 2016-07-13T19:10:42  <Chris_Stewart_5> I should preface this with it is testnet, perhaps the behavior is different?
275 2016-07-13T19:10:58  <sipa> it's very likely that on testnet the peer just does not know about many peers
276 2016-07-13T19:11:21  <jtimon> xinxi_: yes
277 2016-07-13T19:11:24  <sipa> also, if you send getaddr multiple times, the second and later times it is just ignored
278 2016-07-13T19:11:47  <xinxi_> jtimon: great. I want to discuss with you about the libconsensus that you are working on.
279 2016-07-13T19:11:58  <xinxi_> it's a great project.
280 2016-07-13T19:12:14  <jtimon> xinxi_: absolutely, always interested in discussing that
281 2016-07-13T19:12:46  <Chris_Stewart_5> sipa: I found that out the hard way :-). But it seems unlikely that a response to a getaddr message would have just ONE ip address even on testnet, right?
282 2016-07-13T19:13:22  <xinxi_> my I know when this will be merged into the master branch?
283 2016-07-13T19:14:01  <jtimon> xinxi_: right now the longest branch I have is https://github.com/jtimon/bitcoin/tree/0.12.99-consensus but I still have things to cherry pick from https://github.com/jtimon/bitcoin/commits/jt (plus some things that will just have to be new)
284 2016-07-13T19:15:41  <jtimon> xinxi_: uff, no idea myself, I wish I knew, I believe the idea is focusing on refactors like this and the ones going on in net.o and wallet right after forking 0.13's branch, which should be soon
285 2016-07-13T19:16:07  <jtimon> whether the PRs will be merged or not, I cannot know, depends on review
286 2016-07-13T19:16:11  *** sipa has quit IRC
287 2016-07-13T19:16:22  <xinxi_> does libconsensus include the wallet and the network parts?
288 2016-07-13T19:16:28  <jtimon> no
289 2016-07-13T19:16:48  <jtimon> but there's other people doing refactors both in the net and wallet parts
290 2016-07-13T19:17:18  <jtimon> ie cfields and jonasschnelli respectively
291 2016-07-13T19:17:19  <morcos> sipa: i think on machines with a lot of cores, the sig cache actually slows down block validation.   my guess is its the contention on the lock since we erase every time
292 2016-07-13T19:17:46  <morcos> sipa: i think it might be better to batch erase, or randomly erase or something else
293 2016-07-13T19:18:06  <cfields> morcos: i've scratched my head at that one several times. seems we'd be better off nuking a bucket at a time or so
294 2016-07-13T19:18:38  <xinxi_> jtimon: cool. so all the code in libconsensus is in this directory https://github.com/jtimon/bitcoin/tree/0.12.99-consensus/src/consensus, right?
295 2016-07-13T19:18:43  *** spudowiar has quit IRC
296 2016-07-13T19:19:03  <morcos> cfields: it defaults to nuking a bucket at a time if it gets full anyway (although i'm not sure thats super efficient either, but that tends to happen in ATMP, so who cares)
297 2016-07-13T19:19:14  <jtimon> morcos: I once tried with a huge sigcache just to see if the swap partition was working. didn't benchmarked but it certainly didn't seem to accelerate things
298 2016-07-13T19:19:31  <cfields> morcos: eh? i thought it just killed a single entry each time
299 2016-07-13T19:20:50  <jtimon> xinxi_: no, there's still some in main.cpp, but the idea it's that it all should end up there (without counting the code in src/crypto and libsecp256k1)
300 2016-07-13T19:20:51  <cfields> morcos: looks to me like it just picks a random bucket and removes the first entry in it. am i misreading?
301 2016-07-13T19:20:57  <morcos> cfields: the problem i'm referring to is that on validating a block, you erase each stored signature as you read it
302 2016-07-13T19:20:58  *** sipa has joined #bitcoin-core-dev
303 2016-07-13T19:21:29  <morcos> but yes, if it's gotten to big anyway, then it'll pick a random bucket and remove entires until its below the limit, but isn't that what you said?
304 2016-07-13T19:21:38  <jtimon> I will keep updating jtimon/0.12.99-consensus moving more things there though
305 2016-07-13T19:21:42  <xinxi_> jtimon: I guess most of the work has already been done?
306 2016-07-13T19:21:51  <Chris_Stewart_5> sipa: also you said it can relay 1000 addresses at a time, doesn't that effectively give a history of the nodes they are connected to?
307 2016-07-13T19:22:01  <morcos> the second thing happens when trying to set a new entry, which is in ATMP, so not important to shave micros... as it is in block validation
308 2016-07-13T19:22:05  <sipa> Chris_Stewart_5: nodes typically know way more 1000 addresses
309 2016-07-13T19:22:20  <sipa> Chris_Stewart_5: and typically know of many addresses they are not currently, or even never have been, connected to
310 2016-07-13T19:22:39  <Chris_Stewart_5> sipa: So basically caching addresses from addr messages?
311 2016-07-13T19:22:56  <sipa> Chris_Stewart_5: yes, and from dns seed responses
312 2016-07-13T19:22:56  <Chris_Stewart_5> unsolicited ones
313 2016-07-13T19:23:01  <sipa> Chris_Stewart_5: that's addrman
314 2016-07-13T19:23:17  <cfields> morcos: ah yes, i was talking about the full case
315 2016-07-13T19:23:54  <xinxi_> jtimon: we plan to prove the correctness of the consensus part.
316 2016-07-13T19:24:20  <sipa> xinxi_: again, correctness compared to which specification?
317 2016-07-13T19:24:37  <morcos> cfields: yeah the problem is lock contention on validation since each of your script validating cores is contending on the sig cache lock since they each have to erase from it.  if you eliminate or batch those erases, it speeds up validation significantly
318 2016-07-13T19:24:47  <jtimon> xinxi_: well, the part of making a complete verifyBlock, I would say it has been done several times. exposing it as a C API is another thing, specially since probably the APIs will be discussed extensively (plus I never decoupled Consensus::Params from uint256)
319 2016-07-13T19:24:48  <sipa> morcos: interesting
320 2016-07-13T19:24:49  <xinxi_> sipa: as discussed, we are going to derive a specification from the code.
321 2016-07-13T19:25:04  <jtimon> xinxi_: interesting
322 2016-07-13T19:25:13  <sipa> xinxi_: a specification derived from the code will by definition match the code, no? :)
323 2016-07-13T19:25:32  <Chris_Stewart_5> sipa: Is there documentation for addrman any where? Doesn't seem to be in the p2p networking part: https://bitcoin.org/en/developer-reference#p2p-network
324 2016-07-13T19:25:41  <sipa> Chris_Stewart_5: in addrman.h :)
325 2016-07-13T19:25:45  <jtimon> sipa: I guess the goal is that from then on you can't change the code without appropriately changing the specifications or the tests will fail
326 2016-07-13T19:25:58  <sipa> jtimon, xinxi_: yes, that would be awesome
327 2016-07-13T19:26:00  <cfields> morcos: hmm. ignoring larger fixes, if the contention is that high, i wonder if the shared lock is doing more harm than good
328 2016-07-13T19:26:00  <xinxi_> sipa: i would say it's going to match the code as much as possible unless we find bugs/inconsistencies in the specification.
329 2016-07-13T19:26:12  <sipa> xinxi_: then the specification is wrong
330 2016-07-13T19:26:17  <jtimon> hehe
331 2016-07-13T19:26:41  <sipa> (but it would still be very interesting to know)
332 2016-07-13T19:26:46  <jtimon> the specification is clearly my branch
333 2016-07-13T19:26:56  <xinxi_> sipa: yeah, as discussed, we don't want to have a specification with flaws or bugs. if we find that, we should upgrade.
334 2016-07-13T19:27:11  <sipa> xinxi_: i think you're dreaming if you think that's going to happen
335 2016-07-13T19:27:18  <morcos> sipa: on my 16 core machine, i think it speeds up the "Connect total" part by about 40% to just not erase..   But I didnt' run that over a long enough period to model the lower hit rate that'll eventually result from not having smart erasing.  If we can conveniently batch the erases from the block that'll be best
336 2016-07-13T19:27:27  <xinxi_> sipa: why is that?
337 2016-07-13T19:27:49  <xinxi_> a hard fork is not possible?
338 2016-07-13T19:27:56  <sipa> xinxi_: because i expect a hard fork to switch to something with only theoretical advantages to be very controversial
339 2016-07-13T19:28:41  <xinxi_> sipa: well, we will definitely test that first.
340 2016-07-13T19:28:53  <cfields> morcos: whoa
341 2016-07-13T19:28:56  <jtimon> xinxi_: yes, but a hardfork doesn't magically get deployed in all computers when you update the specification, whatever is deployed is the specification in practice
342 2016-07-13T19:29:16  <sipa> xinxi_: not trying to discourage you... i would be very exciting if you can contribute to the correctness of the system using formal proofs, at whatever level
343 2016-07-13T19:29:27  <sipa> xinxi_: but please... keep it about analysing the actual code
344 2016-07-13T19:29:35  <sipa> *excited
345 2016-07-13T19:29:47  <xinxi_> sipa: the actual code is in C++, which is almost impossible to prove.
346 2016-07-13T19:30:01  <xinxi_> if you rewrite it in C, we may be able to help.
347 2016-07-13T19:30:12  <sipa> even that has been controversial in the past
348 2016-07-13T19:30:20  <jtimon> xinxi_: in the end the new tests are going to fail when you change either the specification or the implementation, right?
349 2016-07-13T19:30:52  <xinxi_> jtimon: what do you mean by failure?
350 2016-07-13T19:31:09  <sipa> xinxi_: you could of course do a rewrite in C, and analyse that... we could learn very interesting things from that
351 2016-07-13T19:31:20  <sipa> xinxi_: but that still doesn't mean we need to actually switch to that C code
352 2016-07-13T19:31:30  <jtimon> if you changed the specification, sipa was right, the bug was there, if you change the implementation but you were't expecting change in funtionality, then your changes are wrong
353 2016-07-13T19:31:55  <xinxi_> jtimon: how did you guys fix bugs in the past I am wondering.
354 2016-07-13T19:32:01  <sipa> xinxi_: soft forks
355 2016-07-13T19:32:05  <xinxi_> so you just leave the bugs there?
356 2016-07-13T19:32:06  <jtimon> let me search for a ccc talk about what I'm thinking you wanted to do
357 2016-07-13T19:32:22  <sipa> xinxi_: if they're not dangerous and merely annoying.. yes, we have no other choice
358 2016-07-13T19:32:23  <jtimon> xinxi_: no, sipa answered: softforks
359 2016-07-13T19:32:29  <sipa> xinxi_: we don't decide the rules of the network
360 2016-07-13T19:32:59  <xinxi_> OK. so can we just deploy the slightly different bug free version using a soft fork?
361 2016-07-13T19:33:05  <sipa> maybe
362 2016-07-13T19:33:16  <sipa> that depends on the kind of changes you want to make
363 2016-07-13T19:33:18  <jtimon> for example, the code in bip99 fixes a known bug which is not a priority
364 2016-07-13T19:33:27  <jtimon> but that fix is a hardfork
365 2016-07-13T19:33:33  *** d_t has joined #bitcoin-core-dev
366 2016-07-13T19:34:09  <xinxi_> so can you explain to me what kind of changes require a soft fork and what kind of changes require a hard fork?
367 2016-07-13T19:34:23  <xinxi_> and what's the difference between the hard and soft forks?
368 2016-07-13T19:34:24  <sipa> xinxi_: everything that leaves things that were previously invalid invalid is a softfork
369 2016-07-13T19:35:05  <xinxi_> how do you define invalid?
370 2016-07-13T19:35:20  <jtimon> meh, can't find the ccc video about proving correctness, I though you were coming from there
371 2016-07-13T19:35:27  <sipa> xinxi_: abstractly speaking, the consensus rules are black box which you feed a sequence of blocks
372 2016-07-13T19:35:47  <sipa> xinxi_: and it returns whether that sequence of blocks is valid or not
373 2016-07-13T19:35:51  <jtimon> their "specification" was basically another program for an abstract machine
374 2016-07-13T19:36:08  <sipa> it cannot depend on the order in which those blocks were received, or whatever earlier states the node went through
375 2016-07-13T19:36:46  <sipa> xinxi_: a softfork leaves all previously invalid inputs invalid
376 2016-07-13T19:37:44  <xinxi_> so a soft fork can only invalid a previously valid input.
377 2016-07-13T19:37:48  <sipa> correct
378 2016-07-13T19:37:51  *** frankenmint has quit IRC
379 2016-07-13T19:37:55  <jtimon> or in other words, it only add restrictions, doesn't remove them: everything that was invalid remains invalid
380 2016-07-13T19:37:55  <Chris_Stewart_5> xinxi_: You can think of the set of all consensus rules R with cardinality n, a soft fork increments the cardinality of R one.
381 2016-07-13T19:38:03  <xinxi_> and a hard fork has no such restriction.
382 2016-07-13T19:38:07  <Chris_Stewart_5> xinxi_: Any removal from the set R is a hard fork though
383 2016-07-13T19:38:08  <gmaxwell> xinxi_: the whole purpose of the system it to come to a consistent worldwide agreement on the history of the ledger. This is the first and foremost definition of correctness in the system.
384 2016-07-13T19:38:21  <sipa> xinxi_: and it's very surprising how many things are actually possible with softforks
385 2016-07-13T19:38:46  <xinxi_> sipa: sure, i can imaging that.
386 2016-07-13T19:38:52  <sipa> xinxi_: one analogy is seeing the consensus rules a block of marble from which pieces are cut away that give the complexity
387 2016-07-13T19:39:32  <xinxi_> so how do hard forks and soft forks differ in terms of deployment?
388 2016-07-13T19:39:39  <jtimon> some people theorize that everything that is possible via hardfork is also possible via softfork in some other way, but I'm not convinced that means we should always prefer softforks
389 2016-07-13T19:39:47  <sipa> a hard fork: convince the whole world to switch to different code; period
390 2016-07-13T19:39:55  <sipa> a soft fork: read BIP34 and BIP9
391 2016-07-13T19:40:05  <sipa> (BIP9 is newer)
392 2016-07-13T19:40:25  <xinxi_> has any hard fork happened?
393 2016-07-13T19:40:37  <sipa> very early on in the history of the system, yes
394 2016-07-13T19:40:43  <sipa> in the first 1.5 years
395 2016-07-13T19:40:52  <jtimon> xinxi_: for softforks miners need to coordinate for deployment, then other nodes can upgrade when they want/can. for hardforks everybody needs to upgrade before deployment, which makes them slower to deploy
396 2016-07-13T19:41:35  <sipa> xinxi_: there is also BIP50 which is a post-mortem of a consensus failure that occurred between old and new versions of the system
397 2016-07-13T19:41:42  <xinxi_> jtimon: how can you upgrade without deployment?
398 2016-07-13T19:42:05  <xinxi_> basically, you can automatically activate when a certain height is reached?
399 2016-07-13T19:42:09  <sipa> xinxi_: softforks only apply new rules to blocks after some changeover point
400 2016-07-13T19:42:18  <sipa> they use old rules for everything before
401 2016-07-13T19:42:32  <xinxi_> sipa: yeah, you've explained that well.
402 2016-07-13T19:42:38  <sipa> BIP30 used a fixed timestamp for the changeover point
403 2016-07-13T19:42:51  <jtimon> xinxi_: for softforks bip9 describes it, for hardforks there's different opinions on what would be the best way to signal activation
404 2016-07-13T19:43:22  <xinxi_> jtimon: got it.
405 2016-07-13T19:43:30  *** d_t has quit IRC
406 2016-07-13T19:43:46  *** d_t has joined #bitcoin-core-dev
407 2016-07-13T19:43:56  <sipa> BIP34 used block versions above a certain number to let miners indicate they are enforcing the new rules
408 2016-07-13T19:44:01  <xinxi_> so who makes the final decision on whether a fork should be deployed or not?
409 2016-07-13T19:44:10  <sipa> xinxi_: for a soft fork: miners
410 2016-07-13T19:44:34  <sipa> as it's really just miners deciding to start enforcing some additional rules in addition to what the network requires them to
411 2016-07-13T19:44:43  <sipa> for a hard fork: politics
412 2016-07-13T19:44:52  <sipa> (let's not go there)
413 2016-07-13T19:45:01  <xinxi_> the politics between whom?
414 2016-07-13T19:45:05  <jtimon> xinxi_: here's a few different ways to coordinate activation, the last 3 in there use bip 9 : https://github.com/jtimon/bitcoin/blob/0.12.99-consensus/src/consensus/versionbits.cpp#L172
415 2016-07-13T19:45:11  <xinxi_> I think it's better to clarify things.
416 2016-07-13T19:45:12  <petertodd> xinxi_: a hard fork is arguably the creation of a new currency for starters...
417 2016-07-13T19:45:20  <sturles> xinxi_: All bitcoin users.
418 2016-07-13T19:45:38  <sipa> xinxi_: some people believe developers or some committee should decide on hard forks, and then assume everyone will follow them
419 2016-07-13T19:46:30  <sipa> xinxi_: others think hard forks should be reserved for clearly uncontroversial things
420 2016-07-13T19:46:36  <xinxi_> is there any democratic decision making process?
421 2016-07-13T19:46:56  <sipa> democracy is pretty hard in a system designed to avoid fixed identities
422 2016-07-13T19:47:00  <petertodd> xinxi_: there's no agreement on what a "democratic decision making process" even looks like
423 2016-07-13T19:47:13  <jtimon> bip2 tries to clarify things, but it can only decided on whether something was "accepted" or not observing adoption a posteriori. BIP99 also tries to clarify things, but it's based on a vaguely defined concept of "uncontroversial"
424 2016-07-13T19:47:20  <xinxi_> sipa: sybil attack.
425 2016-07-13T19:47:24  <petertodd> xinxi_: whose votes do you count? miner-triggered soft-forks at least have a natural metric: hashing power
426 2016-07-13T19:47:46  <sdaftuar> 15:43 < sipa> xinxi_: for a soft fork: miners
427 2016-07-13T19:47:49  <xinxi_> then, can we just let miners vote?
428 2016-07-13T19:47:56  <sdaftuar> ^ this is not correct, users have to adopt the rules as well
429 2016-07-13T19:48:07  <sdaftuar> but please, this discussion should be elsewhere
430 2016-07-13T19:48:09  <jtimon> I wouldn't call that a vote, but just confirmation that they have upgraded
431 2016-07-13T19:48:16  <jtimon> for coordination
432 2016-07-13T19:48:20  <sipa> xinxi_: miners should absolutely not be allowed to decide on hardforks (they could for example decide to increase their own reward that way)
433 2016-07-13T19:48:29  <jtimon> the softfork is supposed to be uncontroversial in the first place
434 2016-07-13T19:48:29  <xinxi_> sipa: why not?
435 2016-07-13T19:48:52  <petertodd> xinxi_: "why not?" is a political question...
436 2016-07-13T19:48:55  <sipa> xinxi_: "who watches the watchmen?"
437 2016-07-13T19:49:20  <jeremyrubin> xinxi_: probably discuss this on #bitcoin
438 2016-07-13T19:49:20  <jtimon> sipa: not only they shouldn't not be allowed but they have no power over hardforks in practice, "should" or "shouldn't" is irrelevant here
439 2016-07-13T19:49:24  <sipa> xinxi_: they're called consensus rules because every user in the system choose to accept them
440 2016-07-13T19:49:37  <xinxi_> sipa: sure. that's a good reason.
441 2016-07-13T19:49:44  <sipa> agree, this is getting off topic
442 2016-07-13T19:49:45  <sipa> sorry
443 2016-07-13T19:50:01  <jtimon> yep, sorry, #bitcoin I guess
444 2016-07-13T19:50:02  <xinxi_> well, it's not off topic actually.
445 2016-07-13T19:50:15  <jtimon> well, it's not really development
446 2016-07-13T19:50:15  <sipa> it is off topic for the developer of bitcoin core, which is what this channel is about
447 2016-07-13T19:50:28  <xinxi_> i need to make sure that tremendous amount of effort that could make Bitcoin even greater will be deployed.
448 2016-07-13T19:50:49  <petertodd> xinxi_: I'd suggest you take that kind of discussion to #bitcoin-wizards at least
449 2016-07-13T19:51:17  <sipa> xinxi_: if your tremendous amount of effort requires a hard fork, the answer is that nobody can promise you it will happen
450 2016-07-13T19:51:32  <xinxi_> if there is no certainty but extremely high risk, we will just have to give up.
451 2016-07-13T19:52:05  <sipa> xinxi_: i think there are very interesting things you can do if you aim at something a bit less ambitious than proofs about the entire consensus system
452 2016-07-13T19:52:20  <petertodd> xinxi_: if you want certainty, then yes, I'd suggest you give up. but I'd prefer you took sipa's advice
453 2016-07-13T19:52:54  <xinxi_> petertodd: so you mean i should at least try no matter what the result is?
454 2016-07-13T19:53:38  <sipa> xinxi_: you can focus on for example just a scripting language... rewriting that part in C would not be very hard, and the result could be very interesting
455 2016-07-13T19:54:04  <petertodd> xinxi_: I'm just saying, that anything that for any work you do that may require a hard-fork to deploy, you should accept that you may need to create a new currency, economically distint from bitcoin, to deploy your work in production
456 2016-07-13T19:54:07  <sipa> proofs about its complexity, memory usage, equivalence after certain changes
457 2016-07-13T19:54:35  *** fengling_ has joined #bitcoin-core-dev
458 2016-07-13T19:54:36  <petertodd> xinxi_: whereas if you want high certainty of being able to deploy your work in production on bitcoin, you'll need to set your sights lower to thinks that can be done in a soft-fork
459 2016-07-13T19:55:02  <xinxi_> sipa: complexity, memory usage, etc are certainly important but not the most critical thing.
460 2016-07-13T19:55:10  <sipa> xinxi_: also, we can always propose a softfork that effectively delegates validation under certain conditions to a completely new scripting language
461 2016-07-13T19:55:15  <petertodd> xinxi_: for example basically all my scalability research is stuff that I do knowing full well that it may never be able to be deployed on bitcoin
462 2016-07-13T19:55:18  <Chris_Stewart_5> sipa: Does it make any sense at all that a dns seed on testnet would only relay one addr from a getaddr message? Sorry if answered already but I'm still unclear on this
463 2016-07-13T19:55:22  <sipa> xinxi_: which can be implemented in a different language
464 2016-07-13T19:55:43  <sipa> xinxi_: BIP143 (which we're in the process of finished up) makes that very easy
465 2016-07-13T19:56:01  <sipa> xinxi_: it's effectively designed to allow new scripting language to be plugged in in a softfork compatible way
466 2016-07-13T19:56:25  <xinxi_> petertodd: do you have a fulltime job?
467 2016-07-13T19:56:48  <sipa> xinxi_: there are people working on thinking about improvements that could be made in a new scripting language
468 2016-07-13T19:57:07  <sipa> xinxi_: and for that, all options are open
469 2016-07-13T19:57:12  <xinxi_> sipa: why does a scripting language make a difference?
470 2016-07-13T19:57:25  <petertodd> xinxi_: I'm a consultant, who works pretty much full time on contracts in this space (including Bitcoin Core development contracts)
471 2016-07-13T19:57:32  *** d_t has quit IRC
472 2016-07-13T19:57:44  <sipa> xinxi_: by 'scripting' we mean bitcoin Script here, which is the language used inside transaction outputs to describe the condition under which they can be spent
473 2016-07-13T19:57:51  <sipa> xinxi_: it's not scripting as in bash or python
474 2016-07-13T19:57:57  <xinxi_> petertodd: So Bitcoin Core developers get paid?
475 2016-07-13T19:58:15  <sipa> it's a bytecode language inspired by Forth
476 2016-07-13T19:58:41  <sipa> xinxi_: and arguable, the implementation of that language interpreter is the most complicated piece of the consensus rules
477 2016-07-13T19:58:43  <petertodd> xinxi_: many do - there's no "bitcoin foundation" that pays core developers, but there's lots of ways to get paid
478 2016-07-13T19:58:50  <xinxi_> sipa: yeah, i know what you are talking about. but doesn't that require a hard fork?
479 2016-07-13T19:58:56  <sipa> xinxi_: nope!
480 2016-07-13T19:59:04  <petertodd> xinxi_: (there is a "bitcoin foundation", but they don't pay anyone, and are essentially bankrupt last I checked)
481 2016-07-13T19:59:17  <sipa> xinxi_: the basic idea is that we can redefine some of the remaining NOP opcodes in the language
482 2016-07-13T19:59:26  *** fengling_ has quit IRC
483 2016-07-13T19:59:32  <sipa> xinxi_: it goes further than that, but it would lead me to far to explain it all here
484 2016-07-13T19:59:49  <xinxi_> sipa: i know what you mean.
485 2016-07-13T20:00:03  <sipa> and redefining NOPs can be done as a softfork
486 2016-07-13T20:00:17  <sipa> (as can the new script versions introduced by BIP143)
487 2016-07-13T20:00:56  <xinxi_> sipa: you invalid more blocks syntatically, but you add more semantics.
488 2016-07-13T20:01:27  <sipa> xinxi_: we basically take a particular script whose meaning was "anyone can spend this", and replace it with a new meaning
489 2016-07-13T20:01:51  <sipa> aka moving to a new piece of marable
490 2016-07-13T20:01:53  <sipa> marble
491 2016-07-13T20:02:09  <xinxi_> now I also have a feeling that softforks can solve all problems.
492 2016-07-13T20:02:59  <sipa> to give an extreme example... we can switch to a different proof of work function using softforks
493 2016-07-13T20:03:16  <petertodd> sipa: explain :)
494 2016-07-13T20:03:29  <sipa> (in a hacky way that likely wouldn't actually work, as it would go against the interest of miners who decide on softforks...)
495 2016-07-13T20:04:02  <sipa> petertodd: define a height-dependent function that maps difficulty of the old PoW to difficulty of the new one
496 2016-07-13T20:04:12  <sipa> petertodd: now demand that every block satisfies both PoWs
497 2016-07-13T20:04:31  <sipa> and choose the function so that the difficulty of the old one becomes almost irrelevant over time
498 2016-07-13T20:04:38  <petertodd> sipa: eh... that's not "switching" to a different PoW, that's additing an additional PoW
499 2016-07-13T20:04:46  <Chris_Stewart_5> interesting
500 2016-07-13T20:04:57  <sipa> of course... in a softfork we can by definition only 'add' rules
501 2016-07-13T20:05:08  <sipa> but in practice it would mean switching
502 2016-07-13T20:05:36  <sipa> (this is a contrived example, and not something i'd ever actually propose)
503 2016-07-13T20:05:36  <petertodd> sipa: also, I'm not sure I'd want to call that a soft-fork, given that the cost for an attacker to attack old clients goes down to zero - I'd limit the use of the term "soft-fork" to things that retain lite-client level security for non-adopting clients indefinitely
504 2016-07-13T20:05:48  *** d_t has joined #bitcoin-core-dev
505 2016-07-13T20:06:00  <petertodd> sipa: example: the switch to most-work-chain from bitcoin 0.1's longest-chain rule is something I'd definitely call a hard-fork
506 2016-07-13T20:06:02  <sipa> petertodd: fair point
507 2016-07-13T20:06:35  <sipa> interesting... i'm not sure i would see the chain selection rule as a part of consensus
508 2016-07-13T20:06:42  *** moli has joined #bitcoin-core-dev
509 2016-07-13T20:06:44  <petertodd> I certainly would!
510 2016-07-13T20:06:54  <sipa> it's necessary for convergence... but many things are necessary for convergence, like a working p2p system
511 2016-07-13T20:07:38  <xinxi_> about soft fork, can you explain to me how it gets deployed and why we can only be more restrictive?
512 2016-07-13T20:07:56  <sipa> xinxi_: read BIP9 about how to deploy it
513 2016-07-13T20:08:10  <petertodd> I could make the chain selection rule be "only headers whose merkle-root end with the byte string 'peter todd'", and we'd quickly e in a situation where far less than 50% of hashing power is sufficient to attack non-adopting clients
514 2016-07-13T20:08:19  <sipa> xinxi_: why we can only be more restrictive... so that all rules demanded by old nodes remain satisfied by the majority of the hash rate
515 2016-07-13T20:08:37  <petertodd> sipa: _exactly_: your example fails the "majority of the hash rate" condition
516 2016-07-13T20:08:52  <sipa> petertodd: interesting
517 2016-07-13T20:08:55  <sipa> i agree
518 2016-07-13T20:09:04  *** molz has quit IRC
519 2016-07-13T20:09:22  <sipa> it is a validation-rule-more-restrictive change then, but not a soft forking change
520 2016-07-13T20:09:42  <petertodd> sipa: having said that, I think we need a new term for the case where a hard-fork is introduced gradually; maybe staged hard-fork?
521 2016-07-13T20:10:05  <sipa> we should name the different types of consensus rule changes after pokemon
522 2016-07-13T20:10:26  * sipa hides
523 2016-07-13T20:10:40  <xinxi_> Ha, I see the key is backward compatibility.
524 2016-07-13T20:10:47  * petertodd glares at sipa
525 2016-07-13T20:10:53  <petertodd> xinxi_: yup
526 2016-07-13T20:10:55  <xinxi_> nodes accepted by new nodes should be accepted by old nodes too.
527 2016-07-13T20:10:59  <sipa> xinxi_: otherwise, any old node will end up not accepting the majority chain
528 2016-07-13T20:11:06  <xinxi_> that's the easiest explanation to me.
529 2016-07-13T20:11:08  <petertodd> xinxi_: you mean, blocks accepted by...
530 2016-07-13T20:11:19  <xinxi_> petertodd: yep
531 2016-07-13T20:11:25  <xinxi_> that's a typo.
532 2016-07-13T20:11:26  <sipa> xinxi_: and the ledger will split in half, where each pre-existing coin becomes spendable on both sides of the fork
533 2016-07-13T20:11:56  <xinxi_> that's interesting.
534 2016-07-13T20:11:57  <petertodd> xinxi_: spendable on both sides of the fork, because the hard-fork _is_ the creation of a new currency
535 2016-07-13T20:12:19  <xinxi_> do you know COM+ invented by Microsoft? It's also backward compatible.
536 2016-07-13T20:12:43  <xinxi_> But they can keep releasing new versions of DirectX, which is based on COM+.
537 2016-07-13T20:12:44  <sipa> yup, just a currency that implicitly assigns the new coins exactly according to how they were distributed at the forking point in the old currency
538 2016-07-13T20:12:49  <petertodd> xinxi_: heck, I've been asked two or three times now by exchanges and the like to vet their plans for separating UTXO's cleanly onto both sides post-hard-fork, so they can trade/withdraw them separately (likely to be a legal requirement)
539 2016-07-13T20:13:28  <xinxi_> petertodd: that's funny.
540 2016-07-13T20:13:30  *** Chris_Stewart_5 has quit IRC
541 2016-07-13T20:18:56  <xinxi_> so how to nodes check whether a block is valid or not?
542 2016-07-13T20:19:17  <xinxi_> do they just check the syntactics?
543 2016-07-13T20:19:44  *** afk11 has joined #bitcoin-core-dev
544 2016-07-13T20:19:45  *** afk11 has quit IRC
545 2016-07-13T20:19:45  *** afk11 has joined #bitcoin-core-dev
546 2016-07-13T20:19:56  <petertodd> xinxi_: what do you mean by "the syntactics"?
547 2016-07-13T20:20:09  <xinxi_> i mean the format of the block data.
548 2016-07-13T20:20:23  <petertodd> xinxi_: you mean, without the context of previous blocks?
549 2016-07-13T20:20:55  <xinxi_> petertodd: that's unlikely to be true. they need to verify transactions, which are dependent on history.
550 2016-07-13T20:21:17  <petertodd> xinxi_: indeed
551 2016-07-13T20:21:32  <xinxi_> petertodd: yeah, so how do they check?
552 2016-07-13T20:21:36  <sipa> xinxi_: 1) we receive headers and verify them syntactically, and check whether their PoW matches, then check whether they connect to previous headers, and their expected difficulty is correct, then store them
553 2016-07-13T20:22:08  <xinxi_> how about the block size? the op codes used?
554 2016-07-13T20:22:10  <petertodd> xinxi_: I'm not clear what you mean by "how" - are you thinking that full nodes check blocks differently than miners?
555 2016-07-13T20:22:30  <sipa> xinxi_: 2) we download blocks along the best headers from our peers, and when a block arrives, check it syntactically and that its transactions' hash matches what is in the claimed header we already know; and if so, then store them
556 2016-07-13T20:23:06  <xinxi_> petertodd: should miners and full nodes use the same set of rules to check blocks?
557 2016-07-13T20:23:10  <sipa> yes
558 2016-07-13T20:23:21  <petertodd> xinxi_: if they didn't, how would we ever get miners to enforce rules that we wanted enforced?
559 2016-07-13T20:23:44  <xinxi_> yeah, that's what I expected.
560 2016-07-13T20:23:44  <xinxi_> what if a node see a newer version of blocks?
561 2016-07-13T20:24:07  <sipa> xinxi_: 3) once we have all blocks along a chain whose total work exceeds that of our previous best chain, we look up its input in a database of unspent transaction outputs, validate the scripts, check resource limits, and various other things, ... and if succesful, remove the outputs it spent from the database, and add the outputs it created
562 2016-07-13T20:24:12  <sipa> xinxi_: what does that mean?
563 2016-07-13T20:24:33  <petertodd> xinxi_: then a soft-fork may have happened, adding rules to the existing set of rules, so the node should warn the user that the consensus rules enforced by it are no longer complete
564 2016-07-13T20:25:00  <petertodd> xinxi_: of course, changing the version number for soft-forks is just a nicity - we can't force miners to do that
565 2016-07-13T20:25:15  <petertodd> xinxi_: (modulo radical redesigns of how bitcoin works, like my client-side validation concepts)
566 2016-07-13T20:25:18  <xinxi_> i mean, if a Bitcoin client of a very low version sees a block generated by the newest version of Bitcoin client, what will happen?
567 2016-07-13T20:25:31  <sipa> xinxi_: it will just accept it
568 2016-07-13T20:25:44  <sipa> xinxi_: all software down to 0.2.10 should still work
569 2016-07-13T20:25:53  <sipa> (and that was because of a change in the p2p protocol)
570 2016-07-13T20:26:06  <xinxi_> so 0.2.10 is a hard fork.
571 2016-07-13T20:26:09  <petertodd> xinxi_: all that can happen is the node can warn the user that a soft-fork may have happened
572 2016-07-13T20:26:14  <sipa> xinxi_: nope, it was a p2p change
573 2016-07-13T20:26:29  <petertodd> xinxi_: not quite, that was a p2p change that can be easily worked around w/o changing the core consensus rules
574 2016-07-13T20:26:30  <sipa> xinxi_: you could create a bridge between 0.2.9 and the current network
575 2016-07-13T20:26:36  <xinxi_> based on your definition, soft forks should be backward compatible.
576 2016-07-13T20:26:48  <sipa> xinxi_: validation of the blocks should be backward compatible
577 2016-07-13T20:26:50  <sipa> and it is
578 2016-07-13T20:27:09  <sipa> the 0.2.9 software just doesn't know how to talk to newer clients anymore
579 2016-07-13T20:27:54  <xinxi_> OK. although the consensus protocol did not change, it's not backward compatible.
580 2016-07-13T20:28:12  <xinxi_> how did you guys manage to upgrade from 0.2.9 to 0.2.10?
581 2016-07-13T20:28:36  <petertodd> sipa: note that the change from longest-chain to most-work was in 0.3.3 https://github.com/bitcoin/bitcoin/commit/40cd0369419323f8d7385950e20342e998c994e1#diff-623e3fd6da1a45222eeec71496747b31R420
582 2016-07-13T20:28:50  <sipa> xinxi_: long before my time
583 2016-07-13T20:28:59  *** chris200_ has quit IRC
584 2016-07-13T20:29:04  <morcos> is there really no better channel for this discussion?
585 2016-07-13T20:29:07  <petertodd> xinxi_: it'd be very easy for people to continue to run nodes that had backwards compatbility with 0.2.9
586 2016-07-13T20:29:20  <petertodd> morcos makes a good point...
587 2016-07-13T20:29:23  <sipa> ack, let's move to #bitcoin
588 2016-07-13T20:29:49  <xinxi_> so this is not even dev related?
589 2016-07-13T20:29:56  <petertodd> xinxi_: it's covering really basic material
590 2016-07-13T20:30:17  <xinxi_> but that's still dev. anyway, let's move.
591 2016-07-13T20:30:20  *** shatoshi has joined #bitcoin-core-dev
592 2016-07-13T20:30:22  <shatoshi> INCREDIBLE! Send me some bitcoin and I can turn it into MUCH more, using special blockchain accelerating technology. Your bitcoin wallet will explode! Guaranteed to work & vouched by the OPS. PM me to begin!
593 2016-07-13T20:30:29  *** ChanServ sets mode: +o sipa
594 2016-07-13T20:30:33  *** sipa sets mode: +b *!*legitimo@*.dhcp.kgpt.tn.charter.com
595 2016-07-13T20:30:33  *** shatoshi was kicked by sipa (shatoshi)
596 2016-07-13T20:30:37  <morcos> ha ha, you've been replaced by somethign even worse
597 2016-07-13T20:30:39  <sdaftuar> thank you x 2
598 2016-07-13T20:30:56  *** jtimon has quit IRC
599 2016-07-13T20:37:58  *** Chris_Stewart_5 has joined #bitcoin-core-dev
600 2016-07-13T20:38:55  *** frankenmint has joined #bitcoin-core-dev
601 2016-07-13T20:44:56  *** frankenmint has quit IRC
602 2016-07-13T20:48:27  *** spudowiar has joined #bitcoin-core-dev
603 2016-07-13T20:51:18  *** harrymm has quit IRC
604 2016-07-13T20:54:32  *** d_t has quit IRC
605 2016-07-13T20:56:04  *** fengling_ has joined #bitcoin-core-dev
606 2016-07-13T20:59:00  *** Chris_Stewart_5 has quit IRC
607 2016-07-13T21:00:46  *** fengling_ has quit IRC
608 2016-07-13T21:00:51  *** Sosumi has quit IRC
609 2016-07-13T21:14:10  *** Chris_Stewart_5 has joined #bitcoin-core-dev
610 2016-07-13T21:16:12  *** MiraclePerson has joined #bitcoin-core-dev
611 2016-07-13T21:16:14  <MiraclePerson> SUPER!!!! Want more bitcoin? Send me some Bitcoin and I'll instantly send you MORE back. I use special block-chain exploding skills. Totally safe & secure. Vouched by all the OPS! Pm me to begin!
612 2016-07-13T21:16:43  *** ChanServ sets mode: +o sipa
613 2016-07-13T21:16:46  *** sipa sets mode: +b *!*parson@*.range86-171.btcentralplus.com
614 2016-07-13T21:16:46  *** MiraclePerson was kicked by sipa (MiraclePerson)
615 2016-07-13T21:37:34  *** droark has quit IRC
616 2016-07-13T21:39:37  *** xinxi_ has quit IRC
617 2016-07-13T21:40:10  *** xinxi has joined #bitcoin-core-dev
618 2016-07-13T21:41:13  *** xinxi has quit IRC
619 2016-07-13T21:46:30  *** veleiro has joined #bitcoin-core-dev
620 2016-07-13T21:53:45  *** BlueMatt has joined #bitcoin-core-dev
621 2016-07-13T21:57:08  *** fengling_ has joined #bitcoin-core-dev
622 2016-07-13T22:01:39  <morcos> sipa: how do you feel about boost::lockfree::queue?
623 2016-07-13T22:02:00  <morcos> as a proof of concept that solved the lock contention problem really well
624 2016-07-13T22:02:06  *** fengling_ has quit IRC
625 2016-07-13T22:02:42  <morcos> jeremyrubin pointed me in this direction, but we weren't sure if this is somethign we wanted in the code
626 2016-07-13T22:02:48  <morcos> https://github.com/morcos/bitcoin/commit/3e8a10aa78a8a84a44eca9350178ecf7c308c21c
627 2016-07-13T22:07:13  <sipa> morcos: interesting
628 2016-07-13T22:07:58  *** belcher has joined #bitcoin-core-dev
629 2016-07-13T22:07:58  <sipa> morcos: alternatively, there could be thread-local lists of entries to erase, that get applied once there are enough
630 2016-07-13T22:08:19  <sipa> morcos: though i believe performance of thread locals may be platform dependent
631 2016-07-13T22:08:48  <jeremyrubin> sipa: lockfree is also platform dependent unfortunately
632 2016-07-13T22:08:56  <morcos> sipa: yeah i thought about that, but found the boost thing the easiest to code
633 2016-07-13T22:09:06  <sipa> morcos: did you benchmark?
634 2016-07-13T22:09:34  *** belcher is now known as beIcher
635 2016-07-13T22:09:51  <morcos> yes, for validation its equally as fast as not erasing at all
636 2016-07-13T22:09:56  <sipa> cool!
637 2016-07-13T22:09:59  <morcos> so maybe on the order of a 40% speedup
638 2016-07-13T22:10:04  <morcos> there is the added erase time
639 2016-07-13T22:10:16  <sipa> that's much more than i would have expected
640 2016-07-13T22:10:18  <morcos> but the way i coded it it happens the next time ATMP gets called
641 2016-07-13T22:10:19  *** sipa sets mode: -o sipa
642 2016-07-13T22:10:27  <morcos> well 16 cores, so thats a lot of lock contention
643 2016-07-13T22:10:28  *** beIcher is now known as belcher
644 2016-07-13T22:10:41  <sipa> ah, yes
645 2016-07-13T22:10:57  <sipa> we should probably also look for ways to avoid sha256 there
646 2016-07-13T22:11:09  <morcos> the added erase time is small though, maybe 1ms per block, but again happening at a much better time, and a small fraction of the savings
647 2016-07-13T22:12:30  <jeremyrubin> sipa: we compared to using the iterator, but I think it was negligible
648 2016-07-13T22:12:51  <jeremyrubin> but with the benefit of not having to check for iterator validity before erase
649 2016-07-13T22:13:26  <jeremyrubin> (otherwise you would need to construct a set of iterators from the GC-list because concurrent Gets could add the same iter)
650 2016-07-13T22:14:37  <jeremyrubin> sipa: wait could you clarify which you are referring to re: sha256? the uint256_t on the eviction list?
651 2016-07-13T22:18:58  <sipa> jeremyrubin: the sigcache stores sha256's of the cached data
652 2016-07-13T22:19:08  <sipa> sha256 is relatively slow
653 2016-07-13T22:19:11  *** TomMc has quit IRC
654 2016-07-13T22:20:39  <sipa> jeremyrubin: no, not talking about erasing... just the cache in general
655 2016-07-13T22:20:57  <jeremyrubin> sipa: gotcha, should be fine to use siphash right?
656 2016-07-13T22:21:33  <sipa> jeremyrubin: i'm not sure
657 2016-07-13T22:25:39  <sipa> jeremyrubin: it's not unreasonable to have a cache size of a billion with certain hardware today
658 2016-07-13T22:25:54  <jeremyrubin> sipa: if it is salted then it wouldn't be possible to predictably create a block with bad cache lookup performance?
659 2016-07-13T22:26:37  <sipa> if siphash's security claims are correct, salting should make it unpredictablr
660 2016-07-13T22:26:48  <sipa> but the 64-bit hash may not be enough
661 2016-07-13T22:26:58  <sipa> perhaps a 128-bit hash works
662 2016-07-13T22:31:47  <jeremyrubin> sipa: ttyl
663 2016-07-13T22:43:28  *** murch has quit IRC
664 2016-07-13T22:47:50  *** xinxi has joined #bitcoin-core-dev
665 2016-07-13T22:48:10  <gmaxwell> it's quite easy to just create a lockfree queue using atomics. I think bluematt has one in his fibre codebase.
666 2016-07-13T22:48:38  <gmaxwell> so we don't need to take the additional boost dep if its irritating, or if we do it shouldn't be too hard to get rid of.
667 2016-07-13T22:49:18  <BlueMatt> https://github.com/bitcoinfibre/bitcoinfibre/blob/master/src/udpnet.cpp#L1175
668 2016-07-13T22:49:24  <BlueMatt> actually lockless ring buffer, but close enough
669 2016-07-13T22:50:50  <gmaxwell> sipa: we could have a faster wide cryptographic hash, but last I looked low hitrate seemed to be the bigger performance ipediment than the time spent hashing.
670 2016-07-13T22:53:04  *** xinxi has quit IRC
671 2016-07-13T22:54:54  <sipa> gmaxwell: a sparse hash set would also be much more space efficient
672 2016-07-13T22:55:05  <sipa> (no malloc for each entry)
673 2016-07-13T22:56:41  *** Guyver2 has quit IRC
674 2016-07-13T22:58:37  *** fengling_ has joined #bitcoin-core-dev
675 2016-07-13T23:02:19  *** shangzhou has joined #bitcoin-core-dev
676 2016-07-13T23:02:29  <sipa> BlueMatt: interesting!
677 2016-07-13T23:03:04  <sipa> a ring buffer would suffice here, i guess... if it overflows you can always grab the lock and apply the erases regardless
678 2016-07-13T23:03:26  *** fengling_ has quit IRC
679 2016-07-13T23:05:00  <shangzhou> sipa: the bitcoin.sipa.be data looks like not up to date
680 2016-07-13T23:05:06  *** BonyM has quit IRC
681 2016-07-13T23:07:12  *** belcher has quit IRC
682 2016-07-13T23:07:27  <sipa> shangzhou: oh?
683 2016-07-13T23:08:04  <sipa> oh, yds
684 2016-07-13T23:08:06  <sipa> yes
685 2016-07-13T23:08:30  *** Guest64966 has joined #bitcoin-core-dev
686 2016-07-13T23:12:51  *** belcher has joined #bitcoin-core-dev
687 2016-07-13T23:17:12  *** laurentmt has joined #bitcoin-core-dev
688 2016-07-13T23:17:12  *** xinxi has joined #bitcoin-core-dev
689 2016-07-13T23:17:14  *** laurentmt has quit IRC
690 2016-07-13T23:20:01  *** BonyM has joined #bitcoin-core-dev
691 2016-07-13T23:23:20  <BlueMatt> thoughts on decreasing transaction-serialization time?
692 2016-07-13T23:23:28  <BlueMatt> its currently slow as fuck, even when serializing into a static buffer
693 2016-07-13T23:23:41  <BlueMatt> caching it helps a lot, but eats a decent chunk of memory
694 2016-07-13T23:24:16  *** xinxi has quit IRC
695 2016-07-13T23:25:14  <BlueMatt> (and, mostly, caching tx serialization diverges fibre a decent chunk from core, which is annoying)
696 2016-07-13T23:27:18  <cfields> BlueMatt: working on a few optims to tx right now. not sure how applicable though...
697 2016-07-13T23:27:38  <cfields> BlueMatt: easy quick gain is lazy hashing
698 2016-07-13T23:28:31  <BlueMatt> cfields: I said serialization, not deserialization :p
699 2016-07-13T23:29:23  <sipa> BlueMatt: i think transaction data shoild be stored in a songle malloced buffer, with internal pointers
700 2016-07-13T23:29:24  <cfields> BlueMatt: oh right, heh. i just assumed you were working on the other direction. nevermind then :)
701 2016-07-13T23:29:33  <cfields> lazy hashing would be a quick way to slow you down, then :p
702 2016-07-13T23:29:38  <BlueMatt> what we really should do is calculate the hash at the same time as we serialize/deserialize, since thats easy
703 2016-07-13T23:29:40  <sipa> BlueMatt: that does require encapsulating all fields though
704 2016-07-13T23:29:49  <BlueMatt> sipa: yes, that is a bit longer-term, though
705 2016-07-13T23:29:53  <sipa> agree
706 2016-07-13T23:29:54  <BlueMatt> but, yea, that would help a ton of things
707 2016-07-13T23:30:46  <cfields> sipa: speaking of which, after profiling noticing how much time was spent in prevector, i've fixed up copying/moving to be much quicker. you opposed to dinking around in there?
708 2016-07-13T23:34:57  <sipa> cfields: please do!
709 2016-07-13T23:35:27  <sipa> cfields: actually, we should make it class safe...
710 2016-07-13T23:35:41  <cfields> sipa: better, i added static_asserts :p
711 2016-07-13T23:35:46  <sipa> and use std::move etc instead of realloc etx
712 2016-07-13T23:35:53  <BlueMatt> cfields: ironically, no, FIBRE is mostly tx copy/serialize - it copies most tx from mempool into a CBlock, and serializes those txn to build the data-FEC chunks from which to calculate the missing chunks
713 2016-07-13T23:36:07  <BlueMatt> it only ever has to deserialize txn that it missed from mempool (which is relatively few)
714 2016-07-13T23:36:37  <sipa> BlueMatt: we could also add a means to serialize a block from a list of tx shared_pointers
715 2016-07-13T23:36:38  <cfields> sipa: i started by special-casing it with enable_if<> and SFINAE, but I think we're better off just specializing for ~std::is_trivial<>
716 2016-07-13T23:37:00  <gmaxwell> What fiber FECs is not the block directly, but the block with padding to make transactions line up on packet boundaries.
717 2016-07-13T23:37:04  <sipa> BlueMatt: instead first needing a full deep copy of the tx into the block
718 2016-07-13T23:37:04  <gmaxwell> er fibre
719 2016-07-13T23:37:06  <BlueMatt> sipa: who said anything about serializing?
720 2016-07-13T23:37:18  <BlueMatt> oh, what gmaxwell said
721 2016-07-13T23:37:42  <BlueMatt> sipa: no, it just deep-copies into the CBlock that it hands to ProcessNewBlock
722 2016-07-13T23:38:02  <sipa> "FIBRE is mostly tx copy/serializr" -- BlueMatt
723 2016-07-13T23:38:28  <sipa> BlueMatt: ok, maybe we should just make CBlock have shared_ptr to transactions...
724 2016-07-13T23:38:44  <BlueMatt> that would help a lot, but it doesnt solve my serialize-time problem
725 2016-07-13T23:38:47  <BlueMatt> (which is like...all my time)
726 2016-07-13T23:38:49  <sipa> ok
727 2016-07-13T23:47:34  *** adamg has quit IRC
728 2016-07-13T23:50:28  *** adamg has joined #bitcoin-core-dev
729 2016-07-13T23:51:34  *** xinxi has joined #bitcoin-core-dev
730 2016-07-13T23:54:48  *** kadoban has quit IRC
731 2016-07-13T23:56:04  *** xinxi has quit IRC