1 2016-12-04T00:22:01  *** moli has joined #bitcoin-core-dev
  2 2016-12-04T01:13:40  *** jtimon has joined #bitcoin-core-dev
  3 2016-12-04T01:22:13  *** Atomicat has quit IRC
  4 2016-12-04T01:30:28  *** jeremyrubin has quit IRC
  5 2016-12-04T01:30:32  *** Madars has quit IRC
  6 2016-12-04T01:30:36  *** jeremyrubin has joined #bitcoin-core-dev
  7 2016-12-04T01:31:50  *** Madars has joined #bitcoin-core-dev
  8 2016-12-04T01:41:17  *** Atomicat has joined #bitcoin-core-dev
  9 2016-12-04T01:52:55  *** echonaut has quit IRC
 10 2016-12-04T01:53:02  *** echonaut has joined #bitcoin-core-dev
 11 2016-12-04T01:56:09  *** moli has quit IRC
 12 2016-12-04T02:02:55  *** moli has joined #bitcoin-core-dev
 13 2016-12-04T02:29:59  *** MrHodl has quit IRC
 14 2016-12-04T02:30:44  *** fuc has joined #bitcoin-core-dev
 15 2016-12-04T02:53:32  *** Giszmo has quit IRC
 16 2016-12-04T03:22:41  *** Ylbam has quit IRC
 17 2016-12-04T03:33:13  *** Cheeseo has quit IRC
 18 2016-12-04T03:41:51  *** cryptapus has joined #bitcoin-core-dev
 19 2016-12-04T03:41:51  *** cryptapus has joined #bitcoin-core-dev
 20 2016-12-04T03:42:07  *** gribble has quit IRC
 21 2016-12-04T03:42:40  *** blkdb has quit IRC
 22 2016-12-04T03:42:42  *** cryptapus_afk has quit IRC
 23 2016-12-04T03:42:49  *** blkdb has joined #bitcoin-core-dev
 24 2016-12-04T03:47:14  *** wasi has quit IRC
 25 2016-12-04T03:49:35  *** gribble has joined #bitcoin-core-dev
 26 2016-12-04T04:07:01  *** Alopex has quit IRC
 27 2016-12-04T04:08:06  *** Alopex has joined #bitcoin-core-dev
 28 2016-12-04T04:09:12  *** Atomicat has quit IRC
 29 2016-12-04T04:35:37  *** wasi has joined #bitcoin-core-dev
 30 2016-12-04T04:44:21  *** Alopex has quit IRC
 31 2016-12-04T04:45:26  *** Alopex has joined #bitcoin-core-dev
 32 2016-12-04T04:51:24  *** Chris_Stewart_5 has quit IRC
 33 2016-12-04T05:03:44  *** alpalp has quit IRC
 34 2016-12-04T05:07:52  *** Chris_Stewart_5 has joined #bitcoin-core-dev
 35 2016-12-04T05:09:22  *** ThomasV has joined #bitcoin-core-dev
 36 2016-12-04T05:22:30  *** Chris_Stewart_5 has quit IRC
 37 2016-12-04T05:29:05  *** fuc has quit IRC
 38 2016-12-04T05:38:16  *** kadoban has quit IRC
 39 2016-12-04T05:52:21  *** Alopex has quit IRC
 40 2016-12-04T05:53:26  *** Alopex has joined #bitcoin-core-dev
 41 2016-12-04T07:08:41  *** ThomasV has quit IRC
 42 2016-12-04T07:38:06  *** Alopex has quit IRC
 43 2016-12-04T07:39:11  *** Alopex has joined #bitcoin-core-dev
 44 2016-12-04T07:43:51  *** ThomasV has joined #bitcoin-core-dev
 45 2016-12-04T07:59:29  *** To7 has quit IRC
 46 2016-12-04T08:08:04  *** jtimon has quit IRC
 47 2016-12-04T08:08:26  *** Alopex has quit IRC
 48 2016-12-04T08:09:32  *** Alopex has joined #bitcoin-core-dev
 49 2016-12-04T08:25:00  <bitcoin-git> [bitcoin] TheBlueMatt opened pull request #9273: Remove unused CDiskBlockPos* argument from ProcessNewBlock (master...2016-12-remove-unused-pnb-param) https://github.com/bitcoin/bitcoin/pull/9273
 50 2016-12-04T08:35:02  *** Alopex has quit IRC
 51 2016-12-04T08:36:07  *** Alopex has joined #bitcoin-core-dev
 52 2016-12-04T08:37:16  *** Ylbam has joined #bitcoin-core-dev
 53 2016-12-04T08:37:42  *** gabridome has joined #bitcoin-core-dev
 54 2016-12-04T08:56:58  *** ThomasV has quit IRC
 55 2016-12-04T08:59:44  *** gabridome has quit IRC
 56 2016-12-04T09:00:38  *** gabridome has joined #bitcoin-core-dev
 57 2016-12-04T09:00:46  *** gabridome has quit IRC
 58 2016-12-04T09:05:48  *** ThomasV has joined #bitcoin-core-dev
 59 2016-12-04T09:12:55  *** sanada has quit IRC
 60 2016-12-04T09:31:06  *** Alopex has quit IRC
 61 2016-12-04T09:32:11  *** Alopex has joined #bitcoin-core-dev
 62 2016-12-04T10:15:47  *** Lightsword has quit IRC
 63 2016-12-04T10:16:12  *** Lightsword has joined #bitcoin-core-dev
 64 2016-12-04T10:31:42  *** moli has quit IRC
 65 2016-12-04T10:32:06  *** Alopex has quit IRC
 66 2016-12-04T10:33:12  *** Alopex has joined #bitcoin-core-dev
 67 2016-12-04T10:36:47  *** laurentmt has joined #bitcoin-core-dev
 68 2016-12-04T10:37:57  *** laurentmt has quit IRC
 69 2016-12-04T10:45:01  *** Alopex has quit IRC
 70 2016-12-04T10:46:06  *** Alopex has joined #bitcoin-core-dev
 71 2016-12-04T11:01:49  *** AaronvanW has joined #bitcoin-core-dev
 72 2016-12-04T11:33:40  *** ThomasV has quit IRC
 73 2016-12-04T12:03:50  *** ThomasV has joined #bitcoin-core-dev
 74 2016-12-04T12:41:17  *** alpalp has joined #bitcoin-core-dev
 75 2016-12-04T12:41:17  *** alpalp has joined #bitcoin-core-dev
 76 2016-12-04T13:12:44  *** Atomicat has joined #bitcoin-core-dev
 77 2016-12-04T13:26:37  *** moli has joined #bitcoin-core-dev
 78 2016-12-04T13:36:06  *** CubicEarth has quit IRC
 79 2016-12-04T13:36:41  *** CubicEarth has joined #bitcoin-core-dev
 80 2016-12-04T13:41:12  *** CubicEarth has quit IRC
 81 2016-12-04T13:47:17  *** moli has quit IRC
 82 2016-12-04T13:48:01  *** ThomasV has quit IRC
 83 2016-12-04T13:48:13  *** Guyver2 has joined #bitcoin-core-dev
 84 2016-12-04T13:55:32  *** moli has joined #bitcoin-core-dev
 85 2016-12-04T14:14:33  *** moli has quit IRC
 86 2016-12-04T14:17:17  *** afk11 has quit IRC
 87 2016-12-04T14:25:15  *** alpalp has quit IRC
 88 2016-12-04T14:26:09  *** afk11 has joined #bitcoin-core-dev
 89 2016-12-04T14:26:09  *** afk11 has quit IRC
 90 2016-12-04T14:26:09  *** afk11 has joined #bitcoin-core-dev
 91 2016-12-04T14:30:23  *** alpalp has joined #bitcoin-core-dev
 92 2016-12-04T14:31:00  *** murchandamus has quit IRC
 93 2016-12-04T14:31:48  *** murchandamus has joined #bitcoin-core-dev
 94 2016-12-04T14:41:35  *** murchandamus has quit IRC
 95 2016-12-04T14:42:24  *** murchandamus has joined #bitcoin-core-dev
 96 2016-12-04T14:45:33  *** JackH has quit IRC
 97 2016-12-04T14:46:24  *** JackH has joined #bitcoin-core-dev
 98 2016-12-04T14:48:36  *** moli has joined #bitcoin-core-dev
 99 2016-12-04T14:54:46  *** alpalp has quit IRC
100 2016-12-04T14:57:32  *** alpalp has joined #bitcoin-core-dev
101 2016-12-04T15:05:36  *** murchandamus has quit IRC
102 2016-12-04T15:06:08  *** murchandamus has joined #bitcoin-core-dev
103 2016-12-04T15:08:11  *** murchandamus has quit IRC
104 2016-12-04T15:09:17  *** moli has quit IRC
105 2016-12-04T15:12:48  *** moli has joined #bitcoin-core-dev
106 2016-12-04T15:18:21  *** murchandamus has joined #bitcoin-core-dev
107 2016-12-04T15:47:57  *** kadoban has joined #bitcoin-core-dev
108 2016-12-04T15:52:58  *** Chris_Stewart_5 has joined #bitcoin-core-dev
109 2016-12-04T16:09:26  *** Giszmo has joined #bitcoin-core-dev
110 2016-12-04T16:09:42  *** Cheeseo has joined #bitcoin-core-dev
111 2016-12-04T16:09:42  *** Cheeseo has joined #bitcoin-core-dev
112 2016-12-04T16:09:43  *** Chris_Stewart_5 has quit IRC
113 2016-12-04T16:15:02  *** Chris_Stewart_5 has joined #bitcoin-core-dev
114 2016-12-04T16:16:58  *** Chris_Stewart_5 has quit IRC
115 2016-12-04T16:17:18  *** Chris_Stewart_5 has joined #bitcoin-core-dev
116 2016-12-04T16:39:47  *** Sosumi has joined #bitcoin-core-dev
117 2016-12-04T16:48:57  *** wvr has joined #bitcoin-core-dev
118 2016-12-04T16:49:03  *** protomar has joined #bitcoin-core-dev
119 2016-12-04T16:59:30  *** wvr has quit IRC
120 2016-12-04T17:16:37  *** RoyceX has joined #bitcoin-core-dev
121 2016-12-04T17:17:26  *** Giszmo has quit IRC
122 2016-12-04T17:19:33  *** Cheeseo has quit IRC
123 2016-12-04T17:45:28  *** Chris_Stewart_5 has quit IRC
124 2016-12-04T17:47:57  *** Chris_Stewart_5 has joined #bitcoin-core-dev
125 2016-12-04T17:56:13  *** jtimon has joined #bitcoin-core-dev
126 2016-12-04T17:57:26  *** laurentmt has joined #bitcoin-core-dev
127 2016-12-04T17:57:42  *** laurentmt has quit IRC
128 2016-12-04T18:21:13  *** RoyceX has quit IRC
129 2016-12-04T18:24:44  *** MarcoFalke has joined #bitcoin-core-dev
130 2016-12-04T18:24:55  <bitcoin-git> [bitcoin] MarcoFalke opened pull request #9274: [qa] pruning: Use cached utxo set to run faster (master...Mf1612-qaPruningCacheUtxos) https://github.com/bitcoin/bitcoin/pull/9274
131 2016-12-04T18:25:51  *** JackH has quit IRC
132 2016-12-04T18:26:56  *** alpalp has quit IRC
133 2016-12-04T18:40:50  *** RoyceX has joined #bitcoin-core-dev
134 2016-12-04T18:40:50  *** RoyceX has joined #bitcoin-core-dev
135 2016-12-04T18:52:23  *** LeMiner2 has joined #bitcoin-core-dev
136 2016-12-04T18:54:51  *** LeMiner has quit IRC
137 2016-12-04T18:54:52  *** LeMiner2 is now known as LeMiner
138 2016-12-04T19:02:57  *** Guyver2__ has joined #bitcoin-core-dev
139 2016-12-04T19:05:26  *** Guyver2 has quit IRC
140 2016-12-04T19:05:31  *** Guyver2__ is now known as Guyver2
141 2016-12-04T19:07:50  *** Guyver2__ has joined #bitcoin-core-dev
142 2016-12-04T19:11:26  *** Guyver2 has quit IRC
143 2016-12-04T19:11:33  *** Guyver2__ is now known as Guyver2
144 2016-12-04T19:11:42  *** Sosumi has quit IRC
145 2016-12-04T19:13:58  *** Sosumi has joined #bitcoin-core-dev
146 2016-12-04T19:27:55  *** JackH has joined #bitcoin-core-dev
147 2016-12-04T19:32:02  *** wvr has joined #bitcoin-core-dev
148 2016-12-04T19:32:19  *** alpalp has joined #bitcoin-core-dev
149 2016-12-04T19:32:19  *** alpalp has joined #bitcoin-core-dev
150 2016-12-04T19:38:15  *** CubicEarth has joined #bitcoin-core-dev
151 2016-12-04T19:58:22  <bitcoin-git> [bitcoin] jonasschnelli pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/2efcfa5acfac...4d955fc5824b
152 2016-12-04T19:58:23  <bitcoin-git> bitcoin/master 827d9a3 Wladimir J. van der Laan: qt: Replace NetworkToggleStatusBarControl with generic ClickableLabel...
153 2016-12-04T19:58:23  <bitcoin-git> bitcoin/master 042f9fa Wladimir J. van der Laan: qt: Show progress overlay when clicking spinner icon...
154 2016-12-04T19:58:24  <bitcoin-git> bitcoin/master 4d955fc Jonas Schnelli: Merge #9218: qt: Show progress overlay when clicking spinner icon...
155 2016-12-04T19:58:37  <bitcoin-git> [bitcoin] jonasschnelli closed pull request #9218: qt: Show progress overlay when clicking spinner icon (master...2016_11_overlay_when_clicking_sync_icon) https://github.com/bitcoin/bitcoin/pull/9218
156 2016-12-04T20:18:38  <BlueMatt> jtimon: re: libconsensus: have you thought more about it after the discussions in milan? I'm still not a huge fan of exposing a ton of functions and prefer a simple refactor that a) makes it so that everything ProcessNewBlock calls isnt aware of disk positioning/etc and b) move all that stuff into a class which owns mapBlockIndex, chainActive, etc
157 2016-12-04T20:18:42  <BlueMatt> jtimon: see https://github.com/TheBlueMatt/bitcoin/commits/2016-12-cconsensus
158 2016-12-04T20:20:20  <jtimon> I keep thinking the same thing more or less
159 2016-12-04T20:21:25  <jtimon> if you don't want to expose verifyHeader or verifyTx...I know some people want verifyHeader for SPV wallets, but not so sure about verifyTx
160 2016-12-04T20:21:27  *** bobbytux__ has joined #bitcoin-core-dev
161 2016-12-04T20:21:48  <jtimon> regarding process block, as discussed, I think we should expose verifyBlock instead
162 2016-12-04T20:22:33  *** bobbytux_ has quit IRC
163 2016-12-04T20:22:42  <BlueMatt> jtimon: why, though? if you expose ProcessNewBlock{,Headers} someone can implement a full spv client and full client all in the same codepaths, without exposing low-level partial-verification functions
164 2016-12-04T20:23:38  <BlueMatt> (or, excuse me, without requiring users do their own utxo-management shit, while still giving them access to query the utxo state)
165 2016-12-04T20:23:39  <jtimon> how can an SPV node calle ProcessNewBlock?
166 2016-12-04T20:23:48  <BlueMatt> ProcessNewBlockHeaders
167 2016-12-04T20:23:51  <jtimon> why would they want to depend on levelDB ?
168 2016-12-04T20:23:59  <BlueMatt> nono, that part should be abstracted out
169 2016-12-04T20:24:01  <BlueMatt> as discussed
170 2016-12-04T20:24:33  <BlueMatt> eg see the blockstores in bitcoinj: https://github.com/bitcoinj/bitcoinj/tree/master/core/src/main/java/org/bitcoinj/store
171 2016-12-04T20:25:18  <jtimon> oh, right, you were ok with abstracting storage, just wanted libconsensus to manage reorgs and all the rest in an abstracted way
172 2016-12-04T20:25:35  <BlueMatt> yea, best to handle /all/ the complexity imo
173 2016-12-04T20:25:39  <jtimon> isn't ProcessNewBlockHeaders an intermediate function?
174 2016-12-04T20:25:54  <BlueMatt> no? it takes headers and adds them and selects the best headers chain it has seen
175 2016-12-04T20:27:06  <jtimon> then I'm not sure what you mean by "intermediate" in this context
176 2016-12-04T20:27:57  <BlueMatt> ehh, I apologize, I forgot that your definition of verifyBlock also means checking all the txes connect, ie essentially TestBlockValidity
177 2016-12-04T20:28:02  <BlueMatt> instead of existing CheckBlock-type functions
178 2016-12-04T20:28:22  <jtimon> well, no really testblockvelidity since that call connectBlock too
179 2016-12-04T20:28:42  <BlueMatt> hmm?
180 2016-12-04T20:28:44  <jtimon> not really
181 2016-12-04T20:28:58  <BlueMatt> huh
182 2016-12-04T20:29:36  <jtimon> my proposed verifyBlock just tells you whether a block is valid or not, but it doesn't do anything with it
183 2016-12-04T20:29:47  *** MykelSIlver has joined #bitcoin-core-dev
184 2016-12-04T20:29:59  <BlueMatt> indeed, like TestBlockValidity?
185 2016-12-04T20:31:55  <jtimon> mhmm, connect block updates transactions, updates undo info and updates the view of the best block, no?
186 2016-12-04T20:32:12  <BlueMatt> but TestBlockValidity throws away all that info afterwards
187 2016-12-04T20:32:21  <jtimon> oh, I see
188 2016-12-04T20:32:25  <BlueMatt> and you have to keep an updated-utxo-state as you connect the block to test validity
189 2016-12-04T20:32:42  <jtimon> well, then yes, but without generating info than then throws away
190 2016-12-04T20:33:18  <BlueMatt> you cand check a block without doing that
191 2016-12-04T20:33:24  <jtimon> yeah, you need to update at least a view of the utxo
192 2016-12-04T20:33:50  <jtimon> right, that was my idea for verifyBlock
193 2016-12-04T20:39:52  <BlueMatt> jtimon: ok, so back to my original question...what do you think of not exposing that and actually having a utxo/disk-storage-abstraced full consensus layer
194 2016-12-04T20:39:52  <BlueMatt> ?
195 2016-12-04T20:40:56  <jtimon> well, where I have te strongest opinion is in abstracting from storage
196 2016-12-04T20:41:10  <jtimon> and agree on that
197 2016-12-04T20:41:12  <BlueMatt> ok...?
198 2016-12-04T20:41:25  <BlueMatt> oh, well sure, I dont think anyone wants libconsensus to require a certain storage layer
199 2016-12-04T20:41:55  <jtimon> what would be the API? precessnewblock and processnewheaders?
200 2016-12-04T20:43:12  <BlueMatt> jtimon: essentially, yes, PNB, PNBH, some initialization api (see https://github.com/TheBlueMatt/bitcoin/blob/2016-12-cconsensus/src/validation.cpp#L80) and a utxo-state-accessor/loadblock-accessor api
201 2016-12-04T20:43:17  <jtimon> well, I do think some people want to depend on a concrete storage , but since it's neither of us, let's leave that aside
202 2016-12-04T20:44:01  <jtimon> what about caches? any API for those?
203 2016-12-04T20:44:03  <BlueMatt> well once the bitcoin core codebase "eats its own dogfood", as it were, we can take the storage layer we have and expose that optionally
204 2016-12-04T20:44:24  <jtimon> right
205 2016-12-04T20:44:35  <BlueMatt> no, caches are up to the client application, ie the client application gives libbitcoinconsensus pcoinsTip
206 2016-12-04T20:44:52  <BlueMatt> similarly, in the future, maybe we can expose CCoinsViewCache or whatever
207 2016-12-04T20:44:59  <BlueMatt> but for v1, its uneccessary
208 2016-12-04T20:45:18  <jtimon> mhmm, I mean for example the versionbits cache
209 2016-12-04T20:45:40  *** droark has quit IRC
210 2016-12-04T20:45:45  <jtimon> is that hidden to the caller or exposed somehow?
211 2016-12-04T20:46:07  <BlueMatt> since its currently not exposed, hidden and present
212 2016-12-04T20:46:27  <BlueMatt> dont expose anything unless its required, and for v1, dont worry too much about performance or memory usage
213 2016-12-04T20:47:04  <sipa> the reason for not exposing would exactly be performance and memory usage, i think
214 2016-12-04T20:47:23  <sipa> it's pretty hard to design a stable api that can be used efficiently for those things
215 2016-12-04T20:47:45  <sipa> so i'd say make all state (caches, indexes, ...) by default part of the abstracted state
216 2016-12-04T20:48:01  <sipa> and perhaps later introduce a way for the caller to install callbacks to override it, and provide their own
217 2016-12-04T20:48:05  <jtimon> well, the api may change if consensus rules change
218 2016-12-04T20:52:48  <jtimon> BlueMatt: I highly doubt some implementors (say libbitcoin) will use processNewBlock, then again I also doubted they were going to use verifyBlock directly either, I was hoping maybe verifyHeader, verifyTx + GetConsensusFlags
219 2016-12-04T20:53:39  <BlueMatt> sipa: yea, except for v1 it doesnt matter, really - just dont expose anything and later, if we remove caches to make the library lighter, expose hooks to replace the caches
220 2016-12-04T20:53:55  <bitcoin-git> [bitcoin] morcos opened pull request #9276: Some minor testing cleanups (master...rpccleanup) https://github.com/bitcoin/bitcoin/pull/9276
221 2016-12-04T20:53:57  <jtimon> leaving that aside, I would like libconsensus to be the specification of the consensus rules for a given block to be valid, it seems that with your approach it would be more than that
222 2016-12-04T20:54:33  <jtimon> but bitcoin core is currently way more than that, so...
223 2016-12-04T20:54:53  <BlueMatt> jtimon: I mean the way any sane full node is structured there is an abstracted utxo/blocks-on-disk storage and something which checks everything and itnerfaces with that state
224 2016-12-04T20:55:07  <jtimon> I also hope you don't intend to put half of the server package in the consensus package
225 2016-12-04T20:55:27  <BlueMatt> jtimon: I see no reason why libbitcoinconsensus shouldnt repalce everything in between because if the goal is to reduce the risk of consensus incompatibility, we should replace everything we can
226 2016-12-04T20:55:52  <BlueMatt> jtimon: I dont care what it weighs, only what it does....cutting down the weight is for later versions
227 2016-12-04T20:55:53  <BlueMatt> :p
228 2016-12-04T20:56:40  * BlueMatt -> brunch
229 2016-12-04T20:57:55  <jtimon> because some implementors (like libbitcoin) value more the ability to have their own concurrency model (ie without locks) than reducing risks of consensus incompatibility beyond verifyScript (which they also use only optionally)
230 2016-12-04T20:58:22  * jtimon should have brinner soon too
231 2016-12-04T21:00:34  <jtimon> I agree, that we can leave "cutting weight" for later (like we're leaving moving CFeeRate for later for very long), my point is that with your approach, even after you're done, it will be much bigger than "the specification for whether a block is valid or not"
232 2016-12-04T21:01:17  <jtimon> it would be more like the specification of "how to follow and store the longest valid chain"
233 2016-12-04T21:01:54  <jtimon> then someone will want prunning too...
234 2016-12-04T21:02:22  *** droark has joined #bitcoin-core-dev
235 2016-12-04T21:04:02  <sipa> jtimon: yes, it's the specification for whether a blockchain is valid or not
236 2016-12-04T21:04:22  <jtimon> sipa: processNewBlock does much more than that
237 2016-12-04T21:04:40  <sipa> well, yes, due to necessity
238 2016-12-04T21:04:50  <sipa> we can't track the validity of every single chain
239 2016-12-04T21:04:59  <sipa> so we only have a utxo set at a tip
240 2016-12-04T21:05:47  <jtimon> that's we, maybe other callers want to do things differently
241 2016-12-04T21:07:09  <jtimon> then maybe someone else asks for "sharded prunning" or something else, and soon you end up again with a lot of things that don't have anything to do with consensus rules
242 2016-12-04T21:07:38  <sipa> yes, so we can later provide ways to plug in your own state storage (utxo/caches/indexes/...)
243 2016-12-04T21:08:13  <jtimon> sipa: oh, BlueMatt's approach already gives you an abstraction for the utxo and chain index storage
244 2016-12-04T21:08:49  <sipa> oj
245 2016-12-04T21:08:51  <sipa> ok
246 2016-12-04T21:09:36  *** MarcoFalke has left #bitcoin-core-dev
247 2016-12-04T21:09:57  <jtimon> but my example is that if it supports prunning, that needs to be exposed somehow
248 2016-12-04T21:10:05  *** Sosumi has quit IRC
249 2016-12-04T21:10:09  <sipa> heh?
250 2016-12-04T21:10:20  <sipa> how does block storage have anything to do with this
251 2016-12-04T21:10:36  <jtimon> isn't prunning deleting blocks you are storing?
252 2016-12-04T21:10:49  <sipa> yes, but blocks wouldn't be stored by libconsensus
253 2016-12-04T21:11:03  <jtimon> they would with BlueMatt's approach
254 2016-12-04T21:11:15  <sipa> hmm, that seems strange
255 2016-12-04T21:11:32  <jtimon> if I undesrtood him correctly
256 2016-12-04T21:11:47  <sipa> i would have an api where you plug in a way libconsensus to ask you for a block
257 2016-12-04T21:12:06  <sipa> if yiu're abstracting utxo storage, i don't see why you wouldn't abstract nlock storage
258 2016-12-04T21:12:16  <jtimon> that's why he needs something like https://github.com/bitcoinj/bitcoinj/blob/master/core/src/main/java/org/bitcoinj/store/BlockStore.java#L39
259 2016-12-04T21:12:18  <sipa> which is even less consensus critical
260 2016-12-04T21:12:41  <sipa> but that's the user that provides the blockstore
261 2016-12-04T21:12:44  <sipa> not the library
262 2016-12-04T21:14:15  <jtimon> right, the library provides an API and callbacks the implementor for stored data in both cases, but in bluematt's case, it also updates the storage, it doesn't use it in a read only way
263 2016-12-04T21:14:40  <jtimon> maybe prunning was a bad example, I agree you don't need to put it in libconsensus even in that case
264 2016-12-04T21:15:59  <gmaxwell> I think abstracting things like all this storage maybe getting into the realm of imaginary users... is there really many users that want to make no change (not even instrumentation) to consensus but is very interested at implementing their own UTXO database?
265 2016-12-04T21:16:00  <jtimon> in my case, it takes a block and says valid or validation error x and does nothing more than that
266 2016-12-04T21:16:31  <sipa> jtimon: but you need to reimplement a lot of complicated logic in your application for it
267 2016-12-04T21:16:36  <sipa> utxo caching is hard
268 2016-12-04T21:16:48  <sipa> block index tracking with skiplists is not trivial
269 2016-12-04T21:16:54  <jtimon> gmaxwell: I agree that part of the problem here is not having a clear idea of the users or their preferences
270 2016-12-04T21:16:59  <gmaxwell> I thought the goal for libconsensus is that so that when someone wants to make their own custom wallet thing they wouldn't need to reimplment any of the consensus logic, just to create their own application logic.
271 2016-12-04T21:17:20  <jtimon> gmaxwell: but it seems that your objection would apply to both my approach and matt's
272 2016-12-04T21:17:25  <sipa> i really don't care about users at this point - we don't have any, and have no easy way to build somwthing others could use
273 2016-12-04T21:17:39  <sipa> i care about separating consensus from nonconsensus logic
274 2016-12-04T21:18:00  <gmaxwell> if so that would really mean all the chain management would really be inside the library.
275 2016-12-04T21:18:19  <gmaxwell> sipa: okay so for the purpose of consensus safty of changes, the caches and utxo database are all consensus critical--
276 2016-12-04T21:18:35  <sipa> gmaxwell: fair enough, but so is the c++ standard library
277 2016-12-04T21:18:42  <jtimon> gmaxwell: right, matt puts the chain management inside, but also provides an abstraction for storage
278 2016-12-04T21:19:19  <sipa> i think having block storage outside the library is perfectly fine, as consensus logic really does not need blocks except for reorgs
279 2016-12-04T21:19:30  <gmaxwell> jtimon: Providing an abstraction for IO though is different than something that expects the user to implement a complex database.
280 2016-12-04T21:20:01  <sipa> even utxo storage i can be convinced of, i think
281 2016-12-04T21:20:21  <sipa> but block chain tracking (cblockindex/mapblockindex/...) should really be inside
282 2016-12-04T21:20:38  <sipa> it's highly dependent on how we manage chains
283 2016-12-04T21:20:55  <jtimon> providing a wrapper that includes the same storage as bitcoin core, as an extended library should be easy
284 2016-12-04T21:21:20  <gmaxwell> so long as the requirements are very clear and simple... where no bitcoin specific understanding is required... sure.  though doing that without breaking efficiency might be hard.
285 2016-12-04T21:21:29  <sipa> i don't want every cblockindex access to go through wrapper calls
286 2016-12-04T21:22:00  <gmaxwell> e.g. there are ordering requirements for fixation of block data vs utxo data to disk.
287 2016-12-04T21:22:02  <jtimon> yep, efficiency is the main concern with my approach I think
288 2016-12-04T21:23:38  <jtimon> btw, as always comments on https://github.com/bitcoin/bitcoin/pull/8493 welcomed
289 2016-12-04T21:25:10  <jtimon> this kind of thing (for the abstracted storage) is disruptive though https://github.com/bitcoin/bitcoin/pull/8493/commits/0a1ff996c5db5265742e3e90f3690a0b8d9cf045
290 2016-12-04T21:26:51  <jtimon> and the way I implemented it there, is also inneficient for bitcoin core
291 2016-12-04T21:36:35  <jtimon> anyway, there's 2 separated topics here
292 2016-12-04T21:36:47  <jtimon> whether to abstract and expose storage
293 2016-12-04T21:37:22  <jtimon> and whether or not to manage reorgs and other things beyond "check whether a given block is valid or not"
294 2016-12-04T21:39:41  <jtimon> my answers are yes and no, matt's is yes and yes. If gmaxwell's is no and yes, and sipa's are yes and yes, we have all the options represented :p
295 2016-12-04T21:43:02  <jtimon> s/sipa's are no and no/
296 2016-12-04T21:45:35  <sipa> the first thing we should probably do is separate CBlockIndex into block storage data (file, offset, undo, ...), and chain data (block index, work, validation)
297 2016-12-04T21:46:01  <sipa> that would be required if we'd want to abstract block storage out from a library
298 2016-12-04T21:46:08  <sipa> but i thibk it shoukd hapoen regardless
299 2016-12-04T21:46:31  <jtimon> I don't need to do that in my approach
300 2016-12-04T21:46:34  <sipa> the block storage data doesn't even need to be in memory permnantly
301 2016-12-04T21:46:50  <sipa> i understand
302 2016-12-04T21:47:08  <sipa> but my point is that we should do that, even if there never is a library at all
303 2016-12-04T21:48:10  *** Guyver2__ has joined #bitcoin-core-dev
304 2016-12-04T21:48:32  <jtimon> chain.o and coins.o are already separated, I'm not entirely sure I understand what you mean
305 2016-12-04T21:50:33  <sipa> CBlockIndex stores things about where blocks are on disk
306 2016-12-04T21:50:44  <jtimon> oh, I see
307 2016-12-04T21:50:44  <sipa> (which has nothing to do with consensus)
308 2016-12-04T21:50:56  <sipa> and things like validation flags and chainwork
309 2016-12-04T21:51:00  <sipa> and the block header
310 2016-12-04T21:51:02  *** Guyver2 has quit IRC
311 2016-12-04T21:51:03  <sipa> which do
312 2016-12-04T21:51:04  *** Guyver2__ is now known as Guyver2
313 2016-12-04T21:53:45  <jtimon> regarding the validation flags, I replaced #7779 with https://github.com/bitcoin/bitcoin/pull/9271 (although got errors in unittests because we're already relying on the consensus flags being coupled for testing it seems)
314 2016-12-04T21:53:47  <gribble> https://github.com/bitcoin/bitcoin/issues/7779 | An error has occurred and has been logged. Please contact this bot's administrator for more information.
315 2016-12-04T21:56:24  <jtimon> see https://github.com/bitcoin/bitcoin/pull/9271/commits/7a90173b839e682932426f7d6ac91d2df88e2cfb
316 2016-12-04T22:02:36  *** RoyceX has quit IRC
317 2016-12-04T22:04:52  <bitcoin-git> [bitcoin] s-matthew-english opened pull request #9277: clear typo "skipepd" changed to "skipped" (master...patch-12) https://github.com/bitcoin/bitcoin/pull/9277
318 2016-12-04T22:11:50  <bitcoin-git> [bitcoin] MarcoFalke closed pull request #9277: clear typo "skipepd" changed to "skipped" (master...patch-12) https://github.com/bitcoin/bitcoin/pull/9277
319 2016-12-04T22:19:32  *** RoyceX has joined #bitcoin-core-dev
320 2016-12-04T22:20:49  *** protomar has quit IRC
321 2016-12-04T22:26:01  *** Guyver2 has quit IRC
322 2016-12-04T22:38:10  *** bitdev01 has joined #bitcoin-core-dev
323 2016-12-04T22:45:43  *** bitdev01 has quit IRC
324 2016-12-04T22:56:35  <sdaftuar> gmaxwell: regarding bumpfee...  i'd been advising mrbandrews to try to simplify what the rpc command is doing as much as possible, with the hope of getting a helpful utility in place tht we could layer more advanced behavior on top of
325 2016-12-04T22:57:05  <sdaftuar> gmaxwell: so in particular, i wasn't aware of a robust way to identify the change output, so i suggested for this rpc to have the user supply it
326 2016-12-04T22:57:14  <sdaftuar> is there a robust way to identify the change output?
327 2016-12-04T22:58:10  <sipa> yes, IsMine() and not in address book
328 2016-12-04T23:03:20  <sdaftuar> sorry i'm so unfamiliar with this...  i guess there's an IsChange() function, with a TODO saying that this might not work with multisig behavior in the future.  is that not a concern now?
329 2016-12-04T23:09:11  <gmaxwell> sdaftuar: We really should be storing the data in the wallet, but just using IsMine would be better than asking for it.
330 2016-12-04T23:09:41  <gmaxwell> (I think specifying it is basically something that would only be useful to the same people who could use createrawtransaction)
331 2016-12-04T23:10:25  <gmaxwell> Simplifying it is good, but we can't make it a hazard to users-- which is why I was saying it needed to somehow assure that the outputs will not get spent while unconfirmed.
332 2016-12-04T23:11:14  <gmaxwell> Simiarly, just diminishing the change and never adding inputs will produce transactions which won't relay (dust). though I suppose instead of being able to add inputs it could just limit itself to places where that wouldn't be the case.
333 2016-12-04T23:11:43  <sdaftuar> gmaxwell: i haven't looked at the PR lately but i thought it was going to do the right thing with dust.
334 2016-12-04T23:11:54  <gmaxwell> Because of the need to restrict dependant transaction chains I thought a 'bump everything' might be simpler.
335 2016-12-04T23:12:20  <gmaxwell> sdaftuar: it didn't look like it could add inputs to me but equally possible that I was confused.
336 2016-12-04T23:13:02  <sdaftuar> gmaxwell: oh it won't currently add inputs, as i recall, but i think if it would reduce the change output to less than dust, then it would get dropped altogether
337 2016-12-04T23:13:13  <sdaftuar> (that's what i recall some discussion of, anyway)
338 2016-12-04T23:13:33  <sdaftuar> are you saying though that you don't think it should allow feebumping a tx that has descendants?
339 2016-12-04T23:13:54  <sdaftuar> i thought that was part of the point of this RPC, to get that right.
340 2016-12-04T23:17:01  *** Chris_Stewart_5 has quit IRC
341 2016-12-04T23:18:24  <gmaxwell> imagine you have txn  A, and B that spends A.  Then you bump A creating a replaceme A' that replaces A/B.  Now you make nother spend that spends the output of A'  ...  you really need to draft two versions of it one that spends B and one that spends A'... and switch to relaying the other depending on what confirms.
342 2016-12-04T23:18:38  <gmaxwell> I thought that was too advanced, which is why I was suggesting prohibiting descendants.
343 2016-12-04T23:18:48  <sdaftuar> ahh right
344 2016-12-04T23:18:58  <sdaftuar> wow, i totally missed that
345 2016-12-04T23:19:50  <gmaxwell> GAit has implemented bumping in the green address wallet and I think it deals with all this stuff.
346 2016-12-04T23:20:00  <sdaftuar> PRs welcome? :)
347 2016-12-04T23:20:39  <gmaxwell> even my suggestion of making unspendable for one confirm is a bit risky, since it couple be reorged out. :(
348 2016-12-04T23:20:43  <sdaftuar> so i think bumpunconfirmed is a good idea, but i worry about old transactions that have long since been double spent
349 2016-12-04T23:20:52  <gmaxwell> esp since there is non-trivial amounts of hashpower that won't take replacements.
350 2016-12-04T23:20:54  <sdaftuar> or rather, distinguishing those from other unconfirmed transactions
351 2016-12-04T23:21:06  <gmaxwell> well we track conflicted transactions.
352 2016-12-04T23:21:11  <sdaftuar> but not perfectly
353 2016-12-04T23:21:34  <sdaftuar> we can't track conflicted transactions where the conflict chain is broken with tx's not in our wallet
354 2016-12-04T23:22:33  <gmaxwell> Point. Though it would be sufficient for this functionality to only work on transactions where the closure of unconfirmed parents are all in the wallet, I guess.
355 2016-12-04T23:24:20  *** droark has quit IRC
356 2016-12-04T23:25:38  <gmaxwell> I wonder how often someone has multiple unconfirmeds pending and doesn't want to just bump them all. Usually it will save them total fees if they do so.
357 2016-12-04T23:25:54  <gmaxwell> ignoring crazy corner cases like some being doublespent.
358 2016-12-04T23:26:57  <sdaftuar> yeah i think it does seem like a useful feature
359 2016-12-04T23:27:39  <gmaxwell> it just seemed simpler to me than getting all the corner cases right.
360 2016-12-04T23:28:14  <sdaftuar> well i think that has a bunch of corner cases too.  you probably want to make sure you conflict with anything that might possibly confirm, so you have to include even abandoned transactions right?
361 2016-12-04T23:28:26  <sdaftuar> (maybe even 1-confirm tx's!)
362 2016-12-04T23:28:41  <sdaftuar> er 1-conflicted
363 2016-12-04T23:29:11  <gmaxwell> You need to conflict with anything whos outputs you include, so that you don't potentially double-pay.
364 2016-12-04T23:30:07  <sdaftuar> i guess that raises a good UI point
365 2016-12-04T23:30:31  <sdaftuar> probably the user would want to see all the outputs and transactions being bumped, and ack that?
366 2016-12-04T23:30:47  <sdaftuar> in case of the really old wallet tx that the user forgot about problem
367 2016-12-04T23:31:59  <sdaftuar> i dunno, i guess i just assume that heavy wallet users that have been running for a while must have lots of those, maybe that's not a good assumption
368 2016-12-04T23:32:06  <sdaftuar> ?
369 2016-12-04T23:32:57  <gmaxwell> I think it would work like this: take the set of unconfirmed transactions which are AllFromMe (whatever the test that excludes coinjoins is called). Then eliminate all transactions whos parents aren't either all AllFromMe or 6 confirmed.  Then gather up all their outputs, eliminate all the IsMine outputs,  and generate a new transaction which conflicts each transaction in the set, and pays suffi
370 2016-12-04T23:33:03  <gmaxwell> ciently more fees. Then its change output (if any) should be marked so that it will not be treated as IsFromMe for spending purposes. (UI wise we should encourage somewhat overpaying due to this last point)
371 2016-12-04T23:33:07  *** Chris_Stewart_5 has joined #bitcoin-core-dev
372 2016-12-04T23:34:00  <gmaxwell> I don't think most users have a lot of long lost transactions.  I hope not. :) since they'll pretty likely to lose coins if they ditch the wallet thinking the balance was zero.
373 2016-12-04T23:34:06  <sdaftuar> that seems like it ought to work.
374 2016-12-04T23:34:18  <sdaftuar> do we have a way right now to do the mark-change-as-unspendable thing?
375 2016-12-04T23:34:37  <gmaxwell> We have something that doesn't do what we want here.
376 2016-12-04T23:34:47  <gmaxwell> (lockunspent)
377 2016-12-04T23:35:35  <gmaxwell> For this, I think we need to add a boolen to the wallet txn and have isfromme check that.
378 2016-12-04T23:36:07  <gmaxwell> Later we could have more advanced bumping logic that does the make-multiple-transaction-versions.
379 2016-12-04T23:37:40  *** MarcoFalke has joined #bitcoin-core-dev
380 2016-12-04T23:37:56  <gmaxwell> sdaftuar: if you really think that there are many users with txn stuck never confirmed... then we really need to add a 'Pending Payments' -- sum of confirmed outputs which are spent by unconfirmed transactions -- in the UI/rpc. So that people don't discard wallets as empty when they're really not.
381 2016-12-04T23:38:26  <sdaftuar> gmaxwell: actually, maybe i am missing something.  with your proposed algorithm, what would happen if you bumpunconfirmed twice?
382 2016-12-04T23:39:23  <gmaxwell> You would ignore the existing bump (it would fail the AllFrom me in step 1) and make a new bump of all the other unconfirmed transactions in your wallet.
383 2016-12-04T23:40:15  <sdaftuar> i didn't follow why it would fail the AllFromMe step, but if you managed to construct a new tx that didn't conflict with the older bumped fee one, you're screwed right?
384 2016-12-04T23:41:25  <gmaxwell> It would fail the all from me, because I propose the change output of the bumb be flagged specifically so that it does. ... so that under no condition wall the wallet spend it until it is 6 confirmed. Specfically to avoid having to deal with things spending it then having the originals confirm.
385 2016-12-04T23:42:05  <sdaftuar> isn't IsAllFromMe purely a function of the tx inputs?
386 2016-12-04T23:42:27  <gmaxwell> The new tx doesn't need to conflict with it, so long as it doesn't pay to any of its outputs, which it won't because it wasn't included in the set of available txn.
387 2016-12-04T23:43:02  <sdaftuar> i think i am pretty confused right now, why don't i spend another day thinking about the wallet and then rejoin this conversation
388 2016-12-04T23:43:33  <gmaxwell> It's a function of the input's outputs. Is every input an output of mine.
389 2016-12-04T23:44:22  <gmaxwell> sdaftuar: this stuff makes my head hurt too.  I really want to get some kind of bumping in ASAP, it's hard to come up with a minimal feature...
390 2016-12-04T23:45:49  <sdaftuar> one last try before i call it a night -- let's say i have tx1, tx2, tx3 -- each one is IsAllFromMe.  i bump and create tx4, which conflicts with each, and pays all their outputs, and the change output of tx4 is specially flagged or whatever to be unspendable until it confirms.
391 2016-12-04T23:45:59  <sdaftuar> wont't tx4 still be IsAllFromMe?
392 2016-12-04T23:46:59  <sdaftuar> (actually i have to run, back later)
393 2016-12-04T23:47:01  <gmaxwell> Right, okay -- I should have said IsMine and IsAllFromMe  -- the flagging should make it not IsMine.
394 2016-12-04T23:47:05  <gmaxwell> sdaftuar: ttyl
395 2016-12-04T23:50:30  *** Chris_Stewart_5 has quit IRC
396 2016-12-04T23:50:51  *** droark has joined #bitcoin-core-dev
397 2016-12-04T23:58:44  <BlueMatt> gmaxwell: ok, I missed a bunch of scrollback, but hell yes...everyone wants the utxo db in their own format so that they can do other queries against it
398 2016-12-04T23:58:58  <BlueMatt> jtimon: no, I absolutely think we should abstract out block storage
399 2016-12-04T23:59:16  <BlueMatt> jtimon: hell, the branch I sent you did that for ConnectBlock/DisconnectBlock in the first commit
400 2016-12-04T23:59:45  <jtimon> right, I said "yes and yes" for you