1 2017-03-10T00:01:07  *** CubicEarth has joined #bitcoin-core-dev
  2 2017-03-10T00:02:29  <cbits_> Yeah, and it automatically sets rbf checked when you have it set to visible
  3 2017-03-10T00:08:51  *** CubicEarth has quit IRC
  4 2017-03-10T00:19:37  *** CubicEarth has joined #bitcoin-core-dev
  5 2017-03-10T00:21:36  *** GriTBalL has joined #bitcoin-core-dev
  6 2017-03-10T00:27:13  *** CubicEarth has joined #bitcoin-core-dev
  7 2017-03-10T00:32:37  *** CubicEarth has quit IRC
  8 2017-03-10T00:35:14  *** cbits_ has quit IRC
  9 2017-03-10T00:35:38  *** jannes has quit IRC
 10 2017-03-10T00:39:12  *** wudayoda has quit IRC
 11 2017-03-10T00:49:03  *** GriTBalL has quit IRC
 12 2017-03-10T00:58:34  *** abpa has quit IRC
 13 2017-03-10T00:59:07  *** cbits_ has joined #bitcoin-core-dev
 14 2017-03-10T01:00:31  *** arubi has quit IRC
 15 2017-03-10T01:03:20  *** randy-waterhouse has quit IRC
 16 2017-03-10T01:04:24  *** gribble has joined #bitcoin-core-dev
 17 2017-03-10T01:05:55  *** wasi has quit IRC
 18 2017-03-10T01:18:30  *** wasi has joined #bitcoin-core-dev
 19 2017-03-10T01:21:14  *** wudayoda has joined #bitcoin-core-dev
 20 2017-03-10T01:30:10  *** AaronvanW has quit IRC
 21 2017-03-10T01:30:40  *** Victor_sueca has joined #bitcoin-core-dev
 22 2017-03-10T01:30:57  *** Victorsueca has quit IRC
 23 2017-03-10T01:31:03  *** Victor_sueca is now known as Victorsueca
 24 2017-03-10T01:33:50  *** bityogi has quit IRC
 25 2017-03-10T01:38:42  *** cbits_ has quit IRC
 26 2017-03-10T01:56:37  *** gribble has quit IRC
 27 2017-03-10T02:04:34  *** wudayoda has quit IRC
 28 2017-03-10T02:26:24  *** Giszmo has quit IRC
 29 2017-03-10T02:34:34  *** gribble has joined #bitcoin-core-dev
 30 2017-03-10T02:44:05  *** Ylbam has quit IRC
 31 2017-03-10T02:44:56  *** wudayoda has joined #bitcoin-core-dev
 32 2017-03-10T02:50:27  *** wudayoda has quit IRC
 33 2017-03-10T02:56:46  *** dodomojo_ has joined #bitcoin-core-dev
 34 2017-03-10T02:58:39  <gmaxwell> wumpus: would we perhaps want to consider removing zap? we added it because we had no way to abandon transactions, but we do now. I've seen it used in a way that created a lot of damage. (user ran into the unconfirmed depth limit and couldn't make transactions. Then zapped their wallet, then stared paying again, doublspending the @#$@# out of themselves.  .. then were stuck trying to reassemble
 35 2017-03-10T02:58:45  <gmaxwell> the pieces... figure out who they still owed, etc.
 36 2017-03-10T03:02:39  *** CubicEarth has joined #bitcoin-core-dev
 37 2017-03-10T03:02:40  *** PRab has quit IRC
 38 2017-03-10T03:05:15  *** CubicEarth has quit IRC
 39 2017-03-10T03:13:29  *** harrymm1 has joined #bitcoin-core-dev
 40 2017-03-10T03:15:48  *** harrymm has quit IRC
 41 2017-03-10T03:20:53  *** Victor_sueca has joined #bitcoin-core-dev
 42 2017-03-10T03:23:48  *** Victorsueca has quit IRC
 43 2017-03-10T03:38:09  *** wudayoda has joined #bitcoin-core-dev
 44 2017-03-10T03:47:27  *** wudayoda has quit IRC
 45 2017-03-10T03:52:01  *** alpalp has quit IRC
 46 2017-03-10T03:52:19  *** CubicEarth has joined #bitcoin-core-dev
 47 2017-03-10T03:53:22  *** PatBoy has quit IRC
 48 2017-03-10T03:58:21  *** grubles has quit IRC
 49 2017-03-10T03:58:21  *** PatBoy has joined #bitcoin-core-dev
 50 2017-03-10T04:00:18  *** gribble has quit IRC
 51 2017-03-10T04:34:29  *** chris200_ has joined #bitcoin-core-dev
 52 2017-03-10T04:37:27  *** chris2000 has quit IRC
 53 2017-03-10T04:57:38  *** dodomojo_ has quit IRC
 54 2017-03-10T05:01:09  *** Sosumi has quit IRC
 55 2017-03-10T05:01:54  *** juscamarena has quit IRC
 56 2017-03-10T05:09:37  *** tonebox has joined #bitcoin-core-dev
 57 2017-03-10T05:10:40  *** scan_email_bot has joined #bitcoin-core-dev
 58 2017-03-10T05:11:50  <tonebox> It seems like segwit is a temporary fix...  4M is going to be overwhelmed, just like 1mb now...  Wouldn't a better solution be to change the time-base, so now it's 10 minutes per block... Soon, 5 min, then 2.5...  It could revert to 10, and be automatically adjusted just like the difficulty.
 59 2017-03-10T05:13:23  <Lightsword> tonebox, no, that results in higher orphan rates due to latency
 60 2017-03-10T05:14:55  <tonebox> Ok...  Thanks...   Would there be any way to make segwit not fixed at 4mb so this won't be a problem that needs to be solved again in a few years?
 61 2017-03-10T05:17:09  <tonebox> Also, it would seem like a dynamic timebase and fixing the issue with orphans would be a better solution long term.
 62 2017-03-10T05:20:10  <CubicEarth> Lightsword: I always thought moving to a 5 minute block time would be perfectly fine
 63 2017-03-10T05:20:25  <gwillen> This isn't really a good channel for this kind of discussion -- better in #bitcoin probably.
 64 2017-03-10T05:20:36  <CubicEarth> a slightly higher orphan rate wouldn't hurt anything
 65 2017-03-10T05:23:06  *** gribble has joined #bitcoin-core-dev
 66 2017-03-10T05:28:09  *** CubicEarth has quit IRC
 67 2017-03-10T05:43:54  *** wudayoda has joined #bitcoin-core-dev
 68 2017-03-10T05:47:57  *** wudayoda has quit IRC
 69 2017-03-10T05:52:05  *** juscamarena has joined #bitcoin-core-dev
 70 2017-03-10T05:52:15  *** juscamarena_ has joined #bitcoin-core-dev
 71 2017-03-10T05:56:09  *** juscamarena_ has quit IRC
 72 2017-03-10T05:57:29  *** wudayoda has joined #bitcoin-core-dev
 73 2017-03-10T05:59:17  <wumpus> gmaxwell: I'm fine with that...
 74 2017-03-10T06:01:02  <wumpus> gmaxwell: what I mostly don't like about it is that it requires a rescan, and is very non-selective (zap all unconfirmed)
 75 2017-03-10T06:02:32  <wumpus> gmaxwell: mostly it's useful to troubleshoot issues with wallet bugs and corruption, when a certain transaction behavres strangely.  But a tool to remove a single transaction from the database would work much better for that and be less impactful. Back in the day,though,that was hard to do for some reason
 76 2017-03-10T06:02:46  *** wudayoda has quit IRC
 77 2017-03-10T06:03:29  <wumpus> I agree abandontransaction replaces all end-use servicable reasons to use zap
 78 2017-03-10T06:11:42  <gmaxwell> It might just make sense to rename it and hide it for that reason. If there is a reason to not do that, we should probably enhance abandon further.
 79 2017-03-10T06:12:51  <wumpus> me preference would be to hide it in some dangerous-sounding wallet salvage or editing tool, not have it in the main executable at least or maybe not even in the main distribution
 80 2017-03-10T06:13:15  <wumpus> I mean there's a use for low-level-ish wallet editing, but it certainly shouldn't be easily available
 81 2017-03-10T06:14:51  <gmaxwell> Salvage is also pretty raw... I've said before it should be called "savage (verb) wallet"
 82 2017-03-10T06:15:07  <wumpus> hehe
 83 2017-03-10T06:15:52  <wumpus> salvage has been actually useful to a lot of people though, it tends to be the only thing available if the rest if something is corrupted
 84 2017-03-10T06:16:35  <luke-jr> doesn't zap recover from corrupt bdbs we can't open?
 85 2017-03-10T06:16:46  <wumpus> no, zap doesn't do that
 86 2017-03-10T06:17:01  <wumpus> it assumes that records are simply readable, what it does is remove all transactions
 87 2017-03-10T06:17:15  <wumpus> with a mode to keep metadata (by default) and another to trash it
 88 2017-03-10T06:17:23  <gmaxwell> salvage does. though at least historically it would also miss data in perfectly fine wallet.dats (though I think some of that was due to bugs which have been fixed)
 89 2017-03-10T06:17:37  <luke-jr> ah, mixed them up I guess
 90 2017-03-10T06:17:58  <wumpus> I think those issues have been fixed
 91 2017-03-10T06:18:32  <wumpus> thoug there are still some weird issues with salvagewallet, for example berkeleydb can return an error when salvaging an otherwise ok wallet (but it doesn't lose records anymore IIRC)
 92 2017-03-10T06:18:56  <wumpus> then again this is the kind of thing we're asking for with not updating the backend library for years. BDB should die.
 93 2017-03-10T06:20:44  <gmaxwell> so I did some testing and later BDB now appear to be bidirectionally compatible?!
 94 2017-03-10T06:20:55  <gmaxwell> but I couldn't find any announcement of it.
 95 2017-03-10T06:20:59  <gmaxwell> it just worked.
 96 2017-03-10T06:21:05  <wumpus> it works in some cases
 97 2017-03-10T06:21:31  <wumpus> that's been the case when I tried too. But I don't trust it.
 98 2017-03-10T06:21:54  <gmaxwell> ah, I wasn't aware that it ever worked before.
 99 2017-03-10T06:22:41  <wumpus> I think the backward incompatiblity thing is just a matter of no one ever seriously researching this, and what are the edge cases of it, than a sure thing
100 2017-03-10T06:22:55  <wumpus> it doesn't work in *all* cases that is clear
101 2017-03-10T06:26:32  <wumpus> one thing that is not backwards compatible is the log files; so if there are still log files behind, the old version will error out
102 2017-03-10T06:33:47  *** aalex__ has joined #bitcoin-core-dev
103 2017-03-10T06:36:14  <wumpus> it may well be that a "clean" .dat file, like produced by backupwallet, is always backwards compatible. Though that doesn't help a user that crashes on first run with the new version then tries to go back, the wallet being in intermediate state.
104 2017-03-10T06:37:27  <wumpus> of course it's fairly simple to work around this with a conversion tools and/or making and automatic backup at first run. But it was just never deemed worth the trouble
105 2017-03-10T06:37:38  <wumpus> especially as newer BDBs have license issues
106 2017-03-10T06:37:58  <wumpus> the plan was ,and should still be, to move away from it
107 2017-03-10T06:39:28  *** Victor_sueca has quit IRC
108 2017-03-10T06:40:35  *** Victor_sueca has joined #bitcoin-core-dev
109 2017-03-10T07:06:15  *** gribble has quit IRC
110 2017-03-10T07:07:35  *** aalex__ has quit IRC
111 2017-03-10T07:15:51  *** arubi has joined #bitcoin-core-dev
112 2017-03-10T07:33:05  *** Ylbam has joined #bitcoin-core-dev
113 2017-03-10T07:44:40  *** tonebox has quit IRC
114 2017-03-10T07:54:48  *** BashCo has quit IRC
115 2017-03-10T07:58:32  *** wudayoda has joined #bitcoin-core-dev
116 2017-03-10T08:11:17  *** BashCo has joined #bitcoin-core-dev
117 2017-03-10T08:23:05  *** gribble has joined #bitcoin-core-dev
118 2017-03-10T08:27:23  *** Guyver2 has joined #bitcoin-core-dev
119 2017-03-10T08:28:59  *** JackH has joined #bitcoin-core-dev
120 2017-03-10T08:46:34  *** Guyver2 has quit IRC
121 2017-03-10T08:51:35  *** Alina-malina has quit IRC
122 2017-03-10T08:58:10  <wumpus> baahh.. this is the second time I have trouble with the tests due to a stale qa/cache directory
123 2017-03-10T08:58:49  <wumpus> we should probably nix it when a change in bitcoind is detected
124 2017-03-10T09:00:20  <wumpus> e.g. write that path and sha256sum of the bitcoind used into the cache directory, and if that changes, delete it
125 2017-03-10T09:02:11  <sipa> unsure that's worth it... breaks of the cache direction are very infrequent, i think
126 2017-03-10T09:02:25  <sipa> at least in my experience
127 2017-03-10T09:02:40  <wumpus> any change to the wallet, at least
128 2017-03-10T09:03:39  <wumpus> node0 keeps a wallet from mining the initial blocks, which is usually what causes the problems, if the test expects a certain newly introduced property of transactions
129 2017-03-10T09:04:15  <wumpus> maybe it's infrequent but it is really frustrating and can lead to hours of misdirected search for bugs if it happens
130 2017-03-10T09:04:59  <wumpus> conceptually it's also not *valid* to use an older cache, there is no guarantee that your tests passing is worth anything
131 2017-03-10T09:05:18  <wumpus> the new bitcoind may completely mess up the initial steps and it'd still pass because it is cached
132 2017-03-10T09:06:24  <wumpus> but ok I'll just add a message "Using cached node state in %s" to the test output, maybe that's enough to remind people to delete it if they run into weird issues
133 2017-03-10T09:06:56  <sipa> no, i think you're right
134 2017-03-10T09:07:10  <sipa> we shouldn't be using outdated caches
135 2017-03-10T09:08:57  *** gribble has quit IRC
136 2017-03-10T09:14:09  *** Chris_Stewart_5 has quit IRC
137 2017-03-10T09:18:31  *** Victor_sueca is now known as Victorsueca
138 2017-03-10T09:19:59  <jonasschnelli> wumpus: https://github.com/bitcoin/bitcoin/pull/9294#issuecomment-285612826
139 2017-03-10T09:20:08  <jonasschnelli> This is strange... can it be a caching issue?
140 2017-03-10T09:20:33  <jonasschnelli> I can't see a reason why the dump is different on your machine than on mine / travis.
141 2017-03-10T09:21:25  <jonasschnelli> Maybe you have a chance and check the file "wallet.unencrypted.dump" (maybe pastebin it) when running with --nocleanup
142 2017-03-10T09:23:43  * jonasschnelli starting Ubuntu 16.04
143 2017-03-10T09:28:00  <jonasschnelli> sipa: maybe you have a chance to review #9965. It seems that you ate most familiar with that code part. It also touches the segwit.py test (where I'm not sure if I did the right thing).
144 2017-03-10T09:29:33  <wumpus> jonasschnelli: yes it was a caching issue
145 2017-03-10T09:30:06  <jonasschnelli> Great! *relief*
146 2017-03-10T09:30:10  <wumpus> jonasschnelli: that why I wrote the posts above ^^
147 2017-03-10T09:33:06  <wumpus> I think we should store a hash of the bitcoind executable in the cache directory and delete it when it mismatches
148 2017-03-10T09:37:46  *** riemann has joined #bitcoin-core-dev
149 2017-03-10T09:39:25  <jonasschnelli> wumpus: Ah. Thanks. I haven't read the scrollback. Yes. Good idea with the binary-hash mismatch detection.
150 2017-03-10T10:06:18  *** afk11 has quit IRC
151 2017-03-10T10:14:00  *** kadoban has quit IRC
152 2017-03-10T10:26:16  *** gribble has joined #bitcoin-core-dev
153 2017-03-10T10:26:30  *** kadoban has joined #bitcoin-core-dev
154 2017-03-10T10:30:56  *** MarcoFalke has joined #bitcoin-core-dev
155 2017-03-10T10:31:02  <MarcoFalke> 1
156 2017-03-10T10:41:45  *** AaronvanW has joined #bitcoin-core-dev
157 2017-03-10T10:43:06  *** BirneGetreide_ has joined #bitcoin-core-dev
158 2017-03-10T11:22:12  *** MarcoFalke_ has joined #bitcoin-core-dev
159 2017-03-10T11:22:12  *** MarcoFalke has quit IRC
160 2017-03-10T11:22:12  *** MarcoFalke_ is now known as MarcoFalke
161 2017-03-10T11:24:44  *** Victorsueca has quit IRC
162 2017-03-10T11:25:27  *** Victorsueca has joined #bitcoin-core-dev
163 2017-03-10T11:58:57  *** jtimon has quit IRC
164 2017-03-10T12:10:29  *** BashCo_ has joined #bitcoin-core-dev
165 2017-03-10T12:11:07  *** cryptapus_afk is now known as cryptapus
166 2017-03-10T12:12:31  *** wasi has quit IRC
167 2017-03-10T12:13:56  *** BashCo has quit IRC
168 2017-03-10T12:20:08  <MarcoFalke> no, the salvagewallet issue is not yet fixed
169 2017-03-10T12:21:20  <MarcoFalke> At least I am not aware that anyone fixed it and the tests are still disabled
170 2017-03-10T12:22:01  *** jtimon has joined #bitcoin-core-dev
171 2017-03-10T12:22:55  *** jannes has joined #bitcoin-core-dev
172 2017-03-10T12:28:52  <wumpus> MarcoFalke: from what I remember what causes the test to fail is that salvagewallet can return false when the answer should be true. I don't think any keys are still being lost
173 2017-03-10T12:39:05  *** jtimon has quit IRC
174 2017-03-10T12:39:28  <wumpus> at least that was my experience from last time I tried to reproduce the issue
175 2017-03-10T12:40:19  *** cryptapus has joined #bitcoin-core-dev
176 2017-03-10T12:42:58  <MarcoFalke> ok, going to enable the test on the nightly builds and see what happens
177 2017-03-10T12:58:02  <wumpus> good idea.
178 2017-03-10T12:58:57  <wumpus> what I also found back then is that if you have a wallet that it fails on, it's fully reproducible with that. So it's something in the specific database that triggers it
179 2017-03-10T12:59:48  <Victorsueca> be careful, maybe the code becomes self-aware and starts gathering private keys all around the world and then spends all UTXOs to 1Yoink.... :P
180 2017-03-10T13:18:27  <wumpus> :p
181 2017-03-10T13:27:21  *** paveljanik has quit IRC
182 2017-03-10T13:27:23  *** cryptapus is now known as cryptapus_afk
183 2017-03-10T13:31:18  *** Guyver2 has joined #bitcoin-core-dev
184 2017-03-10T13:32:03  *** Chris_Stewart_5 has joined #bitcoin-core-dev
185 2017-03-10T13:45:00  *** alpalp has joined #bitcoin-core-dev
186 2017-03-10T13:45:00  *** alpalp has joined #bitcoin-core-dev
187 2017-03-10T13:54:01  *** alpalp has quit IRC
188 2017-03-10T13:56:31  <morcos> Re: fee estimation and RBF.  My plan for Core wallet is as follows:
189 2017-03-10T13:56:54  <morcos> Fee estimation allowed for targets of: 2, 4, 6, 12, 24, 48, 144, 1008
190 2017-03-10T13:57:41  <morcos> There will be 2 types of estimate for each target, a conservative estimate (probably not too different from todays estimates, but still a bit less conservative) and an actual estimate
191 2017-03-10T13:58:19  <morcos> I was imagining some kind of interaction between those and RBF, such that if you don't have RBF enabled then it prompts you to use the conservative estimate or something
192 2017-03-10T13:58:58  <morcos> Via RPC, you'll be able to get all kinds of more specific information if you choose
193 2017-03-10T14:00:24  <morcos> Turns out its a bit tricky to do the longer time horizon estimates, b/c if your node hasn't been up for that long, you can't really know..  And if you shut your node down, it doesn't currently record all the txs stuck in your mempool as failing.
194 2017-03-10T14:00:50  <morcos> Fixing these issues is what's holding me up now...   And then I think I may have a performance problem to fix.
195 2017-03-10T14:01:44  <petertodd> morcos: whatever you do, I'd suggest you plan for far more opt-in usage in the future; with ppl spamming the network to apparently raise fees the next round may make whatever estimating thing we come up with unreliable; tx replacement otoh is much harder to game
196 2017-03-10T14:02:25  *** dcousens_ has quit IRC
197 2017-03-10T14:02:52  <morcos> Yeah I think a separate but related project is to have an auto-replace mode...
198 2017-03-10T14:02:56  <petertodd> (e.g. something that ignored opt-in txs entirely my fail if the % of them becomes much higher)
199 2017-03-10T14:03:15  <petertodd> yeah. auto-replace mode is nice - gmaxwell has a neat way to do it where nlocktime is used to ensure replacements can be done prematurely
200 2017-03-10T14:03:30  <morcos> As far as fee estimates dealing with opt-in txs.. I don't think thats much of a problem.. The only issue with that was to be cautious when they were knew that some miners might nto accept them
201 2017-03-10T14:03:49  <morcos> prematurely?  oh you mean sign in advance?
202 2017-03-10T14:04:11  <petertodd> morcos: exactly! he suggested signing n contradictory txs, each with a higher nlocktime and higher fee
203 2017-03-10T14:04:25  *** dcousens has joined #bitcoin-core-dev
204 2017-03-10T14:04:30  <petertodd> morcos: particulkarly relevant for hw wallets like trezor
205 2017-03-10T14:04:39  <morcos> interesting...  and maybe you could have a differential relay too... :)
206 2017-03-10T14:05:05  <petertodd> morcos: yeah, differential relay would be nice, although at least replacement BW just increases the average; peak BW is our actual bottleneck
207 2017-03-10T14:16:20  *** mol has joined #bitcoin-core-dev
208 2017-03-10T14:16:44  *** molz_ has quit IRC
209 2017-03-10T14:20:45  *** Giszmo has joined #bitcoin-core-dev
210 2017-03-10T14:35:40  *** voyager_ has quit IRC
211 2017-03-10T14:44:30  *** aalex__ has joined #bitcoin-core-dev
212 2017-03-10T14:46:49  *** BashCo has joined #bitcoin-core-dev
213 2017-03-10T14:47:38  *** BashCo__ has joined #bitcoin-core-dev
214 2017-03-10T14:48:06  *** voyager_ has joined #bitcoin-core-dev
215 2017-03-10T14:49:28  *** BashCo_ has quit IRC
216 2017-03-10T14:51:04  *** BashCo has quit IRC
217 2017-03-10T15:06:23  *** Magma has quit IRC
218 2017-03-10T15:07:14  *** Magma has joined #bitcoin-core-dev
219 2017-03-10T15:07:35  *** dcousens has quit IRC
220 2017-03-10T15:17:46  *** roidster has joined #bitcoin-core-dev
221 2017-03-10T15:17:47  *** roidster is now known as Guest65095
222 2017-03-10T15:19:58  *** schmidty has joined #bitcoin-core-dev
223 2017-03-10T15:24:13  *** whphhg has quit IRC
224 2017-03-10T15:37:10  *** schmidty has quit IRC
225 2017-03-10T15:38:49  *** schmidty has joined #bitcoin-core-dev
226 2017-03-10T15:47:47  *** whphhg has joined #bitcoin-core-dev
227 2017-03-10T15:59:32  <bsm1175322> I have a need to deliver witness data to SPV clients, which necessitates a one-line change: https://github.com/VidaID/bitcoin/commit/2a3052622596db9b1fe29cd357cfc58a831b050c
228 2017-03-10T15:59:55  *** riemann has quit IRC
229 2017-03-10T16:00:08  <bsm1175322> Would we make this an option or something?
230 2017-03-10T16:00:11  <bsm1175322> *could
231 2017-03-10T16:06:03  *** bityogi has joined #bitcoin-core-dev
232 2017-03-10T16:09:07  <BlueMatt> bsm1175322: you need to do this over p2p?
233 2017-03-10T16:09:14  <bsm1175322> yes
234 2017-03-10T16:09:57  <BlueMatt> bsm1175322: if you can do it over rpc via the gettxoutproof/verifytxoutproof stuff we could probably tweak the format to include proofs of witnesses as well, but p2p....ugh, i think everyone wants to completely remove that code sooner or later, its not good
235 2017-03-10T16:10:11  <BlueMatt> bsm1175322: (need to replace it with bloom filter commitments in blocks or so)
236 2017-03-10T16:10:12  <bsm1175322> BlueMatt: agreed on that.
237 2017-03-10T16:10:26  <bsm1175322> Well giving every random client direct access to RPC is not a good idea for a number of reasons
238 2017-03-10T16:10:44  <bsm1175322> Stage 2 of this project will be to redesign BIP37.  I'm well aware of its flaws.
239 2017-03-10T16:11:06  <bsm1175322> But, for the moment we're just using it, warts and all.
240 2017-03-10T16:11:10  <BlueMatt> bsm1175322: well the other thing we can do is extend the rpc to support it and then your patch will be simpler :)
241 2017-03-10T16:11:31  <BlueMatt> (right now your patch isnt providing any proof of the witnesses, only the tx data, and providing the witness itself just as extra)
242 2017-03-10T16:12:33  <bsm1175322> Oh I hadn't thought of that...there's a Merkle proof that's possible for the witness data too...
243 2017-03-10T16:12:56  <bsm1175322> But...who would care about that?  The client can verify that the witness signature is correct...
244 2017-03-10T16:13:08  <BlueMatt> bsm1175322: yea, would have to provide the merkle path to coinbase + merkle path to the witness in question
245 2017-03-10T16:13:24  <BlueMatt> well assuming they have the transactions being spent, sure
246 2017-03-10T16:14:14  <bsm1175322> Hmmm since you're here in NYC...we should get together soon and have a brainstorming session about a BIP37 replacement...
247 2017-03-10T16:16:22  <BlueMatt> sure, iirc there was some ml post not too long ago on it
248 2017-03-10T16:16:40  <BlueMatt> something about committed bloom filters, I dont recall if they concluded that the filters were too big to be practical or if they were excited though
249 2017-03-10T16:16:44  <BlueMatt> maybe it was a few months ago.....
250 2017-03-10T16:17:13  <bsm1175322> Oh that one...and I calculated the size was unreasonable...
251 2017-03-10T16:17:21  <BlueMatt> awww, damn
252 2017-03-10T16:17:31  <BlueMatt> well, ok, brainstorming it is
253 2017-03-10T16:17:47  <bsm1175322> https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2016-May/012674.html
254 2017-03-10T16:17:58  <BlueMatt> oh wow that was months ago
255 2017-03-10T16:18:06  <bsm1175322> original: https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2016-May/012636.html
256 2017-03-10T16:18:19  *** CubicEarth has joined #bitcoin-core-dev
257 2017-03-10T16:20:56  <bsm1175322> UTXO set commitment in some form are #1 on my wishlist for improving light client security.
258 2017-03-10T16:20:58  <gribble> https://github.com/bitcoin/bitcoin/issues/1 | JSON-RPC support for mobile devices ("ultra-lightweight" clients) · Issue #1 · bitcoin/bitcoin · GitHub
259 2017-03-10T16:22:59  <bitcoin-git> [bitcoin] sdaftuar opened pull request #9970: Improve readability of segwit.py (master...2017-03-segwit-test-improvements) https://github.com/bitcoin/bitcoin/pull/9970
260 2017-03-10T16:23:44  <BlueMatt> bsm1175322: hmm, yea, 12 GB total isnt trivial, though I dont think its insane...I mean its not like you download that whole dataset unless you dont know when your keys were created
261 2017-03-10T16:23:53  <BlueMatt> bsm1175322: does have lots of challenges, though :(
262 2017-03-10T16:24:04  <bsm1175322> There's probably a workable mutation of that idea...
263 2017-03-10T16:24:18  <BlueMatt> bsm1175322: utxo commitments dont help you sync, though, but, yea, are a huge win
264 2017-03-10T16:24:49  <BlueMatt> its unclear what the "right" solution is, I mean scanning the chain for your transactions makes less and less sense every day, especially given that folks are moving towards off-chain txn
265 2017-03-10T16:24:53  <BlueMatt> cant scan for those.....
266 2017-03-10T16:25:41  <bsm1175322> That's a different problem entirely ;-)  SPV-lightning...
267 2017-03-10T16:26:38  <bsm1175322> Another idea I'm a fan of on this topic is andytoshi's PoW skiplists...
268 2017-03-10T16:29:03  *** Alina-malina has joined #bitcoin-core-dev
269 2017-03-10T16:29:09  <BlueMatt> yea, PoW skiplists are cool, though you have to be careful with them depending on your use-case
270 2017-03-10T16:29:24  <BlueMatt> what use-case do you have for them? I mean 80 bytes * 400k blocks isnt that much, still, today?
271 2017-03-10T16:29:43  <bsm1175322> Linear algorithms suck. ;-)
272 2017-03-10T16:30:02  <bsm1175322> Initial sync in SPV mode still takes a non-trivial amount of time on phones.
273 2017-03-10T16:30:19  <bsm1175322> It's just hashing those 400k blocks...
274 2017-03-10T16:32:16  <BlueMatt> i mean we can improve that....we need a "give me all headers without the previous block hash in binary form" thing
275 2017-03-10T16:32:26  <BlueMatt> mabye not p2p, just connect to blockchainheaders.com and get it
276 2017-03-10T16:32:28  <BlueMatt> its 18MB
277 2017-03-10T16:32:36  <BlueMatt> and hashing that cant take that long, no?
278 2017-03-10T16:34:01  <bsm1175322> It does take that long, for whatever reason.  Bcoin only verifies headers at a rate of ~20/s on the phone.
279 2017-03-10T16:35:09  <dgenr8> bsm1175322: did you mean committed bloom filters?
280 2017-03-10T16:35:11  <bsm1175322> @chjj is there any other reason you can think of besides sha256 speed which might be causing spv sync to be slower on the phone?  Obviously it's quite fast in nodejs.
281 2017-03-10T16:35:37  <bsm1175322> dgenr8: yes that's what we were discussing
282 2017-03-10T16:36:25  <BlueMatt> bsm1175322: I mean I assume its also p2p latency, which isnt fun
283 2017-03-10T16:36:28  <dgenr8> BlueMatt: size unreasonable with what fp rate
284 2017-03-10T16:36:31  <BlueMatt> 20/s seems supperrr slow
285 2017-03-10T16:37:22  <bsm1175322> dgenr8: see the referenced post with my calculation, I used fp rate= 1/height ~ 10^-6
286 2017-03-10T16:37:37  <bsm1175322> came out to about 12GB of committed filters.
287 2017-03-10T16:37:46  <bsm1175322> obviously this can be tuned...
288 2017-03-10T16:37:53  <bsm1175322> At the cost of holding blocks you don't need.
289 2017-03-10T16:37:56  <BlueMatt> (though we may want a similar command to remove checkpoints entirely for ibd - much easier to connect to a few peers and ask them for 4MB of headers (~100k blocks) at a time and then get a header chain and ban anyone who tried to spam you
290 2017-03-10T16:38:02  <BlueMatt> instead of the current getheaders stuff
291 2017-03-10T16:38:38  <BlueMatt> bsm1175322: to be fair, you probably want something much higher than 10^-6
292 2017-03-10T16:38:47  <bsm1175322> yes
293 2017-03-10T16:38:50  <BlueMatt> bsm1175322: you definitely want to download some extra blocks
294 2017-03-10T16:39:14  *** kadoban has quit IRC
295 2017-03-10T16:40:12  <bsm1175322> The desirable false positive rate is still (constant)/(height) though, or it's still a linear algorithm...
296 2017-03-10T16:41:14  <BlueMatt> bsm1175322: welcome to the real world, low-cost linear is perfectly ok :P
297 2017-03-10T16:42:11  <BlueMatt> as long as batteries and lte improve faster than your linear increases, at least for as long as you're not using 2nd layer stuff, you should be fine :)
298 2017-03-10T16:42:42  <bsm1175322> Only in the bitcoin world...it makes for a horrible user experience. :-P
299 2017-03-10T16:42:45  <BlueMatt> (well, and data caps...fuck data caps)
300 2017-03-10T16:43:02  <bsm1175322> I'll keep looking for logarithmic solutions :-P
301 2017-03-10T16:43:23  <BlueMatt> 10 seconds in a linear scan isnt all that much different from 10 seconds in a magical logarithmic scan, at least for users :P
302 2017-03-10T16:43:35  <BlueMatt> I see your point, but I'm less worried
303 2017-03-10T16:43:49  <BlueMatt> superlinear, well...lets not do that
304 2017-03-10T16:44:16  <bsm1175322> Well...having been doing dev work on testnet for a few months...where it takes 30 minutes for bcoin to do an initial spv sync...
305 2017-03-10T16:45:00  <BlueMatt> lol, ok, fair point
306 2017-03-10T16:45:14  <BlueMatt> see previous comment about chunking header requests with new p2p messages
307 2017-03-10T16:45:18  <BlueMatt> :)
308 2017-03-10T16:45:41  <BlueMatt> its slow because we've been busy optimizing other things, should be easy to optimize, though
309 2017-03-10T16:47:25  <bsm1175322> Yeah I'm going to have to look into that on bcoin.  For now we're beta testing on regtest so are avoiding the problem.
310 2017-03-10T16:49:21  *** abpa has joined #bitcoin-core-dev
311 2017-03-10T16:53:04  *** aalex__ has quit IRC
312 2017-03-10T16:54:53  *** riemann has joined #bitcoin-core-dev
313 2017-03-10T17:00:12  *** neha has quit IRC
314 2017-03-10T17:01:37  *** neha has joined #bitcoin-core-dev
315 2017-03-10T17:02:17  *** aalex__ has joined #bitcoin-core-dev
316 2017-03-10T17:03:38  *** Alina-malina has quit IRC
317 2017-03-10T17:03:39  *** Alina-malina has joined #bitcoin-core-dev
318 2017-03-10T17:05:46  *** arubi has quit IRC
319 2017-03-10T17:07:15  *** arubi has joined #bitcoin-core-dev
320 2017-03-10T17:11:51  *** nsh has quit IRC
321 2017-03-10T17:11:52  *** ensign- has quit IRC
322 2017-03-10T17:12:42  *** musalbas has quit IRC
323 2017-03-10T17:13:49  *** musalbas has joined #bitcoin-core-dev
324 2017-03-10T17:14:02  *** nsh has joined #bitcoin-core-dev
325 2017-03-10T17:17:03  *** ensign_ has joined #bitcoin-core-dev
326 2017-03-10T17:21:59  *** voyager_ has quit IRC
327 2017-03-10T17:22:49  *** voyager_ has joined #bitcoin-core-dev
328 2017-03-10T17:29:57  *** Alina-malina has quit IRC
329 2017-03-10T17:30:13  *** Alina-malina has joined #bitcoin-core-dev
330 2017-03-10T17:32:15  *** Alina-malina has quit IRC
331 2017-03-10T17:32:16  *** Alina-malina has joined #bitcoin-core-dev
332 2017-03-10T17:40:01  <bsm1175322> BlueMatt: FYI another idea that's floating in my head for improving SPV is whether we could use some form of https://en.wikipedia.org/wiki/Oblivious_transfer
333 2017-03-10T17:44:38  <BlueMatt> bsm1175322: hmm, possibly useful to receive blocks after a high-fp-rate filter commitment or something? Maybe too high overhead, though, gmaxwell might have more to say
334 2017-03-10T17:45:09  <bsm1175322> To be clear, I haven't figured out any algorithm that works and I'm not making a proposal...but I want to find a way...
335 2017-03-10T17:45:52  <BlueMatt> heh, ok
336 2017-03-10T17:46:01  *** CubicEarth has quit IRC
337 2017-03-10T17:46:09  <BlueMatt> the old "that sounds cool, we should use it somewhere" approach :)
338 2017-03-10T17:46:52  <bsm1175322> exactly
339 2017-03-10T18:02:27  *** tripleslash has quit IRC
340 2017-03-10T18:03:44  *** tripleslash has joined #bitcoin-core-dev
341 2017-03-10T18:24:38  <gmaxwell> .... there have been many concrete proposals before. the performance is bad though.
342 2017-03-10T18:32:33  <bsm1175322> Yes...in order to be oblivous about which of N bytes are sent, you have to read/process all N bytes, for *each* request...
343 2017-03-10T18:32:52  <bsm1175322> I'm still hoping there's a way around that observation...
344 2017-03-10T18:35:44  <BlueMatt> uhh, that would be kinda obvious if you didnt....
345 2017-03-10T18:35:58  <BlueMatt> "hey, i dont know what I'm sending you, but its one of A or B, and I never read B from my hdd......"
346 2017-03-10T18:36:36  <bsm1175322> Insert some preprocessing/Merkle tree magic...
347 2017-03-10T18:38:22  <BlueMatt> heh
348 2017-03-10T18:42:45  *** BirneGetreide_ has quit IRC
349 2017-03-10T18:43:25  <gmaxwell> bsm1175322: https://bitcointalk.org/index.php?topic=1762589.0
350 2017-03-10T18:44:43  *** BirneGetreide_ has joined #bitcoin-core-dev
351 2017-03-10T18:45:58  *** BashCo__ has quit IRC
352 2017-03-10T18:46:34  *** BashCo has joined #bitcoin-core-dev
353 2017-03-10T18:46:55  *** adiabat has joined #bitcoin-core-dev
354 2017-03-10T18:51:04  *** BashCo has quit IRC
355 2017-03-10T18:52:57  *** Guest65095 has quit IRC
356 2017-03-10T19:09:44  *** BashCo has joined #bitcoin-core-dev
357 2017-03-10T19:56:47  <bitcoin-git> [bitcoin] MarcoFalke opened pull request #9971: qa: Initialize log in TestManager (master...Mf1703-logFixup) https://github.com/bitcoin/bitcoin/pull/9971
358 2017-03-10T19:57:46  *** jtimon has joined #bitcoin-core-dev
359 2017-03-10T20:14:28  *** shesek has quit IRC
360 2017-03-10T20:18:26  *** BirneGetreide_ has quit IRC
361 2017-03-10T20:19:41  *** str4d has joined #bitcoin-core-dev
362 2017-03-10T20:24:03  <bitcoin-git> [bitcoin] jnewbery opened pull request #9972: Fix extended rpc tests broken by #9768 (master...test_logging_fixups) https://github.com/bitcoin/bitcoin/pull/9972
363 2017-03-10T20:24:56  *** chjj has quit IRC
364 2017-03-10T20:27:48  *** shesek has joined #bitcoin-core-dev
365 2017-03-10T20:30:27  *** Chris_Stewart_5 has quit IRC
366 2017-03-10T20:38:20  *** chjj has joined #bitcoin-core-dev
367 2017-03-10T20:40:12  <gmaxwell> https://www.reddit.com/r/Bitcoin/comments/5yojyp/on_the_recent_bout_of_malleated_transactions/
368 2017-03-10T20:40:41  <midnightmagic> \o/
369 2017-03-10T20:52:48  *** Lauda has quit IRC
370 2017-03-10T20:54:19  *** Lauda has joined #bitcoin-core-dev
371 2017-03-10T21:04:54  *** jtimon has quit IRC
372 2017-03-10T21:05:12  *** CubicEarth has joined #bitcoin-core-dev
373 2017-03-10T21:08:22  *** Chris_Stewart_5 has joined #bitcoin-core-dev
374 2017-03-10T21:08:42  *** roidster has joined #bitcoin-core-dev
375 2017-03-10T21:08:45  *** roidster is now known as Guest95471
376 2017-03-10T21:15:24  *** jnewbery has quit IRC
377 2017-03-10T21:21:49  *** Guyver2 has quit IRC
378 2017-03-10T21:53:04  <bitcoin-git> [bitcoin] theuni opened pull request #9973: depends: fix zlib build on osx (master...fix-zlib-osx) https://github.com/bitcoin/bitcoin/pull/9973
379 2017-03-10T21:53:27  *** tripleslash has quit IRC
380 2017-03-10T21:55:50  *** tripleslash has joined #bitcoin-core-dev
381 2017-03-10T21:57:00  <bitcoin-git> [bitcoin] ryanofsky opened pull request #9974: Add basic Qt wallet test (master...pr/qt-test) https://github.com/bitcoin/bitcoin/pull/9974
382 2017-03-10T22:00:30  *** tripleslash has quit IRC
383 2017-03-10T22:01:28  *** str4d has quit IRC
384 2017-03-10T22:01:44  *** fanquake has joined #bitcoin-core-dev
385 2017-03-10T22:08:28  <bitcoin-git> [bitcoin] MarcoFalke pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/8910b4717e5b...21833f9456f6
386 2017-03-10T22:08:28  <bitcoin-git> bitcoin/master d055bd6 John Newbery: Fix extended rpc tests broken by 8910b4717e5bb946ee6988f7fe9fd461f53a5935
387 2017-03-10T22:08:29  <bitcoin-git> bitcoin/master 21833f9 MarcoFalke: Merge #9972: Fix extended rpc tests broken by #9768...
388 2017-03-10T22:08:56  <bitcoin-git> [bitcoin] MarcoFalke closed pull request #9972: Fix extended rpc tests broken by #9768 (master...test_logging_fixups) https://github.com/bitcoin/bitcoin/pull/9972
389 2017-03-10T22:10:59  *** fanquake has quit IRC
390 2017-03-10T22:28:32  *** harrymm1 has quit IRC
391 2017-03-10T22:35:25  *** tripleslash has joined #bitcoin-core-dev
392 2017-03-10T22:39:26  *** tripleslash has quit IRC
393 2017-03-10T22:43:50  *** harrymm has joined #bitcoin-core-dev
394 2017-03-10T22:49:53  *** tripleslash has joined #bitcoin-core-dev
395 2017-03-10T22:52:04  *** str4d has joined #bitcoin-core-dev
396 2017-03-10T22:54:22  *** tripleslash has quit IRC
397 2017-03-10T23:01:29  *** CubicEarth has quit IRC
398 2017-03-10T23:04:56  *** riemann has quit IRC
399 2017-03-10T23:09:46  *** tripleslash has joined #bitcoin-core-dev
400 2017-03-10T23:23:34  <bitcoin-git> [bitcoin] MarcoFalke closed pull request #9971: qa: Initialize log in TestManager (master...Mf1703-logFixup) https://github.com/bitcoin/bitcoin/pull/9971
401 2017-03-10T23:27:35  *** Guest95471 has quit IRC
402 2017-03-10T23:30:09  *** Telmo has joined #bitcoin-core-dev
403 2017-03-10T23:30:31  *** Telmo has quit IRC
404 2017-03-10T23:32:09  *** Telmo has joined #bitcoin-core-dev
405 2017-03-10T23:32:23  <Telmo> Ola
406 2017-03-10T23:33:10  *** Telmo has quit IRC
407 2017-03-10T23:33:57  *** str4d has quit IRC
408 2017-03-10T23:45:10  *** CubicEarth has joined #bitcoin-core-dev
409 2017-03-10T23:52:24  *** aalex__ has quit IRC