1 2016-07-01T00:00:19  *** ybit has quit IRC
  2 2016-07-01T00:01:32  *** ybit has joined #bitcoin-core-dev
  3 2016-07-01T00:02:43  *** frankenmint has joined #bitcoin-core-dev
  4 2016-07-01T00:03:30  *** cryptapus_afk is now known as cryptapus
  5 2016-07-01T00:05:47  *** molz has joined #bitcoin-core-dev
  6 2016-07-01T00:06:07  *** Retric has joined #bitcoin-core-dev
  7 2016-07-01T00:08:34  *** moli has quit IRC
  8 2016-07-01T00:18:42  <sdaftuar> oddly, travis doesn't appear to have run #8295
  9 2016-07-01T00:18:46  *** cryptapus is now known as cryptapus_afk
 10 2016-07-01T00:38:16  *** frankenmint has quit IRC
 11 2016-07-01T00:43:59  *** TheFactory7 has quit IRC
 12 2016-07-01T00:51:44  *** zooko has joined #bitcoin-core-dev
 13 2016-07-01T00:55:54  *** fengling has joined #bitcoin-core-dev
 14 2016-07-01T00:58:36  *** pedrobranco has joined #bitcoin-core-dev
 15 2016-07-01T01:02:38  *** PRab has quit IRC
 16 2016-07-01T01:03:24  *** pedrobranco has quit IRC
 17 2016-07-01T01:12:35  *** xiangfu has joined #bitcoin-core-dev
 18 2016-07-01T01:32:23  *** face has joined #bitcoin-core-dev
 19 2016-07-01T01:35:46  *** Ylbam has quit IRC
 20 2016-07-01T01:36:59  *** BCBot has joined #bitcoin-core-dev
 21 2016-07-01T01:36:59  *** bsm1175321 has joined #bitcoin-core-dev
 22 2016-07-01T01:36:59  *** Taek has joined #bitcoin-core-dev
 23 2016-07-01T01:38:33  <luke-jr> fwiw, there appears to be no changes affecting bitcoin-cli between 0.12.0 and 0.12.1
 24 2016-07-01T01:40:19  *** belcher has quit IRC
 25 2016-07-01T01:53:04  *** cryptapus_afk is now known as cryptapus
 26 2016-07-01T01:55:53  *** cryptapus is now known as cryptapus_afk
 27 2016-07-01T02:08:38  *** wangchun has quit IRC
 28 2016-07-01T02:09:14  *** wangchun has joined #bitcoin-core-dev
 29 2016-07-01T02:15:28  *** Chris_Stewart_5 has quit IRC
 30 2016-07-01T02:16:24  *** Chris_Stewart_5 has joined #bitcoin-core-dev
 31 2016-07-01T02:20:20  *** adiabat has joined #bitcoin-core-dev
 32 2016-07-01T02:38:37  *** Chris_Stewart_5 has quit IRC
 33 2016-07-01T03:30:59  *** adiabat has left #bitcoin-core-dev
 34 2016-07-01T03:49:34  *** pedrobranco has joined #bitcoin-core-dev
 35 2016-07-01T03:54:21  *** pedrobranco has quit IRC
 36 2016-07-01T03:57:12  *** fengling has quit IRC
 37 2016-07-01T04:14:34  *** afk11 has quit IRC
 38 2016-07-01T04:19:31  *** afk11 has joined #bitcoin-core-dev
 39 2016-07-01T04:19:31  *** afk11 has quit IRC
 40 2016-07-01T04:19:31  *** afk11 has joined #bitcoin-core-dev
 41 2016-07-01T04:52:23  *** Guest27612 has joined #bitcoin-core-dev
 42 2016-07-01T04:52:38  *** Guest27612 is now known as roidster
 43 2016-07-01T04:59:27  *** kadoban has quit IRC
 44 2016-07-01T04:59:42  <gmaxwell> Can anyone think of why we'd get a "CommitTransaction(): Error: Transaction not valid"  on a new transaction being created that paid twice the nodes minrelay fee, ... and then managed to broadcast when the rebroadcast ran?
 45 2016-07-01T04:59:47  <gmaxwell> it looks like it was spending an unconfirmed input.
 46 2016-07-01T05:10:31  <gmaxwell> okay I think the issue was he managed to make a 25 deep unconfirmed chain, and got the last txn in it rejected.
 47 2016-07-01T05:10:38  <gmaxwell> Doesn't look like the wallet handles that well.
 48 2016-07-01T05:13:00  <gmaxwell> I think we should avoid using coins that are at the unconfirmed limit maximum unless not otherwise possible.
 49 2016-07-01T05:13:16  <gmaxwell> Also ooRBF to sendmany would have eliminated 25 transactions here.
 50 2016-07-01T05:20:41  *** zooko has quit IRC
 51 2016-07-01T05:26:34  *** Guyver2 has joined #bitcoin-core-dev
 52 2016-07-01T05:41:53  *** Ylbam has joined #bitcoin-core-dev
 53 2016-07-01T06:17:27  *** Guyver2 has quit IRC
 54 2016-07-01T06:38:18  *** Inaltoas1nistra has joined #bitcoin-core-dev
 55 2016-07-01T06:40:25  *** Inaltoasinistra has quit IRC
 56 2016-07-01T06:48:12  *** roidster has quit IRC
 57 2016-07-01T07:10:05  *** Ginnarr has joined #bitcoin-core-dev
 58 2016-07-01T07:11:51  *** owowo has quit IRC
 59 2016-07-01T07:13:43  *** cjcj has joined #bitcoin-core-dev
 60 2016-07-01T07:52:44  <GitHub68> [bitcoin] laanwj pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/6a87eb0e4b47...da50997a3ee7
 61 2016-07-01T07:52:45  <GitHub68> bitcoin/master 0ce8e99 Wladimir J. van der Laan: windows: Add testnet link to installer
 62 2016-07-01T07:52:45  <GitHub68> bitcoin/master 975a41d Wladimir J. van der Laan: windows: Add testnet icon for testnet link...
 63 2016-07-01T07:52:46  <GitHub68> bitcoin/master da50997 Wladimir J. van der Laan: Merge #8285: windows: Add testnet link to installer...
 64 2016-07-01T07:52:52  <GitHub129> [bitcoin] laanwj closed pull request #8285: windows: Add testnet link to installer (master...2016_06_testnet_link_windows) https://github.com/bitcoin/bitcoin/pull/8285
 65 2016-07-01T07:55:41  *** MarcoFalke has joined #bitcoin-core-dev
 66 2016-07-01T08:10:00  <instagibbs> gmaxwell, yes the maximum ancestor/descendant stuff means that the transaction will be marked invalid and the funds deducted from your wallet until rescan :/
 67 2016-07-01T08:10:08  <instagibbs> err reindex or something
 68 2016-07-01T08:11:32  <gmaxwell> instagibbs: in this case the transaction when through after the parents confirmed.
 69 2016-07-01T08:11:47  <gmaxwell> the wallet rebroadcast accepted it the next time around and tada.
 70 2016-07-01T08:12:09  <gmaxwell> probably the coinselection process, when it considers 0conf inputs should only consider thoes with a low enough depth.
 71 2016-07-01T08:12:40  <gmaxwell> (perhaps two below the maximum to leave room for CPFP), and only if it's unable to meet that should it fall back to considering all inputs.
 72 2016-07-01T08:14:39  <instagibbs> gmaxwell, oh so even in failure it rebroadcasts?
 73 2016-07-01T08:14:59  <gmaxwell> yea, it saves it in the wallet before it tries to mempool it.
 74 2016-07-01T08:15:07  <gmaxwell> and since its saved, it keeps trying to rebroadcast.
 75 2016-07-01T08:15:08  <wumpus> as long as the transaction is in the wallet it rebroadcasts
 76 2016-07-01T08:15:49  <gmaxwell> I dunno if the depth of an unconfirmed coin is easily discernable though from the wallet.
 77 2016-07-01T08:20:31  <wumpus> not at is, it would need another (ugly) mempool dependency
 78 2016-07-01T08:20:34  *** Giszmo has quit IRC
 79 2016-07-01T08:21:05  <gmaxwell> bleh
 80 2016-07-01T08:22:48  <wumpus> then again we already do an InMempool() check in IsTrusted, which determines whether the outputs of an transaction are considered spendable, so...
 81 2016-07-01T08:23:16  <wumpus> if looking up the depth in the mempool is relatiely cheap that could be added too
 82 2016-07-01T08:24:17  <wumpus> but yes I agree with the 'bleh' sentiment
 83 2016-07-01T08:24:45  <wumpus> ideally mempool implementation details shouldn't matter to the wallet
 84 2016-07-01T08:25:03  <wumpus> "if it cannot be submitted yet due to depth, try again later" seems a better approach
 85 2016-07-01T08:25:28  <wumpus> the wallet already tries to avoid generating chains of unconfirmed
 86 2016-07-01T08:25:33  <wumpus> so if it does, it really needs to
 87 2016-07-01T08:26:35  <gmaxwell> it doesn't really: so it tries to avoid spending unconfirmed coins, but if it must it treats short and long chains the same.
 88 2016-07-01T08:26:52  <gmaxwell> So in the example with the user I was supposting today, they had plenty of unconfirmed coins that were only 1 deep.
 89 2016-07-01T08:27:16  <wumpus> yes I don't mean to imply that it looks at the mempool depth, it obviously doesn't
 90 2016-07-01T08:27:19  <gmaxwell> and one chain of 25.
 91 2016-07-01T08:27:58  <wumpus> yes that would make sense - count *unconfirmed* depth
 92 2016-07-01T08:28:08  <wumpus> then prefer coins that are shallow
 93 2016-07-01T08:28:17  <gmaxwell> it ended up that way because they had multiple unconfirmed few-bitcoin outputs, and then a 1 bitcoin output and were making many not huge payments... so it decided to keep reusing the change because it considered them all equal.
 94 2016-07-01T08:28:19  <wumpus> that wouldn't require asking the mempool at all
 95 2016-07-01T08:28:49  <gmaxwell> yea, don't' really care about the mempool, the ismine tracing could count the depth until confirmed I guess.
 96 2016-07-01T08:29:13  <wumpus> coins with a smaller unconfirmed depth could get some advantage in selection
 97 2016-07-01T08:29:40  <wumpus> another thing in the long list of things that could be better in coin selection
 98 2016-07-01T08:31:54  <gmaxwell> well the way to do the advantage is the same way it prefers confirmed coins.
 99 2016-07-01T08:32:16  <gmaxwell> try the selection first with any too-deep coins excluded, and only if it fails attempt without that restriction.
100 2016-07-01T08:32:37  <wumpus> that's a possiblity, yes
101 2016-07-01T08:33:20  *** jtimon has joined #bitcoin-core-dev
102 2016-07-01T08:36:18  <wumpus> though at some point the 'exclude these coins and do the selection again' list would become very long, and is less suited for non-boolean properties like input size (https://github.com/bitcoin/bitcoin/issues/7664)
103 2016-07-01T08:36:38  <wumpus> but sure, this could be bolted on too :)
104 2016-07-01T08:37:32  <gmaxwell> well, it fits here since the failure results in an (temporarily) invalid transaction. ... just leaving them out is the right call for max depth, at least.
105 2016-07-01T08:37:33  <wumpus> I think at some point it would be nice to have a per-coin scoring system, then make the coin selection use that
106 2016-07-01T08:38:08  <gmaxwell> yes, for some things, but a score for this should be infinite unless there is no other choice... since it guarentees the txn will not relay until its ancestors confirm some. :)
107 2016-07-01T08:38:30  <wumpus> but you make assumptions about other people's mempool depth
108 2016-07-01T08:38:41  <wumpus> I think a better approach would just be to *prefer* shallower transactions
109 2016-07-01T08:38:50  <wumpus> without an absolute theshold
110 2016-07-01T08:39:21  <gmaxwell> They're not exclusive, prefer shorter but absolute threshold at the point where you won't relay the thing yourself anymore.
111 2016-07-01T08:39:53  <gmaxwell> this usecase would actually have been better off with some kind of replacement, as they could have saved a good 20 transactions.
112 2016-07-01T08:39:55  <wumpus> and if it does manage to generates a temporarily invalid transaction, handle that more friendly, e.g. a warning instead of an error
113 2016-07-01T08:41:30  <wumpus> having the wallet have a hard dependency on a property of the mempool seems really brittle
114 2016-07-01T08:42:12  <wumpus> and it's not a solution that wallets that don't have such a close coupling to a node could use
115 2016-07-01T08:42:13  <gmaxwell> I don't think this is really a mempool dependency.
116 2016-07-01T08:42:35  *** fengling has joined #bitcoin-core-dev
117 2016-07-01T08:42:43  <wumpus> in principle it's a matter of optimizing the speed at which the transaction can be submitted, longer chains result in slower confirmation times
118 2016-07-01T08:42:54  <wumpus> very long chains even in temporary rejection
119 2016-07-01T08:42:59  <gmaxwell> If it's spending here the transaction is ismine, which means all unconfirmed ancestors are wallet txn.
120 2016-07-01T08:43:10  <gmaxwell> So it just needs to know the longest depth.
121 2016-07-01T08:43:13  <wumpus> but long chains, even if they are accepted, are not a good thing in itself
122 2016-07-01T08:44:33  <gmaxwell> (er not IsMine, but IsFromMe)
123 2016-07-01T08:44:45  *** jannes has joined #bitcoin-core-dev
124 2016-07-01T08:46:04  <gmaxwell> wumpus: they're not that bad, esp with ancestor feerate mining; and relay being not totally broken for them in 0.13.
125 2016-07-01T08:46:28  <wumpus> but I hope you agree that they are worse than shallow transaction chains
126 2016-07-01T08:46:46  *** fengling has quit IRC
127 2016-07-01T08:46:48  <wumpus> 'not that bad' yes there are always much worse things :)
128 2016-07-01T08:46:51  <gmaxwell> all things equal... but compared to a choice that is less private or pays more fees?
129 2016-07-01T08:47:16  <wumpus> well the individual scoring weights would have to be determined, yes, or even depend on some setting
130 2016-07-01T08:47:51  <wumpus> paying more fees may be ok if it means the transaction has a larger chance of being picked
131 2016-07-01T08:47:53  <gmaxwell> because we never split large change output, a lot of usage patterns result in no real choice in any case.
132 2016-07-01T08:48:36  <wumpus> less private, well long transaction chains are a great way to say 'hey this is a wallet sending out transactions serially'
133 2016-07-01T08:49:58  <gmaxwell> sure, but often better to just respend change than join a dozen otherwise unrelated inputs.
134 2016-07-01T08:51:14  <wumpus> if it requires joining them, yes I agree
135 2016-07-01T08:52:55  * wumpus wishes someone would do a serious study about what wallet behavior would make sense
136 2016-07-01T08:53:04  *** AaronvanW has quit IRC
137 2016-07-01T08:53:12  *** fengling has joined #bitcoin-core-dev
138 2016-07-01T08:55:45  <gmaxwell> Would we do anything with it?... a while back someone proposed a change (to remove extranious inputs), I suggested that it might result in wallets grinding down coins into small amouts more often. He made a simulator that showed it would, then we took the change. Then later users showed up complaining about the wallet grinding down inputs when they didn't use to...
139 2016-07-01T08:56:00  *** AaronvanW has joined #bitcoin-core-dev
140 2016-07-01T08:56:39  <wumpus> well the problem is that we're too busy running from issue to issue to look at a higher level
141 2016-07-01T08:56:45  <wumpus> well at least I am
142 2016-07-01T08:56:53  <gmaxwell> right. I agree with that.
143 2016-07-01T08:57:12  <gmaxwell> just collectively we are.
144 2016-07-01T08:57:29  <wumpus> so now long chains are an issue, long chains are fixed by adding yet another special case, but without considering impact on other things
145 2016-07-01T08:57:47  <gmaxwell> You're right that someone looking at it more holistically would be good, part of the problem in that issue was that even there it was only asking a very narrow question.
146 2016-07-01T08:57:53  <wumpus> at some point I just worry all those hacks make things worse, instead of picking some simple algorithm that does fine in most cases
147 2016-07-01T08:58:13  <gmaxwell> I don't really think there is much to consider on the subject of "avoid going over your own maximum chain depth if at all possible".
148 2016-07-01T08:58:16  <wumpus> but I don't know, I don't have the ovierview
149 2016-07-01T08:58:36  <gmaxwell> since producing a txn even you won't relay when you could have avoided it seems clearly wrong.
150 2016-07-01T08:58:50  <phantomcircuit> gmaxwell: the wallet should probably have all of the unconfirmed depdencies of a transaction
151 2016-07-01T08:59:07  <wumpus> so dropping of extraneous inputs was bad?
152 2016-07-01T08:59:08  <phantomcircuit> and then remove the dependencies after some high number of confirms which aren't relevant
153 2016-07-01T09:00:06  <gmaxwell> phantomcircuit: in these cases all the unconfirmed dependencies will be wallet transactions-- IsFromMe otherwise it won't spend the coin!
154 2016-07-01T09:00:07  <wumpus> normally you'd say that choosing a minimum set that covers the value to be spent would be optimal
155 2016-07-01T09:00:09  <phantomcircuit> but yeah you're right it has to pass IsMine which means the wallet already has all the info you need to calculate depth with just coins view
156 2016-07-01T09:00:38  <wumpus> but yes, I'm sure there are lots of other contraints and scores that should be taken in to account
157 2016-07-01T09:00:38  <phantomcircuit> reading top down so my comments might be already worked out :P
158 2016-07-01T09:00:40  <gmaxwell> wumpus: the result ends up being the smallest possible change. so it breaks your wallet into lots of tiny little coins.
159 2016-07-01T09:00:41  <wumpus> which was my point before
160 2016-07-01T09:00:48  *** molly has joined #bitcoin-core-dev
161 2016-07-01T09:01:43  <wumpus> so why wasn't that reverted?
162 2016-07-01T09:02:03  <gmaxwell> I don't know/understand why it wasn't.
163 2016-07-01T09:02:46  <gmaxwell> I think when it went in there was some expectation that further improvements would come, and they didn't. Then it was released. Then people showed up and complaints (opening an issue) and we figured out that the behavior change was due to removing extranious inputs.. and then?  I don't know
164 2016-07-01T09:02:52  <wumpus> we just have too much on our plate
165 2016-07-01T09:02:55  <gmaxwell> then something else caught fire.
166 2016-07-01T09:03:11  <wumpus> sometimes I really want to run around screaming with my hands on my head
167 2016-07-01T09:03:43  <phantomcircuit> wumpus: that sounds like fun
168 2016-07-01T09:03:48  <wumpus> it's not possible to handle this all anymore
169 2016-07-01T09:04:17  *** molz has quit IRC
170 2016-07-01T09:04:29  <wumpus> we really need someone that focuses on wallet improvements
171 2016-07-01T09:04:31  <gmaxwell> (privacy wants you to spend all payments to a single address at once, so that you don't get a rolling linkage that eventually cross taints every address in your wallet--- so maybe just attempting to do this would counteract most of the grinding. thats the kind of thing you want tne step back and overview to answer)
172 2016-07-01T09:05:31  *** Retric has quit IRC
173 2016-07-01T09:06:01  <wumpus> someone that has an overview of what the heck is happening in the wallet
174 2016-07-01T09:06:01  <gmaxwell> There are a number of positive wallet things going on, at least. But there is a lot of time spent twiddling things in the weeds rather than setting up prioties and identifying larger scale issues.
175 2016-07-01T09:06:12  <wumpus> but how to sort 'bad' improvements from good ones?
176 2016-07-01T09:06:22  <wumpus> I don't know anymore
177 2016-07-01T09:07:40  <phantomcircuit> i've been trying to improve the wallet
178 2016-07-01T09:07:55  <wumpus> yes, thanks for that
179 2016-07-01T09:08:14  <phantomcircuit> unfortunately yeah it's a lot of running around in the weeds trying to fix things up... to make it easier to do bigger things
180 2016-07-01T09:08:30  <wumpus> all in all the utxo/coin based approach does cause a lot of non-trivial difficulties, both at the node (utxo set sprawl) level as in the wallet
181 2016-07-01T09:08:38  <gmaxwell> in any case, I didn't bring any of this up here to complain-- I brought it up because initially I couldn't figure out the cause, and then updated when I did.
182 2016-07-01T09:09:31  <gmaxwell> it avoids a lot of replay problems however, which avoids other kinds of sprawl
183 2016-07-01T09:09:49  <wumpus> no, I'm not trying to complain either, but sometimes I just don't know anymroe
184 2016-07-01T09:10:26  *** fengling has quit IRC
185 2016-07-01T09:10:30  *** paveljanik has joined #bitcoin-core-dev
186 2016-07-01T09:10:30  *** paveljanik has joined #bitcoin-core-dev
187 2016-07-01T09:10:51  <gmaxwell> well, from another perspective-- we have no deadline. These same issues existed in 2011 (actually, much worse), but it didn't matter because hardly anyone used the system. :)
188 2016-07-01T09:11:15  <wumpus> but that's part of what worries me, yes these same issues existed in 2011
189 2016-07-01T09:11:58  <wumpus> no, no deadline, but e.g. the utxo growth is worrying
190 2016-07-01T09:12:02  <wumpus> we're running against (soft) walls
191 2016-07-01T09:12:10  <gmaxwell> yes, I'm concerned about utxo growth.
192 2016-07-01T09:12:36  <gmaxwell> I happened to chart data from all those reindexing tests I just did, and the time to verify a block is increasing (for the same amount of txn data).
193 2016-07-01T09:13:15  <wumpus> and all the while joe user is complaining about lack of *scaling*, while the system is already seemingly bursting at its seems
194 2016-07-01T09:13:24  <gmaxwell> I was theorizing that this was from polylog behavior in the database and worrying, but phantomcircuit gave an alternative argument that the reduction in spammy transactions relative to non-spammy ones may be resulting in lower cache hitrates.
195 2016-07-01T09:13:51  <gmaxwell> so I'm going to try to figure out if thats the case.
196 2016-07-01T09:14:38  <gmaxwell> (by increasing, I mean increasing enough that it was clearly visible on a last-8064 block plot)
197 2016-07-01T09:15:15  <wumpus> "the time to verify a block is increasing (for the same amount of txn data" yes, indeed, I also intended to plot time per block, but that much was clear just from looking at timestamps :(
198 2016-07-01T09:16:23  <gmaxwell> well the difference between 1MB and smaller blocks was expected, but the increase still existed when looking at only large blocks.
199 2016-07-01T09:16:34  <wumpus> yes
200 2016-07-01T09:16:49  <wumpus> recent blocks verify slower
201 2016-07-01T09:17:03  <wumpus> even of the same size
202 2016-07-01T09:17:20  <wumpus> so *let alone* what bigger blocks would do
203 2016-07-01T09:17:53  <gmaxwell> At least segwit doesn't increase the worst case amount of utxo growth or accessing...
204 2016-07-01T09:18:03  *** fengling has joined #bitcoin-core-dev
205 2016-07-01T09:19:01  *** xiangfu has quit IRC
206 2016-07-01T09:30:35  *** Ginnarr has quit IRC
207 2016-07-01T09:38:51  *** owowo has joined #bitcoin-core-dev
208 2016-07-01T09:38:54  *** owowo has joined #bitcoin-core-dev
209 2016-07-01T09:42:00  *** pedrobranco has joined #bitcoin-core-dev
210 2016-07-01T09:56:48  <pedrobranco> sipa: hi, when you have the time can you help on the doubts in https://github.com/bitcoin/bitcoin/pull/7551? Thanks :)
211 2016-07-01T10:04:14  <sipa> pedrobranco: i answered quickly
212 2016-07-01T10:04:57  <sipa> pedrobranco: feel free to disagree by the way... if my view is not obvious here, perhaps the semantics should be different and more clear
213 2016-07-01T10:09:06  <pedrobranco> sipa: thanks.
214 2016-07-01T10:35:10  *** fengling has quit IRC
215 2016-07-01T10:37:59  <wumpus> wow, that wallet drop-unnecessary-inputs-from-best-subset change had been open from sep 2014
216 2016-07-01T10:38:07  <wumpus> https://github.com/bitcoin/bitcoin/pull/4906
217 2016-07-01T10:40:46  <wumpus> it has not yet made it into any release
218 2016-07-01T10:41:04  <MarcoFalke> it has
219 2016-07-01T10:41:14  <MarcoFalke> it was backported to .12
220 2016-07-01T10:41:19  <wumpus> fuck
221 2016-07-01T10:41:34  <wumpus> yes I see: 96e8d120336cf4312cd5f42ba2f9aff17d4ad414
222 2016-07-01T10:42:04  <wumpus> that was my own stupid idea probably
223 2016-07-01T10:43:37  <MarcoFalke> Up to now I have not seen proof that this made things considerably worse
224 2016-07-01T10:44:06  <MarcoFalke> Someone should come up with a framwork that mimics common wallet use cases
225 2016-07-01T10:44:16  <wumpus> ok
226 2016-07-01T10:44:25  <MarcoFalke> e.g 'exchange': something with a huge in-out volume/count
227 2016-07-01T10:44:55  <MarcoFalke> then a "normal user": always waits for confirmations, low volume
228 2016-07-01T10:45:20  <wumpus> right, that would make sense, for a project taking the wallet part seriously
229 2016-07-01T10:45:45  <MarcoFalke> and then different behaviors like 'always send small coins' (pay for coffee each day)
230 2016-07-01T10:45:57  *** jtimon has quit IRC
231 2016-07-01T10:46:11  <MarcoFalke> and 'receive large coins' (get payed in btc)
232 2016-07-01T10:46:13  <MarcoFalke> etc
233 2016-07-01T10:47:57  <wumpus> right.
234 2016-07-01T10:49:19  <wumpus> gmaxwell mentioned above that he did see some reports of worse behavior after the change
235 2016-07-01T10:49:29  <wumpus> now it's not clear whether to revert it or not
236 2016-07-01T10:50:13  <gmaxwell> We recieved an issue with a business with a large wallet reporting specifically the expected behavior there.
237 2016-07-01T10:50:45  <wumpus> why don't businesses with large wallets never help with the wallet development?
238 2016-07-01T10:51:03  <wumpus> it seems it would be in their best interest
239 2016-07-01T10:51:10  <gmaxwell> because there are very few left, most are using fully hosted things that have custom software.
240 2016-07-01T10:51:14  <wumpus> but I don't think this was even reported on github
241 2016-07-01T10:51:19  <gmaxwell> I thought it was.
242 2016-07-01T10:51:23  <wumpus> it's just frustrating
243 2016-07-01T10:51:26  <MarcoFalke> it was
244 2016-07-01T10:51:30  <wumpus> oh?
245 2016-07-01T10:51:38  <gmaxwell> And utxo growth coincided with it. This was previously discussed in meetings IIRC and then ... I'm not sure what happened.
246 2016-07-01T10:52:13  <wumpus> that's not important. What is important is what to do now
247 2016-07-01T10:52:16  <wumpus> revert it?
248 2016-07-01T10:52:38  <MarcoFalke> #7664 #7657
249 2016-07-01T10:53:02  <gmaxwell> maybe it was dropped for good reason. I just can't recall.
250 2016-07-01T10:54:01  <gmaxwell> At the end of the day, the pruning itself was correcting a bug. The potential issue is that the bug was covering up that the non-bugged behavior is bad (and the simulations showed that it, as I said, would grind down wallets into lots of tiny coins)
251 2016-07-01T10:54:09  <MarcoFalke> Imo it shouldn't matter if we revert it or not. I haven't yet seen simulations which prove that either would place an advatage/disatvantage
252 2016-07-01T10:54:38  <gmaxwell> MarcoFalke: IIRC there were simulations posted that showed that the behavior caused far more utxo.
253 2016-07-01T10:55:08  <MarcoFalke> No one verified that the simulations were implemented according to the current coin selection code
254 2016-07-01T10:55:18  <MarcoFalke> I have seen single satoshi outputs in those simulations
255 2016-07-01T10:55:25  <MarcoFalke> which does not happen with our code
256 2016-07-01T10:55:42  <MarcoFalke> Also the author found a bug in the implementation about two weeks ago or so
257 2016-07-01T10:55:44  <gmaxwell> The result is also intutive to me. It results in making the smallest possible outputs. This is pessimal when you consider that avoiding utxo size wants, for a given users balance, the largest value outputs.
258 2016-07-01T10:56:40  <MarcoFalke> Ideally, we'd implement the wallet benchmark framework in cpp and have it just use our wallet code?
259 2016-07-01T10:57:00  <wumpus> if it would be modular enough for that :)
260 2016-07-01T10:57:44  <wumpus> possibly separate the coin selection logic out to a separate file, which could also be used by the simulation framework
261 2016-07-01T10:58:03  <wumpus> it could use an abstraction of a list of coins with some properties instead of a wallet as input
262 2016-07-01T10:58:41  <wumpus> it would make some things a lot easier, like trying out what the algo does in certain circumstances, without actually having to fake a wallet
263 2016-07-01T10:59:46  <gmaxwell> An even better behavior would be to add all other inputs to the same addresses being spent, up to some limit to prevent very large transactions, to sweep them up. This is a privacy improving strategy, as well as a fee minimizing strategy under an assumption that fees will increase in the future.   My guess is that I didn't NO DONT the pruning change because in and of itself it was right, and the
264 2016-07-01T10:59:52  <gmaxwell>  grinding should be fixed by something like this.
265 2016-07-01T11:00:11  <gmaxwell> but it went off my radar, also changes to coinselection are a pain because a bunch of th tests freeking hard code the behavior.
266 2016-07-01T11:00:40  <wumpus> if that was the only thing making changes to coin selection a pain :)
267 2016-07-01T11:01:18  <wumpus> it's just almost impossible to agree what is better behavior, which is what talled #4906 for so long and resulted in it eventually being merged, probably wrongly
268 2016-07-01T11:02:06  <wumpus> so a) it should be better behavior b) it needs to be implemented correctly, and indeed, we have no good tests or simulation framework
269 2016-07-01T11:04:32  <wumpus> sweeping up as many as possible inputs to the same address would make sense
270 2016-07-01T11:04:48  <wumpus> there is no privacy benefit from having more spends to one address
271 2016-07-01T11:04:56  <wumpus> then again it assumes bad-case behavior from the user, reusing addresses
272 2016-07-01T11:05:29  <gmaxwell> I don't want to be hard on you, but Xekyo ran simulations which showed 25% to 400% increase in the UTXO set size under some simulation loads.
273 2016-07-01T11:05:38  <wumpus> I think it would make sense to design a standard interface for wallet coin selection algorithms, separately from any sepecific wallet
274 2016-07-01T11:05:38  <gmaxwell> We had simulations, they might not have been ideal.
275 2016-07-01T11:05:57  <wumpus> so that research and simuation can be done outside of the specific scope of bitcoin core
276 2016-07-01T11:06:08  <gmaxwell> They showed bad behavior from this change. From intution I predited specifically this bad behavior and asked for the simulations.  Our issue is not that we need more simulations.
277 2016-07-01T11:06:16  <gmaxwell> (well we do, but that didn't help here!)
278 2016-07-01T11:06:29  <wumpus> ok, never mind then
279 2016-07-01T11:06:37  <gmaxwell> and still MarcoFalke is not agreeing that there is a potential issue.
280 2016-07-01T11:06:46  <wumpus> just trying to think of some things that might help, out of the blue
281 2016-07-01T11:06:53  <gmaxwell> yea.
282 2016-07-01T11:06:56  <wumpus> if it's all hopeless I'll just shut up
283 2016-07-01T11:07:27  <gmaxwell> xekyo also identifyied a useful strategy, making the coinselection target double the value instead of the value. This results in change of useful sizes.
284 2016-07-01T11:07:41  <wumpus> *more* simulations may not be a solution, but better, correctly implemented ones would
285 2016-07-01T11:08:32  <wumpus> if the answer of this is up to whether the simulation is implemented correctly, then we're not really helped a bit, I agree
286 2016-07-01T11:09:27  <gmaxwell> re: assumes bad-case behavior from the user, well, indeed, it's no effect if the user doesn't reuse. But no harm, and reuse is ubiquitious. The (vast?) majority of bitcoins in circulation are held in reused addresses.
287 2016-07-01T11:09:56  <wumpus> the vast majority of those may not  be using bitcoin core in the first place
288 2016-07-01T11:10:38  <wumpus> many wallets encourage address reuse, or can only do address resuse
289 2016-07-01T11:10:41  <wumpus> that rule may help them a lot :)
290 2016-07-01T11:11:15  <gmaxwell> pretty much every active user has some reuse though, consider that thing that sends tips for commits.
291 2016-07-01T11:11:35  <wumpus> anyhow that speaks too of generalizing the coin selection algorithm beyond the specific software level
292 2016-07-01T11:11:37  <gmaxwell> means that all of us have reuse, even if we otherwise act perfectly ourselves. :)
293 2016-07-01T11:11:51  <wumpus> yes, that is true, ideally that should use some BIP32 construction
294 2016-07-01T11:12:12  <wumpus> to be honest I always use manual coin selection
295 2016-07-01T11:12:48  <wumpus> (apart from testing)
296 2016-07-01T11:13:13  <gmaxwell> I do too. (and always spend all the coins connected to an address, when I spend any)
297 2016-07-01T11:14:00  <wumpus> and that hould ideally coincide with changing the donation address
298 2016-07-01T11:14:07  <gmaxwell> they've gotta get spent someday or the coins are lost, fees are likely lower now than in the future... and spending at once avoids privacy harm from constantly interlinking inside the wallet.
299 2016-07-01T11:14:35  <gmaxwell> yea, I do that with my bct address. change the one on the site, when I spend all the coins.
300 2016-07-01T11:15:02  <gmaxwell> for that tip commit thing, I just haven't been spending its payments, I think... as I dunno how to change it.
301 2016-07-01T11:16:10  <wumpus> they have a little web interface where you can log in and change the address IIRC
302 2016-07-01T11:18:18  <gmaxwell> I wonder what the average coin value is in the utxo set and how thats evolved over time. (it's not quite as simple as the set size evolution over time, since more coins have also been introduced)
303 2016-07-01T11:18:21  *** murch has joined #bitcoin-core-dev
304 2016-07-01T11:19:31  <gmaxwell>   "errors": "WARNING: abnormally high number of blocks generated, 4477 blocks received in the last 4 hours (24 expected)"
305 2016-07-01T11:19:34  <gmaxwell> lol
306 2016-07-01T11:20:11  <wumpus> yes that would be an interesting utxo statistic
307 2016-07-01T11:20:48  <wumpus> gmaxwell: https://github.com/bitcoin/bitcoin/pull/8275 :p
308 2016-07-01T11:27:14  <GitHub78> [bitcoin] laanwj opened pull request #8298: wallet: Revert input selection post-pruning (master...2016_06_revert_wallet_input_postprune) https://github.com/bitcoin/bitcoin/pull/8298
309 2016-07-01T11:33:40  *** molly is now known as moli
310 2016-07-01T11:37:33  <sturles> I can see some HD related pull requests with no milestone set.  Any chance of HD support in 0.13, or is it 0.14 material?
311 2016-07-01T11:37:47  <wumpus> basic HD support has been merged for 0.13
312 2016-07-01T11:38:11  <wumpus> the feature freeze for 0.13 is long past now though, so everything else is for 0.14 soonest
313 2016-07-01T11:38:18  <wumpus> feel free to help testing them though, that helps a lot
314 2016-07-01T11:38:58  <wumpus> if anything there's a lack of involvement and testing and review for wallet PRs, especially UI oriented ones
315 2016-07-01T11:39:40  <gmaxwell> we certantly had wallet features slip due to lack of love.
316 2016-07-01T11:40:57  *** Prattler has quit IRC
317 2016-07-01T11:42:38  <sturles> I haven't used the UI for years..  Will start testing HD stuff.
318 2016-07-01T11:45:34  *** blur3d has joined #bitcoin-core-dev
319 2016-07-01T11:46:23  *** cryptapus_afk is now known as cryptapus
320 2016-07-01T11:51:12  <wumpus> at least for BIP32, in contrast to coin selection, the goal is well-defined and delimited :)
321 2016-07-01T11:56:45  <murch> wumpus: I'm working on the latter. :)
322 2016-07-01T11:57:36  <wumpus> murch: awesome!
323 2016-07-01T11:58:28  <murch> I'm writing a Master thesis on Coin Selection. I've been talking about that to some people here before.
324 2016-07-01T11:58:41  <murch> (Pieter, Greg, Greg, Marco)
325 2016-07-01T11:59:02  <wumpus> great, some more general reserach there would be very welcome
326 2016-07-01T11:59:37  <murch> One focus will be trying to analyze goals, prerequisites and constraints, then the results of varying approaches for selection
327 2016-07-01T12:00:15  <wumpus> yea exactly, what I posted here https://github.com/bitcoin/bitcoin/pull/8298#issuecomment-229928583  I think what is needed first is a list of criteria to judge coin selection algorithms by
328 2016-07-01T12:00:48  <wumpus> right now we can't tell up from down
329 2016-07-01T12:01:11  <wumpus> (unless it is obvious breakage)
330 2016-07-01T12:03:54  <murch> Thanks, I hadn't seen that one yet.
331 2016-07-01T12:04:16  <murch> I was actually the one that provided the simulation in 4096
332 2016-07-01T12:05:05  <wumpus> I mean there are obvious concerns such as CPU usage and memory usage, fee minimization, but also more 'tragedy of the commons' issues such as privacy concerns, utxo growth concerns (though that also coincides with performance a bit, keeping the wallet's utxo set small keeps the global set also smaller)
333 2016-07-01T12:06:05  <murch> Actually, I think that the pruning should have had little effect, as there should only be anything to prune when a second pass is needed. Otherwise, since the last added UTXO would be the smallest, there should not be any UTXO prunable.
334 2016-07-01T12:06:45  <wumpus> it has very little effect
335 2016-07-01T12:06:55  <wumpus> (but apparantly visible in some cases)
336 2016-07-01T12:07:00  *** justanotheruser is now known as UserNoJustanuu
337 2016-07-01T12:07:39  *** UserNoJustanuu is now known as justanotheruser
338 2016-07-01T12:08:00  <wumpus> though of course it's not entirely certain whether the reported problems were due to this change, or another change, or a change in the general usage of bitcoin not directly related to a change in our wallet
339 2016-07-01T12:08:50  <wumpus> or a combination of factors including this change
340 2016-07-01T12:09:59  <wumpus> in any case, if it has 'little effect' that's also enough reason to revert, I think. If it achieves hardly anything good, it shouldn't have been changed.
341 2016-07-01T12:10:13  <murch> wumpus: Completely agree.
342 2016-07-01T12:11:47  <murch> Well, I hope that I'll be able to provide some real improvements in the following months. :)
343 2016-07-01T12:12:41  <murch> Although I was surprised how well the current implementation does. It's not trivial to improve on it, and not make it deterministic in some fashion. :)
344 2016-07-01T12:13:41  <murch> Oh, and providing some metrics to compare Coin Selection approaches is the focus of the work
345 2016-07-01T12:13:45  <wumpus> I hope so too!
346 2016-07-01T12:14:14  <wumpus> good to hear the current approach is fairly good, on the other hand, that is going to make it even harder to replace by something better :)
347 2016-07-01T12:14:45  <wumpus> I think the most common complaints are that it does bad with very large wallets, or containing lots of small inputs
348 2016-07-01T12:14:51  <murch> By the way, wumpus, do you know of any other wallet usage data? I got the moneypot.com wallet from #4096, but more good testcases would be grand.
349 2016-07-01T12:14:58  <wumpus> apart from that it's not something people usually stumble upon
350 2016-07-01T12:15:46  <wumpus> no, unfortunately not
351 2016-07-01T12:16:00  <wumpus> I don't have any realistic big wallets
352 2016-07-01T12:16:13  <murch> Okay, too bad. :)
353 2016-07-01T12:16:58  <wumpus> (this is also an issue which has existed from 2011, which affects both coin selection and general wallet scaling work, people with big wallets are very reluctant to share them even with developers :-) )
354 2016-07-01T12:17:34  <murch> Yeah, I'd imagine. :-/
355 2016-07-01T12:18:15  <wumpus> which is also why I always encourage companies which have to cope with them to get involved in development themselves, that avoids having to share anything with third parties
356 2016-07-01T12:18:27  <murch> The code in wallet.cpp is pretty hard to understand, too. I'd love to refactor it, maybe even to just understand it properly…
357 2016-07-01T12:18:46  <wumpus> phantomcircuit is working on that too
358 2016-07-01T12:18:53  <murch> oh really?!
359 2016-07-01T12:19:19  *** Chris_Stewart_5 has joined #bitcoin-core-dev
360 2016-07-01T12:19:21  <murch> mh, just refactoring it, or also improving?
361 2016-07-01T12:19:30  <wumpus> in a way it's kind of a chicken and egg problem though, people rarely dare to change things *becuase* they're not sure how they work exactly, but evaluating refactors is also made hard by that because it's non-trivial to see that behavior stays the same
362 2016-07-01T12:20:35  <wumpus> both, but not coin selection AFAIK
363 2016-07-01T12:20:36  <murch> Yeah, on the other hand it's in a state where it is really hard to add unittests to check whether the behavior remains unchanged.
364 2016-07-01T12:20:46  <murch> okay, excellent
365 2016-07-01T12:20:49  <wumpus> yes, that too.
366 2016-07-01T12:23:54  <wumpus> and years of proposals of doing things differently have made me think that it being hard to understand is not so much a result of the organizatin of wallet.cpp, but that the underlying subject matter is hard
367 2016-07-01T12:24:45  <wumpus> people tend to read the code and then blame that the code is difficult, but it's not entirely clear whether it is *unnecessarily* difficult
368 2016-07-01T12:25:22  <wumpus> it's clearly a complex problem too
369 2016-07-01T12:25:41  <wumpus> maybe it could be divided up into better manageb;e sub-problems though
370 2016-07-01T12:25:47  <wumpus> manageable*
371 2016-07-01T12:27:46  <wumpus> the thing is, before wallet.cpp was written, cryptocurrency wallets was a problem no one ever considered before, it was necessarily ad-hoc
372 2016-07-01T12:28:24  <wumpus> this is very different from say, a web server, where everyone knows what is expected from it, how it usually should be structured, and so on
373 2016-07-01T12:30:46  <murch> wumpus: It's surely complicated, but there are some methods that could be extract, and some things are just somewhat obfuscated by very brief variable names. E.g. I've been looking at how and when the fee gets decided for the transaction and it's still baffling me.
374 2016-07-01T12:30:48  <wumpus> so the refactor is also a discovery process, learning how to best structure this novel application
375 2016-07-01T12:32:12  *** randy-waterhouse has quit IRC
376 2016-07-01T12:32:28  <wumpus> well yes it's complicated - but does it need to be complicated, is there a less complex way that would be just as good ,or better? would that simpler way still satisfy the requirements? (and in some cases: what are the requirements, even?)
377 2016-07-01T12:34:01  <murch> wumpus: I think it could be more comprehensible if key management and coin selection were separated into different classe for example. They are pretty far apart for being part of one class.
378 2016-07-01T12:34:04  <wumpus> things like variable names are trivialities, sure, on first reading of the code it helps to have better names, but once you learn what they are it doesn't really matter what they are anymore (e.g. mathematicians do fine with names such as a b c :)
379 2016-07-01T12:34:36  <murch> Most of Coin Selection could actually be completely static, as it basically is just reliant on the spending target, the UTXO set and the desired fee level.
380 2016-07-01T12:34:49  <wumpus> yes coin selection is a clear unit
381 2016-07-01T12:35:18  <wumpus> I think it would be nice to factor it out to a separate unit with a separate interface, that just gets the information that it needs and returns the selected list
382 2016-07-01T12:35:29  <wumpus> this would also be useful for simulations
383 2016-07-01T12:35:31  <murch> well, wallet.cpp is just a bit of a moloch to delve into at 3500 LOC. ;)
384 2016-07-01T12:35:46  <murch> wumpus: exactly
385 2016-07-01T12:35:49  <wumpus> I've seen so much worse at some companies I've worked at :-)
386 2016-07-01T12:36:40  <murch> Also, adding, watching and tracking UTXO could probably be separated off
387 2016-07-01T12:36:54  <wumpus> yes, that would make sense
388 2016-07-01T12:37:19  <wumpus> did you see https://github.com/bitcoin/bitcoin/pull/7823 ?
389 2016-07-01T12:38:07  <murch> nope, but I just subscribed.
390 2016-07-01T12:38:27  <murch> I am not nearly as familiar with the github repository as I'd like. :-/
391 2016-07-01T12:38:40  <wumpus> the idea is quite nice, but it's *hard* to see what its interaction with different things would be, such as reorganizations and conflicts
392 2016-07-01T12:38:46  <murch> I should probably check out all issues tagged with "wallet" soon.
393 2016-07-01T12:38:52  <wumpus> yes :)
394 2016-07-01T12:39:44  <wumpus> in any case, coin selection in itself is enough of a subject to fill a master's thesis I think, I'd warn against scope creep, or trying to fix the world at once :)
395 2016-07-01T12:39:45  *** davec_ has quit IRC
396 2016-07-01T12:39:53  <murch> hehe
397 2016-07-01T12:40:16  *** davec_ has joined #bitcoin-core-dev
398 2016-07-01T12:41:02  <murch> yeah, it's a bit overwhelming at times. I've been reading about Subset Sum solvers the past days, and delving through wallet.cpp to understand how fees are handled.
399 2016-07-01T12:42:36  <murch> Anyway, back to work. ;) TTYL
400 2016-07-01T12:43:09  <wumpus> later :) hope you manage to make head and tail of it
401 2016-07-01T12:43:23  <wumpus> hm the new option -blockmaxcost is going to give user understanding problems: https://github.com/bitcoin/bitcoin/blob/master/src/init.cpp#L455
402 2016-07-01T12:43:51  <wumpus> should this be a hidden/debug option? if not, it should be documented better
403 2016-07-01T12:53:42  *** TheFactory7 has joined #bitcoin-core-dev
404 2016-07-01T13:06:21  *** Prattler has joined #bitcoin-core-dev
405 2016-07-01T13:10:43  *** Prattler has quit IRC
406 2016-07-01T13:18:24  <jonasschnelli> sipa, wumpus: Do I get this right: -reindex does re-create the block index and recreates the utxo set (including all signature validation)    while -reindex-chainstate will only recreate the utxo set with the current block index?
407 2016-07-01T13:20:26  <sdaftuar> wumpus: -blockmaxcost is supposed to be the recommended way to configure the mining code, going forward
408 2016-07-01T13:21:25  <sdaftuar> wumpus: and -blockmaxsize is in the we-want-to-deprecate-in-the-future category
409 2016-07-01T13:21:44  <sdaftuar> (the mining code is optimizing for fee-per-block-cost, not fee-per-serialized-byte)
410 2016-07-01T13:22:05  <sdaftuar> but yes, we need to document all this before release
411 2016-07-01T13:22:55  <sdaftuar> heh, yeah that --help message text is not very informative
412 2016-07-01T13:27:59  *** Retric has joined #bitcoin-core-dev
413 2016-07-01T13:29:21  *** tadasv has joined #bitcoin-core-dev
414 2016-07-01T13:30:55  <wumpus> sdaftuar: a translator was wondering what kind of cost this was, I think they assume that it's something with fee
415 2016-07-01T13:31:10  <wumpus> jonasschnelli: correct
416 2016-07-01T13:31:12  <sdaftuar> oops.  yeah that is bad language
417 2016-07-01T13:31:50  *** zooko has joined #bitcoin-core-dev
418 2016-07-01T13:31:58  <sdaftuar> what's the timeline for improving that text, is it too late for making changes now that affect translation?
419 2016-07-01T13:32:06  <sdaftuar> i guess we can just document in the release notes?
420 2016-07-01T13:32:31  <wumpus> well given that translators are confused by it (and hence unable to translate it effectiely) I wouldn't mind changing the message
421 2016-07-01T13:33:20  <wumpus> there also needs to be mention in the release notes, but release notes aren't really documentation, just 'news'
422 2016-07-01T13:33:46  <wumpus> e.g. you wouldn't assume that someone that is looking for documentation for an option to go through all previous release notes to find it documented
423 2016-07-01T13:33:55  <sdaftuar> a simple change might just be to reference the BIP where it's defined; that wouldn't necessarily impose an additional burden on translators if it's just "(BIP 141)" or something at the end of it
424 2016-07-01T13:34:04  *** Chris_Stewart_5 has quit IRC
425 2016-07-01T13:34:13  <wumpus> what needs to be made clear is that this is just an abstract cost
426 2016-07-01T13:34:30  <wumpus> referencing the BIp would make sense, yes
427 2016-07-01T13:34:52  <wumpus> in any case, even if it's too late for this to be translated, a better english documentation message would go further than a confusing translated one :)
428 2016-07-01T13:35:17  <sdaftuar> agreed!
429 2016-07-01T13:35:43  <sdaftuar> perhaps block cost shouldn't be translated at all actually, if we're referencing the BIP
430 2016-07-01T13:39:31  <sdaftuar> i updated #8294 so we don't forget this
431 2016-07-01T13:43:21  <MarcoFalke> gmaxwell: I don't think lack of progress in improving of coinselection is due to the test hardcoding the behavior.
432 2016-07-01T13:43:38  <MarcoFalke> Right now they save us from introducing accidental regressions, which is nice
433 2016-07-01T13:44:09  <MarcoFalke> If someone comes up with a new idea, the unit test need to go anyway and will be replaced by new ones, so not really an issue
434 2016-07-01T13:45:25  <MarcoFalke> Also the 25% to 400% performance loss may be flawed, as the coin generator was not adjusted to what happens in the real network
435 2016-07-01T13:45:33  <MarcoFalke> (re simulation)
436 2016-07-01T13:47:19  <MarcoFalke> I remember there was a site which put every address which was ever in a wallet. (Combining whenever two addresses happen to be used in the inputs of the same tx)
437 2016-07-01T13:47:46  <MarcoFalke> This could be useful to verify coin generators in simulations are doing their job properly
438 2016-07-01T13:48:08  *** cjcj has quit IRC
439 2016-07-01T13:51:10  *** JackH has quit IRC
440 2016-07-01T14:09:06  *** blur3d has quit IRC
441 2016-07-01T14:13:34  *** Retric has quit IRC
442 2016-07-01T14:18:04  *** cjcj has joined #bitcoin-core-dev
443 2016-07-01T14:22:46  *** zooko has quit IRC
444 2016-07-01T14:23:23  *** Cats2 has joined #bitcoin-core-dev
445 2016-07-01T14:25:48  *** ClockCat has quit IRC
446 2016-07-01T14:26:14  *** droark has quit IRC
447 2016-07-01T14:28:29  *** Giszmo has joined #bitcoin-core-dev
448 2016-07-01T14:28:42  *** pedrobranco has quit IRC
449 2016-07-01T14:34:25  *** MarcoFalke has left #bitcoin-core-dev
450 2016-07-01T14:38:22  *** Chris_Stewart_5 has joined #bitcoin-core-dev
451 2016-07-01T14:38:55  *** Chris_Stewart_5 has quit IRC
452 2016-07-01T14:39:07  *** Chris_Stewart_5 has joined #bitcoin-core-dev
453 2016-07-01T14:53:58  *** pedrobranco has joined #bitcoin-core-dev
454 2016-07-01T15:00:57  <morcos> wumpus: gmaxwell: i don’t feel strongly enough to make a big argument about it, but if it was up to me i wouldn’t bother reverting 4906.  I agree that we didn’t have sufficient justification to merge it in the first place, but we already crossed that bridge, and discussed it after the fact (more than once).   I’m not sure we’re confident enough that its clearly worse to risk making the same mistake in the other directi
455 2016-07-01T15:01:10  <morcos> direction by reverting it
456 2016-07-01T15:01:22  <morcos> I suppose my point is I’d err on the side of being conservative with changes to coin selection and only making them when someone has put in the effort to really study them.  Since 0.12 has been out for a while running with 4906, it now seems like a change to me to revert it.
457 2016-07-01T15:05:48  *** Chris_Stewart_5 has quit IRC
458 2016-07-01T15:06:01  *** kadoban has joined #bitcoin-core-dev
459 2016-07-01T15:14:51  <jonasschnelli> sipa: A full IBD (against random peers) with current master took ~3h4min, a -reindex-chainstate took ~1h10min
460 2016-07-01T15:15:07  <jonasschnelli> http://bitcoinsrv.jonasschnelli.ch/log.reindex-chainstate.nodebug.da5099.txt, http://bitcoinsrv.jonasschnelli.ch/log.reindex-chainstate.nodebug.da5099.full.txt
461 2016-07-01T15:15:28  <jonasschnelli> IBD: http://bitcoinsrv.jonasschnelli.ch/log.ibd.nodebug.da5099.txt, http://bitcoinsrv.jonasschnelli.ch/log.ibd.nodebug.da5099.full.txt
462 2016-07-01T15:15:39  <jonasschnelli> now doing a full -reindex
463 2016-07-01T15:15:57  <jonasschnelli> Used -dbcache=8000 for both runs
464 2016-07-01T15:37:06  <wumpus> morcos: I don't think 'we crossed that bridge' is a good argument, if this was no improvement, it should not have been merged, and it should be reverted (it should have been reverted a long time ago!)
465 2016-07-01T15:37:21  <wumpus> morcos: I don't see any reason to keep the code if it i not an improvement
466 2016-07-01T15:37:52  <wumpus> and if we should be conservative about changes to coin selection, which we should have been in the first place, then again we shouldn't have this change without understanding it
467 2016-07-01T15:38:19  <morcos> wumpus: ok like i said i won't argue, it just makes me nervous that somehow reverting could be worse (interaction with something else that changed that we're not considering), but i don't have any reason to believe that
468 2016-07-01T15:38:20  <wumpus> I don't think having the mistake out in 0.12 is  a reason to keep it now
469 2016-07-01T15:38:25  <wumpus> don't get attached to your mistakes :)
470 2016-07-01T15:38:49  <morcos> i won't get attached to somebody's mistakes  :)
471 2016-07-01T15:39:07  <wumpus> then again I don't feel strongly about it either, but I'm tired of it coming up every time
472 2016-07-01T15:39:16  <wumpus> let's just make a damn decision about it
473 2016-07-01T15:39:29  <wumpus> I'm sure if we decide not to revert it now, then someone will bring it up again after a month or so
474 2016-07-01T15:39:54  <wumpus> I don't want this following me around forever
475 2016-07-01T15:41:00  <wumpus> I don't see how this could interact with anything else
476 2016-07-01T15:44:17  <bsm1175321> Woah.  I just built github master and on testnet something has gone horribly wrong: I have a negative balance!
477 2016-07-01T15:45:02  <morcos> bsm1175321: checked how?
478 2016-07-01T15:45:14  <bsm1175321> |mcelrath@jane:~> bitcoin-cli -testnet getbalance ''
479 2016-07-01T15:45:14  <bsm1175321> -56.85287200
480 2016-07-01T15:45:24  <sipa> drop the ''
481 2016-07-01T15:45:27  <bsm1175321> It seems to have suddenly fixed itself.
482 2016-07-01T15:45:30  <sipa> you're computing an account balance
483 2016-07-01T15:45:36  <sipa> not the wallet balance
484 2016-07-01T15:45:50  <wumpus> you used to wrong way to query your balance! woe you!
485 2016-07-01T15:46:06  <bsm1175321> Without the quotes I got:
486 2016-07-01T15:46:07  <bsm1175321> |mcelrath@jane:~> bitcoin-cli -testnet getbalance
487 2016-07-01T15:46:07  <bsm1175321> 0.00000000
488 2016-07-01T15:46:10  <morcos> i hesitate to ask this question, but can we finally remove accounts for 0.14
489 2016-07-01T15:46:15  <wumpus> (we're crazy that there are so many, including wrong ways to query your balance)
490 2016-07-01T15:46:22  <morcos> talk about having an issue follow us around
491 2016-07-01T15:46:22  <wumpus> morcos: I tried to push for a label API to replace it
492 2016-07-01T15:46:30  <bsm1175321> Which is also wrong.  A few minutes later it seems to have synced up...and is now showing correct balances.  What happened?
493 2016-07-01T15:46:42  <sipa> there was a 4000 deep reorg
494 2016-07-01T15:46:48  <wumpus> morcos: unnecessary to say, it didn't get enough attention, and is still lingering, I hope to pick it up again for 0.14
495 2016-07-01T15:47:12  <bsm1175321> +1 for getting rid of accounts...
496 2016-07-01T15:47:25  <wumpus> morcos: https://github.com/bitcoin/bitcoin/pull/7729   I think the release after that we can really remove the account functionality
497 2016-07-01T15:47:56  <morcos> wumpus: ok 3rd topic. i thought i head you guys mention that something has gotten slower recently.  is it just reindex, or actually something with connecting blocks?
498 2016-07-01T15:48:11  <bsm1175321> Next question: I need to fund transactions and be sure that their txid is not malleable.  I was hoping 'fundrawtransaction' had an option to take only segwit inputs, but it seems it does not.  What's the best way to achieve this?
499 2016-07-01T15:48:20  <wumpus> morcos: AFAIK the summary there was: gmaxwell accidentally -txindex
500 2016-07-01T15:48:45  <sipa> morcos: and nobody recently tested -reindex with default -dbcache
501 2016-07-01T15:48:56  <morcos> wumpus: oh i definitely missed that conclusion. :)
502 2016-07-01T15:48:56  <wumpus> morcos: yes, more recent blocks are slower to validate, comapred to older blocks of the same size - but this isn't a reversion in the code
503 2016-07-01T15:49:11  <morcos> wumpus: wait, yes its that last thing i'm asking about
504 2016-07-01T15:49:14  <morcos> why is that?
505 2016-07-01T15:49:25  <wumpus> it's likely a by-product of increasing utxo size
506 2016-07-01T15:50:06  <morcos> and that affecting cache hit rate?
507 2016-07-01T15:50:07  <wumpus> and the utxo database is reaching the size that leveldb can handle w/ good performance, e.g. now lots of crazy seeking and reading all over your disk, can't be good for performance
508 2016-07-01T15:50:14  <morcos> oh
509 2016-07-01T15:50:37  <wumpus> yes that was another hypothesis
510 2016-07-01T15:50:51  <wumpus> <gmaxwell> I was theorizing that this was from polylog behavior in the database and worrying, but phantomcircuit gave an alternative argument that the reduction in spammy transactions relative to non-spammy ones may be resulting in lower cache hitrates.
511 2016-07-01T15:50:59  <morcos> eyeballing the numbers i'm not seeing a noticeable slowdown since march
512 2016-07-01T15:51:04  <wumpus> which probably means it's a combination of both
513 2016-07-01T15:52:07  <wumpus> in any case the increased default dbcache should alleviate either problem, at least for a while
514 2016-07-01T15:53:26  <morcos> ok, yeah my numbers are with a big dbcache (2G)
515 2016-07-01T15:53:28  <wumpus> there may be a better way to structure the utxo set on disk - but having looked into other databases it seems that leveldb has the best performance for our use, at least during sync
516 2016-07-01T15:54:17  <wumpus> if you set a huge dbcache you won't notice much, as sipa says it's probably why the surprise that indexing was so slow with default dbcache
517 2016-07-01T15:54:30  <sipa> for 0.14 we should prioritize working on chainstate backups
518 2016-07-01T15:55:05  <sipa> (and vigorously fight the potential push for servers with 'helpful' backups to download...)
519 2016-07-01T15:58:53  <bsm1175321> I need to fund transactions and be sure that their txid is not malleable.  I was hoping 'fundrawtransaction' had an option to take only segwit inputs, but it seems it does not.  What's the best way to achieve this?  (I'm willing to write such an option if there isn't a better way) @sipa?
520 2016-07-01T15:59:18  <sipa> bsm1175321: patches welcome :)
521 2016-07-01T15:59:28  <sipa> seems like a very useful feature
522 2016-07-01T15:59:31  <bsm1175321> Ok so the answer is that this isn't possible currently?
523 2016-07-01T15:59:46  <sipa> for now, you'll need listunspent
524 2016-07-01T15:59:48  <bsm1175321> Cool, I'll see what I can do.
525 2016-07-01T16:00:01  <sipa> i believe i've heard someone else ask for the same feature
526 2016-07-01T16:09:01  *** Guyver2 has joined #bitcoin-core-dev
527 2016-07-01T16:09:30  <bsm1175321> One complication with such an option is that it's possible for your wallet to have enough funds to fund the transaction, but not enough segregated witness inputs.  A solution would be to create an intermediate transaction whose outputs have segregated witness.  Thoughts?
528 2016-07-01T16:09:48  <sipa> fundrawtransaction shouldn't do such thing
529 2016-07-01T16:09:51  <sipa> imho
530 2016-07-01T16:10:04  <sipa> but it can fail with a nice error code, instructing you to clean up your wallet
531 2016-07-01T16:10:12  <bsm1175321> You'd rather it fail in such a case?  That's fine with me too...
532 2016-07-01T16:10:23  <sipa> yes, insufficient funds :)
533 2016-07-01T16:10:42  <sipa> (or perhaps you should use separate wallets if dealing with that situation is hard)
534 2016-07-01T16:10:51  <bsm1175321> What would you expect a user to do in such a case?  Is there a procedure to "clean up your wallet"?  That people will know?
535 2016-07-01T16:11:25  <sipa> it's no different from the situation where you have a wallet with some watchonly coins, and the non-watchonly together are not enough to pay
536 2016-07-01T16:12:03  <bsm1175321> True, sort of, except you have the funds.  I'll make a descriptive error message.
537 2016-07-01T16:15:47  <sipa> i think the real solution is that if you have a need for segwit-only inputs, you run with a segwit-only wallet
538 2016-07-01T16:17:23  <bsm1175321> In effect that's what I'll do.  But have to start from a non-segwit wallet.  Or is there a spent-to-myself-segwit-out wallet RPC call?
539 2016-07-01T16:18:48  <sipa> you can just use sendtoaddress to send from the old wallet to the new?
540 2016-07-01T16:19:28  <sipa> also, wallet integration of segwit is really preliminary at this point
541 2016-07-01T16:19:39  <sipa> for example, change outputs won't be segwit
542 2016-07-01T16:19:55  <bsm1175321> ooohhh...hmmm...
543 2016-07-01T16:20:09  <sipa> (which they should be when it becomes ready for production, but that's not done yet)
544 2016-07-01T16:21:29  <bsm1175321> Well if you 'fundrawtransaction ... segwitOnly' I can make it use a segwit change address.
545 2016-07-01T16:21:50  <bsm1175321> Or you can add an explicit change address
546 2016-07-01T16:22:03  <sipa> true
547 2016-07-01T16:22:31  <bsm1175321> If no objections, I'll make the 'segwitOnly' option take only segwit inputs, and generate segwit change.
548 2016-07-01T16:33:08  *** Chris_Stewart_5 has joined #bitcoin-core-dev
549 2016-07-01T16:35:12  <instagibbs> what is/is there a way to link to an external secp library during compilation?
550 2016-07-01T16:41:43  <bsm1175321> sipa: do you have a "wallet wishlist" for segwit?  I'm seeing that a lot of infrastructure is missing here.
551 2016-07-01T16:43:52  <sipa> bsm1175321: thinks like default witness addresses
552 2016-07-01T16:43:57  <sipa> post softfork
553 2016-07-01T16:53:45  <bsm1175321> I'm confused regarding addresses.
554 2016-07-01T16:54:24  *** jtimon has joined #bitcoin-core-dev
555 2016-07-01T16:54:30  <bsm1175321> What is the output of 'addwitnessaddress'?  (what kind of address is that?)
556 2016-07-01T16:55:42  *** jtimon has quit IRC
557 2016-07-01T16:57:32  <sipa> p2sh
558 2016-07-01T16:58:29  <bsm1175321> Ok, duh.  (hadn't made a testnet p2sh before, was confused by the 2)
559 2016-07-01T17:04:43  <bsm1175321> Ok I think I understand.  A wallet bitcoin.conf really should have 3 possible values then: non-segwit, segwit, and segwit-in-p2sh.
560 2016-07-01T17:04:50  <bsm1175321> *wallet setting
561 2016-07-01T17:05:16  <bsm1175321> Do I understand correctly: The point of P2SH nesting is that non-upgraded wallets will check the destinations (though not the signatures)?
562 2016-07-01T17:15:56  <jl2012> bsm1175321: I'd say the point of P2SH nesting is to allow non-upgraded wallets to send money to upgraded wallet
563 2016-07-01T17:19:34  *** Chris_Stewart_5 has quit IRC
564 2016-07-01T17:20:11  <bsm1175321> Can't they always do that though?  Do you mean the other way around?
565 2016-07-01T17:21:36  <arubi> it's made for upgraded nodes to be able to accept payment from unupgraded nodes to a segwit scriptpubkey, yes
566 2016-07-01T17:23:09  <jl2012> for upgraded -> not upgraded, you just use P2PKH address (1xyz......)
567 2016-07-01T17:25:37  <arubi> bsm1175321, I'm curious, what kind of malleability are you trying to avoid?  are you the one signing the transaction?  can't you use normal inputs too and just sign it properly?
568 2016-07-01T17:26:41  <bsm1175321> arubi: I'm going to be handing out txids, including txids that may be in the mempool, not mined yet.
569 2016-07-01T17:27:58  <arubi> signed by you and you only?  you could still "maul"(?) if you wanted to, even with segwit
570 2016-07-01T17:28:31  <jl2012> arubi: without BIP62, that's not reliable
571 2016-07-01T17:28:47  <bsm1175321> The non-upgraded nodes that receive funds from an upgraded node doesn't verify signatures though.  (correct?)
572 2016-07-01T17:29:32  <arubi> jl2012, talking about the anyonecanpay|single sighash, sign once, permute how many you'd like
573 2016-07-01T17:30:15  <arubi> bsm1175321, the script that they see doesn't talk about checksigs, so they don't check any signatures
574 2016-07-01T17:30:21  <jl2012> arubi: there are still other forms of third party malleability, e.g. non-canonical push
575 2016-07-01T17:30:37  <bsm1175321> arubi: signed by me and me only.
576 2016-07-01T17:31:25  <arubi> jl2012, sure, using standard scripts is a must here if you don't want 3rd parties doing things, but as a single signer, you could still do it to your own txs, even with segwit
577 2016-07-01T17:33:11  <jl2012> well, if you want to guarantee non-malleability to only yourself, segwit is the only way
578 2016-07-01T17:33:16  <bsm1175321> jl2012: can you elaborate?  Malleation of the script (which is part of the witness data) doesn't change the txid, and I don't think I care about that.
579 2016-07-01T17:33:30  *** Chris_Stewart_5 has joined #bitcoin-core-dev
580 2016-07-01T17:34:05  <arubi> jl2012, true.  I am asking because bsm1175321 explains that he will hand out txids (not the transactions themselves, as I understand)
581 2016-07-01T17:34:29  *** pedrobranco has quit IRC
582 2016-07-01T17:34:29  <arubi> so as a single signer, he could still cause malleability, even if he manages to convince the other party all inputs are segwit
583 2016-07-01T17:34:52  <arubi> *and standard scripts, or even better, even only with p2wpkh
584 2016-07-01T17:34:53  <jl2012> arubi: that's double-spending
585 2016-07-01T17:34:57  *** pedrobranco has joined #bitcoin-core-dev
586 2016-07-01T17:35:02  <jl2012> not malleability
587 2016-07-01T17:35:26  <arubi> how so?  payments are still paid, but it's the order of the inputs outputs that's changed
588 2016-07-01T17:35:33  <arubi> you're using the same inputs
589 2016-07-01T17:36:06  <bsm1175321> arubi: Are you saying I could change the order of outputs, keeping the same segwit txid?
590 2016-07-01T17:36:18  <arubi> no, you're changing it
591 2016-07-01T17:37:54  <bsm1175321> Ok then someone looking for one txid I handed them won't see it at all, and there's no point in me giving the wrong txid to someone.  (A txid which never get mined because I malleated it is useless)
592 2016-07-01T17:38:31  <arubi> right
593 2016-07-01T17:39:01  <arubi> they should be checking their addresses for incoming transactions, and that's it.  txid is something that's only relevant when spending
594 2016-07-01T17:39:11  <arubi> *should only be, I guess
595 2016-07-01T17:39:19  <bsm1175321> However if they act on a txid in the mempool, and then I broadcast a malleated version which gets mined.  That's your usual double spend.
596 2016-07-01T17:39:40  *** pedrobranco has quit IRC
597 2016-07-01T17:40:16  <arubi> how is it a double spend?  there is no spending from the malleated version
598 2016-07-01T17:40:23  <pigeons> well if you only malleate the signature its not a double spend
599 2016-07-01T17:40:59  <bsm1175321> Let's say I malleate the output order.  This is a weird thing to do.  I'll have to think on it...
600 2016-07-01T17:41:42  <arubi> same can be done for inputs
601 2016-07-01T17:42:12  <pigeons> if you have someone taking action on the txid of the unconfirmed transaction they will be affected, otherwise, not right
602 2016-07-01T17:43:09  <arubi> a spender still has to reference a txid as input, so even if the outputs order isn't changed, the input is invalid in case of malleability
603 2016-07-01T17:43:56  <arubi> oh you mean if the bad one wasn't mined, right
604 2016-07-01T17:48:21  <bsm1175321> In this case, I'm proving control of coins.  Malleating my own txn is shooting myself in the foot.
605 2016-07-01T17:55:28  *** laurentmt has joined #bitcoin-core-dev
606 2016-07-01T17:58:42  <arubi> I'm guessing, you're proving control by announcing the txid before the transaction is seen on the network?
607 2016-07-01T18:08:50  <bsm1175321> Not *before* but it could be simultaneous.  And I'm wondering what the consequences are of fiddling around...
608 2016-07-01T18:09:21  <bsm1175321> But If I claim to prove something with a txid, and you look for it on the network, I'm only hurting myself by messing with it.  I'm having trouble thinking of any negative consequences here.
609 2016-07-01T18:11:24  <arubi> why not use signed messages to prove ownership of addresses?  as long you commit to an address, which is kinda like commiting to specific coins
610 2016-07-01T18:12:52  *** kadoban has quit IRC
611 2016-07-01T18:17:38  <arubi> hm.  I guess if it's a p2sh that's not so simple, because you'd have to disclose the redeemscript
612 2016-07-01T18:18:54  *** spudowiar has joined #bitcoin-core-dev
613 2016-07-01T18:23:00  *** pedrobranco has joined #bitcoin-core-dev
614 2016-07-01T18:26:41  *** spudowiar has quit IRC
615 2016-07-01T18:27:01  *** spudowiar has joined #bitcoin-core-dev
616 2016-07-01T18:45:06  *** pedrobranco has quit IRC
617 2016-07-01T18:45:39  *** pedrobranco has joined #bitcoin-core-dev
618 2016-07-01T18:45:53  *** pedrobranco has joined #bitcoin-core-dev
619 2016-07-01T18:49:39  *** pedrobranco has quit IRC
620 2016-07-01T18:50:18  *** pedrobranco has joined #bitcoin-core-dev
621 2016-07-01T18:50:34  *** pedrobranco has joined #bitcoin-core-dev
622 2016-07-01T18:51:50  *** pedrobranco has quit IRC
623 2016-07-01T18:56:06  *** pedrobranco has joined #bitcoin-core-dev
624 2016-07-01T19:03:12  *** pedrobranco has quit IRC
625 2016-07-01T19:03:46  *** pedrobranco has joined #bitcoin-core-dev
626 2016-07-01T19:07:51  *** CubicEarth has joined #bitcoin-core-dev
627 2016-07-01T19:08:04  *** pedrobranco has quit IRC
628 2016-07-01T19:23:59  *** TheFactory7 has quit IRC
629 2016-07-01T19:31:10  *** molz has joined #bitcoin-core-dev
630 2016-07-01T19:31:34  *** zooko has joined #bitcoin-core-dev
631 2016-07-01T19:33:34  *** moli has quit IRC
632 2016-07-01T19:34:05  *** laurentmt has quit IRC
633 2016-07-01T19:36:16  *** laurentmt has joined #bitcoin-core-dev
634 2016-07-01T19:39:18  *** Lysander1 has joined #bitcoin-core-dev
635 2016-07-01T19:41:15  *** CyrusV has quit IRC
636 2016-07-01T19:41:51  *** Cory has quit IRC
637 2016-07-01T19:41:53  *** Expanse has quit IRC
638 2016-07-01T19:42:22  *** jannes has quit IRC
639 2016-07-01T19:42:28  *** Lysanders has quit IRC
640 2016-07-01T19:42:28  *** LeMiner has quit IRC
641 2016-07-01T19:42:28  *** OxADADA has quit IRC
642 2016-07-01T19:42:29  *** TD-Linux has quit IRC
643 2016-07-01T19:42:29  *** nanotube has quit IRC
644 2016-07-01T19:42:29  *** crescendo has quit IRC
645 2016-07-01T19:42:52  *** spudowiar has quit IRC
646 2016-07-01T19:42:54  *** kinlo has quit IRC
647 2016-07-01T19:42:54  *** hexis has quit IRC
648 2016-07-01T19:42:54  *** petertodd has quit IRC
649 2016-07-01T19:42:55  *** windsok has quit IRC
650 2016-07-01T19:42:56  *** windsok_ has joined #bitcoin-core-dev
651 2016-07-01T19:43:53  *** Pasha has joined #bitcoin-core-dev
652 2016-07-01T19:43:53  *** crescendo has joined #bitcoin-core-dev
653 2016-07-01T19:44:17  *** OxADADA has joined #bitcoin-core-dev
654 2016-07-01T19:44:21  *** kinlo has joined #bitcoin-core-dev
655 2016-07-01T19:44:28  *** petertodd has joined #bitcoin-core-dev
656 2016-07-01T19:44:59  *** adiabat has joined #bitcoin-core-dev
657 2016-07-01T19:45:10  *** Expanse has joined #bitcoin-core-dev
658 2016-07-01T19:45:36  <adiabat> hey, I'm spamming testnet and have some unexpected behavior
659 2016-07-01T19:45:40  *** laurentmt has quit IRC
660 2016-07-01T19:46:05  <adiabat> I think I get what "size", "cost", "strippedsize", "Vsize" all mean
661 2016-07-01T19:46:50  <adiabat> "cost" is basically "Vsize" * 4
662 2016-07-01T19:47:05  <adiabat> so a block "cost" must be < 4M
663 2016-07-01T19:47:06  *** cryptapus is now known as cryptapus_afk
664 2016-07-01T19:47:10  <adiabat> (<=, whatever)
665 2016-07-01T19:47:47  *** TD-Linux has joined #bitcoin-core-dev
666 2016-07-01T19:48:07  <adiabat> so in .conf, blockmaxsize should set the cost limit of created blocks
667 2016-07-01T19:48:23  <adiabat> but it seems to be targeting "size" instead
668 2016-07-01T19:48:43  *** CyrusV has joined #bitcoin-core-dev
669 2016-07-01T19:49:07  <bsm1175321> arubi: I want to be able to commit to coins regardless of the address type, and as you say, I'd need the redeemScript.
670 2016-07-01T19:50:47  *** Pasha is now known as Cory
671 2016-07-01T19:52:22  *** kvnn has joined #bitcoin-core-dev
672 2016-07-01T19:53:58  *** nanotube has joined #bitcoin-core-dev
673 2016-07-01T19:54:34  <kvnn> Hello everyone. I'm offering a 1BTC bounty for 3&4 here: https://github.com/drivechain-project/docs . Please let me know if you are interested. (if this is not okay in this channel I apologize & will discontinue posting about it)
674 2016-07-01T19:55:28  *** jannes has joined #bitcoin-core-dev
675 2016-07-01T19:55:58  *** hexis has joined #bitcoin-core-dev
676 2016-07-01T19:56:22  *** spudowiar has joined #bitcoin-core-dev
677 2016-07-01T19:59:44  *** sdaftuar_ has joined #bitcoin-core-dev
678 2016-07-01T20:02:55  <sdaftuar_> adiabat: -blockmaxsize continues to refer to total serialized bytes of a block, counting witness and non-witness parts the same. -blockmaxcost is the option that specifies the limit you probably want here.
679 2016-07-01T20:04:43  <adiabat> hmmm ok
680 2016-07-01T20:04:48  <adiabat> so there's both
681 2016-07-01T20:09:18  <sdaftuar_> Yeah.  The hope is that -blockmaxsize is deprecated in the future.
682 2016-07-01T20:10:21  *** spudowiar has quit IRC
683 2016-07-01T20:10:38  <sdaftuar_> (My preference was to change the semantics of -blockmaxsize to refer to segwit's virtual size, but that's not what we ended up with)
684 2016-07-01T20:11:02  <luke-jr> well, depends what you want to limit
685 2016-07-01T20:11:23  <roasbeef> segwit miners on testnet seem to be limiting to ~750k cost
686 2016-07-01T20:11:31  <luke-jr> sdaftuar_: then miners would need to set blockmaxsize to 250k to avoid making blocks >1M
687 2016-07-01T20:11:55  <gmaxwell> 06:43 < MarcoFalke> gmaxwell: I don't think lack of progress in improving of coinselection is due to the test hardcoding the behavior.
688 2016-07-01T20:12:01  <luke-jr> the goal of blockmaxsize is to avoid blocks larger than the size. cost/vsize doesn't matter here
689 2016-07-01T20:12:20  <gmaxwell> I can tell you that personally I've written improvements that I haven't PRed because they required throwing out all the existing tests.
690 2016-07-01T20:12:21  <sdaftuar_> My preference was to introduce a new option that would control serialized bytes
691 2016-07-01T20:12:51  <luke-jr> sdaftuar_: why a new oppppption when we have one for it already?
692 2016-07-01T20:13:27  <sdaftuar_> Because the mining code doesn't optimize for it
693 2016-07-01T20:13:40  <sdaftuar_> So it's misleading to suggest it's supported
694 2016-07-01T20:14:05  <luke-jr> it does what it's supposed to do.
695 2016-07-01T20:14:08  <gmaxwell> I don't think it makes sense to even have a seralized size setting, we don't have a setting to limit the size of the sighashed bytes, or a setting to limit the size of the block's CTransaction encoding.  We don't have a limit to control the size of the compactblock form, or a limit to control the size of a zlib compressed block.
696 2016-07-01T20:14:46  <gmaxwell> But I don't bother complaining because luke was going pyric over this before, and it shouldn't hurt that much having it either.
697 2016-07-01T20:14:51  <luke-jr> gmaxwell: at this time, serialised size is a critical factor in practice
698 2016-07-01T20:15:22  <sdaftuar_> gmaxwell: +1
699 2016-07-01T20:15:32  <gmaxwell> luke-jr: critical for what? it doesn't reflect block propagation time especially well.
700 2016-07-01T20:15:58  <luke-jr> on the p2p network it sure does..
701 2016-07-01T20:16:06  <gmaxwell> validation time is more closely related to the number of utxo operations and how well relayed the txn in hte block were.
702 2016-07-01T20:16:24  <gmaxwell> luke-jr: we have BIP152 now.
703 2016-07-01T20:16:36  <luke-jr> not deployed, unfortunately.
704 2016-07-01T20:17:14  <luke-jr> to be clear, I'm all for removing blockmaxsize once it doesn't matter and isn't used.
705 2016-07-01T20:17:23  <luke-jr> hence why I don't think optimising block creation for it is important
706 2016-07-01T20:17:28  <gmaxwell> (but even without it, seralized size is still not the ideal predictor of propagation time.)
707 2016-07-01T20:18:20  <luke-jr> I just worry that we'll end up with >1 MB blocks before the network can handle it sanely
708 2016-07-01T20:18:20  <gmaxwell> K. indeed 0.12 doesn't have BIP152. Perhaps that does suggest that it should default to four million in 0.13?
709 2016-07-01T20:18:27  <gmaxwell> luke-jr: thats inevitable...
710 2016-07-01T20:18:48  <gmaxwell> see also the utxo comments by wumpus last night.
711 2016-07-01T20:24:36  *** Cats2 is now known as ClockCat
712 2016-07-01T20:30:13  *** zooko has quit IRC
713 2016-07-01T20:47:39  <sturles> Am I correct that the 0.13 HD wallet implementation will only support new wallets, and only support hardened keys?
714 2016-07-01T20:47:49  <gmaxwell> sturles: correct.
715 2016-07-01T20:48:09  <sturles> Will it support importing private keys from an old style wallet?
716 2016-07-01T20:48:37  <gmaxwell> I believe it does, I've not actually tested that!
717 2016-07-01T20:53:27  <sturles> A HD wallet won't work with older versions, right?  Should use the opportunity to switch DB for the wallet to something > libdb 4.8?
718 2016-07-01T20:54:07  <gmaxwell> Do you want software that never makes progress? Thats how you get software that never makes progress.
719 2016-07-01T20:54:45  <gmaxwell> It can't use a later libdb or it will make all non-HD wallets also unreadable by other versions.
720 2016-07-01T20:55:45  <helo> i believe a hd wallet should work with older versions
721 2016-07-01T20:56:18  <gmaxwell> helo: hows that?
722 2016-07-01T20:56:44  <sturles> Yes, I know.  I guess it is impossible to use different versions depending on wallet type, e.g. via dlopen?
723 2016-07-01T20:57:16  <gmaxwell> sturles: in theory possible but that would be a lot of work without any benefit.
724 2016-07-01T20:57:31  <helo> it will work to a limited extent... hd keys already added will be visible
725 2016-07-01T20:57:48  <helo> untested, ofc
726 2016-07-01T20:57:59  <gmaxwell> if so, thats a bug. it should get rejected due to having a too new version.
727 2016-07-01T20:58:02  <Lightsword> can we just migrate to sqlite?
728 2016-07-01T20:58:09  <gmaxwell> if it isn't then you could put yourself in a weird state.
729 2016-07-01T20:58:18  <gmaxwell> I will have to start stabbing people.
730 2016-07-01T20:59:22  <helo> nah, that's not necessary. the version check probably functions as intended :)
731 2016-07-01T20:59:40  <gmaxwell> hah
732 2016-07-01T21:01:33  <sturles> I'd say it is benificial to switch to a newer, supported version of libdb
733 2016-07-01T21:02:42  <gmaxwell> the latest versions have incompatible licenses.
734 2016-07-01T21:04:39  <sturles> Oh.
735 2016-07-01T21:05:05  <gmaxwell> there is newer than 4.8 that is okay, but the latest stuff has a much stronger copyleft than we'd normally use.
736 2016-07-01T21:07:19  <gmaxwell> really the 'database' isn't actually used for much--- the wallet is largely entirely in memory, and the database is only really used for persistance.
737 2016-07-01T21:08:52  *** murch has quit IRC
738 2016-07-01T21:09:36  *** murch has joined #bitcoin-core-dev
739 2016-07-01T21:14:07  <luke-jr> eh, -walletupgrade won't work? O.o
740 2016-07-01T21:30:31  *** afk11 has quit IRC
741 2016-07-01T21:36:54  <Lightsword> would working on migrating the wallet to sqlite be something worthwhile at this point?
742 2016-07-01T21:38:16  *** afk11 has joined #bitcoin-core-dev
743 2016-07-01T21:38:16  *** afk11 has quit IRC
744 2016-07-01T21:38:16  *** afk11 has joined #bitcoin-core-dev
745 2016-07-01T21:39:50  <CubicEarth> Block validation is, theoretically, very amenable to parallelization, correct?  Is the main serial component the depth of the merkel tree?
746 2016-07-01T21:39:52  *** murch has quit IRC
747 2016-07-01T21:42:09  *** moli has joined #bitcoin-core-dev
748 2016-07-01T21:43:50  <gmaxwell> signature validation is, and bitcoin core runs that in paralle... but normally almost all of the signatures in a block are already validated before it shows up.
749 2016-07-01T21:44:19  *** molz has quit IRC
750 2016-07-01T21:44:22  <gmaxwell> parallel*
751 2016-07-01T21:44:27  <gmaxwell> the general database handling ends up taking much of the time, which isn't really paralizable.
752 2016-07-01T21:48:37  *** mkarrer_ has quit IRC
753 2016-07-01T21:51:01  *** sdaftuar1 has joined #bitcoin-core-dev
754 2016-07-01T21:51:18  *** ybit_ has joined #bitcoin-core-dev
755 2016-07-01T21:51:28  *** OxADADA_ has joined #bitcoin-core-dev
756 2016-07-01T21:51:32  *** trippysalmon has joined #bitcoin-core-dev
757 2016-07-01T21:52:20  *** mkarrer has joined #bitcoin-core-dev
758 2016-07-01T21:53:06  *** tadasv has quit IRC
759 2016-07-01T21:53:10  *** [b__b] has quit IRC
760 2016-07-01T21:53:12  *** sdaftuar has quit IRC
761 2016-07-01T21:53:12  *** OxADADA has quit IRC
762 2016-07-01T21:53:12  *** trippysa1mon has quit IRC
763 2016-07-01T21:53:12  *** ybit has quit IRC
764 2016-07-01T21:53:13  *** sdaftuar_ is now known as sdaftuar
765 2016-07-01T21:53:37  *** tadasv has joined #bitcoin-core-dev
766 2016-07-01T21:54:03  *** [b__b] has joined #bitcoin-core-dev
767 2016-07-01T21:55:05  *** luke-jr has quit IRC
768 2016-07-01T22:01:11  *** luke-jr has joined #bitcoin-core-dev
769 2016-07-01T22:05:09  <CubicEarth> gmaxwell: so thinking about the initial chain sync and validation, it could (with lots and lots of careful work) be made to harness GPU's, and spread the signature validation across *many* cores?  I'm not familiar with the nature of the general database handling you are referring to, but for the initial sync is that mainly keeping track of the utxo's as it increments tough the blocks?
770 2016-07-01T22:05:32  *** Chris_Stewart_5 has quit IRC
771 2016-07-01T22:07:53  <gmaxwell> I've seen no reason to believe that a gpu would help at all, GPU cores aren't particularly good at the work of validating. (64 bit arithmetic helps a lot!)
772 2016-07-01T22:08:26  <gmaxwell> and nodes don't even verify the signatures far back in the chain.
773 2016-07-01T22:08:31  *** luke-jr has quit IRC
774 2016-07-01T22:08:39  *** luke-jr has joined #bitcoin-core-dev
775 2016-07-01T22:09:44  <gmaxwell> but sure more work could be done to extract more parallelism out of it.
776 2016-07-01T22:11:16  <gmaxwell> but even with signature validation turned off completely, with default settings a reindex takes almost 9 hours currently.
777 2016-07-01T22:13:02  <gmaxwell> increasing the db cache to a really huge size (such that the sync runs entirely out of ram) lets the whole sync _with_ signature checking run in about 3.5 hours.
778 2016-07-01T22:14:04  *** zooko has joined #bitcoin-core-dev
779 2016-07-01T22:14:43  *** kadoban has joined #bitcoin-core-dev
780 2016-07-01T22:16:12  <CubicEarth> Is the default 100MB?  I never knew that!!  I'll be curious too see what kind of speedup I can achieve on an i5 - dual core.  Are we talking 100GB of ram here, or would setting it to 4 or 8 GB make a substantial difference?
781 2016-07-01T22:17:11  <sipa> setting it to 2 GB is more than sufficient
782 2016-07-01T22:17:32  <sipa> higher numbers don't matter much
783 2016-07-01T22:18:38  *** Chris_Stewart_5 has joined #bitcoin-core-dev
784 2016-07-01T22:19:01  <CubicEarth> Is it a rolling-window thing?
785 2016-07-01T22:19:17  <sipa> no
786 2016-07-01T22:19:23  <sipa> it's a cache
787 2016-07-01T22:19:32  <sipa> when the cache is full, we write it to disk
788 2016-07-01T22:19:38  <sipa> and start over
789 2016-07-01T22:19:38  <sipa> very silly
790 2016-07-01T22:22:10  <gmaxwell> I think you actually need more than 2GB now to get full performance, but 8GB is more than enough.
791 2016-07-01T22:22:31  <midnightmagic> maybe an iovec scatter write might help some systems write faster
792 2016-07-01T22:24:24  <midnightmagic> or is it just a plain sequential write thing and the rest is leveldb
793 2016-07-01T22:24:40  <CubicEarth> Well it's easy enough to set it higher.  I just never knew.  A default of 100MB can make sense for older systems, but if there was place for 'performance tips', that would be a good place to let people know.  Maybe everyone already (node operators) knows except for me.
794 2016-07-01T22:25:15  <gmaxwell> I think a lot of people don't realize what a big difference it makes.
795 2016-07-01T22:25:46  <gmaxwell> looks like we'll increase the default to 300MB.
796 2016-07-01T22:26:46  <gmaxwell> wumpus: on the leveldb stuff, without txindex, I'd suggest making that 4mb instead of 2. 4MB was something like 1% faster for me when testing leveldb in isolation.
797 2016-07-01T22:27:29  <gmaxwell> maybe 2 is a win at the 300mb limit, but with it bumped 4mb would be... and probably is more conservative overall.
798 2016-07-01T22:31:03  *** sdaftuar has quit IRC
799 2016-07-01T22:35:30  *** pedrobranco has joined #bitcoin-core-dev
800 2016-07-01T22:36:23  *** Guyver2 has quit IRC
801 2016-07-01T22:39:42  *** pedrobranco has quit IRC
802 2016-07-01T22:39:57  <CubicEarth> I did a fresh sync a week ago, 2013-era i5 dual core laptop.  Ubuntu, 4GB of ram, spinning HD.  75Mbps cable internet.  It took about 20 hours to finish.  What was interesting though was how the last 20 weeks just crawled.  Seemed less than linear, even estimating about the effects of fuller blocks.
803 2016-07-01T22:42:35  <sipa> yeah, try dbcache 4000 or so
804 2016-07-01T22:43:17  <CubicEarth> after I order more ram :)
805 2016-07-01T22:47:42  <CubicEarth> gmaxwell: earlier you wrote "and nodes don't even verify the signatures far back in the chain.".  That's due to checkpoints, right?  They don't seem like a good thing to rely on.  I understand their practicality, but yeah.
806 2016-07-01T22:49:16  <CubicEarth> I'm just trying to understand the theoretical performance limits... I'm not crusading against checkpoints
807 2016-07-01T22:52:44  <gmaxwell> well you'd be welcome here to do that, I want them gone and have for years.
808 2016-07-01T22:53:51  <gmaxwell> But even with them gone, one can avoid skipping signature validation during the initial sync for blocks burried by thousands of additional blocks with negligible risk-- consider, if miners went rogue enough to reorg thousands of blocks the system is already screwed... and if its only during initial sync they'd only trick new nodes in any case.
809 2016-07-01T22:54:10  <gmaxwell> if it makes a difference between a user running a node or not-- the best choice is clear.
810 2016-07-01T22:54:54  <gmaxwell> Meanwhile bitcoinxt and classic have disabled signature checking for any block where the miner supplied timestamp is a day back or more... meaning they can be fooled without a reorg at all. And no one seems to care.
811 2016-07-01T22:56:57  <CubicEarth> Arn't miners only afforded the option to have their timestamp be off by more than 2-hours?
812 2016-07-01T22:57:13  <CubicEarth> (as an aside)
813 2016-07-01T22:57:28  <gmaxwell> no, not in the past direction. In the future direction yes.
814 2016-07-01T22:58:05  <gmaxwell> The past direction is limited by the median of the last 11 blocks-- but that median can be arbritarily far back.
815 2016-07-01T22:59:42  <CubicEarth> So practically speaking, the way to detect the issue is when the client said it was 'synced' and you noticed that is was reporting yesterdays date?
816 2016-07-01T22:59:56  <midnightmagic> ..  did they actually commit that change? the >24hr non-t-verify change?
817 2016-07-01T23:01:32  <gmaxwell> CubicEarth: no, because miners can simply add a correctly timed block immediately after.
818 2016-07-01T23:02:15  *** zooko has quit IRC
819 2016-07-01T23:02:55  <CubicEarth> That would appear as if the hashrate had fallen precipitously... not blocks for 24hr!
820 2016-07-01T23:03:22  <gmaxwell> midnightmagic: XT did and released, it's merged in classic, but I don't know if it made it into a release yet.
821 2016-07-01T23:03:31  <gmaxwell> These folks are extremely and dangerously incompetent.
822 2016-07-01T23:04:03  <gmaxwell> CubicEarth: no, just a wonky timestamp, miners provide the timestamps and sometimes they're pretty wonky.
823 2016-07-01T23:04:30  <gmaxwell> and while _syncing_ you have no idea that the timestamps were wonky.
824 2016-07-01T23:05:19  <gmaxwell> e.g. someone can produce blocks that are -24h, -23h, -22h .... normal.
825 2016-07-01T23:05:23  <midnightmagic> so.. the peer preference thing means some of the alt-branch software will soon start disconnecting the only nodes that are immune to the attack, making themselves *more* vulnerable to it?
826 2016-07-01T23:07:30  <gmaxwell> midnightmagic: I don't think that matters that much, but right now bitcoin "xt" is only not totally partitioned due to inbound connections from core to XT, but part of segwit is that segwit capable nodes will strongly prefer to connect out to only non-segwit nodes (because they must in order to get blocks once segwit is activated)-- the result will be making that partition complete.
827 2016-07-01T23:08:43  <midnightmagic> i thought -classic was now disconnecting non-classic nodes
828 2016-07-01T23:09:22  <gmaxwell> no, Xt. I don't think classic is doing that yet.
829 2016-07-01T23:10:16  <slackircbridge1> <alp> let them lose money for people
830 2016-07-01T23:10:19  <slackircbridge1> <alp> its the only way theyll learn
831 2016-07-01T23:10:37  <gmaxwell> It's not that simple. See also: Ethereum and DAO.
832 2016-07-01T23:11:23  <gmaxwell> which adversely impacted the bitcoin price even though many of us have been pointing out the related risks and distinguishing bitcoin on the basis of them for some time.
833 2016-07-01T23:13:18  <slackircbridge1> <brg444> :O
834 2016-07-01T23:13:23  <CubicEarth> gmaxwell: well I'm pretty sure the rise of ETH was hurting the bitcoin price to some extent.  I think this event was a decoupling, where Bitcoin shakes the monkey off its back.
835 2016-07-01T23:13:31  <slackircbridge1> <brg444> did he read your post :O
836 2016-07-01T23:16:03  *** jannes has quit IRC
837 2016-07-01T23:24:28  <slackircbridge1> <alp> Short term pain, long term gain
838 2016-07-01T23:27:56  <slackircbridge1> <moli> guys, we can read your posts on IRC lol
839 2016-07-01T23:28:02  *** ChanServ sets mode: +o btcdrak
840 2016-07-01T23:32:09  *** Chris_Stewart_5 has quit IRC
841 2016-07-01T23:32:30  *** btcdrak sets mode: +scirg 
842 2016-07-01T23:32:31  *** ChanServ sets mode: -c 
843 2016-07-01T23:33:10  <btcdrak> wait that's wrong. I'm trying to set slackircbridge1 +q
844 2016-07-01T23:33:59  <gmaxwell> lol
845 2016-07-01T23:34:03  <gmaxwell> undo the damage first.
846 2016-07-01T23:36:16  *** btcdrak sets mode: +o gmaxwell
847 2016-07-01T23:37:02  *** gmaxwell sets mode: -sirg 
848 2016-07-01T23:38:11  *** gmaxwell sets mode: +q slackircbridge*!*@*
849 2016-07-01T23:38:31  *** btcdrak sets mode: -o btcdrak
850 2016-07-01T23:42:03  *** gmaxwell sets mode: -o gmaxwell