12016-07-01T00:00:19 *** ybit has quit IRC
22016-07-01T00:01:32 *** ybit has joined #bitcoin-core-dev
32016-07-01T00:02:43 *** frankenmint has joined #bitcoin-core-dev
42016-07-01T00:03:30 *** cryptapus_afk is now known as cryptapus
52016-07-01T00:05:47 *** molz has joined #bitcoin-core-dev
62016-07-01T00:06:07 *** Retric has joined #bitcoin-core-dev
72016-07-01T00:08:34 *** moli has quit IRC
82016-07-01T00:18:42 <sdaftuar> oddly, travis doesn't appear to have run #8295
92016-07-01T00:18:46 *** cryptapus is now known as cryptapus_afk
102016-07-01T00:38:16 *** frankenmint has quit IRC
112016-07-01T00:43:59 *** TheFactory7 has quit IRC
122016-07-01T00:51:44 *** zooko has joined #bitcoin-core-dev
132016-07-01T00:55:54 *** fengling has joined #bitcoin-core-dev
142016-07-01T00:58:36 *** pedrobranco has joined #bitcoin-core-dev
152016-07-01T01:02:38 *** PRab has quit IRC
162016-07-01T01:03:24 *** pedrobranco has quit IRC
172016-07-01T01:12:35 *** xiangfu has joined #bitcoin-core-dev
182016-07-01T01:32:23 *** face has joined #bitcoin-core-dev
192016-07-01T01:35:46 *** Ylbam has quit IRC
202016-07-01T01:36:59 *** BCBot has joined #bitcoin-core-dev
212016-07-01T01:36:59 *** bsm1175321 has joined #bitcoin-core-dev
222016-07-01T01:36:59 *** Taek has joined #bitcoin-core-dev
232016-07-01T01:38:33 <luke-jr> fwiw, there appears to be no changes affecting bitcoin-cli between 0.12.0 and 0.12.1
242016-07-01T01:40:19 *** belcher has quit IRC
252016-07-01T01:53:04 *** cryptapus_afk is now known as cryptapus
262016-07-01T01:55:53 *** cryptapus is now known as cryptapus_afk
272016-07-01T02:08:38 *** wangchun has quit IRC
282016-07-01T02:09:14 *** wangchun has joined #bitcoin-core-dev
292016-07-01T02:15:28 *** Chris_Stewart_5 has quit IRC
302016-07-01T02:16:24 *** Chris_Stewart_5 has joined #bitcoin-core-dev
312016-07-01T02:20:20 *** adiabat has joined #bitcoin-core-dev
322016-07-01T02:38:37 *** Chris_Stewart_5 has quit IRC
332016-07-01T03:30:59 *** adiabat has left #bitcoin-core-dev
342016-07-01T03:49:34 *** pedrobranco has joined #bitcoin-core-dev
352016-07-01T03:54:21 *** pedrobranco has quit IRC
362016-07-01T03:57:12 *** fengling has quit IRC
372016-07-01T04:14:34 *** afk11 has quit IRC
382016-07-01T04:19:31 *** afk11 has joined #bitcoin-core-dev
392016-07-01T04:19:31 *** afk11 has quit IRC
402016-07-01T04:19:31 *** afk11 has joined #bitcoin-core-dev
412016-07-01T04:52:23 *** Guest27612 has joined #bitcoin-core-dev
422016-07-01T04:52:38 *** Guest27612 is now known as roidster
432016-07-01T04:59:27 *** kadoban has quit IRC
442016-07-01T04:59:42 <gmaxwell> Can anyone think of why we'd get a "CommitTransaction(): Error: Transaction not valid" on a new transaction being created that paid twice the nodes minrelay fee, ... and then managed to broadcast when the rebroadcast ran?
452016-07-01T04:59:47 <gmaxwell> it looks like it was spending an unconfirmed input.
462016-07-01T05:10:31 <gmaxwell> okay I think the issue was he managed to make a 25 deep unconfirmed chain, and got the last txn in it rejected.
472016-07-01T05:10:38 <gmaxwell> Doesn't look like the wallet handles that well.
482016-07-01T05:13:00 <gmaxwell> I think we should avoid using coins that are at the unconfirmed limit maximum unless not otherwise possible.
492016-07-01T05:13:16 <gmaxwell> Also ooRBF to sendmany would have eliminated 25 transactions here.
502016-07-01T05:20:41 *** zooko has quit IRC
512016-07-01T05:26:34 *** Guyver2 has joined #bitcoin-core-dev
522016-07-01T05:41:53 *** Ylbam has joined #bitcoin-core-dev
532016-07-01T06:17:27 *** Guyver2 has quit IRC
542016-07-01T06:38:18 *** Inaltoas1nistra has joined #bitcoin-core-dev
552016-07-01T06:40:25 *** Inaltoasinistra has quit IRC
562016-07-01T06:48:12 *** roidster has quit IRC
572016-07-01T07:10:05 *** Ginnarr has joined #bitcoin-core-dev
582016-07-01T07:11:51 *** owowo has quit IRC
592016-07-01T07:13:43 *** cjcj has joined #bitcoin-core-dev
602016-07-01T07:52:44 <GitHub68> [bitcoin] laanwj pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/6a87eb0e4b47...da50997a3ee7
612016-07-01T07:52:45 <GitHub68> bitcoin/master 0ce8e99 Wladimir J. van der Laan: windows: Add testnet link to installer
622016-07-01T07:52:45 <GitHub68> bitcoin/master 975a41d Wladimir J. van der Laan: windows: Add testnet icon for testnet link...
632016-07-01T07:52:46 <GitHub68> bitcoin/master da50997 Wladimir J. van der Laan: Merge #8285: windows: Add testnet link to installer...
642016-07-01T07:52:52 <GitHub129> [bitcoin] laanwj closed pull request #8285: windows: Add testnet link to installer (master...2016_06_testnet_link_windows) https://github.com/bitcoin/bitcoin/pull/8285
652016-07-01T07:55:41 *** MarcoFalke has joined #bitcoin-core-dev
662016-07-01T08:10:00 <instagibbs> gmaxwell, yes the maximum ancestor/descendant stuff means that the transaction will be marked invalid and the funds deducted from your wallet until rescan :/
672016-07-01T08:10:08 <instagibbs> err reindex or something
682016-07-01T08:11:32 <gmaxwell> instagibbs: in this case the transaction when through after the parents confirmed.
692016-07-01T08:11:47 <gmaxwell> the wallet rebroadcast accepted it the next time around and tada.
702016-07-01T08:12:09 <gmaxwell> probably the coinselection process, when it considers 0conf inputs should only consider thoes with a low enough depth.
712016-07-01T08:12:40 <gmaxwell> (perhaps two below the maximum to leave room for CPFP), and only if it's unable to meet that should it fall back to considering all inputs.
722016-07-01T08:14:39 <instagibbs> gmaxwell, oh so even in failure it rebroadcasts?
732016-07-01T08:14:59 <gmaxwell> yea, it saves it in the wallet before it tries to mempool it.
742016-07-01T08:15:07 <gmaxwell> and since its saved, it keeps trying to rebroadcast.
752016-07-01T08:15:08 <wumpus> as long as the transaction is in the wallet it rebroadcasts
762016-07-01T08:15:49 <gmaxwell> I dunno if the depth of an unconfirmed coin is easily discernable though from the wallet.
772016-07-01T08:20:31 <wumpus> not at is, it would need another (ugly) mempool dependency
782016-07-01T08:20:34 *** Giszmo has quit IRC
792016-07-01T08:21:05 <gmaxwell> bleh
802016-07-01T08:22:48 <wumpus> then again we already do an InMempool() check in IsTrusted, which determines whether the outputs of an transaction are considered spendable, so...
812016-07-01T08:23:16 <wumpus> if looking up the depth in the mempool is relatiely cheap that could be added too
822016-07-01T08:24:17 <wumpus> but yes I agree with the 'bleh' sentiment
832016-07-01T08:24:45 <wumpus> ideally mempool implementation details shouldn't matter to the wallet
842016-07-01T08:25:03 <wumpus> "if it cannot be submitted yet due to depth, try again later" seems a better approach
852016-07-01T08:25:28 <wumpus> the wallet already tries to avoid generating chains of unconfirmed
862016-07-01T08:25:33 <wumpus> so if it does, it really needs to
872016-07-01T08:26:35 <gmaxwell> it doesn't really: so it tries to avoid spending unconfirmed coins, but if it must it treats short and long chains the same.
882016-07-01T08:26:52 <gmaxwell> So in the example with the user I was supposting today, they had plenty of unconfirmed coins that were only 1 deep.
892016-07-01T08:27:16 <wumpus> yes I don't mean to imply that it looks at the mempool depth, it obviously doesn't
902016-07-01T08:27:19 <gmaxwell> and one chain of 25.
912016-07-01T08:27:58 <wumpus> yes that would make sense - count *unconfirmed* depth
922016-07-01T08:28:08 <wumpus> then prefer coins that are shallow
932016-07-01T08:28:17 <gmaxwell> it ended up that way because they had multiple unconfirmed few-bitcoin outputs, and then a 1 bitcoin output and were making many not huge payments... so it decided to keep reusing the change because it considered them all equal.
942016-07-01T08:28:19 <wumpus> that wouldn't require asking the mempool at all
952016-07-01T08:28:49 <gmaxwell> yea, don't' really care about the mempool, the ismine tracing could count the depth until confirmed I guess.
962016-07-01T08:29:13 <wumpus> coins with a smaller unconfirmed depth could get some advantage in selection
972016-07-01T08:29:40 <wumpus> another thing in the long list of things that could be better in coin selection
982016-07-01T08:31:54 <gmaxwell> well the way to do the advantage is the same way it prefers confirmed coins.
992016-07-01T08:32:16 <gmaxwell> try the selection first with any too-deep coins excluded, and only if it fails attempt without that restriction.
1002016-07-01T08:32:37 <wumpus> that's a possiblity, yes
1012016-07-01T08:33:20 *** jtimon has joined #bitcoin-core-dev
1022016-07-01T08:36:18 <wumpus> though at some point the 'exclude these coins and do the selection again' list would become very long, and is less suited for non-boolean properties like input size (https://github.com/bitcoin/bitcoin/issues/7664)
1032016-07-01T08:36:38 <wumpus> but sure, this could be bolted on too :)
1042016-07-01T08:37:32 <gmaxwell> well, it fits here since the failure results in an (temporarily) invalid transaction. ... just leaving them out is the right call for max depth, at least.
1052016-07-01T08:37:33 <wumpus> I think at some point it would be nice to have a per-coin scoring system, then make the coin selection use that
1062016-07-01T08:38:08 <gmaxwell> yes, for some things, but a score for this should be infinite unless there is no other choice... since it guarentees the txn will not relay until its ancestors confirm some. :)
1072016-07-01T08:38:30 <wumpus> but you make assumptions about other people's mempool depth
1082016-07-01T08:38:41 <wumpus> I think a better approach would just be to *prefer* shallower transactions
1092016-07-01T08:38:50 <wumpus> without an absolute theshold
1102016-07-01T08:39:21 <gmaxwell> They're not exclusive, prefer shorter but absolute threshold at the point where you won't relay the thing yourself anymore.
1112016-07-01T08:39:53 <gmaxwell> this usecase would actually have been better off with some kind of replacement, as they could have saved a good 20 transactions.
1122016-07-01T08:39:55 <wumpus> and if it does manage to generates a temporarily invalid transaction, handle that more friendly, e.g. a warning instead of an error
1132016-07-01T08:41:30 <wumpus> having the wallet have a hard dependency on a property of the mempool seems really brittle
1142016-07-01T08:42:12 <wumpus> and it's not a solution that wallets that don't have such a close coupling to a node could use
1152016-07-01T08:42:13 <gmaxwell> I don't think this is really a mempool dependency.
1162016-07-01T08:42:35 *** fengling has joined #bitcoin-core-dev
1172016-07-01T08:42:43 <wumpus> in principle it's a matter of optimizing the speed at which the transaction can be submitted, longer chains result in slower confirmation times
1182016-07-01T08:42:54 <wumpus> very long chains even in temporary rejection
1192016-07-01T08:42:59 <gmaxwell> If it's spending here the transaction is ismine, which means all unconfirmed ancestors are wallet txn.
1202016-07-01T08:43:10 <gmaxwell> So it just needs to know the longest depth.
1212016-07-01T08:43:13 <wumpus> but long chains, even if they are accepted, are not a good thing in itself
1222016-07-01T08:44:33 <gmaxwell> (er not IsMine, but IsFromMe)
1232016-07-01T08:44:45 *** jannes has joined #bitcoin-core-dev
1242016-07-01T08:46:04 <gmaxwell> wumpus: they're not that bad, esp with ancestor feerate mining; and relay being not totally broken for them in 0.13.
1252016-07-01T08:46:28 <wumpus> but I hope you agree that they are worse than shallow transaction chains
1262016-07-01T08:46:46 *** fengling has quit IRC
1272016-07-01T08:46:48 <wumpus> 'not that bad' yes there are always much worse things :)
1282016-07-01T08:46:51 <gmaxwell> all things equal... but compared to a choice that is less private or pays more fees?
1292016-07-01T08:47:16 <wumpus> well the individual scoring weights would have to be determined, yes, or even depend on some setting
1302016-07-01T08:47:51 <wumpus> paying more fees may be ok if it means the transaction has a larger chance of being picked
1312016-07-01T08:47:53 <gmaxwell> because we never split large change output, a lot of usage patterns result in no real choice in any case.
1322016-07-01T08:48:36 <wumpus> less private, well long transaction chains are a great way to say 'hey this is a wallet sending out transactions serially'
1332016-07-01T08:49:58 <gmaxwell> sure, but often better to just respend change than join a dozen otherwise unrelated inputs.
1342016-07-01T08:51:14 <wumpus> if it requires joining them, yes I agree
1352016-07-01T08:52:55 * wumpus wishes someone would do a serious study about what wallet behavior would make sense
1362016-07-01T08:53:04 *** AaronvanW has quit IRC
1372016-07-01T08:53:12 *** fengling has joined #bitcoin-core-dev
1382016-07-01T08:55:45 <gmaxwell> Would we do anything with it?... a while back someone proposed a change (to remove extranious inputs), I suggested that it might result in wallets grinding down coins into small amouts more often. He made a simulator that showed it would, then we took the change. Then later users showed up complaining about the wallet grinding down inputs when they didn't use to...
1392016-07-01T08:56:00 *** AaronvanW has joined #bitcoin-core-dev
1402016-07-01T08:56:39 <wumpus> well the problem is that we're too busy running from issue to issue to look at a higher level
1412016-07-01T08:56:45 <wumpus> well at least I am
1422016-07-01T08:56:53 <gmaxwell> right. I agree with that.
1432016-07-01T08:57:12 <gmaxwell> just collectively we are.
1442016-07-01T08:57:29 <wumpus> so now long chains are an issue, long chains are fixed by adding yet another special case, but without considering impact on other things
1452016-07-01T08:57:47 <gmaxwell> You're right that someone looking at it more holistically would be good, part of the problem in that issue was that even there it was only asking a very narrow question.
1462016-07-01T08:57:53 <wumpus> at some point I just worry all those hacks make things worse, instead of picking some simple algorithm that does fine in most cases
1472016-07-01T08:58:13 <gmaxwell> I don't really think there is much to consider on the subject of "avoid going over your own maximum chain depth if at all possible".
1482016-07-01T08:58:16 <wumpus> but I don't know, I don't have the ovierview
1492016-07-01T08:58:36 <gmaxwell> since producing a txn even you won't relay when you could have avoided it seems clearly wrong.
1502016-07-01T08:58:50 <phantomcircuit> gmaxwell: the wallet should probably have all of the unconfirmed depdencies of a transaction
1512016-07-01T08:59:07 <wumpus> so dropping of extraneous inputs was bad?
1522016-07-01T08:59:08 <phantomcircuit> and then remove the dependencies after some high number of confirms which aren't relevant
1532016-07-01T09:00:06 <gmaxwell> phantomcircuit: in these cases all the unconfirmed dependencies will be wallet transactions-- IsFromMe otherwise it won't spend the coin!
1542016-07-01T09:00:07 <wumpus> normally you'd say that choosing a minimum set that covers the value to be spent would be optimal
1552016-07-01T09:00:09 <phantomcircuit> but yeah you're right it has to pass IsMine which means the wallet already has all the info you need to calculate depth with just coins view
1562016-07-01T09:00:38 <wumpus> but yes, I'm sure there are lots of other contraints and scores that should be taken in to account
1572016-07-01T09:00:38 <phantomcircuit> reading top down so my comments might be already worked out :P
1582016-07-01T09:00:40 <gmaxwell> wumpus: the result ends up being the smallest possible change. so it breaks your wallet into lots of tiny little coins.
1592016-07-01T09:00:41 <wumpus> which was my point before
1602016-07-01T09:00:48 *** molly has joined #bitcoin-core-dev
1612016-07-01T09:01:43 <wumpus> so why wasn't that reverted?
1622016-07-01T09:02:03 <gmaxwell> I don't know/understand why it wasn't.
1632016-07-01T09:02:46 <gmaxwell> I think when it went in there was some expectation that further improvements would come, and they didn't. Then it was released. Then people showed up and complaints (opening an issue) and we figured out that the behavior change was due to removing extranious inputs.. and then? I don't know
1642016-07-01T09:02:52 <wumpus> we just have too much on our plate
1652016-07-01T09:02:55 <gmaxwell> then something else caught fire.
1662016-07-01T09:03:11 <wumpus> sometimes I really want to run around screaming with my hands on my head
1672016-07-01T09:03:43 <phantomcircuit> wumpus: that sounds like fun
1682016-07-01T09:03:48 <wumpus> it's not possible to handle this all anymore
1692016-07-01T09:04:17 *** molz has quit IRC
1702016-07-01T09:04:29 <wumpus> we really need someone that focuses on wallet improvements
1712016-07-01T09:04:31 <gmaxwell> (privacy wants you to spend all payments to a single address at once, so that you don't get a rolling linkage that eventually cross taints every address in your wallet--- so maybe just attempting to do this would counteract most of the grinding. thats the kind of thing you want tne step back and overview to answer)
1722016-07-01T09:05:31 *** Retric has quit IRC
1732016-07-01T09:06:01 <wumpus> someone that has an overview of what the heck is happening in the wallet
1742016-07-01T09:06:01 <gmaxwell> There are a number of positive wallet things going on, at least. But there is a lot of time spent twiddling things in the weeds rather than setting up prioties and identifying larger scale issues.
1752016-07-01T09:06:12 <wumpus> but how to sort 'bad' improvements from good ones?
1762016-07-01T09:06:22 <wumpus> I don't know anymore
1772016-07-01T09:07:40 <phantomcircuit> i've been trying to improve the wallet
1782016-07-01T09:07:55 <wumpus> yes, thanks for that
1792016-07-01T09:08:14 <phantomcircuit> unfortunately yeah it's a lot of running around in the weeds trying to fix things up... to make it easier to do bigger things
1802016-07-01T09:08:30 <wumpus> all in all the utxo/coin based approach does cause a lot of non-trivial difficulties, both at the node (utxo set sprawl) level as in the wallet
1812016-07-01T09:08:38 <gmaxwell> in any case, I didn't bring any of this up here to complain-- I brought it up because initially I couldn't figure out the cause, and then updated when I did.
1822016-07-01T09:09:31 <gmaxwell> it avoids a lot of replay problems however, which avoids other kinds of sprawl
1832016-07-01T09:09:49 <wumpus> no, I'm not trying to complain either, but sometimes I just don't know anymroe
1842016-07-01T09:10:26 *** fengling has quit IRC
1852016-07-01T09:10:30 *** paveljanik has joined #bitcoin-core-dev
1862016-07-01T09:10:30 *** paveljanik has joined #bitcoin-core-dev
1872016-07-01T09:10:51 <gmaxwell> well, from another perspective-- we have no deadline. These same issues existed in 2011 (actually, much worse), but it didn't matter because hardly anyone used the system. :)
1882016-07-01T09:11:15 <wumpus> but that's part of what worries me, yes these same issues existed in 2011
1892016-07-01T09:11:58 <wumpus> no, no deadline, but e.g. the utxo growth is worrying
1902016-07-01T09:12:02 <wumpus> we're running against (soft) walls
1912016-07-01T09:12:10 <gmaxwell> yes, I'm concerned about utxo growth.
1922016-07-01T09:12:36 <gmaxwell> I happened to chart data from all those reindexing tests I just did, and the time to verify a block is increasing (for the same amount of txn data).
1932016-07-01T09:13:15 <wumpus> and all the while joe user is complaining about lack of *scaling*, while the system is already seemingly bursting at its seems
1942016-07-01T09:13:24 <gmaxwell> I was theorizing that this was from polylog behavior in the database and worrying, but phantomcircuit gave an alternative argument that the reduction in spammy transactions relative to non-spammy ones may be resulting in lower cache hitrates.
1952016-07-01T09:13:51 <gmaxwell> so I'm going to try to figure out if thats the case.
1962016-07-01T09:14:38 <gmaxwell> (by increasing, I mean increasing enough that it was clearly visible on a last-8064 block plot)
1972016-07-01T09:15:15 <wumpus> "the time to verify a block is increasing (for the same amount of txn data" yes, indeed, I also intended to plot time per block, but that much was clear just from looking at timestamps :(
1982016-07-01T09:16:23 <gmaxwell> well the difference between 1MB and smaller blocks was expected, but the increase still existed when looking at only large blocks.
1992016-07-01T09:16:34 <wumpus> yes
2002016-07-01T09:16:49 <wumpus> recent blocks verify slower
2012016-07-01T09:17:03 <wumpus> even of the same size
2022016-07-01T09:17:20 <wumpus> so *let alone* what bigger blocks would do
2032016-07-01T09:17:53 <gmaxwell> At least segwit doesn't increase the worst case amount of utxo growth or accessing...
2042016-07-01T09:18:03 *** fengling has joined #bitcoin-core-dev
2052016-07-01T09:19:01 *** xiangfu has quit IRC
2062016-07-01T09:30:35 *** Ginnarr has quit IRC
2072016-07-01T09:38:51 *** owowo has joined #bitcoin-core-dev
2082016-07-01T09:38:54 *** owowo has joined #bitcoin-core-dev
2092016-07-01T09:42:00 *** pedrobranco has joined #bitcoin-core-dev
2102016-07-01T09:56:48 <pedrobranco> sipa: hi, when you have the time can you help on the doubts in https://github.com/bitcoin/bitcoin/pull/7551? Thanks :)
2112016-07-01T10:04:14 <sipa> pedrobranco: i answered quickly
2122016-07-01T10:04:57 <sipa> pedrobranco: feel free to disagree by the way... if my view is not obvious here, perhaps the semantics should be different and more clear
2132016-07-01T10:09:06 <pedrobranco> sipa: thanks.
2142016-07-01T10:35:10 *** fengling has quit IRC
2152016-07-01T10:37:59 <wumpus> wow, that wallet drop-unnecessary-inputs-from-best-subset change had been open from sep 2014
2162016-07-01T10:38:07 <wumpus> https://github.com/bitcoin/bitcoin/pull/4906
2172016-07-01T10:40:46 <wumpus> it has not yet made it into any release
2182016-07-01T10:41:04 <MarcoFalke> it has
2192016-07-01T10:41:14 <MarcoFalke> it was backported to .12
2202016-07-01T10:41:19 <wumpus> fuck
2212016-07-01T10:41:34 <wumpus> yes I see: 96e8d120336cf4312cd5f42ba2f9aff17d4ad414
2222016-07-01T10:42:04 <wumpus> that was my own stupid idea probably
2232016-07-01T10:43:37 <MarcoFalke> Up to now I have not seen proof that this made things considerably worse
2242016-07-01T10:44:06 <MarcoFalke> Someone should come up with a framwork that mimics common wallet use cases
2252016-07-01T10:44:16 <wumpus> ok
2262016-07-01T10:44:25 <MarcoFalke> e.g 'exchange': something with a huge in-out volume/count
2272016-07-01T10:44:55 <MarcoFalke> then a "normal user": always waits for confirmations, low volume
2282016-07-01T10:45:20 <wumpus> right, that would make sense, for a project taking the wallet part seriously
2292016-07-01T10:45:45 <MarcoFalke> and then different behaviors like 'always send small coins' (pay for coffee each day)
2302016-07-01T10:45:57 *** jtimon has quit IRC
2312016-07-01T10:46:11 <MarcoFalke> and 'receive large coins' (get payed in btc)
2322016-07-01T10:46:13 <MarcoFalke> etc
2332016-07-01T10:47:57 <wumpus> right.
2342016-07-01T10:49:19 <wumpus> gmaxwell mentioned above that he did see some reports of worse behavior after the change
2352016-07-01T10:49:29 <wumpus> now it's not clear whether to revert it or not
2362016-07-01T10:50:13 <gmaxwell> We recieved an issue with a business with a large wallet reporting specifically the expected behavior there.
2372016-07-01T10:50:45 <wumpus> why don't businesses with large wallets never help with the wallet development?
2382016-07-01T10:51:03 <wumpus> it seems it would be in their best interest
2392016-07-01T10:51:10 <gmaxwell> because there are very few left, most are using fully hosted things that have custom software.
2402016-07-01T10:51:14 <wumpus> but I don't think this was even reported on github
2412016-07-01T10:51:19 <gmaxwell> I thought it was.
2422016-07-01T10:51:23 <wumpus> it's just frustrating
2432016-07-01T10:51:26 <MarcoFalke> it was
2442016-07-01T10:51:30 <wumpus> oh?
2452016-07-01T10:51:38 <gmaxwell> And utxo growth coincided with it. This was previously discussed in meetings IIRC and then ... I'm not sure what happened.
2462016-07-01T10:52:13 <wumpus> that's not important. What is important is what to do now
2472016-07-01T10:52:16 <wumpus> revert it?
2482016-07-01T10:52:38 <MarcoFalke> #7664 #7657
2492016-07-01T10:53:02 <gmaxwell> maybe it was dropped for good reason. I just can't recall.
2502016-07-01T10:54:01 <gmaxwell> At the end of the day, the pruning itself was correcting a bug. The potential issue is that the bug was covering up that the non-bugged behavior is bad (and the simulations showed that it, as I said, would grind down wallets into lots of tiny coins)
2512016-07-01T10:54:09 <MarcoFalke> Imo it shouldn't matter if we revert it or not. I haven't yet seen simulations which prove that either would place an advatage/disatvantage
2522016-07-01T10:54:38 <gmaxwell> MarcoFalke: IIRC there were simulations posted that showed that the behavior caused far more utxo.
2532016-07-01T10:55:08 <MarcoFalke> No one verified that the simulations were implemented according to the current coin selection code
2542016-07-01T10:55:18 <MarcoFalke> I have seen single satoshi outputs in those simulations
2552016-07-01T10:55:25 <MarcoFalke> which does not happen with our code
2562016-07-01T10:55:42 <MarcoFalke> Also the author found a bug in the implementation about two weeks ago or so
2572016-07-01T10:55:44 <gmaxwell> The result is also intutive to me. It results in making the smallest possible outputs. This is pessimal when you consider that avoiding utxo size wants, for a given users balance, the largest value outputs.
2582016-07-01T10:56:40 <MarcoFalke> Ideally, we'd implement the wallet benchmark framework in cpp and have it just use our wallet code?
2592016-07-01T10:57:00 <wumpus> if it would be modular enough for that :)
2602016-07-01T10:57:44 <wumpus> possibly separate the coin selection logic out to a separate file, which could also be used by the simulation framework
2612016-07-01T10:58:03 <wumpus> it could use an abstraction of a list of coins with some properties instead of a wallet as input
2622016-07-01T10:58:41 <wumpus> it would make some things a lot easier, like trying out what the algo does in certain circumstances, without actually having to fake a wallet
2632016-07-01T10:59:46 <gmaxwell> An even better behavior would be to add all other inputs to the same addresses being spent, up to some limit to prevent very large transactions, to sweep them up. This is a privacy improving strategy, as well as a fee minimizing strategy under an assumption that fees will increase in the future. My guess is that I didn't NO DONT the pruning change because in and of itself it was right, and the
2642016-07-01T10:59:52 <gmaxwell> grinding should be fixed by something like this.
2652016-07-01T11:00:11 <gmaxwell> but it went off my radar, also changes to coinselection are a pain because a bunch of th tests freeking hard code the behavior.
2662016-07-01T11:00:40 <wumpus> if that was the only thing making changes to coin selection a pain :)
2672016-07-01T11:01:18 <wumpus> it's just almost impossible to agree what is better behavior, which is what talled #4906 for so long and resulted in it eventually being merged, probably wrongly
2682016-07-01T11:02:06 <wumpus> so a) it should be better behavior b) it needs to be implemented correctly, and indeed, we have no good tests or simulation framework
2692016-07-01T11:04:32 <wumpus> sweeping up as many as possible inputs to the same address would make sense
2702016-07-01T11:04:48 <wumpus> there is no privacy benefit from having more spends to one address
2712016-07-01T11:04:56 <wumpus> then again it assumes bad-case behavior from the user, reusing addresses
2722016-07-01T11:05:29 <gmaxwell> I don't want to be hard on you, but Xekyo ran simulations which showed 25% to 400% increase in the UTXO set size under some simulation loads.
2732016-07-01T11:05:38 <wumpus> I think it would make sense to design a standard interface for wallet coin selection algorithms, separately from any sepecific wallet
2742016-07-01T11:05:38 <gmaxwell> We had simulations, they might not have been ideal.
2752016-07-01T11:05:57 <wumpus> so that research and simuation can be done outside of the specific scope of bitcoin core
2762016-07-01T11:06:08 <gmaxwell> They showed bad behavior from this change. From intution I predited specifically this bad behavior and asked for the simulations. Our issue is not that we need more simulations.
2772016-07-01T11:06:16 <gmaxwell> (well we do, but that didn't help here!)
2782016-07-01T11:06:29 <wumpus> ok, never mind then
2792016-07-01T11:06:37 <gmaxwell> and still MarcoFalke is not agreeing that there is a potential issue.
2802016-07-01T11:06:46 <wumpus> just trying to think of some things that might help, out of the blue
2812016-07-01T11:06:53 <gmaxwell> yea.
2822016-07-01T11:06:56 <wumpus> if it's all hopeless I'll just shut up
2832016-07-01T11:07:27 <gmaxwell> xekyo also identifyied a useful strategy, making the coinselection target double the value instead of the value. This results in change of useful sizes.
2842016-07-01T11:07:41 <wumpus> *more* simulations may not be a solution, but better, correctly implemented ones would
2852016-07-01T11:08:32 <wumpus> if the answer of this is up to whether the simulation is implemented correctly, then we're not really helped a bit, I agree
2862016-07-01T11:09:27 <gmaxwell> re: assumes bad-case behavior from the user, well, indeed, it's no effect if the user doesn't reuse. But no harm, and reuse is ubiquitious. The (vast?) majority of bitcoins in circulation are held in reused addresses.
2872016-07-01T11:09:56 <wumpus> the vast majority of those may not be using bitcoin core in the first place
2882016-07-01T11:10:38 <wumpus> many wallets encourage address reuse, or can only do address resuse
2892016-07-01T11:10:41 <wumpus> that rule may help them a lot :)
2902016-07-01T11:11:15 <gmaxwell> pretty much every active user has some reuse though, consider that thing that sends tips for commits.
2912016-07-01T11:11:35 <wumpus> anyhow that speaks too of generalizing the coin selection algorithm beyond the specific software level
2922016-07-01T11:11:37 <gmaxwell> means that all of us have reuse, even if we otherwise act perfectly ourselves. :)
2932016-07-01T11:11:51 <wumpus> yes, that is true, ideally that should use some BIP32 construction
2942016-07-01T11:12:12 <wumpus> to be honest I always use manual coin selection
2952016-07-01T11:12:48 <wumpus> (apart from testing)
2962016-07-01T11:13:13 <gmaxwell> I do too. (and always spend all the coins connected to an address, when I spend any)
2972016-07-01T11:14:00 <wumpus> and that hould ideally coincide with changing the donation address
2982016-07-01T11:14:07 <gmaxwell> they've gotta get spent someday or the coins are lost, fees are likely lower now than in the future... and spending at once avoids privacy harm from constantly interlinking inside the wallet.
2992016-07-01T11:14:35 <gmaxwell> yea, I do that with my bct address. change the one on the site, when I spend all the coins.
3002016-07-01T11:15:02 <gmaxwell> for that tip commit thing, I just haven't been spending its payments, I think... as I dunno how to change it.
3012016-07-01T11:16:10 <wumpus> they have a little web interface where you can log in and change the address IIRC
3022016-07-01T11:18:18 <gmaxwell> I wonder what the average coin value is in the utxo set and how thats evolved over time. (it's not quite as simple as the set size evolution over time, since more coins have also been introduced)
3032016-07-01T11:18:21 *** murch has joined #bitcoin-core-dev
3042016-07-01T11:19:31 <gmaxwell> "errors": "WARNING: abnormally high number of blocks generated, 4477 blocks received in the last 4 hours (24 expected)"
3052016-07-01T11:19:34 <gmaxwell> lol
3062016-07-01T11:20:11 <wumpus> yes that would be an interesting utxo statistic
3072016-07-01T11:20:48 <wumpus> gmaxwell: https://github.com/bitcoin/bitcoin/pull/8275 :p
3082016-07-01T11:27:14 <GitHub78> [bitcoin] laanwj opened pull request #8298: wallet: Revert input selection post-pruning (master...2016_06_revert_wallet_input_postprune) https://github.com/bitcoin/bitcoin/pull/8298
3092016-07-01T11:33:40 *** molly is now known as moli
3102016-07-01T11:37:33 <sturles> I can see some HD related pull requests with no milestone set. Any chance of HD support in 0.13, or is it 0.14 material?
3112016-07-01T11:37:47 <wumpus> basic HD support has been merged for 0.13
3122016-07-01T11:38:11 <wumpus> the feature freeze for 0.13 is long past now though, so everything else is for 0.14 soonest
3132016-07-01T11:38:18 <wumpus> feel free to help testing them though, that helps a lot
3142016-07-01T11:38:58 <wumpus> if anything there's a lack of involvement and testing and review for wallet PRs, especially UI oriented ones
3152016-07-01T11:39:40 <gmaxwell> we certantly had wallet features slip due to lack of love.
3162016-07-01T11:40:57 *** Prattler has quit IRC
3172016-07-01T11:42:38 <sturles> I haven't used the UI for years.. Will start testing HD stuff.
3182016-07-01T11:45:34 *** blur3d has joined #bitcoin-core-dev
3192016-07-01T11:46:23 *** cryptapus_afk is now known as cryptapus
3202016-07-01T11:51:12 <wumpus> at least for BIP32, in contrast to coin selection, the goal is well-defined and delimited :)
3212016-07-01T11:56:45 <murch> wumpus: I'm working on the latter. :)
3222016-07-01T11:57:36 <wumpus> murch: awesome!
3232016-07-01T11:58:28 <murch> I'm writing a Master thesis on Coin Selection. I've been talking about that to some people here before.
3242016-07-01T11:58:41 <murch> (Pieter, Greg, Greg, Marco)
3252016-07-01T11:59:02 <wumpus> great, some more general reserach there would be very welcome
3262016-07-01T11:59:37 <murch> One focus will be trying to analyze goals, prerequisites and constraints, then the results of varying approaches for selection
3272016-07-01T12:00:15 <wumpus> yea exactly, what I posted here https://github.com/bitcoin/bitcoin/pull/8298#issuecomment-229928583 I think what is needed first is a list of criteria to judge coin selection algorithms by
3282016-07-01T12:00:48 <wumpus> right now we can't tell up from down
3292016-07-01T12:01:11 <wumpus> (unless it is obvious breakage)
3302016-07-01T12:03:54 <murch> Thanks, I hadn't seen that one yet.
3312016-07-01T12:04:16 <murch> I was actually the one that provided the simulation in 4096
3322016-07-01T12:05:05 <wumpus> I mean there are obvious concerns such as CPU usage and memory usage, fee minimization, but also more 'tragedy of the commons' issues such as privacy concerns, utxo growth concerns (though that also coincides with performance a bit, keeping the wallet's utxo set small keeps the global set also smaller)
3332016-07-01T12:06:05 <murch> Actually, I think that the pruning should have had little effect, as there should only be anything to prune when a second pass is needed. Otherwise, since the last added UTXO would be the smallest, there should not be any UTXO prunable.
3342016-07-01T12:06:45 <wumpus> it has very little effect
3352016-07-01T12:06:55 <wumpus> (but apparantly visible in some cases)
3362016-07-01T12:07:00 *** justanotheruser is now known as UserNoJustanuu
3372016-07-01T12:07:39 *** UserNoJustanuu is now known as justanotheruser
3382016-07-01T12:08:00 <wumpus> though of course it's not entirely certain whether the reported problems were due to this change, or another change, or a change in the general usage of bitcoin not directly related to a change in our wallet
3392016-07-01T12:08:50 <wumpus> or a combination of factors including this change
3402016-07-01T12:09:59 <wumpus> in any case, if it has 'little effect' that's also enough reason to revert, I think. If it achieves hardly anything good, it shouldn't have been changed.
3412016-07-01T12:10:13 <murch> wumpus: Completely agree.
3422016-07-01T12:11:47 <murch> Well, I hope that I'll be able to provide some real improvements in the following months. :)
3432016-07-01T12:12:41 <murch> Although I was surprised how well the current implementation does. It's not trivial to improve on it, and not make it deterministic in some fashion. :)
3442016-07-01T12:13:41 <murch> Oh, and providing some metrics to compare Coin Selection approaches is the focus of the work
3452016-07-01T12:13:45 <wumpus> I hope so too!
3462016-07-01T12:14:14 <wumpus> good to hear the current approach is fairly good, on the other hand, that is going to make it even harder to replace by something better :)
3472016-07-01T12:14:45 <wumpus> I think the most common complaints are that it does bad with very large wallets, or containing lots of small inputs
3482016-07-01T12:14:51 <murch> By the way, wumpus, do you know of any other wallet usage data? I got the moneypot.com wallet from #4096, but more good testcases would be grand.
3492016-07-01T12:14:58 <wumpus> apart from that it's not something people usually stumble upon
3502016-07-01T12:15:46 <wumpus> no, unfortunately not
3512016-07-01T12:16:00 <wumpus> I don't have any realistic big wallets
3522016-07-01T12:16:13 <murch> Okay, too bad. :)
3532016-07-01T12:16:58 <wumpus> (this is also an issue which has existed from 2011, which affects both coin selection and general wallet scaling work, people with big wallets are very reluctant to share them even with developers :-) )
3542016-07-01T12:17:34 <murch> Yeah, I'd imagine. :-/
3552016-07-01T12:18:15 <wumpus> which is also why I always encourage companies which have to cope with them to get involved in development themselves, that avoids having to share anything with third parties
3562016-07-01T12:18:27 <murch> The code in wallet.cpp is pretty hard to understand, too. I'd love to refactor it, maybe even to just understand it properly…
3572016-07-01T12:18:46 <wumpus> phantomcircuit is working on that too
3582016-07-01T12:18:53 <murch> oh really?!
3592016-07-01T12:19:19 *** Chris_Stewart_5 has joined #bitcoin-core-dev
3602016-07-01T12:19:21 <murch> mh, just refactoring it, or also improving?
3612016-07-01T12:19:30 <wumpus> in a way it's kind of a chicken and egg problem though, people rarely dare to change things *becuase* they're not sure how they work exactly, but evaluating refactors is also made hard by that because it's non-trivial to see that behavior stays the same
3622016-07-01T12:20:35 <wumpus> both, but not coin selection AFAIK
3632016-07-01T12:20:36 <murch> Yeah, on the other hand it's in a state where it is really hard to add unittests to check whether the behavior remains unchanged.
3642016-07-01T12:20:46 <murch> okay, excellent
3652016-07-01T12:20:49 <wumpus> yes, that too.
3662016-07-01T12:23:54 <wumpus> and years of proposals of doing things differently have made me think that it being hard to understand is not so much a result of the organizatin of wallet.cpp, but that the underlying subject matter is hard
3672016-07-01T12:24:45 <wumpus> people tend to read the code and then blame that the code is difficult, but it's not entirely clear whether it is *unnecessarily* difficult
3682016-07-01T12:25:22 <wumpus> it's clearly a complex problem too
3692016-07-01T12:25:41 <wumpus> maybe it could be divided up into better manageb;e sub-problems though
3702016-07-01T12:25:47 <wumpus> manageable*
3712016-07-01T12:27:46 <wumpus> the thing is, before wallet.cpp was written, cryptocurrency wallets was a problem no one ever considered before, it was necessarily ad-hoc
3722016-07-01T12:28:24 <wumpus> this is very different from say, a web server, where everyone knows what is expected from it, how it usually should be structured, and so on
3732016-07-01T12:30:46 <murch> wumpus: It's surely complicated, but there are some methods that could be extract, and some things are just somewhat obfuscated by very brief variable names. E.g. I've been looking at how and when the fee gets decided for the transaction and it's still baffling me.
3742016-07-01T12:30:48 <wumpus> so the refactor is also a discovery process, learning how to best structure this novel application
3752016-07-01T12:32:12 *** randy-waterhouse has quit IRC
3762016-07-01T12:32:28 <wumpus> well yes it's complicated - but does it need to be complicated, is there a less complex way that would be just as good ,or better? would that simpler way still satisfy the requirements? (and in some cases: what are the requirements, even?)
3772016-07-01T12:34:01 <murch> wumpus: I think it could be more comprehensible if key management and coin selection were separated into different classe for example. They are pretty far apart for being part of one class.
3782016-07-01T12:34:04 <wumpus> things like variable names are trivialities, sure, on first reading of the code it helps to have better names, but once you learn what they are it doesn't really matter what they are anymore (e.g. mathematicians do fine with names such as a b c :)
3792016-07-01T12:34:36 <murch> Most of Coin Selection could actually be completely static, as it basically is just reliant on the spending target, the UTXO set and the desired fee level.
3802016-07-01T12:34:49 <wumpus> yes coin selection is a clear unit
3812016-07-01T12:35:18 <wumpus> I think it would be nice to factor it out to a separate unit with a separate interface, that just gets the information that it needs and returns the selected list
3822016-07-01T12:35:29 <wumpus> this would also be useful for simulations
3832016-07-01T12:35:31 <murch> well, wallet.cpp is just a bit of a moloch to delve into at 3500 LOC. ;)
3842016-07-01T12:35:46 <murch> wumpus: exactly
3852016-07-01T12:35:49 <wumpus> I've seen so much worse at some companies I've worked at :-)
3862016-07-01T12:36:40 <murch> Also, adding, watching and tracking UTXO could probably be separated off
3872016-07-01T12:36:54 <wumpus> yes, that would make sense
3882016-07-01T12:37:19 <wumpus> did you see https://github.com/bitcoin/bitcoin/pull/7823 ?
3892016-07-01T12:38:07 <murch> nope, but I just subscribed.
3902016-07-01T12:38:27 <murch> I am not nearly as familiar with the github repository as I'd like. :-/
3912016-07-01T12:38:40 <wumpus> the idea is quite nice, but it's *hard* to see what its interaction with different things would be, such as reorganizations and conflicts
3922016-07-01T12:38:46 <murch> I should probably check out all issues tagged with "wallet" soon.
3932016-07-01T12:38:52 <wumpus> yes :)
3942016-07-01T12:39:44 <wumpus> in any case, coin selection in itself is enough of a subject to fill a master's thesis I think, I'd warn against scope creep, or trying to fix the world at once :)
3952016-07-01T12:39:45 *** davec_ has quit IRC
3962016-07-01T12:39:53 <murch> hehe
3972016-07-01T12:40:16 *** davec_ has joined #bitcoin-core-dev
3982016-07-01T12:41:02 <murch> yeah, it's a bit overwhelming at times. I've been reading about Subset Sum solvers the past days, and delving through wallet.cpp to understand how fees are handled.
3992016-07-01T12:42:36 <murch> Anyway, back to work. ;) TTYL
4002016-07-01T12:43:09 <wumpus> later :) hope you manage to make head and tail of it
4012016-07-01T12:43:23 <wumpus> hm the new option -blockmaxcost is going to give user understanding problems: https://github.com/bitcoin/bitcoin/blob/master/src/init.cpp#L455
4022016-07-01T12:43:51 <wumpus> should this be a hidden/debug option? if not, it should be documented better
4032016-07-01T12:53:42 *** TheFactory7 has joined #bitcoin-core-dev
4042016-07-01T13:06:21 *** Prattler has joined #bitcoin-core-dev
4052016-07-01T13:10:43 *** Prattler has quit IRC
4062016-07-01T13:18:24 <jonasschnelli> sipa, wumpus: Do I get this right: -reindex does re-create the block index and recreates the utxo set (including all signature validation) while -reindex-chainstate will only recreate the utxo set with the current block index?
4072016-07-01T13:20:26 <sdaftuar> wumpus: -blockmaxcost is supposed to be the recommended way to configure the mining code, going forward
4082016-07-01T13:21:25 <sdaftuar> wumpus: and -blockmaxsize is in the we-want-to-deprecate-in-the-future category
4092016-07-01T13:21:44 <sdaftuar> (the mining code is optimizing for fee-per-block-cost, not fee-per-serialized-byte)
4102016-07-01T13:22:05 <sdaftuar> but yes, we need to document all this before release
4112016-07-01T13:22:55 <sdaftuar> heh, yeah that --help message text is not very informative
4122016-07-01T13:27:59 *** Retric has joined #bitcoin-core-dev
4132016-07-01T13:29:21 *** tadasv has joined #bitcoin-core-dev
4142016-07-01T13:30:55 <wumpus> sdaftuar: a translator was wondering what kind of cost this was, I think they assume that it's something with fee
4152016-07-01T13:31:10 <wumpus> jonasschnelli: correct
4162016-07-01T13:31:12 <sdaftuar> oops. yeah that is bad language
4172016-07-01T13:31:50 *** zooko has joined #bitcoin-core-dev
4182016-07-01T13:31:58 <sdaftuar> what's the timeline for improving that text, is it too late for making changes now that affect translation?
4192016-07-01T13:32:06 <sdaftuar> i guess we can just document in the release notes?
4202016-07-01T13:32:31 <wumpus> well given that translators are confused by it (and hence unable to translate it effectiely) I wouldn't mind changing the message
4212016-07-01T13:33:20 <wumpus> there also needs to be mention in the release notes, but release notes aren't really documentation, just 'news'
4222016-07-01T13:33:46 <wumpus> e.g. you wouldn't assume that someone that is looking for documentation for an option to go through all previous release notes to find it documented
4232016-07-01T13:33:55 <sdaftuar> a simple change might just be to reference the BIP where it's defined; that wouldn't necessarily impose an additional burden on translators if it's just "(BIP 141)" or something at the end of it
4242016-07-01T13:34:04 *** Chris_Stewart_5 has quit IRC
4252016-07-01T13:34:13 <wumpus> what needs to be made clear is that this is just an abstract cost
4262016-07-01T13:34:30 <wumpus> referencing the BIp would make sense, yes
4272016-07-01T13:34:52 <wumpus> in any case, even if it's too late for this to be translated, a better english documentation message would go further than a confusing translated one :)
4282016-07-01T13:35:17 <sdaftuar> agreed!
4292016-07-01T13:35:43 <sdaftuar> perhaps block cost shouldn't be translated at all actually, if we're referencing the BIP
4302016-07-01T13:39:31 <sdaftuar> i updated #8294 so we don't forget this
4312016-07-01T13:43:21 <MarcoFalke> gmaxwell: I don't think lack of progress in improving of coinselection is due to the test hardcoding the behavior.
4322016-07-01T13:43:38 <MarcoFalke> Right now they save us from introducing accidental regressions, which is nice
4332016-07-01T13:44:09 <MarcoFalke> If someone comes up with a new idea, the unit test need to go anyway and will be replaced by new ones, so not really an issue
4342016-07-01T13:45:25 <MarcoFalke> Also the 25% to 400% performance loss may be flawed, as the coin generator was not adjusted to what happens in the real network
4352016-07-01T13:45:33 <MarcoFalke> (re simulation)
4362016-07-01T13:47:19 <MarcoFalke> I remember there was a site which put every address which was ever in a wallet. (Combining whenever two addresses happen to be used in the inputs of the same tx)
4372016-07-01T13:47:46 <MarcoFalke> This could be useful to verify coin generators in simulations are doing their job properly
4382016-07-01T13:48:08 *** cjcj has quit IRC
4392016-07-01T13:51:10 *** JackH has quit IRC
4402016-07-01T14:09:06 *** blur3d has quit IRC
4412016-07-01T14:13:34 *** Retric has quit IRC
4422016-07-01T14:18:04 *** cjcj has joined #bitcoin-core-dev
4432016-07-01T14:22:46 *** zooko has quit IRC
4442016-07-01T14:23:23 *** Cats2 has joined #bitcoin-core-dev
4452016-07-01T14:25:48 *** ClockCat has quit IRC
4462016-07-01T14:26:14 *** droark has quit IRC
4472016-07-01T14:28:29 *** Giszmo has joined #bitcoin-core-dev
4482016-07-01T14:28:42 *** pedrobranco has quit IRC
4492016-07-01T14:34:25 *** MarcoFalke has left #bitcoin-core-dev
4502016-07-01T14:38:22 *** Chris_Stewart_5 has joined #bitcoin-core-dev
4512016-07-01T14:38:55 *** Chris_Stewart_5 has quit IRC
4522016-07-01T14:39:07 *** Chris_Stewart_5 has joined #bitcoin-core-dev
4532016-07-01T14:53:58 *** pedrobranco has joined #bitcoin-core-dev
4542016-07-01T15:00:57 <morcos> wumpus: gmaxwell: i don’t feel strongly enough to make a big argument about it, but if it was up to me i wouldn’t bother reverting 4906. I agree that we didn’t have sufficient justification to merge it in the first place, but we already crossed that bridge, and discussed it after the fact (more than once). I’m not sure we’re confident enough that its clearly worse to risk making the same mistake in the other directi
4552016-07-01T15:01:10 <morcos> direction by reverting it
4562016-07-01T15:01:22 <morcos> I suppose my point is I’d err on the side of being conservative with changes to coin selection and only making them when someone has put in the effort to really study them. Since 0.12 has been out for a while running with 4906, it now seems like a change to me to revert it.
4572016-07-01T15:05:48 *** Chris_Stewart_5 has quit IRC
4582016-07-01T15:06:01 *** kadoban has joined #bitcoin-core-dev
4592016-07-01T15:14:51 <jonasschnelli> sipa: A full IBD (against random peers) with current master took ~3h4min, a -reindex-chainstate took ~1h10min
4602016-07-01T15:15:07 <jonasschnelli> http://bitcoinsrv.jonasschnelli.ch/log.reindex-chainstate.nodebug.da5099.txt, http://bitcoinsrv.jonasschnelli.ch/log.reindex-chainstate.nodebug.da5099.full.txt
4612016-07-01T15:15:28 <jonasschnelli> IBD: http://bitcoinsrv.jonasschnelli.ch/log.ibd.nodebug.da5099.txt, http://bitcoinsrv.jonasschnelli.ch/log.ibd.nodebug.da5099.full.txt
4622016-07-01T15:15:39 <jonasschnelli> now doing a full -reindex
4632016-07-01T15:15:57 <jonasschnelli> Used -dbcache=8000 for both runs
4642016-07-01T15:37:06 <wumpus> morcos: I don't think 'we crossed that bridge' is a good argument, if this was no improvement, it should not have been merged, and it should be reverted (it should have been reverted a long time ago!)
4652016-07-01T15:37:21 <wumpus> morcos: I don't see any reason to keep the code if it i not an improvement
4662016-07-01T15:37:52 <wumpus> and if we should be conservative about changes to coin selection, which we should have been in the first place, then again we shouldn't have this change without understanding it
4672016-07-01T15:38:19 <morcos> wumpus: ok like i said i won't argue, it just makes me nervous that somehow reverting could be worse (interaction with something else that changed that we're not considering), but i don't have any reason to believe that
4682016-07-01T15:38:20 <wumpus> I don't think having the mistake out in 0.12 is a reason to keep it now
4692016-07-01T15:38:25 <wumpus> don't get attached to your mistakes :)
4702016-07-01T15:38:49 <morcos> i won't get attached to somebody's mistakes :)
4712016-07-01T15:39:07 <wumpus> then again I don't feel strongly about it either, but I'm tired of it coming up every time
4722016-07-01T15:39:16 <wumpus> let's just make a damn decision about it
4732016-07-01T15:39:29 <wumpus> I'm sure if we decide not to revert it now, then someone will bring it up again after a month or so
4742016-07-01T15:39:54 <wumpus> I don't want this following me around forever
4752016-07-01T15:41:00 <wumpus> I don't see how this could interact with anything else
4762016-07-01T15:44:17 <bsm1175321> Woah. I just built github master and on testnet something has gone horribly wrong: I have a negative balance!
4772016-07-01T15:45:02 <morcos> bsm1175321: checked how?
4782016-07-01T15:45:14 <bsm1175321> |mcelrath@jane:~> bitcoin-cli -testnet getbalance ''
4792016-07-01T15:45:14 <bsm1175321> -56.85287200
4802016-07-01T15:45:24 <sipa> drop the ''
4812016-07-01T15:45:27 <bsm1175321> It seems to have suddenly fixed itself.
4822016-07-01T15:45:30 <sipa> you're computing an account balance
4832016-07-01T15:45:36 <sipa> not the wallet balance
4842016-07-01T15:45:50 <wumpus> you used to wrong way to query your balance! woe you!
4852016-07-01T15:46:06 <bsm1175321> Without the quotes I got:
4862016-07-01T15:46:07 <bsm1175321> |mcelrath@jane:~> bitcoin-cli -testnet getbalance
4872016-07-01T15:46:07 <bsm1175321> 0.00000000
4882016-07-01T15:46:10 <morcos> i hesitate to ask this question, but can we finally remove accounts for 0.14
4892016-07-01T15:46:15 <wumpus> (we're crazy that there are so many, including wrong ways to query your balance)
4902016-07-01T15:46:22 <morcos> talk about having an issue follow us around
4912016-07-01T15:46:22 <wumpus> morcos: I tried to push for a label API to replace it
4922016-07-01T15:46:30 <bsm1175321> Which is also wrong. A few minutes later it seems to have synced up...and is now showing correct balances. What happened?
4932016-07-01T15:46:42 <sipa> there was a 4000 deep reorg
4942016-07-01T15:46:48 <wumpus> morcos: unnecessary to say, it didn't get enough attention, and is still lingering, I hope to pick it up again for 0.14
4952016-07-01T15:47:12 <bsm1175321> +1 for getting rid of accounts...
4962016-07-01T15:47:25 <wumpus> morcos: https://github.com/bitcoin/bitcoin/pull/7729 I think the release after that we can really remove the account functionality
4972016-07-01T15:47:56 <morcos> wumpus: ok 3rd topic. i thought i head you guys mention that something has gotten slower recently. is it just reindex, or actually something with connecting blocks?
4982016-07-01T15:48:11 <bsm1175321> Next question: I need to fund transactions and be sure that their txid is not malleable. I was hoping 'fundrawtransaction' had an option to take only segwit inputs, but it seems it does not. What's the best way to achieve this?
4992016-07-01T15:48:20 <wumpus> morcos: AFAIK the summary there was: gmaxwell accidentally -txindex
5002016-07-01T15:48:45 <sipa> morcos: and nobody recently tested -reindex with default -dbcache
5012016-07-01T15:48:56 <morcos> wumpus: oh i definitely missed that conclusion. :)
5022016-07-01T15:48:56 <wumpus> morcos: yes, more recent blocks are slower to validate, comapred to older blocks of the same size - but this isn't a reversion in the code
5032016-07-01T15:49:11 <morcos> wumpus: wait, yes its that last thing i'm asking about
5042016-07-01T15:49:14 <morcos> why is that?
5052016-07-01T15:49:25 <wumpus> it's likely a by-product of increasing utxo size
5062016-07-01T15:50:06 <morcos> and that affecting cache hit rate?
5072016-07-01T15:50:07 <wumpus> and the utxo database is reaching the size that leveldb can handle w/ good performance, e.g. now lots of crazy seeking and reading all over your disk, can't be good for performance
5082016-07-01T15:50:14 <morcos> oh
5092016-07-01T15:50:37 <wumpus> yes that was another hypothesis
5102016-07-01T15:50:51 <wumpus> <gmaxwell> I was theorizing that this was from polylog behavior in the database and worrying, but phantomcircuit gave an alternative argument that the reduction in spammy transactions relative to non-spammy ones may be resulting in lower cache hitrates.
5112016-07-01T15:50:59 <morcos> eyeballing the numbers i'm not seeing a noticeable slowdown since march
5122016-07-01T15:51:04 <wumpus> which probably means it's a combination of both
5132016-07-01T15:52:07 <wumpus> in any case the increased default dbcache should alleviate either problem, at least for a while
5142016-07-01T15:53:26 <morcos> ok, yeah my numbers are with a big dbcache (2G)
5152016-07-01T15:53:28 <wumpus> there may be a better way to structure the utxo set on disk - but having looked into other databases it seems that leveldb has the best performance for our use, at least during sync
5162016-07-01T15:54:17 <wumpus> if you set a huge dbcache you won't notice much, as sipa says it's probably why the surprise that indexing was so slow with default dbcache
5172016-07-01T15:54:30 <sipa> for 0.14 we should prioritize working on chainstate backups
5182016-07-01T15:55:05 <sipa> (and vigorously fight the potential push for servers with 'helpful' backups to download...)
5192016-07-01T15:58:53 <bsm1175321> I need to fund transactions and be sure that their txid is not malleable. I was hoping 'fundrawtransaction' had an option to take only segwit inputs, but it seems it does not. What's the best way to achieve this? (I'm willing to write such an option if there isn't a better way) @sipa?
5202016-07-01T15:59:18 <sipa> bsm1175321: patches welcome :)
5212016-07-01T15:59:28 <sipa> seems like a very useful feature
5222016-07-01T15:59:31 <bsm1175321> Ok so the answer is that this isn't possible currently?
5232016-07-01T15:59:46 <sipa> for now, you'll need listunspent
5242016-07-01T15:59:48 <bsm1175321> Cool, I'll see what I can do.
5252016-07-01T16:00:01 <sipa> i believe i've heard someone else ask for the same feature
5262016-07-01T16:09:01 *** Guyver2 has joined #bitcoin-core-dev
5272016-07-01T16:09:30 <bsm1175321> One complication with such an option is that it's possible for your wallet to have enough funds to fund the transaction, but not enough segregated witness inputs. A solution would be to create an intermediate transaction whose outputs have segregated witness. Thoughts?
5282016-07-01T16:09:48 <sipa> fundrawtransaction shouldn't do such thing
5292016-07-01T16:09:51 <sipa> imho
5302016-07-01T16:10:04 <sipa> but it can fail with a nice error code, instructing you to clean up your wallet
5312016-07-01T16:10:12 <bsm1175321> You'd rather it fail in such a case? That's fine with me too...
5322016-07-01T16:10:23 <sipa> yes, insufficient funds :)
5332016-07-01T16:10:42 <sipa> (or perhaps you should use separate wallets if dealing with that situation is hard)
5342016-07-01T16:10:51 <bsm1175321> What would you expect a user to do in such a case? Is there a procedure to "clean up your wallet"? That people will know?
5352016-07-01T16:11:25 <sipa> it's no different from the situation where you have a wallet with some watchonly coins, and the non-watchonly together are not enough to pay
5362016-07-01T16:12:03 <bsm1175321> True, sort of, except you have the funds. I'll make a descriptive error message.
5372016-07-01T16:15:47 <sipa> i think the real solution is that if you have a need for segwit-only inputs, you run with a segwit-only wallet
5382016-07-01T16:17:23 <bsm1175321> In effect that's what I'll do. But have to start from a non-segwit wallet. Or is there a spent-to-myself-segwit-out wallet RPC call?
5392016-07-01T16:18:48 <sipa> you can just use sendtoaddress to send from the old wallet to the new?
5402016-07-01T16:19:28 <sipa> also, wallet integration of segwit is really preliminary at this point
5412016-07-01T16:19:39 <sipa> for example, change outputs won't be segwit
5422016-07-01T16:19:55 <bsm1175321> ooohhh...hmmm...
5432016-07-01T16:20:09 <sipa> (which they should be when it becomes ready for production, but that's not done yet)
5442016-07-01T16:21:29 <bsm1175321> Well if you 'fundrawtransaction ... segwitOnly' I can make it use a segwit change address.
5452016-07-01T16:21:50 <bsm1175321> Or you can add an explicit change address
5462016-07-01T16:22:03 <sipa> true
5472016-07-01T16:22:31 <bsm1175321> If no objections, I'll make the 'segwitOnly' option take only segwit inputs, and generate segwit change.
5482016-07-01T16:33:08 *** Chris_Stewart_5 has joined #bitcoin-core-dev
5492016-07-01T16:35:12 <instagibbs> what is/is there a way to link to an external secp library during compilation?
5502016-07-01T16:41:43 <bsm1175321> sipa: do you have a "wallet wishlist" for segwit? I'm seeing that a lot of infrastructure is missing here.
5512016-07-01T16:43:52 <sipa> bsm1175321: thinks like default witness addresses
5522016-07-01T16:43:57 <sipa> post softfork
5532016-07-01T16:53:45 <bsm1175321> I'm confused regarding addresses.
5542016-07-01T16:54:24 *** jtimon has joined #bitcoin-core-dev
5552016-07-01T16:54:30 <bsm1175321> What is the output of 'addwitnessaddress'? (what kind of address is that?)
5562016-07-01T16:55:42 *** jtimon has quit IRC
5572016-07-01T16:57:32 <sipa> p2sh
5582016-07-01T16:58:29 <bsm1175321> Ok, duh. (hadn't made a testnet p2sh before, was confused by the 2)
5592016-07-01T17:04:43 <bsm1175321> Ok I think I understand. A wallet bitcoin.conf really should have 3 possible values then: non-segwit, segwit, and segwit-in-p2sh.
5602016-07-01T17:04:50 <bsm1175321> *wallet setting
5612016-07-01T17:05:16 <bsm1175321> Do I understand correctly: The point of P2SH nesting is that non-upgraded wallets will check the destinations (though not the signatures)?
5622016-07-01T17:15:56 <jl2012> bsm1175321: I'd say the point of P2SH nesting is to allow non-upgraded wallets to send money to upgraded wallet
5632016-07-01T17:19:34 *** Chris_Stewart_5 has quit IRC
5642016-07-01T17:20:11 <bsm1175321> Can't they always do that though? Do you mean the other way around?
5652016-07-01T17:21:36 <arubi> it's made for upgraded nodes to be able to accept payment from unupgraded nodes to a segwit scriptpubkey, yes
5662016-07-01T17:23:09 <jl2012> for upgraded -> not upgraded, you just use P2PKH address (1xyz......)
5672016-07-01T17:25:37 <arubi> bsm1175321, I'm curious, what kind of malleability are you trying to avoid? are you the one signing the transaction? can't you use normal inputs too and just sign it properly?
5682016-07-01T17:26:41 <bsm1175321> arubi: I'm going to be handing out txids, including txids that may be in the mempool, not mined yet.
5692016-07-01T17:27:58 <arubi> signed by you and you only? you could still "maul"(?) if you wanted to, even with segwit
5702016-07-01T17:28:31 <jl2012> arubi: without BIP62, that's not reliable
5712016-07-01T17:28:47 <bsm1175321> The non-upgraded nodes that receive funds from an upgraded node doesn't verify signatures though. (correct?)
5722016-07-01T17:29:32 <arubi> jl2012, talking about the anyonecanpay|single sighash, sign once, permute how many you'd like
5732016-07-01T17:30:15 <arubi> bsm1175321, the script that they see doesn't talk about checksigs, so they don't check any signatures
5742016-07-01T17:30:21 <jl2012> arubi: there are still other forms of third party malleability, e.g. non-canonical push
5752016-07-01T17:30:37 <bsm1175321> arubi: signed by me and me only.
5762016-07-01T17:31:25 <arubi> jl2012, sure, using standard scripts is a must here if you don't want 3rd parties doing things, but as a single signer, you could still do it to your own txs, even with segwit
5772016-07-01T17:33:11 <jl2012> well, if you want to guarantee non-malleability to only yourself, segwit is the only way
5782016-07-01T17:33:16 <bsm1175321> jl2012: can you elaborate? Malleation of the script (which is part of the witness data) doesn't change the txid, and I don't think I care about that.
5792016-07-01T17:33:30 *** Chris_Stewart_5 has joined #bitcoin-core-dev
5802016-07-01T17:34:05 <arubi> jl2012, true. I am asking because bsm1175321 explains that he will hand out txids (not the transactions themselves, as I understand)
5812016-07-01T17:34:29 *** pedrobranco has quit IRC
5822016-07-01T17:34:29 <arubi> so as a single signer, he could still cause malleability, even if he manages to convince the other party all inputs are segwit
5832016-07-01T17:34:52 <arubi> *and standard scripts, or even better, even only with p2wpkh
5842016-07-01T17:34:53 <jl2012> arubi: that's double-spending
5852016-07-01T17:34:57 *** pedrobranco has joined #bitcoin-core-dev
5862016-07-01T17:35:02 <jl2012> not malleability
5872016-07-01T17:35:26 <arubi> how so? payments are still paid, but it's the order of the inputs outputs that's changed
5882016-07-01T17:35:33 <arubi> you're using the same inputs
5892016-07-01T17:36:06 <bsm1175321> arubi: Are you saying I could change the order of outputs, keeping the same segwit txid?
5902016-07-01T17:36:18 <arubi> no, you're changing it
5912016-07-01T17:37:54 <bsm1175321> Ok then someone looking for one txid I handed them won't see it at all, and there's no point in me giving the wrong txid to someone. (A txid which never get mined because I malleated it is useless)
5922016-07-01T17:38:31 <arubi> right
5932016-07-01T17:39:01 <arubi> they should be checking their addresses for incoming transactions, and that's it. txid is something that's only relevant when spending
5942016-07-01T17:39:11 <arubi> *should only be, I guess
5952016-07-01T17:39:19 <bsm1175321> However if they act on a txid in the mempool, and then I broadcast a malleated version which gets mined. That's your usual double spend.
5962016-07-01T17:39:40 *** pedrobranco has quit IRC
5972016-07-01T17:40:16 <arubi> how is it a double spend? there is no spending from the malleated version
5982016-07-01T17:40:23 <pigeons> well if you only malleate the signature its not a double spend
5992016-07-01T17:40:59 <bsm1175321> Let's say I malleate the output order. This is a weird thing to do. I'll have to think on it...
6002016-07-01T17:41:42 <arubi> same can be done for inputs
6012016-07-01T17:42:12 <pigeons> if you have someone taking action on the txid of the unconfirmed transaction they will be affected, otherwise, not right
6022016-07-01T17:43:09 <arubi> a spender still has to reference a txid as input, so even if the outputs order isn't changed, the input is invalid in case of malleability
6032016-07-01T17:43:56 <arubi> oh you mean if the bad one wasn't mined, right
6042016-07-01T17:48:21 <bsm1175321> In this case, I'm proving control of coins. Malleating my own txn is shooting myself in the foot.
6052016-07-01T17:55:28 *** laurentmt has joined #bitcoin-core-dev
6062016-07-01T17:58:42 <arubi> I'm guessing, you're proving control by announcing the txid before the transaction is seen on the network?
6072016-07-01T18:08:50 <bsm1175321> Not *before* but it could be simultaneous. And I'm wondering what the consequences are of fiddling around...
6082016-07-01T18:09:21 <bsm1175321> But If I claim to prove something with a txid, and you look for it on the network, I'm only hurting myself by messing with it. I'm having trouble thinking of any negative consequences here.
6092016-07-01T18:11:24 <arubi> why not use signed messages to prove ownership of addresses? as long you commit to an address, which is kinda like commiting to specific coins
6102016-07-01T18:12:52 *** kadoban has quit IRC
6112016-07-01T18:17:38 <arubi> hm. I guess if it's a p2sh that's not so simple, because you'd have to disclose the redeemscript
6122016-07-01T18:18:54 *** spudowiar has joined #bitcoin-core-dev
6132016-07-01T18:23:00 *** pedrobranco has joined #bitcoin-core-dev
6142016-07-01T18:26:41 *** spudowiar has quit IRC
6152016-07-01T18:27:01 *** spudowiar has joined #bitcoin-core-dev
6162016-07-01T18:45:06 *** pedrobranco has quit IRC
6172016-07-01T18:45:39 *** pedrobranco has joined #bitcoin-core-dev
6182016-07-01T18:45:53 *** pedrobranco has joined #bitcoin-core-dev
6192016-07-01T18:49:39 *** pedrobranco has quit IRC
6202016-07-01T18:50:18 *** pedrobranco has joined #bitcoin-core-dev
6212016-07-01T18:50:34 *** pedrobranco has joined #bitcoin-core-dev
6222016-07-01T18:51:50 *** pedrobranco has quit IRC
6232016-07-01T18:56:06 *** pedrobranco has joined #bitcoin-core-dev
6242016-07-01T19:03:12 *** pedrobranco has quit IRC
6252016-07-01T19:03:46 *** pedrobranco has joined #bitcoin-core-dev
6262016-07-01T19:07:51 *** CubicEarth has joined #bitcoin-core-dev
6272016-07-01T19:08:04 *** pedrobranco has quit IRC
6282016-07-01T19:23:59 *** TheFactory7 has quit IRC
6292016-07-01T19:31:10 *** molz has joined #bitcoin-core-dev
6302016-07-01T19:31:34 *** zooko has joined #bitcoin-core-dev
6312016-07-01T19:33:34 *** moli has quit IRC
6322016-07-01T19:34:05 *** laurentmt has quit IRC
6332016-07-01T19:36:16 *** laurentmt has joined #bitcoin-core-dev
6342016-07-01T19:39:18 *** Lysander1 has joined #bitcoin-core-dev
6352016-07-01T19:41:15 *** CyrusV has quit IRC
6362016-07-01T19:41:51 *** Cory has quit IRC
6372016-07-01T19:41:53 *** Expanse has quit IRC
6382016-07-01T19:42:22 *** jannes has quit IRC
6392016-07-01T19:42:28 *** Lysanders has quit IRC
6402016-07-01T19:42:28 *** LeMiner has quit IRC
6412016-07-01T19:42:28 *** OxADADA has quit IRC
6422016-07-01T19:42:29 *** TD-Linux has quit IRC
6432016-07-01T19:42:29 *** nanotube has quit IRC
6442016-07-01T19:42:29 *** crescendo has quit IRC
6452016-07-01T19:42:52 *** spudowiar has quit IRC
6462016-07-01T19:42:54 *** kinlo has quit IRC
6472016-07-01T19:42:54 *** hexis has quit IRC
6482016-07-01T19:42:54 *** petertodd has quit IRC
6492016-07-01T19:42:55 *** windsok has quit IRC
6502016-07-01T19:42:56 *** windsok_ has joined #bitcoin-core-dev
6512016-07-01T19:43:53 *** Pasha has joined #bitcoin-core-dev
6522016-07-01T19:43:53 *** crescendo has joined #bitcoin-core-dev
6532016-07-01T19:44:17 *** OxADADA has joined #bitcoin-core-dev
6542016-07-01T19:44:21 *** kinlo has joined #bitcoin-core-dev
6552016-07-01T19:44:28 *** petertodd has joined #bitcoin-core-dev
6562016-07-01T19:44:59 *** adiabat has joined #bitcoin-core-dev
6572016-07-01T19:45:10 *** Expanse has joined #bitcoin-core-dev
6582016-07-01T19:45:36 <adiabat> hey, I'm spamming testnet and have some unexpected behavior
6592016-07-01T19:45:40 *** laurentmt has quit IRC
6602016-07-01T19:46:05 <adiabat> I think I get what "size", "cost", "strippedsize", "Vsize" all mean
6612016-07-01T19:46:50 <adiabat> "cost" is basically "Vsize" * 4
6622016-07-01T19:47:05 <adiabat> so a block "cost" must be < 4M
6632016-07-01T19:47:06 *** cryptapus is now known as cryptapus_afk
6642016-07-01T19:47:10 <adiabat> (<=, whatever)
6652016-07-01T19:47:47 *** TD-Linux has joined #bitcoin-core-dev
6662016-07-01T19:48:07 <adiabat> so in .conf, blockmaxsize should set the cost limit of created blocks
6672016-07-01T19:48:23 <adiabat> but it seems to be targeting "size" instead
6682016-07-01T19:48:43 *** CyrusV has joined #bitcoin-core-dev
6692016-07-01T19:49:07 <bsm1175321> arubi: I want to be able to commit to coins regardless of the address type, and as you say, I'd need the redeemScript.
6702016-07-01T19:50:47 *** Pasha is now known as Cory
6712016-07-01T19:52:22 *** kvnn has joined #bitcoin-core-dev
6722016-07-01T19:53:58 *** nanotube has joined #bitcoin-core-dev
6732016-07-01T19:54:34 <kvnn> Hello everyone. I'm offering a 1BTC bounty for 3&4 here: https://github.com/drivechain-project/docs . Please let me know if you are interested. (if this is not okay in this channel I apologize & will discontinue posting about it)
6742016-07-01T19:55:28 *** jannes has joined #bitcoin-core-dev
6752016-07-01T19:55:58 *** hexis has joined #bitcoin-core-dev
6762016-07-01T19:56:22 *** spudowiar has joined #bitcoin-core-dev
6772016-07-01T19:59:44 *** sdaftuar_ has joined #bitcoin-core-dev
6782016-07-01T20:02:55 <sdaftuar_> adiabat: -blockmaxsize continues to refer to total serialized bytes of a block, counting witness and non-witness parts the same. -blockmaxcost is the option that specifies the limit you probably want here.
6792016-07-01T20:04:43 <adiabat> hmmm ok
6802016-07-01T20:04:48 <adiabat> so there's both
6812016-07-01T20:09:18 <sdaftuar_> Yeah. The hope is that -blockmaxsize is deprecated in the future.
6822016-07-01T20:10:21 *** spudowiar has quit IRC
6832016-07-01T20:10:38 <sdaftuar_> (My preference was to change the semantics of -blockmaxsize to refer to segwit's virtual size, but that's not what we ended up with)
6842016-07-01T20:11:02 <luke-jr> well, depends what you want to limit
6852016-07-01T20:11:23 <roasbeef> segwit miners on testnet seem to be limiting to ~750k cost
6862016-07-01T20:11:31 <luke-jr> sdaftuar_: then miners would need to set blockmaxsize to 250k to avoid making blocks >1M
6872016-07-01T20:11:55 <gmaxwell> 06:43 < MarcoFalke> gmaxwell: I don't think lack of progress in improving of coinselection is due to the test hardcoding the behavior.
6882016-07-01T20:12:01 <luke-jr> the goal of blockmaxsize is to avoid blocks larger than the size. cost/vsize doesn't matter here
6892016-07-01T20:12:20 <gmaxwell> I can tell you that personally I've written improvements that I haven't PRed because they required throwing out all the existing tests.
6902016-07-01T20:12:21 <sdaftuar_> My preference was to introduce a new option that would control serialized bytes
6912016-07-01T20:12:51 <luke-jr> sdaftuar_: why a new oppppption when we have one for it already?
6922016-07-01T20:13:27 <sdaftuar_> Because the mining code doesn't optimize for it
6932016-07-01T20:13:40 <sdaftuar_> So it's misleading to suggest it's supported
6942016-07-01T20:14:05 <luke-jr> it does what it's supposed to do.
6952016-07-01T20:14:08 <gmaxwell> I don't think it makes sense to even have a seralized size setting, we don't have a setting to limit the size of the sighashed bytes, or a setting to limit the size of the block's CTransaction encoding. We don't have a limit to control the size of the compactblock form, or a limit to control the size of a zlib compressed block.
6962016-07-01T20:14:46 <gmaxwell> But I don't bother complaining because luke was going pyric over this before, and it shouldn't hurt that much having it either.
6972016-07-01T20:14:51 <luke-jr> gmaxwell: at this time, serialised size is a critical factor in practice
6982016-07-01T20:15:22 <sdaftuar_> gmaxwell: +1
6992016-07-01T20:15:32 <gmaxwell> luke-jr: critical for what? it doesn't reflect block propagation time especially well.
7002016-07-01T20:15:58 <luke-jr> on the p2p network it sure does..
7012016-07-01T20:16:06 <gmaxwell> validation time is more closely related to the number of utxo operations and how well relayed the txn in hte block were.
7022016-07-01T20:16:24 <gmaxwell> luke-jr: we have BIP152 now.
7032016-07-01T20:16:36 <luke-jr> not deployed, unfortunately.
7042016-07-01T20:17:14 <luke-jr> to be clear, I'm all for removing blockmaxsize once it doesn't matter and isn't used.
7052016-07-01T20:17:23 <luke-jr> hence why I don't think optimising block creation for it is important
7062016-07-01T20:17:28 <gmaxwell> (but even without it, seralized size is still not the ideal predictor of propagation time.)
7072016-07-01T20:18:20 <luke-jr> I just worry that we'll end up with >1 MB blocks before the network can handle it sanely
7082016-07-01T20:18:20 <gmaxwell> K. indeed 0.12 doesn't have BIP152. Perhaps that does suggest that it should default to four million in 0.13?
7092016-07-01T20:18:27 <gmaxwell> luke-jr: thats inevitable...
7102016-07-01T20:18:48 <gmaxwell> see also the utxo comments by wumpus last night.
7112016-07-01T20:24:36 *** Cats2 is now known as ClockCat
7122016-07-01T20:30:13 *** zooko has quit IRC
7132016-07-01T20:47:39 <sturles> Am I correct that the 0.13 HD wallet implementation will only support new wallets, and only support hardened keys?
7142016-07-01T20:47:49 <gmaxwell> sturles: correct.
7152016-07-01T20:48:09 <sturles> Will it support importing private keys from an old style wallet?
7162016-07-01T20:48:37 <gmaxwell> I believe it does, I've not actually tested that!
7172016-07-01T20:53:27 <sturles> A HD wallet won't work with older versions, right? Should use the opportunity to switch DB for the wallet to something > libdb 4.8?
7182016-07-01T20:54:07 <gmaxwell> Do you want software that never makes progress? Thats how you get software that never makes progress.
7192016-07-01T20:54:45 <gmaxwell> It can't use a later libdb or it will make all non-HD wallets also unreadable by other versions.
7202016-07-01T20:55:45 <helo> i believe a hd wallet should work with older versions
7212016-07-01T20:56:18 <gmaxwell> helo: hows that?
7222016-07-01T20:56:44 <sturles> Yes, I know. I guess it is impossible to use different versions depending on wallet type, e.g. via dlopen?
7232016-07-01T20:57:16 <gmaxwell> sturles: in theory possible but that would be a lot of work without any benefit.
7242016-07-01T20:57:31 <helo> it will work to a limited extent... hd keys already added will be visible
7252016-07-01T20:57:48 <helo> untested, ofc
7262016-07-01T20:57:59 <gmaxwell> if so, thats a bug. it should get rejected due to having a too new version.
7272016-07-01T20:58:02 <Lightsword> can we just migrate to sqlite?
7282016-07-01T20:58:09 <gmaxwell> if it isn't then you could put yourself in a weird state.
7292016-07-01T20:58:18 <gmaxwell> I will have to start stabbing people.
7302016-07-01T20:59:22 <helo> nah, that's not necessary. the version check probably functions as intended :)
7312016-07-01T20:59:40 <gmaxwell> hah
7322016-07-01T21:01:33 <sturles> I'd say it is benificial to switch to a newer, supported version of libdb
7332016-07-01T21:02:42 <gmaxwell> the latest versions have incompatible licenses.
7342016-07-01T21:04:39 <sturles> Oh.
7352016-07-01T21:05:05 <gmaxwell> there is newer than 4.8 that is okay, but the latest stuff has a much stronger copyleft than we'd normally use.
7362016-07-01T21:07:19 <gmaxwell> really the 'database' isn't actually used for much--- the wallet is largely entirely in memory, and the database is only really used for persistance.
7372016-07-01T21:08:52 *** murch has quit IRC
7382016-07-01T21:09:36 *** murch has joined #bitcoin-core-dev
7392016-07-01T21:14:07 <luke-jr> eh, -walletupgrade won't work? O.o
7402016-07-01T21:30:31 *** afk11 has quit IRC
7412016-07-01T21:36:54 <Lightsword> would working on migrating the wallet to sqlite be something worthwhile at this point?
7422016-07-01T21:38:16 *** afk11 has joined #bitcoin-core-dev
7432016-07-01T21:38:16 *** afk11 has quit IRC
7442016-07-01T21:38:16 *** afk11 has joined #bitcoin-core-dev
7452016-07-01T21:39:50 <CubicEarth> Block validation is, theoretically, very amenable to parallelization, correct? Is the main serial component the depth of the merkel tree?
7462016-07-01T21:39:52 *** murch has quit IRC
7472016-07-01T21:42:09 *** moli has joined #bitcoin-core-dev
7482016-07-01T21:43:50 <gmaxwell> signature validation is, and bitcoin core runs that in paralle... but normally almost all of the signatures in a block are already validated before it shows up.
7492016-07-01T21:44:19 *** molz has quit IRC
7502016-07-01T21:44:22 <gmaxwell> parallel*
7512016-07-01T21:44:27 <gmaxwell> the general database handling ends up taking much of the time, which isn't really paralizable.
7522016-07-01T21:48:37 *** mkarrer_ has quit IRC
7532016-07-01T21:51:01 *** sdaftuar1 has joined #bitcoin-core-dev
7542016-07-01T21:51:18 *** ybit_ has joined #bitcoin-core-dev
7552016-07-01T21:51:28 *** OxADADA_ has joined #bitcoin-core-dev
7562016-07-01T21:51:32 *** trippysalmon has joined #bitcoin-core-dev
7572016-07-01T21:52:20 *** mkarrer has joined #bitcoin-core-dev
7582016-07-01T21:53:06 *** tadasv has quit IRC
7592016-07-01T21:53:10 *** [b__b] has quit IRC
7602016-07-01T21:53:12 *** sdaftuar has quit IRC
7612016-07-01T21:53:12 *** OxADADA has quit IRC
7622016-07-01T21:53:12 *** trippysa1mon has quit IRC
7632016-07-01T21:53:12 *** ybit has quit IRC
7642016-07-01T21:53:13 *** sdaftuar_ is now known as sdaftuar
7652016-07-01T21:53:37 *** tadasv has joined #bitcoin-core-dev
7662016-07-01T21:54:03 *** [b__b] has joined #bitcoin-core-dev
7672016-07-01T21:55:05 *** luke-jr has quit IRC
7682016-07-01T22:01:11 *** luke-jr has joined #bitcoin-core-dev
7692016-07-01T22:05:09 <CubicEarth> gmaxwell: so thinking about the initial chain sync and validation, it could (with lots and lots of careful work) be made to harness GPU's, and spread the signature validation across *many* cores? I'm not familiar with the nature of the general database handling you are referring to, but for the initial sync is that mainly keeping track of the utxo's as it increments tough the blocks?
7702016-07-01T22:05:32 *** Chris_Stewart_5 has quit IRC
7712016-07-01T22:07:53 <gmaxwell> I've seen no reason to believe that a gpu would help at all, GPU cores aren't particularly good at the work of validating. (64 bit arithmetic helps a lot!)
7722016-07-01T22:08:26 <gmaxwell> and nodes don't even verify the signatures far back in the chain.
7732016-07-01T22:08:31 *** luke-jr has quit IRC
7742016-07-01T22:08:39 *** luke-jr has joined #bitcoin-core-dev
7752016-07-01T22:09:44 <gmaxwell> but sure more work could be done to extract more parallelism out of it.
7762016-07-01T22:11:16 <gmaxwell> but even with signature validation turned off completely, with default settings a reindex takes almost 9 hours currently.
7772016-07-01T22:13:02 <gmaxwell> increasing the db cache to a really huge size (such that the sync runs entirely out of ram) lets the whole sync _with_ signature checking run in about 3.5 hours.
7782016-07-01T22:14:04 *** zooko has joined #bitcoin-core-dev
7792016-07-01T22:14:43 *** kadoban has joined #bitcoin-core-dev
7802016-07-01T22:16:12 <CubicEarth> Is the default 100MB? I never knew that!! I'll be curious too see what kind of speedup I can achieve on an i5 - dual core. Are we talking 100GB of ram here, or would setting it to 4 or 8 GB make a substantial difference?
7812016-07-01T22:17:11 <sipa> setting it to 2 GB is more than sufficient
7822016-07-01T22:17:32 <sipa> higher numbers don't matter much
7832016-07-01T22:18:38 *** Chris_Stewart_5 has joined #bitcoin-core-dev
7842016-07-01T22:19:01 <CubicEarth> Is it a rolling-window thing?
7852016-07-01T22:19:17 <sipa> no
7862016-07-01T22:19:23 <sipa> it's a cache
7872016-07-01T22:19:32 <sipa> when the cache is full, we write it to disk
7882016-07-01T22:19:38 <sipa> and start over
7892016-07-01T22:19:38 <sipa> very silly
7902016-07-01T22:22:10 <gmaxwell> I think you actually need more than 2GB now to get full performance, but 8GB is more than enough.
7912016-07-01T22:22:31 <midnightmagic> maybe an iovec scatter write might help some systems write faster
7922016-07-01T22:24:24 <midnightmagic> or is it just a plain sequential write thing and the rest is leveldb
7932016-07-01T22:24:40 <CubicEarth> Well it's easy enough to set it higher. I just never knew. A default of 100MB can make sense for older systems, but if there was place for 'performance tips', that would be a good place to let people know. Maybe everyone already (node operators) knows except for me.
7942016-07-01T22:25:15 <gmaxwell> I think a lot of people don't realize what a big difference it makes.
7952016-07-01T22:25:46 <gmaxwell> looks like we'll increase the default to 300MB.
7962016-07-01T22:26:46 <gmaxwell> wumpus: on the leveldb stuff, without txindex, I'd suggest making that 4mb instead of 2. 4MB was something like 1% faster for me when testing leveldb in isolation.
7972016-07-01T22:27:29 <gmaxwell> maybe 2 is a win at the 300mb limit, but with it bumped 4mb would be... and probably is more conservative overall.
7982016-07-01T22:31:03 *** sdaftuar has quit IRC
7992016-07-01T22:35:30 *** pedrobranco has joined #bitcoin-core-dev
8002016-07-01T22:36:23 *** Guyver2 has quit IRC
8012016-07-01T22:39:42 *** pedrobranco has quit IRC
8022016-07-01T22:39:57 <CubicEarth> I did a fresh sync a week ago, 2013-era i5 dual core laptop. Ubuntu, 4GB of ram, spinning HD. 75Mbps cable internet. It took about 20 hours to finish. What was interesting though was how the last 20 weeks just crawled. Seemed less than linear, even estimating about the effects of fuller blocks.
8032016-07-01T22:42:35 <sipa> yeah, try dbcache 4000 or so
8042016-07-01T22:43:17 <CubicEarth> after I order more ram :)
8052016-07-01T22:47:42 <CubicEarth> gmaxwell: earlier you wrote "and nodes don't even verify the signatures far back in the chain.". That's due to checkpoints, right? They don't seem like a good thing to rely on. I understand their practicality, but yeah.
8062016-07-01T22:49:16 <CubicEarth> I'm just trying to understand the theoretical performance limits... I'm not crusading against checkpoints
8072016-07-01T22:52:44 <gmaxwell> well you'd be welcome here to do that, I want them gone and have for years.
8082016-07-01T22:53:51 <gmaxwell> But even with them gone, one can avoid skipping signature validation during the initial sync for blocks burried by thousands of additional blocks with negligible risk-- consider, if miners went rogue enough to reorg thousands of blocks the system is already screwed... and if its only during initial sync they'd only trick new nodes in any case.
8092016-07-01T22:54:10 <gmaxwell> if it makes a difference between a user running a node or not-- the best choice is clear.
8102016-07-01T22:54:54 <gmaxwell> Meanwhile bitcoinxt and classic have disabled signature checking for any block where the miner supplied timestamp is a day back or more... meaning they can be fooled without a reorg at all. And no one seems to care.
8112016-07-01T22:56:57 <CubicEarth> Arn't miners only afforded the option to have their timestamp be off by more than 2-hours?
8122016-07-01T22:57:13 <CubicEarth> (as an aside)
8132016-07-01T22:57:28 <gmaxwell> no, not in the past direction. In the future direction yes.
8142016-07-01T22:58:05 <gmaxwell> The past direction is limited by the median of the last 11 blocks-- but that median can be arbritarily far back.
8152016-07-01T22:59:42 <CubicEarth> So practically speaking, the way to detect the issue is when the client said it was 'synced' and you noticed that is was reporting yesterdays date?
8162016-07-01T22:59:56 <midnightmagic> .. did they actually commit that change? the >24hr non-t-verify change?
8172016-07-01T23:01:32 <gmaxwell> CubicEarth: no, because miners can simply add a correctly timed block immediately after.
8182016-07-01T23:02:15 *** zooko has quit IRC
8192016-07-01T23:02:55 <CubicEarth> That would appear as if the hashrate had fallen precipitously... not blocks for 24hr!
8202016-07-01T23:03:22 <gmaxwell> midnightmagic: XT did and released, it's merged in classic, but I don't know if it made it into a release yet.
8212016-07-01T23:03:31 <gmaxwell> These folks are extremely and dangerously incompetent.
8222016-07-01T23:04:03 <gmaxwell> CubicEarth: no, just a wonky timestamp, miners provide the timestamps and sometimes they're pretty wonky.
8232016-07-01T23:04:30 <gmaxwell> and while _syncing_ you have no idea that the timestamps were wonky.
8242016-07-01T23:05:19 <gmaxwell> e.g. someone can produce blocks that are -24h, -23h, -22h .... normal.
8252016-07-01T23:05:23 <midnightmagic> so.. the peer preference thing means some of the alt-branch software will soon start disconnecting the only nodes that are immune to the attack, making themselves *more* vulnerable to it?
8262016-07-01T23:07:30 <gmaxwell> midnightmagic: I don't think that matters that much, but right now bitcoin "xt" is only not totally partitioned due to inbound connections from core to XT, but part of segwit is that segwit capable nodes will strongly prefer to connect out to only non-segwit nodes (because they must in order to get blocks once segwit is activated)-- the result will be making that partition complete.
8272016-07-01T23:08:43 <midnightmagic> i thought -classic was now disconnecting non-classic nodes
8282016-07-01T23:09:22 <gmaxwell> no, Xt. I don't think classic is doing that yet.
8292016-07-01T23:10:16 <slackircbridge1> <alp> let them lose money for people
8302016-07-01T23:10:19 <slackircbridge1> <alp> its the only way theyll learn
8312016-07-01T23:10:37 <gmaxwell> It's not that simple. See also: Ethereum and DAO.
8322016-07-01T23:11:23 <gmaxwell> which adversely impacted the bitcoin price even though many of us have been pointing out the related risks and distinguishing bitcoin on the basis of them for some time.
8332016-07-01T23:13:18 <slackircbridge1> <brg444> :O
8342016-07-01T23:13:23 <CubicEarth> gmaxwell: well I'm pretty sure the rise of ETH was hurting the bitcoin price to some extent. I think this event was a decoupling, where Bitcoin shakes the monkey off its back.
8352016-07-01T23:13:31 <slackircbridge1> <brg444> did he read your post :O
8362016-07-01T23:16:03 *** jannes has quit IRC
8372016-07-01T23:24:28 <slackircbridge1> <alp> Short term pain, long term gain
8382016-07-01T23:27:56 <slackircbridge1> <moli> guys, we can read your posts on IRC lol
8392016-07-01T23:28:02 *** ChanServ sets mode: +o btcdrak
8402016-07-01T23:32:09 *** Chris_Stewart_5 has quit IRC
8412016-07-01T23:32:30 *** btcdrak sets mode: +scirg
8422016-07-01T23:32:31 *** ChanServ sets mode: -c
8432016-07-01T23:33:10 <btcdrak> wait that's wrong. I'm trying to set slackircbridge1 +q
8442016-07-01T23:33:59 <gmaxwell> lol
8452016-07-01T23:34:03 <gmaxwell> undo the damage first.
8462016-07-01T23:36:16 *** btcdrak sets mode: +o gmaxwell
8472016-07-01T23:37:02 *** gmaxwell sets mode: -sirg
8482016-07-01T23:38:11 *** gmaxwell sets mode: +q slackircbridge*!*@*
8492016-07-01T23:38:31 *** btcdrak sets mode: -o btcdrak
8502016-07-01T23:42:03 *** gmaxwell sets mode: -o gmaxwell