1 2016-12-05T00:00:04  <jtimon> but let me check that commit
  2 2016-12-05T00:00:04  <BlueMatt> oh, what? you told sipa that the "it updates the storage"
  3 2016-12-05T00:00:53  *** Chris_Stewart_5 has joined #bitcoin-core-dev
  4 2016-12-05T00:02:11  <jtimon> I mean it updates what you store, through the api (ie the library handles reorgs and updates things)
  5 2016-12-05T00:02:46  <BlueMatt> well of course, you tell the library about a block and it connects what it can, possibly asking the library owner for blocks that it previously saw but (ofc) didnt store
  6 2016-12-05T00:03:00  <BlueMatt> I dont see how else you'd possibly do it
  7 2016-12-05T00:03:33  <BlueMatt> gmaxwell: see, eg, the bitcoinj stuff, where all the contributions to "full mode" since it was added have been to add it to database X, so that people can put it in their own db
  8 2016-12-05T00:03:35  <jtimon> see the backlog for when I say for people answer the 4 possible combinations to 2 yes/no questions
  9 2016-12-05T00:05:38  <jtimon> you and me agree on abstracting storage
 10 2016-12-05T00:05:57  <jtimon> I mean, if I understood everyone's position correctly
 11 2016-12-05T00:06:38  <gmaxwell> BlueMatt: without using a highly efficient format like ours, I'm dubious that the system can stay in sync without insane hardware.
 12 2016-12-05T00:06:46  <jtimon> the other question is only check that a block is valid or also handle reorgs, update tables etc
 13 2016-12-05T00:06:52  <BlueMatt> gmaxwell: thats not our problem, thats theirs
 14 2016-12-05T00:06:58  <BlueMatt> gmaxwell: hell, they can use leveldb if they want
 15 2016-12-05T00:07:02  <jtimon> ie verifyBlock vs processBlock
 16 2016-12-05T00:07:20  <BlueMatt> gmaxwell: and, if we prefer, its also easy to add eg a single flag like "use X MB of utxo cache, because I dont want to implement that myself"
 17 2016-12-05T00:07:23  <gmaxwell> BlueMatt: not just about leveldb or not, but the compressed ccoins representation is worthless for queries.
 18 2016-12-05T00:08:07  *** Chris_Stewart_5 has quit IRC
 19 2016-12-05T00:08:27  <BlueMatt> gmaxwell: the compressed ccoins thing doesnt matter all that much if you're talking about an actually-high-performance db on high-performance hardware....if folks need their shit in their own db and are willing to pay $$$$$ for it to run, more power to them
 20 2016-12-05T00:08:33  <gmaxwell> As far as 'their problem' goes, we shouldn't waste our resources (or code base clarity, or performance) supporting functionality that won't be useful to anyone (or to anyone beyond a couple centeralized API services).
 21 2016-12-05T00:08:45  <jtimon> we can add a wrapper with our own implementation of the interfaces, beyond that, right, it is their problem
 22 2016-12-05T00:08:55  <BlueMatt> gmaxwell: ok, then what is the point of libconsensus at all?
 23 2016-12-05T00:09:19  <sipa> BlueMatt: i don't care about libconsensus. i care about abstracting consensus logic out
 24 2016-12-05T00:09:20  <BlueMatt> and whats your answer to folks like btcd/the new javascript one, who do have dbs that are performant enough to stay in sync
 25 2016-12-05T00:09:20  <gmaxwell> The way it was pitched to me is so that people could make wallets and other similar applications without having to reimplement consensus logic.
 26 2016-12-05T00:09:34  <jtimon> right
 27 2016-12-05T00:09:38  *** RoyceX has quit IRC
 28 2016-12-05T00:09:53  <BlueMatt> gmaxwell: sure, but that doesnt mean we have to handle db logic ourselves
 29 2016-12-05T00:09:58  <sipa> well, the hardest part of that is already available: we expose script verification
 30 2016-12-05T00:10:02  <jtimon> how is that incompatible with allowing them to chose their database implementation?
 31 2016-12-05T00:10:14  <BlueMatt> sipa: I dont think thats the only hard part, really
 32 2016-12-05T00:10:16  <sipa> but i also don't think it's very hard to abstract utxo storage out... so if there is a use case, sure
 33 2016-12-05T00:10:44  <BlueMatt> sipa: indeed, abstracting out utxo/block storage is also abstracting consensus logic out of other crap
 34 2016-12-05T00:10:45  <sipa> maybe it is.
 35 2016-12-05T00:11:11  <sipa> something like changing to per-output rather than per-tx utxo model would be impossible with a stable utxo storage abstraction
 36 2016-12-05T00:11:38  <jtimon> I assume the use cases come from the fact that they want to reuse that database for some of their logic somehow
 37 2016-12-05T00:11:44  <sipa> s/impossible/inefficient or complicated/
 38 2016-12-05T00:11:47  <gmaxwell> I'm not opposed to it being abstractable-- but I don't see how this is related to that goal-- it's the opposite of it, the blockchain storage and utxo set is consensus and may even be quite normative in its behavior (e.g. if we have a commited utxo set of some kind), and if it trashes performance or code clarity then it's not a good move.
 39 2016-12-05T00:12:15  *** MarcoFalke has left #bitcoin-core-dev
 40 2016-12-05T00:12:52  <BlueMatt> sipa: oh? if you query per-utxo then per-tx can be hidden on the backend and could still be pretty performant....indeed, the other way around doesnt really work
 41 2016-12-05T00:12:53  <gmaxwell> and for an example of the kind of complexity it creates: if you think you can just query a utxo database it means we can't have writeback caching internally.
 42 2016-12-05T00:13:03  <BlueMatt> gmaxwell: sure, no one wants to do anything that ends up introducing performance regressions
 43 2016-12-05T00:13:41  <sipa> abstracting storage is more to avoid dependencies rather than it being reusable
 44 2016-12-05T00:13:55  <gmaxwell> I think callers that want a database probably don't want to replace the database used for consensus-- what they want is a node that maintains an external database for their application.
 45 2016-12-05T00:14:09  <gmaxwell> Which is probably not the same thing due to consistency requirements.
 46 2016-12-05T00:15:19  <jtimon> gmaxwell: it shouldn't trash performance or code clarity, I agree
 47 2016-12-05T00:15:32  <BlueMatt> gmaxwell: it might be worse performance, but syncing after every block after ibd (ie having a flag to sync) isnt all that hard, either....
 48 2016-12-05T00:15:42  <BlueMatt> eg ProcessNewBlock(block); SyncToDB();
 49 2016-12-05T00:16:10  <sipa> BlueMatt: but for a full node, you probably don't want to sync after every block
 50 2016-12-05T00:16:11  <BlueMatt> gmaxwell: and its easy to do that without introducing our own performance regressions....just dont call Sync after every block....
 51 2016-12-05T00:16:21  <BlueMatt> sipa: depending on your application, maybe you do
 52 2016-12-05T00:16:30  <jtimon> sipa: once abstracted out, changing the interface in certain ways may be painful, correct
 53 2016-12-05T00:16:33  <BlueMatt> at least after IBD
 54 2016-12-05T00:16:37  <gmaxwell> Not writing out the utxo set constantly is critical for performance. Leveldb is slow.
 55 2016-12-05T00:16:47  <sipa> BlueMatt: but it'd be using our block validation code, which has known performance characteristics
 56 2016-12-05T00:17:04  <sipa> BlueMatt: and if you don't care about that, you wouldn't be using libconsensus at all
 57 2016-12-05T00:17:14  <BlueMatt> sipa: as long as we dont drop the cache when we flush (like we do now, which we already need to fix), I dont see a performance issue there?
 58 2016-12-05T00:17:30  <sipa> BlueMatt: fair enough
 59 2016-12-05T00:17:42  <sipa> BlueMatt: that's a good point - but the current code doesn't do that
 60 2016-12-05T00:17:52  <gmaxwell> I can't imagine an application which needs to muck around storing the utxo database in an application format which wouldn't be equally or better served by block processing callback that maintains an external database that isn't used for validation.
 61 2016-12-05T00:17:57  <BlueMatt> suresure, but these are all minor issues that are trivial to fix
 62 2016-12-05T00:17:57  <jtimon> avoiding dependencies is also a great gain, although I think we should have a version depending on levelDB and our own implementation too
 63 2016-12-05T00:18:00  <sipa> again, i'm not against abstracting things out
 64 2016-12-05T00:18:17  <gmaxwell> Widespread application visiblity into the actual utxo database would be pretty toxic for commited utxo or stxo improvements.
 65 2016-12-05T00:18:22  <BlueMatt> gmaxwell: duplicate databases? people who run shit on modern services where you literally have no local persistent storage?
 66 2016-12-05T00:18:25  <sipa> i very much feel that utxo storage is one of the things that is abstractable
 67 2016-12-05T00:18:45  <sipa> but that doesn't mean it's necessarily useful for sharing that information with other purposes
 68 2016-12-05T00:18:49  <sipa> it also doesn't mean it's not
 69 2016-12-05T00:19:15  <jtimon> yeah, I assume one use case could be having everything in memory
 70 2016-12-05T00:19:25  <gmaxwell> BlueMatt: basically any standard database approach would have horrible performance to the point that it would only be useable on very high end hardware.  Having two copies of the utxo set would hardly be a consideration there, our copy is only about 2GB data.
 71 2016-12-05T00:20:00  <BlueMatt> gmaxwell: I seriously dont believe that...maybe it takes 10 seconds or 30 seconds to sync a block to the db, but so what? you just dont flush all the time during IBD and then wait it out
 72 2016-12-05T00:20:19  <BlueMatt> (and, yes, I get that in most dbs it actually will take 30 seconds, but it wont take much longer than that)
 73 2016-12-05T00:20:25  <BlueMatt> or a minute with segwit blocks
 74 2016-12-05T00:20:29  * BlueMatt -> out
 75 2016-12-05T00:20:46  <gmaxwell> BlueMatt: see also the electrum channel with people complaing about their servers falling multiple blocks behind.
 76 2016-12-05T00:20:57  <gmaxwell> and two minute updates.
 77 2016-12-05T00:21:36  *** Chris_Stewart_5 has joined #bitcoin-core-dev
 78 2016-12-05T00:22:05  <BlueMatt> gmaxwell: ok...? that doesnt mean its impossible to build a db that can store the utxo set with reasonable performance, even if not blazing fast performance?
 79 2016-12-05T00:22:22  <gmaxwell> I just feel that the purpose of these changes is no longer clear. Expecting the user to implement complex interfaces with bitcoin specific and consensus critical behavior is at odds with what I understood to be the stated goal.
 80 2016-12-05T00:22:31  * BlueMatt doesnt see the problem with it taking a minute for your db to sync the utxo set....if you're just a merchant and its slow, so what?
 81 2016-12-05T00:22:38  <jtimon> it is also possible to build something faster than our solution, isn't it?
 82 2016-12-05T00:23:02  <BlueMatt> gmaxwell: I highly disagree that selecting a sane DB (ie not using some external SQL thing) is "implementing complex interfaces"
 83 2016-12-05T00:23:08  <gmaxwell> And if maintaining a database for a block explorer is a goal-- then it can be done in a better way then also trying to use that database for consensus... just run it in parallel, the resource overhead will be moderate.
 84 2016-12-05T00:23:27  <BlueMatt> and I dont expect every exchange to do so, I'd expect folks like btcd/bitcoinj/javascript thinggy to do it
 85 2016-12-05T00:23:29  <BlueMatt> and people to use it
 86 2016-12-05T00:23:44  <BlueMatt> gmaxwell: go look at the bitcoinj users...people actually use its full validation shitshow so that they can do exactly this
 87 2016-12-05T00:23:47  <jtimon> gmaxwell: we don't need to expect the user to reimplement it, we should provide our own implementation to the interfaces
 88 2016-12-05T00:23:47  <BlueMatt> and its slow, but they dont care
 89 2016-12-05T00:23:54  <BlueMatt> (and the interface is trivial)
 90 2016-12-05T00:24:11  <gmaxwell> BlueMatt: Complex interfaces is that you need to actually pass the data as fields and not opaque blobs.   And what happens when the set of data needed for consensus changes (as sequence locktiming did).
 91 2016-12-05T00:24:15  <BlueMatt> its like 4 functions and only a single non-trivial requirement for which there are unit tests (the utxo-replacement-thing)
 92 2016-12-05T00:24:39  <jtimon> again, you keep discussing the slow case, what if I'm faster than our implementation?
 93 2016-12-05T00:24:54  <gmaxwell> BlueMatt: that usage doesn't even need full node support at all, run your bitcoinj behind a full node, have it log blocks to a database. Hurray.
 94 2016-12-05T00:25:09  <BlueMatt> gmaxwell: yes, except people dont do that
 95 2016-12-05T00:25:20  <sipa> maybe we should write a simple daemon that maintains the utxo set in SQL
 96 2016-12-05T00:25:22  <BlueMatt> people prefer to run full validation shit in btcd or bitcoinj, despite knowingly putting themselves at risk
 97 2016-12-05T00:25:27  <sipa> (without validation)
 98 2016-12-05T00:25:28  <jtimon> or are we discarding that as a possibility?
 99 2016-12-05T00:25:31  <gmaxwell> sipa: that is what I was saying.
100 2016-12-05T00:25:38  <gmaxwell> BlueMatt: they don't know they're putting themselves at risk.
101 2016-12-05T00:26:12  <BlueMatt> gmaxwell: ok, well either way I dont see an alternative great solution here? most developers do want a library that handles all the background validation shit for them
102 2016-12-05T00:26:19  <BlueMatt> and as long as such things exist, they will use them
103 2016-12-05T00:26:20  <gmaxwell> jtimon: because there exists nothing faster currently, or else we'd be using it.
104 2016-12-05T00:26:24  <BlueMatt> proxy-nodes be damned
105 2016-12-05T00:26:42  <gmaxwell> BlueMatt: the alternative is to have support for maintaining a database synced with the blockchain-- that doesn't mean inserting things into the consensus logic.
106 2016-12-05T00:27:15  <gmaxwell> It just means having a simple set of hooks that run post tip change and update the database.
107 2016-12-05T00:27:39  <BlueMatt> gmaxwell: you missed my previous point about how lots of "modern" developers run shit on services with ~no persistent storage
108 2016-12-05T00:27:48  <BlueMatt> anyway, I actually need to run, have an apt to keep
109 2016-12-05T00:27:58  <sipa> run, BlueMatt, run
110 2016-12-05T00:28:10  <jtimon> there's no storage solution optimized for having everything in memory that outperforms leveldb or offers some other advantage?
111 2016-12-05T00:28:11  <sipa> apt-keep BlueMatt
112 2016-12-05T00:28:24  <sipa> jtimon: -dbcache=8000
113 2016-12-05T00:28:31  <gmaxwell> We already have that built in.
114 2016-12-05T00:28:31  <sipa> jtimon: leveldb won't be used at all
115 2016-12-05T00:29:11  <jtimon> sipa: right, I know, but I doubt there's nothing better than levle db with unbounded cache
116 2016-12-05T00:29:23  <jtimon> sipa: levledb is what we're using, no?
117 2016-12-05T00:29:25  *** e4xit has quit IRC
118 2016-12-05T00:29:38  <gmaxwell> jtimon: that doesn't use leveldb (except to persist across restarts)
119 2016-12-05T00:29:54  <jtimon> oh
120 2016-12-05T00:30:09  <jtimon> I see, I didn't know that, thanks
121 2016-12-05T00:32:09  <jtimon> even if we only expose the version with our implementation, I think it would be good to abstract consensus storage even for bitcoin core rewardless of libconsensus users
122 2016-12-05T00:32:48  <gmaxwell> our storage is already abstracted.
123 2016-12-05T00:34:16  <jtimon> I really think not trashing preformance for our own implementation should be the priority at first, if the interface needs to go through a few iterations not to trash other implementations too, I think that's fine since we will be offering the "don't use storage abstractions" version too anyway
124 2016-12-05T00:35:30  *** alpalp has quit IRC
125 2016-12-05T00:35:35  <jtimon> yes, our storage is abstracted in more ways than we would want to expose in an storage-independent libconsensus C API
126 2016-12-05T00:37:15  <jtimon> I mean, since I'm in favor of exposing both, one storage independent and one that is not, I'm fine with starting with the one that is not, I'm just more interested technically in the other one
127 2016-12-05T00:39:27  <jtimon> can we talk a little bit about the other question?
128 2016-12-05T00:42:55  <jtimon> ie whether libconsensus should fully validate a given block, or also accept it and update the state, manage reorgs, etc
129 2016-12-05T00:44:11  <jtimon> again I'm ok with offering both but I'm more interested in the smaller one
130 2016-12-05T00:44:46  *** droark has quit IRC
131 2016-12-05T00:47:42  <jtimon> or at least that's what I have been imagining all this time, I wasn't counting reorgs or updating the tip as part of the validation
132 2016-12-05T00:50:52  *** e4xit has joined #bitcoin-core-dev
133 2016-12-05T01:01:06  *** alpalp has joined #bitcoin-core-dev
134 2016-12-05T01:01:06  *** alpalp has joined #bitcoin-core-dev
135 2016-12-05T01:05:30  <morcos> gmaxwell: sdaftuar: to return to the fee bumping, in suhas' example, where you first try to bump tx1,2,3 with tx4, and then you try again.
136 2016-12-05T01:05:54  <morcos> i can see how you could invent a way to prevent tx4 from getting bumped, but how are you stopping 1,2,3 from being bumped again?
137 2016-12-05T01:06:30  <morcos> or more generally, lets say you manually had tried to bump and tx1 and tx1a both are trying to pay the same guy (maybe you were smart enought to conflict, maybe not)
138 2016-12-05T01:06:33  <gmaxwell> They should be marked abandoned when the bump is created.
139 2016-12-05T01:06:40  <morcos> how do you stop a bumpunconfirmed from bumping both
140 2016-12-05T01:06:47  <gmaxwell> (but even if they weren't they're conflicted at that point)
141 2016-12-05T01:06:54  <morcos> they aren't conflicted
142 2016-12-05T01:07:03  <gmaxwell> by tx4, which is in your mempool.
143 2016-12-05T01:07:06  <morcos> conflicted means the conflicting tx is in the block
144 2016-12-05T01:07:23  <morcos> a conflicting tx in the mempool is something you are not even aware of
145 2016-12-05T01:08:03  <morcos> which brings me to the second point i wanted to make, aobut what you said about people abandoning wallets they think are empty
146 2016-12-05T01:08:14  <morcos> thats a fantastic point, that i wish had been made a while ago
147 2016-12-05T01:08:31  <gmaxwell> you're right, I'd been assuming it would be conflicted without thinking carefully what that test actually means right now.
148 2016-12-05T01:08:53  <gmaxwell> I know for sure people do abandon wallets that they think are empty (or even 'almost empty')
149 2016-12-05T01:09:09  <morcos> before we made the change to the confliction logic (for 0.12 ?)  then if your spend was not in your mempool, it was considered conflicted (regardless of whether it was by an in-block, in-mempool tx, or nothing)
150 2016-12-05T01:09:23  <morcos> so it would be kind of rare for you to think you were out of money but not
151 2016-12-05T01:09:31  <morcos> but now, for sure that might happen
152 2016-12-05T01:10:01  <morcos> now if you issue a tx that never makes it into a block or for some reason can't ever make it into your own mempool
153 2016-12-05T01:10:13  <gmaxwell> I seem to vaguely recall that something else would still prevent us from doublespending the inputs on txn that weren't in the mempool, even then.
154 2016-12-05T01:10:15  <morcos> regardless of any conflicting txs, its uses up your balance until you abandon it
155 2016-12-05T01:10:25  <morcos> i don't think so
156 2016-12-05T01:11:04  <morcos> it seems like we need another notion of balance (or maybe 2 more)
157 2016-12-05T01:11:53  <morcos> potential balance (which is not reduced by non-in chain (6 deep?) spends) and maybe even a pending receive balance (although i guess that hasn't been a problem historically, that hasnt' changed)
158 2016-12-05T01:15:16  <gmaxwell> Yes, agreed. I worry about the use 'balance' for a number which will go down without user interaction. :)
159 2016-12-05T01:15:20  *** CubicEarth has quit IRC
160 2016-12-05T01:15:56  *** CubicEarth has joined #bitcoin-core-dev
161 2016-12-05T01:16:13  <gmaxwell> More like "pending outgoing payments: outbound payments which are not N confirms deep yet" and "pending incoming payments".
162 2016-12-05T01:16:26  <Taek> Would it be enough to have a [confirmed balance] and an [unconfirmed diff]?
163 2016-12-05T01:16:44  <morcos> if this is really the only use case, it would be easy enough to make a rpc call that just give you a report on how "empty" your wallet is
164 2016-12-05T01:17:14  <morcos> Taek: we already have that..  , oh yeah, nm, thats the second thing i was talking abou tthen, the pending received balance
165 2016-12-05T01:17:22  <morcos> getunconfirmedbalance
166 2016-12-05T01:17:49  <Taek> if I'm following correctly, the worry is about coins that become unconfirmed due to e.g. change outputs?
167 2016-12-05T01:18:15  *** droark has joined #bitcoin-core-dev
168 2016-12-05T01:18:16  <morcos> i think the primary worry is that if you spend some coins , but your spend never makes it into a block
169 2016-12-05T01:18:27  <morcos> your wallet still deducts that spend from your balance
170 2016-12-05T01:18:29  <morcos> forever
171 2016-12-05T01:19:04  <morcos> until you manually mark it as abandoned (which is sort of an advanced feature, that we don't often recommend)
172 2016-12-05T01:19:38  <Taek> technically some adversary could un-abandon any transaction that hasn't been double-spent
173 2016-12-05T01:19:57  *** CubicEarth has quit IRC
174 2016-12-05T01:20:04  <morcos> exactly why its an advanced manual feature
175 2016-12-05T01:20:05  <Taek> imo your confirmed balance should not change until the tx is in the blockchain
176 2016-12-05T01:20:54  <Taek> and the confirmed balance should be what is presented to the user as the primary balance
177 2016-12-05T01:21:03  <morcos> perhaps confirmed balance is the wrong word for what getbalance returns...  it returns your spendable balance..  which certainly should be decremented for spends that haven't yet made it into the blockchain
178 2016-12-05T01:21:16  <morcos> and i think thats what people expect to see when they ask their balance
179 2016-12-05T01:22:53  <Taek> I don't think it's safe enough to show the user just one number.
180 2016-12-05T01:23:14  <Taek> simply because the whole unconfirmed uncertainty is unseparatable from the blockchain way of doing thigns
181 2016-12-05T01:23:30  <Taek> (well, lightning doesn't really have this issue)
182 2016-12-05T01:25:40  <BlueMatt> gmaxwell: thinking about it more, the way we'd probably do it is, initially (ie v1) you make the libconsensus consumer provide a k-v store api, and we use that the same way we use leveldb, and then add functionality to parse the blobs we provide the k-v store into things like scriptPubKeys later
183 2016-12-05T01:25:41  <jtimon> perhaphs both confirmed and spendable balances should be shown?
184 2016-12-05T01:25:52  <BlueMatt> this provides functionality, without breaking our ability to change the format to add new things
185 2016-12-05T01:26:56  <jtimon> I understand k-v is key-value kind of storage
186 2016-12-05T01:27:30  <jtimon> BlueMatt: if so, what would the values be? C structs ?
187 2016-12-05T01:27:47  <BlueMatt> the values are binary blobs that libconsensus provides which the user does not have any visibility into
188 2016-12-05T01:28:02  <BlueMatt> (in our case its the serialization of CCoins or whatever with our compression stuff)
189 2016-12-05T01:28:19  <sipa> BlueMatt: yup, that's what i imagined
190 2016-12-05T01:28:32  <BlueMatt> if the user wants to know whats inside, there is some api which can parse it into a c struct or whatever
191 2016-12-05T01:28:49  <sipa> a batch key-value write operation, and a key read operation
192 2016-12-05T01:29:05  <jtimon> kind of like https://github.com/bitcoin/bitcoin/pull/8493/commits/ad3ac37387b231378573f4996c59467247fccd44 ?
193 2016-12-05T01:30:15  <jtimon> ^ for the "binary blobs the caller doesn't know about"
194 2016-12-05T01:30:59  <sipa> jtimon: i don't see such a thing at all
195 2016-12-05T01:31:22  <sipa> that commit is about bitcoinconsensus_create_consensus_parameters
196 2016-12-05T01:32:06  <jtimon> well, yeah, sorry, this are void pointers the other it's just data like in the tx for current verifyScript
197 2016-12-05T01:33:06  <jtimon> in this case we would use the serialize lib to interpret and produce the "blobs", correct?
198 2016-12-05T01:33:55  <sipa> if there needs to be a way to view the utxo set, i'd just provide a separate api for that
199 2016-12-05T01:34:00  <sipa> not a parser for the database
200 2016-12-05T01:34:09  <sipa> and not in the first stage
201 2016-12-05T01:36:01  <jtimon> I am extremely interested in hearing in what other people's next steps, or more feedback on my own proposed next step
202 2016-12-05T01:38:30  <jtimon> sipa: I'm still not sure if you prefer to expose verifyBlock or processBlock
203 2016-12-05T01:39:41  <sipa> jtimon: i don't think we can do so right now anyway, without having a way to abstract state out
204 2016-12-05T01:40:12  <sipa> imho the first step is just continuing refactoring so that consensus logic and other things become better separated internally
205 2016-12-05T01:40:24  <sipa> and not focus on exposing things
206 2016-12-05T01:40:49  <sipa> but others may disagree - i think wumpus prefers first having a clear idea of what will be exposed
207 2016-12-05T01:41:29  <sipa> even a verifyBlock will need a way to pass in the utxo set and the block index
208 2016-12-05T01:42:27  <sipa> the only difference is that a processBlock doesn't need a way to update set utxo set, and doesn't need to be able to request other blocks in case of a reorg
209 2016-12-05T01:42:41  *** Ylbam has quit IRC
210 2016-12-05T01:44:07  <jtimon> althought I tend to agree, I feel that's very vague and doesn't help on clarifying priorities, thinking of the next thing to expose, I think, helps clarify what the goal of the refactors should be and where are we supposed to be moving towards to
211 2016-12-05T01:44:36  <sipa> you know my opinion - i don't care about exposing anything at all at this point, so i'm the wrong person to ask
212 2016-12-05T01:44:47  <jtimon> yes verifyBlock would need an interface to access data from the utxo
213 2016-12-05T01:44:51  <sipa> i think we have harder problems to solve before exposing even comes into question
214 2016-12-05T01:45:22  *** Giszmo has joined #bitcoin-core-dev
215 2016-12-05T01:45:52  <jtimon> dcousens: proposal was to pass all required data explicitly for the block you were validating
216 2016-12-05T01:46:15  <sipa> essentially, i think we should first introduce clean abstractions between certain modules inside bitcoin core, in such a way that it's effectively bitcoind using a consensus library already, without it being exposed
217 2016-12-05T01:46:31  <sipa> when it's good enough for us to use, we can think about exposing it to others
218 2016-12-05T01:47:06  <sipa> (but again, others may see things differently)
219 2016-12-05T01:48:01  <jtimon> right, and I think the module that should be a priority to cleanly separate is the part of the code that is required to fully validate whether a block is valid or not from everything else
220 2016-12-05T01:48:54  <sipa> but that's very tightly coupled with validation of the whole chain, through CBlockIndex
221 2016-12-05T01:49:26  <jtimon> right, it basically depends on chain.o and coins.o
222 2016-12-05T01:50:09  <sipa> you can't validate a block without knowing its CBlockIndex
223 2016-12-05T01:50:24  <jtimon> more specifically on two existing classes on them, an API for that is not hard
224 2016-12-05T01:50:26  <sipa> so i'm not sure whether "single block validation" is a useful abstraction on its own
225 2016-12-05T01:50:47  <sipa> transaction validation may be useful
226 2016-12-05T01:51:36  <jtimon> CBlockIndex is the storage interface I abstract (or abstract from its own exsiting abstraction) in 8493
227 2016-12-05T01:51:48  <sipa> i don't want an abstraction for CBlockIndex
228 2016-12-05T01:52:24  <jtimon> my proposed next steps are single header validation or single tx validation
229 2016-12-05T01:52:33  <jtimon> but without policy checks
230 2016-12-05T01:53:34  <sipa> as i said, i don't think we should focus on exposing interfaces now, but on separating modules
231 2016-12-05T01:53:49  <sipa> and i think separating block validation from chain validation is hard
232 2016-12-05T01:54:03  <BlueMatt> jtimon: I'm with sipa here - The Main Split was the first of many steps that make sense on their own to abstract out consensus and non-consensus code
233 2016-12-05T01:54:19  <BlueMatt> jtimon: the few commits I sent you earlier form the very tiny beginning of what I think are the next steps
234 2016-12-05T01:54:48  <jtimon> separating network things was absolutely brilliant, thanks again
235 2016-12-05T01:54:50  <BlueMatt> jtimon: ie having a state object internally which keeps chainstate in it and calls out to things for disk access and has ProcessNewBlock as a member function
236 2016-12-05T01:55:09  <sipa> BlueMatt: chainstate include mapBlockIndex?
237 2016-12-05T01:55:25  <BlueMatt> and its ~no code changes, just some function splitting and putting ClassName:: in front of them
238 2016-12-05T01:55:34  <BlueMatt> sipa: yes, mapBlockIndex and chainActive and related variables
239 2016-12-05T01:55:51  <BlueMatt> sipa: but calling out for ReadBlockFromDisk, and pcoinsTip is just a pointer that is passed to it
240 2016-12-05T01:56:03  <sipa> BlueMatt: got it
241 2016-12-05T01:56:08  <sipa> seems like an easy first step
242 2016-12-05T01:56:34  <jtimon> sipa: if you don't care about exposing, that's fine, let's talk about dependencies, I want the consensus module to fully verify a single tx and a single header and a single block without depending on coins.o or chain.o
243 2016-12-05T01:56:34  <BlueMatt> yea, should be pretty clean...I dont have time to do it for the next week or two...do you want to take it up jtimon?
244 2016-12-05T01:56:50  <BlueMatt> I also want to work on splitting up net_processing more so that we can multithread ProcessMessages
245 2016-12-05T01:57:07  <BlueMatt> jtimon: I dont see how thats possibel?
246 2016-12-05T01:57:19  <sipa> yes, i think those are possible
247 2016-12-05T01:57:22  <jtimon> I don't know what you want to do, how can I pick it up?
248 2016-12-05T01:57:23  <sipa> (don't
249 2016-12-05T01:57:36  <BlueMatt> jtimon: literally the point of "fully validating" a tx is to validate it against a CCoins-holding UTXO db
250 2016-12-05T01:57:51  <BlueMatt> jtimon: did you look at the top commit on the branch I sent you?
251 2016-12-05T01:57:53  <sipa> efficiency of validation is highly dependent on low-level access to coins and chain
252 2016-12-05T01:57:57  <jtimon> sigh, I thought I had proved it was possible repeated times...
253 2016-12-05T01:58:05  <BlueMatt> jtimon: https://github.com/TheBlueMatt/bitcoin/commit/54c967e73a1d1abba8f426fce0f14c5eeaf277e6
254 2016-12-05T01:58:08  <sipa> it's possible if you introduce abstractions everywhere
255 2016-12-05T01:58:23  <jtimon> is it possible to fully validate a header without depending on chain.o?
256 2016-12-05T01:58:33  <sipa> no
257 2016-12-05T01:58:59  <sipa> (unless you abstract it out, of course)
258 2016-12-05T01:59:02  <jtimon> then what's happening in 8493
259 2016-12-05T01:59:06  <jtimon> right
260 2016-12-05T01:59:32  <sipa> but i think such abstraction are both a performance issue and an unnecessary code complication
261 2016-12-05T02:00:22  <jtimon> well, more than half of 8493 is purely for demonstrating the exposed api and without benchmarking of any kind
262 2016-12-05T02:03:50  <jtimon> my goal was to separate the code the verify a full block, depending on chain.o and coins.o (but only on those related to storage) [maybe put it all in the consensus folder? or wait for later?] but not putting it in the consensus module until you want to expose more and abstract it from chain and coins
263 2016-12-05T02:05:42  <jtimon> anyway, I'm happy reviewing any related refactors, please ping me
264 2016-12-05T02:07:01  <jtimon> sipa: does the GetConsensusFlag make any sense to you at a first glance? at least more than the previous version?
265 2016-12-05T02:08:22  <jtimon> without exposing anything, just as a refactor (note that calling GetConsensusFlag inside ContextualCheckBlock is painful performance-wise)
266 2016-12-05T02:27:28  *** netzin has joined #bitcoin-core-dev
267 2016-12-05T02:32:08  *** netzin has quit IRC
268 2016-12-05T02:38:54  *** wasi has quit IRC
269 2016-12-05T02:43:22  *** To7 has joined #bitcoin-core-dev
270 2016-12-05T02:46:57  *** wasi has joined #bitcoin-core-dev
271 2016-12-05T03:24:20  *** Guest13131 has quit IRC
272 2016-12-05T03:30:12  *** Guest13131 has joined #bitcoin-core-dev
273 2016-12-05T03:36:46  *** molz has joined #bitcoin-core-dev
274 2016-12-05T03:38:06  *** Alopex has quit IRC
275 2016-12-05T03:39:11  *** Alopex has joined #bitcoin-core-dev
276 2016-12-05T03:39:57  *** moli has quit IRC
277 2016-12-05T03:49:02  *** moli has joined #bitcoin-core-dev
278 2016-12-05T03:50:12  *** molz has quit IRC
279 2016-12-05T03:52:37  *** alpalp has quit IRC
280 2016-12-05T03:57:54  *** wasi has quit IRC
281 2016-12-05T04:03:53  *** justan0theruser has joined #bitcoin-core-dev
282 2016-12-05T04:04:00  *** justanotheruser has quit IRC
283 2016-12-05T04:16:07  *** Alopex has quit IRC
284 2016-12-05T04:17:12  *** Alopex has joined #bitcoin-core-dev
285 2016-12-05T04:17:24  *** molz has joined #bitcoin-core-dev
286 2016-12-05T04:20:09  *** moli has quit IRC
287 2016-12-05T04:42:41  *** PaulCapestany has quit IRC
288 2016-12-05T04:45:27  *** PaulCapestany has joined #bitcoin-core-dev
289 2016-12-05T05:02:45  *** MykelSIlver has quit IRC
290 2016-12-05T05:04:41  *** Giszmo has quit IRC
291 2016-12-05T05:07:19  *** arubi_ has joined #bitcoin-core-dev
292 2016-12-05T05:09:44  *** arubi has quit IRC
293 2016-12-05T05:45:51  <bitcoin-git> [bitcoin] jtimon opened pull request #9279: Consensus: Move CFeeRate out of libconsensus (master...0.13-consensus-dust-out-minimal) https://github.com/bitcoin/bitcoin/pull/9279
294 2016-12-05T05:46:14  <bitcoin-git> [bitcoin] jtimon closed pull request #7820: Consensus: Policy: Move CFeeRate out of consensus module and create CPolicy interface (master...0.12.99-consensus-dust-out) https://github.com/bitcoin/bitcoin/pull/7820
295 2016-12-05T05:52:04  *** Atomicat_ has joined #bitcoin-core-dev
296 2016-12-05T05:53:09  *** _mn3monic has quit IRC
297 2016-12-05T05:53:17  *** _mn3monic has joined #bitcoin-core-dev
298 2016-12-05T05:53:41  *** To7 has quit IRC
299 2016-12-05T05:54:13  *** Atomicat has quit IRC
300 2016-12-05T05:54:16  *** Atomicat_ is now known as Atomicat
301 2016-12-05T06:10:03  *** NielsvG has quit IRC
302 2016-12-05T06:15:28  *** NielsvG has joined #bitcoin-core-dev
303 2016-12-05T06:15:28  *** NielsvG has joined #bitcoin-core-dev
304 2016-12-05T06:40:49  *** Ylbam has joined #bitcoin-core-dev
305 2016-12-05T06:47:20  <gmaxwell> FWIW, I'm noticing connection slots full on my nodes.
306 2016-12-05T06:47:37  *** BCBot_ has joined #bitcoin-core-dev
307 2016-12-05T06:48:20  *** bobbytux_ has joined #bitcoin-core-dev
308 2016-12-05T06:48:59  <gmaxwell> including some clown at who looks like he's connected three times to everyone; while pretending to be android wallet (he's not).
309 2016-12-05T06:50:03  *** Alina-malina_ has joined #bitcoin-core-dev
310 2016-12-05T06:50:16  *** ville- has joined #bitcoin-core-dev
311 2016-12-05T06:51:45  *** Cory has quit IRC
312 2016-12-05T06:52:30  *** bobbytux__ has quit IRC
313 2016-12-05T06:52:41  *** CubicEarth has joined #bitcoin-core-dev
314 2016-12-05T06:53:50  *** owowo has quit IRC
315 2016-12-05T06:53:50  *** Alina-malina has quit IRC
316 2016-12-05T06:53:51  *** ville-- has quit IRC
317 2016-12-05T06:53:51  *** BCBot has quit IRC
318 2016-12-05T06:54:56  *** Cory has joined #bitcoin-core-dev
319 2016-12-05T06:55:52  *** owowo has joined #bitcoin-core-dev
320 2016-12-05T06:57:01  *** paveljanik has quit IRC
321 2016-12-05T07:00:13  *** dermoth has quit IRC
322 2016-12-05T07:01:07  *** dermoth has joined #bitcoin-core-dev
323 2016-12-05T07:06:15  <bitcoin-git> [bitcoin] laanwj pushed 5 new commits to master: https://github.com/bitcoin/bitcoin/compare/4d955fc5824b...46904ee5d2ce
324 2016-12-05T07:06:16  <bitcoin-git> bitcoin/master a188353 Pieter Wuille: Switch GetTransaction to returning a CTransactionRef
325 2016-12-05T07:06:16  <bitcoin-git> bitcoin/master c3f5673 Pieter Wuille: Make CWalletTx store a CTransactionRef instead of inheriting
326 2016-12-05T07:06:17  <bitcoin-git> bitcoin/master 42fd8de Pieter Wuille: Make DecodeHexTx return a CMutableTransaction
327 2016-12-05T07:08:08  <bitcoin-git> [bitcoin] laanwj pushed 7 new commits to master: https://github.com/bitcoin/bitcoin/compare/46904ee5d2ce...d04aebaec7bb
328 2016-12-05T07:08:09  <bitcoin-git> bitcoin/master 6fdd43b Matt Corallo: Add struct to track block-connect-time-generated info for callbacks
329 2016-12-05T07:08:10  <bitcoin-git> bitcoin/master fd9d890 Matt Corallo: Keep blocks as shared_ptrs, instead of copying txn in ConnectTip
330 2016-12-05T07:08:10  <bitcoin-git> bitcoin/master ae4db44 Matt Corallo: Create a shared_ptr for the block we're connecting in ActivateBCS
331 2016-12-05T07:08:20  <bitcoin-git> [bitcoin] laanwj closed pull request #9014: Fix block-connection performance regression (master...2016-10-fix-tx-regression) https://github.com/bitcoin/bitcoin/pull/9014
332 2016-12-05T07:15:25  *** luke-jr has quit IRC
333 2016-12-05T07:27:07  *** fanquake has joined #bitcoin-core-dev
334 2016-12-05T07:36:05  *** Alina-malina_ has quit IRC
335 2016-12-05T07:36:06  *** Alina-malina_ has joined #bitcoin-core-dev
336 2016-12-05T07:36:31  *** AaronvanW has quit IRC
337 2016-12-05T07:36:37  *** fanquake has quit IRC
338 2016-12-05T07:42:44  *** kadoban has quit IRC
339 2016-12-05T07:44:06  <sipa> BlueMatt: i can't build flto with -O3, but i can with -O2
340 2016-12-05T08:00:51  <BlueMatt> sipa: strange...I was not doing that, iirc
341 2016-12-05T08:01:05  <BlueMatt> I think I tried -march=native originally, but turned it off when it didnt work
342 2016-12-05T08:01:08  <BlueMatt> (and ccache and all that shit)
343 2016-12-05T08:01:22  *** murchandamus has quit IRC
344 2016-12-05T08:02:03  <BlueMatt> gmaxwell: this is why I auto-ban on strange-ass nodes like that (and had already banned that ip)
345 2016-12-05T08:02:34  <BlueMatt> and why my public node is behind stupid-ridiculous ddos protection :)
346 2016-12-05T08:03:00  *** murchandamus has joined #bitcoin-core-dev
347 2016-12-05T08:04:51  <gmaxwell> sipa: why does O3 break it?!
348 2016-12-05T08:05:09  <sipa> gmaxwell: sense it makes none
349 2016-12-05T08:05:29  <sipa> a linker error with ~boost::filesystem::path
350 2016-12-05T08:07:17  <wumpus> never tried flto with -O3, no issues with -O2
351 2016-12-05T08:09:18  <sipa> (which afaik was what BlueMatt was seeing as well)
352 2016-12-05T08:10:02  <BlueMatt> sipa: yes, it was, though I thought I went back and disabled all the strange optimizations for testing when it failed
353 2016-12-05T08:10:06  <BlueMatt> actually, pretty sure I did
354 2016-12-05T08:17:13  <gmaxwell> I have a newly updated mass-connector/spy banlist which added 13 new entries:
355 2016-12-05T08:17:16  <gmaxwell> https://0bin.net/paste/iSGeBTgtmA8cSHGX#VBd97-ME4hUiIqJIOsd24oLfM57UXP3F3QfgzuoVHAD  -- with commandline syntax
356 2016-12-05T08:17:19  <gmaxwell> https://0bin.net/paste/zplzkwGmn6oeP6Ec#oTFWsZFpGQ1zR10Ofv3mJbIOGCnoshxQBQBJoWhSaip -- with debug console syntax
357 2016-12-05T08:19:18  *** LeMiner2 has joined #bitcoin-core-dev
358 2016-12-05T08:21:18  *** molz has quit IRC
359 2016-12-05T08:21:38  <wumpus> thanks
360 2016-12-05T08:21:43  *** LeMiner has quit IRC
361 2016-12-05T08:21:44  *** LeMiner2 is now known as LeMiner
362 2016-12-05T08:24:19  *** LeMiner2 has joined #bitcoin-core-dev
363 2016-12-05T08:25:27  *** afk11 has quit IRC
364 2016-12-05T08:26:15  *** afk11 has joined #bitcoin-core-dev
365 2016-12-05T08:26:15  *** afk11 has quit IRC
366 2016-12-05T08:26:15  *** afk11 has joined #bitcoin-core-dev
367 2016-12-05T08:26:51  *** LeMiner has quit IRC
368 2016-12-05T08:26:52  *** LeMiner2 is now known as LeMiner
369 2016-12-05T08:30:57  <Lightsword> gmaxwell, maybe as well
370 2016-12-05T08:31:45  <Lightsword> and and and
371 2016-12-05T08:31:48  <bitcoin-git> [bitcoin] jonasschnelli opened pull request #9280: [Qt] Show ModalOverlay by pressing the progress bar, allow hiding (master...2016/12/qt_modal) https://github.com/bitcoin/bitcoin/pull/9280
372 2016-12-05T08:32:37  <gmaxwell> Lightsword: I've only been including ones that show up on all my input hosts, unfortunately since everyone is at limits, that conceals a few.
373 2016-12-05T08:35:06  <BlueMatt> gmaxwell: just take the unlimited-connection-slots patch?
374 2016-12-05T08:37:25  <sipa> you can set -maxconnections=1000 without any patches
375 2016-12-05T08:41:41  <BlueMatt> sipa: huh? I thought that sets you at 125?
376 2016-12-05T08:42:09  <sipa> no
377 2016-12-05T08:42:16  <sipa> 125 is just the default
378 2016-12-05T08:42:29  <BlueMatt> sipa: its limited by available sockets
379 2016-12-05T08:42:31  <wumpus> depends on what the fd lmiit is
380 2016-12-05T08:42:35  <BlueMatt> which can be super low, because select()
381 2016-12-05T08:42:57  <BlueMatt> or this used to be the case
382 2016-12-05T08:43:00  <sipa> Warning: Reducing -maxconnections from 1000 to 873, because of system limitations.
383 2016-12-05T08:43:04  <sipa> ok, 873.
384 2016-12-05T08:43:10  <BlueMatt> oh, 873, hum, I thought it was lower
385 2016-12-05T08:43:12  <wumpus> select() can handle 1024 on most systems, that's pretty much enough for most cases
386 2016-12-05T08:43:22  <BlueMatt> whatever, I carry a patch to use poll() to make it actually higher....
387 2016-12-05T08:44:24  <wumpus> I guess we've held up switching to poll because we expected to switch to libevent any day, that's taking a bit longer than expected :)
388 2016-12-05T08:44:42  <sipa> Soon! (tm)
389 2016-12-05T08:44:47  <wumpus> yea :-)
390 2016-12-05T08:44:52  <gmaxwell> IIRC matt's patch is darn near trivial.
391 2016-12-05T08:45:14  *** JackH has quit IRC
392 2016-12-05T08:45:36  *** JackH has joined #bitcoin-core-dev
393 2016-12-05T08:46:26  <BlueMatt> gmaxwell: yes, but given that its 873 not 1XX as I'd thought, probably not worth it
394 2016-12-05T08:46:39  <BlueMatt> and libevent is actually sooner now
395 2016-12-05T08:49:00  *** BashCo has quit IRC
396 2016-12-05T09:01:13  *** jl2012 has quit IRC
397 2016-12-05T09:01:34  *** jl2012 has joined #bitcoin-core-dev
398 2016-12-05T09:03:29  *** Guest13131 has quit IRC
399 2016-12-05T09:07:05  *** ChillazZ has joined #bitcoin-core-dev
400 2016-12-05T09:07:28  *** ChillazZ is now known as Guest15721
401 2016-12-05T09:13:43  <bitcoin-git> [bitcoin] kallewoof opened pull request #9281: Refactor: Removed using namespace <xxx> from bench/ & test/ sources (master...no-using-namespace-bench-test) https://github.com/bitcoin/bitcoin/pull/9281
402 2016-12-05T09:18:44  *** wvr has quit IRC
403 2016-12-05T09:27:38  *** BashCo has joined #bitcoin-core-dev
404 2016-12-05T09:28:51  *** Guest15721 has quit IRC
405 2016-12-05T09:28:51  *** ChillazZ has joined #bitcoin-core-dev
406 2016-12-05T09:29:20  *** ChillazZ is now known as Guest62446
407 2016-12-05T09:33:54  *** Victorsueca has joined #bitcoin-core-dev
408 2016-12-05T09:36:12  *** Victor_sueca has quit IRC
409 2016-12-05T09:55:36  *** jl2012 has quit IRC
410 2016-12-05T09:55:57  *** jl2012 has joined #bitcoin-core-dev
411 2016-12-05T09:56:53  *** murchandamus has quit IRC
412 2016-12-05T09:57:16  *** murchandamus has joined #bitcoin-core-dev
413 2016-12-05T09:57:42  *** murchandamus has joined #bitcoin-core-dev
414 2016-12-05T09:57:46  *** wvr has joined #bitcoin-core-dev
415 2016-12-05T09:58:58  *** jl2012 has quit IRC
416 2016-12-05T09:59:03  <bitcoin-git> [bitcoin] laanwj pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/d04aebaec7bb...613bda418f83
417 2016-12-05T09:59:04  <bitcoin-git> bitcoin/master 634ad51 Pieter Wuille: Squashed 'src/leveldb/' changes from 20ca81f..a31c8aa...
418 2016-12-05T09:59:05  <bitcoin-git> bitcoin/master 605d701 Pieter Wuille: Merge in LevelDB 1.19 changes
419 2016-12-05T09:59:05  <bitcoin-git> bitcoin/master 613bda4 Wladimir J. van der Laan: Merge #8613: LevelDB 1.19...
420 2016-12-05T10:00:40  *** jl2012 has joined #bitcoin-core-dev
421 2016-12-05T10:02:38  *** AaronvanW has joined #bitcoin-core-dev
422 2016-12-05T10:02:38  *** AaronvanW has quit IRC
423 2016-12-05T10:02:38  *** AaronvanW has joined #bitcoin-core-dev
424 2016-12-05T10:03:07  <sipa> wumpus: yeah, i didn't mean to say we should use reuse_logs, just that the feature over time may be interesting to us
425 2016-12-05T10:03:13  <sipa> *roght now
426 2016-12-05T10:05:15  <wumpus> sipa: right
427 2016-12-05T10:09:00  *** ClockCat has joined #bitcoin-core-dev
428 2016-12-05T10:09:51  <bitcoin-git> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/613bda418f83...43e8150ef690
429 2016-12-05T10:09:52  <bitcoin-git> bitcoin/master 2efc438 Pieter Wuille: Align struct COrphan definition
430 2016-12-05T10:09:52  <bitcoin-git> bitcoin/master 43e8150 Wladimir J. van der Laan: Merge #9269: Align struct COrphan definition...
431 2016-12-05T10:10:05  <bitcoin-git> [bitcoin] laanwj closed pull request #9269: Align struct COrphan definition (master...oneorphan) https://github.com/bitcoin/bitcoin/pull/9269
432 2016-12-05T10:16:08  <bitcoin-git> [bitcoin] paveljanik opened pull request #9282: CMutableTransaction is defined as struct (master...20161205_CMutableTransaction_is_struct) https://github.com/bitcoin/bitcoin/pull/9282
433 2016-12-05T10:29:30  *** Victor_sueca has joined #bitcoin-core-dev
434 2016-12-05T10:31:55  *** Victorsueca has quit IRC
435 2016-12-05T10:34:51  *** MarcoFalke has joined #bitcoin-core-dev
436 2016-12-05T10:35:21  *** luke-jr has joined #bitcoin-core-dev
437 2016-12-05T10:36:47  <bitcoin-git> [bitcoin] sipa opened pull request #9283: A few more CTransactionRef optimizations (master...sharedblock2) https://github.com/bitcoin/bitcoin/pull/9283
438 2016-12-05T10:55:57  *** NielsvG has quit IRC
439 2016-12-05T11:19:55  *** MarcoFalke has left #bitcoin-core-dev
440 2016-12-05T11:47:54  <bitcoin-git> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/43e8150ef690...c01f16aaa0ed
441 2016-12-05T11:47:55  <bitcoin-git> bitcoin/master ea83d00 instagibbs: SendMoney: use already-calculated balance
442 2016-12-05T11:47:55  <bitcoin-git> bitcoin/master c01f16a Wladimir J. van der Laan: Merge #9165: SendMoney: use already-calculated balance...
443 2016-12-05T11:48:04  <bitcoin-git> [bitcoin] laanwj closed pull request #9165: SendMoney: use already-calculated balance (master...triv-curbal) https://github.com/bitcoin/bitcoin/pull/9165
444 2016-12-05T11:53:09  *** dcousens has quit IRC
445 2016-12-05T11:55:06  *** dcousens has joined #bitcoin-core-dev
446 2016-12-05T11:55:25  <bitcoin-git> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/c01f16aaa0ed...7d5d44969b4a
447 2016-12-05T11:55:25  <bitcoin-git> bitcoin/master c4b6fa8 Pavel Janík: CMutableTransaction is defined as struct.
448 2016-12-05T11:55:26  <bitcoin-git> bitcoin/master 7d5d449 Wladimir J. van der Laan: Merge #9282: CMutableTransaction is defined as struct...
449 2016-12-05T11:55:41  <bitcoin-git> [bitcoin] laanwj closed pull request #9282: CMutableTransaction is defined as struct (master...20161205_CMutableTransaction_is_struct) https://github.com/bitcoin/bitcoin/pull/9282
450 2016-12-05T11:59:27  *** _biO_ has joined #bitcoin-core-dev
451 2016-12-05T12:28:42  *** Victor_sueca has quit IRC
452 2016-12-05T12:29:15  *** Alina-malina_ is now known as Alina-malina
453 2016-12-05T12:29:51  *** Victor_sueca has joined #bitcoin-core-dev
454 2016-12-05T12:30:55  *** CubicEarth has quit IRC
455 2016-12-05T12:32:45  *** alpalp has joined #bitcoin-core-dev
456 2016-12-05T13:02:44  <bitcoin-git> [bitcoin] jonasschnelli opened pull request #9284: Suppress some annoying deprecation warnings (OSX) (master...2016/12/osx_warnings) https://github.com/bitcoin/bitcoin/pull/9284
457 2016-12-05T13:08:09  *** moli has joined #bitcoin-core-dev
458 2016-12-05T13:27:19  *** PaulCapestany has quit IRC
459 2016-12-05T13:28:07  *** BashCo_ has joined #bitcoin-core-dev
460 2016-12-05T13:30:03  *** Chris_Stewart_5 has quit IRC
461 2016-12-05T13:31:17  *** BashCo has quit IRC
462 2016-12-05T13:31:21  *** CubicEarth has joined #bitcoin-core-dev
463 2016-12-05T13:36:11  *** CubicEarth has quit IRC
464 2016-12-05T13:37:08  *** alpalp has quit IRC
465 2016-12-05T13:38:56  <jonasschnelli> And comments on our keypoolrefil RPC call behavior?
466 2016-12-05T13:38:56  <jonasschnelli> https://github.com/bitcoin/bitcoin/blob/master/qa/rpc-tests/keypool.py#L45
467 2016-12-05T13:39:10  <jonasschnelli> The tests proof, that nodes[0].keypoolrefill(3) result in 4 available keys.
468 2016-12-05T13:39:22  <jonasschnelli> But reading the API docs, it should be 3.
469 2016-12-05T13:39:34  <jonasschnelli> IMO the +1 is wrong here
470 2016-12-05T13:39:54  <jonasschnelli> (I'd ask because I'd like to fix this with the HD split in ext/int chain)
471 2016-12-05T13:46:48  *** Chris_Stewart_5 has joined #bitcoin-core-dev
472 2016-12-05T13:56:30  <dcousens> hmm, CTransaction assignment was totally removed aye
473 2016-12-05T14:00:05  *** Cory has quit IRC
474 2016-12-05T14:00:51  <dcousens> sipa: was removing CTransaction& operator=(const CTransaction& tx); necessary, or just a safety precaution?
475 2016-12-05T14:02:43  *** Cory has joined #bitcoin-core-dev
476 2016-12-05T14:08:47  <dcousens> meh, I guess I can just CTransactionRef anyway
477 2016-12-05T14:20:24  *** jtimon has quit IRC
478 2016-12-05T14:21:19  <dcousens> eh, nvm, rebased all my local code, tl;dr was juts changing CTransaction to CTransactionRef, .vout to ->vout ... and thats it.
479 2016-12-05T14:21:25  <dcousens> LGTM :)
480 2016-12-05T14:25:28  <instagibbs> dcousens, I love happy endings
481 2016-12-05T14:32:41  *** Guyver2 has joined #bitcoin-core-dev
482 2016-12-05T14:32:45  *** CubicEarth has joined #bitcoin-core-dev
483 2016-12-05T14:37:42  *** CubicEarth has quit IRC
484 2016-12-05T14:49:45  <dcousens> instagibbs: not so happy yet ha
485 2016-12-05T14:50:49  *** murchandamus has quit IRC
486 2016-12-05T14:51:18  *** murchandamus has joined #bitcoin-core-dev
487 2016-12-05T14:52:40  <dcousens> trying a fresh-recompile,  but master seems to just lock up for me atm
488 2016-12-05T14:54:42  *** Giszmo has joined #bitcoin-core-dev
489 2016-12-05T15:03:28  *** thib has joined #bitcoin-core-dev
490 2016-12-05T15:08:59  *** murchandamus has quit IRC
491 2016-12-05T15:09:18  *** murchandamus has joined #bitcoin-core-dev
492 2016-12-05T15:10:10  *** bsm1175321 has joined #bitcoin-core-dev
493 2016-12-05T15:11:32  *** Chris_Stewart_5 has quit IRC
494 2016-12-05T15:13:03  *** bsm1175321 has quit IRC
495 2016-12-05T15:18:13  *** bsm1175321 has joined #bitcoin-core-dev
496 2016-12-05T15:20:42  *** laurentmt has joined #bitcoin-core-dev
497 2016-12-05T15:21:59  *** bsm1175321 has quit IRC
498 2016-12-05T15:27:14  *** Chris_Stewart_5 has joined #bitcoin-core-dev
499 2016-12-05T15:28:12  *** bsm1175321 has joined #bitcoin-core-dev
500 2016-12-05T15:30:34  *** bsm1175321 has quit IRC
501 2016-12-05T15:30:48  *** Chris_Stewart_5 has quit IRC
502 2016-12-05T15:31:08  *** Chris_Stewart_5 has joined #bitcoin-core-dev
503 2016-12-05T15:32:22  *** bsm1175321 has joined #bitcoin-core-dev
504 2016-12-05T15:33:29  *** CubicEarth has joined #bitcoin-core-dev
505 2016-12-05T15:36:41  *** laurentmt has quit IRC
506 2016-12-05T15:37:57  *** CubicEarth has quit IRC
507 2016-12-05T15:46:48  *** murchandamus has quit IRC
508 2016-12-05T15:48:09  *** murchandamus has joined #bitcoin-core-dev
509 2016-12-05T15:49:02  *** morcos has quit IRC
510 2016-12-05T15:49:43  *** zxzzt has quit IRC
511 2016-12-05T15:51:06  *** morcos has joined #bitcoin-core-dev
512 2016-12-05T15:51:14  *** zxzzt has joined #bitcoin-core-dev
513 2016-12-05T15:56:33  *** squidicuz has quit IRC
514 2016-12-05T15:58:06  *** paveljanik has joined #bitcoin-core-dev
515 2016-12-05T15:58:06  *** paveljanik has joined #bitcoin-core-dev
516 2016-12-05T16:30:00  *** TomMc has joined #bitcoin-core-dev
517 2016-12-05T16:34:12  *** CubicEarth has joined #bitcoin-core-dev
518 2016-12-05T16:38:37  *** CubicEarth has quit IRC
519 2016-12-05T16:39:01  *** laurentmt has joined #bitcoin-core-dev
520 2016-12-05T16:47:40  *** paveljanik has quit IRC
521 2016-12-05T16:48:12  *** abpa has joined #bitcoin-core-dev
522 2016-12-05T16:48:31  *** paveljanik has joined #bitcoin-core-dev
523 2016-12-05T16:48:32  *** paveljanik has joined #bitcoin-core-dev
524 2016-12-05T16:55:31  *** laurentmt has quit IRC
525 2016-12-05T17:00:40  *** laurentmt has joined #bitcoin-core-dev
526 2016-12-05T17:04:46  *** wvr has quit IRC
527 2016-12-05T17:10:19  *** BCBot_ has quit IRC
528 2016-12-05T17:10:41  *** BCBot has joined #bitcoin-core-dev
529 2016-12-05T17:29:56  <jl2012> in what situation, the "Warning: We do not appear to fully agree with our peers! You may need to upgrade, or other nodes may need to upgrade" will be shown?
530 2016-12-05T17:34:58  *** CubicEarth has joined #bitcoin-core-dev
531 2016-12-05T17:39:49  *** CubicEarth has quit IRC
532 2016-12-05T17:40:09  <morcos> jonasschnelli: you around?
533 2016-12-05T17:41:33  <morcos> re: #8501, I agree with not fixing the frequency..  But i'm unsure about the no duplicating the same value over and over again..
534 2016-12-05T17:41:35  <gribble> https://github.com/bitcoin/bitcoin/issues/8501 | Add mempool statistics collector by jonasschnelli · Pull Request #8501 · bitcoin/bitcoin · GitHub
535 2016-12-05T17:42:14  <morcos> It might depend on the use case, but for instance it might be valuable to know that it stayed the same for a while and then incremented all at once, as opposed to not being able to tell that it just hadn't been simpled in between
536 2016-12-05T17:43:04  <morcos> my thought was if we saved having to record the time stamp, we might be able to put up with lots of duplicate values..  especially if we're saving for instance only 1000 data points at second frequency, it just won't use all that much memory
537 2016-12-05T17:43:23  *** kadoban has joined #bitcoin-core-dev
538 2016-12-05T17:43:27  <morcos> anywya, sorry, dont mean to redesign your whole PR months after you opened it
539 2016-12-05T17:44:08  *** Victor_sueca has quit IRC
540 2016-12-05T17:49:56  *** davec has quit IRC
541 2016-12-05T18:15:34  *** BashCo_ has quit IRC
542 2016-12-05T18:16:11  *** BashCo has joined #bitcoin-core-dev
543 2016-12-05T18:19:36  *** laurentmt has quit IRC
544 2016-12-05T18:20:28  *** BashCo has quit IRC
545 2016-12-05T18:35:42  *** CubicEarth has joined #bitcoin-core-dev
546 2016-12-05T18:37:06  *** BashCo has joined #bitcoin-core-dev
547 2016-12-05T18:40:23  *** CubicEarth has quit IRC
548 2016-12-05T18:46:54  <Chris_Stewart_5> Does -txindex significantly impact performance of IBD? I tried to sync last night and only synced ~10K blocks, which seems slow. Is that reasonable?
549 2016-12-05T18:48:56  <sipa> that's totally unreasonable
550 2016-12-05T18:49:04  <sipa> is it stuck?
551 2016-12-05T18:49:08  <sipa> or just slow?
552 2016-12-05T18:52:28  <Chris_Stewart_5> extremely slow it seems. I'm using out of box settings on 0.13.1 with -txindex.
553 2016-12-05T18:52:43  *** CubicEarth has joined #bitcoin-core-dev
554 2016-12-05T18:52:53  <sipa> does increasing dbcache help?
555 2016-12-05T18:53:13  *** CubicEarth has quit IRC
556 2016-12-05T18:53:43  <Chris_Stewart_5> I'll try it later and report back, default is 2GB?
557 2016-12-05T18:54:32  <sipa> default is 300MB
558 2016-12-05T18:55:20  <Chris_Stewart_5> mmm that is probably why. I thought it was significantly higher. How long does IBD take other people with that setting as default?
559 2016-12-05T18:59:03  <sipa> what height are you at now?
560 2016-12-05T19:00:03  *** pavel_ has joined #bitcoin-core-dev
561 2016-12-05T19:01:06  *** pavel_ has quit IRC
562 2016-12-05T19:01:36  *** paveljanik has quit IRC
563 2016-12-05T19:03:13  *** aalex__ has quit IRC
564 2016-12-05T19:03:33  *** aalex__ has joined #bitcoin-core-dev
565 2016-12-05T19:04:28  *** aalex has joined #bitcoin-core-dev
566 2016-12-05T19:05:45  *** davec has joined #bitcoin-core-dev
567 2016-12-05T19:07:53  <Chris_Stewart_5> 403817
568 2016-12-05T19:08:03  *** aalex__ has quit IRC
569 2016-12-05T19:08:36  *** harrymm has quit IRC
570 2016-12-05T19:11:14  *** moli has quit IRC
571 2016-12-05T19:12:07  *** MarcoFalke has joined #bitcoin-core-dev
572 2016-12-05T19:12:37  *** dcousens has quit IRC
573 2016-12-05T19:22:14  *** lightningbot has joined #bitcoin-core-dev
574 2016-12-05T19:22:32  *** timothy has quit IRC
575 2016-12-05T19:22:35  *** drizztbsd has joined #bitcoin-core-dev
576 2016-12-05T19:23:03  *** drizztbsd is now known as timothy
577 2016-12-05T19:23:28  *** sdaftuar_ has joined #bitcoin-core-dev
578 2016-12-05T19:23:46  *** kanzure_ has joined #bitcoin-core-dev
579 2016-12-05T19:24:35  *** laurentmt has joined #bitcoin-core-dev
580 2016-12-05T19:24:54  *** harrymm has joined #bitcoin-core-dev
581 2016-12-05T19:25:14  *** bobbytux_ has quit IRC
582 2016-12-05T19:26:16  *** jyap_ has joined #bitcoin-core-dev
583 2016-12-05T19:27:15  *** adam3us_ has joined #bitcoin-core-dev
584 2016-12-05T19:27:20  *** BlueMatt_ has joined #bitcoin-core-dev
585 2016-12-05T19:28:23  *** BlueMatt has quit IRC
586 2016-12-05T19:28:26  *** BlueMatt_ is now known as BlueMatt
587 2016-12-05T19:28:26  *** BlueMatt has joined #bitcoin-core-dev
588 2016-12-05T19:28:32  *** ville- has quit IRC
589 2016-12-05T19:28:32  *** Atomicat has quit IRC
590 2016-12-05T19:28:33  *** sdaftuar has quit IRC
591 2016-12-05T19:28:33  *** adam3us has quit IRC
592 2016-12-05T19:28:33  *** kanzure has quit IRC
593 2016-12-05T19:28:33  *** jyap has quit IRC
594 2016-12-05T19:28:33  *** Bootvis has quit IRC
595 2016-12-05T19:28:33  *** jyap_ is now known as jyap
596 2016-12-05T19:28:34  *** jyap has joined #bitcoin-core-dev
597 2016-12-05T19:28:39  *** Atomicat_ is now known as Atomicat
598 2016-12-05T19:28:41  *** ville-- has joined #bitcoin-core-dev
599 2016-12-05T19:29:53  *** Bootvis has joined #bitcoin-core-dev
600 2016-12-05T19:29:58  *** harrymm has quit IRC
601 2016-12-05T19:37:42  *** aalex_ has joined #bitcoin-core-dev
602 2016-12-05T19:41:16  *** aalex has quit IRC
603 2016-12-05T19:49:26  *** kanzure_ is now known as kanzure
604 2016-12-05T20:00:43  *** Madars has quit IRC
605 2016-12-05T20:03:18  *** Madars has joined #bitcoin-core-dev
606 2016-12-05T20:08:42  *** arubi_ is now known as arubi
607 2016-12-05T20:33:43  <Chris_Stewart_5> sipa: I should have been more clear, I have been trying to do IBD over the course of a few nights, with results like I said ~10k blocks a night.
608 2016-12-05T20:34:46  <Chris_Stewart_5> the first ~250k blocks went relatively fast (a couple hour period) but I think some might have already been on disk? Perhaps i'm using the term IBD a little too loosely
609 2016-12-05T20:34:58  <Chris_Stewart_5> but it is a major sync
610 2016-12-05T20:40:11  <instagibbs> sdaftuar_, why would you want to spend a coin from you wallet that has descendants(already spent)? I'm surely thinking of this wrong
611 2016-12-05T20:41:48  <instagibbs> or are descendants calculated from a tx point of view, ie other output has been spent in a chain, therefore that adds to that count
612 2016-12-05T20:42:23  <sdaftuar_> instagibbs: oh, yeah i meant in-mempool descendants
613 2016-12-05T20:42:35  <sdaftuar_> say you have a tx that has 2 outputs, you send me money and give yourself change.
614 2016-12-05T20:42:39  <sdaftuar_> then i chain 24 transactions off it
615 2016-12-05T20:42:48  <sdaftuar_> you try to spend your change, but that'll fail
616 2016-12-05T20:42:54  <instagibbs> ok, didnt think of the fact that outputs are linked re:descendants
617 2016-12-05T20:43:21  <sdaftuar_> that does seem to be a confusing property of the mempool limiting :)
618 2016-12-05T20:43:31  *** sdaftuar_ is now known as sdaftuar
619 2016-12-05T20:43:31  <instagibbs> but that is obv in hindsight. Ok, well one issue is you might have asymmetrical limits.
620 2016-12-05T20:43:39  <sdaftuar> yeah, i suggested using min()
621 2016-12-05T20:43:50  <gmaxwell> it might have been better if that limit was split across outputs.
622 2016-12-05T20:44:06  <sdaftuar> gmaxwell: that would then fail to capture the issue at hand, i think
623 2016-12-05T20:44:09  <instagibbs> sdaftuar, hmm says max on my screen
624 2016-12-05T20:44:21  <gmaxwell> e.g. A can have up to 24 decendants, it has two outputs, each can have 12 under it.
625 2016-12-05T20:44:35  <sdaftuar> sorry, max(tx->ancestor, tx->descendants()) should be less than min(ancestorlimit, descendantlimit)
626 2016-12-05T20:44:39  <instagibbs> oh i see nvm
627 2016-12-05T20:45:02  <sdaftuar> gmaxwell: oh, hm.
628 2016-12-05T20:47:04  <gmaxwell> in the worst case though it reduces your maximum to a log_outputs(depth), which isn't awesome.
629 2016-12-05T20:47:17  <sdaftuar> maybe doable, but kind of yuck to implement i think
630 2016-12-05T20:48:00  <gmaxwell> but it would prevent other people from chewing up your limit. I think this hasn't actually been a problem, though I could imagine it being one in certian kinds of transaction protocols.
631 2016-12-05T20:48:42  <sdaftuar> what kinds of protocols do you have in mind?
632 2016-12-05T20:49:39  <gmaxwell> In the abstract, protocols where someone delaying your transaction can allow the party to cheat like atomic swaps. Not that big of a concern since unless the head transaction is confirmed those protocols are not secure against miners.
633 2016-12-05T20:52:28  <sdaftuar> right, if someone comes up with a use case that does rely on the parent not necessarily being confirmed, then that should alter our thinking
634 2016-12-05T20:53:38  *** wasi has joined #bitcoin-core-dev
635 2016-12-05T20:59:40  *** windsok has quit IRC
636 2016-12-05T21:19:41  *** aalex__ has joined #bitcoin-core-dev
637 2016-12-05T21:23:24  *** aalex_ has quit IRC
638 2016-12-05T21:53:30  *** windsok has joined #bitcoin-core-dev
639 2016-12-05T21:58:07  *** jtimon has joined #bitcoin-core-dev
640 2016-12-05T22:13:41  *** jl2012 has quit IRC
641 2016-12-05T22:15:54  *** laurentmt has quit IRC
642 2016-12-05T22:23:33  *** Chris_Stewart_5 has quit IRC
643 2016-12-05T22:28:30  *** aalex has joined #bitcoin-core-dev
644 2016-12-05T22:31:32  *** aalex__ has quit IRC
645 2016-12-05T22:44:11  *** Guyver2 has quit IRC
646 2016-12-05T22:58:48  *** grubles has quit IRC
647 2016-12-05T22:59:30  *** JackH has quit IRC
648 2016-12-05T23:01:03  *** tunafizz has joined #bitcoin-core-dev
649 2016-12-05T23:12:42  *** laurentmt has joined #bitcoin-core-dev
650 2016-12-05T23:18:24  *** laurentmt has quit IRC
651 2016-12-05T23:31:07  *** cryptapus is now known as cryptapus_afk
652 2016-12-05T23:33:08  *** dcousens has joined #bitcoin-core-dev
653 2016-12-05T23:39:13  *** MarcoFalke has quit IRC
654 2016-12-05T23:50:22  <dcousens> hmph
655 2016-12-05T23:50:29  <dcousens> So I'm running master, no changes at all
656 2016-12-05T23:50:54  <dcousens> And my bitcoind finishes up to verify, then just sits there on 100% CPU usage (probably forever, but who knows)
657 2016-12-05T23:51:13  <gmaxwell> 'up to verify'?
658 2016-12-05T23:51:15  <dcousens> It fails to open up the RPC, or start synchronizing
659 2016-12-05T23:51:20  <dcousens> checkblocks
660 2016-12-05T23:51:27  <gmaxwell> what is the last log entry?
661 2016-12-05T23:51:34  <gmaxwell> can you attach GDB?
662 2016-12-05T23:51:47  <dcousens> It still keeps logging, but solely the tor control messages
663 2016-12-05T23:52:25  <gmaxwell> that sounds like a deadlock then.
664 2016-12-05T23:52:29  <dcousens> What do I need to do to attach GDB? Happy to do it
665 2016-12-05T23:52:38  <gmaxwell> dcousens: what OS are you on?
666 2016-12-05T23:53:47  <dcousens> just collecting info, sec
667 2016-12-05T23:54:27  <gmaxwell> on *nix:   ps aux | grep bitcoin   to get the bitcoind pid    then gdb -p <pid> to attach.     then run thread apply all bt full   to get backtraces from every thread, and 0bin that to me,  and then you can type q<enter> to quit
668 2016-12-05T23:58:04  *** moli has joined #bitcoin-core-dev
669 2016-12-05T23:58:37  <dcousens> http://pastie.org/private/a40ppfohw2kxxxundn1iw - for the info, doing gdb now