1 2015-11-12T00:00:07  *** evoskuil has joined #bitcoin-core-dev
  2 2015-11-12T00:01:57  *** [1]evoskuil has joined #bitcoin-core-dev
  3 2015-11-12T00:04:26  *** evoskuil has quit IRC
  4 2015-11-12T00:04:26  *** [1]evoskuil is now known as evoskuil
  5 2015-11-12T00:12:18  *** skyraider has quit IRC
  6 2015-11-12T01:26:38  *** gribble has quit IRC
  7 2015-11-12T01:32:23  *** zooko has joined #bitcoin-core-dev
  8 2015-11-12T01:40:18  *** gribble has joined #bitcoin-core-dev
  9 2015-11-12T01:42:51  *** zooko has quit IRC
 10 2015-11-12T01:44:25  *** Ylbam has quit IRC
 11 2015-11-12T01:54:50  *** randy-waterhouse has quit IRC
 12 2015-11-12T02:00:06  *** evoskuil has quit IRC
 13 2015-11-12T02:00:39  *** evoskuil has joined #bitcoin-core-dev
 14 2015-11-12T02:00:46  *** trippysalmon has joined #bitcoin-core-dev
 15 2015-11-12T02:25:41  *** trippysalmon has quit IRC
 16 2015-11-12T02:26:47  <GitHub16> [bitcoin] pstratem opened pull request #6993: Add -blocksonly option (master...blocksonly) https://github.com/bitcoin/bitcoin/pull/6993
 17 2015-11-12T02:30:46  *** evoskuil has quit IRC
 18 2015-11-12T02:30:47  <gmaxwell> phantomcircuit: does that actually work?!
 19 2015-11-12T02:30:59  <gmaxwell> add docs.
 20 2015-11-12T02:31:51  *** evoskuil has joined #bitcoin-core-dev
 21 2015-11-12T02:49:49  *** jgarzik has joined #bitcoin-core-dev
 22 2015-11-12T02:55:36  <phantomcircuit> gmaxwell, yeah it does
 23 2015-11-12T02:57:23  <phantomcircuit> i had completely forgotten about the fRelay flag
 24 2015-11-12T03:08:49  <dcousens> phantomcircuit: its cool tbh haha
 25 2015-11-12T03:21:11  *** jgarzik has quit IRC
 26 2015-11-12T03:23:15  *** jgarzik has joined #bitcoin-core-dev
 27 2015-11-12T03:33:37  *** dcousens has quit IRC
 28 2015-11-12T03:50:08  *** fanquake has quit IRC
 29 2015-11-12T03:50:24  *** randy-waterhouse has joined #bitcoin-core-dev
 30 2015-11-12T04:19:02  *** CodeShark has joined #bitcoin-core-dev
 31 2015-11-12T04:19:25  *** Arnavion has quit IRC
 32 2015-11-12T04:19:41  *** [1]evoskuil has joined #bitcoin-core-dev
 33 2015-11-12T04:19:43  *** Arnavion has joined #bitcoin-core-dev
 34 2015-11-12T04:21:46  *** evoskuil has quit IRC
 35 2015-11-12T04:21:46  *** [1]evoskuil is now known as evoskuil
 36 2015-11-12T04:27:17  *** dcousens has joined #bitcoin-core-dev
 37 2015-11-12T04:31:30  *** gribble has quit IRC
 38 2015-11-12T04:44:01  *** gribble has joined #bitcoin-core-dev
 39 2015-11-12T04:56:04  *** jtimon has quit IRC
 40 2015-11-12T05:14:03  *** Thireus has quit IRC
 41 2015-11-12T05:18:09  *** NLNico has joined #bitcoin-core-dev
 42 2015-11-12T05:38:18  *** dcousens has left #bitcoin-core-dev
 43 2015-11-12T05:48:13  *** dcousens has joined #bitcoin-core-dev
 44 2015-11-12T05:49:07  *** d_t has joined #bitcoin-core-dev
 45 2015-11-12T05:49:21  <dcousens> conceptually, phantomcircuit I feel like you could separate mempool from bitcoind with -blocksonly :P
 46 2015-11-12T05:51:09  <dcousens> phantomcircuit: also, would it be worth just dropping the transactions before they even get to AcceptToMempool?
 47 2015-11-12T05:51:32  <dcousens> or would advertising you don't care about txs be a BIP requiring protocol change?
 48 2015-11-12T05:55:28  <dcousens> gmaxwell: getdata being dropping the transactions before ATM? :P
 49 2015-11-12T05:56:16  <gmaxwell> jinx I guess. :)
 50 2015-11-12T05:56:42  <dcousens> haha
 51 2015-11-12T06:00:00  <gmaxwell> but yea, I think he should change it to not getdata any announcements instead from non-whitebound peers, and leave the mempool alone.
 52 2015-11-12T06:00:28  <dcousens> gmaxwell: agreed, most likely the mempool would always be empty
 53 2015-11-12T06:00:31  <dcousens> but that is intended
 54 2015-11-12T06:00:43  <dcousens> (obviously)
 55 2015-11-12T06:01:20  <gmaxwell> Right but then you could still use this node to e.g. relay transactions created by an offline wallet behind it and such.
 56 2015-11-12T06:01:47  <dcousens> gmaxwell: or as I was thinking, have multiple relay nodes with decent bandwidth, but low memory, backed by a [lower] bandwidth blockchain node
 57 2015-11-12T06:02:10  <dcousens> assuming you had a service big enough where that is warranted anyway
 58 2015-11-12T06:02:45  <dcousens> hmmm, though, its the mempool that takes all the memory anyway haha
 59 2015-11-12T06:03:15  <dcousens> well, it'll be interesting to play with for sure
 60 2015-11-12T06:03:43  <phantomcircuit> dcousens, im dropping them in AcceptToMempool to prevent leaking which transactions are yours
 61 2015-11-12T06:03:52  <phantomcircuit> im not sure i did a thorough job of that though
 62 2015-11-12T06:04:40  <gmaxwell> phantomcircuit: nah, don't do that. Turning off wallet broadcast (already an option) is how thats achieved.
 63 2015-11-12T06:05:01  <dcousens> gmaxwell: what about if it was submitted via RPC?
 64 2015-11-12T06:05:30  <gmaxwell> Then we want to relay it.
 65 2015-11-12T06:05:58  <dcousens> gmaxwell: sure, but, that seems counter to the point of preventing leakage
 66 2015-11-12T06:06:02  <gmaxwell> blocksonly shouldn't refuse to relay transactions when you've specifically asked it to so. :)
 67 2015-11-12T06:06:18  <gmaxwell> dcousens: if you don't want it to broadcast a transaction then don't ask it to!
 68 2015-11-12T06:06:34  <dcousens> gmaxwell: sure, but, you could say the same about using it as a wallet
 69 2015-11-12T06:06:39  <phantomcircuit> gmaxwell, well then it's a one line patch
 70 2015-11-12T06:07:30  <gmaxwell> phantomcircuit: I think not, you need to avoid getdataing if they ignore your flags.
 71 2015-11-12T06:07:49  <dcousens> gmaxwell: I agree with you, just pointing out the logic didn't work with what phantomcircuit logic said "preventing leaking [of] transactions which are yours"
 72 2015-11-12T06:08:02  *** CodeShark_ has joined #bitcoin-core-dev
 73 2015-11-12T06:08:15  <gmaxwell> dcousens: his current behavior will not relay a sendrawtransaction.
 74 2015-11-12T06:08:27  *** CodeShark_ has quit IRC
 75 2015-11-12T06:08:32  <gmaxwell> I thought you were asking what would happen in my proposed change.
 76 2015-11-12T06:08:37  <dcousens> dw, I was
 77 2015-11-12T06:08:49  *** CodeShark has quit IRC
 78 2015-11-12T06:09:01  *** CodeShark_ has joined #bitcoin-core-dev
 79 2015-11-12T06:09:05  <dcousens> unimportant, semantics are clear either way ;)
 80 2015-11-12T06:09:11  *** CodeShark_ has quit IRC
 81 2015-11-12T06:09:27  *** CodeShark has joined #bitcoin-core-dev
 82 2015-11-12T06:10:20  <gmaxwell> phantomcircuit: so I think behavior should be when set, you will set the flag (to suppress inv announcements), and ignore getdatas.  But only from non-white peers.  And then anything sendraw/wallet will still work like normal, and if you don't want that for privacy reasons..then disable wallet broadcast and don't use sendraw. :)
 83 2015-11-12T06:10:57  <gmaxwell> I imagine post 0.12 we'll also make this behavior one of the things we do in response to running into bandwidth limits.
 84 2015-11-12T06:11:09  <gmaxwell> (e.g. if you get too close to your limit, go into blocks only mode.)
 85 2015-11-12T06:11:40  *** Thireus has joined #bitcoin-core-dev
 86 2015-11-12T06:12:12  <phantomcircuit> gmaxwell, that reminds me, thoughts on having a global settings object? (something that ya know.. actually acquired a lock)
 87 2015-11-12T06:13:29  <gmaxwell> Big lock protecting bag-of-assorted things isn't generally great.
 88 2015-11-12T06:14:24  <gmaxwell> in any case, not needed here, auto-blocks only I think is out of scope for 0.12.  It's a pretty big behavior change, and we'd have to do something about the transaction privacy loss. (hopefully fix that via a seperate private announcement mechenism)
 89 2015-11-12T06:15:22  <dcousens> gmaxwell: by auto, do you mean default?
 90 2015-11-12T06:15:32  <dcousens> oh, you mean auto bandwidth response
 91 2015-11-12T06:15:33  <dcousens> nvm
 92 2015-11-12T06:17:35  <phantomcircuit> gmaxwell, could equally be a separate lock on each of the settings
 93 2015-11-12T06:18:18  <gmaxwell> phantomcircuit: right for settings that should chang at runtime the setting should be read in init and set in a datastructure with its own lock for further changes.
 94 2015-11-12T06:20:12  <phantomcircuit> gmaxwell, well and most of them can just be atomic ints
 95 2015-11-12T06:21:41  <gmaxwell> We are not yet using C++ new enough to have standard atomic types.
 96 2015-11-12T06:21:50  <gmaxwell> And plain integers ARE NOT ATOMIC.
 97 2015-11-12T06:23:30  <jgarzik> without help
 98 2015-11-12T06:24:06  <gmaxwell> sure, you can use locks. Or custom asm or what not.
 99 2015-11-12T06:25:04  <jgarzik> gcc & clang provide atomic intrinsics to give C/C++ level access to cmpxchg, ints with lock prefixes and more.
100 2015-11-12T06:25:16  <jgarzik> not C++ standard but it is available on all our platforms
101 2015-11-12T06:25:51  <phantomcircuit> gmaxwell, something something boost atomic
102 2015-11-12T06:26:33  <phantomcircuit> (im opposed to using boost in places where replacing it would be a giant headache, this is not one of them)
103 2015-11-12T06:26:44  <gmaxwell> Well we'll upgrade to C++11 probably sooner rather than later and pick up language standard atomics.
104 2015-11-12T06:27:40  <jgarzik> phantomcircuit, nod, http://www.boost.org/doc/libs/1_59_0/doc/html/atomic.html
105 2015-11-12T06:27:50  <gmaxwell> /usually/ one shouldn't be directly using atomics in any case except in performance crticial algorithims or when building lower level constructs to be used from other things.
106 2015-11-12T06:28:06  <gmaxwell> If the boost stuff is drop in for c++11 too than thats not so bad.
107 2015-11-12T06:28:21  <jgarzik> that's the idea
108 2015-11-12T06:29:10  <gmaxwell> I think we're already in a mode of only using boost features that aren't in C++11 if we have reason good enough to justify reimplementing them internally in any case. (e.g. mempool indexing)
109 2015-11-12T06:32:34  <jgarzik> https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html
110 2015-11-12T06:52:33  <phantomcircuit> gmaxwell, hmm so im thinking blocksonly should be undocumented as there are actually very few people who should be running such a thing
111 2015-11-12T06:53:03  <phantomcircuit> the privacy loss form running with blocksonly and walletbroadcast=1 is potentially huge
112 2015-11-12T06:54:01  *** Ylbam has joined #bitcoin-core-dev
113 2015-11-12T06:55:21  <dcousens> phantomcircuit: that seems wrong
114 2015-11-12T06:55:51  <dcousens> maybe a compile flag to allow it
115 2015-11-12T06:55:52  <dcousens> or something
116 2015-11-12T06:56:07  <wumpus> just disable walletbroadcast by default if it's enabled?
117 2015-11-12T06:56:09  <dcousens> but not undocumented
118 2015-11-12T06:56:16  <dcousens> or that ^
119 2015-11-12T06:56:20  <wumpus> and emit a warning to the log
120 2015-11-12T06:57:00  <wumpus> PRIVACYWARNING: wallet broadcasting has been disabled, because any transaction you send while not relaying transactions will stand out like a sore thumb
121 2015-11-12T06:57:28  <phantomcircuit> wumpus, maybe... that's just what we need a more complicated tree of which settings are compatible :P
122 2015-11-12T06:57:39  <wumpus> if you are sure you want to enable this, use -walletbroadcast
123 2015-11-12T06:57:53  <gmaxwell> phantomcircuit: yea, you can imply walletbroadcast off for that case. We have handling like this for many options related to privacy.
124 2015-11-12T06:57:55  <wumpus> you call that complicated?
125 2015-11-12T06:57:55  <sipa> phantomcircuit: wallet broadcasting and whitelisting are already inherently a privacy leak
126 2015-11-12T06:58:17  <gmaxwell> though I think you are GREATLY overestimating the improvement relay has on privacy.
127 2015-11-12T06:58:20  <dcousens> wumpus: he just doesn't want to make it more than 1 or 2 lines
128 2015-11-12T06:58:45  <gmaxwell> I think either way (defaulting walletbroadcast off or on) is okay with me, though I would leave it defaulting on.
129 2015-11-12T06:59:06  <phantomcircuit> i uh dcousens *shhhh*
130 2015-11-12T06:59:10  <wumpus> 'it is a already bad' is no excuse for allowing things to get silently even worse, hence the warning.
131 2015-11-12T07:00:06  <wumpus> that's the problem with privacy everywhere, people think well we have no privacy already anyway, so why not do this and this also, it's not like it could get any worse. Yes, it can get worse.
132 2015-11-12T07:00:45  <sipa> wumpus: agree, but to someone already listening on many nodes in the network, if they're connected to the original broadcaster too, they have a fairly good chance of finding you
133 2015-11-12T07:00:48  <gmaxwell> wumpus: I don't know if it makes it worse or not, it might-- so fair enough.
134 2015-11-12T07:01:17  <sipa> not relaying transactions otherwise just reduces the noticed; it doesn't fix the fact that information is being leaked
135 2015-11-12T07:01:18  <wumpus> gmaxwell: well if you never send transactions then suddenlly send one, that makes it extremely clear it's yours, doesn't it?
136 2015-11-12T07:01:23  <gmaxwell> sipa: research so far suggests that its currently about 100% effective if you transact more than once.
137 2015-11-12T07:01:49  <gmaxwell> wumpus: well with whitebind, perhaps not! (also if we start turning this on dynamically in the future e.g. when low on bandwidth) perhaps not.  But fair enough.
138 2015-11-12T07:01:54  <sipa> s/noticed/noise/ i do not comprehend how i made that typo
139 2015-11-12T07:02:04  <gmaxwell> phantomcircuit: so go softset off wallet broadcast and log, see the other examples in init.cpp.
140 2015-11-12T07:02:23  <wumpus> I just don't like a setting that can silently make your privacy worse
141 2015-11-12T07:02:37  <gmaxwell> wumpus: yup easily avoided, I agree.
142 2015-11-12T07:02:40  <sipa> i'm fine with default off
143 2015-11-12T07:02:41  <wumpus> the other option would be to indeed not document the option
144 2015-11-12T07:03:15  <wumpus> but I think a warning + disable wallet broadcast by default is good enough
145 2015-11-12T07:03:20  <sipa> what about whitebind?
146 2015-11-12T07:03:26  <wumpus> what's with whitebind?
147 2015-11-12T07:03:46  <wumpus> you mean - do relay transactions for whitelisters?
148 2015-11-12T07:03:47  <sipa> whitelisted peers that relay transactions
149 2015-11-12T07:04:01  <gmaxwell> I think it'll be a nice bandwidth reduction feature for watching wallets. It's a >2x reduction in bandwidth (probably more like 3x typically) and also saves you from a number of dos attacks. :)
150 2015-11-12T07:04:36  <gmaxwell> I think it should relay for whitelisted peers. So you can use a node configured like this as a gateway to get your transactions to the outside world.
151 2015-11-12T07:04:40  <wumpus> gmaxwell: it is a great feature, but the catch is that you have to make sure you're silent yourself, as there is no noise to hide in. That's acceptable, but needs to be transparent.
152 2015-11-12T07:05:09  <sipa> how is whitelisting different from wallet broadcasting yourself?
153 2015-11-12T07:06:13  <gmaxwell> because you choose and configure whitelists, it's like setting broadcast=1
154 2015-11-12T07:06:16  <sipa> even if it doesn't disclose the ip of the actual sender anymore, they do know an ip address that the sender trusts
155 2015-11-12T07:06:32  <wumpus> right it's something you have to explicitly enable
156 2015-11-12T07:06:46  <gmaxwell> That could be a seperate setting. I've wanted that before, to control the forced relay for whitehosts.
157 2015-11-12T07:06:47  <wumpus> ... which could come with a warning, of course
158 2015-11-12T07:07:04  <sipa> wumpus: then how is it different from walletbroadcast?
159 2015-11-12T07:07:23  <wumpus> sipa: my proposal is that walletbroadcast would be disabled by default if block-only
160 2015-11-12T07:07:30  <wumpus> whitelist already *is* disabled by default
161 2015-11-12T07:07:39  <sipa> hmm, ok
162 2015-11-12T07:07:44  <wumpus> walletbroadcast is enabled by default
163 2015-11-12T07:07:53  <sipa> ok, that is a fair point
164 2015-11-12T07:08:15  <gmaxwell> Whitelist is a bot too overloaded. We might want to add a whiterelay that controls the forced relay for whitelisted hosts, which defaults on, but is soft-off with blocksonly
165 2015-11-12T07:08:20  <wumpus> there are probably more ways that whitelisted peers can screw up your privacy :D
166 2015-11-12T07:08:23  <gmaxwell> s/bot/bit/
167 2015-11-12T07:08:27  <wumpus> they're sort of trusted
168 2015-11-12T07:08:33  <gmaxwell> yea, we already bypass mempool policy
169 2015-11-12T07:08:53  <sipa> wumpus: they are very trusted... but you yourself are also trusted
170 2015-11-12T07:09:41  <sipa> if you do a wallet transaction, you expect it to be relayed; if a whitelisted peer relays a transaction, you also expect it to be relayed
171 2015-11-12T07:10:02  <wumpus> the wallet may also rebroadcast old transactions
172 2015-11-12T07:10:05  <wumpus> that's the scary part
173 2015-11-12T07:10:18  <sipa> whitelisted peers will also rebroadcast old transactions
174 2015-11-12T07:10:20  <wumpus> even transactions sent to you
175 2015-11-12T07:10:29  <wumpus> ok  -I feel this is not constructive anymore
176 2015-11-12T07:10:48  <wumpus> don't introduce features that may silently reduce user's privacy even more, that's all I ask
177 2015-11-12T07:11:04  <sipa> ok
178 2015-11-12T07:11:26  <wumpus> if there is warning, or a knob that has to be explicitly enabled, I'm ok with it
179 2015-11-12T07:12:26  <gmaxwell> yea, seems simple enough, softset off walletbroadcast when this is set.
180 2015-11-12T07:12:47  <wumpus> right
181 2015-11-12T07:12:58  <wumpus> and yes, whitelisted peers have to be careful, we should document that
182 2015-11-12T07:13:19  <gmaxwell> yea, we should document that already because they are already deanonymizing.
183 2015-11-12T07:14:07  <gmaxwell> (not even the primary reason I really don't like the whitepeer relay rules--)
184 2015-11-12T07:14:07  <wumpus> and yeah we could have an option to not relay transactions from whitelisted peers...
185 2015-11-12T07:14:15  *** [b__b] has quit IRC
186 2015-11-12T07:14:35  *** [b__b] has joined #bitcoin-core-dev
187 2015-11-12T07:14:36  *** Apocalyptic has quit IRC
188 2015-11-12T07:14:39  <gmaxwell> (I mostly don't like them because I whitebinded the relaynetwork client and p2pool for obvious reasons, and then found they were blowing past my relay policy :-/ )
189 2015-11-12T07:14:47  <sipa> gmaxwell: i think the rebroadcasting is fundamentally the problem, because nobody but the owner of the transaction will rebroadcast
190 2015-11-12T07:15:11  <sipa> an alternative is peers syncing the mempool with each other...
191 2015-11-12T07:15:42  <gmaxwell> sipa: It's fairly busted without that, in fact.
192 2015-11-12T07:15:48  <sipa> so transaction owners don't need to rebroadcast randomly
193 2015-11-12T07:15:53  <wumpus> policy 192.168.1.* { allow relay; bypass relaypolicy; } policy bind :1234 { ... }   *ducks*
194 2015-11-12T07:16:29  <sipa> it is... in theory you can end up with a cordon of nodes that already know your transaction between you and miners
195 2015-11-12T07:16:53  <gmaxwell> Right now on any given publically accessible node >=15 peers are spynodes that connect to everyone. They also use addr behavior to infer things about the non-listening peers behind them; and research shows high success rates for even single transaction origins. If you make a couple transactions in the current network you should assume your origin WILL be idenfied.
196 2015-11-12T07:17:14  <gmaxwell> To avoid this we need an actually private broadcast mechenism.
197 2015-11-12T07:17:24  <wumpus> yes
198 2015-11-12T07:17:37  *** Apocalyptic has joined #bitcoin-core-dev
199 2015-11-12T07:18:14  <gmaxwell> and once we have that, it would work just as well for rebroadcast. :)
200 2015-11-12T07:18:28  <wumpus> we need a private transaction mechanism badly, but we already knew that I think :)
201 2015-11-12T07:18:45  <gmaxwell> and also would remove some of these discussions since then the p2p protocol would not have any privacy implications for full nodes (beyond running a node at all. :) )
202 2015-11-12T07:19:16  <wumpus> right, it would, but it doesn't affect our decisions right now because it's entirely hypothetical :)
203 2015-11-12T07:19:51  <gmaxwell> Just another benefit. It's really nice to not have to consider privacy implications for things; especially when the current state is bad enough that it can be hard to tell if you're making it worse. :)
204 2015-11-12T07:20:04  <gmaxwell> E.g. one of the problems with peer rotation, ... in privacy.
205 2015-11-12T07:20:07  <gmaxwell> er is privacy.
206 2015-11-12T07:20:31  <sipa> we just need separate networks for block relay and tx relay :)
207 2015-11-12T07:20:47  <gmaxwell> One of the problems with pruning ... is privacy (makes nodes more fingerprintable; so you can track nodes across tor and then correlate transactions they originate)
208 2015-11-12T07:21:16  <sipa> well... i guess the same arguments apply to mining actually, it's just much less data to analyse, but ideally you also don't want to be able to pinpoint block creators
209 2015-11-12T07:21:18  <gmaxwell> yadda yadda. All becomes more tidy if these things are seperated.  In one sense you can see that privacy and partition resistance are actually in conflict.
210 2015-11-12T07:21:43  <gmaxwell> For privacy you want to tell your transactions to very few parties, hopefully not attackers. For partition resistance you want to talk to everyone. :)
211 2015-11-12T07:22:27  <gmaxwell> sipa: yea, though different problems; mining is very latency sensntive; tx broadcast is not.
212 2015-11-12T07:22:50  <GitHub183> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/cbf9609c71c6...5fcc14ee0562
213 2015-11-12T07:22:50  <GitHub183> bitcoin/master b5cbd39 James O'Beirne: Add basic coverage reporting for RPC tests...
214 2015-11-12T07:22:50  <sipa> and partitioning for tx relay is less of a problem
215 2015-11-12T07:22:51  <GitHub183> bitcoin/master 5fcc14e Wladimir J. van der Laan: Merge pull request #6804...
216 2015-11-12T07:22:55  <GitHub198> [bitcoin] laanwj closed pull request #6804: [tests] Add basic coverage reporting for RPC tests (master...rpc_coverage) https://github.com/bitcoin/bitcoin/pull/6804
217 2015-11-12T07:23:30  <wumpus> even dropping off your transactions to a random node over Tor is better than publicly broadcasting your transaction from your own node
218 2015-11-12T07:29:39  <gmaxwell> yea, but requires having a tor client. :(
219 2015-11-12T07:30:23  <wumpus> it does, having a tor client seems more and more a requirement for any kind of privacy unfortunately
220 2015-11-12T07:31:25  <gmaxwell> it's just a lot of code and exposure, hate having reason to mix things like that with other things that have critical security requirements.
221 2015-11-12T07:31:31  <wumpus> a tor bitcoin bundle would be pretty nice
222 2015-11-12T07:31:58  <wumpus> well at least it gives plenty of noise to hide in
223 2015-11-12T07:32:57  <wumpus> if you were to say, set up an onion routing network just for transactions, even partaking in it would still leak a lot information about you. Also, if you write it yourself doesn't make it any less 'code and exposure'
224 2015-11-12T07:33:32  <gmaxwell> It's true, for tx broadcast though, one could send a transaction through a high latency mixmaster like system entered via normal internet. And the daemon could just send an empty message an interior hop one ever couple minutes to hide the traffic. Doing the first hop over tor is superior, but I think not strictly needed.
225 2015-11-12T07:34:11  <wumpus> I agree that such a solution could theoretically be better, the problem as we've seen for years is that no one is implementing such a thing, and tor already exists
226 2015-11-12T07:34:30  <gmaxwell> wumpus: the client could be quite simple relative to tor (which at the very begining immediately needs a full SSL stack)
227 2015-11-12T07:34:36  <gmaxwell> Indeed.
228 2015-11-12T07:34:49  <gmaxwell> Actually someone contacted me recently about implementing such a thing, but I don't know if they will.
229 2015-11-12T07:35:18  <gmaxwell> (well they said they will, and asked if I'd be willing/able to review; but you know most things don't actually happen :) )
230 2015-11-12T07:36:20  <gmaxwell> wumpus: e.g. sending client for such a thing could encrypt a message, connect to a TCP socket, and send it over. Never reading a byte off the socket even. Attack surface would be incompariable to tor. :)
231 2015-11-12T07:38:26  <wumpus> the reason that Tor uses SSL, by the way, is that that masquerading as SSL is one of the few feasible ways to hide an encrypted connection in plain sight
232 2015-11-12T07:39:20  <gmaxwell> Indeed, for good reason. I don't think you're going to manage to hide that you're sending a transaction at all unless you use tor... but you well could hide which transaction you were sending. Esp if you don't mind broadcast taking a minute.
233 2015-11-12T07:39:21  <wumpus> if you start with your own such system you'd very soon run against the same issue
234 2015-11-12T07:41:49  <gmaxwell> (in particular it's completely reasonable to have high cover traffic, if you don't mind latency... just send a 100k message into the routing network every N minutes, containing some random assortment of transactions from the public network, and any of your own).  If you're not running this over tor, you'll stand out as a user of the software but not more than that.
235 2015-11-12T07:41:54  <gmaxwell> )
236 2015-11-12T07:42:12  <wumpus> well you could still hide that you're sending a transaction by hiding in noise, e.g. also using the system to automatically send other people's transactions
237 2015-11-12T07:42:48  *** Thireus has quit IRC
238 2015-11-12T07:43:01  <wumpus> just like the relaying system we have now but encrypted, and high-latency, and with padding so that you can't look at the envelope and identify by the size, etc etc
239 2015-11-12T07:43:44  <gmaxwell> exactly. which is also good in that if someone finds out how to distinguish transactions exiting the system they might seek to punish their authors, but if you're sending random other transactions into the system, what exits will not just be from witting users.
240 2015-11-12T07:45:03  <kanzure> we are still missing out on a high-latency mixnet of some kind
241 2015-11-12T07:45:20  <wumpus> but anyhow - just like 5 years ago, I'd be pleasantly surprised if someone build something like that
242 2015-11-12T07:45:41  <wumpus> exactly kanzure
243 2015-11-12T07:45:49  <gmaxwell> prior to bitcoin they were hard(er) because of flooding attacks, but we have a one-two punch here: we have highly valuable tiny messages AND mechenisms to stop flooding attacks.
244 2015-11-12T07:52:34  <wumpus> lol "On January 23, 2006, the Russian FSB accused Britain of using wireless dead drops concealed inside hollowed-out rocks to collect espionage information from agents in Russia. According to the Russian authorities, the agent delivering information would approach the rock and transmit data wirelessly into it from a hand-held device, and later his British handlers would   pick up the stored data by similar means." If
245 2015-11-12T07:52:35  <wumpus> everything fails, we could always use fake rocks :-)
246 2015-11-12T07:53:57  <gmaxwell> "some of us already are"
247 2015-11-12T07:54:27  <randy-waterhouse> yes, btcoin has been accused of using "fake rocks" many times already
248 2015-11-12T07:55:03  <wumpus> randy-waterhouse: any links?
249 2015-11-12T07:55:17  * phantomcircuit looks at scrollback
250 2015-11-12T07:55:36  *** MarcoFalke has joined #bitcoin-core-dev
251 2015-11-12T07:56:29  <randy-waterhouse> wumpus: tongue-in-cheek reference to "fake gold" accusations
252 2015-11-12T07:56:43  <MarcoFalke> wumpus, using `-proxy` I get "2015-11-12 07:43:06 ERROR: Proxy error: connection not allowed
253 2015-11-12T07:56:43  <MarcoFalke> " for .onion and "2015-11-12 07:43:56 ERROR: Proxy error: host unreachable
254 2015-11-12T07:56:44  <MarcoFalke> " for the seeds
255 2015-11-12T07:57:01  <MarcoFalke> Are those implying censorship?
256 2015-11-12T07:57:03  <wumpus> randy-waterhouse: ahh old pyrite, or fool's gold :-)
257 2015-11-12T07:57:25  <randy-waterhouse> heh, right
258 2015-11-12T07:58:21  <wumpus> MarcoFalke: it may just mean what it says - the proxy cannot reach those hosts
259 2015-11-12T07:58:52  <wumpus> MarcoFalke: sometimes hosts are truly unreachable. I think you need to look at tor's logging to see if it can reach the first hop to troubleshoot this
260 2015-11-12T07:59:34  <wumpus> but at least you're one step further- you get errors from the proxy, instead of the level before of connecting to the proxy
261 2015-11-12T08:00:44  <wumpus> it may just have a hard time finding a working .onion peer here try this one: nns4r54x3lfbrkq5.onion
262 2015-11-12T08:13:25  <MarcoFalke> It only connects to  nkf5e6b7pl4jfd4a.onion (TheBlueMatt)
263 2015-11-12T08:14:44  <wumpus> ok, in that case it's at least not censorship, if it can connect to one onion it can in principle connect to all without your ISP knowing anything
264 2015-11-12T08:18:15  *** Thireus has joined #bitcoin-core-dev
265 2015-11-12T08:26:04  *** NLNico has quit IRC
266 2015-11-12T08:26:11  <MarcoFalke> I don't have to set `-proxy=127...`?
267 2015-11-12T08:26:24  <wumpus> you do have to set a proxy
268 2015-11-12T08:26:35  <wumpus> otherwise it won't know how to connect to onions
269 2015-11-12T08:26:47  <MarcoFalke> Right now I am doing `-onlynet=onion  -torpassword=`
270 2015-11-12T08:26:58  <MarcoFalke> and I can `-connect=` to it
271 2015-11-12T08:27:01  <MarcoFalke> but not sync
272 2015-11-12T08:27:18  <wumpus> it isn't able to derive the Tor proxy from the control port. Maybe this is possible, but it is not implemented.
273 2015-11-12T08:28:07  <MarcoFalke> I am doing `-connect=moheur3skn7jbca4.onion   -proxy= -testnet` locally and get the connection
274 2015-11-12T08:29:45  <MarcoFalke> But I need the proxy for outgoing connections, I see.
275 2015-11-12T08:29:52  <wumpus> ohh testnet
276 2015-11-12T08:30:15  <wumpus> probably very, very few onion testnet peers
277 2015-11-12T08:30:36  <MarcoFalke> Still, why wouldn't they sync each others blocks if connected?
278 2015-11-12T08:32:01  <MarcoFalke> Could you try  `-connect=moheur3skn7jbca4.onion -proxy= -testnet` on your machine?
279 2015-11-12T08:32:27  <wumpus> sure
280 2015-11-12T08:32:38  <wumpus> with an empty datadir?
281 2015-11-12T08:33:03  <MarcoFalke> yes, I have 2400 blocks
282 2015-11-12T08:33:17  <MarcoFalke> *47762
283 2015-11-12T08:34:13  <wumpus> 2015-11-12 08:34:02 SOCKS5 connecting moheur3skn7jbca4.onion / 2015-11-12 08:34:08 SOCKS5 connected moheur3skn7jbca4.onion/ 2015-11-12 08:34:08 receive version message: /Satoshi:0.11.99/: version 70011, blocks=47762, us=, peer=1
284 2015-11-12T08:34:27  <wumpus> and nothing after that
285 2015-11-12T08:34:53  <wumpus> could mean two things a) I'm not sending getheaders b) peer is ignoring my getheaders
286 2015-11-12T08:35:27  <wumpus> as for (b) I know this issue all too well: https://github.com/bitcoin/bitcoin/issues/6971
287 2015-11-12T08:36:25  <wumpus> but this required P2P-level debugging, it's not tor's fault :)
288 2015-11-12T08:36:39  *** gribble has quit IRC
289 2015-11-12T08:37:08  <MarcoFalke> I don't run master. Still running `f1d548d squashme: mention torcontrol in doc/tor.md
290 2015-11-12T08:37:09  <MarcoFalke> ` with no merge
291 2015-11-12T08:37:41  <MarcoFalke> I guess thats an tested ACK on 6639, then?
292 2015-11-12T08:37:46  <wumpus> that doesn't matter - if it's the *other* peer that's ignoring you, nothing you can do
293 2015-11-12T08:38:54  <MarcoFalke> We are both in IBD?
294 2015-11-12T08:39:03  <wumpus> using debug=net I can see that I'm sending getheaders, but he's not responding
295 2015-11-12T08:39:26  <wumpus> or he's forked off the chain completely
296 2015-11-12T08:39:45  <wumpus> anything that would make the tip older than 24 hours old will preent him from sending headers
297 2015-11-12T08:40:14  <wumpus> I remember there were some issues on testnet
298 2015-11-12T08:41:02  <MarcoFalke> block 47762 is Jan 2013
299 2015-11-12T08:41:50  <wumpus> but he seems stuck there. blocks=... doesn't go up after reconnecting :)
300 2015-11-12T08:42:06  <wumpus> so we'll have to find a good testnet onion peer to check this
301 2015-11-12T08:42:16  <MarcoFalke> one sec...
302 2015-11-12T08:42:55  <MarcoFalke> going to sync and then `-connect=` again
303 2015-11-12T08:43:31  <wumpus> I can spin up one, after catching up (or I'll just comment out that check in #6971, so anyone-can-getheaders)
304 2015-11-12T08:44:14  <wumpus> ok one min
305 2015-11-12T08:45:03  *** JackH has joined #bitcoin-core-dev
306 2015-11-12T08:47:26  *** gribble has joined #bitcoin-core-dev
307 2015-11-12T08:48:30  *** BashCo has quit IRC
308 2015-11-12T08:50:03  <wumpus> MarcoFalke:  tor: Got service ID vbzzvljohcyzlfzg, advertizing service vbzzvljohcyzlfzg.onion:8333   you should be able to sync from that
309 2015-11-12T08:50:24  <wumpus> eh wait that's not testnet
310 2015-11-12T08:51:00  <wumpus> it is: ytiwtue6no6c3cit.onion:18333
311 2015-11-12T08:51:33  <wumpus> may take some seconds for the new service id to propagate over the tor network
312 2015-11-12T08:53:07  <MarcoFalke> Could you make that `-connect=moheur3skn7jbca4.onion -proxy= -testnet` and then sync me?
313 2015-11-12T08:53:27  <MarcoFalke> because I am still running with no `-proxy` to prevent outgoing connections
314 2015-11-12T08:55:39  <wumpus> ok
315 2015-11-12T08:57:03  <wumpus> I can connect, but I cannot sync from you, because your node is in IBD
316 2015-11-12T08:58:23  <wumpus> /Satoshi:0.11.99(hi_wumpus)/ hehe
317 2015-11-12T09:00:01  <wumpus> anyhow yes that's a tested ACK on #6639 - it's clear that creating the hidden service and makng it connectible works
318 2015-11-12T09:06:10  *** molly has quit IRC
319 2015-11-12T09:09:19  <MarcoFalke> Ok, then. Will do some more testing and comment on the pull. ;)
320 2015-11-12T09:09:45  *** BashCo has joined #bitcoin-core-dev
321 2015-11-12T09:12:12  *** MarcoFalke has left #bitcoin-core-dev
322 2015-11-12T10:22:58  *** rubensayshi has joined #bitcoin-core-dev
323 2015-11-12T10:25:43  *** zarathustra has joined #bitcoin-core-dev
324 2015-11-12T10:34:41  *** d_t has quit IRC
325 2015-11-12T10:38:42  <phantomcircuit> gmaxwell, so im (soft) disabling walletbroadcast but what am i doing about whitelisted peers
326 2015-11-12T10:38:56  <phantomcircuit> it seems that there's an argument to be made both ways
327 2015-11-12T10:40:29  <gmaxwell> phantomcircuit: relay for them; and add an option to control their relay special casing (e.g. so you can turn off the behavior that irritates me too. :) ) and then you can soft set that off too when blocksonly is on.
328 2015-11-12T10:41:11  <sipa> that seems neat
329 2015-11-12T10:41:59  <phantomcircuit> -relaywhitelisted ?
330 2015-11-12T10:43:29  *** pmienk has quit IRC
331 2015-11-12T10:43:41  <gmaxwell> whitelistalwaysrelay  might be more descriptive.
332 2015-11-12T10:44:07  <gmaxwell> since turning it off should turn off the current behavior where we will accept txn from a whitelisted peer that we wouldn't otherwise.
333 2015-11-12T10:44:42  <sipa> right, normal relay for whitelisted peers would keep working even without the special case behaviour
334 2015-11-12T10:45:09  <phantomcircuit> default to true or false?
335 2015-11-12T10:45:23  <phantomcircuit> i guess default to true unless blocksonly=1
336 2015-11-12T10:45:29  <gmaxwell> default to true (current behavior) and soft-off it in blocksonly mode.
337 2015-11-12T10:46:16  <gmaxwell> then people who want to relay for nodes behind them while in blocksonly can turn it on.
338 2015-11-12T10:46:19  *** pmienk has joined #bitcoin-core-dev
339 2015-11-12T10:56:04  <phantomcircuit> gmaxwell, bleh it's now a hideously huge 13 loc patch
340 2015-11-12T10:56:32  <phantomcircuit> :P
341 2015-11-12T10:57:30  <gmaxwell> obviously you have to find some unrelated code to remove at the same time... (heh no not really)
342 2015-11-12T10:59:34  *** paveljanik has joined #bitcoin-core-dev
343 2015-11-12T11:00:08  <wumpus> phantomcircuit: we don't make it easy to make a small patch, do we :)
344 2015-11-12T11:01:27  <sipa> we need patches that are C++-neutral
345 2015-11-12T11:02:34  <phantomcircuit> wumpus, lol @ constant
346 2015-11-12T11:02:38  <phantomcircuit> BUT BUT
347 2015-11-12T11:03:18  <phantomcircuit> wumpus, main.h or net.h?
348 2015-11-12T11:03:48  <wumpus> phantomcircuit: depends on where it's used
349 2015-11-12T11:03:52  <wumpus> I thought main.h
350 2015-11-12T11:04:21  *** randy-waterhouse has quit IRC
351 2015-11-12T11:05:51  <sipa> phantomcircuit: your patch will go into history books as the one with the higher discussion per lines of code patch ever that got merged
352 2015-11-12T11:06:26  <phantomcircuit> lol
353 2015-11-12T11:07:54  <paveljanik> if it will be merged at all! :-)
354 2015-11-12T11:26:41  <gmaxwell> sipa: pfft.
355 2015-11-12T11:27:39  <gmaxwell> https://github.com/bitcoin/bitcoin/commit/b196b685c9089b74fd4ff3d9a28ea847ab36179b
356 2015-11-12T11:36:35  <phantomcircuit> oy the mix of logic between net.cpp and main.cpp is super annoying
357 2015-11-12T11:36:57  <sipa> phantomcircuit: the certainties of life :)
358 2015-11-12T11:37:13  <phantomcircuit> net.cpp:454:89: error: ‘DEFAULT_BLOCKSONLY’ was not declared in this scope
359 2015-11-12T11:37:20  <phantomcircuit> well yes because it's defined in main.h
360 2015-11-12T11:37:23  <phantomcircuit> >.>
361 2015-11-12T11:37:58  <sipa> put it in the node state in main
362 2015-11-12T11:38:33  <phantomcircuit> sipa, put it in CNode ?
363 2015-11-12T11:38:37  <sipa> no
364 2015-11-12T11:38:42  <sipa> in CNodeState
365 2015-11-12T11:39:09  <sipa> CNode is the network layer's kitchen sink
366 2015-11-12T11:39:19  <sipa> (with a ton of state from processing)
367 2015-11-12T11:39:25  <sipa> CNodeState is just processing
368 2015-11-12T11:40:20  <phantomcircuit> sipa, is that accessible from CNode::Cnode ?
369 2015-11-12T11:40:25  <sipa> no
370 2015-11-12T11:40:27  <sipa> that's the point
371 2015-11-12T11:40:31  <sipa> none of this should be in net
372 2015-11-12T11:45:23  <gmaxwell> http://blog.nordeus.com/dev-ops/power-failure-testing-with-ssds.htm
373 2015-11-12T11:52:16  <wumpus> <phantomcircuit> oy the mix of logic between net.cpp and main.cpp is super annoying <- oh, you're only now discovering that? :-)
374 2015-11-12T11:52:32  <phantomcircuit> sipa, ok then i have to move PushVerison out of Cnode::CNode :/
375 2015-11-12T11:52:52  <phantomcircuit> (it actually shouldn't be there anyways so i guess net win... horray)
376 2015-11-12T11:59:20  <wumpus> but yes, the split between net.cpp and main.cpp is not optimal, it left too many P2P related code in main
377 2015-11-12T11:59:39  <GitHub139> [bitcoin] sipa opened pull request #6996: Add preciousblock RPC (master...preciousblock) https://github.com/bitcoin/bitcoin/pull/6996
378 2015-11-12T12:00:18  <sipa> wumpus: and too much processing in net
379 2015-11-12T12:00:54  <sipa> vRecvGetData has no business in CNode, for example
380 2015-11-12T12:01:02  <wumpus> right
381 2015-11-12T12:01:22  <sipa> how do i run the python tests from the command line?
382 2015-11-12T12:01:49  <wumpus> cd qa/pulltester; ./rpc-tests.py
383 2015-11-12T12:02:57  <sipa> ha, ok!
384 2015-11-12T12:04:41  <wumpus> and if you want to run a seperate test you can execute it manually from qa/rpc-tests
385 2015-11-12T12:05:23  <sipa> oh lol, i can't remember it being so easy
386 2015-11-12T12:05:46  *** jtimon has joined #bitcoin-core-dev
387 2015-11-12T12:06:43  <wumpus> whoa does anyone ever read README.md? :-) it still mentions the pull-tester
388 2015-11-12T12:06:47  <GitHub93> [bitcoin] sturles opened pull request #6997: Don't use hard-coded AllowFree, as this is usually far too low. (master...no-default-free-priority) https://github.com/bitcoin/bitcoin/pull/6997
389 2015-11-12T12:09:56  <GitHub55> [bitcoin] laanwj opened pull request #6998: doc: Remove mention of pulltester from README.md (master...2015_11_readme_pull_pulltester) https://github.com/bitcoin/bitcoin/pull/6998
390 2015-11-12T12:13:46  <wumpus> I also think the "Manual Quality Assurance (QA) Testing" is very much out of date
391 2015-11-12T12:15:53  <GitHub86> [bitcoin] jonasschnelli opened pull request #6999: add (max)uploadtarget infos to getnetinfos RPC help (master...2015/11/maxuploadtarget_rpc_help) https://github.com/bitcoin/bitcoin/pull/6999
392 2015-11-12T12:15:56  <GitHub29> [bitcoin] jonasschnelli opened pull request #7000: [Qt] add shortcurts for debug-/console-window (master...2015/11/qt_shortcuts) https://github.com/bitcoin/bitcoin/pull/7000
393 2015-11-12T12:15:58  <jonasschnelli> Yeah. PR #7000!
394 2015-11-12T12:16:28  <phantomcircuit> sipa, where else would vRecvGetData be?
395 2015-11-12T12:16:38  <sipa> phantomcircuit: CNodeState
396 2015-11-12T12:16:45  <phantomcircuit> ah
397 2015-11-12T12:17:32  <phantomcircuit> sipa, is the distinction you're thinking between the socket and the protocol things?
398 2015-11-12T12:17:48  <phantomcircuit> if so maybe CBitcoinNode : CNode
399 2015-11-12T12:18:37  <sipa> phantomcircuit: the idea (well, my idea... i don't know how many people share it) is that CNode should hold pointers to various module-specific state objects
400 2015-11-12T12:18:52  <sipa> phantomcircuit: and that when invoking code from one module, it just gets its own state objected handed to it
401 2015-11-12T12:19:43  <sipa> so ping handling can be entirely separate from block synchronization for example, and use completely independent locks, and even be totally unaware of the other's existance
402 2015-11-12T12:23:39  <wumpus> sounds like a sensible way to handle a protocol with different (mostly) independent concerns
403 2015-11-12T12:28:42  <phantomcircuit> yes except for the part where we decided to maintain the order of responses
404 2015-11-12T12:29:35  <wumpus> uhm, you mean that responses come in the same order as commands that trigger them?
405 2015-11-12T12:30:03  <wumpus> that's a pretty sane assumption that holds for most network protocols AFAIK
406 2015-11-12T12:30:39  <sipa> phantomcircuit: that's not incompatible
407 2015-11-12T12:31:08  <sipa> phantomcircuit: the ping module could be processing a message from node A, while the block sync logic is processing a message from node B
408 2015-11-12T12:32:51  <phantomcircuit> ah
409 2015-11-12T12:33:23  <phantomcircuit> wumpus, there's afaict no really good reason to enforce ordering of the responses actually
410 2015-11-12T12:33:54  <phantomcircuit> the only thing that relies on it in anyway is the initial block download thing
411 2015-11-12T12:34:02  <phantomcircuit> and im not sure it really does anymore
412 2015-11-12T12:34:06  <sipa> it does
413 2015-11-12T12:34:09  <sipa> normal block sync does too
414 2015-11-12T12:34:20  <sipa> bip37-based downloading does too
415 2015-11-12T12:34:27  <wumpus> phantomcircuit: I don't see a reason to make it different though
416 2015-11-12T12:34:36  <sipa> and a ton of test infrastructure relies on it
417 2015-11-12T12:36:34  <wumpus> is there anything specific that could be done better if it was not the case?
418 2015-11-12T12:59:56  *** go1111111 has quit IRC
419 2015-11-12T13:02:10  *** go1111111 has joined #bitcoin-core-dev
420 2015-11-12T13:09:51  *** fkhan has quit IRC
421 2015-11-12T13:24:56  *** fkhan has joined #bitcoin-core-dev
422 2015-11-12T13:37:39  <GitHub129> [bitcoin] sipa pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/5fcc14ee0562...54e8bfec8387
423 2015-11-12T13:37:40  <GitHub129> bitcoin/master 06d81ad Alex Morcos: Skip BIP30 check after BIP34 activation
424 2015-11-12T13:37:40  <GitHub129> bitcoin/master 33c90cf Alex Morcos: Make skipping BIP30 check chain agnostic
425 2015-11-12T13:37:41  <GitHub129> bitcoin/master 54e8bfe Pieter Wuille: Merge pull request #6931...
426 2015-11-12T13:37:42  <GitHub166> [bitcoin] sipa closed pull request #6931: Skip BIP 30 verification where not necessary (master...skipBIP30) https://github.com/bitcoin/bitcoin/pull/6931
427 2015-11-12T13:45:40  *** jgarzik has quit IRC
428 2015-11-12T13:49:38  *** fkhan has quit IRC
429 2015-11-12T13:51:56  *** molly has joined #bitcoin-core-dev
430 2015-11-12T14:01:09  *** jtimon has quit IRC
431 2015-11-12T14:04:46  <phantomcircuit> wumpus, yes we could improve the throughput of getdata requests if they were out of order
432 2015-11-12T14:05:06  <phantomcircuit> as it is there's a large latency hit if the block requested isn't in cache
433 2015-11-12T14:05:06  <wumpus> how?
434 2015-11-12T14:06:05  <phantomcircuit> wumpus, parallel fetch from disk
435 2015-11-12T14:06:41  <wumpus> getdata is already enough of a i/o DoS attack as it is, thank you
436 2015-11-12T14:06:49  <sipa> i don't think we should go write our own optimizer for disk fetches
437 2015-11-12T14:06:53  <phantomcircuit> hell we could even rewrite the requests so that they hit disk sequentially
438 2015-11-12T14:07:17  <sipa> phantomcircuit: you'd also need a reorganizer that rewrites the block files to be sequential
439 2015-11-12T14:07:25  <sipa> please don't go there
440 2015-11-12T14:07:29  <wumpus> TBH this doesn't sound very reasonable nor useful, please profile first before you think about optimizing anything
441 2015-11-12T14:08:12  <phantomcircuit> sipa, parallel fetch isn't anywhere near as complex as writing our own optimizer.... but it might need one if it's not to completely break hdds
442 2015-11-12T14:08:23  <wumpus> optimizing the disk component of getdata doesn't rank anywhere in the 'concerns that slow down bitcoin core in practice' I'm sure
443 2015-11-12T14:08:57  *** BashCo has quit IRC
444 2015-11-12T14:10:04  <sipa> phantomcircuit: so if that's really a concern at a time when the protocol too strongly relies on the ordering to meaningfully change it, we can always add a pargetdata which allows answers in any order. fixed.
445 2015-11-12T14:10:45  <wumpus> we already do parallel getdata on different nodes, that's even better paralleism thanks to sipa :)
446 2015-11-12T14:13:18  <phantomcircuit> wumpus, it basically only effects people like me syncing over gigabit lans from systems with the entire thing in page cache
447 2015-11-12T14:13:31  <wumpus> ... right
448 2015-11-12T14:13:59  <phantomcircuit> increasing the fetch depth to 100 mostly fixes it
449 2015-11-12T14:23:58  *** jgarzik has joined #bitcoin-core-dev
450 2015-11-12T14:23:58  *** jgarzik has joined #bitcoin-core-dev
451 2015-11-12T14:24:49  *** Guyver2 has joined #bitcoin-core-dev
452 2015-11-12T14:28:56  <morcos> sipa: ha, i was chasing my own tail on that insecure_rand.  Ok so should I fix the extra 30 bits of randomness or take our chances on only 32.  I vote take our chances.
453 2015-11-12T14:29:46  <sipa> morcos: will the test fail if it actually is identical?
454 2015-11-12T14:30:48  <morcos> i was going to consider testing that.  i was thinking that it might allow you to spend a duplicate coinbase which has already been spent
455 2015-11-12T14:30:51  *** fkhan has joined #bitcoin-core-dev
456 2015-11-12T14:31:36  <morcos> but maybe even thats ok, unless you randomly spend the second copy in exactly the same way.  in which case it would break
457 2015-11-12T14:31:39  <morcos> i think?
458 2015-11-12T14:32:14  <sipa> morcos: hmm, your cache will be very slow
459 2015-11-12T14:32:16  <sipa> eh, small
460 2015-11-12T14:32:23  *** Guyver2 has quit IRC
461 2015-11-12T14:32:29  <morcos> it is slow, but what do you mean small?
462 2015-11-12T14:33:10  <sipa> eh, for some reason i was assuming that normal transactions decreased the number of entries... but they create a new output too of course
463 2015-11-12T14:33:23  <morcos> it about triples the time of that test from 300ms to 1sec
464 2015-11-12T14:33:24  *** Guyver2 has joined #bitcoin-core-dev
465 2015-11-12T14:34:09  <sipa> i guess you grow to around 4000 entries
466 2015-11-12T14:34:28  <morcos> yeah so on a typical test you generate around 4000 coinbases about 40 of which are duplicate and you end up towards the end of the test, failing on about 40 of them (if you have the broken fix)
467 2015-11-12T14:34:38  <morcos> that should be right
468 2015-11-12T14:35:08  <morcos> sorry around 400 of which are duplicate
469 2015-11-12T14:37:09  <morcos> b/c you only fail on the duplicate if it happens to get spent before its flushed.   so i didn't check for coverage of that, since its a bit tricky, but that seemed a high enough failure rate that slight tweaks to the test shouldn't affect it.
470 2015-11-12T14:38:21  <sipa> you have around 1.7% chance to hit a duplicate during a test run :)
471 2015-11-12T14:38:56  <sipa> oh, wait, that's assuming there are only coinbases
472 2015-11-12T14:41:30  <sipa> around 0.2% chance per run; i think you need 64-bit amounts :)
473 2015-11-12T14:41:59  <morcos> how did you get 1.7%?  does that sound really high?  do you have to have the same random 32 bit number in 4000 tries (or 40k for the calculation you did)
474 2015-11-12T14:42:28  <sipa> i calculated 1 - multiply(1 - x/2^32, x=1..4000)
475 2015-11-12T14:42:33  <sipa> that's 0.2%
476 2015-11-12T14:42:53  <morcos> oh silly me
477 2015-11-12T14:43:06  <morcos> yes i calculated that wrong
478 2015-11-12T14:44:27  <morcos> ok i'll make it a 64-bit number with the cast.
479 2015-11-12T14:44:50  <sipa> of course, you could just use the iteration counter instead :)
480 2015-11-12T14:45:31  <morcos> good even easier
481 2015-11-12T14:45:52  <morcos> in a car now..  i'll push it in a bit
482 2015-11-12T14:58:23  <morcos> ok lots of traffic, squashed the fix in.
483 2015-11-12T14:59:06  <instagibbs> Hopefully not while driving :)
484 2015-11-12T14:59:40  <morcos> heh, no...
485 2015-11-12T15:00:04  <jgarzik> heh
486 2015-11-12T15:00:47  <sipa> Hopefully not while trafficking
487 2015-11-12T15:03:06  <jgarzik> with Uber & self driving cars, non stop hacking is possible :)  Ah, the future.
488 2015-11-12T15:03:33  <sipa> Where is my hoverboard?
489 2015-11-12T15:23:51  *** ParadoxSpiral has joined #bitcoin-core-dev
490 2015-11-12T15:25:29  <jgarzik> sipa, http://hendohover.com
491 2015-11-12T15:26:34  <sipa> jgarzik: ha, yes, i've heard about that
492 2015-11-12T15:27:17  <wumpus> hah
493 2015-11-12T15:28:24  *** treehug88 has joined #bitcoin-core-dev
494 2015-11-12T15:29:09  *** Thireus has quit IRC
495 2015-11-12T15:48:53  *** zooko has joined #bitcoin-core-dev
496 2015-11-12T15:49:52  *** jtimon has joined #bitcoin-core-dev
497 2015-11-12T16:06:19  *** zooko has quit IRC
498 2015-11-12T16:32:52  *** zooko has joined #bitcoin-core-dev
499 2015-11-12T16:38:14  <GitHub77> [bitcoin] laanwj pushed 4 new commits to master: https://github.com/bitcoin/bitcoin/compare/54e8bfec8387...eb6172a8ca7e
500 2015-11-12T16:38:15  <GitHub77> bitcoin/master 830e3f3 Pieter Wuille: Make sigcache faster and more efficient
501 2015-11-12T16:38:15  <GitHub77> bitcoin/master 0b9e9dc Pieter Wuille: Evict sigcache entries that are seen in a block
502 2015-11-12T16:38:16  <GitHub77> bitcoin/master 69d373f Pieter Wuille: Don't wipe the sigcache in TestBlockValidity
503 2015-11-12T16:38:21  <GitHub101> [bitcoin] laanwj closed pull request #6918: Make sigcache faster, more efficient, larger (master...smallsigcache) https://github.com/bitcoin/bitcoin/pull/6918
504 2015-11-12T16:48:17  *** rubensayshi has quit IRC
505 2015-11-12T16:52:57  *** d_t has joined #bitcoin-core-dev
506 2015-11-12T16:56:36  *** Thireus has joined #bitcoin-core-dev
507 2015-11-12T16:58:17  *** d_t has quit IRC
508 2015-11-12T17:37:23  *** MarcoFalke has joined #bitcoin-core-dev
509 2015-11-12T17:45:57  *** skyraider has joined #bitcoin-core-dev
510 2015-11-12T18:33:46  <GitHub51> [bitcoin] laanwj pushed 6 new commits to master: https://github.com/bitcoin/bitcoin/compare/eb6172a8ca7e...bd629d77edbe
511 2015-11-12T18:33:46  <GitHub33> [bitcoin] laanwj closed pull request #6639: net: Automatically create hidden service, listen on Tor (master...2015_08_tor_hs_v2) https://github.com/bitcoin/bitcoin/pull/6639
512 2015-11-12T18:33:47  <GitHub51> bitcoin/master 8f4e67f Wladimir J. van der Laan: net: Automatically create hidden service, listen on Tor...
513 2015-11-12T18:33:47  <GitHub51> bitcoin/master 2f796e5 Peter Todd: Better error message if Tor version too old
514 2015-11-12T18:33:48  <GitHub51> bitcoin/master 09c1ae1 Wladimir J. van der Laan: torcontrol improvements and fixes...
515 2015-11-12T18:44:30  *** morcos has quit IRC
516 2015-11-12T18:45:41  *** morcos has joined #bitcoin-core-dev
517 2015-11-12T18:59:00  *** d_t has joined #bitcoin-core-dev
518 2015-11-12T18:59:41  *** d_t has joined #bitcoin-core-dev
519 2015-11-12T19:00:20  *** d_t has joined #bitcoin-core-dev
520 2015-11-12T19:00:58  *** d_t has joined #bitcoin-core-dev
521 2015-11-12T19:02:01  <jcorgan> wow, i missed this PR entirely.  what a great idea, glad it is merged.
522 2015-11-12T19:12:13  <MarcoFalke> yes, good to see this in 0.12!
523 2015-11-12T19:35:13  <GitHub56> [bitcoin] sdaftuar closed pull request #6992: [WIP] [Mempool] Add space for priority transactions (master...add-priority-to-mempool) https://github.com/bitcoin/bitcoin/pull/6992
524 2015-11-12T19:55:52  *** randy-waterhouse has joined #bitcoin-core-dev
525 2015-11-12T19:57:28  <Luke-Jr> morcos: ok, so back to your stuff? :P
526 2015-11-12T19:58:22  <gmaxwell> Luke-Jr: on priority. the exact current behavior requires walking all inputs for all transactioin in the mempool on every CNB to update priorities. With large mempools this is an enormous amount of time, and we cannot continue doing this.
527 2015-11-12T19:58:27  <Luke-Jr> morcos: can you make it so the accurate priority gets used with a policy option miners can enable? that makes current policy possible, right?
528 2015-11-12T19:58:34  <gmaxwell> There are multiple alternatives but most result in lots of software complexity, which we also probably don't want.
529 2015-11-12T19:58:46  <morcos> Hmm.. I think that would be possible
530 2015-11-12T19:58:47  <Luke-Jr> gmaxwell: it can be disabled by default
531 2015-11-12T19:58:59  <morcos> The original version of my pull did the expensive calculation
532 2015-11-12T19:59:14  <morcos> it'll make the code slightly uglier casing that out
533 2015-11-12T19:59:21  <gmaxwell> Luke-Jr: I think miners would be stupid to enable something that makes CNB 10x + slower, and we'd be stupid to offer it since slow CNB means more 'spv mining' and other bad effects.
534 2015-11-12T19:59:32  <morcos> but yes, i agree with that concern
535 2015-11-12T19:59:44  <sdaftuar> what's the motivation to keep the current version of priority rather than starting priority as the metric?
536 2015-11-12T19:59:51  <Luke-Jr> gmaxwell: CNB is not time-critical with Eloipool, and doesn't imply SPV mining
537 2015-11-12T19:59:53  <gmaxwell> I wouldn't _oppose_ an option, default off, except on the basis of carrying around code, and lack of testing (but I assume you'll test)
538 2015-11-12T19:59:56  <morcos> Luke-Jr: it may be something you can just easily patch in yourself
539 2015-11-12T20:00:07  <petertodd> sdaftuar: I think for most uses of priority, updating every time shouldn't make much of a difference - we're talking about old inputs after all
540 2015-11-12T20:00:21  <petertodd> sdaftuar: whats the difference betweek 1 month and 1 month + 1 hour?
541 2015-11-12T20:00:31  <Luke-Jr> I suggest we have it in Core for 0.12, and try to optimise (or remove) by 0.13
542 2015-11-12T20:00:45  <gmaxwell> sdaftuar: I think starting priority is a pretty poor approximation, though on the basis that I wouldn't be too opposed to ditching priority entirely, using a poor approximation isn't so bad.
543 2015-11-12T20:00:56  <sdaftuar> it just seems pretty arbitrary to me to decide to age inputs rather than use the age when relayed to you
544 2015-11-12T20:01:04  <sdaftuar> so given two arbitrary calculations, might as well do the faster one
545 2015-11-12T20:01:19  <gmaxwell> sdaftuar: age inputs has a property that the tx shouldn't hang around forever, but will actually bubble up and get mined at some point.
546 2015-11-12T20:01:24  <Luke-Jr> sdaftuar: well, aging inputs properly may in some cases be necessary for txns to confirm ever
547 2015-11-12T20:01:44  <sdaftuar> well we're about to deploy RBF and expiry
548 2015-11-12T20:01:46  <petertodd> the only time this matters is very high value outputs which we're expecting to accumulate priority quickly, and why incentivise people to put everything in on etxout? can have bad privacy implications
549 2015-11-12T20:02:02  <Luke-Jr> sdaftuar: as optional policies
550 2015-11-12T20:02:18  *** skyraider has quit IRC
551 2015-11-12T20:02:22  <phantomcircuit> Luke-Jr, then write a background thread to update priority
552 2015-11-12T20:02:27  <petertodd> phantomcircuit: +1
553 2015-11-12T20:02:31  <morcos> Luke-Jr: I'll see if I can whip up a dependent PR that adds back in the accurate calculation with an option, that could be included.
554 2015-11-12T20:02:43  <Luke-Jr> phantomcircuit: that would be one way to optimise it, but I hope to have something better for 0.13 anyway
555 2015-11-12T20:03:13  <gmaxwell> I do agree if the functionality was retained update in the background would be prudent.
556 2015-11-12T20:03:15  <Luke-Jr> morcos: well, I don't want to merge the existing PR without that included, so might as well put it in the same PR..
557 2015-11-12T20:03:16  <morcos> If you want to properly calculate priority, you need to do it dynamicalyl as in 6357 otherwise you destroy your UTXO cache
558 2015-11-12T20:03:26  <morcos> 6357 is strictly better than that
559 2015-11-12T20:03:40  <morcos> Luke-Jr: yes but that may not be the prevailing view.
560 2015-11-12T20:03:58  <morcos> Well lets argue about it when we see what it looks like anyway
561 2015-11-12T20:04:04  <phantomcircuit> Luke-Jr, the priority calculation as it existed before mempool limiting cannot be reasonably calculated in realtime, the only way to do it is as a background thread
562 2015-11-12T20:04:11  <wumpus>  * [new tag]         v0.11.2 -> v0.11.2
563 2015-11-12T20:04:23  <petertodd> wumpus: +1
564 2015-11-12T20:04:24  <morcos> First I have to fix the existing pull, it uses the wrong calculation now anyway
565 2015-11-12T20:04:30  <petertodd> wumpus: or really, -rc
566 2015-11-12T20:04:39  <Luke-Jr> petertodd: you missed the rc
567 2015-11-12T20:04:50  <morcos> phantomcircuit: no, background thread is bad b/c of cache.  see 6357
568 2015-11-12T20:04:54  <petertodd> Luke-Jr: lol, I mean, we're taking the rc off of v0.11.2rc1 :)
569 2015-11-12T20:05:08  <Luke-Jr> o
570 2015-11-12T20:05:27  <phantomcircuit> morcos, a background thread is equally as bad as any other way of doing it in that regard
571 2015-11-12T20:06:00  <morcos> 6357 does not destroy your cache, and is way less computational complexity
572 2015-11-12T20:06:26  <gmaxwell> as far as this priority discussion, I think people are being far too dismissive about priority. In particular, this idea of fee competition vs dos attacks is a situation where the defender has no advantage on an attacker at all... they spend the same. Having a tie breaker that rewards coins that haven't recently moved is, I think, an elegent way to create a gap so that attackers don't merely nee
573 2015-11-12T20:06:32  <gmaxwell> d to pay market rate but slightly above.  And impact to miner income can be made negligible.
574 2015-11-12T20:07:09  <morcos> gmaxwell: that argument doesn't make sense to me
575 2015-11-12T20:07:24  <morcos> i think the HOV/fast lane arguement makes sense
576 2015-11-12T20:07:32  <morcos> but to have priority boost fee doesn't make sense
577 2015-11-12T20:08:04  <petertodd> gmaxwell: why do we assume the defender always has more old coins than the attacker?
578 2015-11-12T20:08:07  <morcos> so it boosts it by delta.  then attackers just have to pay market rate + delta, who cares, unless delta is large, in which case then you have prioirty txs crowding out fee paying txs and hurting miners profits
579 2015-11-12T20:08:10  <gmaxwell> morcos: right now the attacker need only bid episilon more than market rate to replace the honest users at market rate in the mempool.
580 2015-11-12T20:08:52  <gmaxwell> petertodd: it's a hurestic, and it maps to the actual attack pattern in that by their nature an attack requires making lots of transactions. Honest use may or may not have that property; but it remains a distinguisher, if not a super strong one.
581 2015-11-12T20:08:59  <morcos> its impossible to pick the conversion so that it accomplishes exactly your goal.. unless you reserve only a certain amount of space for priority, which is what the existing code does
582 2015-11-12T20:09:01  <petertodd> e.g. IIRC in lightning you can be in a situation where you need to get a recently mined tx spent, and the attacker may be trying to flood mempools to keep you from doing that
583 2015-11-12T20:09:21  <morcos> the existing code makes sense for that purpose, i agree.  but combining to just one bucket filled on a combined metric does not
584 2015-11-12T20:09:34  <Luke-Jr> petertodd: I'm sure there are lightning-optimised policies that are possible?
585 2015-11-12T20:09:35  <petertodd> morcos: and *that* can be easily done with two separate mempools, each separately limited, which keeps the codebase simple
586 2015-11-12T20:09:54  <petertodd> Luke-Jr: better for privacy if you can't distingish lightning txs from othre txs
587 2015-11-12T20:10:06  <Luke-Jr> …
588 2015-11-12T20:10:12  <gmaxwell> morcos: Okay, you're right there. I was thinking that even having the attacker have to pay the extra delta is a benefit, but you're right that its probably negligible.
589 2015-11-12T20:10:13  <Luke-Jr> but you can
590 2015-11-12T20:11:13  <phantomcircuit> morcos, 6357 is less computationally complex, but significantly more complex code
591 2015-11-12T20:11:19  <wumpus> and another: * [new tag]         v0.10.4 -> v0.10.4
592 2015-11-12T20:11:27  <morcos> phantomcircuit: ok that i will agree with!
593 2015-11-12T20:11:34  <gmaxwell> The two pools approach can also resolve the repricing cost issue, I think, if transactions cannot move back from the feerate pool to the priority pool, and if the priority pool is small.
594 2015-11-12T20:11:53  <morcos> sdaftuar's pull was 2 pools
595 2015-11-12T20:12:04  <morcos> maintained in the same structure
596 2015-11-12T20:12:11  <phantomcircuit> morcos, and it's a bunch of caching of values which is genuinely hard to get right
597 2015-11-12T20:12:15  <gmaxwell> (as you could limit repricing to the priority pool if the above assumptions are kept)
598 2015-11-12T20:13:04  <gmaxwell> maybe I just need to come around to bluematt's thinking that priority is not worth maintaining...
599 2015-11-12T20:13:27  <dgenr8> different-sized mempools mean that attacker doesn't have a perfectly defined function to erase a tx with a given fee
600 2015-11-12T20:13:35  <morcos> gmaxwell: i mostly agree, i wanted to keep priority, but it is a lot of complication, and maybe its better to realize that if we can come up with some incentive compatible metric like priority
601 2015-11-12T20:13:41  <morcos> then we can just add that back in in a clean way
602 2015-11-12T20:13:54  <morcos> we dont' have to permanently shut the door on a metric like this
603 2015-11-12T20:14:10  <dgenr8> also, attacker has to erase 299M of transactions to affect the lowest-feerate tx that's going into the next block
604 2015-11-12T20:14:26  <dgenr8> at default mempool setting
605 2015-11-12T20:14:33  <sipa> dgenr8: highest
606 2015-11-12T20:15:55  <dgenr8> 300M to affect highest, no?
607 2015-11-12T20:16:06  <gmaxwell> morcos: yes, if block costing is changed, then it becomes incentive compatible and works better; though I am really really not fond of multi-metric block costs.
608 2015-11-12T20:16:34  <morcos> but block costs are inherently multi-metric with sig-op limit
609 2015-11-12T20:16:58  <morcos> you could buy extra block size with older age coins even
610 2015-11-12T20:17:06  <phantomcircuit> morcos, the sigop/byte limit is so high that no legitimate user is going to hit it
611 2015-11-12T20:17:10  <Luke-Jr> removing policy actually increases maintenance costs
612 2015-11-12T20:17:15  <sipa> dgenr8: oh, i see what you're saying; in theory, yes, but in practice chains of transactions are harder to kick out and tend to linger, making it more spread out
613 2015-11-12T20:17:15  <gmaxwell> morcos: yes, and I want that to go away (it's busted anyways. :) )  fortunately the sigop limit is high enough that under ordinary use it's never hit.
614 2015-11-12T20:17:44  <phantomcircuit> you basically need to have a transaction that consists of OP_DUP OP_DUP OP_CHECKSIG  repeated 100+ times to hit the limit
615 2015-11-12T20:17:44  <morcos> gmaxwell: ha, but you're the one tryign to worry about hitting it now b/c of the flood
616 2015-11-12T20:18:02  <morcos> if you dont' have multi metrics, then you have conversion factors between metrics which are hard to get right
617 2015-11-12T20:18:11  <gmaxwell> yea I know, I know, short term effects because of people specifically trying to exploit it.
618 2015-11-12T20:18:21  <gmaxwell> morcos: I don't know if for incentive reasons it matters if they're "right"
619 2015-11-12T20:18:45  <gmaxwell> in that I think mostly the sign matters.  "oh I should prefer to make transactions that do X because it'll cost less"
620 2015-11-12T20:18:57  <gmaxwell> or at least only matters to within an order of magnitude.
621 2015-11-12T20:18:58  <morcos> gmaxwell: fair enough, but i'd say heuristics for optimizing multi metrics are also good enough for mining
622 2015-11-12T20:19:20  *** d_t has quit IRC
623 2015-11-12T20:19:32  <gmaxwell> morcos: but will anyone ever implement them?  Multiple limits also complicate the implementation of fraud proofs, making it less likely that we'll ever have them.
624 2015-11-12T20:20:07  <morcos> sure i will!  but fraud proofs, hmm...  because you ahve to be able to prove any one of those things?
625 2015-11-12T20:20:53  <morcos> i don't understand enough about how fraud proofs work
626 2015-11-12T20:21:01  <gmaxwell> yes, and efficiently proving a global limit requires commiting to a tree over its computation.
627 2015-11-12T20:22:30  * michagogo has his gitian-build script already running
628 2015-11-12T20:23:01  <gmaxwell> E.g. you commit to the value of of each input and output, and hashtree it, and in the interior nodes include the sum fees of the children. Now you can make an efficient proof that a block creates inflation.   This can be done for any limit, e.g. sigops, whatever, but the interior nodes in the hashtree have to track each limit.
629 2015-11-12T20:24:41  *** d_t has joined #bitcoin-core-dev
630 2015-11-12T20:24:42  <gmaxwell> morcos: I worry less that you won't implement it, and more that miners won't bother running code that almost never fires. "Morcos solved it" is a stable solution only if we assume people are running your code, which they might not for smarter or dumber reasons.
631 2015-11-12T20:26:03  *** skyraider has joined #bitcoin-core-dev
632 2015-11-12T20:26:11  <gmaxwell> morcos: also I think seperate limits are somewhat backwards in the sense that what we really seek to limit is "cost"; and turning cost into seperate limits is something we do because we don't know how to weigh for cost, but we do know what demand looks like, and demand uses a little of each dimension.
633 2015-11-12T20:27:49  <gmaxwell> in any case the issue with fraud proofs is mostly just making things even more complex reduces the odds that they get correctly implemented.  Mempool hurestics are one thing-- they're not consensus critical, but limits are.
634 2015-11-12T20:27:51  <wangchun> Luke-Jr: We have priority size set to 0 for maybe one year. But I have blockminsize set to 250 KB.
635 2015-11-12T20:28:04  <sipa> wangchun: good to know!
636 2015-11-12T20:28:19  <michagogo> Also, I found out I'm a bit of an idiot. The issue I was having where as soon as I upgraded the LXC base, I had to keep it updated or the build would fail?
637 2015-11-12T20:28:22  *** d4de has joined #bitcoin-core-dev
638 2015-11-12T20:28:36  <Luke-Jr> wangchun: let's chat in #bitcoin since there is conversation here already
639 2015-11-12T20:28:51  <michagogo> The fix is just to on-target -u root rm /var/cache/gitian/initial-upgrade
640 2015-11-12T20:29:13  <michagogo> (on the target, before copying it back over the base)
641 2015-11-12T20:29:29  <michagogo> (That's just a flag file that the upgrade script touches)
642 2015-11-12T20:30:01  <kanzure> fraud proofs require each unique validation rule has separate fraud proof implementation to check https://bitcointalk.org/index.php?topic=1103281.msg11743498#msg11743498
643 2015-11-12T20:31:15  <morcos> gmaxwell: ok, i'm not against weighted cost, i just worry about getting the weights right..  especially if those are consensus rules!
644 2015-11-12T20:31:27  <jgarzik> giving #6771 (chain limits) one last test run
645 2015-11-12T20:31:49  <gmaxwell> kanzure: that link is not correct in the details fwiw.
646 2015-11-12T20:31:58  <michagogo> Hm, weird lines from configure:
647 2015-11-12T20:31:59  <kanzure> quick overview of wrongness?
648 2015-11-12T20:32:04  <gmaxwell> (like saying utxo commitments are required, they're not)
649 2015-11-12T20:32:09  <michagogo> Checking for QT... no
650 2015-11-12T20:32:10  <michagogo> Checking for QT... yes
651 2015-11-12T20:32:12  <kanzure> okay, i'll keep that in mind.
652 2015-11-12T20:32:19  <michagogo> One right after the other
653 2015-11-12T20:32:37  <sipa> gmaxwell: until a few days ago i would have claimed that utxo commitments are required too :)
654 2015-11-12T20:33:02  <gmaxwell> sipa: pshaw, I've known they were for like ... months now. You need to spend more time in my head. :)
655 2015-11-12T20:33:26  <sipa> *walks back slowly*
656 2015-11-12T20:33:45  <kanzure> why few days ago?
657 2015-11-12T20:34:03  <sipa> kanzure: because that's when gmaxwell told me how to do it without commitments in a PM :)
658 2015-11-12T20:34:09  <kanzure> PMs don't count
659 2015-11-12T20:34:12  <kanzure> that's cheating
660 2015-11-12T20:34:20  <gmaxwell> it's been discussed in #bitcoin-wizards.
661 2015-11-12T20:34:23  <sipa> Oh.
662 2015-11-12T20:34:26  <kanzure> ah okay
663 2015-11-12T20:34:41  <kanzure> great
664 2015-11-12T20:35:05  <gmaxwell> in any case, you do them without a utxo commitment by instead commiting to the input block and offset that the input came from.
665 2015-11-12T20:35:35  <gmaxwell> then fraud can be proven by simply showing whatever is at that location, if its not the thing its supposted to be.
666 2015-11-12T20:38:00  <gmaxwell> I mean, I'm probably at fault for making people think that its the only way to get a fraud proof for spending non-existing coins, since in the very first writeup I suggest that https://en.bitcoin.it/wiki/User:Gmaxwell/features#Proofs
667 2015-11-12T20:38:18  <gmaxwell> but always things like that should be considered necessary, back then I didn't give it more than a moment's thought.
668 2015-11-12T20:38:28  <gmaxwell> er considered sufficient not necessary.
669 2015-11-12T20:40:56  <kanzure> ok, just posted follow-up to that thread linking to this explanation.
670 2015-11-12T20:42:34  <gmaxwell> (and obviously, spends of already spent coins proven by showing the other spend)
671 2015-11-12T20:48:56  <phantomcircuit> morcos, i dont really see the point of using a complex cost metric today as the principle limiting factor is bandwidth (and presumably will remain that way for many years)
672 2015-11-12T20:49:50  <phantomcircuit> if i was building a cost function is would be weighted to size so heavily as to be redundant
673 2015-11-12T20:50:53  <jgarzik> morcos, currently waiting on travis for #6771, then go for launch
674 2015-11-12T20:51:13  <gmaxwell> phantomcircuit: uh, even with libsecp256k1 dsa signature validation run at something like only 9mbit/sec/core (in terms of transaction size).
675 2015-11-12T20:51:19  <morcos> phantomcircuit: i thought you were arguing otherwise?  but yes personally i think the biggest costs are bandwidth and utxo bloat.  my idea was the metric should be something simple like .5 * total_bytes + 1 * output_bytes, but then maybe discounting provably unspendable output bytes
676 2015-11-12T20:52:08  <morcos> jgarzik: did it have a problem or you just rerun it for completeness
677 2015-11-12T20:52:53  <jgarzik> morcos, the latter - I always travis in jgarzik/bitcoin on the merged tree before pushing
678 2015-11-12T20:53:01  <jgarzik> travis results get stale rapidly
679 2015-11-12T20:54:14  <gmaxwell> phantomcircuit: and those figures are without multisig, it's worse with multisig. (like 2 of 3 is like 6mb/s/core)
680 2015-11-12T20:55:38  <morcos> gmaxwell: but you're saying that number is too low?
681 2015-11-12T20:56:31  <morcos> oh, i guess for a degenerate block that has all new sigs...
682 2015-11-12T20:56:51  <morcos> so maybe there is a standard cost metric, and then separate limits like sig op is now for the degenerate cases
683 2015-11-12T20:59:10  *** grieve has joined #bitcoin-core-dev
684 2015-11-12T21:00:46  *** Guyver2 has quit IRC
685 2015-11-12T21:26:36  <phantomcircuit> gmaxwell, hmm that's slower than i was thinking
686 2015-11-12T21:26:51  <phantomcircuit> why was i thinking it was faster?
687 2015-11-12T21:26:58  * phantomcircuit doesn't know
688 2015-11-12T21:28:30  <gmaxwell> phantomcircuit: thats the only reason I mentioned it.
689 2015-11-12T21:28:57  <gmaxwell> Not that I thought it was super important here, but just because you were clearly thinking it was faster than that; you're probably thinking in terms of IBD with signature checking disabled.
690 2015-11-12T21:33:09  <phantomcircuit> gmaxwell, hmm maybe
691 2015-11-12T21:33:21  <phantomcircuit> gmaxwell, i know that we cant do >50mbps for that
692 2015-11-12T21:33:50  <phantomcircuit> possibly with some of the recent performance improvements we can improve on that significantly though
693 2015-11-12T21:39:39  <michagogo> CXX    test/test_test_bitcoin-Checkpoints_tests.o
694 2015-11-12T21:39:46  <michagogo> That capital C seems out of place
695 2015-11-12T21:40:15  <michagogo> So does _DoS just below
696 2015-11-12T21:40:55  <michagogo> -DoS*
697 2015-11-12T21:41:08  <michagogo> (Contrast to -pow which is kept lowercase)
698 2015-11-12T21:41:56  <michagogo> Okay, 0.11.2 built, PR imminent
699 2015-11-12T21:42:10  * michagogo starts the script to build 0.10.;
700 2015-11-12T21:42:13  <michagogo> .4*
701 2015-11-12T21:43:38  <GitHub107> [bitcoin] jgarzik pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/bd629d77edbe...38ed190eefcc
702 2015-11-12T21:43:38  <GitHub107> bitcoin/master 971a4e6 Alex Morcos: Lower default policy limits...
703 2015-11-12T21:43:39  <GitHub107> bitcoin/master 38ed190 Jeff Garzik: Merge #6771 from branch 'lowerLimits' of git://github.com/morcos/bitcoin
704 2015-11-12T21:43:44  <GitHub188> [bitcoin] jgarzik closed pull request #6771: Policy: Lower default limits for tx chains (master...lowerLimits) https://github.com/bitcoin/bitcoin/pull/6771
705 2015-11-12T21:43:47  <jgarzik> morcos, email req :)
706 2015-11-12T21:45:03  <morcos> jgarzik: sent
707 2015-11-12T21:45:53  <morcos> I think I have 2 backed up in moderation queue now
708 2015-11-12T21:46:23  <michagogo> Hm, I wonder if my script will break if there's an outstanding PR
709 2015-11-12T21:47:47  <michagogo> Actually, probably not. The Ruby oneliner that makes the PR is the last line in the script, so even if it fails and exits non-zero, the script is over anyway and there's nothing to abort
710 2015-11-12T21:48:12  <petertodd> gmaxwell: "utxo commitment by instead commiting to the input block and offset" <- that falls into the category of !@#$ why didn't I think of that :)
711 2015-11-12T21:49:42  <michagogo> (set -ex)
712 2015-11-12T21:54:46  *** ParadoxSpiral has quit IRC
713 2015-11-12T21:56:20  <CodeShark> petertodd: that was actually my stumbling block to not fully getting some of the potential advantages of MMR trees
714 2015-11-12T21:57:27  *** treehug88 has quit IRC
715 2015-11-12T21:57:58  *** jgarzik has quit IRC
716 2015-11-12T21:58:16  <CodeShark> in retrospect, it makes a lot more sense to index outputs that way
717 2015-11-12T21:59:43  <CodeShark> at least for existence/nonexistence proofs - obviously for spending we need to reference the transaction in some way
718 2015-11-12T22:20:08  *** MarcoFalke has quit IRC
719 2015-11-12T22:37:14  *** aburan28 has joined #bitcoin-core-dev
720 2015-11-12T23:52:09  *** murr4y has quit IRC
721 2015-11-12T23:59:15  *** murr4y has joined #bitcoin-core-dev