1 2015-11-18T00:00:47  <gmaxwell> I think davec's comment was about the difference between disconnect instantly vs the score?  I don't think the score has much use, but most of the reason we bother disconnecting is to avoid resource wasting attacks, and a little bit of vioation is fine on those terms. ... and handwave handwave, maybe the network is a bit more robust if its a little tolerant there.  I'm skeptical.
  2 2015-11-18T00:01:40  <davec> indeed.  We're extremely strict in btcd.  For example, if you claim to be a protocol version after the nonce was added to ping/pong, but you don't include them, you aren't following the protocol, so you get disconnected, etc
  3 2015-11-18T00:02:02  <gmaxwell> Thats good feedback.
  4 2015-11-18T00:02:04  <davec> unless things have changed recently that I missed, core doesn't care
  5 2015-11-18T00:02:20  <gmaxwell> davec: do you disconnect on unsolicited TX messages (no getdata first?)
  6 2015-11-18T00:02:34  <davec> we used to, but BitcoinJ didn't work, so we allow them now
  7 2015-11-18T00:04:56  <phantomcircuit> gmaxwell, hmm that's kind of nasty the only way to relay a transaction is to place it into the mempool
  8 2015-11-18T00:05:02  <davec> Another thing we added which I don't believe core does is maximums for every protocol message type
  9 2015-11-18T00:05:19  <gmaxwell> phantomcircuit: Thats fine, we'll only allow whitelisted peers to do that.
 10 2015-11-18T00:05:22  <davec> so like a transaction can't possibly be < x or > y so it's disallowed
 11 2015-11-18T00:06:15  <phantomcircuit> gmaxwell, could soft change mempoolexpirey to 1 so at least they're thrown away rapidly
 12 2015-11-18T00:06:27  <phantomcircuit> or not
 13 2015-11-18T00:06:37  <gmaxwell> phantomcircuit: nah, I don't see a reason to.
 14 2015-11-18T00:06:59  <gmaxwell> they'll only come in from whitelisted peers. Don't whitelist (and allow relay for them) stuff you don't want adding things to your mempool.
 15 2015-11-18T00:07:12  <phantomcircuit> ok comments added and that commit removed
 16 2015-11-18T00:07:30  <gmaxwell> phantomcircuit: I understood this completely and was comfortable with it before merging your prior code, FWIW!
 17 2015-11-18T00:09:17  <phantomcircuit> gmaxwell, heh
 18 2015-11-18T00:09:49  <gmaxwell> davec: any idea if bitcoinj still sends unsolicited transactions if its connected to something that has set the relay flag to false?
 19 2015-11-18T00:09:52  <phantomcircuit> yes i tried to "fix" something which after thinking about it more is basically not fixable unless you're willing to push the tx messages instead of inv
 20 2015-11-18T00:10:03  <phantomcircuit> which isn't very nice
 21 2015-11-18T00:14:01  <gmaxwell> phantomcircuit: your first commit is changing too much of the logic, e.g. bypassing nSendsize check.
 22 2015-11-18T00:15:47  <phantomcircuit> gmaxwell, the nSendSize check still runs if we do something that could change it
 23 2015-11-18T00:17:46  <gmaxwell> phantomcircuit: right now. Creates non-local action; I'd prefer you just make the minimial change and then elseif the other reasons, and emit the log entry.
 24 2015-11-18T00:18:07  <phantomcircuit> ok
 25 2015-11-18T00:20:02  <davec> gmaxwell: I don't watch it's dev to know if it has changed, but back when we were getting them to work together the answer is yes.  It always sent unsolicited txns regardless.
 26 2015-11-18T00:20:11  <davec> its*
 27 2015-11-18T00:20:37  <gmaxwell> davec: thanks. Yea, thats all I needed to know (expected the answer too)
 28 2015-11-18T00:20:50  <davec> I believe the reasoning was that if it authored the transaction they are 100% sure you don't have it, so no point in the round trip
 29 2015-11-18T00:21:02  <davec> except it did it for more than that case
 30 2015-11-18T00:21:22  <sipa> that's only true when relaying to just a single peer
 31 2015-11-18T00:21:54  <gmaxwell> also only true if you'd never relay a third party txn and don't care about privacy at all.
 32 2015-11-18T00:22:37  <davec> I agree, but that was the reasoning provided when we asked about it
 33 2015-11-18T00:23:11  <davec> at any rate, we either had to loosen up and allow them or not work with BitcoinJ, so we chose to interop
 34 2015-11-18T00:23:40  <tulip> I didn't think btcd implemented BIP37 anyway?
 35 2015-11-18T00:24:02  <gmaxwell> davec: sure, I'll see if I can convince the new maintership to change.
 36 2015-11-18T00:24:12  <davec> it does
 37 2015-11-18T00:24:40  <davec> We allow it to be disabled via --nobloom, but it fully supports it
 38 2015-11-18T00:24:43  <tulip> ok, crossed wires.
 39 2015-11-18T00:24:45  <dcousens> gmaxwell: to change [it]
 40 2015-11-18T00:24:54  <gmaxwell> Even if it doesn't ... we may start kicking when fRelay is off-- because it's in their interest if they're relaying to us and we're just going to drop it.
 41 2015-11-18T00:25:26  <dcousens> :P
 42 2015-11-18T00:28:27  <davec> oh, if it's of any interest to you, we do d/c on unsolicited blocks and have not seen anything that doesn't follow the inv/getdata model there
 43 2015-11-18T00:29:44  <sipa> ah, that is good to know!
 44 2015-11-18T00:32:03  <tulip> I don't think the relaynode software sends inv for blocks either, I'll have to check though.
 45 2015-11-18T00:33:49  <tulip> that looks right.
 46 2015-11-18T00:35:18  <phantomcircuit> gmaxwell, i can do it as if(inv.type != MSG_BLOCK) {if (fBlocksOnly) Log() else if (!fAlreadyHave && !fImporting && !fReindex) AskFor();}
 47 2015-11-18T00:35:58  <phantomcircuit> i think that's the cleanest change
 48 2015-11-18T00:36:07  <phantomcircuit> without it being slightly insane
 49 2015-11-18T00:36:12  <davec> tulip: hmmm I don't recall seeing any of them hit that, but I can comb back through the logs.  We might have to change that as well.
 50 2015-11-18T00:38:48  <davec> Although it seems pretty wasteful to send a full block only to have it ignored by the other end because they already have it or have already reusted it from another peer and its in-flight, etc
 51 2015-11-18T00:39:10  <gmaxwell> phantomcircuit: I think you can do it clear, take existing code, add the whitelist. Then add an else that checks everything except the whitelist and the blocks only)
 52 2015-11-18T00:39:37  <gmaxwell> tulip: relaynetwork client purposefully does not, but it's local.
 53 2015-11-18T00:39:47  <tulip> davec: the relaynode software is only meant to communicate with local nodes anyway.
 54 2015-11-18T00:40:29  <GitHub144> [bitcoin] luke-jr opened pull request #7047: [WIP] Backports for 0.11.3 (updated to e8df8a5) (0.11...backports-for-0.11.3) https://github.com/bitcoin/bitcoin/pull/7047
 55 2015-11-18T00:42:12  <davec> tulpi: Oh, right.  I thought you were talking about the relay network (which I know does things quite a bit differently)
 56 2015-11-18T00:42:26  <sipa> davec: he is, i think
 57 2015-11-18T00:43:45  <tulip> davec: we're talking about the same thing, just different name
 58 2015-11-18T00:43:53  <gmaxwell> he's talking about the relay network client local gateway.
 59 2015-11-18T00:44:13  <gmaxwell> As far as I know the normal relay network bitcoin nodes do nothing special on the bitcoin p2p protocol.
 60 2015-11-18T00:45:37  <davec> Yeah that makes sense.  I would say that's probably a case for whitelisted nodes accepting them but not allowing random unknown p2p nodes to do it
 61 2015-11-18T00:45:37  <phantomcircuit> gmaxwell, like this?
 62 2015-11-18T00:45:39  <phantomcircuit> https://gist.github.com/pstratem/a8bc031b959cd2225ef1
 63 2015-11-18T00:46:09  <phantomcircuit> that definitely does what we want but... well it's the kind of thing im going to see in six months and go "what idiot did this! oh right me"
 64 2015-11-18T00:47:53  <gmaxwell> phantomcircuit: you can reorg the branches to eliminate the duplication  but be functionally equal then.
 65 2015-11-18T00:48:19  <gmaxwell> also why are you keeping up with the fBlocks only flag. If its only getting used once, no need to make a new flag.
 66 2015-11-18T00:48:30  <davec> or even from local conns
 67 2015-11-18T00:49:37  <phantomcircuit> gmaxwell, without it as a flag the expression is insanely difficult to understand
 68 2015-11-18T00:52:55  <phantomcircuit> gmaxwell, to be pedantic fRelayTxes literally only prevents transactions from being relayed, so if we were to add a new inv type in the future that isn't MSG_BLOCK this logic would be wrong
 69 2015-11-18T00:53:18  <phantomcircuit> it would still do the right thing except for the logging
 70 2015-11-18T00:54:50  <davec> iirc there is already a MSG_FILTERED_BLOCK too
 71 2015-11-18T00:55:50  <phantomcircuit> davec, yeah but that's not relevant for "inv"
 72 2015-11-18T00:55:51  <sipa> davec: only in getdata, not in iv
 73 2015-11-18T00:57:40  <GitHub150> [bitcoin] fanquake opened pull request #7048: [doc][trivial] Update miniupnpc version in build-unix (master...miniupnpc-build-unix) https://github.com/bitcoin/bitcoin/pull/7048
 74 2015-11-18T00:57:51  <phantomcircuit> gmaxwell, oh and using the flag in "inv" prevents calling GetBoolArg in a potentially tight loop
 75 2015-11-18T00:58:17  <gmaxwell> phantomcircuit: K leave the flag but ditch the continue.
 76 2015-11-18T00:58:24  <davec> sipa/phantomcircuit: right.  thanks -- been a while since I looked so forgot the particulars there
 77 2015-11-18T00:58:28  <phantomcircuit> gmaxwell, done
 78 2015-11-18T01:10:54  *** zooko has joined #bitcoin-core-dev
 79 2015-11-18T01:23:07  *** randy-waterhouse has quit IRC
 80 2015-11-18T01:24:27  *** Ylbam has quit IRC
 81 2015-11-18T01:52:17  *** Guest16267 has quit IRC
 82 2015-11-18T01:52:17  *** Guest16267 has joined #bitcoin-core-dev
 83 2015-11-18T01:52:22  *** Guest16267 is now known as s1w
 84 2015-11-18T02:06:37  <dcousens> gmaxwell: maybe I should clarify, my NACK on that was weak
 85 2015-11-18T02:06:56  <dcousens> I just didn't see a need for it except in highly custom setups which IMHO don't fit the bill for most node operators, I may be wrong though
 86 2015-11-18T02:07:01  <gmaxwell> dcousens: ya thats fine, if I sounded too eager there in responding I'm sorry; wasn't communicating well.
 87 2015-11-18T02:08:04  <dcousens> On reviewing the code, its plain enough (less than 40-50 lines), the rest is tests
 88 2015-11-18T02:08:08  <gmaxwell> Does the thought of removing the old mechenism make it more appealing to you?
 89 2015-11-18T02:08:24  <dcousens> I just saw +240 and jumped the gun
 90 2015-11-18T02:09:03  <dcousens> and yes
 91 2015-11-18T02:09:20  <dcousens> If the alterior method is removed, then this is an easy win
 92 2015-11-18T02:09:48  <gmaxwell> whew
 93 2015-11-18T02:11:25  <gmaxwell> Okay, I confess some paranoia here-- I don't want this to be like the rpc hangs issue. Where we pigheadily avoid improvements for some silly footgun which is making people stop using bitcoin core and switch to hosted apis.
 94 2015-11-18T02:11:38  <gmaxwell> no reason to let me ramrod the feature though.
 95 2015-11-18T02:11:48  <gmaxwell> If it goes in it should be the best possible.
 96 2015-11-18T02:14:31  <dcousens> IMHO, 1 day, walletd, bitcoind, indexd.  1 day.  Probably 10 years haha
 97 2015-11-18T02:15:09  <instagibbs> it's a beautiful dream :)
 98 2015-11-18T02:15:40  <dcousens> We need some Shia to make it happen
 99 2015-11-18T02:16:30  <gmaxwell> dcousens: agreed, though we can't halt incremental progress on the way to that vision; ... we'll never get there if we lose all the users who might help support that work. :)
100 2015-11-18T02:16:37  <dcousens> haha, indeed
101 2015-11-18T02:16:41  <instagibbs> Oh, I was thinking "Shia Muslim" and was quite confused...
102 2015-11-18T02:16:50  <dcousens> instagibbs: haha, I meant Laboef
103 2015-11-18T02:17:07  <dcousens> Labouef*
104 2015-11-18T02:18:04  *** belcher has quit IRC
105 2015-11-18T02:18:25  <gmaxwell> #include <culturally_insentive_joke_of_some_form.h>
106 2015-11-18T03:00:26  *** neha has quit IRC
107 2015-11-18T03:01:17  *** neha has joined #bitcoin-core-dev
108 2015-11-18T03:01:26  *** helo has quit IRC
109 2015-11-18T03:02:20  *** helo has joined #bitcoin-core-dev
110 2015-11-18T03:05:53  <morcos> sipa: that "Leaving block file" showing 0's thing.  There is no reason not to fix that right?
111 2015-11-18T03:24:27  <morcos> nm, just did it.. :)
112 2015-11-18T03:24:28  <GitHub85> [bitcoin] morcos opened pull request #7050: Fix debug log message for block files (master...nitfix) https://github.com/bitcoin/bitcoin/pull/7050
113 2015-11-18T03:32:04  *** parazyd has joined #bitcoin-core-dev
114 2015-11-18T03:59:53  <phantomcircuit> gmaxwell, ok i've stopped updating 7046 every ten minutes
115 2015-11-18T04:12:46  <phantomcircuit> gmaxwell, i still really do not like having Get*Arg everywhere
116 2015-11-18T04:12:59  <phantomcircuit> but alas my brain is currently jello so im not doing that tonight
117 2015-11-18T04:14:23  *** Greybits has joined #bitcoin-core-dev
118 2015-11-18T04:23:56  *** jgarzik has joined #bitcoin-core-dev
119 2015-11-18T04:23:56  *** jgarzik has joined #bitcoin-core-dev
120 2015-11-18T04:27:54  *** guest234234 has joined #bitcoin-core-dev
121 2015-11-18T04:32:56  *** guest234234 has quit IRC
122 2015-11-18T04:42:56  *** d_t has quit IRC
123 2015-11-18T05:12:34  *** Greybits has quit IRC
124 2015-11-18T05:39:58  *** dcousens has quit IRC
125 2015-11-18T05:40:15  *** dcousens has joined #bitcoin-core-dev
126 2015-11-18T05:40:44  *** d_t has joined #bitcoin-core-dev
127 2015-11-18T05:51:04  *** tulip has quit IRC
128 2015-11-18T05:53:42  *** randy-waterhouse has joined #bitcoin-core-dev
129 2015-11-18T06:07:59  *** jtimon has quit IRC
130 2015-11-18T06:12:25  *** tulip has joined #bitcoin-core-dev
131 2015-11-18T06:22:14  *** bsm117532 has quit IRC
132 2015-11-18T06:23:39  *** bsm117532 has joined #bitcoin-core-dev
133 2015-11-18T07:14:06  *** Ylbam has joined #bitcoin-core-dev
134 2015-11-18T07:28:05  *** dcousens has quit IRC
135 2015-11-18T07:29:04  *** baldur has quit IRC
136 2015-11-18T07:29:27  *** baldur has joined #bitcoin-core-dev
137 2015-11-18T08:10:37  <GitHub23> [bitcoin] laanwj opened pull request #7051: ui: Add "Copy raw transaction data" to transaction list context menu (master...2015_11_transaction_hex2) https://github.com/bitcoin/bitcoin/pull/7051
138 2015-11-18T08:11:30  *** jcorgan has quit IRC
139 2015-11-18T08:11:42  *** jcorgan has joined #bitcoin-core-dev
140 2015-11-18T08:11:42  *** jcorgan has joined #bitcoin-core-dev
141 2015-11-18T08:20:28  <GitHub147> [bitcoin] gmaxwell pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/e8df8a5077df...f3ea48ad8b85
142 2015-11-18T08:20:28  <GitHub147> bitcoin/master e855b01 Alex Morcos: Fix debug log message for block files
143 2015-11-18T08:20:29  <GitHub147> bitcoin/master f3ea48a Gregory Maxwell: Merge pull request #7050...
144 2015-11-18T08:20:38  <GitHub197> [bitcoin] gmaxwell closed pull request #7050: Fix debug log message for block files (master...nitfix) https://github.com/bitcoin/bitcoin/pull/7050
145 2015-11-18T08:50:17  <GitHub3> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/f3ea48ad8b85...7f8e90da335e
146 2015-11-18T08:50:17  <GitHub3> bitcoin/master 013a364 MarcoFalke: [contrib] Delete test-patches
147 2015-11-18T08:50:18  <GitHub3> bitcoin/master 7f8e90d Wladimir J. van der Laan: Merge pull request #7030...
148 2015-11-18T08:50:26  <GitHub125> [bitcoin] laanwj closed pull request #7030: [contrib] Delete test-patches (master...MarcoFalke-2015-contribC) https://github.com/bitcoin/bitcoin/pull/7030
149 2015-11-18T08:53:28  *** ParadoxSpiral has joined #bitcoin-core-dev
150 2015-11-18T08:58:05  *** Guyver2 has joined #bitcoin-core-dev
151 2015-11-18T09:27:28  *** d_t has quit IRC
152 2015-11-18T09:40:12  *** cjcj has joined #bitcoin-core-dev
153 2015-11-18T09:45:44  *** rubensayshi has joined #bitcoin-core-dev
154 2015-11-18T10:08:49  *** jtimon has joined #bitcoin-core-dev
155 2015-11-18T10:26:42  *** cjcj has quit IRC
156 2015-11-18T10:31:41  *** randy-waterhouse has quit IRC
157 2015-11-18T11:15:48  <GitHub184> [bitcoin] MarcoFalke opened pull request #7052: [qa] python-bitcoinrpc is no longer a subtree (master...MarcoFalke-2015-qaSubtree) https://github.com/bitcoin/bitcoin/pull/7052
158 2015-11-18T11:16:06  *** MarcoFalke has joined #bitcoin-core-dev
159 2015-11-18T11:51:42  <GitHub111> [bitcoin] jtimon opened pull request #7053: Globals: Remove a bunch of Params() from main.cpp before 0.12 (master...globals-chainparams-main) https://github.com/bitcoin/bitcoin/pull/7053
160 2015-11-18T11:54:00  <GitHub25> [bitcoin] jtimon closed pull request #5970: DEPENDENT: Globals: Avoid calling Params() (master...chainparams_cleanup) https://github.com/bitcoin/bitcoin/pull/5970
161 2015-11-18T12:33:46  *** guest234234 has joined #bitcoin-core-dev
162 2015-11-18T12:47:35  *** rav3n_pl has quit IRC
163 2015-11-18T13:21:38  *** MarcoFalke has quit IRC
164 2015-11-18T13:23:14  *** dcousens has joined #bitcoin-core-dev
165 2015-11-18T13:23:57  *** sdaftuar has joined #bitcoin-core-dev
166 2015-11-18T13:28:47  <GitHub151> [bitcoin] jonasschnelli pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/7f8e90da335e...03403d8c0f3b
167 2015-11-18T13:28:47  <GitHub151> bitcoin/master 513686d MarcoFalke: [qt] Use maxTxFee instead of 10000000
168 2015-11-18T13:28:48  <GitHub151> bitcoin/master 03403d8 Jonas Schnelli: Merge pull request #6951...
169 2015-11-18T13:28:52  <GitHub190> [bitcoin] jonasschnelli closed pull request #6951: [qt] Use maxTxFee instead of 10000000 (master...MarcoFalke-2015-qtMaxFee) https://github.com/bitcoin/bitcoin/pull/6951
170 2015-11-18T13:46:23  *** zooko has quit IRC
171 2015-11-18T13:54:07  <dcousens> out of interest, why username:password vs just token for the RPC?
172 2015-11-18T13:54:13  <dcousens> they're both sent in the clear right?
173 2015-11-18T13:56:25  *** Sanjay has joined #bitcoin-core-dev
174 2015-11-18T14:01:00  <dcousens> ping gmaxwell: thoughts on https://github.com/bitcoin/bitcoin/pull/7044#issuecomment-157719965
175 2015-11-18T14:01:45  *** zooko has joined #bitcoin-core-dev
176 2015-11-18T14:02:17  *** MarcoFalke has joined #bitcoin-core-dev
177 2015-11-18T14:06:40  <sipa> dcousens: if it was just a token, you wouldn't know which one to try
178 2015-11-18T14:06:53  <sipa> dcousens: which is needed if you want strengthening
179 2015-11-18T14:07:28  *** MarcoFalke has quit IRC
180 2015-11-18T14:15:03  <jonasschnelli> dcousens: The hashing improves the attack vectors a little bit (tiny, i agree). Isn't the hashing is a little step towards a http-auth-digest likes authentication?
181 2015-11-18T14:20:19  *** guest234234 has quit IRC
182 2015-11-18T14:22:18  <gmaxwell> dcousens: username is useful because you can log it. also it fits into the http auth scheme.
183 2015-11-18T14:22:31  <gmaxwell> of course, you could set all your usernames to x and just use passwords.
184 2015-11-18T14:22:54  <gmaxwell> oh yea, sipas point too.
185 2015-11-18T14:30:57  <jgarzik> jonasschnelli, http digest auth has been brought up quite a few times...
186 2015-11-18T14:33:23  <jonasschnelli> jgarzik: Yes. I once looked at a possible implementation and decided that it's not worth doing it.
187 2015-11-18T14:34:46  <gmaxwell> It's also not compatible with a surprising amount of stuff.
188 2015-11-18T14:34:56  <jgarzik> the question really comes down to SSL, IMO.  We are sending plaintext passwords.  Is that a problem on localhost?  no.  Is that a problem on WAN?  Yes.  Should you be sending unencrypted RPC over WAN?  No, use stunnel.   So now you're back to not needing digest.
189 2015-11-18T14:35:47  <gmaxwell> right.
190 2015-11-18T14:40:00  <jonasschnelli> agreed
191 2015-11-18T14:40:14  *** jgarzik has quit IRC
192 2015-11-18T14:41:57  <wumpus> right: if you want secure RPC, tunnel it over something secure. Digest auth would be pretty pointless. Yes, you protect the auth, but your walletpassphrase will still go over the line unencrypted.
193 2015-11-18T14:43:41  <wumpus> a challenge/response authentication system with the key storage/signer would be nice, as it'd also solve the 'walletpassphrase leaks into process memory' problem, but then again, you'd still be leaking actual commands over the line, which can (at the least) hurt privacy
194 2015-11-18T14:43:59  <jonasschnelli> Right. #7044 (multiple RPC users) is interesting and it would allow to make permissions levels
195 2015-11-18T14:44:02  <wumpus> so no, I think digest auth (or seemingly more secure) RPC auth is optimziing the wrong thing
196 2015-11-18T14:44:44  <wumpus> but we want to get rid of the account stuff; what would be the use of multiple RPC accounts?
197 2015-11-18T14:45:45  <wumpus> if you have multiple, completely separate wallets it could make sense I guess
198 2015-11-18T14:46:28  <dcousens> gmaxwell: sure, but he could have been showing you a curl script with the rpcpassword in it
199 2015-11-18T14:46:29  <jonasschnelli> companies that run services based on bitcoind could have multiple users, one that only accessed public data (like blocks / transaction), another user could have permissions to change things, like ban, etc.
200 2015-11-18T14:46:37  <dcousens> the use case just isn't relevant imho, since, I doubt he minded lol
201 2015-11-18T14:47:08  <gmaxwell> wumpus: not for seperate wallets, but access list management purposes; e.g. you with to withdraw one host, and add a new one. Otherwise you must disrupt everything tfor a flag-day change, and take an outage.
202 2015-11-18T14:47:08  <wumpus> I don't want a complex, complete authentication and permissions system in bitcoin core
203 2015-11-18T14:47:12  <wumpus> we're not buliding an OS here
204 2015-11-18T14:47:29  <wumpus> it has to stop somewhere with importing all kind of ancillary concens
205 2015-11-18T14:47:44  <dcousens> gmaxwell: I'm just not sure if its worth adding, since it'll give people impression that RPC is safer across a non-localized env?  Weren't we discouraging that?
206 2015-11-18T14:47:45  <jonasschnelli> Right. This could make things really complicate.
207 2015-11-18T14:47:47  <wumpus> do you want access control? build an RPC proxy and have that check user permissions
208 2015-11-18T14:48:02  <gmaxwell> Sure, but this is basic access management that any other network facing application provides (and basically any web api provides)
209 2015-11-18T14:48:08  <wumpus> sigh...
210 2015-11-18T14:48:08  <jonasschnelli> Yes. Thats a better approach.
211 2015-11-18T14:48:09  *** wumpus has left #bitcoin-core-dev
212 2015-11-18T14:48:09  <dcousens> (crap, just saw all the text above, notifications daemon died, reading now)
213 2015-11-18T14:48:40  <dcousens> gmaxwell: isn't the point that this doesn't face the network?
214 2015-11-18T14:48:55  <dcousens> All the docs have warnings to avoid that
215 2015-11-18T14:49:10  <gmaxwell> It most certantly does-- not public networks, but even localhost is a network. Stunnel to it is network access.
216 2015-11-18T14:49:19  <gmaxwell> If it's not facing things it cannot be used at all.
217 2015-11-18T14:49:31  <dcousens> Sure,  in which case, your on the machine, and, unless its a multi-user env, the hashing is pointless?
218 2015-11-18T14:49:41  <gmaxwell> The secondary use, not enabled yet, is so that logging can also record which account did it, which is useful for debugging and forensics.
219 2015-11-18T14:49:43  <dcousens> by multi-user env, I mean, the syystem itself, not bitcoind
220 2015-11-18T14:50:41  <gmaxwell> The fact that the credentials are hased is a basic security practice that prevents leaking them when handling the credentials themselves; it's also a checklist requirement on many security standards (and not an unreasonable one)
221 2015-11-18T14:51:30  <sipa> a single sha256 would suffice for that, though
222 2015-11-18T14:52:23  <gmaxwell> It would. and if the credential is properly machine generated, then nothing more is actually useful.
223 2015-11-18T14:53:33  <dcousens> gmaxwell: for logging, you could even just use hash(token), not untraceable
224 2015-11-18T14:53:57  <dcousens> but even then, fine, user:pass still if you want make that easier
225 2015-11-18T14:54:08  <gmaxwell> dcousens: doesn't really fit how people use things, the http auth already sends a user/password in any case.
226 2015-11-18T14:54:21  *** Greyboy has joined #bitcoin-core-dev
227 2015-11-18T14:54:37  *** MarcoFalke has joined #bitcoin-core-dev
228 2015-11-18T14:54:56  <dcousens> gmaxwell: it just seems to me like the hashed password is implying a different security model to what has been advised up until now
229 2015-11-18T14:55:26  <dcousens> and, aside from the auditing aspect, the username:password scheme is also misleading since its entirely in the clear
230 2015-11-18T14:55:43  <dcousens> so, I guess, are we changing the security model? or keeping it the same
231 2015-11-18T14:56:02  <gmaxwell> dcousens: almost 40 years of system security says otherwise.
232 2015-11-18T14:56:28  <dcousens> gmaxwell: I have *no* issue with hashing, if we're protecting against local breach
233 2015-11-18T14:56:34  *** Greyboy has quit IRC
234 2015-11-18T14:56:35  *** Greyboy has joined #bitcoin-core-dev
235 2015-11-18T14:56:42  <dcousens> but, AFAIK, we haven't protected that to date
236 2015-11-18T14:56:43  <gmaxwell> Seriously, people get _fired_ for keeping unhashed password databases. When a breach results from one it is an embarssment that results in litigation.
237 2015-11-18T14:57:57  <GitHub96> [bitcoin] ptschip opened pull request #7055: MAX_PROTOCOL_MESSAGE_LENGTH description (master...maxmessage) https://github.com/bitcoin/bitcoin/pull/7055
238 2015-11-18T14:58:37  <dcousens> gmaxwell: does that make sense? I'm asking if we're wanting to be more permissive in telling users "local breach isn't catastrophic"
239 2015-11-18T14:58:39  *** wumpus has joined #bitcoin-core-dev
240 2015-11-18T14:58:57  <dcousens> To date, that isn't the case
241 2015-11-18T15:00:53  <Greyboy> I'm late to this convo, but like security. may i ask what passwords you are discussing?
242 2015-11-18T15:01:16  * instagibbs reading backlog
243 2015-11-18T15:03:19  <instagibbs> Greyboy, https://github.com/bitcoin/bitcoin/pull/7044
244 2015-11-18T15:04:31  <Greyboy> Thank you.
245 2015-11-18T15:04:46  <dcousens> instagibbs: good PR :), just noticed it changes the security model, so, just wanted that to be clear, because it'll likely mean we'll need to update docs which currently mention "dont ever expose the RPC", and, "dont use SSL", etc
246 2015-11-18T15:04:57  <instagibbs> yeah I'm thinking through all this too
247 2015-11-18T15:05:32  <dcousens> and everything being in the clear made that pretty obvious, whereas, now this seems half-half, in what was previously a blatantly unprotected env
248 2015-11-18T15:10:50  <dcousens> wumpus: lol: https://en.bitcoin.it/wiki/Enabling_SSL_on_original_client_daemon
249 2015-11-18T15:10:55  <Greyboy> instagibbs, i know i'm new and my opinion is relatively worthless, but i think it's a good idea.  my question is what percent of people who run bitcoin-core actually need this solution?
250 2015-11-18T15:11:28  <wumpus> dcousens: hah - the 0.12 release notes specify how to use stunnel as well as transparant proxying, would be more useful to add that
251 2015-11-18T15:12:14  <sipa> Greyboy: or what percentage doesn't run it because it misses features like this? :)
252 2015-11-18T15:12:29  <instagibbs> dcousens, I understand and agree with the concerns, but if we want some basic audit logging, I don't think we should avoid basic improvements
253 2015-11-18T15:12:30  <dcousens> or more recently
254 2015-11-18T15:12:31  <dcousens> https://bitcoin.stackexchange.com/questions/39117/why-is-json-rpc-over-ssl-strongly-discouraged
255 2015-11-18T15:12:33  <instagibbs> keep the warnings, in general
256 2015-11-18T15:12:44  <dcousens> though, thats a bit circular
257 2015-11-18T15:12:48  <instagibbs> btw where are the warnings today
258 2015-11-18T15:12:55  *** zooko has quit IRC
259 2015-11-18T15:13:11  <dcousens> instagibbs: my main question doesn't stand in the way of doing logging
260 2015-11-18T15:13:34  <dcousens> instagibbs: I'm just talking about the hashing, I'm ACK on everything else, even the hashing, just, clarifying that that is intended
261 2015-11-18T15:13:36  <Greyboy> On a different note, I was wondering what bitcoin-core does that requires it to connect to dyndns?
262 2015-11-18T15:13:48  <instagibbs> dcousens, would you feel better if it was a single hash :P
263 2015-11-18T15:14:45  <gmaxwell> Well that would also drop any worry about where the pbkdf2 comes from.
264 2015-11-18T15:14:52  <instagibbs> ^
265 2015-11-18T15:15:20  <instagibbs> warn people the same things we warned before (strong random keys only)
266 2015-11-18T15:15:25  <gmaxwell> I don't consider the iterated hash actually important, it only potentially saves misusers from themselves slightly.
267 2015-11-18T15:16:59  <dcousens> instagibbs: sure, I mean, the key-stretching seemed a bit unnecessary if we encourage what is already encouraged, strong random keys etc
268 2015-11-18T15:17:17  <sipa> Greyboy: what?
269 2015-11-18T15:17:39  <Greyboy> sipa, i was wondering why bitcoin-core tries to connect to dyndns and alerts my IDS
270 2015-11-18T15:17:44  <instagibbs> I was going for "why not", but if it makes it more palatable, and strictly superior to today, and drops openssl dep....
271 2015-11-18T15:17:52  <sipa> Greyboy: what version?
272 2015-11-18T15:18:27  <dcousens> anyway, its late here, hope I didn't stir up to much of a hornet nest haha.  Just, recognizing this was a changed expectation.  maybe its a better one,  hard to tell
273 2015-11-18T15:18:35  <Greyboy> I don't have it on this machine or i would check, but i've seen this behaviour in many versions including recent ones.
274 2015-11-18T15:18:45  <instagibbs> dcousens, no this is really appreciated. My first Core PR getting tons of attention from smart folks is nice
275 2015-11-18T15:21:19  <dcousens> instagibbs: jgarzik summed it up nicely
276 2015-11-18T15:21:20  <dcousens> "the question really comes down to SSL, IMO.  We are sending plaintext passwords.  Is that a problem on localhost?  no.  Is that a problem on WAN?  Yes. Should you be sending unencrypted RPC over WAN?  No, use stunnel.   So now you're back to not needing digest.
277 2015-11-18T15:21:24  <dcousens> "
278 2015-11-18T15:22:12  <instagibbs> I get the concern. Single hash at least gets the "peek over shoulder" security model.
279 2015-11-18T15:22:24  <sipa> and sha256 we already have
280 2015-11-18T15:22:39  <instagibbs> *the village rejoices*
281 2015-11-18T15:22:48  <gmaxwell> dcousens: the digest thing is specifically referring to http digest auth, orthorgonal issue.
282 2015-11-18T15:22:57  <dcousens> yeah just realised, was re-writing that
283 2015-11-18T15:23:17  <dcousens> the other issue was, well, read up about 10 lines, night all
284 2015-11-18T15:23:49  *** dcousens has quit IRC
285 2015-11-18T15:24:29  *** dcousens has joined #bitcoin-core-dev
286 2015-11-18T15:26:51  <instagibbs> dcousens, cheers
287 2015-11-18T15:29:46  <sipa> if we want PBKDF2 in there, it's trivial to implement, but if we're going with just hashed passwords... even easier\
288 2015-11-18T15:34:08  *** Guyver2 has quit IRC
289 2015-11-18T15:35:12  <wumpus> right - I think we don't want something with a lot of key stretching, as it'd increase the latency for RPC calls
290 2015-11-18T15:35:46  <wumpus> but hashing the passwords in the config file is a great idea, not sure why we've not been doing that yet
291 2015-11-18T15:36:14  <sipa> we can even do salting by using HMAC-SHA256, but without stretching
292 2015-11-18T15:36:24  <sipa> though i don't think that adds much
293 2015-11-18T15:36:40  <sipa> except "certification"
294 2015-11-18T15:36:50  <gmaxwell> if the key is randomly generated as it should be, the salting does nothing.
295 2015-11-18T15:37:03  <gmaxwell> but yea, it might fail some checkbox.
296 2015-11-18T15:37:22  <wumpus> salting is pretty standard for password checks, yes
297 2015-11-18T15:37:23  <gmaxwell> I suggest the format be such that we could add different types in the future and if we find out its a problem we can just make a new password type.
298 2015-11-18T15:37:33  <gmaxwell> or keep the salt, I don't care. :)
299 2015-11-18T15:37:35  * wumpus wonders what Tor uses for torcontrolpassword
300 2015-11-18T15:37:56  <instagibbs> I was just going to drop salt, but either way is fine
301 2015-11-18T15:38:14  <instagibbs> we're assuming people aren't foot-gunning bad passwords already
302 2015-11-18T15:38:55  <gmaxwell> only reason to keep it is certificational, which is an issue here. But it might be answerable with "the salt is _in_ the password" :)
303 2015-11-18T15:39:22  <instagibbs> Excuse my noobishness, what is "certificational" about a salt
304 2015-11-18T15:39:42  <sipa> instagibbs: rainbow tables!
305 2015-11-18T15:39:49  <wumpus> tor apparently does salting, hashing 'bla' repeatedly gives  16:DD2D9F35664C69D7604862845D52D4238AD5E702E8C9DF771108459C00 16:EC737D2A880C18996066882BAE3BA73E32543DA0F74D308572288E38F0 16:35D59809F0AA181160F34D05D6504F89958E2A1D393A29959937C577A0
306 2015-11-18T15:40:07  <gmaxwell> thats a +1 for salt. :P
307 2015-11-18T15:40:14  <instagibbs> damnit
308 2015-11-18T15:40:20  * instagibbs re-adds
309 2015-11-18T15:40:22  <sipa> http://www.antitree.com/onion-porn-hashedcontrolpassword/
310 2015-11-18T15:40:22  <instagibbs> :P
311 2015-11-18T15:40:35  <gmaxwell> instagibbs: the certificational part is that there are security checklists that ask if you're using salted hashed passwords.
312 2015-11-18T15:40:46  <instagibbs> ohhhh like actualy certificates. Got it.
313 2015-11-18T15:41:14  <wumpus> ah, thanks sipa, so 16 is the algorithm identifier, then follows the salt and password concatenated
314 2015-11-18T15:41:23  <sipa> wumpus: nope, the 16 means it's encoded in hex :)
315 2015-11-18T15:41:32  <sipa> seriously
316 2015-11-18T15:41:44  <sipa> it's also the only thing supported
317 2015-11-18T15:41:54  <wumpus> huh. ok
318 2015-11-18T15:42:00  <sipa> but ehm... interoperability with tor would be nice imho
319 2015-11-18T15:42:14  <wumpus> I'm confused with $3$ or /etc/shadow etc
320 2015-11-18T15:42:22  <sipa> yeah
321 2015-11-18T15:42:47  <gmaxwell> oh interesting actually thinking of using the same scheme as tor? is it complex to implement?
322 2015-11-18T15:43:13  <sipa> 47 lines of python
323 2015-11-18T15:43:15  <wumpus> we can just steal there code I guess :p although they may lean on OpenSSL as well
324 2015-11-18T15:43:23  <Greyboy> *their
325 2015-11-18T15:43:26  <sipa> it's just sha1
326 2015-11-18T15:43:29  <sipa> iterated
327 2015-11-18T15:45:14  <GitHub43> [bitcoin] ptschip closed pull request #7055: MAX_PROTOCOL_MESSAGE_LENGTH description (master...maxmessage) https://github.com/bitcoin/bitcoin/pull/7055
328 2015-11-18T15:45:29  <wumpus> there's one difference though: tor only authenticates on opening the connection, bitcoin http authenticates for every request. So they can use more key stretching without creating command latency issues.
329 2015-11-18T15:45:35  <gmaxwell> well we already have a sha1 dependency.
330 2015-11-18T15:45:58  <instagibbs> hmm
331 2015-11-18T15:46:07  <instagibbs> link to related tor source?
332 2015-11-18T15:46:52  *** Guyver2 has joined #bitcoin-core-dev
333 2015-11-18T15:49:00  <wumpus> https://github.com/torproject/tor/blob/master/src/or/main.c#L3210
334 2015-11-18T15:50:07  <wumpus> https://github.com/torproject/tor/blob/master/src/common/crypto_s2k.c
335 2015-11-18T15:50:36  <wumpus> but the python script sipa linked is more easily readable
336 2015-11-18T15:50:53  *** Greyboy has quit IRC
337 2015-11-18T15:55:12  <sipa> oh, it has a 1-byte iteration count which uses exponential notation
338 2015-11-18T15:55:21  <sipa> 4 bits of mantissa, and 4 bits of exponent
339 2015-11-18T15:55:23  <wumpus> yes, the 'indicator'
340 2015-11-18T15:55:40  <sipa> up to max 65M iterations
341 2015-11-18T15:56:02  <sipa> and min 1024
342 2015-11-18T15:56:16  <gmaxwell> meh. thats kinda sad.
343 2015-11-18T15:56:34  <gmaxwell> Again I remind: so long as the format has a type field we could add other types in the future if needed.
344 2015-11-18T15:57:27  <sipa> the tor thing actually doesn't (though it starts with an undefined "16:" prefix, which could be redefined into being a type, if necessary, i guess)
345 2015-11-18T15:59:04  <wumpus> normally iteration is done by feeding the previous hash back into the hasher, not in this case
346 2015-11-18T15:59:25  <wumpus> "hash the salty password as many times as the length of the password divides into the count value"
347 2015-11-18T15:59:47  <wumpus> so it defines (roughly) how many bytes will be passed into the hash function
348 2015-11-18T16:00:19  <wumpus> yes, the 16: in case of tor is kind of a red herring, I think anyone would have expected it to be an algorithm id
349 2015-11-18T16:01:38  *** MarcoFalke has quit IRC
350 2015-11-18T16:12:05  *** MarcoFalke has joined #bitcoin-core-dev
351 2015-11-18T16:13:36  <gmaxwell> wumpus: what are your thoughts on trying to make the final push to get openssl out of bitcoind?  I think all code needed for it has already been written, then abandoned at varrious points in the past. The two big blockers were rpcssl and ecc, and both are resolved now.
352 2015-11-18T16:14:07  <wumpus> we should definitely do it, but for 0.13 imo
353 2015-11-18T16:14:24  <gmaxwell> OKAY, yes, that seems more reasonable I guess.
354 2015-11-18T16:14:38  <sipa> agree
355 2015-11-18T16:14:46  <wumpus> the deadline for 0.12 is getting closer and I'm sure there is high-prio stuff that still needs to get in
356 2015-11-18T16:14:56  <gmaxwell> RNG replacement isn't a small project or one we'd want to do poorly. I think if not for that we could do the rest.
357 2015-11-18T16:14:58  <wumpus> but as soon as 0.12 is branched off let's kill openssl
358 2015-11-18T16:15:26  <sipa> i'll try to find the time after branching to turn the fortuna/entropy gathering code into an external library
359 2015-11-18T16:15:51  <wumpus> I also have that on my list sipa, but feel free to do it :)
360 2015-11-18T16:15:55  <gmaxwell> I am so tired of spurrious valgrind errors on various systems because after all these years openssl can't be bothered to not avoid undefined behavior in integrity critical RNG code. :-/
361 2015-11-18T16:16:40  <wumpus> yes - RNG is kind of critical in our case
362 2015-11-18T16:18:21  <wumpus> foremost importance is that if no good entropy source can be found on a system, it's better to fail than continue with a lousy one, many wallets made that mistake
363 2015-11-18T16:19:01  <gmaxwell> yes, if you have no good entropy and know it. Irritatingly this failure mode is not as common as it should be. :(
364 2015-11-18T16:19:04  <sipa> "Oh, you're using Internet Explorer 3? I'll just go use { return 4; // decided by a fair dice roll } as entry!
365 2015-11-18T16:19:25  <wumpus> exactly :D
366 2015-11-18T16:19:36  <sipa> s/ry/ropy/
367 2015-11-18T16:19:46  <gmaxwell> openssl and libressl both fail that test though, even if they can't get any strong OS rng input, they keep on trucking.
368 2015-11-18T16:19:49  <wumpus> gmaxwell: well I mean if the OS APIs somehow fail
369 2015-11-18T16:21:20  <gmaxwell> wumpus: yea. the design Sipa had done before was also targeting improving the strength when they silently fail (happened somewhat recently in several OSes, :( ).. but certantly if the strong input fails it should full stop.
370 2015-11-18T16:21:26  <wumpus> sure, it may happen that the OS happily returns values which happen to be perfectly predictable because it's only based on the timestamp of VM startup and timings on synthethic devices
371 2015-11-18T16:22:42  <gmaxwell> for the GUI you can always pop up a initial start friendly game of 2048 to record user timings from; but alas, the enviroments most likely to be entropy starved are not the ones with a UI.
372 2015-11-18T16:22:53  <wumpus> lol!
373 2015-11-18T16:23:26  <wumpus> blockchain 2048
374 2015-11-18T16:24:12  <wumpus> "combine the blocks to double the block reward"
375 2015-11-18T16:24:27  <sipa> freicoin can use that 2048 version whose pawn disappear after a certain time...
376 2015-11-18T16:24:51  <wumpus> hehehe
377 2015-11-18T16:26:08  <gmaxwell> I've wondered what simple games extract the most entropy from users in the least time.
378 2015-11-18T16:26:34  <sipa> gmaxwell: things with many keypresses, i'm sure :)
379 2015-11-18T16:27:36  <gmaxwell> maybe a flashcard typing game.
380 2015-11-18T16:28:07  <gmaxwell> oh that shows captcha flashcards so you make mistakes. :P
381 2015-11-18T16:32:35  <wumpus> it's a nice change from the usual 'bash on the keyboard and move the mouse around like you have parkinson disease'
382 2015-11-18T16:33:09  <sipa> LOL
383 2015-11-18T16:37:03  <gmaxwell> wumpus: for a procedure at work I want to just instruct people to roll dice. actual dice rolls produce less entropy than the timing from typing them in, but its fun to do.
384 2015-11-18T16:40:28  <wumpus> or combine the ideas, add a game that involves throwing dice
385 2015-11-18T16:43:03  <wumpus> shuffling a pack of cards may also work, but it's so much work it enter it all :)
386 2015-11-18T16:47:43  <wumpus> so sneaky, would have been so easy to miss "This should not be called with the 2 historical coinbase duplicate pairs because the new coins are marked fresh, and in the event the duplicate coinbase was spent before a flush, the now pruned coins would not properly overwrite the first coinbase of the pair. Simultaneous modifications are not allowed."
387 2015-11-18T16:48:01  <wumpus> (re: #6932)
388 2015-11-18T16:50:14  *** treehug88 has joined #bitcoin-core-dev
389 2015-11-18T16:51:09  *** zooko has joined #bitcoin-core-dev
390 2015-11-18T16:53:52  <GitHub151> [bitcoin] laanwj pushed 4 new commits to master: https://github.com/bitcoin/bitcoin/compare/03403d8c0f3b...73fa5e604356
391 2015-11-18T16:53:53  <GitHub151> bitcoin/master 14470f9 Alex Morcos: ModifyNewCoins saves database lookups...
392 2015-11-18T16:53:53  <GitHub151> bitcoin/master 03c8282 Alex Morcos: Make CCoinsViewTest behave like CCoinsViewDB
393 2015-11-18T16:53:54  <GitHub151> bitcoin/master 1cf3dd8 Alex Morcos: Add unit test for UpdateCoins
394 2015-11-18T16:53:57  <GitHub31> [bitcoin] laanwj closed pull request #6932: ModifyNewCoins saves database lookups (master...newCoinsThinAir) https://github.com/bitcoin/bitcoin/pull/6932
395 2015-11-18T16:55:28  <gmaxwell> \O/
396 2015-11-18T16:56:36  <sipa> someone should put a logprint in CCoinsViewDb on read and write, and reindex with max size dbcache
397 2015-11-18T16:56:57  <sipa> there should only be 1 db operation per block left (the coinsbase overwrite)
398 2015-11-18T16:57:49  <gmaxwell> save the logging until we've gotten that one fixed too? :)
399 2015-11-18T16:58:43  <sipa> that needs #5967
400 2015-11-18T17:01:59  *** davec has quit IRC
401 2015-11-18T17:06:42  *** davec has joined #bitcoin-core-dev
402 2015-11-18T17:07:19  *** MarcoFalke has quit IRC
403 2015-11-18T17:22:22  <morcos> here you go:
404 2015-11-18T17:22:27  <GitHub106> [bitcoin] morcos opened pull request #7056: Save last db read (master...saveLastDBRead) https://github.com/bitcoin/bitcoin/pull/7056
405 2015-11-18T17:27:37  <gmaxwell> bam. theres to all the people shed painting over database engines, so long as you never stop your node the database no longer matters. :P
406 2015-11-18T17:28:22  <sipa> gmaxwell: and have infinite dbcache
407 2015-11-18T17:28:46  <gmaxwell> doesn't everyone have 128GB ram?
408 2015-11-18T17:28:49  *** d_t has joined #bitcoin-core-dev
409 2015-11-18T17:30:04  <wumpus> gmaxwell: is the UTXO set that big now? :(
410 2015-11-18T17:30:35  <sipa> wumpus: he runs 20 bitcoind in parallel, i'm sure
411 2015-11-18T17:30:38  <gmaxwell> wumpus: it's about 6GB in ram right now. Less with sipa's sold script vector replacement thing.
412 2015-11-18T17:30:40  <wumpus> hehhe
413 2015-11-18T17:31:16  <gmaxwell> right now I have a host importing one of the satoshi dice addresses into the wallet...
414 2015-11-18T17:31:44  <gmaxwell> this might not have been a good idea; and .. I picked one with impossibly low odds too.
415 2015-11-18T17:31:55  *** rubensayshi has quit IRC
416 2015-11-18T17:32:21  <gmaxwell> (does anyone have an address to sugget importing that has a lot of txn but not ALL the transactions?)
417 2015-11-18T17:34:48  <btcdrak> gmaxwell: you can find a bunch here https://www.walletexplorer.com
418 2015-11-18T18:00:14  *** Greyboy has joined #bitcoin-core-dev
419 2015-11-18T18:07:50  <davec> gmaxwell: if you still need one I typically use 14WCrzmqVCD7Thf79vutWTB6y1hUc8pP19 for such stress tests since it has ~62k
420 2015-11-18T18:09:21  <gmaxwell> davec: thanks.
421 2015-11-18T18:13:26  *** paveljanik has joined #bitcoin-core-dev
422 2015-11-18T18:13:26  *** paveljanik has joined #bitcoin-core-dev
423 2015-11-18T18:16:46  <gmaxwell> hm. this might be a bit suboptimal; reindex with dbcach=8192. leave sitting for a day. start a wallet rescan and decide that I'd rather not, kill -9 the process.  start back up with a deleted wallet.. chainstate is at height=174109
424 2015-11-18T18:27:50  <wumpus> need more db flushes :)
425 2015-11-18T18:28:16  <sipa> if we had flushes that didn't wipe everything, we could do them more frequently
426 2015-11-18T18:28:49  <wumpus> I also once lost almost a whole sync in a kill -9, that's the cost of using -dbcache=8000 I guess. On the other hand it's great to be able to do a sync with almost no i/o.
427 2015-11-18T18:30:23  <wumpus> with the standard dbcache setting it's not an issue
428 2015-11-18T18:30:44  <gmaxwell> heh, yea, it wasn't a complaint so much as a "Oh, yea. thats ... not technically surprising."
429 2015-11-18T18:32:34  <wumpus> does it still do a flush after IsInitialBlockDownload is complete?
430 2015-11-18T18:33:10  <gmaxwell> I think it does, in my case I'd been reindexing and it doesn't on reindex complete AFAIK.
431 2015-11-18T18:33:35  <wumpus> hmm maybe it should
432 2015-11-18T18:42:14  <gmaxwell> I think so.
433 2015-11-18T18:43:04  <wumpus> (I can't think of a reason why it'd be a bad idea to do a one-time flush after it completes indexing)
434 2015-11-18T18:44:23  <gmaxwell> the only downside I can think of is that right now doing that will blow out the cache.
435 2015-11-18T18:44:31  *** zooko has quit IRC
436 2015-11-18T18:46:01  *** zooko has joined #bitcoin-core-dev
437 2015-11-18T18:54:09  *** Greyboy has quit IRC
438 2015-11-18T18:54:45  *** evoskuil has joined #bitcoin-core-dev
439 2015-11-18T19:15:22  *** zooko has quit IRC
440 2015-11-18T19:45:48  *** randy-waterhouse has joined #bitcoin-core-dev
441 2015-11-18T20:05:48  *** trippysalmon has joined #bitcoin-core-dev
442 2015-11-18T20:32:39  *** belcher has joined #bitcoin-core-dev
443 2015-11-18T21:20:42  *** arubi has quit IRC
444 2015-11-18T21:28:36  *** arubi has joined #bitcoin-core-dev
445 2015-11-18T21:30:29  <instagibbs> For debugging core where does one put the flag to do different optimization levels?
446 2015-11-18T21:47:04  <phantomcircuit> instagibbs, CFLAGS="" ./configure
447 2015-11-18T21:47:16  <phantomcircuit> grep for CFLAGS in config.log to see what they are by default
448 2015-11-18T21:48:20  <BlueMatt> morcos: so I'm kinda confused about the proposed "score index" to mempool
449 2015-11-18T21:48:39  <BlueMatt> is there any case where we really want a feerate index and do not intend to use it as "priority to get into next block"
450 2015-11-18T21:48:50  <phantomcircuit> Luke-Jr, posted performance numbers in 6851
451 2015-11-18T21:48:52  <BlueMatt> it seems to me we shouldnt need to indexes for this
452 2015-11-18T21:48:58  <morcos> BlueMatt: Well it probably wont' last as its current incarnation anyway
453 2015-11-18T21:49:10  <morcos> Why 2 indices?
454 2015-11-18T21:49:19  <morcos> we definitely need a mining index and an eviction index?
455 2015-11-18T21:49:40  <morcos> are those the 2 you are referring to?
456 2015-11-18T21:49:46  <phantomcircuit> yeah see this is what i was worried about
457 2015-11-18T21:49:51  <phantomcircuit> "just add another index"
458 2015-11-18T21:50:26  <BlueMatt> oh, this is the inverse index of the one we have? yea, nvm, ignore me
459 2015-11-18T21:50:35  <morcos> phantomcircuit: we right now have 3 indices.  time and descendant package fee rate for eviction.  and hash for lookup.
460 2015-11-18T21:50:48  <morcos> we have to have a 4th index for mining
461 2015-11-18T21:51:03  <morcos> because descendant package fee rate is useless
462 2015-11-18T21:51:22  <phantomcircuit> probably dont actually need an index for time if the time based eviction has 1 hour granularity
463 2015-11-18T21:51:48  <morcos> phantomcircuit: agreed, we could probably skip that index, and just scan the whole mempool periodically, it should be super fast
464 2015-11-18T21:52:07  <morcos> but since it doesn't change it probably also doesnt hurt
465 2015-11-18T21:52:23  <phantomcircuit> memory usage?
466 2015-11-18T21:53:02  <morcos> BlueMatt: but sdaftuar was trying to add feeDelta prioritisation to your eviction code and realized that the way I did score made it hard.  because he doesn't want the modified fee rate.  he wants the modified fee
467 2015-11-18T21:53:44  <morcos> So I think he will change it around, but the index will still be sorting by the same thing for mining, the internal data structure might just be a bit different
468 2015-11-18T21:53:51  <BlueMatt> morcos: yea, fair enough...I do wonder, however, if we're gonna do a modified-feerate thing for tx selection, we probably dont want any indexes based on feerate
469 2015-11-18T21:53:55  <BlueMatt> just base everything on modified feerate
470 2015-11-18T21:54:06  <morcos> phantomcircuit: sure, i suppose, its really rather minor memory usage though
471 2015-11-18T21:54:14  <morcos> BlueMatt: yes, thats what will happen
472 2015-11-18T21:54:19  <BlueMatt> even if its modified-feereate-with-package-tracking
473 2015-11-18T21:54:20  <BlueMatt> yea, ok
474 2015-11-18T21:54:32  <morcos> But we need to store the actual fee without modification as well (even though there is no index on it)
475 2015-11-18T21:54:41  <morcos> because it is now required for block construction
476 2015-11-18T21:54:50  <morcos> so it doesn't have to be recalculated
477 2015-11-18T21:56:15  *** Guyver2 has quit IRC
478 2015-11-18T21:56:17  <phantomcircuit> morcos, i assumed as much, but do we have actual numbers for it?
479 2015-11-18T21:57:11  <morcos> phantomcircuit: memory usage?  i think sipa was approximating 3 pointers per entry per index, but i'm not sure where he got that?
480 2015-11-18T21:57:54  <morcos> sigh... just realized i forgot to fix that for my 4th index!
481 2015-11-18T21:59:07  <morcos> sipa: is that right, was it 3
482 2015-11-18T21:59:17  <morcos> 3 pointers per index, or did i just make that up?
483 2015-11-18T22:04:40  *** paveljanik has quit IRC
484 2015-11-18T22:09:02  *** pmienk has joined #bitcoin-core-dev
485 2015-11-18T22:11:08  *** treehug88 has quit IRC
486 2015-11-18T22:52:55  <phantomcircuit> morcos, wait why is the descendent fee index useless for mining?
487 2015-11-18T22:53:22  <phantomcircuit> oh because you need to also track ancestor fees
488 2015-11-18T22:53:24  <phantomcircuit> ha
489 2015-11-18T23:20:25  *** trippysalmon has quit IRC
490 2015-11-18T23:33:09  <dcousens> gmaxwell: I wonder if the auth aspect will play into the RPC v REST speed,  RPC can batch commands/per request which makes me think it'll lead for a while yet
491 2015-11-18T23:34:01  <dcousens> though, without benchmarks, its hard to tell whether univalue is a factor too
492 2015-11-18T23:35:38  <phantomcircuit> hmm
493 2015-11-18T23:36:03  <phantomcircuit> there's no way for us to have more than 1 handle open for the berkeley db in the wallet code
494 2015-11-18T23:36:18  <gmaxwell> univalue is pretty quick, at least my impression from AFLing it.
495 2015-11-18T23:36:42  <phantomcircuit> so Flush is really only used for durability, right?
496 2015-11-18T23:39:23  <phantomcircuit> ha
497 2015-11-18T23:39:41  <phantomcircuit> i was going to ask why nMinutes is only set in CDB::Flush when we're in readonly mode, but satoshi wrote that code
498 2015-11-18T23:40:24  <phantomcircuit> why would we be flushing the database at all if we were in read only mode?
499 2015-11-18T23:40:44  <phantomcircuit> damn it satoshi come back so i can ask mundane questions about code
500 2015-11-18T23:41:23  *** andytoshi has quit IRC
501 2015-11-18T23:41:24  *** andytoshi has joined #bitcoin-core-dev
502 2015-11-18T23:50:46  *** ParadoxSpiral has quit IRC
503 2015-11-18T23:59:37  <phantomcircuit> huh