1 2016-09-29T00:05:28  *** jnewbery has joined #bitcoin-core-dev
   2 2016-09-29T00:05:28  *** aureianimus has quit IRC
   3 2016-09-29T00:05:37  *** aureianimus has joined #bitcoin-core-dev
   4 2016-09-29T00:27:00  *** Ylbam has quit IRC
   5 2016-09-29T00:29:39  *** shesek has joined #bitcoin-core-dev
   6 2016-09-29T00:38:13  *** aureianimus_ has joined #bitcoin-core-dev
   7 2016-09-29T00:38:14  *** aureianimus has quit IRC
   8 2016-09-29T00:49:07  *** aureianimus has joined #bitcoin-core-dev
   9 2016-09-29T00:49:08  *** aureianimus_ has quit IRC
  10 2016-09-29T00:59:03  *** jnewbery has quit IRC
  11 2016-09-29T01:01:50  *** aureianimus_ has joined #bitcoin-core-dev
  12 2016-09-29T01:02:00  *** aureianimus has quit IRC
  13 2016-09-29T01:18:46  *** aureianimus has joined #bitcoin-core-dev
  14 2016-09-29T01:19:13  *** aureianimus_ has quit IRC
  15 2016-09-29T01:19:23  *** aureianimus has joined #bitcoin-core-dev
  16 2016-09-29T01:56:04  *** Chris_Stewart_5 has quit IRC
  17 2016-09-29T02:00:09  *** aureianimus has quit IRC
  18 2016-09-29T02:00:20  *** aureianimus has joined #bitcoin-core-dev
  19 2016-09-29T02:00:37  *** fengling has joined #bitcoin-core-dev
  20 2016-09-29T02:12:07  *** Chris_Stewart_5 has joined #bitcoin-core-dev
  21 2016-09-29T02:21:27  *** Alopex has quit IRC
  22 2016-09-29T02:22:32  *** Alopex has joined #bitcoin-core-dev
  23 2016-09-29T02:29:38  *** Chris_Stewart_5 has quit IRC
  24 2016-09-29T02:31:22  *** aureianimus_ has joined #bitcoin-core-dev
  25 2016-09-29T02:31:33  *** aureianimus has quit IRC
  26 2016-09-29T02:32:50  *** moli has joined #bitcoin-core-dev
  27 2016-09-29T02:40:06  *** Alopex has quit IRC
  28 2016-09-29T02:41:12  *** Alopex has joined #bitcoin-core-dev
  29 2016-09-29T02:42:15  *** aureianimus_ has quit IRC
  30 2016-09-29T02:42:18  *** aureianimus has joined #bitcoin-core-dev
  31 2016-09-29T02:49:24  *** tunafizz has joined #bitcoin-core-dev
  32 2016-09-29T03:03:24  *** aureianimus has quit IRC
  33 2016-09-29T03:03:35  *** aureianimus has joined #bitcoin-core-dev
  34 2016-09-29T03:13:10  *** droark has quit IRC
  35 2016-09-29T03:14:54  *** aureianimus_ has joined #bitcoin-core-dev
  36 2016-09-29T03:15:04  *** aureianimus has quit IRC
  37 2016-09-29T03:26:04  *** aureianimus has joined #bitcoin-core-dev
  38 2016-09-29T03:26:16  *** aureianimus_ has quit IRC
  39 2016-09-29T03:36:50  *** aureianimus has quit IRC
  40 2016-09-29T03:37:02  *** aureianimus has joined #bitcoin-core-dev
  41 2016-09-29T03:46:00  *** cryptapus has joined #bitcoin-core-dev
  42 2016-09-29T03:46:34  *** Alina-malina has quit IRC
  43 2016-09-29T03:47:11  *** Alopex has quit IRC
  44 2016-09-29T03:47:12  *** cryptapus_afk has quit IRC
  45 2016-09-29T03:48:16  *** Alopex has joined #bitcoin-core-dev
  46 2016-09-29T03:49:11  *** Alina-malina has joined #bitcoin-core-dev
  47 2016-09-29T03:49:31  *** Bootvis has quit IRC
  48 2016-09-29T03:50:21  *** davec has quit IRC
  49 2016-09-29T03:50:39  *** davec has joined #bitcoin-core-dev
  50 2016-09-29T03:50:48  *** jouke has quit IRC
  51 2016-09-29T03:54:57  *** Bootvis has joined #bitcoin-core-dev
  52 2016-09-29T03:57:36  *** jouke has joined #bitcoin-core-dev
  53 2016-09-29T03:57:57  *** aureianimus has quit IRC
  54 2016-09-29T03:58:01  *** aureianimus_ has joined #bitcoin-core-dev
  55 2016-09-29T04:24:50  *** aureianimus_ has quit IRC
  56 2016-09-29T04:25:01  *** aureianimus has joined #bitcoin-core-dev
  57 2016-09-29T04:34:57  *** Giszmo has quit IRC
  58 2016-09-29T04:36:01  *** Alopex has quit IRC
  59 2016-09-29T04:36:12  *** aureianimus_ has joined #bitcoin-core-dev
  60 2016-09-29T04:36:19  *** aureianimus has quit IRC
  61 2016-09-29T04:37:06  *** Alopex has joined #bitcoin-core-dev
  62 2016-09-29T04:37:10  *** veleiro has joined #bitcoin-core-dev
  63 2016-09-29T04:47:18  *** aureianimus_ has quit IRC
  64 2016-09-29T04:47:26  *** aureianimus has joined #bitcoin-core-dev
  65 2016-09-29T05:08:40  *** aureianimus has quit IRC
  66 2016-09-29T05:08:51  *** aureianimus has joined #bitcoin-core-dev
  67 2016-09-29T05:16:31  *** Alopex has quit IRC
  68 2016-09-29T05:17:37  *** Alopex has joined #bitcoin-core-dev
  69 2016-09-29T05:21:37  *** aureianimus has quit IRC
  70 2016-09-29T05:21:48  *** aureianimus has joined #bitcoin-core-dev
  71 2016-09-29T05:27:33  *** fengling has quit IRC
  72 2016-09-29T05:32:51  *** aureianimus has quit IRC
  73 2016-09-29T05:32:52  *** aureianimus_ has joined #bitcoin-core-dev
  74 2016-09-29T05:40:50  *** Ylbam has joined #bitcoin-core-dev
  75 2016-09-29T05:44:27  *** aureianimus_ has quit IRC
  76 2016-09-29T05:44:38  *** aureianimus has joined #bitcoin-core-dev
  77 2016-09-29T05:58:38  <GitHub190> [bitcoin] laanwj pushed 5 new commits to master: https://github.com/bitcoin/bitcoin/compare/dc641415e75e...d675984fdfa4
  78 2016-09-29T05:58:39  <GitHub190> bitcoin/master f4dffdd Luke Dashjr: Add MIT license to Makefiles
  79 2016-09-29T05:58:39  <GitHub190> bitcoin/master 3b4b6dc Luke Dashjr: Add MIT license to autogen.sh and share/genbuild.sh
  80 2016-09-29T05:58:40  <GitHub190> bitcoin/master 3f8a5d8 Luke Dashjr: Trivial: build-aux/m4/l_atomic: Fix typo
  81 2016-09-29T05:58:53  <GitHub162> [bitcoin] laanwj closed pull request #8784: Copyright headers for build scripts (master...license_build) https://github.com/bitcoin/bitcoin/pull/8784
  82 2016-09-29T06:01:25  <wumpus> this is strange, https://github.com/bitcoin/bitcoin/pull/8832   "waxmihs2902 approved these changes 10 hours ago"   clicking on that users name gives a 404 page
  83 2016-09-29T06:03:20  <paveljanik> User deleted himself?
  84 2016-09-29T06:03:52  <wumpus> I guess
  85 2016-09-29T06:04:02  <wumpus> or github deleted him
  86 2016-09-29T06:05:22  *** aureianimus has quit IRC
  87 2016-09-29T06:05:31  *** aureianimus has joined #bitcoin-core-dev
  88 2016-09-29T06:06:58  <wumpus> as it is impossible to remove approvals, random-approval-spamming trolls wouldn't be unthinkable. Note that for normal posts, the user gets changed to 'ghost' if they disappear, so the link doesn't break.
  89 2016-09-29T06:08:24  <paveljanik> I think the approval stuff is half-baked...
  90 2016-09-29T06:09:20  <paveljanik> So maybe someone is trying DoS on us...
  91 2016-09-29T06:09:49  <wumpus> it's only on one issue AFAIK, so I doubt it'd be targetted on us :)
  92 2016-09-29T06:10:26  <paveljanik> ./DoS ... and then while :; do ./DoS; done ;-)
  93 2016-09-29T06:10:40  <wumpus> or they got caught *really* early; more likely such a troll/bot would just randomly approve pulls all over the site
  94 2016-09-29T06:11:29  <paveljanik> wumpus, search for the username on the github...
  95 2016-09-29T06:11:33  <paveljanik> complete github...
  96 2016-09-29T06:12:08  <wumpus> luckily they didn't add a message, the message added with review approval is also un-removable/un-editable IIRC
  97 2016-09-29T06:12:34  <wumpus> yes its half-baked :)
  98 2016-09-29T06:12:45  <wumpus> I'm sure they'll fix this eventually
  99 2016-09-29T06:13:13  <GitHub131> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/d675984fdfa4...7d563cc16d64
 100 2016-09-29T06:13:13  <GitHub131> bitcoin/master fa05cfd MarcoFalke: [rpc] throw JSONRPCError when utxo set can not be read
 101 2016-09-29T06:13:14  <GitHub131> bitcoin/master 7d563cc Wladimir J. van der Laan: Merge #8832: [rpc] throw JSONRPCError when utxo set can not be read...
 102 2016-09-29T06:13:26  <GitHub93> [bitcoin] laanwj closed pull request #8832: [rpc] throw JSONRPCError when utxo set can not be read (master...Mf1610-rpcUtxoFail) https://github.com/bitcoin/bitcoin/pull/8832
 103 2016-09-29T06:20:43  *** fengling has joined #bitcoin-core-dev
 104 2016-09-29T06:33:29  *** AaronvanW has quit IRC
 105 2016-09-29T06:54:12  *** aureianimus has quit IRC
 106 2016-09-29T06:54:21  *** aureianimus has joined #bitcoin-core-dev
 107 2016-09-29T06:57:20  *** rubensayshi has joined #bitcoin-core-dev
 108 2016-09-29T07:05:08  *** aureianimus has quit IRC
 109 2016-09-29T07:05:16  *** aureianimus has joined #bitcoin-core-dev
 110 2016-09-29T07:19:48  <GitHub3> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/7d563cc16d64...489a6ab5073c
 111 2016-09-29T07:19:48  <GitHub3> bitcoin/master 64047f8 Wladimir J. van der Laan: depends: Add libevent compatibility patch for windows...
 112 2016-09-29T07:19:49  <GitHub3> bitcoin/master 489a6ab Wladimir J. van der Laan: Merge #8730: depends: Add libevent compatibility patch for windows...
 113 2016-09-29T07:19:58  <GitHub165> [bitcoin] laanwj closed pull request #8730: depends: Add libevent compatibility patch for windows (master...2016_09_libevent_windows_gcc_531) https://github.com/bitcoin/bitcoin/pull/8730
 114 2016-09-29T07:20:46  *** aureianimus has quit IRC
 115 2016-09-29T07:20:52  *** aureianimus_ has joined #bitcoin-core-dev
 116 2016-09-29T07:21:13  <GitHub162> [bitcoin] laanwj closed pull request #7522: Bugfix: Only use git for build info if the repository is actually the right one (master...bugfix_gitdir) https://github.com/bitcoin/bitcoin/pull/7522
 117 2016-09-29T07:24:47  <wumpus> cfields_: I've assigned some build-system issues to you, hope you don't mind
 118 2016-09-29T07:31:39  *** aureianimus_ has quit IRC
 119 2016-09-29T07:31:51  *** aureianimus has joined #bitcoin-core-dev
 120 2016-09-29T07:33:06  *** DigiByteDev has joined #bitcoin-core-dev
 121 2016-09-29T07:34:56  *** Alina-malina has quit IRC
 122 2016-09-29T07:34:56  *** Alina-malina has joined #bitcoin-core-dev
 123 2016-09-29T07:37:43  *** DigiByteDev has quit IRC
 124 2016-09-29T07:46:00  *** aureianimus has quit IRC
 125 2016-09-29T07:46:12  *** aureianimus has joined #bitcoin-core-dev
 126 2016-09-29T07:58:00  *** DigiByteDev has joined #bitcoin-core-dev
 127 2016-09-29T07:58:15  *** ghtdak has quit IRC
 128 2016-09-29T08:02:40  *** ghtdak has joined #bitcoin-core-dev
 129 2016-09-29T08:34:04  *** DigiByteDev_ has joined #bitcoin-core-dev
 130 2016-09-29T08:34:09  *** Guyver2 has joined #bitcoin-core-dev
 131 2016-09-29T08:34:45  *** DigiByteDev has quit IRC
 132 2016-09-29T08:34:48  *** DigiByteDev_ is now known as DigiByteDev
 133 2016-09-29T08:49:07  *** aureianimus has quit IRC
 134 2016-09-29T08:49:18  *** aureianimus has joined #bitcoin-core-dev
 135 2016-09-29T08:49:42  *** DigiByteDev has quit IRC
 136 2016-09-29T08:50:33  *** MarcoFalke has joined #bitcoin-core-dev
 137 2016-09-29T08:50:57  <GitHub87> [bitcoin] MarcoFalke pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/489a6ab5073c...8ca69a2a88a7
 138 2016-09-29T08:50:58  <GitHub87> bitcoin/master 54e5d7c jnewbery: Add bitcoin-tx JSON tests
 139 2016-09-29T08:50:58  <GitHub87> bitcoin/master 8ca69a2 MarcoFalke: Merge #8829: Add bitcoin-tx JSON tests...
 140 2016-09-29T08:51:07  <GitHub21> [bitcoin] MarcoFalke closed pull request #8829: Add bitcoin-tx JSON tests (master...test-bitcoin-tx-json) https://github.com/bitcoin/bitcoin/pull/8829
 141 2016-09-29T08:55:45  *** pedrobranco has joined #bitcoin-core-dev
 142 2016-09-29T08:59:08  *** pedrobranco has quit IRC
 143 2016-09-29T08:59:12  *** pedrobra_ has joined #bitcoin-core-dev
 144 2016-09-29T09:00:41  <wumpus> weird, I'm on testnet and trying to rebroadcast a transaction using "sendrawtransaction" but it's not working, I don't see anything appear in my wireshark session
 145 2016-09-29T09:01:48  <wumpus> maybe all the current nodes already know of it, otoh the "broadcast through 1 node(s)" in status doesn't increase either
 146 2016-09-29T09:03:27  <sipa> could it be that your peers knew about it, but evicted it?
 147 2016-09-29T09:03:49  <wumpus> but for they they'd have to request it first
 148 2016-09-29T09:04:26  <MarcoFalke> What about resendwallettransactions
 149 2016-09-29T09:04:28  <sipa> not necessarily from you
 150 2016-09-29T09:05:03  <wumpus> MarcoFalke: this should work with sendrawtransaction
 151 2016-09-29T09:05:38  <wumpus> could be that the net refactoring work changed the assumptions, will add some debugging...
 152 2016-09-29T09:06:31  <sipa> i don't think so
 153 2016-09-29T09:06:42  <MarcoFalke> its a noop when fHaveMempool?
 154 2016-09-29T09:07:00  <sipa> no
 155 2016-09-29T09:07:05  <wumpus> I don't hope so
 156 2016-09-29T09:07:24  <MarcoFalke> sendrawtransaction will only put in in your mempool
 157 2016-09-29T09:07:36  <sipa> and then loop over all peers
 158 2016-09-29T09:07:41  <sipa> and call PushInventory
 159 2016-09-29T09:07:48  <MarcoFalke> oh
 160 2016-09-29T09:08:30  <sipa> but PushInventory is a noop if filterInventoryKnown for that peer already contains the tx
 161 2016-09-29T09:09:08  <wumpus> but it can't be there unless the peer requested it from *us*right?
 162 2016-09-29T09:09:40  <wumpus> hm or if they inved it for relay to us, I guess
 163 2016-09-29T09:10:46  <wumpus> I don't have a capture of the whole session so we'll never know for sure
 164 2016-09-29T09:10:48  <sipa> well you can enable -debug=net and see if any messages go out in response to sendrawtransaction
 165 2016-09-29T09:10:56  <sipa> oh
 166 2016-09-29T09:11:00  <sipa> you're already doing that
 167 2016-09-29T09:11:14  <wumpus> no, no messages went out
 168 2016-09-29T09:11:30  <wumpus> and according to the tx metadata apparently one peer ever requested the transaction
 169 2016-09-29T09:11:38  <wumpus> but that was before I started logging
 170 2016-09-29T09:13:33  <wumpus> it could also be that that counter is broken, of course, I don't think that functionality is tested anywhere it's UI only
 171 2016-09-29T09:16:01  *** aureianimus has quit IRC
 172 2016-09-29T09:16:10  *** aureianimus has joined #bitcoin-core-dev
 173 2016-09-29T09:33:17  <wumpus> ok, seems the behaviour was correct. After restart the transaction is no longer in the filter, so I do sendrawtransaction again, it sends out an inv to every node.
 174 2016-09-29T09:33:17  <sipa> i wonder if we should add some random memory/cpu intensive task occasionally during block validation (so it doesn't consume more than 1% cpu rate or so)
 175 2016-09-29T09:33:29  <sipa> and if that fails, tell the user their hardware is too unreliable
 176 2016-09-29T09:33:47  <wumpus> yes, some games have that, it's not a bad idea
 177 2016-09-29T09:33:56  <wumpus> it allows distinguishing bugs from hw failures
 178 2016-09-29T09:34:57  <wumpus> we have the same problem, 'support' is overflowed with issues that are probably hw failures but we can't be sure so it wastes a lot of time
 179 2016-09-29T09:38:42  <paveljanik> We could ask users for the result of bitcoind -sanitychecks or something if we suspect HW issue...
 180 2016-09-29T09:39:05  *** jannes has joined #bitcoin-core-dev
 181 2016-09-29T09:39:13  <sipa> paveljanik: the problem is that such a sanitycheck would need to run for several hours or so to be reliable
 182 2016-09-29T09:40:02  <paveljanik> sure, but 90% rule...
 183 2016-09-29T09:40:46  <sipa> if a failure is detectable within minutes, it also means their block validation like fails within minutes
 184 2016-09-29T09:40:51  <sipa> and many other things
 185 2016-09-29T09:41:03  <sipa> most random failure i hear about happen after several hours of sync
 186 2016-09-29T09:42:16  <wumpus> the sanity check definitely needs to run automatically and periodically for it to be useful
 187 2016-09-29T09:42:26  <wumpus> because otherwise it won't run in the right conditinos
 188 2016-09-29T09:43:19  <wumpus> we should start selling a hardware quality certificates: has synced the bitcoin block chain succesfully
 189 2016-09-29T09:43:44  *** Guyver2_ has joined #bitcoin-core-dev
 190 2016-09-29T09:43:55  <wumpus> if it can do that, it won't crash on games either. Well. Maybe after GPU secp256k1 verification is implemented ;)
 191 2016-09-29T09:44:18  <sipa> maybe we should instead just market secp256k1 asics
 192 2016-09-29T09:45:12  <wumpus> would be a very interesting project if there's a market for that
 193 2016-09-29T09:46:11  <wumpus> hm, scrap that. THe market for that is key-cracking :(
 194 2016-09-29T09:47:00  *** aureianimus has quit IRC
 195 2016-09-29T09:47:01  *** Guyver2 has quit IRC
 196 2016-09-29T09:47:05  *** aureianimus_ has joined #bitcoin-core-dev
 197 2016-09-29T09:47:08  *** Guyver2_ is now known as Guyver2
 198 2016-09-29T09:50:45  <wumpus> searching around a bit, apparently, many people *started* on a verilog/vhdl FPGA implementation of secp256k1, but there's no code to be found anywhere
 199 2016-09-29T09:57:49  *** aureianimus_ has quit IRC
 200 2016-09-29T09:58:01  *** aureianimus has joined #bitcoin-core-dev
 201 2016-09-29T09:58:40  <wumpus> I'm pleasantly surprised how well the wireshark dissector for the bitcoin protocol works, can just do "tcp port 18333" then use a display filter of "bitcoin" and gets a running list of bitcoin packets easy to inspect using the tree structure
 202 2016-09-29T10:01:27  <wumpus> compared to trying to mentally parse debug.log output this is a breeze
 203 2016-09-29T10:02:16  <waxwing> wumpus: they've had that dissector for years, i remember being pleasantly surprised in like 2013/14
 204 2016-09-29T10:02:29  <waxwing> not that i need it or anything, but it's cool :)
 205 2016-09-29T10:02:49  <wumpus> yeah yeah it's probably not new
 206 2016-09-29T10:03:14  <wumpus> but just in case someone didn't discover it 20 years ago yet, there you go...
 207 2016-09-29T10:03:30  <waxwing> sorry wasn't trying to be a hipster :)
 208 2016-09-29T10:03:44  <wumpus> :-)
 209 2016-09-29T10:03:47  <sipa> damn. i believe this is a blocker for bip151.
 210 2016-09-29T10:04:26  <wumpus> agree sipa
 211 2016-09-29T10:08:25  <wumpus> though it is possible to have dissectors with parameters, such as a key, but it's so much less user friendly!
 212 2016-09-29T10:08:48  *** aureianimus has quit IRC
 213 2016-09-29T10:08:58  *** aureianimus has joined #bitcoin-core-dev
 214 2016-09-29T10:12:28  <wumpus> a more practical way to go at this, post-bip151, would be add functionality to dump packets to disk after decryption on receive and before encryption on send, it's for debugging anyhow not snooping
 215 2016-09-29T10:14:04  <waxwing> iirc there are per-dissector settings, for ssl you can enter the session key there etc.
 216 2016-09-29T10:14:26  <wumpus> yes
 217 2016-09-29T10:14:30  <waxwing> there's an env variable you can set when running firefox that dumps the keys for you, and you can import it in
 218 2016-09-29T10:14:40  <sipa> implementing bip151 in wireshark will be fu
 219 2016-09-29T10:14:40  <waxwing> and chrome i think
 220 2016-09-29T10:14:48  <wumpus> waxwing: but if you deal with lots of different sessions, or sessions that are created on the fly
 221 2016-09-29T10:15:21  <waxwing> right, which i guess might be of particular import in your case (/me hasn't read bip151 tho)
 222 2016-09-29T10:15:44  <wumpus> it's certainly possible, the only remark I was making was in regard to what was the most practical way :)
 223 2016-09-29T10:16:45  <wumpus> one advantage of doing the decryption in the dissector would be that it could detect crypto problems
 224 2016-09-29T10:18:31  <sipa> i don't know what language wireshark uses... but reimplementing sha256, chacha20 and poly1305 doesn't sound trivial
 225 2016-09-29T10:18:48  <sipa> (unless those are already available as primitives)
 226 2016-09-29T10:18:49  <wumpus> C
 227 2016-09-29T10:19:09  <sipa> oh, the filters too?
 228 2016-09-29T10:19:20  <wumpus> it's also possible to write dissectors in lua, but that's only recommended for one-off projects, as performance is abysmal
 229 2016-09-29T10:19:23  <sipa> i thought those would be in a plugin language like lua or so
 230 2016-09-29T10:19:24  <wumpus> yes, most of them
 231 2016-09-29T10:19:31  <sipa> ah.
 232 2016-09-29T10:20:27  <sipa> the bitcoin dissector is in c currently?
 233 2016-09-29T10:20:43  <wumpus> let me see
 234 2016-09-29T10:21:31  <sipa> i vaguely remember seeing the code for it, years ago
 235 2016-09-29T10:22:52  <wumpus> https://github.com/wireshark/wireshark/blob/master/epan/dissectors/packet-bitcoin.c
 236 2016-09-29T10:24:23  <GitHub156> [bitcoin] MarcoFalke opened pull request #8834: [qa] blockstore: Switch to dumb dbm (master...Mf1610-qaBlockstoreDumb) https://github.com/bitcoin/bitcoin/pull/8834
 237 2016-09-29T10:24:52  <wumpus> apparently it's not yet updated for 0.12.0+, no handler for sendheaders, feefilter etc
 238 2016-09-29T10:27:21  *** mmeijeri has joined #bitcoin-core-dev
 239 2016-09-29T10:27:29  <mmeijeri> The same environment variable (SSLKEYLOGFILE) also works for Chrome.
 240 2016-09-29T10:29:17  <wumpus> oh it's a dynamic file? that's a good idea
 241 2016-09-29T10:29:19  <waxwing> mmeijeri: it's coming back to me, i have a feeling that it handles multiple sessions transparently, by just checking which key material works for which session. not that this matters, sorry for OT :)
 242 2016-09-29T10:30:31  <waxwing> ah yes that was it, i think it stores the premaster secrets for all sessions, then in each handshake it tries them out until it finds what works. something like that.
 243 2016-09-29T10:34:21  *** wumpus has quit IRC
 244 2016-09-29T10:42:16  *** droark has joined #bitcoin-core-dev
 245 2016-09-29T10:57:38  *** aureianimus has quit IRC
 246 2016-09-29T10:57:47  *** aureianimus has joined #bitcoin-core-dev
 247 2016-09-29T11:08:29  *** aureianimus has quit IRC
 248 2016-09-29T11:08:42  *** aureianimus has joined #bitcoin-core-dev
 249 2016-09-29T11:10:28  <GitHub26> [bitcoin] MarcoFalke pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/8ca69a2a88a7...cc9e8aca5f95
 250 2016-09-29T11:10:28  <GitHub26> bitcoin/master a0f8482 Suhas Daftuar: [qa] Split up slow RPC calls to avoid pruning test timeouts
 251 2016-09-29T11:10:29  <GitHub26> bitcoin/master cc9e8ac MarcoFalke: Merge #8827: [qa] Split up slow RPC calls to avoid pruning test timeouts...
 252 2016-09-29T11:10:48  <GitHub122> [bitcoin] MarcoFalke closed pull request #8827: [qa] Split up slow RPC calls to avoid pruning test timeouts (master...fix-pruning-timeout) https://github.com/bitcoin/bitcoin/pull/8827
 253 2016-09-29T11:15:22  *** molz has joined #bitcoin-core-dev
 254 2016-09-29T11:16:06  *** moli has quit IRC
 255 2016-09-29T11:19:43  <luke-jr> any reason for me to NOT switch to 64-bit after I sleep?
 256 2016-09-29T11:22:21  *** aureianimus has quit IRC
 257 2016-09-29T11:22:33  *** aureianimus has joined #bitcoin-core-dev
 258 2016-09-29T11:22:46  <phantomcircuit> luke-jr: something about  #8828 results in a null pointer dereference
 259 2016-09-29T11:22:58  <phantomcircuit> afaict i just moved code around basically
 260 2016-09-29T11:23:03  <phantomcircuit> you wrote the original method
 261 2016-09-29T11:23:07  <phantomcircuit> can you take a look?
 262 2016-09-29T11:23:17  *** benma has joined #bitcoin-core-dev
 263 2016-09-29T11:23:53  <luke-jr> phantomcircuit: too sleepy now, can you ask me after I finish switching into 64-bit (probably tomorrow)? :x
 264 2016-09-29T11:24:09  <phantomcircuit> luke-jr: whoa now
 265 2016-09-29T11:24:13  <phantomcircuit> you're moving to 64bit?
 266 2016-09-29T11:24:23  <luke-jr> phantomcircuit: unless someone stops me before I wake up
 267 2016-09-29T11:24:56  <phantomcircuit> also the line gdb thinks it's segfaulting on makes no sense
 268 2016-09-29T11:25:01  <luke-jr> ?
 269 2016-09-29T11:25:01  <phantomcircuit> nOrderPosNext = 0;
 270 2016-09-29T11:25:03  *** cryptapus_ has joined #bitcoin-core-dev
 271 2016-09-29T11:25:09  <luke-jr> so this is NULL? O.o
 272 2016-09-29T11:25:46  <luke-jr> +    int64_t& nOrderPosNext = nOrderPosNext;
 273 2016-09-29T11:25:48  <luke-jr> wtf is this
 274 2016-09-29T11:26:13  <luke-jr> bet that's your problem, it makes no sense
 275 2016-09-29T11:26:16  <luke-jr> prob undefined behaviour
 276 2016-09-29T11:26:35  <luke-jr> but I'm half asleep, so who knows
 277 2016-09-29T11:28:09  *** aureianimus has quit IRC
 278 2016-09-29T11:28:21  *** aureianimus has joined #bitcoin-core-dev
 279 2016-09-29T11:35:12  *** harrymm has left #bitcoin-core-dev
 280 2016-09-29T11:40:01  *** fengling has quit IRC
 281 2016-09-29T11:40:04  *** aureianimus has quit IRC
 282 2016-09-29T11:40:17  *** aureianimus has joined #bitcoin-core-dev
 283 2016-09-29T11:45:18  *** harrymm has joined #bitcoin-core-dev
 284 2016-09-29T11:52:55  *** mol has joined #bitcoin-core-dev
 285 2016-09-29T11:55:04  <luke-jr> oh well, hope that helped, otherwise ping me tomorrow I guess :x
 286 2016-09-29T11:55:06  <luke-jr> night
 287 2016-09-29T11:55:41  *** molz has quit IRC
 288 2016-09-29T11:57:15  *** fengling has joined #bitcoin-core-dev
 289 2016-09-29T12:01:36  <phantomcircuit> luke-jr: that's weird but apparently correct
 290 2016-09-29T12:05:13  *** Soligor has quit IRC
 291 2016-09-29T12:10:16  <midnightmagic> that's a weird scoping problem
 292 2016-09-29T12:10:26  <midnightmagic> lol
 293 2016-09-29T12:13:37  *** Soligor has joined #bitcoin-core-dev
 294 2016-09-29T12:15:19  *** fengling has quit IRC
 295 2016-09-29T12:16:29  *** Squidicc has joined #bitcoin-core-dev
 296 2016-09-29T12:19:01  *** Squidicuz has quit IRC
 297 2016-09-29T12:31:15  *** aureianimus has quit IRC
 298 2016-09-29T12:31:23  *** aureianimus has joined #bitcoin-core-dev
 299 2016-09-29T12:40:12  *** jnewbery has joined #bitcoin-core-dev
 300 2016-09-29T12:50:51  *** droark has quit IRC
 301 2016-09-29T13:00:43  <GitHub52> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/cc9e8aca5f95...9b94cca41f3e
 302 2016-09-29T13:00:43  <GitHub52> bitcoin/master 64d9507 Pavel Janík: [WIP] Remove unused statement in serialization
 303 2016-09-29T13:00:44  <GitHub52> bitcoin/master 9b94cca Wladimir J. van der Laan: Merge #8658: Remove unused statements in serialization...
 304 2016-09-29T13:00:53  <GitHub41> [bitcoin] laanwj closed pull request #8658: Remove unused statements in serialization (master...20160902_nVersion_serialization_cleanup) https://github.com/bitcoin/bitcoin/pull/8658
 305 2016-09-29T13:02:24  <kanzure> wumpus: tcpdump if you want to generate .pcap files without wireshark's gui
 306 2016-09-29T13:05:09  *** paveljanik has quit IRC
 307 2016-09-29T13:05:37  *** limpkin_ is now known as limpkin
 308 2016-09-29T13:05:43  <GitHub166> [bitcoin] jgarzik closed pull request #6451: BIP 102: Increase block size limit to 2MB (master...2015_2mb_blocksize) https://github.com/bitcoin/bitcoin/pull/6451
 309 2016-09-29T13:06:21  * midnightmagic stabs wireshark's gui
 310 2016-09-29T13:07:26  *** Chris_Stewart_5 has joined #bitcoin-core-dev
 311 2016-09-29T13:09:13  *** veleiro has quit IRC
 312 2016-09-29T13:12:19  *** aureianimus has quit IRC
 313 2016-09-29T13:12:23  *** aureianimus_ has joined #bitcoin-core-dev
 314 2016-09-29T13:24:06  *** aureianimus_ has quit IRC
 315 2016-09-29T13:24:18  *** aureianimus has joined #bitcoin-core-dev
 316 2016-09-29T13:27:46  *** benma has quit IRC
 317 2016-09-29T13:33:18  *** AaronvanW has joined #bitcoin-core-dev
 318 2016-09-29T13:35:34  *** To7 has joined #bitcoin-core-dev
 319 2016-09-29T13:36:13  <GitHub180> [bitcoin] MarcoFalke opened pull request #8835: [qa] nulldummy.py: Don't run unused code (master...Mf1610-qaNulldummyUnused) https://github.com/bitcoin/bitcoin/pull/8835
 320 2016-09-29T13:45:11  *** aureianimus has quit IRC
 321 2016-09-29T13:45:12  *** aureianimus_ has joined #bitcoin-core-dev
 322 2016-09-29T13:51:23  *** Chris_Stewart_5 has quit IRC
 323 2016-09-29T14:01:04  *** aureianimus_ has quit IRC
 324 2016-09-29T14:01:15  *** aureianimus has joined #bitcoin-core-dev
 325 2016-09-29T14:02:29  *** pedrobra_ has quit IRC
 326 2016-09-29T14:05:09  *** MarcoFalke has left #bitcoin-core-dev
 327 2016-09-29T14:22:14  *** aureianimus_ has joined #bitcoin-core-dev
 328 2016-09-29T14:22:24  <GitHub8> [bitcoin] jnewbery opened pull request #8836: bitcoin-util-test.py should fail if the output file is empty (master...bitcoin-tx-no-empty-outputs) https://github.com/bitcoin/bitcoin/pull/8836
 329 2016-09-29T14:22:25  *** aureianimus has quit IRC
 330 2016-09-29T14:25:26  *** rubensayshi has quit IRC
 331 2016-09-29T14:26:55  *** wumpus has joined #bitcoin-core-dev
 332 2016-09-29T14:27:04  <GitHub100> [bitcoin] jnewbery opened pull request #8837: allow bitcoin-tx to parse partial transactions (master...bitcoin-tx-partial-transactions) https://github.com/bitcoin/bitcoin/pull/8837
 333 2016-09-29T14:31:12  *** Giszmo has joined #bitcoin-core-dev
 334 2016-09-29T14:34:23  <GitHub16> [bitcoin] jnewbery opened pull request #8838: Only log block size if block size is being accounted (master...dont_log_size) https://github.com/bitcoin/bitcoin/pull/8838
 335 2016-09-29T14:35:39  *** mmeijeri has quit IRC
 336 2016-09-29T14:41:02  *** aalex_ has quit IRC
 337 2016-09-29T14:41:03  *** paveljanik has joined #bitcoin-core-dev
 338 2016-09-29T14:46:14  <GitHub177> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/9b94cca41f3e...2dd57e4f9f58
 339 2016-09-29T14:46:15  <GitHub177> bitcoin/master fa156c6 MarcoFalke: [qa] nulldummy: Don't run unused code
 340 2016-09-29T14:46:15  <GitHub177> bitcoin/master 2dd57e4 Wladimir J. van der Laan: Merge #8835: [qa] nulldummy.py: Don't run unused code...
 341 2016-09-29T14:46:29  <GitHub194> [bitcoin] laanwj closed pull request #8835: [qa] nulldummy.py: Don't run unused code (master...Mf1610-qaNulldummyUnused) https://github.com/bitcoin/bitcoin/pull/8835
 342 2016-09-29T15:03:16  *** aureianimus has joined #bitcoin-core-dev
 343 2016-09-29T15:03:29  *** aureianimus_ has quit IRC
 344 2016-09-29T15:04:33  <GitHub44> [bitcoin] laanwj opened pull request #8839: test: Avoid ConnectionResetErrors during RPC tests (master...2016_09_freebsd_rpctest_fix) https://github.com/bitcoin/bitcoin/pull/8839
 345 2016-09-29T15:05:56  *** aalex has joined #bitcoin-core-dev
 346 2016-09-29T15:07:38  *** laurentmt has joined #bitcoin-core-dev
 347 2016-09-29T15:08:53  <GitHub42> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/2dd57e4f9f58...c84181665f34
 348 2016-09-29T15:08:54  <GitHub42> bitcoin/master 16f8823 fanquake: [depends] Boost 1.61.0
 349 2016-09-29T15:08:54  <GitHub42> bitcoin/master c841816 Wladimir J. van der Laan: Merge #8819: [depends] Boost 1.61.0...
 350 2016-09-29T15:09:08  <GitHub98> [bitcoin] laanwj closed pull request #8819: [depends] Boost 1.61.0 (master...depends-boost-1-61-0) https://github.com/bitcoin/bitcoin/pull/8819
 351 2016-09-29T15:10:26  *** laurentmt has quit IRC
 352 2016-09-29T15:18:31  *** moli has joined #bitcoin-core-dev
 353 2016-09-29T15:21:57  *** mol has quit IRC
 354 2016-09-29T15:23:48  <GitHub53> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/c84181665f34...c9d7b0de2fc9
 355 2016-09-29T15:23:48  <GitHub53> bitcoin/master fa9cd25 MarcoFalke: [qa] blockstore: Switch to dumb dbm
 356 2016-09-29T15:23:49  <GitHub53> bitcoin/master c9d7b0d Wladimir J. van der Laan: Merge #8834: [qa] blockstore: Switch to dumb dbm...
 357 2016-09-29T15:23:58  <GitHub36> [bitcoin] laanwj closed pull request #8834: [qa] blockstore: Switch to dumb dbm (master...Mf1610-qaBlockstoreDumb) https://github.com/bitcoin/bitcoin/pull/8834
 358 2016-09-29T15:27:49  <GitHub17> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/c9d7b0de2fc9...f560d9564f74
 359 2016-09-29T15:27:49  <GitHub17> bitcoin/master 7e5fd71 Pavel Janík: Do not include env_win.cc on non-Windows systems
 360 2016-09-29T15:27:49  <GitHub17> bitcoin/master f560d95 Wladimir J. van der Laan: Merge #8826: Do not include env_win.cc on non-Windows systems...
 361 2016-09-29T15:27:59  <GitHub75> [bitcoin] laanwj closed pull request #8826: Do not include env_win.cc on non-Windows systems (master...20160928_leveldb_no_win) https://github.com/bitcoin/bitcoin/pull/8826
 362 2016-09-29T15:44:16  *** aureianimus has quit IRC
 363 2016-09-29T15:44:27  *** aureianimus has joined #bitcoin-core-dev
 364 2016-09-29T15:51:57  *** Chris_Stewart_5 has joined #bitcoin-core-dev
 365 2016-09-29T15:55:00  *** cryptapus__ has joined #bitcoin-core-dev
 366 2016-09-29T15:58:01  *** laurentmt has joined #bitcoin-core-dev
 367 2016-09-29T15:58:41  *** cryptapus_ has quit IRC
 368 2016-09-29T16:03:49  *** aureianimus has quit IRC
 369 2016-09-29T16:04:01  *** aureianimus has joined #bitcoin-core-dev
 370 2016-09-29T16:05:24  <GitHub51> [bitcoin] laanwj opened pull request #8840: test: Explicitly set encoding to utf8 when opening text files (master...2016_09_textfiles_locale) https://github.com/bitcoin/bitcoin/pull/8840
 371 2016-09-29T16:08:03  *** laurentmt has quit IRC
 372 2016-09-29T16:23:13  <GitHub57> [bitcoin] jl2012 opened pull request #8841: [qa] fix nulldummy test (master...nulldummytest) https://github.com/bitcoin/bitcoin/pull/8841
 373 2016-09-29T16:32:53  *** aalex has quit IRC
 374 2016-09-29T16:33:46  *** aalex has joined #bitcoin-core-dev
 375 2016-09-29T16:35:24  *** aureianimus has quit IRC
 376 2016-09-29T16:35:28  *** Giszmo has quit IRC
 377 2016-09-29T16:35:35  *** aureianimus has joined #bitcoin-core-dev
 378 2016-09-29T16:46:25  *** jnewbery has quit IRC
 379 2016-09-29T16:47:05  *** jnewbery has joined #bitcoin-core-dev
 380 2016-09-29T16:51:53  *** jnewbery has quit IRC
 381 2016-09-29T16:57:32  *** molz has joined #bitcoin-core-dev
 382 2016-09-29T16:57:39  *** moli has quit IRC
 383 2016-09-29T17:03:49  *** jnewbery has joined #bitcoin-core-dev
 384 2016-09-29T17:16:21  <cfields_> wumpus: no problem
 385 2016-09-29T17:17:04  <cfields_> jonasschnelli: i had a look at your circular dep issue. No luck here. The (hack) solution is to use grouping libs, but I'd really rather not go down that road. I've started untangling dependencies instead.
 386 2016-09-29T17:20:18  *** Arnavion has quit IRC
 387 2016-09-29T17:23:27  *** Arnavion has joined #bitcoin-core-dev
 388 2016-09-29T17:25:56  <wumpus> which dependencies are the problem there? the circular dependency between libbitcoin_server and libbitcoin_wallet?
 389 2016-09-29T17:26:45  <wumpus> that would "just" require solving https://github.com/bitcoin/bitcoin/issues/7965
 390 2016-09-29T17:26:52  <arubi> jnewbery, :)  I was just about to whine about 'if (!DecodeHexTx(txDecodeTmp, strHexTx)'
 391 2016-09-29T17:26:57  <arubi> thanks!
 392 2016-09-29T17:36:22  *** aureianimus has quit IRC
 393 2016-09-29T17:36:31  *** aureianimus has joined #bitcoin-core-dev
 394 2016-09-29T17:38:07  *** mrkent has joined #bitcoin-core-dev
 395 2016-09-29T17:38:10  *** jnewbery has quit IRC
 396 2016-09-29T17:47:57  *** cryptapus__ has quit IRC
 397 2016-09-29T17:48:11  *** cryptapus__ has joined #bitcoin-core-dev
 398 2016-09-29T17:48:11  *** cryptapus__ has joined #bitcoin-core-dev
 399 2016-09-29T17:48:17  *** cryptapus__ is now known as cryptapus_
 400 2016-09-29T17:54:08  *** laurentmt has joined #bitcoin-core-dev
 401 2016-09-29T17:54:26  *** laurentmt has quit IRC
 402 2016-09-29T17:57:32  <arubi> too bad, I wanted to ask him about it here
 403 2016-09-29T17:57:35  *** jtimon has joined #bitcoin-core-dev
 404 2016-09-29T18:01:21  *** MarcoFalke has joined #bitcoin-core-dev
 405 2016-09-29T18:12:17  *** aureianimus has quit IRC
 406 2016-09-29T18:12:19  *** aureianimus_ has joined #bitcoin-core-dev
 407 2016-09-29T18:20:36  <jonasschnelli> wumpus, cfields_: Yes. I think we need to complete 7965 first
 408 2016-09-29T18:23:11  *** aureianimus_ has quit IRC
 409 2016-09-29T18:23:22  *** aureianimus has joined #bitcoin-core-dev
 410 2016-09-29T18:32:20  *** gabridome has joined #bitcoin-core-dev
 411 2016-09-29T18:36:43  *** Squidicc is now known as squidicuz
 412 2016-09-29T18:37:36  <jl2012> Travis test failed for #8841 with an unrelated test
 413 2016-09-29T18:37:45  <jl2012> https://www.irccloud.com/pastebin/pWEgvccJ/
 414 2016-09-29T18:38:14  <jl2012> https://travis-ci.org/bitcoin/bitcoin/jobs/163788152
 415 2016-09-29T18:39:24  *** jnewbery has joined #bitcoin-core-dev
 416 2016-09-29T18:40:22  <luke-jr> phantomcircuit: correct for what?
 417 2016-09-29T18:40:38  <MarcoFalke> jl2012: We should probably create an issue for those failures
 418 2016-09-29T18:43:18  *** Soligor_ has joined #bitcoin-core-dev
 419 2016-09-29T18:48:11  *** lightningbot has joined #bitcoin-core-dev
 420 2016-09-29T19:00:04  *** BakSAj has joined #bitcoin-core-dev
 421 2016-09-29T19:00:15  <jonasschnelli> bingbong
 422 2016-09-29T19:00:35  <luke-jr> hi
 423 2016-09-29T19:00:41  <sipa> ovatobat
 424 2016-09-29T19:00:45  <CodeShark> yo
 425 2016-09-29T19:01:09  *** cdecker has quit IRC
 426 2016-09-29T19:01:19  <wumpus> #startmeeting
 427 2016-09-29T19:01:19  <lightningbot> Meeting started Thu Sep 29 19:01:19 2016 UTC.  The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
 428 2016-09-29T19:01:19  <lightningbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
 429 2016-09-29T19:01:26  <BakSAj> hi
 430 2016-09-29T19:01:36  <CodeShark> meeting time!
 431 2016-09-29T19:01:46  <MarcoFalke> meeting!
 432 2016-09-29T19:01:56  <wumpus> #bitcoin-core-dev Meeting: wumpus sipa gmaxwell jonasschnelli morcos luke-jr btcdrak sdaftuar jtimon cfields petertodd kanzure bluematt instagibbs phantomcircuit codeshark michagogo marcofalke paveljanik NicolasDorier
 433 2016-09-29T19:02:04  <MarcoFalke> (oh already started)
 434 2016-09-29T19:02:07  <btcdrak> here
 435 2016-09-29T19:02:07  <cfields_> hi
 436 2016-09-29T19:02:12  <kanzure> hi
 437 2016-09-29T19:02:20  <wumpus> any proposed topics?
 438 2016-09-29T19:02:24  <jonasschnelli> topic proposal: pruning and blockrelay
 439 2016-09-29T19:02:56  <petertodd> hi
 440 2016-09-29T19:03:11  <sipa> policy against uncompressed keys or not
 441 2016-09-29T19:03:11  *** Chris_Stewart_5 has quit IRC
 442 2016-09-29T19:03:11  <wumpus> #topic pruning and blockrelay
 443 2016-09-29T19:03:38  <jonasschnelli> I think we should add a service flag for block relay with a min-height
 444 2016-09-29T19:03:52  <jonasschnelli> NODE_PRUNENETWORK or something
 445 2016-09-29T19:03:57  <sipa> there have been multiple ideas around that
 446 2016-09-29T19:04:35  <petertodd> IMO whatever we do, we should recognise that w/ segwit's larger blocks we can expect a lot of full nodes to run out of disk space quite soon
 447 2016-09-29T19:04:35  <sipa> the easiest is just to add a flag that says you relay valid blocks and transactions, but not historical blocks more than a few deep
 448 2016-09-29T19:04:35  <jonasschnelli> I guess people slowly start to prune the blockchain to a max of 80GB or similar... but I guess not everyone is aware of the fact that you don't relay then
 449 2016-09-29T19:04:36  <wumpus> it would be nice to support more than one range, e.g. also archive nodes that host part of the old blocks
 450 2016-09-29T19:04:49  <sipa> it becomes harder when you want multiple ranger
 451 2016-09-29T19:04:57  <petertodd> do we have a reason for more than one range?
 452 2016-09-29T19:05:00  <jonasschnelli> We could introduce another message type... blockrange  or so
 453 2016-09-29T19:05:01  <sipa> it becomes even harder when you want to support sharding in an efficient way
 454 2016-09-29T19:05:08  <wumpus> I'm not sure why itb ecomes hard, just add a query message that returns what ranges are supported
 455 2016-09-29T19:05:11  <petertodd> sipa: what do you mean by sharding exactly?
 456 2016-09-29T19:05:25  <sipa> petertodd: you'd configure your node to maintain a certain % of blocks
 457 2016-09-29T19:05:45  <jonasschnelli> wumpus: query, yes, why not, or just inform like we do with sendheaders
 458 2016-09-29T19:05:47  <petertodd> sipa: see, given that the bitcoin protocol can't be safely sharded right now, I think we can safely say that we don't need to support sharding in block relay yet
 459 2016-09-29T19:05:50  *** Chris_Stewart_5 has joined #bitcoin-core-dev
 460 2016-09-29T19:05:55  <petertodd> sipa: doing so might even be dangerous if people start using it
 461 2016-09-29T19:05:59  *** cdecker has joined #bitcoin-core-dev
 462 2016-09-29T19:06:04  <sipa> petertodd: not in block relay
 463 2016-09-29T19:06:08  <sipa> petertodd: for block archival
 464 2016-09-29T19:06:16  <petertodd> sipa: but why shard vs. keep ranges?
 465 2016-09-29T19:06:23  <petertodd> sipa: (ranges of full blocks)
 466 2016-09-29T19:06:25  <luke-jr> BitTorrent already does this. Surely we can learn from that?
 467 2016-09-29T19:06:38  <petertodd> luke-jr: I don't think so - bittorrent is a very different problem than bitcoin
 468 2016-09-29T19:06:38  <wumpus> this is for letting other peers know what ranges of blocks are hosted, I don't think this should affect releay
 469 2016-09-29T19:06:41  <sipa> so, i've been running statistics on what block depths are being requested from nodes
 470 2016-09-29T19:06:47  <luke-jr> petertodd: learn from it, not use it directly
 471 2016-09-29T19:07:01  <luke-jr> petertodd: BitTorrent's problem isn't very different from IBD
 472 2016-09-29T19:07:08  <jonasschnelli> sipa: interesting.. do you have the stats public available somewhere
 473 2016-09-29T19:07:16  <jonasschnelli> I wanted to do this a long time
 474 2016-09-29T19:07:20  <petertodd> luke-jr: so, the thing is bittorrent has the problem of a diverse set of files, we just don't have that problem and can optimise differently because everyone needs access t othe same set of data
 475 2016-09-29T19:08:03  <sipa> there are something like 4 meaningful 'ranges' 1) the top 2 blocks (just relay) 2) up to ~2500 blocks deep... requested very often 3) up to ~10000 deep... requested a few times more than the next range 4) the rest
 476 2016-09-29T19:08:05  *** veleiro has joined #bitcoin-core-dev
 477 2016-09-29T19:08:15  <wumpus> otoh bittorrent has a fixed block size :)
 478 2016-09-29T19:08:22  <sipa> wumpus: so do we *ducks*
 479 2016-09-29T19:08:38  *** jnewbery has joined #bitcoin-core-dev
 480 2016-09-29T19:08:38  <petertodd> sipa: that probably corresponds to how long people leave their nodes offline :)
 481 2016-09-29T19:08:51  <btcdrak> inb4 Bittorrent XT
 482 2016-09-29T19:08:51  *** dgenr8 has joined #bitcoin-core-dev
 483 2016-09-29T19:09:04  <sipa> jonasschnelli: they're not available, and the ranges i gave above are just from me quickly glancing over the result
 484 2016-09-29T19:09:04  <petertodd> btcdrak: I use Bittorrent Unlimited myself
 485 2016-09-29T19:09:05  <jonasschnelli> What about fingerprinting issued in conjunction with available ranges?
 486 2016-09-29T19:09:16  <jonasschnelli> *issues
 487 2016-09-29T19:09:20  <petertodd> jonasschnelli: make them powers of two?
 488 2016-09-29T19:09:33  <sipa> well 4 ranges can be done with 2 service bit flags
 489 2016-09-29T19:09:45  <sipa> gmaxwell: you've worked on these ideas before, comments?
 490 2016-09-29T19:09:47  <jonasschnelli> But would that work with the flexible pruning option based on MB?
 491 2016-09-29T19:10:05  <petertodd> jonasschnelli: sure, just find the biggest range less than the pruning amount
 492 2016-09-29T19:10:12  <sipa> jonasschnelli: you'd change your service bits on the fly
 493 2016-09-29T19:10:20  <wumpus> why would the ranges need to be in the flags?
 494 2016-09-29T19:10:30  <gmaxwell> sorry, I missed that the meeting started.
 495 2016-09-29T19:10:32  <jonasschnelli> Yes. Why? Better add an explicit message for the range
 496 2016-09-29T19:10:34  <sipa> how would you otherwise discover what nodes to connect to?
 497 2016-09-29T19:10:45  <sipa> just randomly try?
 498 2016-09-29T19:10:45  <petertodd> jonasschnelli: oh right, you mean if MB != blocks... sorry.
 499 2016-09-29T19:10:49  <wumpus> I think you'll need a service flag to show support for the protocol, but not what ranges you have
 500 2016-09-29T19:10:52  <jonasschnelli> query or inform the other node if proto-ver > NODE_PRUNENETWORK
 501 2016-09-29T19:11:00  <wumpus> well that can be negotiated later, like bittorrent does I guess
 502 2016-09-29T19:11:08  <sipa> wumpus: well you do want addr messages to contain this information
 503 2016-09-29T19:11:18  <wumpus> I doubt bitcoin has 'service flags' in its tracker what blocks nodes have
 504 2016-09-29T19:11:18  <petertodd> sipa: so the nice thing about bitcoin, is just randomly try will probably work fairly often due to the low number of ranges out there
 505 2016-09-29T19:11:26  <wumpus> as that changes all the time anyhow
 506 2016-09-29T19:11:29  <gmaxwell> I was strongly of the view that we needed to signal at least two ranges. Sipa's latest measurements make me think at least three are needed.
 507 2016-09-29T19:11:31  <wumpus> s/bitcoin/bittorrent/
 508 2016-09-29T19:11:37  <jonasschnelli> I think informing other nodes ranges over addr is another thing...
 509 2016-09-29T19:11:46  <jonasschnelli> A first step would be a information after connect
 510 2016-09-29T19:11:55  <wumpus> yes, addr is another thing
 511 2016-09-29T19:12:22  <gmaxwell> I think ranges in service bits are no big deal, the harder question is what to do about the history. having nodes with 150GB of history in order to serve the last range is not very viable.
 512 2016-09-29T19:12:23  <wumpus> could be done later if an efficient way is needed to *locate* peers with certain ranges
 513 2016-09-29T19:12:31  <wumpus> but that seems premature optimization
 514 2016-09-29T19:12:36  <gmaxwell> We will need to redo addr sometime relatively soon in any case, as our messages are not compatible with HS-NG.
 515 2016-09-29T19:12:52  <petertodd> gmaxwell: oh, you mean Tor's new hidden services standard right?
 516 2016-09-29T19:12:56  <gmaxwell> petertodd: yes.
 517 2016-09-29T19:13:02  <gmaxwell> (also I2P though thats not new)
 518 2016-09-29T19:13:18  <wumpus> I think the number of ranges should be variable
 519 2016-09-29T19:13:24  <wumpus> redesigning addr is a different topic
 520 2016-09-29T19:13:35  <wumpus> also necessary, but again, doesn't need to be on one heap
 521 2016-09-29T19:13:45  <gmaxwell> wumpus: when I'm saying ranges I am specifically referring to the top-N zomes.
 522 2016-09-29T19:14:14  <petertodd> well, so if we add service bits for recent history ranges, that should be possible to implement as a separate feature to archival history ranges, and it'd be a big first step
 523 2016-09-29T19:14:22  <wumpus> I think it should be possible to, say, only host the first 20GB of blocks
 524 2016-09-29T19:14:37  <jonasschnelli> historic only nodes
 525 2016-09-29T19:14:37  <wumpus> I don't see why it should be restricted to only recent history
 526 2016-09-29T19:14:38  <petertodd> I don't think it's likely we'll see the two different features collide, so maybe implement recent history ranges first
 527 2016-09-29T19:14:55  <wumpus> or I mean first 20GB + last 144 blocks
 528 2016-09-29T19:15:07  <gmaxwell> For history storage, I was previously working on a proposal where nodes could signal a small (32 bit) seed and a size and from that everyone would know what parts of the history they would store.  I was so far unable to unify two different schemes, one which was computationally efficient to figure out who had what, and one which never required a peer to fetch a block it had previously deleted.
 529 2016-09-29T19:15:15  <sipa> so very quick breakdown: out of 7M requested blocks, 100k were for the tip, range 2-2500 has around 200-2000 requests per block, and from 10000 to genesis deep there are around 20 per block
 530 2016-09-29T19:16:19  <gmaxwell> I think for now we should not worry about the old history part and only worry about Top-n vs everything, as that fits into the pruning we already have and can be accomplished purely with service bits.
 531 2016-09-29T19:16:22  <wumpus> the bittorrent problem is different in that there the goal of each node is to have everything
 532 2016-09-29T19:16:25  <petertodd> so a social consideration here, is we can think in terms of recent history as "if there's a flaw, how much would we ever reorg w/o just saying bitcoin has failed?"
 533 2016-09-29T19:16:44  <gmaxwell> petertodd: thats partly why we have the 288 block maximum amount of pruning.
 534 2016-09-29T19:17:13  <petertodd> gmaxwell: indeed, and that's only two days...
 535 2016-09-29T19:17:36  <jonasschnelli> Using multiple service bits for 4 ranges seems to be a hackish-design IMO
 536 2016-09-29T19:17:49  <gmaxwell> at 100 blocks any reorg will _necessarily_ cause unrecoverable losses. So 288 basically gives a day plus an extra day for overhead.
 537 2016-09-29T19:17:50  <petertodd> there's also a natural time criteria from how the difficulty adjustments reduce your resistance to 51% attack - if your node is offline longer, the minimum attacker size to fool you goes down
 538 2016-09-29T19:17:52  <sipa> strangely enough: i see much more requests around 1000 deep than around 100 deep
 539 2016-09-29T19:18:04  <gmaxwell> jonasschnelli: I don't see anything hackish.
 540 2016-09-29T19:18:05  <wumpus> jonasschnelli: I also think it's a strange use of service bits
 541 2016-09-29T19:18:07  <jonasschnelli> I'd prefere using a single service bit to state pruned blockchain and then a new message (or append something to version?)
 542 2016-09-29T19:18:24  <petertodd> sipa: probably because people don't turn their nodes on and off every day
 543 2016-09-29T19:18:25  <gmaxwell> sipa: you probably want to filter out the bitnodes spider, as I believe it requests a block to check the node is working.
 544 2016-09-29T19:18:35  <sipa> gmaxwell: ah.
 545 2016-09-29T19:18:53  <gmaxwell> petertodd: someone who hasn't turned their node on will request all of 0 to -1000. so it will not make 1000 greater.
 546 2016-09-29T19:19:07  <gmaxwell> jonasschnelli: NAK.
 547 2016-09-29T19:19:07  <petertodd> gmaxwell: oh! I didn't know we did that
 548 2016-09-29T19:19:09  <sipa> i'm a bit surprised people think there is no need to have the available block ranges indicated in addr messages
 549 2016-09-29T19:19:30  <sipa> (whether through service bits, or some extension)
 550 2016-09-29T19:19:30  <jonasschnelli> I think there is a need... but it could be a second step
 551 2016-09-29T19:19:38  <wumpus> jonasschnelli: appending to version should be unnecessary, that's also a hack :)
 552 2016-09-29T19:19:50  <sipa> jonasschnelli: if it's a second step, we need to extend addr, and the whole management of addresses
 553 2016-09-29T19:19:51  <jonasschnelli> Okay. Agree. What about a new message type?
 554 2016-09-29T19:19:55  <jonasschnelli> blockrange
 555 2016-09-29T19:19:58  <sipa> jonasschnelli: you don't understand.
 556 2016-09-29T19:20:18  <gmaxwell> jonasschnelli: look at pieter's request figures, if nodes are effectively forced to go to peers that have everything whenever they connect becuase if they don't know they'll be able to fetch any blocks at all, then it will put lots more load on them.. causing people to stop offering blocks... causing more pressure on what remains.
 557 2016-09-29T19:20:56  <sipa> jonasschnelli: the point of having it in service bits is so nodes can find peers that have the range they need
 558 2016-09-29T19:20:58  <wumpus> but addr information gets old really fast
 559 2016-09-29T19:21:17  <sipa> wumpus: much less so with feeler connections
 560 2016-09-29T19:21:26  <wumpus> nodes may dynamically change what blocks they have, so there will always be cases of nodes connecting and realizing they have nothing to offer each other
 561 2016-09-29T19:21:27  <jonasschnelli> Okay. I see the point.
 562 2016-09-29T19:21:29  <sipa> (presumably, i don't have numbers)
 563 2016-09-29T19:21:54  <wumpus> just like currently nodes will try to connect into black holes that no longer host a node
 564 2016-09-29T19:22:24  <petertodd> so another interesting thing here is that ranges are queried linearly - you download blocks in a roughly linear fashion - so we could take advantage of that by making sure that nodes with one range keep track of nodes with adjacent ranges
 565 2016-09-29T19:22:29  <wumpus> sipa: sure, feeler connections make it somewhat better
 566 2016-09-29T19:22:35  <gmaxwell> wumpus: yes, sometimes the data is wrong. But there is a big difference between having 80% of the nodes on the network giving you no idea if they'll be useful at all until after you connect, vs a suggestion that might sometimes be wrong.
 567 2016-09-29T19:22:40  *** Giszmo has joined #bitcoin-core-dev
 568 2016-09-29T19:22:41  <wumpus> but I don't think addr is a very up-to-date information source
 569 2016-09-29T19:22:46  <petertodd> thus, as you sync the first time, ask nodes with the range you're syncing at this moment for the next range you need
 570 2016-09-29T19:23:08  <luke-jr> wumpus: if ranges are deterministic, they don't need to be up to date
 571 2016-09-29T19:23:15  <sipa> petertodd: yes, any sharding plan wouldn't randomly distribute the kept blocks, but keep randomly distributed ranges
 572 2016-09-29T19:23:32  <gmaxwell> wumpus: I don't know if you realize that sipa and I are not thinking in terms of absolute ranges here. but nodes saying "I keep the last 288" or "I keep the last 2016" or "I have all of history".
 573 2016-09-29T19:23:38  <wumpus> gmaxwell: but indeed this is a different problem from the bittorrent problem where everyone's goal is to have everything
 574 2016-09-29T19:23:56  <sipa> gmaxwell: well that's sharding... maybe that is something to postpone for later
 575 2016-09-29T19:24:01  <petertodd> sipa: sure, I'm more talking about how the linearity affects the network p2p design - prefentially peering with peers with the adjacent range may even be a reasonable design
 576 2016-09-29T19:24:03  <luke-jr> wumpus: eh, everyone needs to get everything
 577 2016-09-29T19:24:03  <wumpus> there, nodes can just connect randomly and have a high change the other nodes has something to offer them
 578 2016-09-29T19:24:04  <gmaxwell> wumpus: and I wouldn't expect that data to go out of date fast.. pretty much only when nodes go up and down.
 579 2016-09-29T19:24:04  <sipa> oh, nvm, i'm misreading
 580 2016-09-29T19:24:20  <wumpus> luke-jr: only initially
 581 2016-09-29T19:24:28  <luke-jr> oh, I see the distinction
 582 2016-09-29T19:24:41  <wumpus> luke-jr: bittorrent nodes don't throw away blocks, generally
 583 2016-09-29T19:24:53  <luke-jr> f(best-height, seed-in-addr) -> ranges
 584 2016-09-29T19:25:12  <gmaxwell> for the spreading the history around, as mentioned I came up with concrete schemes (based on consistent hashes) that have nice properties.
 585 2016-09-29T19:25:30  <sipa> i wonder whether we need to have that in the first go at this
 586 2016-09-29T19:26:04  <jonasschnelli> I think a first simple solution that allow to extend it further would be appriciated.
 587 2016-09-29T19:26:10  <sipa> even just having serve-everything and server-the-last-288-and-relay-at-tip would be a good addition
 588 2016-09-29T19:26:15  <wumpus> making the ranges deterministic makes some sense, on the other hand, it does restrict the flexibilty of nodes to choose what ranges they host, it means everything has to be got right in first try
 589 2016-09-29T19:26:23  <gmaxwell> sipa: thats what I am saying.
 590 2016-09-29T19:26:27  <jonasschnelli> sipa: agree
 591 2016-09-29T19:26:30  <gmaxwell> I do not think we can do better immediately anyways.
 592 2016-09-29T19:26:45  <sipa> 21:18:07 < jonasschnelli> I'd prefere using a single service bit to state pruned blockchain and then a new message (or append something to version?)
 593 2016-09-29T19:26:47  <gmaxwell> sipa: though your latest figures suggest that the 2016 depth is important too.
 594 2016-09-29T19:26:48  <sipa> 21:19:07 < gmaxwell> jonasschnelli: NAK.
 595 2016-09-29T19:27:10  <petertodd> if nodes attempt to maintain a few connections to peers that have the next range after they have, maybe it doesn't matter exactly what the ranges actually are? any given node would have a few connections to the next range, and anyone syncing from them could ask for those connections
 596 2016-09-29T19:27:15  <gmaxwell> sipa: my understanding of jonasschnelli comment was there should be a bit that says "I relay blocks but don't have history" I am NAK on that.
 597 2016-09-29T19:27:36  <wumpus> as there is no scope for later optimization, because all nodes have to agree what ranges are implied
 598 2016-09-29T19:27:40  <jonasschnelli> We could add a service bit that says "I relay only the last 288 blocks"
 599 2016-09-29T19:27:50  <wumpus> jannes: yes that would be the initial idea
 600 2016-09-29T19:27:54  <wumpus> jonasschnelli*
 601 2016-09-29T19:28:08  <sipa> gmaxwell: how is that different from what i suggested?
 602 2016-09-29T19:28:11  <sipa> 21:26:10 < sipa> even just having serve-everything and server-the-last-288-and-relay-at-tip would be a good addition
 603 2016-09-29T19:28:21  <jonasschnelli> I think my initial idea with the general pruning sevice bit and a new message type is to complex and inflexible
 604 2016-09-29T19:28:28  <gmaxwell> jonasschnelli: yes, that would be better, though pieter's data suggests that there are a LOT of requests at 1000. I think if I had that data I would have been suggesting the maximum pruning should be 2016, and then had the bit at that dep.
 605 2016-09-29T19:28:50  <gmaxwell> sipa: the ability to relay blocks at depth -10.
 606 2016-09-29T19:29:04  <sipa> gmaxwell: less than 2% of blocks requested from my node are at the tip
 607 2016-09-29T19:29:22  <sipa> (but the tip is still 100x more frequent than any other individual depth)
 608 2016-09-29T19:29:50  <sipa> gmaxwell: "a service bit to indicate pruned blockchain" implies you can serve 288 deep :)
 609 2016-09-29T19:30:08  <petertodd> gmaxwell: re: maximum pruning depth, it's reasonable for that to be a similar % of the total data that storing the UTXO set takes - if you have 10GB of UTXO, 2GB of block data isn't a big change
 610 2016-09-29T19:30:11  <wumpus> yes, you could define it as that
 611 2016-09-29T19:30:15  <gmaxwell> I don't think there is any remaining disagreement on using bit(s) to signal I have a top-n.  But I have some doubt on N. it needs to capture the largest amount of the block realy bandwidth without being unduely pruning incompatible.
 612 2016-09-29T19:30:37  <wumpus> 288 is the minimum pruning amount in bitcoin core already so it'd be a valid choice
 613 2016-09-29T19:30:44  <morcos> as a first pass, i wonder if you preferentially downloaded from pruned peers whenever you were behind by less than 288 blocks, that would take enough load of peers serving full history?
 614 2016-09-29T19:30:50  <gmaxwell> morcos: absolutely.
 615 2016-09-29T19:30:57  *** aureianimus has quit IRC
 616 2016-09-29T19:30:57  *** aureianimus_ has joined #bitcoin-core-dev
 617 2016-09-29T19:31:01  <jonasschnelli> Good idea
 618 2016-09-29T19:31:09  <wumpus> yes, that would make sense
 619 2016-09-29T19:31:23  <gmaxwell> unfortunately, sipa's data suggests that 288 sheds less traffic than measurements years ago suggested.
 620 2016-09-29T19:31:43  <sipa> maybe i should compute statistics in bytes rather than blocks
 621 2016-09-29T19:31:45  <morcos> gmaxwell: it wasn't clear to me what the integral from 1 to 288 was compared to 288 to inf
 622 2016-09-29T19:31:46  <wumpus> well it is a compromise
 623 2016-09-29T19:31:53  <wumpus> putting the threshold higher makes some peers completely useless
 624 2016-09-29T19:31:57  <sipa> to see what percentage of bandwidth is needed in 1-288
 625 2016-09-29T19:31:58  <wumpus> which reduces morcos 's argument
 626 2016-09-29T19:32:04  <jonasschnelli> Yes. I guess you convinced me to use two service bits then. -288 and -2016
 627 2016-09-29T19:32:16  <gmaxwell> which is why it might be useful to use two bits and be able to signal 1-288, 1-2016... and perhaps start encouraging people to not prune shorter than 2016.
 628 2016-09-29T19:32:37  <sipa> i think we're getting into a design discussion here
 629 2016-09-29T19:32:47  <sipa> my number are very premature and not well analysed
 630 2016-09-29T19:32:50  <wumpus> it'd also be possible to add a 288-flag now, and then consider a 2016 flag later
 631 2016-09-29T19:32:54  <gmaxwell> sipa: indeed, thought that was the input you requested from me.
 632 2016-09-29T19:32:57  <morcos> wumpus: yes, thats what i'm saying
 633 2016-09-29T19:32:59  <gmaxwell> wumpus: yes! indeed.
 634 2016-09-29T19:33:03  <jonasschnelli> Agree with wumpus
 635 2016-09-29T19:33:04  <wumpus> if it turns out to be necessary
 636 2016-09-29T19:33:10  <petertodd> wumpus: ACK
 637 2016-09-29T19:33:15  <sipa> yes, i think just a 1-288 one seems useful
 638 2016-09-29T19:33:16  <wumpus> good :)
 639 2016-09-29T19:33:16  <jonasschnelli> Start with a simple tip-288 relay, and get some experience
 640 2016-09-29T19:33:23  <gmaxwell> wumpus: it looks pretty clearly necessary but no need to do everything at once.
 641 2016-09-29T19:33:30  <petertodd> wumpus: basically advice is, turn your node on at least once every two days
 642 2016-09-29T19:33:54  <wumpus> petertodd: yes
 643 2016-09-29T19:33:55  <gmaxwell> petertodd: we really should have cron mode for the daemon where it just syncs up and shuts off. :P
 644 2016-09-29T19:34:06  <gmaxwell> bitcoind -oneshot
 645 2016-09-29T19:34:08  <gmaxwell> :P
 646 2016-09-29T19:34:17  <petertodd> gmaxwell: heh, that's not a crazy idea - I'd use it on my laptop
 647 2016-09-29T19:34:18  <jonasschnelli> didn't we once had a proposal for the pause option?
 648 2016-09-29T19:34:24  <wumpus> right, there's a flag that quits after reindex, but none that exits after sync
 649 2016-09-29T19:34:31  <wumpus> would be easy to add tho
 650 2016-09-29T19:34:41  <morcos> we could just ask for the utxo set, shoudl we discuss ideas how to do that
 651 2016-09-29T19:34:51  <CodeShark> ^ yes :)
 652 2016-09-29T19:34:58  <petertodd> make -oneshot run in the foreground with a progress bar :)
 653 2016-09-29T19:34:59  <wumpus> without utxo commitment that's a no-go
 654 2016-09-29T19:35:05  <morcos> thanks codeshark
 655 2016-09-29T19:35:20  <petertodd> wumpus: +1
 656 2016-09-29T19:35:42  <gmaxwell> morcos: pointless when we were unable to get past the discussion for the security model change to not validate the past history based on proof of work.
 657 2016-09-29T19:35:48  <petertodd> and lets not underestimate how dangerous UTXO commitments can be - I'm very dubious about committing to the (U)TXO set more recently than maybe a month or two
 658 2016-09-29T19:35:51  <CodeShark> would be great to query utxo for quick sync, then go backwards in time fetching blocks to increase security...but yes, this is a design discussion
 659 2016-09-29T19:35:53  <morcos> i was making a joke, sorry
 660 2016-09-29T19:36:18  <CodeShark> alas, quick sync doesn't look feasible in the nearterm
 661 2016-09-29T19:36:20  <wumpus> ok, next topic?
 662 2016-09-29T19:36:36  <gmaxwell> but since that was brought up... Can we talk about removing checkpoints?
 663 2016-09-29T19:36:55  <wumpus> #topic removing checkpoints
 664 2016-09-29T19:36:59  <sipa> what % of transactions are before the last checkpoint
 665 2016-09-29T19:37:01  <sipa> does anyone know?
 666 2016-09-29T19:37:03  <morcos> someone should write up a design proposal for that to be evaluated
 667 2016-09-29T19:37:18  <gmaxwell> Right now they're used for two things, preventing header flooding with low difficulty headers; and skipping signatures in earlier blocks.
 668 2016-09-29T19:37:21  <petertodd> gmaxwell: just removing checkpoints, or assuming sigs are valid if buried deep enough?
 669 2016-09-29T19:37:27  <sipa> gmaxwell: and 3) estimating progress
 670 2016-09-29T19:37:45  <wumpus> keeping something for estimating progress would make sense
 671 2016-09-29T19:37:47  <sipa> i think 1) remains needed and 3) remains useful
 672 2016-09-29T19:37:50  <wumpus> that doesn't need to be checkpoints
 673 2016-09-29T19:38:06  <gmaxwell> because very few percentage of the transactions are below the checkpoint .. since libsecp256k1 (and I expect the checkqueue)-- my point two is basically pointless, and I think it could just be removed
 674 2016-09-29T19:38:29  <gmaxwell> I think on a desktop it only adds 15-20 minutes to the sync.
 675 2016-09-29T19:38:29  <petertodd> gmaxwell: I'd ACK simply removing checkpoints entirely; I'm not happy to see them replaced with another scheme to skip sig checking
 676 2016-09-29T19:38:33  <wumpus> a block-height-to-relative-difficulty map would have much less of a stigma
 677 2016-09-29T19:38:46  <wumpus> eh, verification difficulty that is
 678 2016-09-29T19:38:52  <sipa> gmaxwell: really?
 679 2016-09-29T19:38:54  <gmaxwell> petertodd: I think we could remove CP from reason two without implementing the replcement.
 680 2016-09-29T19:39:06  <gmaxwell> petertodd: morcos is right that needs a design proposal outside of the meeting.
 681 2016-09-29T19:39:12  <sdaftuar> i'm a bit confused about how to think about checkpoints for signature skipping
 682 2016-09-29T19:39:22  <gmaxwell> sipa: I benchmarked before but I'm going off of memory, I could be wildly wrong. I will test again if there is interest.
 683 2016-09-29T19:39:52  *** jl2012 has quit IRC
 684 2016-09-29T19:39:52  <jonasschnelli> Removing checkpoints would slow down (maybe insignificant) a scan in a possible SPV hybrid mode?
 685 2016-09-29T19:39:54  <gmaxwell> For reason (1) the only answer I have is that I think we should proposal a bit to perpetually increase the minimum difficulty from 1 to something else.
 686 2016-09-29T19:40:00  <sdaftuar> for instance the recent ISM change caused us to do less validation for certain blocks in our history (blocks in a softfork between the 75% and 95% thresholds)
 687 2016-09-29T19:40:10  <sipa> jonasschnelli: SPV mode won't validate *anything* at all
 688 2016-09-29T19:40:24  *** mturquette has quit IRC
 689 2016-09-29T19:40:28  <gmaxwell> (with a checkpoint like bypass of that new rule, for existing blocks that break it) As little as 100,000 would eliminate the header flooding vulenrablity.
 690 2016-09-29T19:40:33  <jonasschnelli> Yes. But assume we would add an SPV hibrid mode in oder to received payment during IBD
 691 2016-09-29T19:40:50  <jonasschnelli> One would need to download 400k headers without a checkpoint at h400k
 692 2016-09-29T19:40:53  <luke-jr> maybe checkpoints should just be disabled by default before complete removal?
 693 2016-09-29T19:41:00  <sipa> jonasschnelli: i think you're confused
 694 2016-09-29T19:41:07  <gmaxwell> for Sipa's (3) reason for 'checkpoints' I don't give a darn, use chicken bones for progress estimation for all I care. :P it's historical accident that checkpoints and progress use the same data structure.
 695 2016-09-29T19:41:11  *** jl2012 has joined #bitcoin-core-dev
 696 2016-09-29T19:41:24  <morcos> gmaxwell: :) +1
 697 2016-09-29T19:41:28  <wumpus> gmaxwell: yes, my point too
 698 2016-09-29T19:41:33  *** jl2012_ has joined #bitcoin-core-dev
 699 2016-09-29T19:41:34  <sipa> gmaxwell: agree, those could be completely separated
 700 2016-09-29T19:41:36  <petertodd> gmaxwell: ACK checken bones
 701 2016-09-29T19:41:36  <gmaxwell> Might as well fit a cubic spline to the height vs txn count... and store the parameters.
 702 2016-09-29T19:41:53  <wumpus> right
 703 2016-09-29T19:42:02  *** jl2012 has left #bitcoin-core-dev
 704 2016-09-29T19:42:09  <petertodd> gmaxwell: heh, if we do that with floating point math that has the advantage that it _can't_ be used for consensus :)
 705 2016-09-29T19:42:18  * sipa now remembers a song our student organization wrote to the melody of staying alive, called 'cubic spline'
 706 2016-09-29T19:42:21  <gmaxwell> so my proposal, if there is interest, is that I'll measure the performance impact of removing the signature skippingentirely (esp post checkqueue). And if it's not awful, we'll remove.
 707 2016-09-29T19:42:37  <wumpus> +1
 708 2016-09-29T19:42:44  <sipa> gmaxwell: i'm unconvinced
 709 2016-09-29T19:42:52  <wumpus> it doesn't hurt to benchmark
 710 2016-09-29T19:42:55  *** mturquette has joined #bitcoin-core-dev
 711 2016-09-29T19:43:00  <gmaxwell> and maybe I'll tender a proposal to up the minimum difficulty, but I'd like to know what people think about that.
 712 2016-09-29T19:43:04  <wumpus> measuring is always better than making assumptions
 713 2016-09-29T19:43:11  <sipa> with a replacement for sig skipping that isn't based on checkpoints we could significantly improve things
 714 2016-09-29T19:43:22  *** zmanian___ has quit IRC
 715 2016-09-29T19:43:39  <petertodd> sipa: I don't think such a replacement can exist without changing the security assumptions; I'd *rather* have checkpoints than trusting hashing power for that
 716 2016-09-29T19:43:44  <sipa> the last checkpoint currently is very old for the very reason that we've been planning to replace it
 717 2016-09-29T19:43:47  <gmaxwell> sipa: would you like to help work on a proposal for that? it has been controversial in the past. I'd like to do something good, because otherwise imprudent attempts will be adopted instead.
 718 2016-09-29T19:44:15  <sipa> so it's unfair to use the "the last checkpoint is old" as a given; it's something we've affected indirectly
 719 2016-09-29T19:44:19  <petertodd> sipa: though what checkpoints should do is say "Something big has changed; you can disable checkpoints with --no-checkpoints, but you should find out what this means before doing so."
 720 2016-09-29T19:44:29  <gmaxwell> (for example Bitcoin Classic's current behavior simply looks at block header timestamps and ignores signatures when they're more than 24 hours (*par) old by the local clock. It's easily exploited and makes me sad.
 721 2016-09-29T19:44:40  <sipa> petertodd: it's my opinion that on a timescale of months, it doesn't matter
 722 2016-09-29T19:44:53  <sipa> IF you can guarantee it's actually a timescale of months
 723 2016-09-29T19:44:55  <wumpus> yes that makes me sad too
 724 2016-09-29T19:44:58  <petertodd> sipa: on a timescale of months, checkpoints shouldn't matter either...
 725 2016-09-29T19:45:06  <wumpus> anything based on time seems very brittle
 726 2016-09-29T19:45:27  <sipa> petertodd: look at the current hashrate; what's 3 months worth of chain work at that hashrate
 727 2016-09-29T19:45:31  <petertodd> wumpus: and anything based on work isn't much better if you're running an old client, and mining has advanced significantly
 728 2016-09-29T19:45:37  <jonasschnelli> sipa: I (hope) I'm not confused. If we would add a SPV hybrid mode directly fetch blocks at the tip (in order to received payments), no available checkpoint would result in downloading all headers *losing* maybe 3-4mins before you can start using SPV... minor issue though, I agree
 729 2016-09-29T19:45:38  <petertodd> sipa: that assumes you know what the current hashrate is
 730 2016-09-29T19:45:41  <gmaxwell> wumpus: the prior proposals were based on work,  e.g. skip if the best chain you see dominates the next conflicted chain at that hight by N months of work.
 731 2016-09-29T19:45:44  <Chris_Stewart_5> gmaxwell: How have we solved the problem that checkpoints were originally created for? You have an excerpt in here: https://en.bitcoin.it/wiki/Bitcoin_Core_0.11_(ch_5):_Initial_Block_Download#Checkpoints
 732 2016-09-29T19:45:45  <petertodd> sipa: your node might be surrounded by sybils
 733 2016-09-29T19:45:52  <gmaxwell> wumpus: with a 'minimum total work' coded in as part of the release proces.
 734 2016-09-29T19:46:00  <sipa> Chris_Stewart_5: headers first sync
 735 2016-09-29T19:46:08  <sipa> Chris_Stewart_5: 0.10
 736 2016-09-29T19:46:08  <gmaxwell> Chris_Stewart_5: headers first sync.
 737 2016-09-29T19:46:09  <wumpus> gmaxwell: right. Well, at the least it should be measured whether such a change is really worth it.
 738 2016-09-29T19:46:16  <sipa> petertodd: yes, i know...
 739 2016-09-29T19:46:21  <sipa> so, let's measure.
 740 2016-09-29T19:46:25  <sipa> and discuss later
 741 2016-09-29T19:46:37  <gmaxwell> Chris_Stewart_5: and the signature skipping behavior in checkpoints was actually a result of a bug fixed years ago.. mlock being used on all allocations making script validation INSANELY slow.
 742 2016-09-29T19:46:41  <wumpus> so much of the verification overhead is looking up UTXOs
 743 2016-09-29T19:46:44  *** zmanian___ has joined #bitcoin-core-dev
 744 2016-09-29T19:46:44  <gmaxwell> sipa: okay.
 745 2016-09-29T19:46:47  <wumpus> something you'll not avoid
 746 2016-09-29T19:46:53  *** Lauda_ has joined #bitcoin-core-dev
 747 2016-09-29T19:47:20  <gmaxwell> Chris_Stewart_5: but then with chain growth we became dependant on it to keep sync times reasonable. but libsecp256k1 made signature validation >5x faster.
 748 2016-09-29T19:47:21  <wumpus> especially for recent blocks
 749 2016-09-29T19:47:30  *** Lauda has quit IRC
 750 2016-09-29T19:47:38  *** Lauda_ is now known as Lauda
 751 2016-09-29T19:47:38  <wumpus> if you do any benchmarking please look at the recent blocks, not the first N
 752 2016-09-29T19:47:38  *** Lauda has quit IRC
 753 2016-09-29T19:47:39  *** Lauda has joined #bitcoin-core-dev
 754 2016-09-29T19:48:04  <gmaxwell> wumpus: it's still a major speed up on existing blocks.
 755 2016-09-29T19:48:07  <sipa> in a side node: i've already updated my logging to measure bandwidth vs blockdepth instead of just count.
 756 2016-09-29T19:48:11  <Chris_Stewart_5> So header sync solves the attack of flooding disk space, but not having your entire network hijacked, correct?
 757 2016-09-29T19:48:29  *** harambea has joined #bitcoin-core-dev
 758 2016-09-29T19:48:32  <wumpus> Chris_Stewart_5: huh?
 759 2016-09-29T19:48:41  <wumpus> gmaxwell: sure, could be
 760 2016-09-29T19:48:42  <gmaxwell> Chris_Stewart_5: isolation can be resolved by simply knowing what the total work of the best chain was at release.
 761 2016-09-29T19:48:52  *** lesderid_ has joined #bitcoin-core-dev
 762 2016-09-29T19:49:00  <gmaxwell> Chris_Stewart_5: sorry, this was discussed prior times removing checkpoints had come up, I haven't completely described the background.
 763 2016-09-29T19:49:19  <Chris_Stewart_5> gmaxwell: Thanks for the explanation, i'll keep digging.
 764 2016-09-29T19:49:45  <wumpus> Chris_Stewart_5: ah, you mean being isolated and being fed a wrong chain, sorry I was imaginging some wacky things at having your network hijacked :)
 765 2016-09-29T19:51:01  *** lesderid has quit IRC
 766 2016-09-29T19:51:01  <wumpus> ok, next topic?
 767 2016-09-29T19:51:01  <gmaxwell> wumpus: just the "you got a faithful bitcoin core download but the attacker controls your network"... but that doesn't need a checkpoint to fix, a simple partitioning detction that knows the total work of the best chain at releast time is sufficient.
 768 2016-09-29T19:51:01  <gmaxwell> Thanks for the discussion.
 769 2016-09-29T19:51:13  <wumpus> #topic segwit against uncompressed keys or not
 770 2016-09-29T19:51:17  <wumpus> (10 minutes to go)
 771 2016-09-29T19:51:24  <wumpus> (9 minutes to go)
 772 2016-09-29T19:51:27  <petertodd> so to be clear, *just* segwit right?
 773 2016-09-29T19:51:30  <CodeShark> does anyone still use uncompressed keys?
 774 2016-09-29T19:51:33  <wumpus> yes, only segwit
 775 2016-09-29T19:51:39  <achow101> CodeShark: armory does
 776 2016-09-29T19:51:42  <luke-jr> seems uncontroversial
 777 2016-09-29T19:51:49  <petertodd> I'm happy to ACK that given just segwit
 778 2016-09-29T19:51:55  *** aureianimus_ has quit IRC
 779 2016-09-29T19:51:57  <achow101> having segwit enforce uncompressed keys would delay segwit adoption for armory users
 780 2016-09-29T19:52:01  *** aureianimus has joined #bitcoin-core-dev
 781 2016-09-29T19:52:03  <achow101> *compressed
 782 2016-09-29T19:52:05  <jl2012_> it's in #8499
 783 2016-09-29T19:52:09  <luke-jr> achow101: why? just compress them
 784 2016-09-29T19:52:13  <wumpus> gmaxwell: yes, though we had a lot of trouble with partitioning detection, I remember some code being stripped out and such. But anyhow, yes that's the better approach if it can be gotten to work.
 785 2016-09-29T19:52:22  <sipa> achow101: sigh, does armory still not do that?
 786 2016-09-29T19:52:30  <achow101> luke-jr: we have to change the whole wallet structure (it's still going to happen anyways)
 787 2016-09-29T19:53:34  <wumpus> gmaxwell: without too much false positives
 788 2016-09-29T19:53:34  <luke-jr> achow101: why?
 789 2016-09-29T19:53:34  <sipa> achow101: alan said somewhere in 2013 he was implementing it...
 790 2016-09-29T19:53:34  <achow101> alan's gone now..
 791 2016-09-29T19:53:34  <luke-jr> afaik the only downside to using compressed keys is it changes the address, which segwit is changing anyway
 792 2016-09-29T19:53:34  <CodeShark> it's not a very complicated change
 793 2016-09-29T19:53:34  <wumpus> armory still uses uncompressed keys?!
 794 2016-09-29T19:53:34  <luke-jr> there's no reason you'd need to change the wallet structure I can see
 795 2016-09-29T19:53:34  <wumpus> in any case this only applies to segwit, not to old transactions
 796 2016-09-29T19:53:34  <achow101> the plan is to have a new wallet structure with bip32 that supports segwit and compressed keys
 797 2016-09-29T19:53:41  <gmaxwell> wumpus: "you're partitioned until you see a header chain with at least work X" is a pretty simple critera. :P
 798 2016-09-29T19:53:44  <sipa> luke-jr: it had fixed size records in its wallet format for pubkeys
 799 2016-09-29T19:54:05  <sipa> achow101: well if a new wallet format is needed for segwit anyway, it doesn't matter right?
 800 2016-09-29T19:54:10  <gmaxwell> achow101: oh god please do not use uncompressed keys with segwit. why would you do that?
 801 2016-09-29T19:54:13  <luke-jr> sipa: zero-pad it?
 802 2016-09-29T19:54:35  <achow101> sipa: well no, we don't need a new wallet for segwit as it could still work with the old one with a little bit of hacking
 803 2016-09-29T19:54:48  <achow101> that was the original plan
 804 2016-09-29T19:54:48  <luke-jr> achow101: no less than compressed could
 805 2016-09-29T19:55:15  <luke-jr> sipa: or store the uncompressed key, and compress it at address-generation/signing
 806 2016-09-29T19:55:26  <gmaxwell> achow101: why cant the same hack that indicates segwit is in use indicate compressed.. you just chop off some bytes of the key pretty much.
 807 2016-09-29T19:55:43  <sipa> btw, uncompressed keys account for 0.7% of used keys in succesful sigs on the network (in the past 2 hours)
 808 2016-09-29T19:55:44  <gmaxwell> it could be done entirely inside the process that seralizes the segwit scriptpubkey.
 809 2016-09-29T19:55:54  *** Chris_Stewart_5 has quit IRC
 810 2016-09-29T19:56:06  <achow101> gmaxwell: idk. ask goatpig
 811 2016-09-29T19:56:06  <gmaxwell> achow101: okay
 812 2016-09-29T19:56:09  * michagogo pokes his head in belatedly
 813 2016-09-29T19:56:10  <CodeShark> I think we should encourage all wallets to use compressed keys - achow101, if you need help with this I'd be willing to help
 814 2016-09-29T19:56:25  <sipa> agree - we should help
 815 2016-09-29T19:56:27  <gmaxwell> yes, lots of people would be glad to help.
 816 2016-09-29T19:56:32  <sipa> instead of just yell
 817 2016-09-29T19:56:50  <gmaxwell> well I offered to help armory move off uncompressed keys to alan several times, including offering to pay to do it.
 818 2016-09-29T19:56:56  <gmaxwell> so please don't say anyone just yelled.
 819 2016-09-29T19:58:39  <CodeShark> I initially designed my account structures to only use compressed keys - but later added a compressed bit to support legacy stuff
 820 2016-09-29T19:59:06  <petertodd> CodeShark: what legacy stuff specifically? legacy armory users?
 821 2016-09-29T19:59:08  <wumpus> CodeShark: bah,it's kind of sad that to hear some things seem to be going back instead of forward :)
 822 2016-09-29T19:59:18  <CodeShark> yes, to support other wallets
 823 2016-09-29T19:59:27  *** anchow101 has joined #bitcoin-core-dev
 824 2016-09-29T19:59:27  <wumpus> it's time
 825 2016-09-29T19:59:44  *** lesderid_ has quit IRC
 826 2016-09-29T19:59:50  <CodeShark> but I think we really do need to prod all wallets to move to compressed keys
 827 2016-09-29T19:59:52  *** lesderid has joined #bitcoin-core-dev
 828 2016-09-29T19:59:53  *** anchow101 has quit IRC
 829 2016-09-29T20:00:07  <CodeShark> there's really no reason to continue to support uncompressed keys - other than perhaps some migration tools
 830 2016-09-29T20:00:15  <wumpus> #endmeeting
 831 2016-09-29T20:00:15  *** anchow101 has joined #bitcoin-core-dev
 832 2016-09-29T20:00:15  <lightningbot> Meeting ended Thu Sep 29 20:00:15 2016 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
 833 2016-09-29T20:00:15  <lightningbot> Minutes:        http://www.erisian.com.au/meetbot/bitcoin-core-dev/2016/bitcoin-core-dev.2016-09-29-19.01.html
 834 2016-09-29T20:00:15  <lightningbot> Minutes (text): http://www.erisian.com.au/meetbot/bitcoin-core-dev/2016/bitcoin-core-dev.2016-09-29-19.01.txt
 835 2016-09-29T20:00:15  <lightningbot> Log:            http://www.erisian.com.au/meetbot/bitcoin-core-dev/2016/bitcoin-core-dev.2016-09-29-19.01.log.html
 836 2016-09-29T20:00:15  <gmaxwell> CodeShark: as pieter notes, virutally nothing is already.
 837 2016-09-29T20:00:29  <gmaxwell> 0.7% percent
 838 2016-09-29T20:00:38  <anchow101> Yes, help would be appreciated
 839 2016-09-29T20:01:05  <wumpus> well supporting it in consensus for the normal network keeps making sense, but segwit is just such a great oppertunity to get rid of it
 840 2016-09-29T20:01:31  <petertodd> wumpus: yeah, I don't see how we can remove backwards compatibility for it w/o confiscating funds, but no reason to not remove support in new addresses
 841 2016-09-29T20:01:40  <wumpus> petertodd: indeed
 842 2016-09-29T20:01:56  <gmaxwell> yes, thats why its important to get rid of now. otherwise I wouldn't care if action were taken n months later.
 843 2016-09-29T20:01:56  <luke-jr> if anything, we should be discussing whether to make it a consensus rule rather than a policy ;)
 844 2016-09-29T20:02:10  *** achow101_ has joined #bitcoin-core-dev
 845 2016-09-29T20:02:20  <gmaxwell> luke-jr: I like to but many people feel that addining an additional consensus rule for segwit now wouldn't be prudent.
 846 2016-09-29T20:02:37  <gmaxwell> making it non-standard is sufficient in my view, such that we'd be able to make it a consensus rule later.
 847 2016-09-29T20:02:51  <btcdrak> achow101 seems to be having connection problems
 848 2016-09-29T20:02:51  <luke-jr> sure
 849 2016-09-29T20:02:55  <achow101_> btcdrak: just a little bit. switching computers
 850 2016-09-29T20:02:57  <CodeShark> gmaxwell: having to deal with the additional case complicates implementations
 851 2016-09-29T20:02:59  <luke-jr> just saying the rule shouldn't be controversial itself really
 852 2016-09-29T20:03:02  <CodeShark> not very much, but still
 853 2016-09-29T20:03:07  <petertodd> gmaxwell: well, so long as we loudly warn that this is intended to become unspendable later if you bypass the standardness
 854 2016-09-29T20:03:27  <morcos> gmaxwell: if so, we should be as clear about it being not allowed now as if we were to make it a consensus rule now.
 855 2016-09-29T20:04:25  <gmaxwell> for those thinking that we have to verify all the old stuff for all time, that might be true for bitcoin core, but in the future I could imagine some implementations just not bothering to verify old stuff.
 856 2016-09-29T20:04:25  <gmaxwell> morcos: petertodd: agreed for sure.
 857 2016-09-29T20:04:34  <sdaftuar> petertodd: ntoe that it's standard to create an output with an uncompressed pubkey hash, as we can't detect the issue until a spend attempt (right?)
 858 2016-09-29T20:04:34  <jtimon> I agree with luke-jr on not seeing the controversy in it being consensus rule with the rest of segwit
 859 2016-09-29T20:04:43  <petertodd> gmaxwell: I don't mean verifying old stuff, I mean verifying new txs spending old coins
 860 2016-09-29T20:04:55  <petertodd> sdaftuar: yes, nothing we can do about that though
 861 2016-09-29T20:05:16  <gmaxwell> jtimon: well in the case of low-S which we also wanted to make a consensus rule, jl2012 discovered that there were corner conditions we wanted to think about more carefully before making it a consensus rule.
 862 2016-09-29T20:05:23  <sdaftuar> petertodd: right; just want to make sure we're all on the same page as i think communicating this widely/loudly is important
 863 2016-09-29T20:05:23  <petertodd> for example, we've made hybrid pubkeys non-standard, and given basically no-one's ever used them in production for anything I'd have no issues with making them unspendable in a soft-fork
 864 2016-09-29T20:05:43  <jtimon> gmaxwell: thanks
 865 2016-09-29T20:06:30  <gmaxwell> jtimon: I don't think the same applies to uncompressed keys, because the criteria there is even simpler. but the lowS reason is part of why we punted this collection of improvements to policy for now.
 866 2016-09-29T20:06:49  <jtimon> mhmm
 867 2016-09-29T20:06:57  <michagogo> I saw a movie that depicts a form of distributed/decentralized system, to avoid it getting shut down. Or in the words of the character that explains it, "everyone that logs on is a server". It's said to be "open source", but then that's explained as "anyone can edit the code, like Wikipedia.
 868 2016-09-29T20:07:18  <achow101_> so if anyone wants to help armory with segwit support, bip32, compressed keys, we accept PRs. All our work happens in the dev branch, not master
 869 2016-09-29T20:07:19  <michagogo> And the code is deployed when a majority of users approve it
 870 2016-09-29T20:07:42  *** veleiro has quit IRC
 871 2016-09-29T20:08:01  <wumpus> michagogo: heh, open source in some weird twisted mirror world
 872 2016-09-29T20:08:05  <gmaxwell> achow101: is there a IRC channel where things are discussed? E.g. where should I ask goatpig about compressed pubkeys in segwit.
 873 2016-09-29T20:08:11  <sipa> michagogo: i believe they're mistakingly not describing a computer network, but politics.
 874 2016-09-29T20:08:12  *** BakSAj has quit IRC
 875 2016-09-29T20:08:12  <achow101_> #bitcoin-armory
 876 2016-09-29T20:08:24  <jtimon> michagogo: that is, when sybil decides so...
 877 2016-09-29T20:08:50  <luke-jr> achow101_: meh, should just collapse that into #bitcoin-dev :p
 878 2016-09-29T20:09:27  <michagogo> And it's completely vulnerable to Sybil attacks…
 879 2016-09-29T20:09:30  *** cryptapus_ has quit IRC
 880 2016-09-29T20:09:43  <michagogo> Gah, lagging
 881 2016-09-29T20:09:57  <michagogo> Yeah
 882 2016-09-29T20:10:17  <michagogo> And of course, when the last user logs off, it doesn't just stop working
 883 2016-09-29T20:10:44  <michagogo> The sybil attackers are able to watch it dramatically implode with special effects, "graphical corruption" type stuff
 884 2016-09-29T20:11:50  <michagogo> And there's the obligatory "they're blocking all our foreign IPs" and that kind of stuff, with no explanation of who "they" are
 885 2016-09-29T20:13:45  <jtimon> so what was there a conclusion for the range service bits? nothing/top-288/everything?
 886 2016-09-29T20:13:45  <jtimon> what about the getrange message and "sharding"
 887 2016-09-29T20:13:45  <GitHub159> [bitcoin] laanwj opened pull request #8843: rpc: Handle `getinfo` client-side in bitcoin-cli w/ `-getinfo` (master...2016_09_getinfo_clientside) https://github.com/bitcoin/bitcoin/pull/8843
 888 2016-09-29T20:13:45  <wumpus> conclusion was to add one service bit: last-288-served
 889 2016-09-29T20:13:45  <wumpus> and maybe later one for last-1000-served
 890 2016-09-29T20:13:45  <jtimon> wumpus: I see, and leave the rest for later, thanks
 891 2016-09-29T20:13:45  <luke-jr> 1024 would be rounder. ☺
 892 2016-09-29T20:13:45  <wumpus> and a jackpot for whoever enabled both at once
 893 2016-09-29T20:13:51  <luke-jr> if you set both, does it mean last 288000? :P
 894 2016-09-29T20:14:16  *** aureianimus has quit IRC
 895 2016-09-29T20:14:22  <jtimon> would it be crazy to just have last-1024 without last-288 and just change prunning's default?
 896 2016-09-29T20:14:29  *** aureianimus has joined #bitcoin-core-dev
 897 2016-09-29T20:14:48  <wumpus> 288 is not just the default, it's the minimum
 898 2016-09-29T20:15:05  <wumpus> I'd be okay with changing the default not the minimum, but that'd keep some nodes completely useless
 899 2016-09-29T20:15:18  <wumpus> whereas by far most requests are in the last 288
 900 2016-09-29T20:15:25  <luke-jr> wumpus: useless for syncing*
 901 2016-09-29T20:15:57  <luke-jr> frankly, there are enough full-archive nodes out there that we really don't *need* to do anything right now, so meh :p
 902 2016-09-29T20:16:06  <sipa> wumpus: actually, not true.
 903 2016-09-29T20:16:13  <jtimon> well, the users know what to do to stop being useless...
 904 2016-09-29T20:16:16  <wumpus> which, as morcos remarked, preferentially downloading the last blocks from would take a lot of load of nodes that do keep more blocks
 905 2016-09-29T20:16:22  <sipa> there are more requests in 101-1000 deep then 2-100 deep
 906 2016-09-29T20:16:29  <wumpus> ok...
 907 2016-09-29T20:16:33  <sipa> *than
 908 2016-09-29T20:16:37  <wumpus> I misremembered apparently, never mind
 909 2016-09-29T20:16:39  *** laurentmt has joined #bitcoin-core-dev
 910 2016-09-29T20:16:45  <luke-jr> an unsyncable-from node is still more useful than a syncable node that isn't used for a wallet
 911 2016-09-29T20:16:46  *** laurentmt has quit IRC
 912 2016-09-29T20:17:08  <luke-jr> syncable-from*
 913 2016-09-29T20:17:26  <jtimon> maybe we can changethe prunning minimum if that simplifies things?
 914 2016-09-29T20:17:30  <sipa> wumpus: well, sample of 1 long-term-running node over the course of a few weeks of data
 915 2016-09-29T20:17:35  <sipa> wumpus: more samples welcome
 916 2016-09-29T20:17:57  <wumpus> sipa: do you have a special patch for statistics collection?
 917 2016-09-29T20:18:00  <gmaxwell> sipa: need to filter out bitnotes.
 918 2016-09-29T20:18:13  <sipa> gmaxwell: right; how do you suggest to do that?
 919 2016-09-29T20:18:26  <wumpus> sipa: or a script for parsing logs?
 920 2016-09-29T20:18:39  <sipa> wumpus: both; i'll publish them after a little cleanup
 921 2016-09-29T20:18:46  <wumpus> I could put it up on a few nodes, no problem
 922 2016-09-29T20:19:05  <sipa> it just logs an extra line with depth and block size for each requested block
 923 2016-09-29T20:19:13  <wumpus> nice
 924 2016-09-29T20:19:16  <jtimon> I guess it's not completely crazy, but nobody seem to specially like it
 925 2016-09-29T20:19:35  <sipa> en then
 926 2016-09-29T20:19:36  <sipa> S=0; fgrep DEEP ~/.bitcoin/debug.log | cut -d ' ' -f 4 | sort -g | uniq -c | tac | while read C D; do S=$(($S+$C)); echo "$D $C $S"; done | tac | less
 927 2016-09-29T20:19:39  <sipa> to inspect :)
 928 2016-09-29T20:20:14  *** laurentmt has joined #bitcoin-core-dev
 929 2016-09-29T20:20:14  <wumpus> jtimon: no, it's not completely crazy, using only one service bit is kind of elegant
 930 2016-09-29T20:21:11  <jtimon> :)
 931 2016-09-29T20:22:20  <wumpus> jtimon: if 1000 is really one-size-fits-all, and <1000-keeping nodes may as well be ignored. It's just hard to say without better statistics.
 932 2016-09-29T20:22:34  <wumpus> also statistics about what pruning sizes people prefer
 933 2016-09-29T20:23:00  <wumpus> I mean if everyone prefers the minimum and no one sets 1000 in practice
 934 2016-09-29T20:23:17  *** anchow101 has quit IRC
 935 2016-09-29T20:23:18  <gmaxwell> sipa: just do Satoshi:(recent) useragents.
 936 2016-09-29T20:23:35  <jtimon> well, independenlty of the statistics we will eventually need a more generic solution for flexible sharding, right?
 937 2016-09-29T20:23:45  <sipa> jtimon: maybe
 938 2016-09-29T20:24:05  <sipa> "need" is a big word imho
 939 2016-09-29T20:24:13  <sipa> but i agree it would be nice
 940 2016-09-29T20:24:14  <gmaxwell> jtimon: I think we do, would you like to finish the solution for that I started on?
 941 2016-09-29T20:24:16  <wumpus> jtimon: well there needs to be a different solution for historical block hosting IMO, but that's a different thing
 942 2016-09-29T20:25:26  *** aureianimus has quit IRC
 943 2016-09-29T20:25:26  <gmaxwell> sipa: I think excepting participants to keep around hundreds of gigs of blockchain is not conducive to the surival of the network, the alternative I see is a hardfork that drops off the history past some point. (e.g. just restarts the chain from a utxo commitment made a year before)
 944 2016-09-29T20:25:33  *** aureianimus has joined #bitcoin-core-dev
 945 2016-09-29T20:26:26  <sipa> gmaxwell: well, or just stop supporting historical block fetching more than 1 year or whatever number back on the p2p protocol, and use http
 946 2016-09-29T20:26:40  <wumpus> or bittorrent *ducks*
 947 2016-09-29T20:26:45  *** brainwave has joined #bitcoin-core-dev
 948 2016-09-29T20:26:50  <jtimon> wumpus: yeah, historical hosting is what I mean
 949 2016-09-29T20:27:15  <jtimon> gmaxwell: maybe, but it sounded deterministic like luke-jr proposed instead of flexible like wumpus wanted
 950 2016-09-29T20:27:22  <wumpus> it could be anything that supports downloading ranges of data...
 951 2016-09-29T20:27:46  <brainwave> Under overview, balances, on the right side of  available, pending, total, add ~ exchange rate for dollars, pounds, euro
 952 2016-09-29T20:28:00  <sipa> brainwave: bitcoin core does not and cannot know exchange rates
 953 2016-09-29T20:28:37  <sipa> (because it would require contacting a centralized service, which we don't do by design)
 954 2016-09-29T20:29:09  <wumpus> yes or someone would need to commit them to the chain, but that'd still be trusting a central issuer/signer of the information
 955 2016-09-29T20:29:12  <wumpus> it's just a no-go
 956 2016-09-29T20:30:27  *** laurentmt has quit IRC
 957 2016-09-29T20:32:12  *** brainwave has quit IRC
 958 2016-09-29T20:35:52  <gmaxwell> well if the users of bitcoin accepted that kind of security model change, what I would suggest is something like every 26280 blocks the block is required to have a commitment to the utxo set (could be a linear hash) as of 2016 blocks prior. and then six months of work after that, that commitment becomes usable for initial sync. and so then no one need process more than a year of blocks at sync.
 959 2016-09-29T20:35:58  <gmaxwell> .. though you would have to store three copies of the utxo set (though perhaps deduplicated)
 960 2016-09-29T20:36:46  <gmaxwell> jtimon: I don't know why anyone would find determinstic less desirable.
 961 2016-09-29T20:37:21  <sipa> gmaxwell: well i expect the controversy to not be about the change in security model, but about the perpetual requirement of having a utxo set
 962 2016-09-29T20:37:42  *** bad_duck has quit IRC
 963 2016-09-29T20:37:55  *** aureianimus has quit IRC
 964 2016-09-29T20:38:55  *** aureianimus has joined #bitcoin-core-dev
 965 2016-09-29T20:38:55  *** bad_duck has joined #bitcoin-core-dev
 966 2016-09-29T20:40:06  <wumpus> gmaxwell: I explained that: if you make it deterministic you have to be sure of the parameters in advance, there is no room for tweaking or optimizing later on
 967 2016-09-29T20:42:12  <gmaxwell> wumpus: well you simply extend the protocol to have a new signaling mechenism for the tweaked thing.
 968 2016-09-29T20:42:16  <wumpus> sipa: yes the bigger problem is the ever-growing UTXO set
 969 2016-09-29T20:42:24  <wumpus> gmaxwell: but then it loses backwards compatibility every time
 970 2016-09-29T20:42:29  *** jnewbery has quit IRC
 971 2016-09-29T20:43:03  <gmaxwell> something that just signals absolute heights has the problem that the communicated information will always be out of date. .. or if nodes don't change the ranges they host, we will end up with highly irregular distributions of information.
 972 2016-09-29T20:43:42  <sipa> the type of tweaking needed, and the potentially aging problem depend on the specific proposal
 973 2016-09-29T20:44:09  <sipa> i'm sure we can come up with something that seems reasonable to all
 974 2016-09-29T20:44:28  <wumpus> agree, there may be a compromise that is somewhat flexible and still deterministic
 975 2016-09-29T20:44:50  <gmaxwell> well what I suggested might not be viable after all too. I'm not sure, I wasn't successful in achieving all my goals at once.
 976 2016-09-29T20:45:26  <wumpus> I just don't think setting it all in stone in advance is a good idea, for the whole reason that it's so hard to achieve all your goals all at once
 977 2016-09-29T20:45:40  <wumpus> especially if you don't know some of those goals yet
 978 2016-09-29T20:45:47  <gmaxwell> I wanted a scheme that would result in a uniform distribution of blocks, that didn't depend on peers to look to see what other peers had (because that could be spoofed), required minimal communication (not a long list of blocks in an addr message).. and retainined uniformity as the chain grew, without causing peers to redownload blocks they already forgot.
 979 2016-09-29T20:47:13  <gmaxwell> So I had found two schemes, one where peers had a ID and the amount of blocks they would store, and from that they could determine which they would store, and as new blocks came in they might store then and drop some group of old ones.  The problem with it was that to figure out if a particular peer had block X you had to do computation linear in the number of blocks in the chain.
 980 2016-09-29T20:47:37  <wumpus> darn, also fingerprinting will be hard to avoid
 981 2016-09-29T20:47:45  <gmaxwell> Then I had another scheme that was sublinerar work, BUT a peer might drop a block but later have to go fetch it again.
 982 2016-09-29T20:47:54  <gmaxwell> wumpus: thats unavoidable with any split up scheme.
 983 2016-09-29T20:48:14  <wumpus> yes
 984 2016-09-29T20:48:15  <sipa> make the IP address part of the seed
 985 2016-09-29T20:48:25  <sipa> if your dhcp changes, you have to resync, sorry.
 986 2016-09-29T20:48:30  <wumpus> unless a substantial part of nodes are the same
 987 2016-09-29T20:48:31  <gmaxwell> sipa: then when you change IP, you have to go download a different set of blocks.. :P hah
 988 2016-09-29T20:48:47  <wumpus> e.g. there are only 8 IDs, pick one
 989 2016-09-29T20:48:54  <gmaxwell> wumpus: well I was thinking 32 bits, but perhaps a smaller collection would be fine.
 990 2016-09-29T20:49:18  <gmaxwell> but that gives you at best only 1/8th spitting storage. :( maybe fine now, but not in the long term.
 991 2016-09-29T20:49:47  <wumpus> maybe the number of groups can grow over time, a doubling every so many blocks :)
 992 2016-09-29T20:50:02  <sipa> hah: if you get a request through an IP that doesn't correspond to your local storage, just proxy all requests through to another node which does, and use that to gradually resync for the new seed.
 993 2016-09-29T20:50:05  <gmaxwell> Part of why I haven't given this that much more thought is because I think bitcoin will need to move to the commit state and forget history model; the ever growing sync time is too big a tide to stand against.
 994 2016-09-29T20:50:15  <gmaxwell> sipa: lol!
 995 2016-09-29T20:50:32  <gmaxwell> sipa: I think thats actually how the freenet location swapping works, funny enough.
 996 2016-09-29T20:50:43  <wumpus> hehe
 997 2016-09-29T20:50:46  <sipa> downside: if you want this to be fingerprint resistant, you have no way to determine how many proxies your blocks actually went through
 998 2016-09-29T20:50:49  <sipa> => instant mixnet
 999 2016-09-29T20:51:37  <gmaxwell> sipa: freenet nodes change position over time, and they do it by swapping their location with a direct neighbor, when that location swap makes them both closer to where they want to be, ... when requests come in for the new location, they don't have the data, but it's only one hop away..
1000 2016-09-29T20:52:55  <wumpus> gmaxwell: I've always thought that, it's hard to imagine this continuing for 10's of years, but where to put the anchor...
1001 2016-09-29T20:52:59  <gmaxwell> in any case. if there were only 8 flavors of nodes, then it all becomes simple, block_height//1000 % 8 = flavor.
1002 2016-09-29T20:53:32  * gmaxwell lunch
1003 2016-09-29T20:54:18  <wumpus> that seems kind of elegant and straightforward, there must be a catch
1004 2016-09-29T20:58:57  *** aureianimus has quit IRC
1005 2016-09-29T20:59:06  *** aureianimus has joined #bitcoin-core-dev
1006 2016-09-29T20:59:47  <jtimon> gmaxwell: sorry, well the deterministic seems to come at the cost of less flexibility
1007 2016-09-29T21:01:37  <sipa> wumpus: i'm trying to think about why 8 isn't enough
1008 2016-09-29T21:01:37  <wumpus> if you want to automatically scale the number of flavors up with height you could divide height 0..N into X flavors, the N..3N into 2*N flavors, and so on, where each flavor gets flavor (x<<1)+randbit()
1009 2016-09-29T21:01:37  <jtimon> only 8 flavors requires you to store 1/8 of the blockchain
1010 2016-09-29T21:01:37  <sipa> and we could have names for the first 8 top-level flavours or so... so your wallet could report "Looking for a bittersweet node..."
1011 2016-09-29T21:01:40  <wumpus> (well those numbers are arbitrary but the idea is that if a doubling of the # is needed, the new flavor, a member of a twice as big set, would contain the previous one)
1012 2016-09-29T21:02:21  <wumpus> hehe, yes assigning names would be nice
1013 2016-09-29T21:13:53  *** jasonv76 has quit IRC
1014 2016-09-29T21:14:10  *** jasonv76 has joined #bitcoin-core-dev
1015 2016-09-29T21:18:39  *** jnewbery has joined #bitcoin-core-dev
1016 2016-09-29T21:18:42  *** cryptapus has quit IRC
1017 2016-09-29T21:19:47  *** achow101_ has quit IRC
1018 2016-09-29T21:19:56  *** aureianimus has quit IRC
1019 2016-09-29T21:20:16  *** aureianimus has joined #bitcoin-core-dev
1020 2016-09-29T21:21:59  *** achow101_ has joined #bitcoin-core-dev
1021 2016-09-29T21:22:12  *** achow101_ has quit IRC
1022 2016-09-29T21:31:14  *** cryptapus has joined #bitcoin-core-dev
1023 2016-09-29T21:40:06  *** sipa has quit IRC
1024 2016-09-29T21:40:06  *** sipa has joined #bitcoin-core-dev
1025 2016-09-29T21:40:51  *** aureianimus has quit IRC
1026 2016-09-29T21:40:59  *** aureianimus has joined #bitcoin-core-dev
1027 2016-09-29T21:41:13  *** MarcoFalke has left #bitcoin-core-dev
1028 2016-09-29T21:46:23  *** cdecker has quit IRC
1029 2016-09-29T21:46:24  <GitHub131> [bitcoin] jnewbery opened pull request #8844: change sigops cost to sigops weight (master...sigops_weight) https://github.com/bitcoin/bitcoin/pull/8844
1030 2016-09-29T21:48:11  *** gabridome has quit IRC
1031 2016-09-29T21:49:09  <GitHub72> [bitcoin] jnewbery opened pull request #8845: Don't return the address of a P2SH of a P2SH (master...trivial-P2SH-P2SH) https://github.com/bitcoin/bitcoin/pull/8845
1032 2016-09-29T21:51:37  *** luke-jr has quit IRC
1033 2016-09-29T21:51:55  <gmaxwell> jtimon: "less flexible" -- everything is less flexible short of sending someone arbritary x86 bytecode that they run.
1034 2016-09-29T21:52:02  *** aureianimus_ has joined #bitcoin-core-dev
1035 2016-09-29T21:52:02  *** aureianimus has quit IRC
1036 2016-09-29T21:54:00  <jtimon> less flexible in the amount of data you store, but maybe 8 flavors can be subidivided in 16 flavors half the size as wumpus was suggeting, then 16 to 32, etc. That may be flexible enough
1037 2016-09-29T21:54:30  <gmaxwell> jtimon: I was recommending 2^32 'flavors' but wumpus was concerned about reducing fingerprinting.
1038 2016-09-29T21:55:10  <gmaxwell> the whole reason to reduce the amount was to make it more difficult to follow a node around as it changes network identity.
1039 2016-09-29T21:55:30  <gmaxwell> sipa: 8 isn't enough if the chain is perpetually growing.
1040 2016-09-29T21:56:05  <jtimon> I see
1041 2016-09-29T21:56:15  <sipa> yeah, increasing number, the further back you go, may make sense
1042 2016-09-29T21:56:34  <gmaxwell> a year from now the chain will be 200 gb, a year after 300 gb-- at that size it is larger than the most common ssd size currently. a year after that 400gb.... and at that point an 8 way split is again running common hosts out of disk even if the common ssd size has moved up to 500gb by then.
1043 2016-09-29T21:56:59  <jtimon> well, maybe archive nodes that don't want to store everything have to get a privacy hit
1044 2016-09-29T21:57:32  <gmaxwell> who will bother running one if it takes speical effort above and beyond running a node, and draws more resources?
1045 2016-09-29T21:57:39  <sipa> well if only we'd have a separate network for archivsl
1046 2016-09-29T21:57:53  <sipa> there are no privacy issues at all then
1047 2016-09-29T22:01:17  *** jnewbery has quit IRC
1048 2016-09-29T22:06:25  <gmaxwell> and no one run them.
1049 2016-09-29T22:06:57  <gmaxwell> s/run/running/
1050 2016-09-29T22:07:27  <sipa> i was about to say that separate network doesn't need to imply separate nodes
1051 2016-09-29T22:07:42  <sipa> but of course, that doesn't work because you'd get a privacy leak from correlating
1052 2016-09-29T22:08:46  <sipa> however, you can reconcile those by only having nodes with a long-term IP provide archival further back than some threshold
1053 2016-09-29T22:09:37  *** jannes has quit IRC
1054 2016-09-29T22:10:02  <gmaxwell> sipa: not just that, but if it's a special very resource intensive mode.. few will do it, pliling more resources onto it... causing fewer to do it...
1055 2016-09-29T22:10:37  <sipa> it's true that it's resource intensive, but it's a different kind of resources than most of the rest of running a node
1056 2016-09-29T22:10:43  <sipa> it needs disk space and bandwidth
1057 2016-09-29T22:10:45  <gmaxwell> I might think it's not over the threshold of that, except already people don't run regular nodes due to costs.
1058 2016-09-29T22:11:12  <sipa> rather than memory and cpu
1059 2016-09-29T22:11:13  <gmaxwell> which are what people usually complain about.
1060 2016-09-29T22:11:31  <sipa> then why aren't we seeing more pruned nodes?
1061 2016-09-29T22:11:57  <sipa> one reason may be that pruned nodes don't advertize, so we just don't know about them
1062 2016-09-29T22:12:33  <gmaxwell> because you have to edit a config file or change an obscure setting, we don't advertise it, and it breaks rescan and reindex. (which is part of why we don't really advertise it)
1063 2016-09-29T22:12:46  *** aureianimus_ has quit IRC
1064 2016-09-29T22:12:55  *** aureianimus has joined #bitcoin-core-dev
1065 2016-09-29T22:14:43  <sipa> well people mostly complain about the sync time for a node
1066 2016-09-29T22:15:14  <gmaxwell> yes, though I think thats most because so many stop there and give up before they get a chance to complain about the rest.
1067 2016-09-29T22:15:20  <sipa> perhaps
1068 2016-09-29T22:18:16  <TD-Linux> a first-run dialog box with a slider for disk usage and an estimated sync time would be very nice
1069 2016-09-29T22:18:49  <sipa> except the sync time does not depend on the value of the slider
1070 2016-09-29T22:19:57  <TD-Linux> yes, I meant it there so it'd appear at start. I guess having it in the status bar is sufficient
1071 2016-09-29T22:20:36  <sipa> ah
1072 2016-09-29T22:20:48  <sipa> well there will be an overlay with sync time indication in 0.14
1073 2016-09-29T22:21:16  <gmaxwell> doesn't it still incorrectly say you can't transact while syncing?
1074 2016-09-29T22:21:30  <sipa> we still have a lot time until 0.14
1075 2016-09-29T22:23:08  <gmaxwell> :)
1076 2016-09-29T22:23:14  <wumpus> well you can, but most people probably shouldn't do so
1077 2016-09-29T22:23:34  <gmaxwell> yes they should
1078 2016-09-29T22:23:52  <wumpus> during the initial sync they won't have any coins to send anyway, and receiving them is a bad idea as they'll only see them when the entire thing is done
1079 2016-09-29T22:23:55  <wumpus> oh?
1080 2016-09-29T22:24:02  <wumpus> why?
1081 2016-09-29T22:24:18  <gmaxwell> initial sync isn't my concern there:
1082 2016-09-29T22:24:18  <gmaxwell> probably one of the most common usage patterns for a wallet user is that you start your wallet up in order to pay someone, and it's three weeks behind. You can go ahead and pay, no problems.. why wouldn't you?
1083 2016-09-29T22:24:36  <gmaxwell> during initial sync you just won't have any coins, indeed. :)
1084 2016-09-29T22:24:48  <wumpus> the biggest problem is people giving out addresses during initial sync
1085 2016-09-29T22:24:52  <wumpus> then realizing how long it takes
1086 2016-09-29T22:25:10  <wumpus> this is what the overlay is designed to prevent
1087 2016-09-29T22:25:26  <wumpus> sure, you can send coins if you're three weeks behind, no problem, although fee computation could be off
1088 2016-09-29T22:25:39  <gmaxwell> yes, that is a large source of complaints, but we shouldn't tell people that they cant send funds already in their wallet when they start up and they're a bit behind, it's already a common mistaken belief that they cant (and then they complain about how long it takes to catch up a month of blcks)
1089 2016-09-29T22:25:40  <TD-Linux> the warning could be conditional on having zero funds
1090 2016-09-29T22:26:13  <gmaxwell> TD-Linux: the earlier warning text was fine-- saying that you won't see payments to you yet, but for some reason it was changed to say that you cannot send funds.
1091 2016-09-29T22:26:17  <wumpus> yeah fix one thing and they'll start complaining about another, it's a never ending source of fun...
1092 2016-09-29T22:26:33  <sipa> i don't think anyone will read the text anyway
1093 2016-09-29T22:26:47  <gmaxwell> I also complained that the text is now too long and won't get read.
1094 2016-09-29T22:26:50  <wumpus> of course people will read it
1095 2016-09-29T22:26:55  <sipa> the important thing is that it's in the way, and gives accurate (by then, hopefully) information
1096 2016-09-29T22:26:57  <wumpus> heck, users aren't stupid
1097 2016-09-29T22:26:59  <gmaxwell> The first text was better.
1098 2016-09-29T22:27:10  <sipa> gmaxwell: PR welcome
1099 2016-09-29T22:27:14  <wumpus> maybe some are, but not all of them, some will actually read and understand
1100 2016-09-29T22:27:18  <gmaxwell> Well I'm stupid, and looked at the notice in its updated state and didn't read the list line.
1101 2016-09-29T22:27:32  <gmaxwell> first*
1102 2016-09-29T22:27:48  <gmaxwell> because when there is too much text many people go a bit banner blind and skim past headings and such.
1103 2016-09-29T22:27:51  <wumpus> if we don't believe peopel actually pay attention then why do anything at all
1104 2016-09-29T22:28:14  <gmaxwell> saying that a wall of text is too much is not saying that people don't pay attention.
1105 2016-09-29T22:28:20  <wumpus> I think it's an improvement to what was there, indeed, if you want to imrpvoe further then pulls are welcome
1106 2016-09-29T22:28:49  <sipa> right, that's what i'm saying - having there being an overlay at all is more important than what the text says
1107 2016-09-29T22:29:00  <gmaxwell> and re: being able to send, people already complain that they have to wait a long time after starting to send because they already frequently mistakingly believe they can't.
1108 2016-09-29T22:29:00  <sipa> and we have time to improve the latter
1109 2016-09-29T22:29:05  <wumpus> but I'm a bit tired of people always saying "users won't read anyway" to everything that adds documentation , help or warnings
1110 2016-09-29T22:29:29  <wumpus> a lot of users are definitely looking for more help and guidance when they first open the program, and a bit of text helps there
1111 2016-09-29T22:29:40  <gmaxwell> wumpus: why should I waste my time when I point out that THE TEXT IS OUTRIGHT UNTRUE and your response is to accuse me of thinking users are stupid?  my comment was that the earlier version of the text which was simple and NOT UNTRUE was better.
1112 2016-09-29T22:29:54  <sipa> please guys
1113 2016-09-29T22:30:00  <sipa> gmaxwell: go propose something
1114 2016-09-29T22:30:05  <gmaxwell> I did!
1115 2016-09-29T22:30:15  <wumpus> gmaxwell: well if the text is wrong then it should be fixed obviously, change it to a better text
1116 2016-09-29T22:30:49  <wumpus> I don't know what the previous version of the text was
1117 2016-09-29T22:31:56  <sipa> it's been changed a dozen times in the lifetime of the pull
1118 2016-09-29T22:33:03  <sipa> also, it says "Spending bitcoins may not be possible until synchronization has finished."
1119 2016-09-29T22:33:08  <sipa> which is not untrue.
1120 2016-09-29T22:34:44  <gmaxwell> okay, it was changed after I last saw it.
1121 2016-09-29T22:35:13  <wumpus> ok that was useless :)
1122 2016-09-29T22:35:18  <gmaxwell> by saying 'may' which is still misleading, but worse, that text is the bold.
1123 2016-09-29T22:35:24  <gmaxwell> er is the only bold part.
1124 2016-09-29T22:35:42  <sipa> well, improvements welcome
1125 2016-09-29T22:35:54  <gmaxwell> So now it says "mumble mumble mumble  Spending bitcoins may not be possible during that phase!" :-/
1126 2016-09-29T22:36:14  <gmaxwell> it's a waste of my time, I already raised these issues and it was then merged.
1127 2016-09-29T22:36:45  <wumpus> it had to be merged at some point, with the idea it could be improved later
1128 2016-09-29T22:36:58  <gmaxwell> well to be fair the last change did improve it, its true.
1129 2016-09-29T22:37:38  <gmaxwell> but created the problem that if you skim it is that all you extract is that you can't spend, .. which misses the really critical thing: which is that you wallet may look empty when it isn't.
1130 2016-09-29T22:37:57  *** aureianimus has quit IRC
1131 2016-09-29T22:37:58  <wumpus> that doesn't mean it's final, most will only see the message when it is merged, and can improve it then, there are already some pulls open to improve that overlay
1132 2016-09-29T22:38:00  <gmaxwell> but okay, I can open a PR.
1133 2016-09-29T22:38:17  <wumpus> (but I don't think they change that message)
1134 2016-09-29T22:38:20  *** aureianimus has joined #bitcoin-core-dev
1135 2016-09-29T22:38:53  *** Guyver2 has quit IRC
1136 2016-09-29T22:39:08  <sipa> gmaxwell: i think people didn't really understand the point of your concern (i didn't): if you're looking at it from a point of view that this would be be mostly seen (and intended to convey information) during IBD, it's perfectly reasonable to warn users they won't be able to spend the money they're still to receive... and a simplification to reduce the length of the text may be warranted
1137 2016-09-29T22:39:31  <sipa> it's a good point that it's also seen during non-IBD
1138 2016-09-29T22:39:34  <gmaxwell> It will mostly not be seen during IBD.
1139 2016-09-29T22:39:54  <gmaxwell> during IBD sure someone will see it then, say of course I knew that (even if they didn't) minimize and go on with life. :P
1140 2016-09-29T22:40:13  <gmaxwell> but then users will see it every single time they start.
1141 2016-09-29T22:40:22  <sipa> i'm aware, you don't need to argue about this
1142 2016-09-29T22:40:30  <sipa> i'm just explaining why maybe you felt misunderstood
1143 2016-09-29T22:40:30  <gmaxwell> sorry, not arguing-- clarifying.
1144 2016-09-29T22:40:43  <gmaxwell> Yes, I see that and I didn't before.
1145 2016-09-29T22:41:19  <gmaxwell> when I first saw this PR I even took the time to go through the code carefully to check to see if there was anything that made it IBD only.
1146 2016-09-29T22:42:02  <gmaxwell> because I couldn't understand why people wanted the text that it had.
1147 2016-09-29T22:42:18  <gmaxwell> it did not occure to me that other people might be only thinking about IBD.
1148 2016-09-29T22:42:26  <gmaxwell> sorry for being thoughtless there.
1149 2016-09-29T22:42:46  <wumpus> #8805 fixed a few minor grammar nits, #8821 fixes a blocking problem with the overlay, there are no pulls yet that improve the message
1150 2016-09-29T22:43:17  <sipa> sorry, we (including me) aren't being careful with terminology here... IBD is also used for syncup when you were previously synced to a month ago
1151 2016-09-29T22:43:22  *** To7 has quit IRC
1152 2016-09-29T22:43:47  <wumpus> it's very easy to forget about catching up nodes
1153 2016-09-29T22:44:18  <wumpus> but yes we shouldn't
1154 2016-09-29T22:44:21  <sipa> well it's mostly designed to help with that first sync
1155 2016-09-29T22:45:07  <gmaxwell> more obvious to me just by chance of hering more people complain about it, also I've stopped running a node 24/7 on my laptop because I've been watching the battlestar galactica series in evenings and bitcoin interupts video playback. :)
1156 2016-09-29T22:45:15  <wumpus> during catch-up it's reasonably useful too, people may not know they won't see transactions newer than their sync point and worry, but yes it's mostly important for the initial IBD (lol)
1157 2016-09-29T22:45:21  <gmaxwell> so every time I go to use bitcoin I'm stuck waiting for it to catch up.
1158 2016-09-29T22:45:42  <gmaxwell> yes, we should have this message during catch up. But it's important to not make people think they can't spend funds that they can see.
1159 2016-09-29T22:46:03  <gmaxwell> The important message is that you may not see all payments to you yet (and you can't spend what you can't see).
1160 2016-09-29T22:46:31  <wumpus> bitcoin interrupts video playback? even in steady state mode?
1161 2016-09-29T22:46:34  <TD-Linux> one option would be to put it on the payment request generation page instead. but even in its current state it's far better than what was there before (nothing)
1162 2016-09-29T22:47:29  <gmaxwell> wumpus: yes. on my laptop... playback from a local file. The issue is IO or cpu related, probably the former but I haven't tested extensively to know for sure.
1163 2016-09-29T22:48:03  <wumpus> that's very strange. I'd expect that during intial sync when it maxes out CPU and I/O usage, but not when it's up to date
1164 2016-09-29T22:48:13  <gmaxwell> TD-Linux: the big problem we should be solving here is that people see a balance of zero then delete the wallet. I think thats the priority because any other issue doesn't cause irrecoverable loss.
1165 2016-09-29T22:48:24  *** cryptapus is now known as cryptapus_afk
1166 2016-09-29T22:48:43  <gmaxwell> wumpus: I notice it during ordinary computer use.. causes IO hangs, but its not irritating except when watching video.
1167 2016-09-29T22:49:28  <wumpus> do you have a lot of mlocked memory? is it swapping?
1168 2016-09-29T22:50:01  <gmaxwell> no, not swapping 8gb ram. I think that when a bunch of random writes happen it causes long delays for garbage collection in the SSD.
1169 2016-09-29T22:50:02  <wumpus> swapping seems to be the foremost cause of I/O related hangs here, as essentially the memory subsystem has to wait for I/O to complete
1170 2016-09-29T22:50:40  <wumpus> heh as if 8gb ram means no swapping these days :)
1171 2016-09-29T22:50:59  <gmaxwell> well on my laptop its enough most of the time.
1172 2016-09-29T22:51:29  <gmaxwell> The stalls seemed to get better for a while after I freed up a bunch of space and trimmed the drive, but got worse after which is why I think SSD GC plays a roll.
1173 2016-09-29T22:51:50  <gmaxwell> but in any case, while watching the show every block arrival causes a second-long pause in playback.
1174 2016-09-29T22:51:51  <wumpus> maybe someone is requesting a lot of blocks from you with a bloom filter? :-) it would be interesting to find out what your node is actually doing at those times
1175 2016-09-29T22:52:02  <gmaxwell> nah, outbound only.
1176 2016-09-29T22:52:09  <gmaxwell> I know its at the same time as blocks showing up.
1177 2016-09-29T22:52:19  <TD-Linux> gmaxwell, easy way to verify that would be to increase your video player's lookahead cache
1178 2016-09-29T22:52:21  <wumpus> ok so it's block verification, leveldb seeks
1179 2016-09-29T22:52:57  <gmaxwell> TD-Linux: think mpv uses non-blocking reads of the disk?
1180 2016-09-29T22:53:21  <TD-Linux> gmaxwell, yup it does. I've increase the setting to 10s when using sshfs and it works fine
1181 2016-09-29T22:54:04  <gmaxwell> in any case, performance distraction aside, when this happens I shut down bitcoind then it may stay off for a week before I need to do something with it, then waiting for it to catch up is irritating.
1182 2016-09-29T22:55:23  <gmaxwell> (and of course my systems performance is seriously impacted while it catches up)
1183 2016-09-29T22:55:27  <wumpus> yes, nothing to do about that, I guess if hybrid SPV mode is implemented it could also work during catch-up
1184 2016-09-29T22:55:55  <wumpus> indeed, it's either slow down the catch up or tolerate it hogging the whole system
1185 2016-09-29T22:58:09  <gmaxwell> I have wondered if it might be useful to split the chainstate into two parts, one with txouts created in the most recent N blocks, and one with the rest. Then on start we could just load the whole first one into the cache.
1186 2016-09-29T22:58:10  <wumpus> the default setting of hogging all cores during IBD/catch-up is a bit rude, certainly if it is a background process
1187 2016-09-29T22:58:59  *** aureianimus has quit IRC
1188 2016-09-29T22:59:02  *** aureianimus_ has joined #bitcoin-core-dev
1189 2016-09-29T22:59:04  <gmaxwell> if we did that much of the cost would then be signature valdation instead of random IO, and signature validation could run in the background, following behind the blocks.. and at lower priority.
1190 2016-09-29T22:59:51  <wumpus> so that would be like a 'prefer keeping recent UTXOs' cache policy?
1191 2016-09-29T23:00:14  <gmaxwell> I guess thats a stat that I still haven't collected.  "what is the average age-in-blocks of inputs that are consumed" (/what is the distribution of that age)
1192 2016-09-29T23:00:28  <gmaxwell> we know recent ones are spent more often but I don't have good numbers on it.
1193 2016-09-29T23:00:31  <gmaxwell> wumpus: yes.
1194 2016-09-29T23:02:56  <gmaxwell> probably not the highest priority improvement in any case.
1195 2016-09-29T23:02:58  <wumpus> well, currently the whole cache is emptied at a write, I think there are many eviction policies that would do better
1196 2016-09-29T23:03:32  <gmaxwell> right, also the in memory representation of the cache entries is quite inefficient.
1197 2016-09-29T23:04:03  <gmaxwell> so its effective size could potentially be doubled if its entries were flat allocated.
1198 2016-09-29T23:04:08  <wumpus> though that helps it actually being an efficient cache, a more efficient representation shouldn't come at a higher access cost
1199 2016-09-29T23:06:48  <wumpus> I did an experiment once with storing the UTXOs in serialized form in memory: https://github.com/laanwj/bitcoin/tree/2016_04_dummy_db
1200 2016-09-29T23:09:31  <gmaxwell> interesting!
1201 2016-09-29T23:09:33  <wumpus> although that's for the entire UTXO set, not just a limited cache
1202 2016-09-29T23:10:52  <gmaxwell> yea, the on disk seralization is most efficient, but not fast.
1203 2016-09-29T23:11:11  *** aureianimus_ has quit IRC
1204 2016-09-29T23:11:19  *** aureianimus has joined #bitcoin-core-dev
1205 2016-09-29T23:11:25  <gmaxwell> I wouldn't be surprised though for the current in memory representation if there wasn't more bytes spend in malloc/container overhead than actual transaction data though.
1206 2016-09-29T23:11:29  <wumpus> but given that most time is wasted on disk seeks anyway, it may not make too much difference in practice
1207 2016-09-29T23:11:43  <wumpus> depends on the system...
1208 2016-09-29T23:12:00  <wumpus> yes, the malloc overhead is somewhat bad
1209 2016-09-29T23:12:13  *** aureianimus has quit IRC
1210 2016-09-29T23:12:25  *** aureianimus has joined #bitcoin-core-dev
1211 2016-09-29T23:14:08  <wumpus> but not more than the actual data size, from what I remember
1212 2016-09-29T23:14:54  *** aalex has quit IRC
1213 2016-09-29T23:15:16  <wumpus> in any case improvements are certainly possible there, without any rocket science, it's just that it's such risky code to change
1214 2016-09-29T23:15:53  <wumpus> if it was any other project people would have optimized the shit out of it by now
1215 2016-09-29T23:18:52  *** aalex has joined #bitcoin-core-dev
1216 2016-09-29T23:21:06  <wumpus> unfortunately the damages of a bug there are unfathomable, not just skipping a few video frames
1217 2016-09-29T23:22:24  <gmaxwell> well the hope I had there was that with making the cache more efficient, it could be increased in size and avoid more disk IO. :)
1218 2016-09-29T23:24:30  <gmaxwell> on the earlier subject of 'alt' implementations doing inadvisable things, some of them have a genius performance optimization which they're crowing about, -- where they only validate transactions that weren't already in the mempool; something we explicity decided not to do because of the long history of subtle mempool corruption issues.
1219 2016-09-29T23:25:36  <wumpus> exactly, there are tons of ways to optimize and get things just slightly wrong
1220 2016-09-29T23:25:45  <gmaxwell> I worry that there will be a race to the bottom, where by making risky / security reducing optimizations implementations will gain significant performance advantages, and suffer no cost until the inevitable spectacular failure that results.
1221 2016-09-29T23:26:06  <wumpus> which doesn't matter if no one runs your code anyway, but we have to be really careful
1222 2016-09-29T23:26:33  <gmaxwell> and being safe doesn't matter if peopel don't run it in favor of things that are faster.
1223 2016-09-29T23:26:43  <gmaxwell> people*
1224 2016-09-29T23:26:52  <wumpus> we also shouldn't overestimate how important the performance is to most users, many just run it on a server or otherwise unused computer
1225 2016-09-29T23:27:31  *** aureianimus_ has joined #bitcoin-core-dev
1226 2016-09-29T23:27:41  *** aureianimus has quit IRC
1227 2016-09-29T23:27:51  <wumpus> well you'd say safeness is really important, if the inevitable spectacular failure happens you don't want to be at the center of it
1228 2016-09-29T23:28:32  <gmaxwell> well sure. indeed.
1229 2016-09-29T23:28:39  <wumpus> better slow than dead :)
1230 2016-09-29T23:29:44  <gmaxwell> but for things like the security model change to use a 6 month old utxo commitment instead of syncing the history... the potential for a spectacular failure there which a more conservative approach could have stopped is negligible.
1231 2016-09-29T23:29:52  <TD-Linux> gmaxwell, still think you should verify that it's actually IO that's the problem before going too deep :)
1232 2016-09-29T23:29:56  <gmaxwell> And if we don't investgate things like that, someone will do something dumber.
1233 2016-09-29T23:30:18  <wumpus> TD-Linux: yes, measuring is better than assumptions :)
1234 2016-09-29T23:30:28  <gmaxwell> TD-Linux: Oh IO is an issue for sure regardless of whats causing my mpv stalls. I'll find out tonight (I'm not going to sit here and watch video for an hour just to check)
1235 2016-09-29T23:31:54  <gmaxwell> TD-Linux: SSD vs a fast spinning disk with small dbcache here is a <4 hour sync (from a local peer) vs a >9hour sync.
1236 2016-09-29T23:32:26  <wumpus> and sure, it's good to investigate things like that
1237 2016-09-29T23:33:00  <wumpus> I/O is absolutely a problem sometimes, leveldb generates *tons* of seeks and small reads, better caching would help avoid some of that
1238 2016-09-29T23:34:27  <wumpus> I had no luck with other databases, I remember trying with lmdb at some point, which was faster in reading, but instead... does tons of seeks and small writes at write, so it just moves the problem
1239 2016-09-29T23:35:21  <TD-Linux> yeah, on a SSD at least, the former is much less detrimental to system latency
1240 2016-09-29T23:35:27  <wumpus> indeed!
1241 2016-09-29T23:35:52  <gmaxwell> for operation at the tip, using the mempool instead of validating would be a big aid... but the safty of that remains dubious. :)
1242 2016-09-29T23:36:07  <sipa> gmaxwell: i believe for the mempool it's approximately a factor 2 overhead
1243 2016-09-29T23:36:24  <sipa> gmaxwell: that that is also indexes, orderings, accounting, ...
1244 2016-09-29T23:38:07  <wumpus> yes that remains dubious, especially with regard to isFinal and such
1245 2016-09-29T23:38:27  <wumpus> I mean there is a part of transaction validation that can be obviously cached, and a part that may change in time
1246 2016-09-29T23:39:03  <gmaxwell> fortunately the MTP change made that much safer.
1247 2016-09-29T23:41:48  <gmaxwell> e.g. before a block could come in with a time before your local time, and contain txn which are isfinal invalid according to the block but okay with respect to the local time, and you'd accept it. I wonder if the alt implementations had that bug.
1248 2016-09-29T23:48:12  <wumpus> I remember that one, tricky... and there may be more problems of that kind not yet found
1249 2016-09-29T23:49:43  <gmaxwell> looks like their change dodged it, because the finality test is in the ContextualCheckBlock, and the bypass patch only bypasses checkinputs...
1250 2016-09-29T23:49:55  <gmaxwell> (which also means that it doesn't manage to avoid accessing the utxo cache entries)
1251 2016-09-29T23:51:16  <wumpus> I just realized, if the problem is that the block validation hiccups other things happening on your PC, the solution may be actually to slow it down :)
1252 2016-09-29T23:52:00  <wumpus> put a small sleep between each UTXO lookup, limit the validation to one thread
1253 2016-09-29T23:52:38  <wumpus> not something you'd want to do during initial sync if you're waiting for it, but if you don't care and it runs in the background...
1254 2016-09-29T23:54:14  <wumpus> after all you run it to keep up, you don't need to outrace it
1255 2016-09-29T23:56:05  <gmaxwell> I've thought before that if we have bandwidth limiting enabled we should delay announcement of new blocks to reduce the number of peers that request them from us... but slowing down the validation would work as well.
1256 2016-09-29T23:56:18  <gmaxwell> small sleeps perhaps aren't so good because it may busy spin. :P
1257 2016-09-29T23:57:44  <wumpus> heh, not that small
1258 2016-09-29T23:58:57  <wumpus> or use some OS-dependent way to reduce the I/O priority
1259 2016-09-29T23:59:35  <wumpus> as long as it's done by the time the next block comes in, so taking 10 minutes would take it too far :)