1 2015-11-16T00:19:28  *** d4de has joined #bitcoin-core-dev
  2 2015-11-16T00:27:00  <dcousens> the *.dat files, they are assumed to be in order yeah?  I know the blocks in them may be out of order, but the files themselves are in order yeah?
  3 2015-11-16T00:28:51  <sipa> what does order for files mean, if the blocks are not in order?
  4 2015-11-16T00:28:59  <dcousens> sequential*
  5 2015-11-16T00:29:25  <dcousens> in that, data from blk001.dat "flows" into blk002.dat
  6 2015-11-16T00:29:31  <sipa> yes
  7 2015-11-16T00:29:42  <sipa> :)
  8 2015-11-16T00:29:53  <sipa> though that has been broken in the past
  9 2015-11-16T00:30:01  <dcousens> sipa: oh?
 10 2015-11-16T00:30:26  <dcousens> (having troubles with a parser, it makes it 370k blocks in before blowing up with the magic header being wrong)
 11 2015-11-16T00:30:28  <sipa> it would at startup rescan through all block files the first time a block was written, and if there was space in one, it would go there
 12 2015-11-16T00:30:31  <dcousens> (running a re-index now)
 13 2015-11-16T00:33:11  <dcousens> sipa: is there a separate index for the blocks in the .dat files?
 14 2015-11-16T00:33:31  <sipa> that's called the block index :p
 15 2015-11-16T00:33:38  <sipa> $DATADIR/blocks/indx
 16 2015-11-16T00:33:39  <sipa> $DATADIR/blocks/index
 17 2015-11-16T00:34:05  <dcousens> haha
 18 2015-11-16T00:34:24  <dcousens> Sure, is that referencing file pointers?
 19 2015-11-16T00:34:34  <dcousens> like, file+offset pointers
 20 2015-11-16T00:34:58  <sipa> indeed
 21 2015-11-16T00:35:04  <sipa> file number, byte offset
 22 2015-11-16T00:35:23  <dcousens> and even then, the blocks would still be back to back right, there shouldn't be empty space out of order data etc?
 23 2015-11-16T00:35:43  <dcousens> like,  I wouldn't expect a block to flow from 001.dat to 003.dat,
 24 2015-11-16T00:35:49  <dcousens> (that is, skipping 002.dat)
 25 2015-11-16T00:35:52  <sipa> that's certainly possible
 26 2015-11-16T00:35:56  <sipa> there are really no guarantees
 27 2015-11-16T00:36:24  <sipa> we try to put blocks in order
 28 2015-11-16T00:36:33  <sipa> to the extent possible
 29 2015-11-16T00:36:45  <sipa> but only by limiting how much out-of-order blocks we fetch
 30 2015-11-16T00:36:45  <dcousens> sipa: wouldn't that mean the block index would have to maintain multiple file number/offsets?
 31 2015-11-16T00:36:50  <sipa> how so?
 32 2015-11-16T00:36:52  <dcousens> per block
 33 2015-11-16T00:37:03  <sipa> oh! a block is always in only exactly one file
 34 2015-11-16T00:37:05  <sipa> continuously
 35 2015-11-16T00:37:20  <dcousens> with a possibility to flow into the next file, right?
 36 2015-11-16T00:37:30  <sipa> how do you mean 'flow into' ?
 37 2015-11-16T00:37:38  <dcousens> half in blk001.dat, half in blk002.dat
 38 2015-11-16T00:37:40  <sipa> no
 39 2015-11-16T00:37:49  <sipa> always continuously in one file
 40 2015-11-16T00:37:58  <sipa> if file N can't fit it, it goes into file N+!
 41 2015-11-16T00:38:09  <sipa> which is why the block files aren't exactly the same size :)
 42 2015-11-16T00:40:47  <dcousens> ok! easy
 43 2015-11-16T00:41:06  <dcousens> but the block files themselves could contain complete garbage data mixed with actual data
 44 2015-11-16T00:41:09  <dcousens> conceptually
 45 2015-11-16T00:41:30  <sipa> yes
 46 2015-11-16T00:41:48  <sipa> though that too is something we try to prevent :)
 47 2015-11-16T00:42:41  <dcousens> well, crap. Haha, I'll let you know how I go, but, considered I got 360k blocks in... might be easier to just trash the ?garbage? blocks and see how it goes after
 48 2015-11-16T00:43:02  <sipa> what is the problem?
 49 2015-11-16T00:43:24  <dcousens> "(having troubles with a parser, it makes it ~365k blocks in before blowing up with the magic header being wrong)"
 50 2015-11-16T00:44:12  <sipa> well scan for the first valid header :)
 51 2015-11-16T00:44:13  <dcousens> but the parser makes the assumption the data is contigous with no gaps (aka, no garbage)
 52 2015-11-16T00:44:40  <dcousens> sipa: might do, just wasn't aware that was necessary
 53 2015-11-16T00:45:02  <sipa> i think it's very unlikely to happen in 0.11 or later
 54 2015-11-16T00:45:09  <sipa> would require block index corruption
 55 2015-11-16T00:45:17  <dcousens> is a fresh build and IBD from master
 56 2015-11-16T00:45:25  <sipa> weird
 57 2015-11-16T00:45:30  <sipa> maybe i'm wrong
 58 2015-11-16T00:45:41  <sipa> maybe your block.data is just actually corrupted? :)
 59 2015-11-16T00:45:45  <dcousens> maybe I am, like I said, trying a re-index now
 60 2015-11-16T00:45:53  <dcousens> see how bitcoind goes, if it gets past it
 61 2015-11-16T00:45:55  <dcousens> well
 62 2015-11-16T00:46:21  <dcousens> i know exactly where its blowing up, so I can hex dump and have a looksy :)
 63 2015-11-16T00:46:35  <dcousens> thankfully the parser only takes 23s to run
 64 2015-11-16T00:47:45  <sipa> that's impressive
 65 2015-11-16T00:47:52  <sipa> wait
 66 2015-11-16T00:48:03  <dcousens> no script validation, obviously
 67 2015-11-16T00:48:05  <sipa> you parse 2 Gbyte per second?
 68 2015-11-16T00:48:47  <dcousens> that sounds a bit majestic doesn't it
 69 2015-11-16T00:49:00  <dcousens> and not right...
 70 2015-11-16T00:49:26  <dcousens> but, its a full parse.  I've got a 100MB buffer and the disk IO is going hard
 71 2015-11-16T00:49:30  <dcousens> its on AWS
 72 2015-11-16T00:50:08  <dcousens> so reading 100MB at a time, lazy loads the block in terms of what I want out of it (in this case, for a script index)
 73 2015-11-16T00:50:09  <sipa> well how else do you parse 50 Gbyte of block data in 23s?
 74 2015-11-16T00:50:29  <sipa> oh, you don't actually read the block data, just the headers?
 75 2015-11-16T00:50:42  <dcousens> sipa: technically I do though, like, I'm not doing an ftell...
 76 2015-11-16T00:51:01  <sipa> not following
 77 2015-11-16T00:51:02  <dcousens> in terms of IO...
 78 2015-11-16T00:51:30  <dcousens> sipa: 2GB seems a bit fast for what IO should be, AFAIK the AWS SSD's are only about 600mb/s
 79 2015-11-16T00:51:59  <dcousens> remembering, this is 365k blocks
 80 2015-11-16T00:52:05  <sipa> ah, a bit less
 81 2015-11-16T00:52:18  <dcousens> assuming 0.5mb for the remaining 20k?
 82 2015-11-16T00:52:22  <dcousens> thats like, 10GB less
 83 2015-11-16T00:52:42  <dcousens> still... 1.7GB/s ?
 84 2015-11-16T01:13:16  <GitHub18> [bitcoin] CodeShark opened pull request #7023: Fixed integer comparison warning. (master...signed_int_comparison_fix) https://github.com/bitcoin/bitcoin/pull/7023
 85 2015-11-16T01:28:42  *** challisto has joined #bitcoin-core-dev
 86 2015-11-16T01:34:17  *** lightningbot has joined #bitcoin-core-dev
 87 2015-11-16T01:34:27  *** luke-jr_ has joined #bitcoin-core-dev
 88 2015-11-16T01:34:45  *** Luke-Jr has quit IRC
 89 2015-11-16T01:37:15  *** cfields has joined #bitcoin-core-dev
 90 2015-11-16T01:38:02  *** jonasschnelli has joined #bitcoin-core-dev
 91 2015-11-16T01:44:26  *** Ylbam has quit IRC
 92 2015-11-16T01:54:33  *** luke-jr_ is now known as Luke-Jr
 93 2015-11-16T02:19:06  *** d_t has joined #bitcoin-core-dev
 94 2015-11-16T02:25:00  *** CodeShark_ has quit IRC
 95 2015-11-16T02:36:01  <phantomcircuit> jgarzik, did you ever benchmark sqlite against leveldb with the WAL settings?
 96 2015-11-16T02:36:37  <jgarzik> phantomcircuit, yes and no - was in the 'obviously too slow' category
 97 2015-11-16T02:36:43  <jgarzik> sqlite4 storage engine looks interesting
 98 2015-11-16T02:37:30  <phantomcircuit> jgarzik, that's what i guessed
 99 2015-11-16T02:37:56  <phantomcircuit> the python data node i wrote in ~2010 had a super optimized sqlite3 db and even then it was comically slow
100 2015-11-16T02:38:10  <phantomcircuit> even with a proper db like postgres it was pretty horrendously slow
101 2015-11-16T03:20:32  <dcousens> jgarzik: even with prepared statements etc?
102 2015-11-16T03:20:41  <jgarzik> dcousens, yes
103 2015-11-16T03:20:41  *** droark has joined #bitcoin-core-dev
104 2015-11-16T03:29:27  *** d_t has quit IRC
105 2015-11-16T03:39:05  *** zooko has joined #bitcoin-core-dev
106 2015-11-16T03:40:58  *** d_t has joined #bitcoin-core-dev
107 2015-11-16T03:54:53  *** arowser has joined #bitcoin-core-dev
108 2015-11-16T04:01:35  *** go1111111 has quit IRC
109 2015-11-16T04:03:20  <gmaxwell> it's not really that sqllite is "slow", it's pretty fast for what it does; it's just that dedicated transactional key value stores are really quite fast.
110 2015-11-16T04:05:15  <gmaxwell> for me bitcoin core sync runs at around 8000tx/s (with secp256k1), even though every tx is involving a non-cachable missing record check for bip30 (in addition to all the cached stuff)... thats really quite remarkable IMO.
111 2015-11-16T04:15:22  *** go1111111 has joined #bitcoin-core-dev
112 2015-11-16T04:20:15  *** zooko has quit IRC
113 2015-11-16T04:22:30  *** challisto has quit IRC
114 2015-11-16T04:44:57  *** Anduck has quit IRC
115 2015-11-16T04:45:10  *** Anduck has joined #bitcoin-core-dev
116 2015-11-16T04:49:40  *** d_t has quit IRC
117 2015-11-16T05:18:04  *** zarathustra has quit IRC
118 2015-11-16T05:39:20  *** Anduck has quit IRC
119 2015-11-16T05:46:57  *** Anduck has joined #bitcoin-core-dev
120 2015-11-16T05:48:05  *** d_t has joined #bitcoin-core-dev
121 2015-11-16T06:41:30  *** Thireus1 has quit IRC
122 2015-11-16T06:41:50  *** Thireus has joined #bitcoin-core-dev
123 2015-11-16T06:45:33  *** Ylbam has joined #bitcoin-core-dev
124 2015-11-16T07:01:18  *** guest234234 has joined #bitcoin-core-dev
125 2015-11-16T07:03:12  *** Guest23423 has joined #bitcoin-core-dev
126 2015-11-16T07:06:11  *** guest234234 has quit IRC
127 2015-11-16T07:11:21  *** gavinand1esen has joined #bitcoin-core-dev
128 2015-11-16T07:11:24  *** btcdrak_ has joined #bitcoin-core-dev
129 2015-11-16T07:12:15  *** Anduck has quit IRC
130 2015-11-16T07:13:41  *** Anduck has joined #bitcoin-core-dev
131 2015-11-16T07:14:14  *** CodeShark_ has joined #bitcoin-core-dev
132 2015-11-16T07:15:00  *** morcos_ has joined #bitcoin-core-dev
133 2015-11-16T07:15:42  *** teward- has joined #bitcoin-core-dev
134 2015-11-16T07:15:48  *** btcdrak has quit IRC
135 2015-11-16T07:15:49  *** teward has quit IRC
136 2015-11-16T07:15:49  *** CodeShark has quit IRC
137 2015-11-16T07:15:49  *** morcos has quit IRC
138 2015-11-16T07:15:50  *** gavinandresen has quit IRC
139 2015-11-16T07:15:50  *** teward- is now known as teward
140 2015-11-16T07:15:55  *** CodeShark_ is now known as CodeShark
141 2015-11-16T07:16:04  *** Thireus has quit IRC
142 2015-11-16T07:16:21  *** btcdrak_ is now known as btcdrak
143 2015-11-16T07:16:22  *** Thireus has joined #bitcoin-core-dev
144 2015-11-16T07:28:37  *** Thireus has quit IRC
145 2015-11-16T07:30:06  *** Thireus has joined #bitcoin-core-dev
146 2015-11-16T07:32:50  *** s1w has quit IRC
147 2015-11-16T07:38:40  *** s1w has joined #bitcoin-core-dev
148 2015-11-16T07:38:56  *** jonasschnelli has quit IRC
149 2015-11-16T07:38:57  *** jonasschnelli has joined #bitcoin-core-dev
150 2015-11-16T07:39:04  *** s1w is now known as Guest16267
151 2015-11-16T07:40:05  <GitHub184> [bitcoin] jonasschnelli pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/b632145edeb3...814697c5569c
152 2015-11-16T07:40:06  <GitHub184> bitcoin/master 773ae46 Jonas Schnelli: [Qt] add shortcurts for debug-/console-window
153 2015-11-16T07:40:06  <GitHub184> bitcoin/master 814697c Jonas Schnelli: Merge pull request #7000...
154 2015-11-16T07:40:10  <GitHub19> [bitcoin] jonasschnelli closed pull request #7000: [Qt] add shortcurts for debug-/console-window (master...2015/11/qt_shortcuts) https://github.com/bitcoin/bitcoin/pull/7000
155 2015-11-16T07:40:44  *** Thireus has quit IRC
156 2015-11-16T07:41:04  *** Thireus has joined #bitcoin-core-dev
157 2015-11-16T07:42:03  * btcdrak claps. jonas' first merge :)
158 2015-11-16T07:43:14  <jonasschnelli> hah. Thanks btcdrak
159 2015-11-16T07:47:03  *** Thireus has quit IRC
160 2015-11-16T07:47:25  *** Thireus has joined #bitcoin-core-dev
161 2015-11-16T07:49:57  *** Thireus has quit IRC
162 2015-11-16T07:50:40  *** Thireus has joined #bitcoin-core-dev
163 2015-11-16T08:09:27  <gmaxwell> BlueMatt: did he mess up the signing? :)
164 2015-11-16T08:17:56  <phantomcircuit> jonasschnelli, *cough*
165 2015-11-16T08:18:14  <phantomcircuit> oh nvm ignore me
166 2015-11-16T08:18:37  <jonasschnelli> gmaxwell: hmm... i testes the verifiy-commit stuff before. Should work if one has a recent master where he verifies commit from.
167 2015-11-16T08:19:12  <jonasschnelli> master needs to have https://github.com/bitcoin/bitcoin/pull/7004
168 2015-11-16T08:19:25  <jonasschnelli> then my first merge should be valid. :) (hopefully)
169 2015-11-16T08:19:41  <phantomcircuit> jonasschnelli, it's properly signed with 32EE 5C4C 3FA1 5CCA DB46  ABE5 29D4 BCB6 416F 53EC
170 2015-11-16T08:20:27  <jonasschnelli> ID 416F53EC, 32EE 5C4C 3FA1 5CCA DB46  ABE5 29D4 BCB6 416F 53EC
171 2015-11-16T08:20:52  <jonasschnelli> and this matched that: https://github.com/bitcoin/bitcoin/pull/7004/files
172 2015-11-16T08:21:31  * jonasschnelli fingers where sweat during his first merge and double/tripple checked everything...
173 2015-11-16T08:22:56  <gmaxwell> heh.
174 2015-11-16T08:25:53  <wumpus> hah
175 2015-11-16T08:28:16  <wumpus> jonasschnelli: looks good to me (w/ --show-signature)
176 2015-11-16T08:29:08  <wumpus> (git log --merges --show-signature)
177 2015-11-16T08:29:29  <gmaxwell> jonasschnelli: Congrats!
178 2015-11-16T08:31:09  <wumpus> jonasschnelli: my plan was to walk you through at least one merge, but really needed some time out this weekend - seems you figured it out!
179 2015-11-16T08:33:17  <jonasschnelli> I'm using the `github-merge.sh` script (1:1) in two other projects (https://github.com/libbtc/libbtc and https://github.com/digitalbitbox/mcu), so I'm a little bit familiar with it (also with signed commits)
180 2015-11-16T08:36:34  <wumpus> great :D
181 2015-11-16T08:38:50  <GitHub194> [bitcoin] gmaxwell pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/814697c5569c...6876a78b863e
182 2015-11-16T08:38:50  <GitHub194> bitcoin/master 9bd3f03 Peter Todd: Clarify 'fee' field in fundrawtransaction help text...
183 2015-11-16T08:38:51  <GitHub194> bitcoin/master 6876a78 Gregory Maxwell: Merge pull request #6991...
184 2015-11-16T08:38:57  <GitHub101> [bitcoin] gmaxwell closed pull request #6991: Minor: Clarify 'fee' field in fundrawtransaction help text (master...clarify-fundrawtransaction-help) https://github.com/bitcoin/bitcoin/pull/6991
185 2015-11-16T08:40:29  *** jtimon has joined #bitcoin-core-dev
186 2015-11-16T08:50:49  <GitHub34> [bitcoin] laanwj closed pull request #7021: add bip65 tests to rpc-tests.sh -extended (in 0.11 branch) (0.11...11rpcfixups) https://github.com/bitcoin/bitcoin/pull/7021
187 2015-11-16T08:50:53  <GitHub193> [bitcoin] laanwj pushed 2 new commits to 0.11: https://github.com/bitcoin/bitcoin/compare/7e278929df53...595c8d6301cf
188 2015-11-16T08:50:54  <GitHub193> bitcoin/0.11 9730051 Alex Morcos: add bip65 tests to rpc-tests.sh -extended
189 2015-11-16T08:50:54  <GitHub193> bitcoin/0.11 595c8d6 Wladimir J. van der Laan: Merge pull request #7021...
190 2015-11-16T08:52:50  *** droark has quit IRC
191 2015-11-16T08:55:06  *** d_t has quit IRC
192 2015-11-16T09:13:03  *** Guyver2 has joined #bitcoin-core-dev
193 2015-11-16T09:13:30  <gmaxwell> bleh, so it looks like my gbt latency measurements going up coincide not with mempool flooding attacks but with updating to master. Now I need to figure out what hurt the performance so much.
194 2015-11-16T09:19:56  *** Thireus has quit IRC
195 2015-11-16T09:21:00  *** Thireus has joined #bitcoin-core-dev
196 2015-11-16T09:24:39  *** dcousens has quit IRC
197 2015-11-16T09:30:06  *** rubensayshi has joined #bitcoin-core-dev
198 2015-11-16T09:34:02  <wumpus> bleh indeed
199 2015-11-16T09:34:21  <gmaxwell> https://people.xiph.org/~greg/p2pool-gbt.png < did you see my graph? :(
200 2015-11-16T09:34:30  <wumpus> on the positive side, updating to master is easy to reproduce
201 2015-11-16T09:34:45  <gmaxwell> well I need to figure out what I was running before! :)
202 2015-11-16T09:34:51  <wumpus> git reflog?
203 2015-11-16T09:35:14  <gmaxwell> yea, I'll try (tomorrow, late here)
204 2015-11-16T09:35:21  <wumpus> no problem, there's always tons of metadata :p
205 2015-11-16T09:37:11  <wumpus> so it's going to be either a) the switch to libevent's http server, there may be some foul interaction with longpoll b) one of the mempool changes
206 2015-11-16T09:38:07  <wumpus> there's also c) something other, contending the cs_main lock  .. may even be related to the pruning-related changes. Ok, conclusion: no clue.
207 2015-11-16T09:38:45  <gmaxwell> your thinking is already much more constructive than mine, I was stuck at "oh no" :)
208 2015-11-16T09:40:00  <gmaxwell> the GBT latency doesn't LP. it's p2pool timing how long it takes it to make a gbt call on its own. The delay is constant, so I doubt contention unless its really hot.
209 2015-11-16T09:40:12  <gmaxwell> if it's libevent it might just be some silly tunable.
210 2015-11-16T09:40:32  <wumpus> well it may even be just some silly mistake
211 2015-11-16T09:40:52  <wumpus> e.g. creating an event but not triggering it
212 2015-11-16T09:47:18  <gmaxwell> hm. I don't see the time when I moved back to test 0.11.2rc on this graph. Interesting.
213 2015-11-16T09:48:32  *** Thireus has quit IRC
214 2015-11-16T09:48:53  *** Thireus has joined #bitcoin-core-dev
215 2015-11-16T09:49:50  *** Thireus has joined #bitcoin-core-dev
216 2015-11-16T09:50:45  <wumpus> btw using just seccomp-bpf it's not possible to limit the scope of opening files, seccomp scripts cannot dereference syscall argument pointers, it's limited to checks based on the numbers themselves  - e.g. disallow PROT_EXEC on mprotect
217 2015-11-16T09:50:59  *** paveljanik has joined #bitcoin-core-dev
218 2015-11-16T09:50:59  *** paveljanik has joined #bitcoin-core-dev
219 2015-11-16T09:51:06  <wumpus> so if you allow open, it can open any file on the file system
220 2015-11-16T09:51:28  <gmaxwell> oh crud, thats .. lame!
221 2015-11-16T09:51:43  <gmaxwell> well of course selinux and such can control what files you can open.
222 2015-11-16T09:52:09  <wumpus> it's possible to get around it with a dedicated process that traps some syscalls and then does the syscall for the restricted thread, but that's a lot more complexity
223 2015-11-16T09:52:53  <wumpus> all those require privileged operations or changes to the OS
224 2015-11-16T09:53:24  <wumpus> seccomp is IIRC the only one that can run unprivileged, chrome has a set-uid helper to do some jailing operations, but I don't like that
225 2015-11-16T09:54:12  <wumpus> something may be possible with namespaces too: https://code.google.com/p/chromium/issues/detail?id=312380
226 2015-11-16T09:54:15  <gmaxwell> Right, selinux requires privledge setup to establish the rules on the system (though after that the tags/contexts can be applied from userspace); I don't know how the apparmor stuff works.
227 2015-11-16T09:54:30  <wumpus> apparmor works the same, it is a global config file per application
228 2015-11-16T09:55:19  <gmaxwell> Hm I would have expected the namespaces thing to have some of the problems chroot has that makes you need privs to use it.
229 2015-11-16T09:55:57  <gmaxwell> oh well even if we can't limit file scope at least that easily done with selinux/apparmor, and we could limit other things.
230 2015-11-16T09:56:03  <wumpus> the advantage is that it can be set up externally, on the other hand that relies on specific files for the OS
231 2015-11-16T10:00:55  <wumpus> nice about seccomp is that it can be set on a per-thread level. Although that doesn't help as much as long as signing and key storage happens within the main process :-)
232 2015-11-16T10:01:40  <gmaxwell> oh I don't think I realized seccomp could be thread scoped.
233 2015-11-16T10:02:17  <wumpus> the idea is to set up a global seccomp filter as soon as possible before any threads are created
234 2015-11-16T10:02:22  <gmaxwell> though use of callbacks and event signaling would then make you have to be careful about what thread things actually excuted in.
235 2015-11-16T10:02:27  <wumpus> but in a thread it can be restricted further
236 2015-11-16T10:02:54  <wumpus> the security mesaure that amounst to is block all the exits but what people want to steal isn't outside, it's in the same memory space
237 2015-11-16T10:02:59  <gmaxwell> e.g. I'd say the network thread could lose all file IO. ... except I'm not sure that it can.
238 2015-11-16T10:03:33  <gmaxwell> sure if you lose control of control flow you've lost, but if the attacker can just call some libc functions, taking away file IO and such can be pretty limiting.
239 2015-11-16T10:03:35  *** Anduck has quit IRC
240 2015-11-16T10:04:10  *** Anduck has joined #bitcoin-core-dev
241 2015-11-16T10:04:18  <wumpus> so I think we need seccomp lockdown of the main process (at least the one that does P2P networking) + key storage and signing in a different process and memory space
242 2015-11-16T10:04:54  <wumpus> P2P networking would never have to see the passphrase or decryption key, for example
243 2015-11-16T10:05:11  *** CodeShark_ has joined #bitcoin-core-dev
244 2015-11-16T10:05:37  *** CodeShark_ has quit IRC
245 2015-11-16T10:06:06  <wumpus> "sure if you lose control of control flow you've lost" the whole idea is to mitigate that, that if you have control flow, you can only keep up doing what the thread was doing anyway because nothing else is allowed :-)
246 2015-11-16T10:06:09  <GitHub45> [bitcoin] jonasschnelli opened pull request #7025: [Qt] refactor and optimize proxy settings behavior (master...2015/11/qt_settingsvalidation) https://github.com/bitcoin/bitcoin/pull/7025
247 2015-11-16T10:06:38  <gmaxwell> yea, I want to do signing in another process in any case for a multitude of reasons. Including being able to keep passphrases out of ram... being able to use HSMs, etc.
248 2015-11-16T10:06:49  <wumpus> right
249 2015-11-16T10:07:24  <wumpus> and my point is that I think that's a more useful step, given concrete thread models, than seccomp right now
250 2015-11-16T10:07:48  <gmaxwell> annoyingly though, if compromising the threads that are doing network deseralization can be escaped to take over the user account, the likely they can compromise the signing process too.
251 2015-11-16T10:08:00  <gmaxwell> Ah. Fair enough.
252 2015-11-16T10:08:07  <wumpus> e.g. the UPnP vulnerability would have allowed getting control of a thread that has network access, no amount of lockdown couldh ave avoided it from leaking secrets from the process memory
253 2015-11-16T10:08:31  <wumpus> *apart* from having the secrets in a seperate process in the first place
254 2015-11-16T10:08:31  <gmaxwell> UPNP could also be easily moved into an external program, FWIW.
255 2015-11-16T10:08:53  <wumpus> that's not my point - UPnP is just an example. The worst case scenario would be an attack through P2P
256 2015-11-16T10:08:53  <gmaxwell> but thats much less interesting than seperating the secrets from the attack surface, as upnp is just one vector.
257 2015-11-16T10:09:33  <wumpus> right, better to protect what has to be protected instead
258 2015-11-16T10:09:44  <gmaxwell> I do think if we decide we'd like to turn upnp back on, it should go in another process and _that_ one we could seccomp to have no access to open(). :)
259 2015-11-16T10:10:01  <gmaxwell> (even if we are otherwise protecting the signing)
260 2015-11-16T10:11:46  <wumpus> sure
261 2015-11-16T10:12:20  <wumpus> "given concrete thread models" eh THREAT models
262 2015-11-16T10:22:07  *** Thireus has quit IRC
263 2015-11-16T10:22:25  *** Thireus has joined #bitcoin-core-dev
264 2015-11-16T10:22:55  *** Thireus has joined #bitcoin-core-dev
265 2015-11-16T10:24:03  *** Thireus has quit IRC
266 2015-11-16T10:24:25  *** Thireus has joined #bitcoin-core-dev
267 2015-11-16T10:28:23  *** Guest23423 has quit IRC
268 2015-11-16T10:37:06  *** guest234234 has joined #bitcoin-core-dev
269 2015-11-16T10:43:35  *** randy-waterhouse has quit IRC
270 2015-11-16T10:48:39  <wumpus> it's possible to use unprivileged user namespaces, but only on linux kernel 3.18+
271 2015-11-16T10:49:05  <btcdrak> the message "UpdateTip: 1 of last 100 blocks above version 4" is not correct. It's above 3.
272 2015-11-16T10:51:31  <wumpus> BTW: if you get "/home/user/bitcoin/src/key.cpp:204: undefined reference to `secp256k1_ecdsa_sign_recoverable'"  errors after updating to master you need to clean your git tree  (since #6983)
273 2015-11-16T11:02:19  *** dcousens has joined #bitcoin-core-dev
274 2015-11-16T11:03:32  <GitHub134> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/6876a78b863e...dafefb79244d
275 2015-11-16T11:03:32  <GitHub134> bitcoin/master aee22bf Gregory Maxwell: Avoid a compile error on hosts with libevent too old for EVENT_LOG_WARN....
276 2015-11-16T11:03:33  <GitHub134> bitcoin/master dafefb7 Wladimir J. van der Laan: Merge pull request #7016...
277 2015-11-16T11:03:37  <GitHub72> [bitcoin] laanwj closed pull request #7016: Avoid a compile error on hosts with libevent too old for EVENT_LOG_WARN. (master...without_EVENT_LOG_WARN) https://github.com/bitcoin/bitcoin/pull/7016
278 2015-11-16T11:03:47  <midnightmagic> yay!
279 2015-11-16T11:23:03  <wumpus> user namespaces create a 'fantasy world' for a process in which it is root, but root is mapped (by the namespace) to the user own uid - it does have capabilities such as chroot *within* this user namespace, but it doesn't let it use its elevated privileges outside that. This gives me a headache.
280 2015-11-16T11:23:35  <wumpus> "Over the years, there have been a lot of features that have been added to the Linux kernel that have been made available only to privileged users because of their potential to confuse set-user-ID-root applications.  In general, it becomes safe to allow the root user in a user namespace to use those features because it is impossible, while in a  user  namespace, to gain more privilege than the root user of a user namespace has"
281 2015-11-16T11:26:35  *** Anduck has quit IRC
282 2015-11-16T11:28:13  *** Anduck has joined #bitcoin-core-dev
283 2015-11-16T11:44:35  *** Anduck has quit IRC
284 2015-11-16T11:45:43  <wumpus> not really promising regarding stable API that the example program in "man user_namespaces" is already broken in kernel 4.2.0
285 2015-11-16T11:48:43  <wumpus> ah there was a security problem, and the kernel as well as man page was updated, but didn't have that version: http://man7.org/linux/man-pages/man7/user_namespaces.7.html#EXAMPLE
286 2015-11-16T11:50:13  *** Anduck has joined #bitcoin-core-dev
287 2015-11-16T12:02:15  *** Anduck has quit IRC
288 2015-11-16T12:02:33  *** Anduck has joined #bitcoin-core-dev
289 2015-11-16T12:03:32  *** paveljanik has quit IRC
290 2015-11-16T12:04:23  <GitHub43> [bitcoin] MarcoFalke opened pull request #7026: [contrib] Update versionprefix to "bitcoin-core" in verify.sh (master...MarcoFalke-2015-verify) https://github.com/bitcoin/bitcoin/pull/7026
291 2015-11-16T12:25:17  <GitHub159> [bitcoin] MarcoFalke opened pull request #7027: [qa] rpc-tests: Properly use test framework (master...MarcoFalke-2015-rpcTestCleanup) https://github.com/bitcoin/bitcoin/pull/7027
292 2015-11-16T12:25:27  <GitHub113> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/dafefb79244d...e54ebbf60097
293 2015-11-16T12:25:28  <GitHub113> bitcoin/master 6e18268 Pieter Wuille: Switch to libsecp256k1-based validation for ECDSA
294 2015-11-16T12:25:28  <GitHub113> bitcoin/master e54ebbf Wladimir J. van der Laan: Merge pull request #6954...
295 2015-11-16T12:25:32  <GitHub119> [bitcoin] laanwj closed pull request #6954: Switch to libsecp256k1-based ECDSA validation (master...secp256k1) https://github.com/bitcoin/bitcoin/pull/6954
296 2015-11-16T12:25:43  *** MarcoFalke has joined #bitcoin-core-dev
297 2015-11-16T12:28:07  * jonasschnelli applaud to the merge of  #6954
298 2015-11-16T12:28:12  *** dcousens has quit IRC
299 2015-11-16T12:32:21  <jonasschnelli> wumpus: sorry for 6900, i could not create a trusty base image. Seems not to work on my debian7 machine. I tried to change the gitian vm builder... but looks like a VitrualBox mkfs issue.
300 2015-11-16T12:32:39  <jonasschnelli> The only gitian host i have is debian...
301 2015-11-16T12:32:47  * jonasschnelli needs more nativ linux machines...
302 2015-11-16T12:43:50  *** mm_1 has quit IRC
303 2015-11-16T12:44:27  *** mm_1 has joined #bitcoin-core-dev
304 2015-11-16T12:48:21  *** ParadoxSpiral has joined #bitcoin-core-dev
305 2015-11-16T12:58:30  <phantomcircuit> wumpus, neat
306 2015-11-16T13:01:29  <sipa> yay :)
307 2015-11-16T13:02:45  <phantomcircuit> sipa, how long have you been working on libsecp256k1?
308 2015-11-16T13:03:18  <sipa> i started somewhere in 2013, i think
309 2015-11-16T13:03:20  *** MarcoFalke has quit IRC
310 2015-11-16T13:03:33  *** MarcoFalke has joined #bitcoin-core-dev
311 2015-11-16T13:11:34  <wumpus> jonasschnelli: doesn't strictly need to be native - w/ lxc you can also use VMs, I think it's even possible to do nested kvm on recent CPUs
312 2015-11-16T13:14:51  <btcdrak> sipa: wow, that is like a life's work
313 2015-11-16T13:24:29  <jl2012> Sorry if this is a stupid question. Is there any contingency plan if libsecp256k1 turns out to be a softfork or hardfork?
314 2015-11-16T13:30:55  <instagibbs> or both? ;)
315 2015-11-16T13:35:12  <sipa> if there is any difference with openssl, it is a hard fork
316 2015-11-16T13:35:37  <sipa> because you can negate the outcome of checksig
317 2015-11-16T13:35:59  <sipa> so any strictening of rules can be turned into widening, in theory
318 2015-11-16T13:37:26  <sipa> though at this point i think it's fair to say that due the amount of testing and formal validation that went into libsecp256k1, that is more likely to be a bug in openssl then
319 2015-11-16T13:38:56  <sipa> also, it may not be exploitable... the BN_sqr bug we found in OpenSSL (thanks to libsecp256k1's unit tests that compared with OpenSSL) was technically a hard fork for bitcoin when it got foxed, but an unexploitable one as far as we know
320 2015-11-16T13:39:06  <sipa> *fixed
321 2015-11-16T13:39:59  <sipa> so i guess it would depend on what sort of consensus change it would be
322 2015-11-16T13:42:14  *** d_t has joined #bitcoin-core-dev
323 2015-11-16T13:42:51  *** d_t has joined #bitcoin-core-dev
324 2015-11-16T13:43:01  <sipa> and if it's before 0.12 is released, we can always just revert it :)
325 2015-11-16T13:45:48  *** MarcoFalke has quit IRC
326 2015-11-16T13:53:07  *** zmanian_ has quit IRC
327 2015-11-16T13:54:43  *** zmanian_ has joined #bitcoin-core-dev
328 2015-11-16T13:54:49  <instagibbs> sipa, is there a summary of the review/testing process for it?
329 2015-11-16T14:00:13  *** MarcoFalke has joined #bitcoin-core-dev
330 2015-11-16T14:00:27  *** davec has quit IRC
331 2015-11-16T14:02:57  *** d_t has quit IRC
332 2015-11-16T14:06:26  *** Guest95455 has quit IRC
333 2015-11-16T14:07:31  <sipa> instagibbs: not really, but i agree we need that
334 2015-11-16T14:07:39  *** pigeons has joined #bitcoin-core-dev
335 2015-11-16T14:08:02  *** pigeons is now known as Guest89319
336 2015-11-16T14:08:51  <sipa> instagibbs: several types of tests are not yet merged (though they pass fine), including sage scripts to algebraically verify the group formulas, and an alterrnative build that results in a very small size curve so we can do high-level exhaustive testing
337 2015-11-16T14:08:54  *** d_t has joined #bitcoin-core-dev
338 2015-11-16T14:09:38  <sipa> those should go in before 0.12, as well as some document describing all the various methods
339 2015-11-16T14:13:02  *** d_t has quit IRC
340 2015-11-16T14:15:23  <jonasschnelli> wumpus: get a linker error when building #6900 (trusty gitian). Check: https://bitcoin.jonasschnelli.ch/pulls/6900/build-linux.log (very bottom)
341 2015-11-16T14:41:58  <GitHub162> [bitcoin] MarcoFalke opened pull request #7028: [doc] qa: Move README.md and update -help (master...MarcoFalke-2015-qaReadme) https://github.com/bitcoin/bitcoin/pull/7028
342 2015-11-16T14:54:42  *** grieve has quit IRC
343 2015-11-16T14:58:10  <GitHub151> [bitcoin] sdaftuar opened pull request #7029: Remove unmaintained example test script_test.py (master...script-test-cleanup) https://github.com/bitcoin/bitcoin/pull/7029
344 2015-11-16T14:58:12  <jgarzik> Great to see libsecp256k1 go in
345 2015-11-16T14:58:51  <btcdrak> and openssl to be shown the door
346 2015-11-16T14:59:22  *** guest234234 has quit IRC
347 2015-11-16T15:08:15  <GitHub155> [bitcoin] MarcoFalke opened pull request #7030: [contrib] Delete test-patches and move testgen to devtools (master...MarcoFalke-2015-contribC) https://github.com/bitcoin/bitcoin/pull/7030
348 2015-11-16T15:12:26  *** davec has joined #bitcoin-core-dev
349 2015-11-16T15:19:16  *** zooko has joined #bitcoin-core-dev
350 2015-11-16T15:23:03  *** fkhan has quit IRC
351 2015-11-16T15:33:13  *** skyraider has joined #bitcoin-core-dev
352 2015-11-16T15:33:35  *** Thireus has quit IRC
353 2015-11-16T15:34:22  *** Thireus has joined #bitcoin-core-dev
354 2015-11-16T15:36:26  *** fkhan has joined #bitcoin-core-dev
355 2015-11-16T15:39:46  *** bsm1175321 has joined #bitcoin-core-dev
356 2015-11-16T15:44:57  *** treehug88 has joined #bitcoin-core-dev
357 2015-11-16T15:47:20  <wumpus> jonasschnelli: don't really understand
358 2015-11-16T15:47:34  <jonasschnelli> wumpus: The 6900 issues?
359 2015-11-16T15:47:38  <wumpus> yes
360 2015-11-16T15:47:49  <jonasschnelli> seems something with boost
361 2015-11-16T15:48:13  <wumpus> I thought the point of gitian was that the results are the same for everyone
362 2015-11-16T15:48:38  <wumpus> you're also using kvm? or lxc?
363 2015-11-16T15:48:51  <jonasschnelli> ha... right.. maybe i do it slightly different..
364 2015-11-16T15:48:57  <jonasschnelli> wumpus: kvm,... not i try to build 0340ffb
365 2015-11-16T15:49:10  <jonasschnelli> not/now
366 2015-11-16T15:54:27  <wumpus> the non-working boost sleep implementation error usually means it cannot link against boost at all
367 2015-11-16T15:54:54  <wumpus> unfortunately it's not really easy to get the config.log in the general case from the gitian builder
368 2015-11-16T15:56:31  *** Thireus has quit IRC
369 2015-11-16T15:56:52  *** Thireus has joined #bitcoin-core-dev
370 2015-11-16T15:58:09  <wumpus> linux seems still to be building properly here - although it has the dependencies cached, could try removing the cache and starting over from the beginning but that's going to take a long time
371 2015-11-16T16:13:48  <jonasschnelli> linux is still building here...
372 2015-11-16T16:20:40  <jonasschnelli> wumpus : still have the linking issue on linux (https://bitcoin.jonasschnelli.ch/pulls/6900/build-linux.log)
373 2015-11-16T16:21:01  <wumpus> can you try removing the cache? maybe something went wrong there
374 2015-11-16T16:22:26  *** MarcoFalke has quit IRC
375 2015-11-16T16:22:35  *** Thireus has quit IRC
376 2015-11-16T16:22:50  *** treehug88 has quit IRC
377 2015-11-16T16:22:51  *** Thireus has joined #bitcoin-core-dev
378 2015-11-16T16:27:24  <jonasschnelli> wumpus: remove cache = rm -Rf gitian-builder/cache/common/*?
379 2015-11-16T16:28:07  <wumpus> not common
380 2015-11-16T16:28:18  <wumpus> I think only bitcoin-linux-0.12
381 2015-11-16T16:28:43  * jonasschnelli ran rm -Rf gitian-builder/cache/bitcoin-linux-0.12/  ... does build again now
382 2015-11-16T16:29:09  <wumpus> I think the problem in your case is that it was using the previous cache (built on precise) on trusty
383 2015-11-16T16:31:56  *** zooko has quit IRC
384 2015-11-16T16:32:14  *** Thireus has quit IRC
385 2015-11-16T16:32:14  <jonasschnelli> right. That would make sense.
386 2015-11-16T16:32:28  *** zooko has joined #bitcoin-core-dev
387 2015-11-16T16:32:36  *** Thireus has joined #bitcoin-core-dev
388 2015-11-16T16:34:43  <wumpus> going to try following gitian-building.md on a new debian8 vm
389 2015-11-16T16:36:26  *** Thireus has quit IRC
390 2015-11-16T16:36:46  *** Thireus has joined #bitcoin-core-dev
391 2015-11-16T16:46:53  *** Thireus has quit IRC
392 2015-11-16T16:47:11  *** Thireus has joined #bitcoin-core-dev
393 2015-11-16T16:52:35  *** treehug88 has joined #bitcoin-core-dev
394 2015-11-16T16:58:23  *** zooko has quit IRC
395 2015-11-16T17:00:40  *** Thireus has quit IRC
396 2015-11-16T17:01:01  *** Thireus has joined #bitcoin-core-dev
397 2015-11-16T17:02:02  *** Thireus has joined #bitcoin-core-dev
398 2015-11-16T17:05:00  *** Thireus has quit IRC
399 2015-11-16T17:05:21  *** Thireus has joined #bitcoin-core-dev
400 2015-11-16T17:22:48  <jonasschnelli> wumpus: seems to work now: https://bitcoin.jonasschnelli.ch/pulls/6900/
401 2015-11-16T17:43:23  *** d_t has joined #bitcoin-core-dev
402 2015-11-16T17:43:29  <morcos_> sipa: Congrats and thank you and everybody who helped with libsecp256k1.  Great work!
403 2015-11-16T17:44:52  <sipa> morcos_: thanks :)
404 2015-11-16T17:46:13  *** morcos_ is now known as morcos
405 2015-11-16T17:56:01  *** rubensayshi has quit IRC
406 2015-11-16T18:03:15  *** Anduck has quit IRC
407 2015-11-16T18:04:02  *** Anduck has joined #bitcoin-core-dev
408 2015-11-16T18:28:01  *** Guest89319 is now known as pigeons
409 2015-11-16T18:30:33  <morcos> jonasschnelli: when you have a chance, i'd like to get your thoughts on #6134.  i want to make sure i understood your comments.
410 2015-11-16T18:55:59  *** PRab_ has joined #bitcoin-core-dev
411 2015-11-16T18:56:59  *** PRab has quit IRC
412 2015-11-16T18:56:59  *** PRab_ is now known as PRab
413 2015-11-16T19:12:24  *** Naphex has quit IRC
414 2015-11-16T19:13:39  *** PRab_ has joined #bitcoin-core-dev
415 2015-11-16T19:14:50  *** PRab__ has joined #bitcoin-core-dev
416 2015-11-16T19:16:59  *** PRab has quit IRC
417 2015-11-16T19:18:40  *** PRab_ has quit IRC
418 2015-11-16T19:18:46  *** PRab has joined #bitcoin-core-dev
419 2015-11-16T19:19:39  *** PRab__ has quit IRC
420 2015-11-16T19:31:01  <jonasschnelli> wumpus: https://bitcoin.jonasschnelli.ch/pulls/6900/build-win.log MinGW still has troubles with boost #6900
421 2015-11-16T19:31:24  <jonasschnelli> ah. wait. I forgot to clean the cache for windows. Nm.
422 2015-11-16T19:31:31  <jonasschnelli> morcos: sure.
423 2015-11-16T19:31:36  <jonasschnelli> Let me open the PR first
424 2015-11-16T19:34:23  <phantomcircuit> 2015-11-16 19:34:00 PartitionCheck : Found 24 blocks in the last 4 hours
425 2015-11-16T19:34:23  <phantomcircuit> 2015-11-16 19:34:00 PartitionCheck : likelihood: 0.0811515
426 2015-11-16T19:34:39  <phantomcircuit> that seems like a small window
427 2015-11-16T19:37:26  <jonasschnelli> morcos: I have to admit, that i haven't fully reviewed #6134, my focus was the GUI part.
428 2015-11-16T19:37:54  <morcos> jonasschnelli: i just wanted to make sure i understood your comment about never returning -1
429 2015-11-16T19:38:12  <jonasschnelli> I think for an user it's strange that he might get the same fee for targeting conf. withing the next 25blocks and the next 18blocks.
430 2015-11-16T19:38:35  <jonasschnelli> morcos: I was calling estimatefee instead of estimateaproxfee
431 2015-11-16T19:38:37  <morcos> part of the point of the PR was to make it so you got -1 a lot less often.  but i didn't change what estimatefee returns, i just made a new function estimateapproxfee which is smarter
432 2015-11-16T19:38:45  <morcos> perhaps i should call it estimatesmartfee instead
433 2015-11-16T19:39:27  <morcos> that one will still return -1, but only at the very startup when there is not enough data period (and you don't have an old estimates file, although thats maybe dangerous for its own reasons)
434 2015-11-16T19:39:30  <jonasschnelli> I think the name is okay. I just wasn't really concentrated while reviewing and mixed up the extimateaproxfee and estimatefee
435 2015-11-16T19:39:41  <morcos> ok
436 2015-11-16T19:39:52  <morcos> as for same fee for different confirmations
437 2015-11-16T19:39:55  <jonasschnelli> UI: why can't you target for 1 block?
438 2015-11-16T19:40:06  <jonasschnelli> (conf. withing next block)
439 2015-11-16T19:40:14  <jonasschnelli> *within
440 2015-11-16T19:40:30  <morcos> i'm not really sure there is anything to be done about that.  its just as accurate an answer as you can get from the data
441 2015-11-16T19:40:42  <morcos> how do you mean, you can target for 1 block
442 2015-11-16T19:40:58  <jonasschnelli> The slider i got was from block 25 to block 2.
443 2015-11-16T19:41:13  <jonasschnelli> So, as enduser i would whish i could select 1 block.
444 2015-11-16T19:41:16  <morcos> ok, so that is the UI change i wanted fee back from
445 2015-11-16T19:41:28  <morcos> if you look at the right end of the slider
446 2015-11-16T19:41:35  <morcos> position 1 and position 2, both say 2 blocks
447 2015-11-16T19:42:08  <jonasschnelli> Right. I saw that. Is that because 1 block is just very hard (impossible to estimate)?
448 2015-11-16T19:42:28  <morcos> so you are actually trying to get an estiamte for 1 block, but estimatefee returns -1.   so estimateapproxfee looks at 2 blocks, gives you an answer there, and lets you know that the answer is for 2, so you're not expecting to get confirmed in 1 with that fee
449 2015-11-16T19:42:29  <jonasschnelli> * ... hard (impossible) to estimate?
450 2015-11-16T19:42:53  <morcos> yes, thats correct, there is not enough data to indicate that any fee will have very high likelihood of being confirmed in 1 block
451 2015-11-16T19:43:29  <morcos> this is primarily because i require 95% (used to be 85%) of txs with that fee or higher to be confirmed within the desired number of blocks, and often miners only recalculate their block every minute
452 2015-11-16T19:43:49  <morcos> so that puts a ceiling of about 90% on the chance you will be confirmed in the next blcok
453 2015-11-16T19:44:53  <morcos> my prior version of the change had the slider snap back to position 2, but that seemed like i did not implement that very well.  so the simpler change is this, but i'm happy for suggestions on how to improve it from a UI standpoint
454 2015-11-16T19:45:29  <morcos> what i'm trying to communicate is we can't give you an answer for 1 block, the best i can do is give you an answer for 2 blocks and that answer is X
455 2015-11-16T19:45:51  <jonasschnelli> hm... hmm.. somehow it feels that the user must be able to select 1 block... i know, it's not possible to estimate... :)
456 2015-11-16T19:46:49  <jonasschnelli> we could also go the way other wallets do: "confirmation speed" : "slow" = ~8b | "avg" ~4b | "fast" ~2b.
457 2015-11-16T19:47:19  <jonasschnelli> we just use the words "slow", "average", "fest"
458 2015-11-16T19:47:26  <jonasschnelli> *"fast"
459 2015-11-16T19:47:35  <morcos> i do think the ability to choose any # is overkill
460 2015-11-16T19:47:44  <morcos> but maybe thats too big a change now for 0.12?
461 2015-11-16T19:48:20  <jonasschnelli> I mean, it's nitpicking. The change loos good to me (need to test more in detail).
462 2015-11-16T19:48:25  <jonasschnelli> *looks
463 2015-11-16T19:48:39  <jonasschnelli> what do you think by aggregating block/fees.
464 2015-11-16T19:48:51  <morcos> ok...  what do you think about changing estimateapproxfee to estimatesmartfee.  i think i like that name better
465 2015-11-16T19:49:14  <morcos> better holds to how i think that function will evolve in the future
466 2015-11-16T19:49:18  <jonasschnelli> yeah. Smart* sounds always good. I also like estimatesmartfee better.
467 2015-11-16T19:49:22  <morcos> aggregating block/fees?
468 2015-11-16T19:49:51  <jonasschnelli> the thing is, if you move the slider, you might get the same fee between block 25 and block 18 (was the case in my test),...
469 2015-11-16T19:50:11  <jonasschnelli> we could just stop at block 18 (as min value, to very left).
470 2015-11-16T19:50:23  <morcos> hmm.. the problem is its not really possible to know any 2 targets would give you the same number until you ask
471 2015-11-16T19:51:08  <jonasschnelli> we could run from 2-25, then aggregate values, invalidate (this cache) after a new block comes in (or similar)
472 2015-11-16T19:51:26  <jonasschnelli> but, yeah. sounds after some lines of code.
473 2015-11-16T19:51:36  <jonasschnelli> could also be done later.
474 2015-11-16T19:51:44  <jonasschnelli> I mean the PR should not be hold back because of the UI!
475 2015-11-16T19:51:46  <morcos> yeah i think after we get 0.12.  we should talk about how to improve it
476 2015-11-16T19:52:19  <morcos> maybe do something like only store results for 1,2,4,8,16  (and then possibly add longer time horizons: 32,64,128,256?)
477 2015-11-16T19:52:28  <morcos> and then thats the only place the slider stops
478 2015-11-16T19:52:39  <morcos> perhaps they will sometimes give the same value, but a lot less often
479 2015-11-16T19:52:42  <jonasschnelli> yes. This could make sense.
480 2015-11-16T19:53:23  <jonasschnelli> IMO we should keep the UI as it is for #6134 and focus on get it in.
481 2015-11-16T19:53:38  <jonasschnelli> What is the reason to have two estimatefee rpc calls. Keep the API?
482 2015-11-16T19:53:46  <morcos> ok agreed.  especially as i am out of town for 10 days starting on friday!
483 2015-11-16T19:53:59  <morcos> yeah i think fee estimation still has some evolving to do
484 2015-11-16T19:54:00  <jonasschnelli> Take your laptop with you. :)
485 2015-11-16T19:54:17  <morcos> so i'd like to have estimatesmartfee be the new API, that will evolve with the functionality
486 2015-11-16T19:54:38  <morcos> but for the meantime keep the old API consistent until we have a sense of where the smart version will end up
487 2015-11-16T19:55:25  <jonasschnelli> will it only be estimation? Or would a RPC call "smartfee estiamte" make more sense?
488 2015-11-16T19:55:48  <morcos> i didn't understand
489 2015-11-16T19:55:49  <jonasschnelli> *estimate
490 2015-11-16T19:56:43  <jonasschnelli> I'm just thinking loud. Maybe other smartfee interaction would be possible in future, setting some parameters or similar.
491 2015-11-16T19:56:45  <morcos> oh, like you could configure state within the estimator, like smartfee timehorizon 1008
492 2015-11-16T19:56:56  <jonasschnelli> Then we could collect everything in the smartfee rpc call.
493 2015-11-16T19:57:33  <morcos> well i think maybe i should just call it smartfee and smartpriority ?
494 2015-11-16T19:57:35  <jonasschnelli> something like "smartfee estimate" and "smartfee set timehorizon 100"
495 2015-11-16T19:57:55  *** tripleslash has quit IRC
496 2015-11-16T19:58:18  <morcos> i'm not sure we'll want to set state like that, i think it would be safer to only be able to configure parameters on startup
497 2015-11-16T19:58:29  *** AtashiCon has quit IRC
498 2015-11-16T19:58:33  *** Arnavion3 has joined #bitcoin-core-dev
499 2015-11-16T19:58:37  *** Arnavion3 is now known as AtashiCon
500 2015-11-16T19:58:38  *** gribble has quit IRC
501 2015-11-16T19:58:44  <jonasschnelli> Hm.. no. better keep it "estimatesmartfee".
502 2015-11-16T19:59:11  <morcos> ok...  estimatesmartfee it is.  i'll rebase at some point with just the name change...
503 2015-11-16T19:59:15  <jonasschnelli> We could still have something like "estiamtesmartfee settimehorzon X"
504 2015-11-16T19:59:35  <jonasschnelli> I'll try to do a full test after your rebase.
505 2015-11-16T19:59:43  <morcos> ok
506 2015-11-16T19:59:47  <jonasschnelli> Keep the UI stuff as it is. We can polish that later.
507 2015-11-16T20:01:26  <morcos> yikes, so many commits, i'm just going to make one final commit which changes the names i think...
508 2015-11-16T20:01:45  <jonasschnelli> Ok.
509 2015-11-16T20:02:08  <jonasschnelli> morcos: one thing: I think we sould remove ""\nWARNING: This interface is unstable and may disappear or change!\n""
510 2015-11-16T20:02:38  <morcos> oh why?  i wanted to have that in there so people didn't write code depending on estimatesmartfee
511 2015-11-16T20:02:51  <morcos> i'm keeping the estimatefee interface the same
512 2015-11-16T20:03:08  <morcos> and for the most part they can design their own smart algorithm by using estimatefee directly
513 2015-11-16T20:03:32  <morcos> but i don't want to tie down the interface to the built-in smart algorithm now
514 2015-11-16T20:03:42  <morcos> i almost didn't expose it as rpc calls, but its nice for testing
515 2015-11-16T20:03:48  <jonasschnelli> Well, i think if we pack that into a release, we need to consider that people write code depending on that... new features in a 0. version are not meant to habe stable API IMO
516 2015-11-16T20:04:08  <jonasschnelli> *have
517 2015-11-16T20:04:21  <morcos> well thats the point of the warning, so they don't write code or they're aware.  i think that was wumpus' recommendation
518 2015-11-16T20:04:25  <jonasschnelli> But no strong opinion on that.
519 2015-11-16T20:04:39  <jonasschnelli> Okay. Then lets keep this.
520 2015-11-16T20:04:43  <morcos> ok thx
521 2015-11-16T20:05:10  *** gribble has joined #bitcoin-core-dev
522 2015-11-16T20:06:05  <jonasschnelli> morcos: nice code/PR by the way. You guys always come directly with a rpc test and a bunch of unittests!
523 2015-11-16T20:06:22  <morcos> ha, sdaftuar makes me!
524 2015-11-16T20:06:37  <jonasschnelli> hah!
525 2015-11-16T20:06:42  <jonasschnelli> Good boy.
526 2015-11-16T20:07:08  *** tripleslash has joined #bitcoin-core-dev
527 2015-11-16T20:42:49  *** belcher has joined #bitcoin-core-dev
528 2015-11-16T20:43:08  *** belcher is now known as Guest49481
529 2015-11-16T20:43:34  *** Guest49481 has quit IRC
530 2015-11-16T20:45:15  *** belcher has joined #bitcoin-core-dev
531 2015-11-16T21:00:25  *** d_t has quit IRC
532 2015-11-16T21:04:55  *** Anduck has quit IRC
533 2015-11-16T21:05:19  *** Anduck has joined #bitcoin-core-dev
534 2015-11-16T21:17:23  *** d_t has joined #bitcoin-core-dev
535 2015-11-16T21:32:18  *** randy-waterhouse has joined #bitcoin-core-dev
536 2015-11-16T21:37:26  <gmaxwell> https://www.reddit.com/r/Bitcoin/comments/3t04fp/psa_please_upgrade_to_bitcoin_core_0112_to/cx2b117 < anyone know whats up with someone saying they can't upgrade bitcoin core without upgrading OSX?
537 2015-11-16T21:45:24  *** Guyver2 has quit IRC
538 2015-11-16T22:11:59  *** paveljanik has joined #bitcoin-core-dev
539 2015-11-16T22:11:59  *** paveljanik has joined #bitcoin-core-dev
540 2015-11-16T22:12:43  <treehug88> gmaxwell : he doesn't mention which OS version he's currently on. I had no trouble upgrading to 0.11.2 on 10.10.5 (Yosemite)
541 2015-11-16T22:18:18  *** treehug88 has quit IRC
542 2015-11-16T22:19:37  *** paveljanik has quit IRC
543 2015-11-16T22:32:42  *** ParadoxSpiral has quit IRC
544 2015-11-16T22:52:26  *** jgarzik has quit IRC
545 2015-11-16T22:54:15  *** davec has quit IRC
546 2015-11-16T22:54:16  *** bsm117532 has quit IRC
547 2015-11-16T22:54:17  *** gmaxwell has quit IRC
548 2015-11-16T22:54:44  *** bsm1175321 has quit IRC
549 2015-11-16T22:54:45  *** berndj has quit IRC
550 2015-11-16T22:54:45  *** isis has quit IRC
551 2015-11-16T22:55:31  *** gmaxwell has joined #bitcoin-core-dev
552 2015-11-16T22:55:55  *** gmaxwell is now known as Guest42878
553 2015-11-16T22:55:58  *** davec has joined #bitcoin-core-dev
554 2015-11-16T22:56:20  *** Guest42878 has quit IRC
555 2015-11-16T22:56:20  *** Guest42878 has joined #bitcoin-core-dev
556 2015-11-16T22:56:30  *** Guest42878 is now known as gmaxwell
557 2015-11-16T22:59:28  *** berndj has joined #bitcoin-core-dev
558 2015-11-16T23:00:46  *** gribble has quit IRC
559 2015-11-16T23:03:25  *** isis has joined #bitcoin-core-dev
560 2015-11-16T23:05:13  *** wump has joined #bitcoin-core-dev
561 2015-11-16T23:06:50  *** wumpus has quit IRC
562 2015-11-16T23:07:13  *** bsm117532 has joined #bitcoin-core-dev
563 2015-11-16T23:09:03  *** zooko has joined #bitcoin-core-dev
564 2015-11-16T23:09:32  *** bsm1175321 has joined #bitcoin-core-dev
565 2015-11-16T23:09:38  *** guest234234 has joined #bitcoin-core-dev
566 2015-11-16T23:10:08  *** gribble has joined #bitcoin-core-dev
567 2015-11-16T23:13:59  *** guest234234 has quit IRC
568 2015-11-16T23:27:34  *** zarathustra has joined #bitcoin-core-dev
569 2015-11-16T23:27:34  *** zarathustra has joined #bitcoin-core-dev
570 2015-11-16T23:48:35  *** dcousens has joined #bitcoin-core-dev
571 2015-11-16T23:52:19  *** skyraider has quit IRC
572 2015-11-16T23:53:14  <dcousens> sipa: so, :S
573 2015-11-16T23:53:29  <dcousens> I have about 10mib garbage at the end of my leading .dat atm
574 2015-11-16T23:53:43  <dcousens> and one of my .dat files in the middle has 23mb garbage at the end
575 2015-11-16T23:54:04  <dcousens> other than that, all blocks accounted for
576 2015-11-16T23:54:28  <dcousens> bitcoind's rescan went without a hitch