1 2016-08-27T00:00:08  *** kadoban has quit IRC
  2 2016-08-27T00:05:34  *** spudowiar has quit IRC
  3 2016-08-27T00:13:40  *** achow101 has quit IRC
  4 2016-08-27T00:14:31  *** tom3 has joined #bitcoin-core-dev
  5 2016-08-27T00:22:16  *** Alopex has quit IRC
  6 2016-08-27T00:23:22  *** Alopex has joined #bitcoin-core-dev
  7 2016-08-27T00:35:21  *** Alopex has quit IRC
  8 2016-08-27T00:36:26  *** Alopex has joined #bitcoin-core-dev
  9 2016-08-27T00:42:12  *** Chris_Stewart_5 has quit IRC
 10 2016-08-27T00:42:30  *** Kexkey has joined #bitcoin-core-dev
 11 2016-08-27T00:44:42  <GitHub4> [bitcoin] nomnombtc opened pull request #8608: Install manpages via make install, also add some autogenerated manpages (master...man_automake2) https://github.com/bitcoin/bitcoin/pull/8608
 12 2016-08-27T00:46:22  *** fengling has joined #bitcoin-core-dev
 13 2016-08-27T00:51:06  *** fengling has quit IRC
 14 2016-08-27T00:52:53  *** Samdney has left #bitcoin-core-dev
 15 2016-08-27T01:05:46  *** achow101 has joined #bitcoin-core-dev
 16 2016-08-27T01:25:53  *** Ylbam has quit IRC
 17 2016-08-27T01:27:05  *** mappum has joined #bitcoin-core-dev
 18 2016-08-27T01:35:07  *** eragmus has joined #bitcoin-core-dev
 19 2016-08-27T01:35:26  *** Alopex has quit IRC
 20 2016-08-27T01:36:31  *** Alopex has joined #bitcoin-core-dev
 21 2016-08-27T01:39:13  *** btcdrak has joined #bitcoin-core-dev
 22 2016-08-27T01:47:48  *** fengling has joined #bitcoin-core-dev
 23 2016-08-27T01:49:44  <btcdrak> is there a way to sign each commit during an interactive rebase?
 24 2016-08-27T01:51:04  *** achow101 has quit IRC
 25 2016-08-27T01:52:26  *** fengling has quit IRC
 26 2016-08-27T01:59:52  *** tom3 has quit IRC
 27 2016-08-27T02:00:21  *** tom3 has joined #bitcoin-core-dev
 28 2016-08-27T02:00:26  *** Alopex has quit IRC
 29 2016-08-27T02:01:31  *** Alopex has joined #bitcoin-core-dev
 30 2016-08-27T02:18:01  *** Alopex has quit IRC
 31 2016-08-27T02:18:43  *** achow101 has joined #bitcoin-core-dev
 32 2016-08-27T02:19:06  *** Alopex has joined #bitcoin-core-dev
 33 2016-08-27T02:35:03  *** Giszmo has quit IRC
 34 2016-08-27T02:35:22  * luke-jr peers at rebroad.
 35 2016-08-27T02:47:22  *** tom3 has quit IRC
 36 2016-08-27T02:48:49  *** fengling has joined #bitcoin-core-dev
 37 2016-08-27T02:53:46  *** fengling has quit IRC
 38 2016-08-27T02:56:16  *** Alopex has quit IRC
 39 2016-08-27T02:57:21  *** Alopex has joined #bitcoin-core-dev
 40 2016-08-27T03:07:33  *** tom3 has joined #bitcoin-core-dev
 41 2016-08-27T03:08:12  *** Chris_Stewart_5 has joined #bitcoin-core-dev
 42 2016-08-27T03:17:52  *** droark has joined #bitcoin-core-dev
 43 2016-08-27T03:19:31  *** Kexkey has quit IRC
 44 2016-08-27T03:26:44  *** achow101 has quit IRC
 45 2016-08-27T03:28:18  *** kadoban has joined #bitcoin-core-dev
 46 2016-08-27T03:42:54  <roasbeef> btcdrak: signoff-rebase = "!GIT_SEQUENCE_EDITOR='sed -i -re s/^pick/e/' sh -c 'git rebase -i $1 && while git rebase --continue; do git commit --amend -S --no-edit; done' -"
 47 2016-08-27T03:43:06  <roasbeef> btcdrak: stuff that into an alias in your .gitconfig
 48 2016-08-27T03:48:28  *** AtashiCon has quit IRC
 49 2016-08-27T03:49:48  *** AtashiCon has joined #bitcoin-core-dev
 50 2016-08-27T03:50:19  *** fengling has joined #bitcoin-core-dev
 51 2016-08-27T03:53:26  *** Alopex has quit IRC
 52 2016-08-27T03:54:31  *** Alopex has joined #bitcoin-core-dev
 53 2016-08-27T03:54:46  *** fengling has quit IRC
 54 2016-08-27T04:02:50  *** cryptapus_afk is now known as cryptapus
 55 2016-08-27T04:04:06  *** Alopex has quit IRC
 56 2016-08-27T04:05:11  *** Alopex has joined #bitcoin-core-dev
 57 2016-08-27T04:32:14  *** fengling has joined #bitcoin-core-dev
 58 2016-08-27T04:35:07  *** Alopex has quit IRC
 59 2016-08-27T04:36:12  *** Alopex has joined #bitcoin-core-dev
 60 2016-08-27T04:46:24  *** tom3 has quit IRC
 61 2016-08-27T04:54:27  *** Alopex has quit IRC
 62 2016-08-27T04:55:32  *** Alopex has joined #bitcoin-core-dev
 63 2016-08-27T04:57:19  *** btcdrak has quit IRC
 64 2016-08-27T04:59:02  *** cryptapus is now known as cryptapus_afk
 65 2016-08-27T05:15:28  <luke-jr> hm, I would have thought verifying blocks was a read-only operation, but it seems to be writing about as much as it's reading?
 66 2016-08-27T05:15:54  <luke-jr> more actually
 67 2016-08-27T05:21:33  <gmaxwell> should be much more,
 68 2016-08-27T05:21:39  <gmaxwell> if it's reading your dbcache is too small.
 69 2016-08-27T05:43:27  <luke-jr> what's it writing?
 70 2016-08-27T05:44:33  <gmaxwell> the chainstate.
 71 2016-08-27T05:51:09  <wumpus> right, it's not the verification that causes writing, but *accepting* the blocks and updating the state
 72 2016-08-27T05:51:29  <wumpus> rejecting blocks should be a read-only operations
 73 2016-08-27T05:51:46  <wumpus> as well as verifying transactions outside blocks
 74 2016-08-27T05:55:35  <wumpus> gmaxwell: re: BU removing outward connection limit, sigh, it was to be expected somehow, you can't stop anti-social people by just telling them not to do a certain behavior, we'll need explicit anti-DoS measures for that
 75 2016-08-27T05:56:29  <wumpus> such a system could block spy-nodes connecting to everyone in one go
 76 2016-08-27T05:57:05  <wumpus> that there is no damn reason to connect to connect to more nodes also won't stop anyone
 77 2016-08-27T05:57:36  <wumpus> 'look mom we're real anarchists, we can misbehave on an open P2P network!'
 78 2016-08-27T06:02:23  <midnightmagic> any such mechanism if successful would at least in the short term eradicate blockchain.info's spying, as well as the various chainalysis mechs.
 79 2016-08-27T06:02:26  <wumpus> their name, their whole raison d'etre, is 'remove limits', so they removed another limit. It doesn't matter whether it had a purpose, they've remved a few lines of very limiting-feeling codes, and now they feel happy
 80 2016-08-27T06:04:30  <wumpus> pre-emtively filling all connection slots would also prevent blockchain.info's spying :-)
 81 2016-08-27T06:05:01  <midnightmagic> :-)
 82 2016-08-27T06:05:08  *** kadoban has quit IRC
 83 2016-08-27T06:07:20  <wumpus> if this escalates and there is no solution, in the longer run, it will turn the P2P network to a ghetto, and I'd expect it to transition it to more of a F2F network w/ authenticated connections
 84 2016-08-27T06:08:02  <wumpus> but also a few more mass-connectors likely won't break the netwerk, it just depends on the scale
 85 2016-08-27T06:12:16  <midnightmagic> Bitcoin Classic -- DDoS'ing the Bitcoin network since 2016.
 86 2016-08-27T06:12:35  <midnightmagic> I wonder if someone could get gavin to scold them for that
 87 2016-08-27T06:12:46  <midnightmagic> or. Unlimited. Same diff.
 88 2016-08-27T06:14:07  <wumpus> it reminds me a bit of seeders versus leechers discussion for bittorrent, with the difference is that 'defecting' there is advantageous to the leecher, here it's just about being a jerk just because
 89 2016-08-27T06:15:53  <wumpus> though connecting to lots of nodes can get you blocks fractially faster, which is useful for miners, you can accomplish the same thing by listening and advertising
 90 2016-08-27T06:26:46  <wumpus> you can set a virtually unlimited number of incoming connection slots, then by having a node with a high uptime, it will drift up in the DNS seeder rankings, which means its address will be dealt out more often, resulting in more connections to other nodes. It's a slower process but the social thing to do.
 91 2016-08-27T06:34:33  <luke-jr> shouldn't those blocks (and the chainstate) already be correct? O.o
 92 2016-08-27T06:35:49  <wumpus> but blocks are a delta to the chainstate
 93 2016-08-27T06:36:18  <wumpus> if you accept a block, by definition, you have to update the chainstate. And undo files are produced, too, a reverse-delta just in case.
 94 2016-08-27T06:36:35  <luke-jr> so it's actually rewinding and re-accepting them? somehow I thought it just verified the current state was sane
 95 2016-08-27T06:37:03  <gmaxwell> wumpus: what you say is true, but its much harder to deal with abusive behavior when you also have many more 'honest' people also being abusive out of ignorance. (in part, because the motivational structure is different; e.g. we can discourage spy nodes by reducing information leaks, moving more users into tor, etc.)
 96 2016-08-27T06:37:28  <wumpus> luke-jr: no, just applying the block results in writes to the utxo state, to mark outputs as spent, add new outputs. it rewinds only on reorg
 97 2016-08-27T06:37:38  <gmaxwell> (But ignorance, "I'm gonna help the network out by connecting to EVERYTHING!" ... is harder to resolve by rational measures)
 98 2016-08-27T06:38:06  <luke-jr> wumpus: I'm talking about the verifying blocks at startup.. checkblocks/checklevel
 99 2016-08-27T06:38:23  <wumpus> luke-jr: ohhh! that wasn't clear to me.
100 2016-08-27T06:38:32  <wumpus> luke-jr: no, that should be read-only
101 2016-08-27T06:38:32  <luke-jr> sorry
102 2016-08-27T06:38:52  <luke-jr> hmm
103 2016-08-27T06:39:34  <wumpus> gmaxwell: sure, at least ignorance can be improved by trying harder to inform people about things
104 2016-08-27T06:42:34  <wumpus> people may assume 'moaaarrrr outgoing connections is better' unless it's explained , .e.g in user interfaces, blog posts, that it's bad for yourself as well as others, with alternatives how to get more connections. Sure, some people will ignore it, or be jerks, but hopefully a miniority and most will heed.
105 2016-08-27T06:43:39  <wumpus> luke-jr: the rewinding with checklevel 3 completely happens in memory, if it is writing anything to disk that'd be very wrong
106 2016-08-27T06:44:01  <wumpus> (there could be a bug of course....)
107 2016-08-27T06:56:15  *** btcdrak has joined #bitcoin-core-dev
108 2016-08-27T06:57:36  <luke-jr> having trouble reproducing now. flushed Linux's disk caches though, and even with just reading, it's going super-slow :|
109 2016-08-27T06:59:58  <luke-jr> (as in 1% every 2 minutes or so, ETA 3 hours at this rate)
110 2016-08-27T07:05:29  * luke-jr ponders if iotop would report writing if it was swapping other processes to disk to do caching for us
111 2016-08-27T07:09:22  <wumpus> yes it reports swapping as writing, but wouldn't account swapping of *other* processes to disk to bitcoind
112 2016-08-27T07:09:46  <wumpus> IIRC there's a kswapd that gets all the blame for swapping
113 2016-08-27T07:09:49  <luke-jr> hmm
114 2016-08-27T07:10:03  <luke-jr> it did pick up speed and finished in 15 mins (just now) fwiw
115 2016-08-27T07:10:18  <luke-jr> didn't see any writing this time either
116 2016-08-27T07:11:28  <wumpus> phew
117 2016-08-27T07:12:38  <luke-jr> but I guess I did manage to reproduce https://www.reddit.com/r/Bitcoin/comments/4zrxs1/qtcore_client_taking_ages_to_start/ after all
118 2016-08-27T07:12:46  <luke-jr> just a matter of dropping caches
119 2016-08-27T07:12:47  *** OxADADA_ has joined #bitcoin-core-dev
120 2016-08-27T07:12:59  <luke-jr> gmaxwell: can you confirm on your slow laptop? echo 3 > /proc/sys/vm/drop_caches
121 2016-08-27T07:14:13  <wumpus> usually if the client takes ages to start there's a backlog of blocks to verify
122 2016-08-27T07:14:50  *** asoltys_ has joined #bitcoin-core-dev
123 2016-08-27T07:14:59  *** Pasha has joined #bitcoin-core-dev
124 2016-08-27T07:15:08  <sipa> question: how many times has the initial verification at startup actually caught corruption
125 2016-08-27T07:15:48  <sipa> nobody knows, of course
126 2016-08-27T07:15:49  <wumpus> I don't know - but I think it would be just as effective to just check a few blocks
127 2016-08-27T07:15:52  <luke-jr> sipa: not sure if it's still a problem, but every time when we had those powerfail-corrupts-db problem? (unless that was caught by something else?)
128 2016-08-27T07:16:06  <wumpus> the latest blocks are the most likely to be corrupt and below that it drops off
129 2016-08-27T07:16:10  <sipa> luke-jr: those result in a leveldb checksum error
130 2016-08-27T07:16:15  * luke-jr didn't realise the slowish startup time because he had checkblocks=4 checklevel=6 in his bitcoin.conf
131 2016-08-27T07:16:31  <sipa> there are only 4 checklevels :)
132 2016-08-27T07:16:41  <wumpus> luke-jr: same here - I think the default checkblocks should be much lower
133 2016-08-27T07:16:42  <luke-jr> well, if there's ever more added, I'm ready! :P
134 2016-08-27T07:16:52  *** owowo has quit IRC
135 2016-08-27T07:16:52  *** ratoder has quit IRC
136 2016-08-27T07:16:53  *** dgenr8 has quit IRC
137 2016-08-27T07:16:53  *** OxADADA has quit IRC
138 2016-08-27T07:16:53  *** droark has quit IRC
139 2016-08-27T07:16:54  *** nobits has quit IRC
140 2016-08-27T07:16:54  *** gluytium has quit IRC
141 2016-08-27T07:16:54  *** Cory has quit IRC
142 2016-08-27T07:16:54  *** asoltys has quit IRC
143 2016-08-27T07:16:57  <wumpus> a as-thorough-as-possible check on just a few blocks
144 2016-08-27T07:16:58  <sipa> and 4 blocks is not very much
145 2016-08-27T07:17:07  <wumpus> well, take 10 then
146 2016-08-27T07:17:14  *** nobits has joined #bitcoin-core-dev
147 2016-08-27T07:17:14  *** owowo has joined #bitcoin-core-dev
148 2016-08-27T07:17:16  <luke-jr> sipa: my PC is on UPS ;)
149 2016-08-27T07:17:24  <sipa> wumpus: jonasschnelli has a patch to switch to txcount based limiting
150 2016-08-27T07:18:06  <luke-jr> hmm, that's an interesting idea.
151 2016-08-27T07:18:07  <wumpus> that's good, but I think the effective default check depth should also be lowered, don't know if it does that
152 2016-08-27T07:18:10  *** dgenr8 has joined #bitcoin-core-dev
153 2016-08-27T07:18:46  <wumpus> or maybe write a flag on 'clean' shutdown, and do a reduced check in that case?
154 2016-08-27T07:18:52  <luke-jr> set it for the equivalent checkblocks back when it was introduced? :p
155 2016-08-27T07:18:54  *** gluytium has joined #bitcoin-core-dev
156 2016-08-27T07:19:17  <sipa> wumpus: it sets the default to 100000 txn
157 2016-08-27T07:19:34  <sipa> i guess it can be lower even
158 2016-08-27T07:19:36  <wumpus> sipa: ok
159 2016-08-27T07:19:51  <sipa> but it also insists on at least 6 blocks
160 2016-08-27T07:19:55  <gmaxwell> we need to do something about the startup check.
161 2016-08-27T07:20:23  <gmaxwell> unfortunately just checking a couple blocks is not much of a test, but anything more takes too long for most people.
162 2016-08-27T07:20:33  <sipa> wumpus: interesting idea
163 2016-08-27T07:20:39  <luke-jr> I wonder how difficult it would be to background it
164 2016-08-27T07:20:44  <gmaxwell> wumpus: neat idea!
165 2016-08-27T07:21:53  *** Pasha is now known as Cory
166 2016-08-27T07:22:10  <gmaxwell> when we were having windows corruption, what was needed to reliably detect that?
167 2016-08-27T07:22:56  <wumpus> just opening the leveldb IIRC
168 2016-08-27T07:22:58  <btcdrak> roasbeef: thanks!!
169 2016-08-27T07:23:19  <wumpus> or maybe the first access. No thorough checking was necessary
170 2016-08-27T07:23:22  * sipa suggests: 10000 txn (which corresponds to ~6 blocks currently)
171 2016-08-27T07:24:03  <wumpus> sounds good to me
172 2016-08-27T07:25:10  <wumpus> it's very helpful that leveldb has its own corruption detection here
173 2016-08-27T07:25:22  <gmaxwell> why bother with the txn count?
174 2016-08-27T07:25:25  <gmaxwell> just set it to 6 blocks?
175 2016-08-27T07:25:30  *** Guyver2 has joined #bitcoin-core-dev
176 2016-08-27T07:25:39  <wumpus> because that auto-adapts
177 2016-08-27T07:25:55  <gmaxwell> so? other than segwit the txn count of blocks won't change much.
178 2016-08-27T07:25:56  <wumpus> if blocks grow again, it won't become  slower
179 2016-08-27T07:26:19  <wumpus> heh yes that's a completely different discission
180 2016-08-27T07:26:22  <sipa> gmaxwell: if you are early in IBD you probably want more blockss
181 2016-08-27T07:26:37  <gmaxwell> sipa: okay, I'll but that.. but kind of a corner case.
182 2016-08-27T07:26:39  <wumpus> but the fact is that transaction could is a better measure of run time
183 2016-08-27T07:26:43  <wumpus> count*
184 2016-08-27T07:26:46  <gmaxwell> I was at the verge of suggesting this just do a single block.
185 2016-08-27T07:26:56  <wumpus> and amount of data checked
186 2016-08-27T07:27:01  *** Ylbam has joined #bitcoin-core-dev
187 2016-08-27T07:27:06  <sipa> heh, fine by me as well
188 2016-08-27T07:27:22  <gmaxwell> It's not yielding a high payoff in detected errors, the ones I know it detected (chainstate version corruption, need checking back to your last restat to reliably report)
189 2016-08-27T07:27:50  <wumpus> well the first priority is to make the insane wait time go away
190 2016-08-27T07:28:01  <wumpus> whether the new number becomes low or ultra-low is less important :)
191 2016-08-27T07:28:15  <gmaxwell> One or two block keeps the code live and working, I think this code is useful around refactorings and changes to the code.
192 2016-08-27T07:48:01  *** Alopex has quit IRC
193 2016-08-27T07:48:11  <btcdrak> i guess the reply to Roger's testnet response is no, BU runs in production and is a menace to the network
194 2016-08-27T07:48:44  <btcdrak> BU has no peer review and no chance in hell of being used for real
195 2016-08-27T07:49:07  *** Alopex has joined #bitcoin-core-dev
196 2016-08-27T08:06:04  <btcdrak> wumpus: rebased #7562
197 2016-08-27T08:07:38  <sipa> btcdrak: what was the problem with the tests?
198 2016-08-27T08:07:42  <btcdrak> gmaxwell: can you explain a bit more about your suggestion regarding my attempt at extracting the MacOSX sdk on linux?
199 2016-08-27T08:09:53  <btcdrak> sipa: changing the defaul tx version affected a couple of tests that were comparing hash values, or sorting txs by hash. obviously the hashes change when tx version is bumped. Those seem innocuous. There is one test however that I dont understand which I commented on. Not sure why it is affected by changing the version number.
200 2016-08-27T08:10:16  <btcdrak> I dont know if it's the tests' fault and innocuous, or revealing an issue.
201 2016-08-27T08:15:46  *** Megaf has joined #bitcoin-core-dev
202 2016-08-27T08:20:26  *** fengling has quit IRC
203 2016-08-27T08:22:24  *** Guyver2 has quit IRC
204 2016-08-27T08:30:54  <gmaxwell> btcdrak: so, first someone extracts it via OSX.  Then we take the extracted binary and find all the offsets for its data in the decompressed file.
205 2016-08-27T08:31:00  <gmaxwell> we can distribute the offset list.
206 2016-08-27T08:31:21  <gmaxwell> actually, a took like xdelta or another binary diff tool might just handle it.
207 2016-08-27T08:34:18  *** MarcoFalke has joined #bitcoin-core-dev
208 2016-08-27T08:40:31  <sipa> but the dmg file is compressed
209 2016-08-27T08:40:48  <sipa> can we decompress it?
210 2016-08-27T08:41:05  <sipa> without "installing", that is
211 2016-08-27T08:41:53  <gmaxwell> it's a 7z file according to drak
212 2016-08-27T08:43:10  <gmaxwell> so I was assuming that.
213 2016-08-27T08:43:11  <btcdrak> gmaxwell: one thing I am not sure about is if 7z is actually extracting it correctly. It seems to be, and the bug seems more like in the linux implementation of hfsplus; but it is feasible that the compression algo was tweaked and 7z is unaware
214 2016-08-27T08:43:24  <gmaxwell> unlikely, 7z has checksums.
215 2016-08-27T08:47:05  <sipa> so we'd xdelta the 7z-decompressed dmg with the resulting compressed .tar?
216 2016-08-27T08:47:21  <gmaxwell> uncompressed tar.
217 2016-08-27T08:47:44  <gmaxwell> oh what we want is a tgz.
218 2016-08-27T08:47:45  <gmaxwell> yes.
219 2016-08-27T08:47:52  <gmaxwell> so the file we normally get out of it.
220 2016-08-27T08:48:05  <gmaxwell> and if the resulting delta is small, we call it done.
221 2016-08-27T08:48:28  <luke-jr> btcdrak: what version of 7zip did you use?
222 2016-08-27T08:48:49  <sipa> btcdrak: where is hfsplus used?
223 2016-08-27T08:49:34  <btcdrak> luke-jr: version 9.20
224 2016-08-27T08:49:43  <luke-jr> hmm, it tells me: Error: Can not open file as archive
225 2016-08-27T08:51:36  *** fengling has joined #bitcoin-core-dev
226 2016-08-27T08:56:44  <btcdrak> sipa: https://www.irccloud.com/pastebin/S3zIzO3D/
227 2016-08-27T08:58:29  <luke-jr> oh, my DMG file is truncated
228 2016-08-27T08:59:04  <btcdrak> so basically the files we want are in Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk, but using the method above you get a bunch of empty files. On a Mac you run "tar -C /Volumes/Xcode/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/ -czf MacOSX10.11.sdk.tar.gz MacOSX10.11.sdk"
229 2016-08-27T08:59:35  <btcdrak> cfields_ documented it here https://github.com/bitcoin/bitcoin/blob/master/doc/README_osx.md
230 2016-08-27T09:00:08  <sipa> but we don't need all of the contents of that dmg, right?
231 2016-08-27T09:00:27  <luke-jr> not nearly
232 2016-08-27T09:00:39  <luke-jr> but it's copyrighted and non-redistributable :<
233 2016-08-27T09:00:39  <btcdrak> sipa right, we just need the files from Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/* which we turn into a tar.gz
234 2016-08-27T09:00:46  <sipa> i see
235 2016-08-27T09:00:55  <btcdrak> That gives us 47MB tar.gz :-p
236 2016-08-27T09:02:09  <btcdrak> we drop that in the gitian-builder/inputs/ folder when performing the Gitian rituals.
237 2016-08-27T09:02:46  <sipa> got it
238 2016-08-27T09:02:53  * luke-jr waits on copy over LAN.. Xcode_7.3.1.dmg   27% 1383MB   6.5MB/s   09:08 ETA
239 2016-08-27T09:03:03  <sipa> 14m here
240 2016-08-27T09:03:19  <luke-jr> you and your fast internet :x :p
241 2016-08-27T09:03:58  <sipa> luke-jr: this is over t-mobile roaming data
242 2016-08-27T09:04:26  *** fengling has quit IRC
243 2016-08-27T09:04:29  <luke-jr> sadly, T-Mobile data here seems to only be about 5 Mbps
244 2016-08-27T09:10:21  <gmaxwell> 2hr 49m remaining here.
245 2016-08-27T09:10:33  *** moli has quit IRC
246 2016-08-27T09:14:20  <btcdrak> RIP sipa's roaming data charges
247 2016-08-27T09:14:25  <sipa> btcdrak: it's free
248 2016-08-27T09:14:41  <sipa> it's effectively cheaper than my swiss wired or wireless internet
249 2016-08-27T09:15:04  <sipa> s/free/fixed price/
250 2016-08-27T09:16:30  <gmaxwell> btcdrak: a while back tmobile ceo put out some open letter about "data abusers" and for some reason it said, 'mining bitcoin' -- for a while we wondered if perhaps he was talking about blockstream. https://newsroom.t-mobile.com/news-and-blogs/stopping-network-abusers.htm
251 2016-08-27T09:16:47  <gmaxwell> apparently blockstream was using several TB a month on tmobile.
252 2016-08-27T09:18:26  <gmaxwell> doesn't everyone run a full node on their phone?
253 2016-08-27T09:18:44  <luke-jr> lol, xdelta is 45 MB
254 2016-08-27T09:18:59  <sipa> :(
255 2016-08-27T09:19:09  <sipa> luke-jr: how did you decompress the dmg?
256 2016-08-27T09:19:12  <luke-jr> 7z
257 2016-08-27T09:19:28  <gmaxwell> lol
258 2016-08-27T09:19:53  <gmaxwell> might need more options for it to actually find the match
259 2016-08-27T09:20:03  <luke-jr> maybe there's some tweaking to the tar we need to put it in a better order?
260 2016-08-27T09:20:10  *** fengling has joined #bitcoin-core-dev
261 2016-08-27T09:20:17  <gmaxwell> yes, that too.
262 2016-08-27T09:20:21  <luke-jr> oh wait
263 2016-08-27T09:20:27  <luke-jr> hmm, nm
264 2016-08-27T09:20:33  <luke-jr> the uncompressed tar is 306 MB FWIW
265 2016-08-27T09:20:38  <luke-jr> but xdelta's patch is compressed.
266 2016-08-27T09:20:47  <gmaxwell> were you using the compressed tar as the input?
267 2016-08-27T09:20:50  <luke-jr> no
268 2016-08-27T09:21:22  <sipa> hpmount, mount -t hfsplus, 7zr... all fail to open the file
269 2016-08-27T09:21:36  <gmaxwell> maybe we should just encrypt the damn tar with the whole dmg file as the key, and then call that sufficient for legal purposes. :P
270 2016-08-27T09:22:03  <luke-jr> sipa: need 7z, not 7zr
271 2016-08-27T09:22:07  <luke-jr> 7zr has formats removed
272 2016-08-27T09:22:31  <luke-jr> gmaxwell: hmm, I wonder if that'd be okay
273 2016-08-27T09:22:54  <luke-jr> I turned off xdelta's patch compression and it's 261 MB :/
274 2016-08-27T09:23:10  <gmaxwell> smaller than 300 so it found something.
275 2016-08-27T09:23:48  <gmaxwell> luke-jr: I think doing that would be defensible.
276 2016-08-27T09:23:59  <luke-jr> xdelta3: warning: input position 310378496 overflowed instruction buffer, needed 43091 (vs. 32768), consider changing -I
277 2016-08-27T09:24:13  <gmaxwell> consider changing -I
278 2016-08-27T09:24:20  * luke-jr sets -I to unlimited
279 2016-08-27T09:24:29  <luke-jr> not doing much better :<
280 2016-08-27T09:24:47  <luke-jr> -rw-r--r-- 1 luke-jr luke-jr 261M Aug 27 09:24 MacOSX.xdelta
281 2016-08-27T09:24:52  <gmaxwell> there are some other binary patching tools, xdelta was the first that came to mind.
282 2016-08-27T09:25:09  <sipa> xdelta and xdelta3 seem to be different things
283 2016-08-27T09:28:03  <gmaxwell> open-vcdiff
284 2016-08-27T09:28:23  <luke-jr> I'm writing something custom
285 2016-08-27T09:28:52  <luke-jr> hmm
286 2016-08-27T09:28:57  <luke-jr> I can't mmap this DMG :<
287 2016-08-27T09:29:07  <btcdrak> well surely the better solution would be to fix the hfsplus driver?
288 2016-08-27T09:29:09  <gmaxwell> luke-jr: upgrade to a darn 64 bit OS already luke!
289 2016-08-27T09:31:51  <luke-jr> oh, if I build with -static I can
290 2016-08-27T09:38:52  <sipa> i can't get it below 46 MB
291 2016-08-27T09:39:17  <sipa> that's with
292 2016-08-27T09:39:19  <sipa> xdelta3 encode -P $((2**30)) -I 0 -B 2147483648 -W 16777216
293 2016-08-27T09:39:31  <sipa> (the highest values for each of the limits that work)
294 2016-08-27T09:41:45  <sipa> it runs too fast, though
295 2016-08-27T09:42:12  <sipa> this is an 8 GB source file, and it only takes a few seconds
296 2016-08-27T09:42:26  <gmaxwell> "I demand my np-complete problems solving software be slow"
297 2016-08-27T09:42:44  <sipa> i demand it actually uses the whole input
298 2016-08-27T09:43:49  <gmaxwell> try open-vcdiff?
299 2016-08-27T09:44:28  *** Ginnarr has joined #bitcoin-core-dev
300 2016-08-27T09:45:29  <sipa> 42 MB
301 2016-08-27T09:45:43  <sipa> gmaxwell: seems that uses the same algorithm as xdelta3
302 2016-08-27T09:46:11  <gmaxwell> failure?
303 2016-08-27T09:47:27  <luke-jr> http://codepad.org/Yi5FEWML <-- any obvious bugs or possible optimizations?
304 2016-08-27T09:49:07  <btcdrak> also, I tried the dmg2img tool, but that doesnt appear to have been updated to handle the altered compression
305 2016-08-27T09:50:20  <GitHub180> [bitcoin] ajtowns closed pull request #8575: leveldb: generate lib independent of locale sort (master...leveldb-locale-reproducible) https://github.com/bitcoin/bitcoin/pull/8575
306 2016-08-27T09:50:21  <luke-jr> welp, failed to find the first 82 byte file :/
307 2016-08-27T09:50:26  <gmaxwell> luke-jr: well the needles may not be in the file in a contigious chunk.
308 2016-08-27T09:50:40  <luke-jr> gmaxwell: surely 82 bytes would be :<
309 2016-08-27T09:50:58  <gmaxwell> unless the file system stores the first n bytes of files in the inode table.
310 2016-08-27T09:51:03  <luke-jr> hmm
311 2016-08-27T09:51:04  <gmaxwell> (so that magic is fast)
312 2016-08-27T09:51:18  <gmaxwell> or just to prevent the indirection
313 2016-08-27T09:51:38  <luke-jr> would it make sense to target the 2nd 4096 bytes of the file then?
314 2016-08-27T09:53:55  <gmaxwell> to test your tool grab the second 4096 bytes of the haystack.
315 2016-08-27T09:54:02  <gmaxwell> but yes, you could try that.
316 2016-08-27T09:54:41  <luke-jr> http://codepad.org/VvfwXbqo isn't getting anything so far either
317 2016-08-27T09:54:59  <luke-jr> you mean of the needle, right?
318 2016-08-27T09:55:25  <gmaxwell> I mean grab a 4096 byte chunk out of the data you are searching in, just to verify your tool works.
319 2016-08-27T09:56:12  <luke-jr> oh
320 2016-08-27T09:56:15  <gmaxwell> dd if=5.hfs of=junk bs=4096 skip=1234 count=1
321 2016-08-27T09:56:18  <luke-jr> "In Mac OS X Snow Leopard 10.6, HFS+ compression was added. In open source and some other areas this is referred to as AppleFSCompression. Compressed data may be stored in either an extended attribute or the resource fork."
322 2016-08-27T09:56:25  <luke-jr> I bet this is what we're dealing with :|
323 2016-08-27T09:57:26  *** fengling has quit IRC
324 2016-08-27T09:57:26  <sipa> seems likely, indeed
325 2016-08-27T09:59:11  <gmaxwell> bleh
326 2016-08-27T10:00:19  <luke-jr> http://www.spinics.net/lists/linux-fsdevel/msg55545.html anyone want to implement? :/
327 2016-08-27T10:02:38  *** fengling has joined #bitcoin-core-dev
328 2016-08-27T10:10:17  <sipa> leveldb 1.19 was released
329 2016-08-27T10:11:27  <sipa>  52 files changed, 1976 insertions(+), 429 deletions(-)
330 2016-08-27T10:11:33  <sipa> (compared to 1.18)
331 2016-08-27T10:11:59  <gmaxwell> doesn't sound too bad to review.
332 2016-08-27T10:12:47  <sipa> they added a cache pruning
333 2016-08-27T10:13:27  <gmaxwell> doesn't sound especially relevant then.
334 2016-08-27T10:13:36  <sipa> it's been 2 years since their previous release
335 2016-08-27T10:13:45  <sipa> glad to see activity :)
336 2016-08-27T10:15:11  <sipa> ARM64 support for memory barriers seems relevant
337 2016-08-27T10:15:16  <sipa> cache size estimation
338 2016-08-27T10:15:26  <gmaxwell> oh the arm64 might fix performance on odroid c2
339 2016-08-27T10:28:06  <sipa> this is interesting but not included: https://github.com/google/leveldb/pull/309
340 2016-08-27T10:29:29  <gmaxwell> wumpus started benchmarking what that might look like.
341 2016-08-27T10:29:41  <gmaxwell> odroid c2 has a crc32c accelerator that is sutiable too.
342 2016-08-27T10:35:26  *** fengling has quit IRC
343 2016-08-27T10:35:52  *** harrymm has quit IRC
344 2016-08-27T10:47:08  *** moli has joined #bitcoin-core-dev
345 2016-08-27T10:51:10  *** harrymm has joined #bitcoin-core-dev
346 2016-08-27T11:03:35  <luke-jr> found implementation of HFS+ compression
347 2016-08-27T11:03:44  <sipa> where?
348 2016-08-27T11:03:49  <luke-jr> SleuthKit
349 2016-08-27T11:04:38  * luke-jr suggests we drop manpages from the tarball
350 2016-08-27T11:06:53  <luke-jr> xz -d <inodes.xz | while read inode filename; do filename="y/$filename"; p=$(dirname $filename); mkdir -vp $p; icat  ../5.hfs $inode >"$filename"; done
351 2016-08-27T11:07:54  * luke-jr tests result in gitian
352 2016-08-27T11:08:18  <luke-jr> the tarball is missing a bunch of stuff that looks like dummy files (but still 66 MB)
353 2016-08-27T11:08:35  <sipa> luke-jr: nice!
354 2016-08-27T11:09:24  <luke-jr> FWIW, I generated the inode list using mount and stat :p
355 2016-08-27T11:10:14  * luke-jr would be surprised if SleuthKit didn't have a way to get that info, but didn't see it
356 2016-08-27T11:12:38  <luke-jr> fls seems to be the tool
357 2016-08-27T11:15:49  <sipa> sleuthkit added support for HFS+ in version 3.1
358 2016-08-27T11:16:14  <luke-jr> I have 4.0.2
359 2016-08-27T11:16:57  <sipa> ubuntu 12.04 has sleuthkit 3.2.3
360 2016-08-27T11:17:08  <sipa> so i think we're good
361 2016-08-27T11:18:03  <luke-jr> oh.
362 2016-08-27T11:18:07  <luke-jr> except gitian doesn't like it :<
363 2016-08-27T11:18:27  <sipa> due to the missing manpages...?
364 2016-08-27T11:18:32  <luke-jr> hmm
365 2016-08-27T11:18:34  <luke-jr> I think symlinks
366 2016-08-27T11:18:38  <sipa> does it do a checksum check perhaps on the .tgz?
367 2016-08-27T11:18:40  <luke-jr> can't find the lib for c++
368 2016-08-27T11:18:50  <luke-jr> it can't, everyone's .tgz is different ;p
369 2016-08-27T11:19:40  <luke-jr> yeah, I think it's missing symlinks
370 2016-08-27T11:27:07  *** belcher has joined #bitcoin-core-dev
371 2016-08-27T11:29:41  <luke-jr> fls ../5.hfs -rpF 154283 | perl -nle 'm/^(r|l)\S*\s(\d+)\:\s*(.*$)/ && print "$1 $2 $3"' | while read type inode filename; do filename="MacOSX10.11.sdk/$filename"; mkdir -p "$(dirname "$filename")"; if [ "$type" = "l" ]; then ln -s $(icat ../5.hfs $inode) "$filename"; else icat ../5.hfs $inode >"$filename"; fi; done
372 2016-08-27T11:32:18  <luke-jr> tempting to figure out some kind of trick to download data on demand for this :P
373 2016-08-27T11:39:28  <luke-jr> hm, surprised there's no single-file curl FUSE fs
374 2016-08-27T11:48:28  *** Ginnarr has quit IRC
375 2016-08-27T11:54:45  *** MarcoFalke has left #bitcoin-core-dev
376 2016-08-27T11:54:48  <luke-jr> gitian build matches
377 2016-08-27T12:09:15  <sipa> \o/
378 2016-08-27T12:22:54  <btcdrak> wow
379 2016-08-27T12:26:19  <luke-jr> ?
380 2016-08-27T12:32:40  <CodeShark> I think btcdrak just considers perl code to be inherently aesthetically pleasing :p
381 2016-08-27T12:33:34  <luke-jr> lol
382 2016-08-27T12:33:40  <luke-jr> latest draft has no perl sadly: http://codepad.org/1wMp5vse
383 2016-08-27T12:33:48  <luke-jr> I'll wait until I'm actually awake to turn it into a PR
384 2016-08-27T12:34:19  <luke-jr> -rw-r--r-- 1 luke-jr luke-jr  21M Aug 27 12:33 MacOSX10.11.sdk.tar.gz
385 2016-08-27T12:35:29  <luke-jr> night
386 2016-08-27T12:38:19  *** achow101 has joined #bitcoin-core-dev
387 2016-08-27T12:42:04  <phantomcircuit> luke-jr, the client can take ages to start if you closed it after you downloaded a bunch of blocks but before you processed them
388 2016-08-27T12:42:14  <phantomcircuit> cause it processes all of them in AppInit2
389 2016-08-27T12:42:27  <phantomcircuit> (actually it doesn't anymore so this shouldn't be an issue in 0.13.x)
390 2016-08-27T12:43:24  <sipa> leveldb 1.19 should start up significantly faster
391 2016-08-27T12:44:18  <phantomcircuit> oh and leveldb reads it's journal which is potentially very slow also
392 2016-08-27T12:52:03  *** achow101 has quit IRC
393 2016-08-27T12:56:44  *** kadoban has joined #bitcoin-core-dev
394 2016-08-27T13:00:12  *** moli has quit IRC
395 2016-08-27T13:05:23  *** achow101 has joined #bitcoin-core-dev
396 2016-08-27T13:23:18  *** Chris_Stewart_5 has quit IRC
397 2016-08-27T13:24:25  <GitHub3> [bitcoin] sipa opened pull request #8610: Share unused mempool memory with coincache (master...sharemem) https://github.com/bitcoin/bitcoin/pull/8610
398 2016-08-27T13:26:27  *** Chris_Stewart_5 has joined #bitcoin-core-dev
399 2016-08-27T13:29:19  <GitHub62> [bitcoin] sipa opened pull request #8611: Reduce default number of blocks to check at startup (master...fastcheck) https://github.com/bitcoin/bitcoin/pull/8611
400 2016-08-27T13:32:32  *** moli has joined #bitcoin-core-dev
401 2016-08-27T13:34:56  <phantomcircuit> sipa, #8610 instead of doing that can you add a framework for limiting memory globally?
402 2016-08-27T13:35:25  <GitHub51> [bitcoin] sipa opened pull request #8612: Check for compatibility with download in FindNextBlocksToDownload (master...fixwitban) https://github.com/bitcoin/bitcoin/pull/8612
403 2016-08-27T13:35:31  <sipa> phantomcircuit: i don't have 5 years
404 2016-08-27T13:35:36  <phantomcircuit> it can probably even be as simple as a global memory limit goal and percentages for now
405 2016-08-27T13:36:06  <sipa> percentages isn't good enough
406 2016-08-27T13:36:35  <sipa> you need something that detects "oh, the mempool actually does not need its maximum usage... let's move some of its allocation elsewhere"
407 2016-08-27T13:36:48  <sipa> but can change that back when the mempool grows
408 2016-08-27T13:37:00  <phantomcircuit> percentages and an atomic of how much everything thinks it's using?
409 2016-08-27T13:37:09  <sipa> what does that solve?
410 2016-08-27T13:37:17  <phantomcircuit> "usage is below 90% of limit i can exceed my limit"
411 2016-08-27T13:37:37  <phantomcircuit> i guess you need callbacks to apply memory pressure then
412 2016-08-27T13:37:42  <sipa> yeah
413 2016-08-27T13:37:47  <phantomcircuit> but you need that for sharing memory at all
414 2016-08-27T13:38:04  <sipa> utxo set and mempool are things that are constantly checked anyway
415 2016-08-27T13:38:24  <sipa> what this PR implements is mempool < maxmempool && coincache + mempool < maxtotal
416 2016-08-27T13:38:44  <sipa> so arguably it already has something like a global limit (though it's just coincache + mempool)
417 2016-08-27T13:39:12  <sipa> the mempool has complicated semantics wrt computing relay fee based on its size
418 2016-08-27T13:39:26  <sipa> making its actual limit dynamic is harder, i think
419 2016-08-27T13:41:07  <sipa> also, i don't think it's needed to for example give your mempool 10 GB of memory even if you have 64 GB available... that'll just make your memory a sewer for spam that nobody else on the network accepts
420 2016-08-27T14:00:38  *** belcher has quit IRC
421 2016-08-27T14:01:46  <GitHub42> [bitcoin] sipa opened pull request #8613: [preview] LevelDB 1.19 (master...leveldb119) https://github.com/bitcoin/bitcoin/pull/8613
422 2016-08-27T15:00:42  *** spudowiar has joined #bitcoin-core-dev
423 2016-08-27T15:08:08  *** fengling has joined #bitcoin-core-dev
424 2016-08-27T15:24:26  *** fengling has quit IRC
425 2016-08-27T15:39:40  *** spudowiar has quit IRC
426 2016-08-27T16:03:18  *** spudowiar has joined #bitcoin-core-dev
427 2016-08-27T16:35:05  *** MarcoFalke has joined #bitcoin-core-dev
428 2016-08-27T16:39:49  *** harrymm has quit IRC
429 2016-08-27T16:53:01  *** harrymm has joined #bitcoin-core-dev
430 2016-08-27T17:02:35  *** Megaf has quit IRC
431 2016-08-27T17:08:00  *** Megaf has joined #bitcoin-core-dev
432 2016-08-27T17:10:09  *** achow101 has quit IRC
433 2016-08-27T17:22:34  *** Megaf has quit IRC
434 2016-08-27T17:25:53  *** Samdney has joined #bitcoin-core-dev
435 2016-08-27T17:36:35  *** JackH has quit IRC
436 2016-08-27T17:42:52  *** kadoban has quit IRC
437 2016-08-27T17:48:14  *** achow101 has joined #bitcoin-core-dev
438 2016-08-27T18:19:15  *** Lysanders has joined #bitcoin-core-dev
439 2016-08-27T18:57:52  *** Guyver2 has joined #bitcoin-core-dev
440 2016-08-27T19:11:02  *** spudowiar has quit IRC
441 2016-08-27T19:15:42  *** JackH has joined #bitcoin-core-dev
442 2016-08-27T19:19:51  *** JackH has quit IRC
443 2016-08-27T19:22:47  *** justanotheruser has quit IRC
444 2016-08-27T19:23:56  *** justanotheruser has joined #bitcoin-core-dev
445 2016-08-27T19:27:44  *** justanotheruser has quit IRC
446 2016-08-27T19:28:10  *** justanotheruser has joined #bitcoin-core-dev
447 2016-08-27T19:45:07  *** dgenr8 has quit IRC
448 2016-08-27T19:45:32  *** justanotheruser has quit IRC
449 2016-08-27T19:53:36  <gmaxwell> sipa: I think we should backport feeler connections. Beyond the security/robustness improvement, they'll help reduce the harm of network density loss w/ segwit.
450 2016-08-27T19:54:55  *** laurentmt has joined #bitcoin-core-dev
451 2016-08-27T20:02:54  *** timothy has quit IRC
452 2016-08-27T20:07:27  *** justanotheruser has joined #bitcoin-core-dev
453 2016-08-27T20:12:10  *** laurentmt has quit IRC
454 2016-08-27T20:18:46  <GitHub126> [bitcoin] luke-jr opened pull request #8617: Include instructions to extract Mac OS X SDK on Linux using 7zip and SleuthKit (master...gitian_osx_extractor) https://github.com/bitcoin/bitcoin/pull/8617
455 2016-08-27T20:20:27  <luke-jr> sipa: so have you confirmed already that LevelDB 1.19 is reasonably certain to not have forking changes? or are you assuming we'll do collectively that before merging/releasing?
456 2016-08-27T20:22:44  <sipa> luke-jr: i'm reasonably certain, yes, but i encourage review
457 2016-08-27T20:23:12  <sipa> i'm just pring it to bitcoin already to make testing easier
458 2016-08-27T20:53:37  *** kadoban has joined #bitcoin-core-dev
459 2016-08-27T21:14:36  *** achow101 has quit IRC
460 2016-08-27T21:43:20  *** spudowiar has joined #bitcoin-core-dev
461 2016-08-27T21:59:07  *** Giszmo has joined #bitcoin-core-dev
462 2016-08-27T22:22:36  *** Guyver2 has quit IRC
463 2016-08-27T22:26:21  *** Alopex has quit IRC
464 2016-08-27T22:27:26  *** Lysanders has quit IRC
465 2016-08-27T22:27:27  *** Alopex has joined #bitcoin-core-dev
466 2016-08-27T22:32:02  *** belcher has joined #bitcoin-core-dev
467 2016-08-27T22:47:16  *** Yogh has quit IRC
468 2016-08-27T22:47:42  *** MarcoFalke has quit IRC
469 2016-08-27T22:49:23  *** Yogh has joined #bitcoin-core-dev
470 2016-08-27T22:50:07  *** Alopex has quit IRC
471 2016-08-27T22:51:12  *** Alopex has joined #bitcoin-core-dev
472 2016-08-27T22:51:24  *** harrymm has quit IRC
473 2016-08-27T23:14:22  *** nibor has quit IRC
474 2016-08-27T23:15:09  *** nibor has joined #bitcoin-core-dev
475 2016-08-27T23:16:02  *** Alopex has quit IRC
476 2016-08-27T23:17:07  *** Alopex has joined #bitcoin-core-dev
477 2016-08-27T23:30:51  *** belcher has quit IRC
478 2016-08-27T23:43:45  *** cryptapus_afk is now known as cryptapus