1 2015-11-17T00:03:34  <dcousens> also, unrelated, but the total parse time (I realised that, before due to lazy evaluation, it was skipping ahead), when the result is piped to /dev/null, its parsing at ~210MiB/s, otherwise its blocked by output to about 110MiB/s
  2 2015-11-17T00:04:26  <dcousens> (meaning, about 7 minutes for a full 50G parse w/ ~50G output)
  3 2015-11-17T00:05:43  *** guest234234 has joined #bitcoin-core-dev
  4 2015-11-17T00:16:01  <dcousens> sipa: I also found a blog post detailing what I'm running into, tl;dr all the garbage is just a long run of 0's
  5 2015-11-17T00:16:58  <sipa> dcousens: that's preallocation
  6 2015-11-17T00:17:30  <dcousens> sipa: is it meant to be written though?
  7 2015-11-17T00:17:36  <sipa> dcousens: we preallocate the block files when first creating them, and then overwrite pieces of it with blocks; at the end, when leaving the file, we truncate
  8 2015-11-17T00:18:16  <dcousens> sipa: well, considering my bitcoind is closed, and the file still has a huge run of 0's at the end
  9 2015-11-17T00:18:16  <sipa> but things can go wrong i guess (for example, if the first write after restart is to a new file, you never 'leave' the old file)
 10 2015-11-17T00:18:26  <dcousens> I'm guessing the truncate isn't 100% solid
 11 2015-11-17T00:18:28  <sipa> dcousens: the last file will always have zeroes at the end
 12 2015-11-17T00:18:34  <sipa> as it isn't full yet
 13 2015-11-17T00:18:44  <dcousens> ok, by leaving the file you meant moving onto the next
 14 2015-11-17T00:18:50  <sipa> right, indeed
 15 2015-11-17T00:20:31  <gmaxwell> we prefill with zeros to prevent fragmenting the crap out of non-fragmentation resistant file systems with our incremental writes.
 16 2015-11-17T00:20:47  <dcousens> gmaxwell: gotcha
 17 2015-11-17T00:21:01  <dcousens> though, it probably shouldn't be in the middle of a .dat though
 18 2015-11-17T00:25:08  <dcousens> a 10MiB gap infact, then 252 more blocks after
 19 2015-11-17T00:27:09  <gmaxwell> dcousens: are these old files? We previously had bugs that might cause behavior like that, but I think they're all fixed.
 20 2015-11-17T00:27:15  <GitHub175> [bitcoin] ptschip opened pull request #7034: Fixed Windows secp256k1 compilation issue #7018 (master...secp256k1_compile) https://github.com/bitcoin/bitcoin/pull/7034
 21 2015-11-17T00:27:32  <dcousens> gmaxwell: its a fresh IBD with a just-compiled 4 days ago bitcoin/bitcoin master
 22 2015-11-17T00:28:00  <dcousens> at most the build would be within 2 weeks
 23 2015-11-17T00:28:49  <gmaxwell> okay, that sounds like a bug then.
 24 2015-11-17T00:29:05  <dcousens> Happy to donate time to debug it, but, not sure where to start :S
 25 2015-11-17T00:29:30  *** Anduck has quit IRC
 26 2015-11-17T00:30:46  *** Anduck has joined #bitcoin-core-dev
 27 2015-11-17T00:35:23  <GitHub69> [bitcoin] gmaxwell pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/e54ebbf60097...0a547d2d5501
 28 2015-11-17T00:35:23  <GitHub69> bitcoin/master 4d29032 Eric Lombrozo: Fixed integer comparison warning.
 29 2015-11-17T00:35:24  <GitHub69> bitcoin/master 0a547d2 Gregory Maxwell: Merge pull request #7023...
 30 2015-11-17T00:35:33  <GitHub106> [bitcoin] gmaxwell closed pull request #7023: Fixed integer comparison warning. (master...signed_int_comparison_fix) https://github.com/bitcoin/bitcoin/pull/7023
 31 2015-11-17T00:39:57  <GitHub66> [bitcoin] gmaxwell pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/0a547d2d5501...972bf9c52987
 32 2015-11-17T00:39:57  <GitHub66> bitcoin/master f6d9d5e Jonas Schnelli: add (max)uploadtarget infos to getnettotals RPC help
 33 2015-11-17T00:39:58  <GitHub66> bitcoin/master 972bf9c Gregory Maxwell: Merge pull request #6999...
 34 2015-11-17T00:40:02  <GitHub149> [bitcoin] gmaxwell closed pull request #6999: add (max)uploadtarget infos to getnettotals RPC help (master...2015/11/maxuploadtarget_rpc_help) https://github.com/bitcoin/bitcoin/pull/6999
 35 2015-11-17T00:41:49  <GitHub25> [bitcoin] ptschip closed pull request #7034: Fixed Windows secp256k1 compilation issue #7018 (master...secp256k1_compile) https://github.com/bitcoin/bitcoin/pull/7034
 36 2015-11-17T00:48:42  *** Guest23423 has joined #bitcoin-core-dev
 37 2015-11-17T00:48:59  *** zooko has quit IRC
 38 2015-11-17T00:52:23  *** guest234234 has quit IRC
 39 2015-11-17T00:54:27  *** Ylbam has quit IRC
 40 2015-11-17T01:00:49  <GitHub18> [bitcoin] gmaxwell pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/972bf9c52987...87ee0e2dbc12
 41 2015-11-17T01:00:50  <GitHub18> bitcoin/master 598e494 Jorge Timón: Chainparams: Explicit CChainParams arg for main (pre miner):...
 42 2015-11-17T01:00:50  <GitHub18> bitcoin/master 6bc9e40 Jorge Timón: Chainparams: Explicit CChainParams arg for miner:...
 43 2015-11-17T01:00:51  <GitHub18> bitcoin/master 87ee0e2 Gregory Maxwell: Merge pull request #6986...
 44 2015-11-17T01:00:54  <GitHub68> [bitcoin] gmaxwell closed pull request #6986: Globals: Don't call Params() from miner.cpp (master...global-chainparams-miner) https://github.com/bitcoin/bitcoin/pull/6986
 45 2015-11-17T01:13:22  <GitHub122> [bitcoin] dcousens opened pull request #7035: torcontrol: only output disconnect if -debug=tor (master...torlog) https://github.com/bitcoin/bitcoin/pull/7035
 46 2015-11-17T01:15:15  *** Anduck has quit IRC
 47 2015-11-17T01:15:33  *** Anduck has joined #bitcoin-core-dev
 48 2015-11-17T01:15:59  *** zooko has joined #bitcoin-core-dev
 49 2015-11-17T01:16:03  <gmaxwell> wumpus: so valgrind is telling me the event_new TorController::disconnected_cb is creating a leak on clean shutdown with stop.  But I've confirmed the destructor that calls event_del is correctly firing.  Is there some extra libevent magic needed to cause this to get freed?
 50 2015-11-17T01:16:33  <dcousens> gmaxwell: wumpus isn't here? :S
 51 2015-11-17T01:16:55  <sipa> it seems he got truncated into wump
 52 2015-11-17T01:17:27  <gmaxwell> I wondered why my client completed 'wump' :) (but apparently I didn't wonder that much)
 53 2015-11-17T01:37:44  *** Guest23423 has quit IRC
 54 2015-11-17T01:49:57  <morcos> gmaxwell: This is not a complaint but just a question, because I do think we should merge more of Jorge's refactoring changes, but we planning on trying to find a time to do a bunch of them so that we can consolidate the merge conflicts?
 55 2015-11-17T01:53:23  <gmaxwell> Feel free to complain in any case; I was mindful of it creating some conflict-- but it didn't move any code and I thought refactors that created structural changes were the big concern there.  I looked at existing open PRs (including yours) and concluded that the disruption from this would be small. (and if I felt that I could have merged yours first now and asked Jorge to rebase, I would have d
 56 2015-11-17T01:53:29  <gmaxwell> one that).
 57 2015-11-17T01:55:04  <morcos> yeah like i said, i'm not complaining, rebasing #7020 took a second, and #6898 is going to need some more work depending on #7020 and #7008 anyway.  But I just wanted to be aware if we were about to have a slew of these, since we're also trying to get a bunch of stuff in before the freeze.
 58 2015-11-17T01:55:15  <gmaxwell> morcos: if you go and fix your pull req, please provide feedback by screaming my name on IRC in proportion to what a pain in the ass iti is. :)
 59 2015-11-17T01:56:10  <gmaxwell> Ah, no I wasn't planning on pulling most of the other ways. (there was another very minor change from jtimon that I just asked him to rebase wrt the one I just merged, that I think won't break anything)
 60 2015-11-17T01:56:44  <morcos> speaking of, #6898 , the CreateNewBlock pull, it is quite a risky change, although with a big benefit.  What else do you think we should do to get comfortable with it.
 61 2015-11-17T01:57:23  <morcos> For instance, I called it out in the initial PR comment, but I didn't realize until reading your comments on IRC this weekend that not stopping to add txs at some minimum feerate is actually a pretty big negative change
 62 2015-11-17T01:58:22  <morcos> I've since fixed that...  But it worries me to change the code that miners everywhere will be relying on without extensive testing.   What should that be?
 63 2015-11-17T02:03:52  <gmaxwell> A couple things: we can probably add some tests that hand it simulated mempools and check some additional invarients, like that the selection is feerate monotone.  We can deploy it early 'mining' (doesn't have to start actually mining, just call createnewblock and sanity check the output) on the main network. We can probably get some miners with non-trival hashpower to deploy it post merge durin
 64 2015-11-17T02:03:58  <gmaxwell> g the 0.12 testing cycle; allowing us to catch an extreme stupidity before release.
 65 2015-11-17T02:04:37  *** belcher has quit IRC
 66 2015-11-17T02:04:41  <gmaxwell> It's also bad news that we have so many speedups in 0.12. That means miners will likely upgrade very quickly.
 67 2015-11-17T02:04:54  <sipa> morcos: would it be possible to add the new getblocktemplate as a new RPC (getblocktemplatebeta or so) ?
 68 2015-11-17T02:05:03  <sipa> and leaving the old one as-is for now
 69 2015-11-17T02:05:04  <gmaxwell> Otherwise we would have some buffering from "well, it'll take a while for everyone to deploy it."
 70 2015-11-17T02:05:59  <gmaxwell> hm. if we do that I'd want to do something to identify the blocks created with it... but I don't know that we can. :(
 71 2015-11-17T02:06:57  <sipa> it probably makes little difference
 72 2015-11-17T02:07:08  *** zooko has quit IRC
 73 2015-11-17T02:07:31  <gmaxwell> well I just mean that if we see miners making pathalogically stupid blocks it would be nice to be able to tell if the new code is to blame. :)
 74 2015-11-17T02:08:44  <sipa> gmaxwell: i mean that miners will likely use it anyway, even if it's called beta (although that does mean changing some software...)
 75 2015-11-17T02:09:24  <morcos> sipa: i'm sure thats easy enough to do.  Maybe that makes sense in that if there is a problem its easy to revert back without having to stop using 0.12
 76 2015-11-17T02:09:31  <dcousens> a rescan shouldn't modify the .dat files, should it?
 77 2015-11-17T02:09:59  <gmaxwell> sipa: that could also be a flag: oldcreatenewblock=0
 78 2015-11-17T02:10:26  <sipa> dcousens: no
 79 2015-11-17T02:10:28  <gmaxwell> Another reason to keep the old one around is if the miners are carrying patches to the old one, they can upgrade to 0.12 and keep them in place... OTOH, more for us to test.
 80 2015-11-17T02:10:59  <morcos> I'd tested previously that it generated the exact same blocks, modulo the comments I put at the top of the pull.  But I did forget one more.  I'll go update that.  I stop trying to fill the last little spot in the block early, but if I increase how much it'll search for that , they were the same
 81 2015-11-17T02:12:00  <morcos> I think I like this idea though.  If its a command line option then all we need is my duplicated CreateNewBlock code , and getblocktemplate checks the argument to see which CreateNewBlock it should call?
 82 2015-11-17T02:12:04  <phantomcircuit> gmaxwell, the coinbase sig flags returned by getblocktemplate could be modified
 83 2015-11-17T02:12:12  <phantomcircuit> im not sure if any miners actually use that though
 84 2015-11-17T02:12:54  <morcos> Hmm..  kind of annoying for unit tests and rpc tests though....
 85 2015-11-17T02:13:43  <phantomcircuit> when is feature freeze for 0.12 ?
 86 2015-11-17T02:13:43  <sipa> morcos: or CreateNewBlock itself dispatches based on the command-line flag?
 87 2015-11-17T02:13:58  <gmaxwell> phantomcircuit: lots of people ignore the coinbaseflags coming out of gbt. :(
 88 2015-11-17T02:14:36  *** guest234234 has joined #bitcoin-core-dev
 89 2015-11-17T02:15:30  <gmaxwell> morcos: it could just be a commandline flag, doubt we want rpc callers picking it on the fly.
 90 2015-11-17T02:15:48  <morcos> One other comment I'll make about 6898 is that its clearly a WIP in the larger sense too.  If we can rip out priority for 0.13, it'll clean up the code a lot, and also we might rework it yet again to do some intelligent CPFP
 91 2015-11-17T02:16:19  <morcos> so we should think a bit about how we might wan tto make future major changes to the code as well?
 92 2015-11-17T02:17:28  <morcos> gmaxwell: yes thats what i meant (checks the command line argument) i agree you should not be able to toggle back and forth
 93 2015-11-17T02:17:29  <gmaxwell> Well I do kind of like having the ability to swap in new algorithims with runtime selection.
 94 2015-11-17T02:17:54  <gmaxwell> I don't know how realistic that is generally, since I think different algorithims will require different indexes for efficient implementation.
 95 2015-11-17T02:18:36  <morcos> yes and keeping two algorithms properly tested is basically impossible
 96 2015-11-17T02:20:41  <gmaxwell> well right now with the current one there is a montone property I think it will obey, in that it will always select the highest feerate transactions, so anything omitted must be equal to or lower than the bottom feerate. So testing on real and simulated loads will at least tell us if its behaving crazily (like failing to mine a particular transaction causing it to clog the mempool)
 97 2015-11-17T02:20:48  <gmaxwell> But fancier algorithms will lose that simple property.
 98 2015-11-17T02:20:55  *** guest234234 has quit IRC
 99 2015-11-17T02:34:24  <morcos> gmaxwell: you're assuming wiht a priority space of 0?  Also that's not exactly correct, in that a high fee rate child could only have become elgible to be included when there was no longer enough space for it.
100 2015-11-17T02:35:32  <gmaxwell> Yes, that was assuming priority space of 0.
101 2015-11-17T02:35:35  <morcos> But I was suggesting we hack up the existing code to change the tie breaker and require that the blocks are identical for testing.  I think that's easy enough to achieve.  The problem is whether there is some degenerate situation where something bad happens
102 2015-11-17T02:35:43  <gmaxwell> ah, indeed, it still needs a topological sort.
103 2015-11-17T02:36:30  <phantomcircuit> morcos, so the answer i would like to give is "it doesn't matter they can run both and the block verification at the end will protect them from bugs"
104 2015-11-17T02:36:44  <phantomcircuit> but i know that lots of them dont have proper setups so that isn't really true
105 2015-11-17T02:36:45  <phantomcircuit> :(
106 2015-11-17T02:37:25  <gmaxwell> phantomcircuit: if you note above I mentioned the case of some high feerate txn not getting mined when it should.
107 2015-11-17T02:38:15  *** davec has quit IRC
108 2015-11-17T02:39:51  <morcos> So there are some very weird edge case differences I guess.  For instance, one of the versions has the sigop check before the block priority space check and the other has it after.  so if you have a transaction that fills the block priority space, you now switch to fee rate sorted, but if it fails the sigop test, then code A will have moved on to fee txs, but code B will still try for another priority tx
109 2015-11-17T02:40:42  <morcos> i can't remember which is which, and i haven't encountered that yet, nor do i expect to...   but at a certain point, mimicking the old behavior is not what we want
110 2015-11-17T02:41:02  <gmaxwell> no reason to think much of the old behavior is any good.
111 2015-11-17T02:43:44  *** zooko has joined #bitcoin-core-dev
112 2015-11-17T02:45:20  *** davec has joined #bitcoin-core-dev
113 2015-11-17T02:57:58  <phantomcircuit> gmaxwell, i assume that was from a block maxing out checksigs?
114 2015-11-17T03:32:06  <dcousens> gmaxwell: "One thought I had on this (before realizing there was already a pool"
115 2015-11-17T03:32:18  <dcousens> did you mean s/pool/pull/?
116 2015-11-17T03:36:07  *** davec has quit IRC
117 2015-11-17T03:38:57  *** davec has joined #bitcoin-core-dev
118 2015-11-17T03:51:12  *** jtimon has quit IRC
119 2015-11-17T03:58:35  <BlueMatt> gmaxwell: nope, actually, master still verifies with all the pgp sigs
120 2015-11-17T04:02:01  <BlueMatt> ofc jonasschnelli's key is completely unsigned, so......
121 2015-11-17T04:02:21  <gmaxwell> dcousens: yep
122 2015-11-17T04:02:47  <sipa> BlueMatt: i'll go see him when i'm in .ch (assuming he doesn't hate my face)
123 2015-11-17T04:26:13  <BlueMatt> sipa: sign wump's key while you're up north :p
124 2015-11-17T04:52:47  *** Squidicuz has joined #bitcoin-core-dev
125 2015-11-17T04:56:30  *** sipa_ has joined #bitcoin-core-dev
126 2015-11-17T04:56:41  *** cfields_ has joined #bitcoin-core-dev
127 2015-11-17T04:59:58  *** dgenr8 has quit IRC
128 2015-11-17T05:00:01  *** jonasschnelli has quit IRC
129 2015-11-17T05:00:03  *** zmanian_ has quit IRC
130 2015-11-17T05:00:03  *** Squidicc has quit IRC
131 2015-11-17T05:00:04  *** da2ce7 has quit IRC
132 2015-11-17T05:00:05  *** sipa has quit IRC
133 2015-11-17T05:00:06  *** andytoshi has quit IRC
134 2015-11-17T05:00:09  *** teward has quit IRC
135 2015-11-17T05:00:10  *** cfields has quit IRC
136 2015-11-17T05:00:10  *** phantomcircuit has quit IRC
137 2015-11-17T05:00:11  *** amiller_ has quit IRC
138 2015-11-17T05:00:12  *** BlueMatt has quit IRC
139 2015-11-17T05:00:22  *** dgenr8 has joined #bitcoin-core-dev
140 2015-11-17T05:00:29  *** Guest802 has joined #bitcoin-core-dev
141 2015-11-17T05:00:40  *** jonasschnelli has joined #bitcoin-core-dev
142 2015-11-17T05:00:57  *** phantomcircuit_ has joined #bitcoin-core-dev
143 2015-11-17T05:02:08  *** BlueMatt has joined #bitcoin-core-dev
144 2015-11-17T05:02:15  *** zooko has quit IRC
145 2015-11-17T05:03:29  *** zmanian_ has joined #bitcoin-core-dev
146 2015-11-17T05:03:54  *** andytoshi has joined #bitcoin-core-dev
147 2015-11-17T05:05:41  *** teward has joined #bitcoin-core-dev
148 2015-11-17T05:09:46  *** Guest802 has quit IRC
149 2015-11-17T05:09:46  *** Guest802 has joined #bitcoin-core-dev
150 2015-11-17T05:09:46  *** Guest802 is now known as amiller
151 2015-11-17T05:10:00  *** da2ce7 has joined #bitcoin-core-dev
152 2015-11-17T05:15:07  *** phantomcircuit_ is now known as phantomcircuit
153 2015-11-17T05:17:14  *** zmanian_ has quit IRC
154 2015-11-17T05:18:51  *** btcdrak has quit IRC
155 2015-11-17T05:20:09  *** CodeShark has quit IRC
156 2015-11-17T05:25:21  *** amiller has quit IRC
157 2015-11-17T05:25:21  *** zarathustra has quit IRC
158 2015-11-17T05:26:01  *** amiller_ has joined #bitcoin-core-dev
159 2015-11-17T05:29:07  *** CodeShark has joined #bitcoin-core-dev
160 2015-11-17T05:35:47  *** btcdrak has joined #bitcoin-core-dev
161 2015-11-17T05:37:11  *** zmanian_ has joined #bitcoin-core-dev
162 2015-11-17T06:07:38  *** zarathustra has joined #bitcoin-core-dev
163 2015-11-17T06:07:38  *** zarathustra has joined #bitcoin-core-dev
164 2015-11-17T06:07:49  *** pigeons has quit IRC
165 2015-11-17T06:08:37  *** pigeons has joined #bitcoin-core-dev
166 2015-11-17T06:09:00  *** pigeons is now known as Guest96832
167 2015-11-17T06:12:20  <dcousens> sipa_: so
168 2015-11-17T06:12:49  <dcousens> I'm still debugging this, and the garbage in my blk00289
169 2015-11-17T06:13:00  <dcousens> is random transactions, but its not at all an actual block
170 2015-11-17T06:13:17  <dcousens> like, its not prefixed by a header
171 2015-11-17T06:14:59  <dcousens> http://pastie.org/10561905
172 2015-11-17T06:15:26  <dcousens> Any ideas?
173 2015-11-17T06:16:08  <dcousens> that pastie is hard to decipher, but tldr I printed some bytes from the previous block to see if maybe the length was off, but, again, that block verifies so :/
174 2015-11-17T06:16:26  <dcousens> no idea, maybe theres a loose buffer overwrite somewhere?
175 2015-11-17T06:17:33  <gmaxwell> dcousens: while syncing, we write to the file and only flush the indexes periodically, if you shut off prematurely, it'll continue at the last index write, and overwrite whatever blocks were there. Might this be consistent with what you're seeing?
176 2015-11-17T06:18:46  <dcousens> gmaxwell: so you're thinking it had written a block, of say size N, it was then shutdown prematurely, and on re-open, it then wrote block of size M at that same index, where M < N?
177 2015-11-17T06:20:19  <gmaxwell> Yes.
178 2015-11-17T06:20:55  <gmaxwell> though unless it were the last block placed in the file (according to the block index), I don't know why it wouldn't have overwrote the lest.
179 2015-11-17T06:21:44  <midnightmagic> out-of-order block storage
180 2015-11-17T06:21:49  <midnightmagic> makes sense.
181 2015-11-17T06:22:27  <midnightmagic> sort of..
182 2015-11-17T06:29:43  <dcousens> gmaxwell: if I nuke this file
183 2015-11-17T06:29:45  <dcousens> and rescan
184 2015-11-17T06:29:50  <dcousens> Will it survive? :P
185 2015-11-17T06:30:07  <gmaxwell> it'll rescan up to that part and begin downloading the rest after it.
186 2015-11-17T06:30:24  <gmaxwell> (by rescan I mean reindex)
187 2015-11-17T06:30:27  <dcousens> uh yeah
188 2015-11-17T06:30:30  <dcousens> reindex*
189 2015-11-17T06:30:36  <dcousens> won't use any of the other .dat files?
190 2015-11-17T06:30:56  <dcousens> (as in, it'll abandon all blocks after it and re-download the remaining 15G?)
191 2015-11-17T06:31:33  <gmaxwell> it'll use all the earlier ones, and none of the later ones.
192 2015-11-17T06:32:00  <dcousens> ok
193 2015-11-17T06:32:07  <gmaxwell> It doesn't have a sophicated indirect membership management because ... that only makes sense if you're going to do crazy things like randomly delete a file in the middle. :)
194 2015-11-17T06:34:06  <dcousens> gmaxwell: :), well, at this point I can do this, or just ignore bytes until I reach a magic int then verify the block :/
195 2015-11-17T06:34:50  <dcousens> the 2.5 minute blockchain parse is too good to give up now haha
196 2015-11-17T06:35:10  *** Anduck has quit IRC
197 2015-11-17T06:35:23  *** Anduck has joined #bitcoin-core-dev
198 2015-11-17T06:36:10  <gmaxwell> dcousens: what are you doing this parse for?
199 2015-11-17T06:37:28  <dcousens> gmaxwell: custom index building for various dbs, its the fastest way to bootstrap, 5minutes to write it all out then just use the RPC to get anything else
200 2015-11-17T06:37:36  *** tulip has joined #bitcoin-core-dev
201 2015-11-17T06:38:02  <dcousens> much nicer then waiting 40+hrs for an RPC level sync, or hacking bitcoind to do it for me
202 2015-11-17T06:38:33  <gmaxwell> You benchmark that rpc scan since we got libevent?
203 2015-11-17T06:38:57  <dcousens> Its just become a bit of a timesink now because of this weird run of 000's and this issue.
204 2015-11-17T06:39:15  <dcousens> gmaxwell: actually I haven't
205 2015-11-17T06:40:09  <gmaxwell> I don't know which changes made it faster, but its much faster than it used to be; still going to be slow compared to scanning the files directly probably.
206 2015-11-17T06:40:24  <dcousens> gmaxwell: but maybe not slow enough to warrant this, point
207 2015-11-17T06:41:01  <gmaxwell> (well also, if its slow but fast enough to use; I'd like to optimize it further!)
208 2015-11-17T06:41:14  <dcousens> gmaxwell: the point was also so I had a tooling to be able to do things like check ancestor/depths super quick etc
209 2015-11-17T06:42:07  <dcousens> for service level bootstrapping, the new RPC speed may be enough,but analytically I'd still like a fast parse.
210 2015-11-17T06:43:30  <gmaxwell> just take care, there have been some things that lost funds because they were doing block scanning and didn't know some blocks were orphans... :-/
211 2015-11-17T06:44:02  <dcousens> gmaxwell: indeed, noticed the orphans
212 2015-11-17T06:44:34  <tulip> there's also traps for people trying to do block parsing as well, some transactions contain blocks.
213 2015-11-17T06:45:13  <dcousens> haha, like an 80byte OP_RETURN?
214 2015-11-17T06:45:52  <tulip> the minimum block size is more than 80 bytes, but something like that.
215 2015-11-17T06:48:40  <dcousens> gmaxwell: I'll give a play at the RPC sync speed
216 2015-11-17T06:48:43  <dcousens> let you know :)
217 2015-11-17T06:59:50  <gmaxwell> anyone around that understands the ZMQ interface? I'm trying to figure out how it hooks in to get a signal of a new block.  I see nothing in the codebase that references CZMQNotificationInterface::UpdatedBlockTip at all.
218 2015-11-17T07:00:22  <dcousens> gmaxwell: give me an hour and I might be sharing that question with you
219 2015-11-17T07:04:03  <aj> gmaxwell: src/validationinterface.cpp sets it up as a signal, and src/main.cpp triggers the signal?
220 2015-11-17T07:05:18  <aj> gmaxwell: or src/init.cpp has RegisterValidationInterface(pzmqNotificationInterface); to set it up in the first place?
221 2015-11-17T07:10:06  <gmaxwell> thanks!
222 2015-11-17T07:12:48  *** Ylbam has joined #bitcoin-core-dev
223 2015-11-17T07:13:55  <GitHub177> [bitcoin] gmaxwell opened pull request #7037: Move the blocknotify callback ahead of peer announcement. (master...notify_early) https://github.com/bitcoin/bitcoin/pull/7037
224 2015-11-17T07:15:51  *** wump is now known as wumpus
225 2015-11-17T07:29:08  <GitHub2> [bitcoin] gmaxwell pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/87ee0e2dbc12...eac53ec99201
226 2015-11-17T07:29:09  <GitHub2> bitcoin/master 141c44e MarcoFalke: [contrib] Update versionprefix to "bitcoin-core" in verify.sh
227 2015-11-17T07:29:09  <GitHub2> bitcoin/master a6d5a65 MarcoFalke: [trivial] contrib: Fix `echo`s in verify.sh
228 2015-11-17T07:29:10  <GitHub2> bitcoin/master eac53ec Gregory Maxwell: Merge pull request #7026...
229 2015-11-17T07:29:13  <GitHub74> [bitcoin] gmaxwell closed pull request #7026: [contrib] Update versionprefix to "bitcoin-core" in verify.sh (master...MarcoFalke-2015-verify) https://github.com/bitcoin/bitcoin/pull/7026
230 2015-11-17T07:33:26  <gmaxwell> Luke-Jr: https://github.com/bitcoin/bitcoin/pull/6851 wants rebase
231 2015-11-17T07:42:04  *** MarcoFalke has joined #bitcoin-core-dev
232 2015-11-17T07:51:47  *** jonasschnelli has quit IRC
233 2015-11-17T07:51:48  *** jonasschnelli has joined #bitcoin-core-dev
234 2015-11-17T08:07:47  <jonasschnelli> sipa: BlueMatt: i'll go see him when i'm in .ch (assuming he doesn't hate my face)  <--- Hah. The question is, if you don't hate my face after bothering your with tons of wallet/main.cpp questions. :)
235 2015-11-17T08:08:32  <jonasschnelli> sipa: I would be in Zurich next Tuesday 24., if you want to sign my key
236 2015-11-17T08:13:55  <btcdrak> I see wumpus has been untruncated thankfully :-P
237 2015-11-17T08:21:29  *** Guyver2 has joined #bitcoin-core-dev
238 2015-11-17T08:38:57  *** MarcoFalke has quit IRC
239 2015-11-17T09:02:42  *** guest234234 has joined #bitcoin-core-dev
240 2015-11-17T09:14:43  *** MarcoFalke has joined #bitcoin-core-dev
241 2015-11-17T09:15:53  *** guest234234 has quit IRC
242 2015-11-17T09:32:33  *** rubensayshi has joined #bitcoin-core-dev
243 2015-11-17T09:39:14  *** MarcoFalke has quit IRC
244 2015-11-17T09:40:32  *** GAit has joined #bitcoin-core-dev
245 2015-11-17T09:43:28  *** tulip has quit IRC
246 2015-11-17T09:56:59  *** d_t has quit IRC
247 2015-11-17T09:58:47  <CodeShark> The usual procedure for BIP proposals is to discuss on mailing list first. But given that at least a few important reviewers/critics either have unsubscribed to the ML entirely or else just ignore it, should we discuss it here or in bitcoin-dev?
248 2015-11-17T10:01:09  <wumpus> well definitely not here, this channel is just for bitcoin core related programming work. The proper answer to your question is stil 'the mailing list'. #bitcoin-dev is fine, but IRC discussion tends to be more ephermal than mailing lists
249 2015-11-17T10:01:42  *** tulip has joined #bitcoin-core-dev
250 2015-11-17T10:01:59  <wumpus> (or #bitcoin-wizards if it is moonmath related :-) )
251 2015-11-17T10:05:00  *** GAit has quit IRC
252 2015-11-17T10:05:15  *** GAit has joined #bitcoin-core-dev
253 2015-11-17T10:05:26  *** GAit has quit IRC
254 2015-11-17T10:10:14  <wumpus> btcdrak: tends to happen when I get disconnected from the IRC server, the client will automatically butcher my name when it is already used
255 2015-11-17T10:10:31  <btcdrak> wumpus: it have us all a giggle yesterday :)
256 2015-11-17T10:10:44  <btcdrak> I was going to suggest we all truncate in solidarity!
257 2015-11-17T10:10:49  <btcdrak> :)
258 2015-11-17T10:12:37  <wumpus> lol :)
259 2015-11-17T10:20:38  <wumpus> gmaxwell: re: valgrind tests, what version of libevent are you using?
260 2015-11-17T10:21:20  <wumpus> please use an up-to-date version of libevent, I know for fact a few (minor) memory leaks have been fixed over time
261 2015-11-17T10:23:33  <wumpus> (it's very possible that there is an actual leak caused by our code, but to avoid getting stuck in unnecessary rabbit holes I'd like to rule out issues that were already fixed years ago :-) )
262 2015-11-17T10:28:30  *** davec has quit IRC
263 2015-11-17T10:34:47  *** Anduck has quit IRC
264 2015-11-17T10:35:04  *** Anduck has joined #bitcoin-core-dev
265 2015-11-17T10:49:38  <btcdrak> who maintains the Ubuntu ppa packages for bitcoin? there are issues being reported https://www.reddit.com/r/Bitcoin/comments/3t04fp/psa_please_upgrade_to_bitcoin_core_0112_to/cx33vpw?context=3
266 2015-11-17T10:51:38  *** GAit has joined #bitcoin-core-dev
267 2015-11-17T10:52:15  *** tulip has quit IRC
268 2015-11-17T11:00:06  *** randy-waterhouse has quit IRC
269 2015-11-17T11:04:11  <sipa_> btcdrak: BlueMatt
270 2015-11-17T11:04:17  *** sipa_ is now known as sipa
271 2015-11-17T11:04:41  <btcdrak> hi
272 2015-11-17T11:05:33  *** davec has joined #bitcoin-core-dev
273 2015-11-17T11:08:43  <BlueMatt> btcdrak: huh? ubuntu 11.04???
274 2015-11-17T11:09:04  <BlueMatt> 11.04 has been fully deprecated since 2013?!
275 2015-11-17T11:12:12  <wumpus> right - 11.04 is not a LTS release
276 2015-11-17T11:12:25  <wumpus> they should upgrade to 12.04, at the least
277 2015-11-17T11:33:17  *** guest234234 has joined #bitcoin-core-dev
278 2015-11-17T11:42:40  *** GAit has quit IRC
279 2015-11-17T11:45:06  *** GAit has joined #bitcoin-core-dev
280 2015-11-17T11:46:16  *** rav3n_pl has joined #bitcoin-core-dev
281 2015-11-17T11:48:29  <rav3n_pl> hello. im looking to simple way to check for double spends of unconfirmed transaction. ive found bitcoinXT that return some info on "gettransaction" rpc, but it can be used only for in-wallet transactions. is there a way, to check it for non-wallet transactions?
282 2015-11-17T11:51:52  <rav3n_pl> i can parse mempool transactions, but trying to avoid it
283 2015-11-17T11:54:38  <phantomcircuit> rav3n_pl, that's impossible
284 2015-11-17T11:55:02  <rav3n_pl> :/ bad
285 2015-11-17T11:55:17  <phantomcircuit> rav3n_pl, also you're way offtopic
286 2015-11-17T11:55:21  <rav3n_pl> :D
287 2015-11-17T11:55:28  <rav3n_pl> sorry
288 2015-11-17T11:55:45  <rav3n_pl> not found anythin closer in channel list
289 2015-11-17T11:55:51  <wumpus> #bitcoin please
290 2015-11-17T11:56:01  <rav3n_pl> okay, ty
291 2015-11-17T12:44:53  *** jtimon has joined #bitcoin-core-dev
292 2015-11-17T12:48:32  *** ParadoxSpiral has joined #bitcoin-core-dev
293 2015-11-17T12:49:52  *** rav3n_pl is now known as rav3n_pl_away
294 2015-11-17T12:52:39  *** ParadoxSpiral has quit IRC
295 2015-11-17T12:56:52  *** Guest96832 is now known as pigeons
296 2015-11-17T13:03:55  *** MarcoFalke has joined #bitcoin-core-dev
297 2015-11-17T13:09:58  *** GAit has quit IRC
298 2015-11-17T13:12:10  *** Anduck has quit IRC
299 2015-11-17T13:13:48  *** Anduck has joined #bitcoin-core-dev
300 2015-11-17T13:15:29  *** ParadoxSpiral has joined #bitcoin-core-dev
301 2015-11-17T13:17:58  <jtimon> what I was supposed to do with "undefined reference to `secp256k1_ecdsa_sign_recoverable'", git clean -f and make clean didn't do it...
302 2015-11-17T13:18:39  <phantomcircuit> jtimon, git clean -fdx
303 2015-11-17T13:19:37  <jtimon> phantomcircuit: thanks
304 2015-11-17T13:22:19  *** MarcoFalke has quit IRC
305 2015-11-17T13:31:48  *** GAit has joined #bitcoin-core-dev
306 2015-11-17T13:42:48  *** AtashiCon has quit IRC
307 2015-11-17T13:42:54  *** Arnavion3 has joined #bitcoin-core-dev
308 2015-11-17T13:42:58  *** Arnavion3 is now known as AtashiCon
309 2015-11-17T13:44:06  *** murr4y has quit IRC
310 2015-11-17T13:48:04  *** murr4y has joined #bitcoin-core-dev
311 2015-11-17T13:49:36  *** fanquake has joined #bitcoin-core-dev
312 2015-11-17T13:50:10  <fanquake> Do you still need to pass —with-gui=qt5 to configure to get a qt5 build, or do we change it to default to qt5 when available?
313 2015-11-17T13:59:22  <jtimon> andytoshi: from what we talked previously, I think you like #6907
314 2015-11-17T14:00:20  <wumpus> fanquake: no longer necessary to provide that
315 2015-11-17T14:00:34  <wumpus> if you have qt4 and qt5 development headers installed it will now default to qt5
316 2015-11-17T14:03:26  <fanquake> wumpus nice; thought I'd seen a pull to change it. Good that we've made that change.
317 2015-11-17T14:03:58  <jtimon> fanquake: wumpus: maybe something for the release notes?
318 2015-11-17T14:05:21  <sipa> jtimon: the releases use qt5 for a while already, this only applies to people building from scratch, i think
319 2015-11-17T14:08:11  *** GAit has quit IRC
320 2015-11-17T14:08:40  <wumpus> jtimon: there's already so many "notable changes" for 0.12 that I doubt this one will make the cut - it'll be in the long list of pulls/commits anyway
321 2015-11-17T14:08:46  <wumpus> sipa: right
322 2015-11-17T14:09:01  <jtimon> sipa: wumpus fair enough
323 2015-11-17T14:11:00  <fanquake> You use [skip-ci] in the body of a pull to skip travis right?
324 2015-11-17T14:12:18  <wumpus> yes, that should be possible
325 2015-11-17T14:12:26  <wumpus> (I'm not sure what the exact incantation is)
326 2015-11-17T14:15:17  <GitHub159> [bitcoin] fanquake opened pull request #7040: [doc] Update OS X build notes for new qt5 configure (master...patch-4) https://github.com/bitcoin/bitcoin/pull/7040
327 2015-11-17T14:23:25  <GitHub45> [bitcoin] fanquake opened pull request #7041: [doc][trivial] Update OS X install instructions (master...patch-5) https://github.com/bitcoin/bitcoin/pull/7041
328 2015-11-17T15:03:32  *** guest234234 has quit IRC
329 2015-11-17T15:09:29  <GitHub64> [bitcoin] fanquake opened pull request #7042: [doc] Update Debian watch & control (master...update-debian) https://github.com/bitcoin/bitcoin/pull/7042
330 2015-11-17T15:13:15  *** zooko has joined #bitcoin-core-dev
331 2015-11-17T15:13:41  *** Anduck has quit IRC
332 2015-11-17T15:30:36  *** treehug88 has joined #bitcoin-core-dev
333 2015-11-17T15:33:12  *** go1111111 has quit IRC
334 2015-11-17T15:36:24  *** MarcoFalke has joined #bitcoin-core-dev
335 2015-11-17T15:38:40  *** paveljanik has joined #bitcoin-core-dev
336 2015-11-17T15:38:40  *** paveljanik has joined #bitcoin-core-dev
337 2015-11-17T15:58:27  *** MarcoFalke has quit IRC
338 2015-11-17T16:00:51  <sdaftuar> sipa: wumpus: any thoughts on 7020?  morcos and i both have pulls that change the CTxMemPoolEntry constructor which causes a lot of work updating unit tests.
339 2015-11-17T16:01:21  <sdaftuar> if 7020 is likely to go in i can rebase #6915 on it though
340 2015-11-17T16:02:08  <sdaftuar> (it needs to be rebased anyway)
341 2015-11-17T16:06:25  *** MarcoFalke has joined #bitcoin-core-dev
342 2015-11-17T16:06:32  *** moli has quit IRC
343 2015-11-17T16:07:22  *** moli has joined #bitcoin-core-dev
344 2015-11-17T16:15:23  *** zooko has quit IRC
345 2015-11-17T16:19:27  *** fanquake has quit IRC
346 2015-11-17T16:22:17  <Luke-Jr> gmaxwell: I can rebase it easily, but I wonder if I should try to get master to actually build first :/
347 2015-11-17T16:28:34  <Luke-Jr> looks like I had to manually re-configure. more build system issues, sigh
348 2015-11-17T16:42:23  *** rubensayshi has quit IRC
349 2015-11-17T16:42:24  <MarcoFalke> Trying to set up gitian in fedora. What should I do when ./bin/gbuild asks  for a password for ubuntu@localhost?
350 2015-11-17T16:42:28  <MarcoFalke> wumpus ?
351 2015-11-17T16:42:58  <sipa> MarcoFalke: try 'ubuntu'
352 2015-11-17T16:43:37  <wumpus> with lxc? you could add a sudoers rule that allows 'lxc-start' without password, there's an example in doc/gitian-building.md
353 2015-11-17T16:43:37  <MarcoFalke> nice
354 2015-11-17T16:44:41  <MarcoFalke> Why would I need to type the password 'ubuntu' five times? I tried it up to three times and assumed it failed because there was no response.
355 2015-11-17T16:44:54  <MarcoFalke> Anyway, why isn't it using the keys I created?
356 2015-11-17T16:45:10  <MarcoFalke> I am using kvm
357 2015-11-17T16:45:56  <wumpus> with kvm it shouldn't be asking passwords at all, kvm sandboxes can be run as unprivileged users, IIRC you only need a few root commands to make the VM in make-base-vm
358 2015-11-17T16:46:13  <wumpus> but not on gbuild, gbuild asking passwords is absolutely wrong
359 2015-11-17T16:46:35  <MarcoFalke> make-base-vm asked for my password on the fedora machine
360 2015-11-17T16:46:44  <wumpus> yes, that's normal
361 2015-11-17T16:47:06  <wumpus> if gbuild asks for a password *inside* the image there's certainly something wrong with it
362 2015-11-17T16:47:15  <wumpus> what vmbuilder version did you use?
363 2015-11-17T16:48:17  <MarcoFalke> 0.12.4
364 2015-11-17T16:49:00  *** d_t has joined #bitcoin-core-dev
365 2015-11-17T16:49:36  <wumpus> installed from source or from a package?
366 2015-11-17T16:50:10  <MarcoFalke> wget http://archive.ubuntu.com/ubuntu/pool/universe/v/vm-builder/vm-builder_0.12.4+bzr494.orig.tar.gz
367 2015-11-17T16:50:15  <MarcoFalke> Now it's asking for root@localhost password
368 2015-11-17T16:50:44  <wumpus> ok, shouldn't be anything wrong with that
369 2015-11-17T16:50:58  <wumpus> not sure what causes your issues then
370 2015-11-17T16:53:44  <wumpus> but my guess is that something went wrong while building the image
371 2015-11-17T16:55:35  <MarcoFalke> The var/id_dsa key is supposed to be used by the ubuntu@localhost user?
372 2015-11-17T16:58:25  <GitHub86> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/eac53ec99201...e8df8a5077df
373 2015-11-17T16:58:25  <GitHub86> bitcoin/master e587bc3 Alex Morcos: Implement helper class for CTxMemPoolEntry constructor...
374 2015-11-17T16:58:26  <GitHub86> bitcoin/master e8df8a5 Wladimir J. van der Laan: Merge pull request #7020...
375 2015-11-17T16:58:30  <GitHub93> [bitcoin] laanwj closed pull request #7020: Implement helper class for CTxMemPoolEntry constructor (master...EntryHelper) https://github.com/bitcoin/bitcoin/pull/7020
376 2015-11-17T17:00:06  <wumpus> yes, those keys are passed into vmbuilder
377 2015-11-17T17:00:28  <wumpus> I think they are supposed to be installed inside the VM for both the root and ubuntu user
378 2015-11-17T17:01:02  <wumpus> --ssh-key=var/id_dsa.pub --ssh-user-key=var/id_dsa.pub
379 2015-11-17T17:03:06  <morcos> thanks wumpus!
380 2015-11-17T17:11:03  <MarcoFalke> Trying with precise, now.
381 2015-11-17T17:22:12  *** MarcoFalke has quit IRC
382 2015-11-17T17:25:54  *** MarcoFalke has joined #bitcoin-core-dev
383 2015-11-17T17:51:55  *** btcdrak has quit IRC
384 2015-11-17T17:53:08  *** MarcoFalke has quit IRC
385 2015-11-17T17:53:40  *** CodeShark has quit IRC
386 2015-11-17T17:55:53  *** zmanian_ has quit IRC
387 2015-11-17T17:59:59  *** davec has quit IRC
388 2015-11-17T18:00:16  *** davec has joined #bitcoin-core-dev
389 2015-11-17T18:05:06  *** d4de has quit IRC
390 2015-11-17T18:06:47  *** zooko has joined #bitcoin-core-dev
391 2015-11-17T18:13:27  *** rav3n_pl_away is now known as rav3n_pl
392 2015-11-17T18:43:26  *** andytoshi has quit IRC
393 2015-11-17T18:51:06  <gmaxwell> wumpus: I'll repro on something very new (though this particular system isn't an old one); mostly making sure that it wasn't known and harmless.
394 2015-11-17T19:04:10  *** zooko has quit IRC
395 2015-11-17T19:11:23  *** zmanian_ has joined #bitcoin-core-dev
396 2015-11-17T19:15:31  *** btcdrak has joined #bitcoin-core-dev
397 2015-11-17T19:17:40  *** andytoshi has joined #bitcoin-core-dev
398 2015-11-17T19:39:27  *** trippysalmon has joined #bitcoin-core-dev
399 2015-11-17T19:40:31  *** CodeShark has joined #bitcoin-core-dev
400 2015-11-17T19:44:03  *** sdaftuar has quit IRC
401 2015-11-17T19:51:56  <GitHub126> [bitcoin] instagibbs opened pull request #7044: RPC: Added additional config option for multiple RPC users. (master...multrpc) https://github.com/bitcoin/bitcoin/pull/7044
402 2015-11-17T20:02:06  *** evoskuil has quit IRC
403 2015-11-17T20:21:03  *** trippysalmon has quit IRC
404 2015-11-17T20:38:46  *** evoskuil has joined #bitcoin-core-dev
405 2015-11-17T20:53:02  <GitHub121> [bitcoin] luke-jr opened pull request #7045: Bugfix: Use unique autostart filenames on Linux for testnet/regtest (master...linux_autostart_unique) https://github.com/bitcoin/bitcoin/pull/7045
406 2015-11-17T20:53:36  *** zooko has joined #bitcoin-core-dev
407 2015-11-17T21:02:31  *** moli has quit IRC
408 2015-11-17T21:02:46  *** evoskuil has quit IRC
409 2015-11-17T21:08:53  *** go1111111 has joined #bitcoin-core-dev
410 2015-11-17T21:17:37  *** paveljanik has quit IRC
411 2015-11-17T21:21:58  *** randy-waterhouse has joined #bitcoin-core-dev
412 2015-11-17T21:30:18  <Luke-Jr> I wonder if #6978 (qt -fPIC stuff) should be backported?
413 2015-11-17T21:32:15  *** SatoshisCat has joined #bitcoin-core-dev
414 2015-11-17T21:32:24  *** MarcoFalke has joined #bitcoin-core-dev
415 2015-11-17T21:37:30  *** tulip has joined #bitcoin-core-dev
416 2015-11-17T21:47:18  *** belcher has joined #bitcoin-core-dev
417 2015-11-17T21:51:18  <GitHub102> [bitcoin] pstratem opened pull request #7046: [WIP] Net: Ignore "tx" messages in blocks only mode. (master...2015-11-17-blocksonly) https://github.com/bitcoin/bitcoin/pull/7046
418 2015-11-17T21:58:26  *** d_t has quit IRC
419 2015-11-17T22:00:38  <phantomcircuit> gmaxwell, oops i just realized i messed up the whitelistalways stuff
420 2015-11-17T22:00:40  * phantomcircuit goes to fix
421 2015-11-17T22:01:04  <phantomcircuit> (it technically works but it drops the inv so the whitelisted peer has to actually push the tx blindly)
422 2015-11-17T22:01:47  *** treehug88 has quit IRC
423 2015-11-17T22:02:46  <phantomcircuit> sipa, i end up calculating bool fAcceptTxs, is that something you would see being in CNodeState? (currently it's constant during runtime)
424 2015-11-17T22:03:08  <phantomcircuit> (unless you can remove the whitelist flag at runtime, can you?)
425 2015-11-17T22:03:27  <sipa> no, whitelist won't change at runtime
426 2015-11-17T22:05:35  <phantomcircuit> BlueMatt, can i disable the mempool by limiting the size to 0?
427 2015-11-17T22:05:42  <phantomcircuit> morcos, same question^
428 2015-11-17T22:06:17  <BlueMatt> i think you have to set it to some big number
429 2015-11-17T22:06:20  <BlueMatt> but i dont recall now
430 2015-11-17T22:06:33  <phantomcircuit> although i guess if im going to be relaying whitelisted peers transactions i might as well also be keeping them in the mempool
431 2015-11-17T22:06:39  <phantomcircuit> gmaxwell, any opinion there?
432 2015-11-17T22:06:44  <sipa> BlueMatt: he means disabling the mempool, not disabling limiting :)
433 2015-11-17T22:06:46  <morcos> phantomcircuit: also don't recall, sorry
434 2015-11-17T22:06:52  <morcos> oh
435 2015-11-17T22:07:16  <phantomcircuit> yes
436 2015-11-17T22:08:29  <tulip> Leaving block file 375: CBlockFileInfo(blocks=0, size=0, heights=0...0, time=1970-01-01...1970-01-01)
437 2015-11-17T22:08:37  <morcos> yikes, it looks like you can't because of bad interaction with descendantlimit
438 2015-11-17T22:08:56  <phantomcircuit> so blocksonly=1 whitelistalwaysrelay=1 should the node be adding the transaction to it's mempool or just blindly passing it to every peer
439 2015-11-17T22:09:09  <sipa> tulip: oh, i think that's harmless
440 2015-11-17T22:09:30  <morcos> BlueMatt: do you think it should be completely prohibited from violating the desired ratio, or just warn
441 2015-11-17T22:10:15  <BlueMatt> phantomcircuit: I assume you'll break things if you do, but it should disable mempool, yes
442 2015-11-17T22:10:32  <BlueMatt> oh, yea, what morcos said
443 2015-11-17T22:10:45  <phantomcircuit> BlueMatt, im not sure what i'd break doing that though since it's already relaying whitelisted peers transactions even if they're rejected from the mempool entirely
444 2015-11-17T22:10:48  <BlueMatt> morcos: hmm? which ratio?
445 2015-11-17T22:11:12  <morcos> BlueMatt: if you look at 6557, i implemented that check differently, so the descendentsizelimit is automatically maxed at the right ratio of your mempool size unless you explicitly set it
446 2015-11-17T22:11:17  <BlueMatt> phantomcircuit: mostly "there be dragons" (of unspecified type)
447 2015-11-17T22:11:19  <tulip> sipa:  nothing seems to be on fire otherwise. is it caused by something broken in my local state, or a recent change in master?
448 2015-11-17T22:11:19  <morcos> that might be a better parameter interaction
449 2015-11-17T22:12:11  <BlueMatt> morcos: sorry, I've been behind on bitcoin core...people were starting to notice the relay network had been broken for a long time so i figured i needed to fix it
450 2015-11-17T22:12:41  <morcos> BlueMatt: you return error if the mempool is less than 40 * the descendant size limit.  i'm saying if the descendant size limit isn't set, you should just silently cap it at 1/40 of mempool.  and if it is explicitly set, let any two values go, evne if lmiting will be adversely affected
451 2015-11-17T22:13:01  <michagogo> 16:11:00 <fanquake> You use [skip-ci] in the body of a pull to skip travis right?
452 2015-11-17T22:13:09  <morcos> 6557 was just our version of the limiting pull, its closed now, i'm just pointing out a different way to do the parameter interaction
453 2015-11-17T22:13:17  <michagogo> I think it's [skip ci] or [ci skip], both work
454 2015-11-17T22:13:23  <michagogo> I don't know if - works
455 2015-11-17T22:13:24  <sipa> tulip: i believe it's due to a recent change in master
456 2015-11-17T22:13:45  <michagogo> And I think it's the commit message, though I'm not sure about that
457 2015-11-17T22:14:07  <phantomcircuit> BlueMatt, hmm well then i'll have to muck about with the "tx" ProcessMessage logic a lot more than i wanted to
458 2015-11-17T22:14:16  <phantomcircuit> would anybody be opposed to moving it to it's own function?
459 2015-11-17T22:14:29  <phantomcircuit> (indeed most of the logic in ProcessMessage would do well in it's own function)
460 2015-11-17T22:16:50  <morcos> phantomcircuit: ha ha, sdaftuar and i would LOVE that.  its the most annoying part of our historical data simulator right now
461 2015-11-17T22:18:02  *** pmienk_ has quit IRC
462 2015-11-17T22:18:28  <phantomcircuit> is anybody going to kill me if i submit a pr that moves each of the message handling routines into it's own function?
463 2015-11-17T22:18:35  <phantomcircuit> speak now or yell at me later
464 2015-11-17T22:19:47  <BlueMatt> morcos: hmm...I'm inclined to say leave it as is...I prefer the documented parameters to just refuse to run if it's gonna end up doing something dumb (people will shoot themselves in the foot if you let them
465 2015-11-17T22:19:55  <BlueMatt> phantomcircuit: please god do
466 2015-11-17T22:20:19  <BlueMatt> morcos: i wouldnt complain loudly if it was changed, but its not like the limit enforced now is crazy
467 2015-11-17T22:20:43  <phantomcircuit> bool static ProcessVersionMessage(CNode* pfrom, CDataStream& vRecv, int64_t nTimeReceived) EXCLUSIVE_LOCKS_REQUIRED(cs_main)
468 2015-11-17T22:20:46  <phantomcircuit> etc
469 2015-11-17T22:22:23  <phantomcircuit> hmm need strCommand too
470 2015-11-17T22:23:01  <tulip> sipa: thanks.
471 2015-11-17T22:24:20  * BlueMatt -> lunch
472 2015-11-17T22:24:29  <sipa> phantomcircuit: i've argued against that before, because it will conflict with pretty much every PR there is
473 2015-11-17T22:24:55  <sipa> phantomcircuit: though that was under the assumption that some decent modularization of msg handling would happen instead, and so far, it hasn't
474 2015-11-17T22:25:18  <phantomcircuit> sipa, it probably will, and it probably always will
475 2015-11-17T22:25:55  <sipa> yes, but moving things now, and then doing clean modularization later means it breaks everything twice :)
476 2015-11-17T22:25:58  <sipa> for the same benefit
477 2015-11-17T22:26:00  *** sdaftuar_ has joined #bitcoin-core-dev
478 2015-11-17T22:26:33  <phantomcircuit> sipa, it'll at least be easier to modularize once it's "cleaned" up a bit
479 2015-11-17T22:26:50  <sipa> hardly, i think
480 2015-11-17T22:27:54  <phantomcircuit> sipa, when you say modularization you're referring to having a ping module etc
481 2015-11-17T22:27:55  <phantomcircuit> right?
482 2015-11-17T22:29:03  <sipa> yes
483 2015-11-17T22:29:40  *** sdaftuar_ has quit IRC
484 2015-11-17T22:31:10  <phantomcircuit> sipa, i would be worried about such a system being too complicated
485 2015-11-17T22:31:24  <phantomcircuit> the basic protocol is really very simple
486 2015-11-17T22:31:45  <phantomcircuit> the only complexity is in scheduling block download really
487 2015-11-17T22:32:18  <phantomcircuit> (which i assume is part of your interest in doing that?)
488 2015-11-17T22:37:17  *** devpo has joined #bitcoin-core-dev
489 2015-11-17T22:37:37  <devpo> hi is anybdy here to help me a little
490 2015-11-17T22:37:38  <devpo> ?
491 2015-11-17T22:38:56  <devpo> :'(
492 2015-11-17T22:39:01  <btcdrak> devpo: what's up?
493 2015-11-17T22:41:18  <devpo> i need a php script
494 2015-11-17T22:41:31  <btcdrak> better move to #bitcoin
495 2015-11-17T22:41:45  <devpo> ok
496 2015-11-17T22:42:02  *** devpo has quit IRC
497 2015-11-17T22:45:32  <sipa> phantomcircuit: no, my interest is allowing the message handler to run in multiple threads
498 2015-11-17T22:46:25  <sipa> (and making things cleaner in the process)
499 2015-11-17T22:48:11  <phantomcircuit> sipa, assuming things are holding the correct locks i dont see why we cant do that now
500 2015-11-17T22:48:23  *** molly has joined #bitcoin-core-dev
501 2015-11-17T22:48:43  <sipa> phantomcircuit: the problem is that it isn't clear what those locks are
502 2015-11-17T22:49:13  <sipa> if all handler ran in their own module, with their own state, with their own encapsulated lock, it would be trivially correct
503 2015-11-17T22:50:13  <sipa> anyway, no big objection if you want to move code out of processblock into smaller functions, but not everything at once...
504 2015-11-17T22:50:40  <phantomcircuit> sipa, it would work today with very if any problems because of cs_vRecvMsg/cs_vSend
505 2015-11-17T22:50:57  <phantomcircuit> (assuming for a second that there's nothing in SendMessages looking at vRecvMsg
506 2015-11-17T22:51:00  <phantomcircuit> )
507 2015-11-17T22:51:12  <sipa> you can't do that now
508 2015-11-17T22:51:21  <sipa> all shared handler state is locked by cs_main
509 2015-11-17T22:51:33  <phantomcircuit> it doesn't get you much of anything though since virtually everything they're doing that's cpu intensive requires cs_main
510 2015-11-17T22:51:43  <sipa> yes
511 2015-11-17T22:52:13  <sipa> but many of those cs_main only protect handler state, and not consensus/block related structures
512 2015-11-17T22:52:25  <sipa> those need to be replaced by their own handler-spoecific locks
513 2015-11-17T22:53:19  <phantomcircuit> what would be an appropriate DoS level for a peer that's sending transactions despite fRelayTxs=false (keeping in mind fRelayTxs will be true for whitelisted peers)
514 2015-11-17T22:53:22  *** d_t has joined #bitcoin-core-dev
515 2015-11-17T22:54:03  <phantomcircuit> (and it's sending a protocol version high enough to support fRelayTxs in version)
516 2015-11-17T22:55:01  <gmaxwell> phantomcircuit: I think we shouldn't DOS now.
517 2015-11-17T22:55:31  <gmaxwell> not without studying more what it'll do. I know BTCD in the past wanted the protocol to define sending unsolicited transactions as misbehavior.
518 2015-11-17T22:56:54  <tulip> phantomcircuit: a least one piece of SPV wallet software pushes all of their transactions directly without inv'ing them.
519 2015-11-17T23:00:34  <gmaxwell> perhaps in the blocks only context is a fine way to introduce that behavior, but I think we shouldn't rush into it.
520 2015-11-17T23:00:50  <gmaxwell> We can try asking davec if he has any insight into observed behavior.
521 2015-11-17T23:01:20  <gmaxwell> phantomcircuit: One thing you could do is log when a non-whitelisted peer sends a tx message in blocksonly mode.
522 2015-11-17T23:11:27  <phantomcircuit> gmaxwell, is it important that GetMainSignals().Inventory(inv.hash); runs regardless of whether we call pfrom->AskFor(inv); or not?
523 2015-11-17T23:11:28  *** Guyver2 has quit IRC
524 2015-11-17T23:11:41  <phantomcircuit> (if so it needs to be moved)
525 2015-11-17T23:12:33  <gmaxwell> phantomcircuit: my belief was no, if we're not relaying we don't care what others have relayed to us.
526 2015-11-17T23:13:09  <gmaxwell> (and in particular, we don't want to know: just another way for a peer to use our memory)
527 2015-11-17T23:13:24  <phantomcircuit> the version in master actually does call GetMainSignals().Inventory(inv.hash); even if we dont ask anybody for the data
528 2015-11-17T23:13:36  <phantomcircuit> yeah that's what i thought
529 2015-11-17T23:27:38  <phantomcircuit> im afraid to ask but, GetBoolArg has it's own locks... right?
530 2015-11-17T23:27:54  <sipa> phantomcircuit: no, but mapArgs etc are immutable :)
531 2015-11-17T23:28:17  <sipa> (apart from initialization which happens single-threadedly)
532 2015-11-17T23:28:24  <phantomcircuit> sipa, so... practically safe but technically a threading violation
533 2015-11-17T23:28:33  <sipa> phantomcircuit: no, no threading violation
534 2015-11-17T23:28:39  <sipa> reading data that can't change is safe
535 2015-11-17T23:30:48  <gmaxwell> phantomcircuit: the memory model is that shared reading is okay, but as soon as there is a potential write (even of the same data) then there must be locks that exclude all other readers and writers.
536 2015-11-17T23:32:29  <gmaxwell> as the complier is allowed to do things like "oh I can see in this block I will, for sure, overwrite location X.. that means I have exclusive access.. so until then I can spill random registers into it."
537 2015-11-17T23:32:38  *** MarcoFalke has quit IRC
538 2015-11-17T23:37:40  *** ParadoxSpiral has quit IRC
539 2015-11-17T23:41:01  <phantomcircuit> gmaxwell, ah right
540 2015-11-17T23:45:20  *** zooko has quit IRC
541 2015-11-17T23:46:41  <phantomcircuit> gmaxwell, pr updated
542 2015-11-17T23:47:09  <phantomcircuit> only thing im not happy with is that it's not obvious why transactions dont end up in the orphan pool
543 2015-11-17T23:50:04  <gmaxwell> phantomcircuit: you should improve your commit messages some;  e.g. https://github.com/pstratem/bitcoin/commit/17e6157c1975a4f5e0afa97d632ca0310b227158  should explain that previously if we recieved an unsolicited TX message (which we shouldn't) we'd process it anyways, even in blocks only mode.
544 2015-11-17T23:50:21  *** SatoshisCat has quit IRC
545 2015-11-17T23:51:32  <davec> Regarding the DoS banning, in general I've never seen much point in allowing protocol level misbehavior.  It's TCP, so if the peer isn't following the protocol, they should be disconnected with prejudice.
546 2015-11-17T23:52:31  <phantomcircuit> gmaxwell, can do
547 2015-11-17T23:52:51  <phantomcircuit> er is there an easy way to edit an arbitrary commit message?
548 2015-11-17T23:53:13  <sipa> git commit --amend
549 2015-11-17T23:53:18  <davec> git commit --amend   for the top commit
550 2015-11-17T23:53:26  <davec> if it's further back, git rebase -i
551 2015-11-17T23:53:32  <davec> and use 'r'
552 2015-11-17T23:56:08  <phantomcircuit> oh rebase e
553 2015-11-17T23:56:11  <phantomcircuit> r*
554 2015-11-17T23:56:12  <phantomcircuit> that's nice
555 2015-11-17T23:56:32  <tulip> davec: in 0.8.x nodes were banned for their own internal misbehaviour which was distinctly non-optimal. it's good to be strict, but there's definitely some risk to it.
556 2015-11-17T23:57:53  <sipa> davec, tulip: i don't think the question is whether we should allow protocol misbehaviour (imho, we shouldn't), but what is considered misbehaviour :)