1 2015-10-27T00:05:06  <phantomcircuit> morcos, my huge sigcache is mostly fine
  2 2015-10-27T00:05:30  <morcos> phantomcircuit: i'm running with 1M now and its still noticeably slower (than infinite)
  3 2015-10-27T00:05:38  <phantomcircuit> changing the max arena 100% fixed the problem without rewritting a bunch of stuff too :)
  4 2015-10-27T00:06:05  <phantomcircuit> hmm it really shouldn't be
  5 2015-10-27T00:06:13  <morcos> well i'm chaning the other stuff anyway  :)
  6 2015-10-27T00:06:58  <morcos> well noticeably slower i mean just that...  say instead of taking 30us to verify a txin on average it takes 40us
  7 2015-10-27T00:07:19  <morcos> but without a cache at all of course its 10x higher
  8 2015-10-27T00:09:51  <morcos> its the process of filling up your 300M mempool for the first time that makes it slow...  i guess it should improve once its full and the mempool gets a minfee
  9 2015-10-27T00:10:48  <morcos> scarily fast filling up to 300M from starting from scratch.  like 45mins
 10 2015-10-27T00:11:39  <phantomcircuit> morcos, without the limiting
 11 2015-10-27T00:11:42  <phantomcircuit> im at about 2GB now
 12 2015-10-27T00:11:50  <phantomcircuit> 3.168GB
 13 2015-10-27T00:12:30  <morcos> i'm still disturbed by how this same backlog of txs is so readily rerelayed
 14 2015-10-27T00:13:19  <gmaxwell> esp since the network behavior is supposted to not relay transactions once they've already been relayed. :(
 15 2015-10-27T00:13:39  <gmaxwell> unfortunately there seem to be some abusive nodes out there that behave strangely, constantly resending transactions and such.
 16 2015-10-27T00:14:40  <morcos> yeah i noticed that, i was wondering if they were nodes who considerd these wallet tx's or if was specifically designed to retransmit them
 17 2015-10-27T00:15:44  <morcos> ok modulo the final TestBlockValidity check, i brought the time of CreateNewBlock from a bit over 1 second to under 10 ms.
 18 2015-10-27T00:15:44  <gmaxwell> the ones I've caught doing it with an instrumented node appeared to regurgitate their whole mempool every block.
 19 2015-10-27T00:16:01  <morcos> unfortunately i wouldn't feel good about dumping that check.
 20 2015-10-27T00:16:22  <morcos> that also doesn't include adding any priority transactions.  i can add that back in, but i think it'll be slow
 21 2015-10-27T00:16:40  <morcos> gmaxwell: oh really?? thats bad
 22 2015-10-27T00:18:05  <midnightmagic> fwiw, gmaxwell recall you wanted me to check and see whether any of those nodes were resending me tx dupes: across the entire time I was monitoring, none of them did that I could detect while parsing the debug.log.
 23 2015-10-27T00:18:26  <midnightmagic> I don't think I actually got back to you about that.
 24 2015-10-27T00:18:38  <midnightmagic> gmaxwell: What's the periodicity of the resends?
 25 2015-10-27T00:19:35  <gmaxwell> midnightmagic: at least some were doing this after every block.
 26 2015-10-27T00:20:04  <gmaxwell> e.g. a bogus assumption that any transaction known would make it in a block, and if it didn't you must have just 'lost' it.
 27 2015-10-27T00:20:09  <gmaxwell> or the like.
 28 2015-10-27T00:20:33  <midnightmagic> I wonder if some services are attempting to "help" the network by propagating tx in order to guarantee reach for nodes that have rebooted or something.
 29 2015-10-27T00:29:04  <CodeShark> sending as in sending the actual transaction? or just an inv?
 30 2015-10-27T00:32:47  <phantomcircuit> CodeShark, same thing for this purpose
 31 2015-10-27T00:32:52  <phantomcircuit> zombie transactions
 32 2015-10-27T00:33:49  <CodeShark> for what purpose? I would think sending the mempool (as in the actual txs, not just the inv) is MUCH worse for bandwidth
 33 2015-10-27T00:34:09  <morcos> would anybody else find useful a feature which split the minimum fee for acceptance to your mempool and the min fee for relay?
 34 2015-10-27T00:34:37  <morcos> often during this testing, i either hack up my node to not relay, or i feel bad about the fact that i'm helping propogate these low fee txs all over again
 35 2015-10-27T00:34:43  <CodeShark> I would find really useful a feature that would allow you to get paid for running a validator/relay node :p
 36 2015-10-27T00:35:01  <morcos> you get paid in food for your soul
 37 2015-10-27T00:35:04  <CodeShark> lol
 38 2015-10-27T00:35:19  <sipa> CodeShark: i think that screws incentives completely
 39 2015-10-27T00:35:37  <CodeShark> sipa: howso?
 40 2015-10-27T00:35:48  <sipa> CodeShark: if you can get paid to run a node, you can equally get paid to run a compromised node
 41 2015-10-27T00:36:15  <CodeShark> presumably you'd have to actually perform the function sufficiently well
 42 2015-10-27T00:36:35  <sipa> you can't prove that you are
 43 2015-10-27T00:37:09  <CodeShark> well, relay might be more easily amenable to micropayments - validation is a little trickier
 44 2015-10-27T00:37:13  *** deepcore has quit IRC
 45 2015-10-27T00:38:37  <CodeShark> but I think ultimately doable...not sure how, but my gut tells me at the very least some efficient probabilistic checks are possible
 46 2015-10-27T00:39:07  <CodeShark> without having to go to SNARKs
 47 2015-10-27T00:39:18  <sipa> their validation is unobservable... it's a feature they provide to their users, not to you
 48 2015-10-27T00:42:42  <CodeShark> it's a feature they provide to the network
 49 2015-10-27T00:43:01  <sipa> no
 50 2015-10-27T00:43:07  <sipa> relay is what they provide to the network
 51 2015-10-27T00:43:16  <sipa> and you can't observe that they validate before relay
 52 2015-10-27T00:43:40  <sipa> 1) they may just be connected to honest nodes them selves, so they'll only relay valid blocks through that
 53 2015-10-27T00:44:23  <sipa> 2) past performance is not an indicator for future success. One awesome way to sybil attack the network is to spin up many totally honest nodes, and then suddenly turn evil once you notice a sufficient portion of the network depends on you.
 54 2015-10-27T00:45:14  <sipa> relay/ibd is a dumb service they offer to the network - which hopefully verifies what you tell them enough to not need to trust you
 55 2015-10-27T00:46:10  <sipa> the power of validation is only through knowing that the person running it has economic activity depending on it
 56 2015-10-27T00:48:16  <CodeShark> so I guess if the network is to subsidize it it would have to either require miners to be doing validation somehow...or to incentivize invalidation
 57 2015-10-27T00:50:16  <CodeShark> in any case, if we could just take care of incentives for relay for now I'd be happy :)
 58 2015-10-27T00:50:29  <CodeShark> that would go a long ways towards solving the mempool issues
 59 2015-10-27T00:57:30  <CodeShark> providing SPV proofs could be incentivized...and perhaps nodes that can provide SPV proofs would also have to perform full validation
 60 2015-10-27T01:00:16  <CodeShark> or it would be in their interest to, I suppose
 61 2015-10-27T01:01:10  *** fkhan has quit IRC
 62 2015-10-27T01:15:29  *** fkhan has joined #bitcoin-core-dev
 63 2015-10-27T01:15:41  <phantomcircuit> morcos, i always return early in relaytransaction when messing with mempool stuff
 64 2015-10-27T01:22:21  <phantomcircuit> ha i can detect which peers are master by the order of their responses to the mempool command
 65 2015-10-27T01:23:52  <phantomcircuit> 2015-10-27 01:23:33 CSignatureCache.setValid.size() 230249
 66 2015-10-27T01:23:53  <phantomcircuit> 2015-10-27 01:23:33 AcceptToMemoryPool: peer=15: accepted 3f5cfddb91105914ddbe9dc8dd3f32542bd9b1c5fe6b73873b4b6a5c87b62f51 (poolsz 3360)
 67 2015-10-27T01:23:55  <phantomcircuit> interesting
 68 2015-10-27T01:25:41  <gmaxwell> whats interesting?
 69 2015-10-27T01:26:13  <morcos> phantomcircuit: is this a new node?  not the 3GB one?  its not surprising.  most of these spam txs have 100txins, so you'd assume about 100 to 1 ration of txs recieved to setValid size
 70 2015-10-27T01:27:35  <phantomcircuit> gmaxwell, sigcache vs mempool size
 71 2015-10-27T01:27:44  <phantomcircuit> morcos, it's currently talking upto being full
 72 2015-10-27T01:27:46  <phantomcircuit> takes ages
 73 2015-10-27T01:34:20  *** Ylbam has quit IRC
 74 2015-10-27T01:53:58  <phantomcircuit> ok so 1,000,000 sigcache ~= 300MiB mempool
 75 2015-10-27T01:54:34  <phantomcircuit> we should definitely try to switch to a single parameter "-memlimit"
 76 2015-10-27T01:54:40  <phantomcircuit> and then split it out as percentages
 77 2015-10-27T02:30:20  *** d_t has quit IRC
 78 2015-10-27T02:43:50  *** belcher has quit IRC
 79 2015-10-27T02:46:32  *** jl2012_ has quit IRC
 80 2015-10-27T02:46:45  *** jl2012 has joined #bitcoin-core-dev
 81 2015-10-27T04:03:40  *** dcousens has quit IRC
 82 2015-10-27T04:28:16  *** CodeShark has quit IRC
 83 2015-10-27T04:41:18  *** dcousens has joined #bitcoin-core-dev
 84 2015-10-27T04:41:49  <dcousens> hmph.    CXX      libbitcoin_server_a-init.o
 85 2015-10-27T04:41:50  <dcousens> g++: internal compiler error: Killed (program cc1plus)
 86 2015-10-27T04:41:58  <dcousens> Awesome :S
 87 2015-10-27T04:42:43  *** fkhan has quit IRC
 88 2015-10-27T04:43:18  <jcorgan> in my experience that is almost certainly an out-of-memory problem
 89 2015-10-27T04:43:47  <dcousens> jcorgan: interesting, I am trying to compile it on a device with only 1GB
 90 2015-10-27T04:45:15  <jcorgan> look in dmesg for linux OOM killer action
 91 2015-10-27T04:45:24  <dcousens> yeah I did, you were right
 92 2015-10-27T04:46:02  <gmaxwell> add swap. There are some options you can give g++ which will drastically reduce memory usage... but I have no idea what they are. :)
 93 2015-10-27T04:46:16  <dcousens> gmaxwell: easier to just put in another stick atm
 94 2015-10-27T04:52:53  *** blur3d has joined #bitcoin-core-dev
 95 2015-10-27T04:56:36  *** fkhan has joined #bitcoin-core-dev
 96 2015-10-27T04:59:22  *** blur3d has quit IRC
 97 2015-10-27T06:08:02  *** Guest20816 has quit IRC
 98 2015-10-27T06:08:51  *** pigeons has joined #bitcoin-core-dev
 99 2015-10-27T06:09:13  *** pigeons is now known as Guest72716
100 2015-10-27T06:12:47  *** d_t has joined #bitcoin-core-dev
101 2015-10-27T06:18:06  *** Thireus has joined #bitcoin-core-dev
102 2015-10-27T06:20:25  *** d_t has quit IRC
103 2015-10-27T06:23:38  *** ParadoxSpiral has joined #bitcoin-core-dev
104 2015-10-27T06:31:16  <dcousens> poop
105 2015-10-27T06:31:20  <dcousens> 2GB still dies lol
106 2015-10-27T06:31:26  <dcousens> time to look up those g++ options...
107 2015-10-27T06:31:34  <dcousens> or swap... fk it I'll add swap
108 2015-10-27T06:32:33  <dcousens> hmmm, dmesg not showing OOM this time
109 2015-10-27T06:32:35  <dcousens> but same error
110 2015-10-27T06:45:17  *** deepcore has joined #bitcoin-core-dev
111 2015-10-27T06:46:39  *** ParadoxSpiral has quit IRC
112 2015-10-27T06:56:09  <wumpus> 2GB should certainly be enough. See e.g. https://github.com/bitcoin/bitcoin/issues/6658 , worst contender main.cpp uses 1.2MB while compiling (on gcc 4.8). Adding swap may help though, linux' memory management works better if swap is enabled even if you don't need the extra memory
113 2015-10-27T06:56:38  <wumpus> if you want to build bitcoind for a smaller device I'd recommend cross compiling though, it's easy with the depends system
114 2015-10-27T06:56:45  <dcousens> I ended up `git clean -xdf`, re-configuring/autconf and it got past that second hurdle
115 2015-10-27T06:56:54  <dcousens> Guessing it configured something different when I only had 1GB?
116 2015-10-27T06:57:05  <wumpus> it doesn't
117 2015-10-27T06:57:11  <gmaxwell> wumpus: some people are managing to run with no overcommit _and_ no swap; ... dunno how anything at all works for them.
118 2015-10-27T06:57:31  <dcousens> Then I have no idea, except that it worked this time,  but again, no OOM from dmesg so not sure
119 2015-10-27T06:57:34  <wumpus> but it may be  possible for gcc, when it crashes, to generate a poop file that will crash it later
120 2015-10-27T06:59:06  <dcousens> gmaxwell: eh, on my own home nodes I always ran without swap
121 2015-10-27T06:59:19  <dcousens> but, they always had a bit more hardware to play with
122 2015-10-27T06:59:45  <gmaxwell> dcousens: note that I said both no swap and no overcommit.
123 2015-10-27T07:02:59  <wumpus> gmaxwell: I guess if the workload is very predictable
124 2015-10-27T07:03:24  <wumpus> it sounds like the way to go for e.g. embedded devices
125 2015-10-27T07:06:15  <wumpus> but certainly not if you run desktop applications like browsers :-)
126 2015-10-27T07:07:02  <Luke-Jr> dcousens: note that Linux likes to OOM-kill processes long before it runs out of RAM, if you don't have swap :P
127 2015-10-27T07:07:16  <dcousens> gmaxwell: I know.  Eh, I've always kept that at the default, though, I'd be half tempted to turn it off if it meant I'd actually get std:error in my logs haha
128 2015-10-27T07:08:57  <dcousens> Luke-Jr: hasn't been my experience haha, I don't run swap on my most of my work machines and if I OOM, it just grinds to a complete halt usually
129 2015-10-27T07:15:26  <wumpus> I used to have no swap enabled when 8GB still seemed like a lot of memory, but I can confirm what Luke-Jr says, even compile with -j3 would invoke the killer penguin
130 2015-10-27T07:18:29  <wumpus> the problem was that any amount of swapping somehow instantly brought the entire system to a halt (even mouse cursor would no longer move) until it was done.  Managed to improve this somewhat by setting high swappiness.
131 2015-10-27T07:18:51  <wumpus> (which makes it swap little used pages in advance instead of as last desperate measure)
132 2015-10-27T07:19:26  <Luke-Jr> nice thing about a desktop PC is I have like 4-6 hard drives I can RAID0 the swap across..
133 2015-10-27T07:23:40  <wumpus> good idea
134 2015-10-27T07:27:28  *** Ylbam has joined #bitcoin-core-dev
135 2015-10-27T07:29:43  <Luke-Jr> (fwiw, that's builtin to Linux by using swapon -o pri=N where N is identical for everything)
136 2015-10-27T07:37:17  <wumpus> oh!
137 2015-10-27T07:48:39  <GitHub175> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/2b625510d374...38369dda325d
138 2015-10-27T07:48:40  <GitHub175> bitcoin/master 0d699fc Jonas Schnelli: fix locking issue with new mempool limiting...
139 2015-10-27T07:48:40  <GitHub175> bitcoin/master 38369dd Wladimir J. van der Laan: Merge pull request #6889...
140 2015-10-27T07:48:48  <GitHub109> [bitcoin] laanwj closed pull request #6889: fix locking issue with new mempool limiting (master...2015/10/fix_mempool_lock) https://github.com/bitcoin/bitcoin/pull/6889
141 2015-10-27T08:04:13  <wumpus> sipa: what is the idea with #6865? I think we should either remove or change the meaningless test
142 2015-10-27T08:59:05  *** BashCo has quit IRC
143 2015-10-27T09:21:00  *** BashCo has joined #bitcoin-core-dev
144 2015-10-27T09:47:28  *** deepcore has quit IRC
145 2015-10-27T09:50:30  *** moli has joined #bitcoin-core-dev
146 2015-10-27T09:53:19  *** molly has quit IRC
147 2015-10-27T10:09:14  *** isis has quit IRC
148 2015-10-27T10:09:21  <GitHub182> [bitcoin] paveljanik closed pull request #5525: Add removeaddress RPC call (remove watch-only address) (master...rpc_removeaddress) https://github.com/bitcoin/bitcoin/pull/5525
149 2015-10-27T10:19:33  *** Arnavion has quit IRC
150 2015-10-27T10:20:54  *** AtashiCon has quit IRC
151 2015-10-27T10:24:41  *** AtashiCon has joined #bitcoin-core-dev
152 2015-10-27T10:24:46  *** Arnavion has joined #bitcoin-core-dev
153 2015-10-27T10:59:00  <GitHub78> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/38369dda325d...8f3b3cdee497
154 2015-10-27T10:59:00  <GitHub78> bitcoin/master 2d8c49d Casey Rodarmor: Clean up tx prioritization when conflict mined
155 2015-10-27T10:59:01  <GitHub78> bitcoin/master 8f3b3cd Wladimir J. van der Laan: Merge pull request #6464...
156 2015-10-27T10:59:06  <GitHub144> [bitcoin] laanwj closed pull request #6464: Always clean up manual transaction prioritization (master...manual-tx-prioritization-cleanup) https://github.com/bitcoin/bitcoin/pull/6464
157 2015-10-27T12:04:40  *** dcousens has quit IRC
158 2015-10-27T12:35:30  *** droark has quit IRC
159 2015-10-27T12:48:03  *** ParadoxSpiral has joined #bitcoin-core-dev
160 2015-10-27T13:02:09  *** isis has joined #bitcoin-core-dev
161 2015-10-27T14:05:17  *** danielsocials has joined #bitcoin-core-dev
162 2015-10-27T14:07:55  <morcos> I have some questions about changes I'd like to make to manual transaction prioritization and CreateNewBlock and block priority space.
163 2015-10-27T14:09:32  <morcos> I added a fee rate sort to the mempool.  Using this and examining 600 calls to CreateNewBlock over the last 12 hours, the time to create a new block has been reduced from 2.337 seconds + 222 ms to validate to 11 ms + 345 ms to validate
164 2015-10-27T14:09:57  <morcos> I think the increased validation time is because the originaly version made the cache warm first
165 2015-10-27T14:10:26  <morcos> However, this does not create a priority space and does not apply manual transaction prioritization
166 2015-10-27T14:11:02  <morcos> Fixing the manual transaction prioritzation shouldn't be a problem as long as we're willing to store that data in the CTxMemPoolEntry.  I know it was previously discussed not to do that because it might only apply to a few txs
167 2015-10-27T14:11:37  <morcos> However I don't think a sort based on the modified fee will work very well otherwise.  So question 1, can I store the fee delta in the CTxMemPoolEntry.
168 2015-10-27T14:12:03  <morcos> Question 2:  I assume I'm not allowed to get rid of block priority space?
169 2015-10-27T14:13:21  <morcos> I was thinking I could just basically repeat the priority sort of the old alogrithm first, but I believe we might lose some advantages of the speedup.  However I think we could just skip that if blockprioritysize=0.  The code will be a bit uglier that way, but sort of optimized for the case where blockprioritysize=0?
170 2015-10-27T14:13:26  <morcos> Any thoughts?
171 2015-10-27T14:19:05  <morcos> Question 3: BlueMatt: In your issue about tie-breaking you seem to suggest its random.  But isn't the current tiebreaker (for the fee sort) priority?  What would you like the tiebreaker to be?
172 2015-10-27T14:20:43  <morcos> BTW, since the current tie-breaker is priority, and its expensive to calculate priority, I'm not going to try to make my code generate the exact same blocks as the old code.
173 2015-10-27T14:21:17  <morcos> sipa: gmaxwell: wumpus: see above questions please
174 2015-10-27T14:28:16  <sipa> morcos: BlueMatt's problem occurs when there is an abundance of transactions with the exact same priority, i think
175 2015-10-27T14:29:23  <sipa> morcos: i too believe we should get rid of priority, and replace it with a custom function that alters size or fee of a transaction
176 2015-10-27T14:40:50  <gavinandresen> sipa: I agree, but a simple function with a miner-tweakable multiplier (maybe zero by default to mean "no priority adjustment") would be fine, I think.
177 2015-10-27T14:41:21  *** bsm1175321 has quit IRC
178 2015-10-27T14:42:40  <sipa> gavinandresen: right, i was taking code-wise... the policy code can implement an overridable function, and we know that everything keeps.workimg whatever that function is
179 2015-10-27T14:42:58  <sipa> the default implementation can be a simple multiplier
180 2015-10-27T14:43:14  <sipa> or a constant that get added to both fee and size
181 2015-10-27T14:43:30  <gavinandresen> spiffy, I was worried somebody would go and implement a javascript interpreter so miners could tweak the function at runtime... :-)
182 2015-10-27T14:43:59  <sipa> ha
183 2015-10-27T14:44:04  <wumpus> should definitely use lua for that *ducks*
184 2015-10-27T14:44:28  <sipa> we should use bitcoin script with the op_x86 extension
185 2015-10-27T14:45:00  <wumpus> morcos: what is the main drawback of storing the fee delta in CTxMemPoolEntry? increasing the size of the structure?
186 2015-10-27T14:47:11  <wumpus> bitcoin script, hah, even better
187 2015-10-27T14:47:37  *** danielsocials has quit IRC
188 2015-10-27T14:47:46  *** bsm1175321 has joined #bitcoin-core-dev
189 2015-10-27T14:50:24  <wumpus> morcos: optimizing for blockprioritysize=0 sounds good to me, removing it is somewhat controversial - but as it seems to be where things are going anyway, and simplifies the algorithm a lot it makes sense to optimize for the case
190 2015-10-27T14:52:03  <morcos> ok thanks everyone.  yes only downside is more bloat to the mempool, but its relatively small.
191 2015-10-27T14:53:01  <morcos> sipa: so perhaps i'll make a CTxMemPoolEntry function GetMiningPriority (ugh I hate overuse of the word priority) which for now = GetFee() + feeDelta
192 2015-10-27T14:56:41  <morcos> or should I store a miningPriority variable, and use an outside of CTxMempool function to calculate it and set it? I guess it could set it at creation and further calls to prioritizeTransaction would adjust it?
193 2015-10-27T14:59:18  *** danielsocials has joined #bitcoin-core-dev
194 2015-10-27T15:13:16  *** treehug88 has joined #bitcoin-core-dev
195 2015-10-27T15:36:44  <jonasschnelli> cfields: any idea why i get "clang: error: unknown argument: '-framework QtCore'" on my osx machine with qt5.5 [homebrew]?
196 2015-10-27T15:39:20  <cfields> jonasschnelli: mm, that's an odd error. is anything quoted funny?
197 2015-10-27T15:39:54  <jonasschnelli> cfields: i just switched back to Qt 5.4.1 .. there i don't have a such problem..
198 2015-10-27T15:40:10  <jonasschnelli> cfields: no additional details... just the clang error output during linking
199 2015-10-27T15:40:35  <cfields> jonasschnelli: can you paste the exact link-line? (make V=1)
200 2015-10-27T15:40:56  <cfields> if there's a problem with the framework, i'd expect "framework not found" or so
201 2015-10-27T15:41:03  <jonasschnelli> cfields: will do soon... currently switched back to qt5.4.1
202 2015-10-27T15:42:54  <bsm1175321> Can discuss c++11 here if you prefer.  But then what's #bitcoin-dev for?
203 2015-10-27T15:44:00  *** zooko has joined #bitcoin-core-dev
204 2015-10-27T15:44:02  <wumpus> for more abstract things, nitty gritty implementation details for bitcoin core belong here
205 2015-10-27T15:46:21  *** danielsocials has quit IRC
206 2015-10-27T15:46:42  <bsm1175321> wumpus: indeed an explicit shared_ptr gets rid of the compiler error.  Let's see if it runs...
207 2015-10-27T15:47:13  <wumpus> welcome zooko
208 2015-10-27T15:47:15  <wumpus> bsm1175321: great!
209 2015-10-27T15:49:58  <cfields> jonasschnelli: https://github.com/Homebrew/homebrew/commit/fdbb338f2274e093b96209e7ca3ff9bf5460714e
210 2015-10-27T15:50:13  <cfields> patch is here: https://gist.githubusercontent.com/UniqMartin/a54542d666be1983dc83/raw/f235dfb418c3d0d086c3baae520d538bae0b1c70/qtbug-47162.patch
211 2015-10-27T15:50:19  <cfields> it seems to leave out the closing quote
212 2015-10-27T15:51:14  <cfields> er, nm, misread
213 2015-10-27T15:51:23  <jonasschnelli> cfields : you mean a syntax error for the two ends?
214 2015-10-27T15:54:46  <cfields> jonasschnelli: oh, according to the bug report, that patchshould actually fix your problem
215 2015-10-27T15:55:10  <cfields> possible you hadn't updated brew lately before building 5.5?
216 2015-10-27T15:55:58  *** malte has left #bitcoin-core-dev
217 2015-10-27T15:57:14  <cfields> (remember that we use static libs for depends builds, so we wouldn't hit this there for our 5.5.0 build)
218 2015-10-27T15:58:50  *** yammq has joined #bitcoin-core-dev
219 2015-10-27T16:14:20  *** d_t has joined #bitcoin-core-dev
220 2015-10-27T16:17:48  *** yammq has quit IRC
221 2015-10-27T16:30:06  *** BashCo has quit IRC
222 2015-10-27T16:32:04  *** rubensayshi has joined #bitcoin-core-dev
223 2015-10-27T16:40:19  <jonasschnelli> cfields: thanks for looking at it! I think i can solve it now... but need to finish stuff with qt 5.4.1 now until i go back to 5.5.
224 2015-10-27T16:41:02  <GitHub34> [bitcoin] Diapolo opened pull request #6891: constify missing catch cases (master...const-ex) https://github.com/bitcoin/bitcoin/pull/6891
225 2015-10-27T16:46:09  <GitHub81> [bitcoin] Diapolo opened pull request #6892: [Trivial] ensure minimal header conventions (master...headers-new) https://github.com/bitcoin/bitcoin/pull/6892
226 2015-10-27T16:54:59  <GitHub176> [bitcoin] Diapolo opened pull request #6893: forward-declare univalue in rpcclient + remove include in rpcserver.cpp (master...univalue) https://github.com/bitcoin/bitcoin/pull/6893
227 2015-10-27T16:56:48  *** treehug88 has quit IRC
228 2015-10-27T16:59:40  *** lclc has joined #bitcoin-core-dev
229 2015-10-27T17:04:12  <wumpus> sigh...
230 2015-10-27T17:09:58  <jonasschnelli> ;-)
231 2015-10-27T17:19:52  *** treehug88 has joined #bitcoin-core-dev
232 2015-10-27T17:27:26  *** MarcoFalke has joined #bitcoin-core-dev
233 2015-10-27T17:28:09  <btcdrak> Trolling for review: https://github.com/bitcoin/bitcoin/pull/6312
234 2015-10-27T17:33:25  *** BashCo has joined #bitcoin-core-dev
235 2015-10-27T17:38:50  *** treehug88 has quit IRC
236 2015-10-27T17:40:39  *** treehug88 has joined #bitcoin-core-dev
237 2015-10-27T17:42:57  *** MarcoFalke has quit IRC
238 2015-10-27T17:55:41  *** treehug88 has quit IRC
239 2015-10-27T18:00:02  <gmaxwell> btcdrak: is no one backporting mediantimepast to 0.11?
240 2015-10-27T18:00:16  <btcdrak> gmaxwell: I backported it...
241 2015-10-27T18:00:42  <btcdrak> gmaxwell: https://github.com/bitcoin/bitcoin/issues/6884
242 2015-10-27T18:01:35  <btcdrak> gmaxwell: Not sure if we need to backport to 0.10, didnt have time yet anyway.
243 2015-10-27T18:01:59  <gmaxwell> ah okay I missed it.
244 2015-10-27T18:04:01  <btcdrak> gmaxwell: I also made a branch with BIP68+OP_CSV in one as some people asked for it. I'm debating if I should make a PR or just leave the two separate PRs as is.
245 2015-10-27T18:06:28  *** rubensayshi has quit IRC
246 2015-10-27T18:10:54  <gmaxwell> btcdrak: this patch does not appear to enforce (as standardness) the new behavior.
247 2015-10-27T18:13:51  <morcos> there are some unit tests in miner_tests.cpp which verify that CreateNewBlock won't mine non-final txs that are in the mempool
248 2015-10-27T18:13:59  <morcos> they were added by petertodd
249 2015-10-27T18:14:26  <morcos> i was under the impression that we don't want to allow non-final txs into the mempool
250 2015-10-27T18:15:18  <morcos> i suppose i should have checked that my code passed the unit tests earlier on, but the whole idea is to not recheck everything while building the block
251 2015-10-27T18:16:08  <gmaxwell> morcos: the finality test is against the mined block itself, while the mempool is running one block behind.
252 2015-10-27T18:16:47  <morcos> i think that was fixed so the mempool checks against +1, but that would have been a problem in the opposite direction
253 2015-10-27T18:16:51  <gmaxwell> So the mepool need a one block lookahead or there will be an extra delay in mining transactions. (which would presumably mean a competative advantage for miners that change that code)
254 2015-10-27T18:17:30  <morcos> the unit test specifically calls addUnchecked with a non-final tx, and then sees if CreateNewBlock would mine it
255 2015-10-27T18:17:48  <morcos> non-final for 1 block ahead (and a time based version as well)
256 2015-10-27T18:18:24  <morcos> what i'm saying is I'd like to be able ot rely on there being no non-final txs in the mempool, i think this was the idea sipa had in mind too?
257 2015-10-27T18:19:08  <morcos> the one thing I can think of is that its possible we need a function analogous to removeCoinbaseSpends for locktime based txs in the event of a reorg if thats not happening already
258 2015-10-27T18:19:16  <gmaxwell> The hight based can be precise, e.g. so it's not possible to add something that won't be final; and I think that would be good enough for what you're doing. But time based can't be precise; but MPL is fixing that.  I'm fine with there beinging nothing that won't be final.
259 2015-10-27T18:20:17  <gmaxwell> I'm not sure if we're on the same page, so I'm saying that I think there should be currently non-final tx in the mempool which will be final from the perspective of createnewblock,  but none that won't be final from the perspective of createnewblock.
260 2015-10-27T18:21:28  <gmaxwell> Before MTP this was hard to achieve without adding additional delays, but MTP will effectively make that additional delay universal.
261 2015-10-27T18:21:30  *** treehug88 has joined #bitcoin-core-dev
262 2015-10-27T18:21:51  <morcos> I guess thats the same page.  Mempool adds 1 to chainActive.height before checking finality, so i would consider it to be final from the perspective of both the mempool and CNB
263 2015-10-27T18:22:10  <morcos> and yes i agree that the current state is no non-final txs will end up in the mempool
264 2015-10-27T18:22:35  <morcos> what i'm saying is the unit tests require that even if they do, CNB will not put them in the template.  (that breaks for me, and i think is unnecessary computation)
265 2015-10-27T18:22:41  <gmaxwell> okay, fair enough, I was defining current as chainActive.Tip().
266 2015-10-27T18:22:53  <morcos> also what i'm saying is that a reorg could probably break that assumption
267 2015-10-27T18:23:19  <gmaxwell> I think we can drop that test, but we need to make sure that reorgs are removing things correctly at the same time. (I don't know if we have a test for that).
268 2015-10-27T18:23:20  <sdaftuar> morcos: bluematt had a pr recently to fix that
269 2015-10-27T18:24:09  <sdaftuar> #6595
270 2015-10-27T18:24:32  <sdaftuar> we need to pick that up again...
271 2015-10-27T18:24:54  <sdaftuar> oh ha. i remember now, we were going to revamp the way reorgs are handled
272 2015-10-27T18:24:58  <morcos> ah, ok good.  yes we need that.
273 2015-10-27T18:25:12  <morcos> and yeah thats what i was thinking, that you only want to run this after the reorg
274 2015-10-27T18:25:17  <morcos> but why is that difficult
275 2015-10-27T18:25:28  <sdaftuar> i'll dust off that code.  after packages went in, bluematt's open pulls needed some rebasing
276 2015-10-27T18:25:53  <sdaftuar> see also #6656
277 2015-10-27T18:25:53  <morcos> you odn't have to redo the whole way reorgs are handled, you just have to call the removeForReorg after ActivateBestChainStep returns
278 2015-10-27T18:26:02  <sdaftuar> yeah i think that's what 6656 does
279 2015-10-27T18:26:04  <sdaftuar> er, did
280 2015-10-27T18:26:11  <morcos> perhaps you need to flag that the height or mediantimepast might have changed
281 2015-10-27T18:26:14  <morcos> down
282 2015-10-27T18:26:17  <gmaxwell> Perhaps that unit test could be changed to addunchecked but then trigger the logic that handles a reorg.
283 2015-10-27T18:27:37  <morcos> heh, i think thats in a different unit test
284 2015-10-27T18:28:02  <morcos> sdaftuar: so 6656 really supercedes 6595 then.  jsut for some reason 6595 was left open
285 2015-10-27T18:29:12  <morcos> gmaxwell: this actually does lead me to a concern.  i was leaving in the final TestBlockValidity test, but what should the code do in the event it fails?
286 2015-10-27T18:29:39  <morcos> I'm thinking it should switch into mining empty block mode, because otherwise there is no way to figure out what part of your block assembly broke
287 2015-10-27T18:30:08  <morcos> the down side of this, is if there is some error in mempool code that lets non valid txs in, all miners will just start switching to mining empty blocks
288 2015-10-27T18:30:12  <morcos> ugh that sounds terrible
289 2015-10-27T18:30:39  <gmaxwell> We assert now, no? .. yea, but I was just about to suggest it could start doing that. OTOH, that doesn't exactly encourage people to fix things, and empty blocks may mean that you don't failover to another daemon.
290 2015-10-27T18:31:14  <morcos> its much less likely to happen now, because we CheckInputs on every tx as we add it anyway, and if that fails we skip that txs
291 2015-10-27T18:31:22  <gmaxwell> morcos: well empty blocks are safe(er). Assuming that _prior_ chain is not invalid, they're no worse than not mining and likely better.
292 2015-10-27T18:31:27  <morcos> perhaps i should put that code back in
293 2015-10-27T18:31:40  <morcos> and just leave it off, unless you hit an invalid block, and then switch it on
294 2015-10-27T18:31:48  <morcos> because that way you'll just skip over the invalid txs
295 2015-10-27T18:31:52  <morcos> at least for that failure mode
296 2015-10-27T18:32:07  <gmaxwell> Running checkinputs again sounds expensive.
297 2015-10-27T18:32:12  <morcos> but then it won't be a very tested code path, unless we have a mode to test it
298 2015-10-27T18:32:23  <morcos> yeah well thats why the existing code is slow
299 2015-10-27T18:33:34  <morcos> its a nice property of the existing code, that if somebody injects an invalid tx into mempools, the mining code will just keep skipping it and assembling valid blocks from the rest of the txs
300 2015-10-27T18:34:15  <gmaxwell> In any case, thinking in terms of what the user of this code wants;  they either want empty blocks (if they have no fail over) or they want the node to stop responding (if they have fail-over).
301 2015-10-27T18:34:30  <gmaxwell> morcos: right, it keeps the mempool behavior seperate from consensus.
302 2015-10-27T18:39:29  *** zooko has quit IRC
303 2015-10-27T18:44:20  <sipa> morcos: i think the mempool should avoid but not outlaw non-final transactions; CNB should skip the nonfinal ones though
304 2015-10-27T18:48:47  <gmaxwell> what about checkinputs redundancy?  at least doing another IsFinal check is cheap.
305 2015-10-27T18:52:15  <sipa> i think all mempool entries should be valid regarding 1) not spending nonexisting inputs 2) script validity
306 2015-10-27T18:53:12  <sipa> if we have that, i think CNB could just not validate tye result
307 2015-10-27T18:53:22  <sipa> but that seems controversial
308 2015-10-27T18:56:31  <morcos> sipa: so you'd like me to put IsFinal check back in?, you might be right, it might be cheap...
309 2015-10-27T18:57:05  <morcos> i suppose it would be nice to not have to worry about removing txs in the event of a reorg, you know they'll be valid again very soon
310 2015-10-27T18:57:51  <morcos> what do you think about BIP68 and sequence numbers though
311 2015-10-27T18:58:06  <morcos> thats not cheap to check b/c you have to look up the inputs
312 2015-10-27T18:58:37  <gmaxwell> sipa: really if we make CNB no longer latency critical (via mechenisms discussed before) then I think validate after remains a cheap protection.
313 2015-10-27T18:59:07  <gmaxwell> but I'd rather remove some of the internal double checking where it is expensive and really shouldn't be needed but continue to validate after.
314 2015-10-27T19:00:37  <sipa> morcos: i think enforcing nonfinalness in the mempool.is expensive, and the worst a reorg can cheaply do is put a bunch of things in that will be check anyway
315 2015-10-27T19:01:10  <sipa> morcos: fair enough, bip68 does make this significantly more expensive
316 2015-10-27T19:01:50  *** Thireus has quit IRC
317 2015-10-27T19:02:11  *** Thireus has joined #bitcoin-core-dev
318 2015-10-27T19:03:31  <morcos> sipa: i feel like i'm really overloading CTxMempoolEntry, but seems like it could store a block height and time at which the tx is final
319 2015-10-27T19:04:08  <morcos> that way the locktime and bip68 checks that you do at ATMP don't have to be repeated (ugh... i guess unless there is a reorg that affects inputs)
320 2015-10-27T19:04:23  *** randy-waterhouse has joined #bitcoin-core-dev
321 2015-10-27T19:07:57  *** zooko has joined #bitcoin-core-dev
322 2015-10-27T19:14:18  <sipa> morcos: sgtm, if that can guarantee not needing to verify after creation
323 2015-10-27T19:16:29  *** zooko has quit IRC
324 2015-10-27T19:18:21  *** zooko has joined #bitcoin-core-dev
325 2015-10-27T19:19:42  *** d_t has quit IRC
326 2015-10-27T19:23:19  *** d_t has joined #bitcoin-core-dev
327 2015-10-27T19:27:31  <morcos> sipa: well then we're back to the reorg problem right?  but maybe that's solvable more quickly by only having to do work on smaller subset of the txs.  for instance if you know the lowest height and mediantPast time you disconnected down to, you only need to look at txs with CTxMemPoolEntry valid time and heights > that
328 2015-10-27T19:32:07  <GitHub163> [bitcoin] giacecco closed pull request #6885: Instructions on how to make the Homebrew OpenSSL headers visible... (master...master) https://github.com/bitcoin/bitcoin/pull/6885
329 2015-10-27T19:33:36  *** belcher has joined #bitcoin-core-dev
330 2015-10-27T19:36:21  *** PaulCapestany has quit IRC
331 2015-10-27T19:37:59  *** PaulCapestany has joined #bitcoin-core-dev
332 2015-10-27T19:39:41  *** zooko` has joined #bitcoin-core-dev
333 2015-10-27T19:40:33  *** CodeShark has joined #bitcoin-core-dev
334 2015-10-27T19:41:35  *** zooko has quit IRC
335 2015-10-27T19:41:55  *** zooko has joined #bitcoin-core-dev
336 2015-10-27T19:43:20  *** zooko` has quit IRC
337 2015-10-27T19:43:25  *** zooko has quit IRC
338 2015-10-27T19:43:46  *** zooko has joined #bitcoin-core-dev
339 2015-10-27T19:45:01  *** CodeShark has quit IRC
340 2015-10-27T19:46:36  *** randy-waterhouse has quit IRC
341 2015-10-27T19:58:43  <GitHub51> [bitcoin] sdaftuar opened pull request #6894: [Tests] Fix BIP65 p2p test (master...fix-bip65-p2p-test) https://github.com/bitcoin/bitcoin/pull/6894
342 2015-10-27T19:59:03  <morcos> sipa: BlueMatt: sdaftuar and i were just talking.  the ccoinsviewcache growing out of control.  the biggest problem is a reorg!  a reorg will cause removeCoinbaseSpends to run, which will pull every single txin into the cache
343 2015-10-27T19:59:19  <morcos> i assume it'll flush right after that, but at that point it'll be too late if you OOM'ed
344 2015-10-27T19:59:41  <sipa> hmmm
345 2015-10-27T20:00:16  <morcos> i think storing this valid height/time will be very useful for coinbase spends, and moderately useful for checklocktime and csv
346 2015-10-27T20:01:04  <morcos> arguably we should actually store 3 numbers, the height/time its valid at, and the height which if you reorg below you need to recalculate validity
347 2015-10-27T20:01:24  <morcos> then on a reorg we just scan the whole mempool checking that last number, and recalculate validity
348 2015-10-27T20:01:48  <morcos> unfortunatley anti-fee sniping will cause any txs which have arrived in the last couple blocks to be triggered for recalc
349 2015-10-27T20:02:17  <morcos> and wihtout storing more info, you'd have to recalc for coinbase, checklocktime and bip68 sequence, bc you don't know which was invalidated
350 2015-10-27T20:02:25  <morcos> so its a bit of an expensive recalc
351 2015-10-27T20:06:37  *** Thireus1 has joined #bitcoin-core-dev
352 2015-10-27T20:10:18  *** Thireus has quit IRC
353 2015-10-27T20:31:34  *** zooko has quit IRC
354 2015-10-27T20:35:08  <BlueMatt> morcos: no, there is no current tie breaker in CreateNewBlock
355 2015-10-27T20:35:37  <morcos> BlueMatt: sure why not, it uses TxPriorityCompare right?  which sorts by fee then priority
356 2015-10-27T20:36:51  <BlueMatt> morcos: the way I read it, it first creates a heap according to priority, then resorts to create a heap using fee
357 2015-10-27T20:36:55  <BlueMatt> it never uses fee then priority
358 2015-10-27T20:37:31  <morcos> yes correct, but if you look at that compartor, in either case it uses the other one as the tie break
359 2015-10-27T20:37:51  <morcos> comparatator
360 2015-10-27T20:38:33  <BlueMatt> hmmmm, indeed, I hadnt seen that before....how is it that ordering seems randomish, then?
361 2015-10-27T20:39:08  <BlueMatt> lemme go look at mempool
362 2015-10-27T20:40:25  <morcos> yeah i was wondering that, you would think that would be fairly consistent across nodes
363 2015-10-27T20:41:25  <morcos> i wonder if in this recent attack there are many things with also the same priority...  seems a bit unlikely..
364 2015-10-27T20:41:50  <sipa> that was my assumption
365 2015-10-27T20:42:04  <sipa> if all outputs are created in the same block, and the spends are the same size
366 2015-10-27T20:42:10  <sipa> that's what you would get
367 2015-10-27T20:57:38  *** zooko has joined #bitcoin-core-dev
368 2015-10-27T21:03:54  *** Thireus has joined #bitcoin-core-dev
369 2015-10-27T21:04:46  *** ParadoxSpiral has quit IRC
370 2015-10-27T21:07:50  *** Thireus1 has quit IRC
371 2015-10-27T21:27:08  *** Thireus1 has joined #bitcoin-core-dev
372 2015-10-27T21:27:47  *** Thireus2 has joined #bitcoin-core-dev
373 2015-10-27T21:28:17  *** Thireus1 has quit IRC
374 2015-10-27T21:28:30  *** Thireus1 has joined #bitcoin-core-dev
375 2015-10-27T21:29:51  *** Thireus3 has joined #bitcoin-core-dev
376 2015-10-27T21:30:28  *** Thireus has quit IRC
377 2015-10-27T21:30:30  *** Thireus2 has quit IRC
378 2015-10-27T21:33:51  *** Thireus1 has quit IRC
379 2015-10-27T21:51:12  *** deepcore has joined #bitcoin-core-dev
380 2015-10-27T21:55:21  <BlueMatt> morcos: hmmmm, now I'm very confused
381 2015-10-27T21:55:32  <BlueMatt> morcos: there are almost no txn in my mempool which have the same fee and prio
382 2015-10-27T21:56:41  <morcos> BlueMatt: I didn't look into the root issue, its just that you are not seeing the blocks you expect to see?  It's not a result of malleated txs?  Where some nodes have one version and others have another
383 2015-10-27T21:59:43  <BlueMatt> it could be related to malleated txn, but ive definitely seen it every time mempools get large, not necessarily when malleation is happening
384 2015-10-27T22:01:43  <sipa> might it just be different miners using different policies?
385 2015-10-27T22:05:50  <BlueMatt> afaiu there are effectively no different policies in use by miners today
386 2015-10-27T22:05:54  <BlueMatt> (aside from eligius)
387 2015-10-27T22:06:30  *** droark has joined #bitcoin-core-dev
388 2015-10-27T22:06:59  *** deepcore has quit IRC
389 2015-10-27T22:12:24  *** deepcore has joined #bitcoin-core-dev
390 2015-10-27T22:14:36  *** dcousens has joined #bitcoin-core-dev
391 2015-10-27T22:50:35  *** deepcore has quit IRC
392 2015-10-27T23:11:17  <dcousens> when are the IRC meetings planned?
393 2015-10-27T23:11:22  <dcousens> ooi
394 2015-10-27T23:15:21  <sipa> thursday 7pm UTC
395 2015-10-27T23:15:28  <dcousens> ta
396 2015-10-27T23:27:08  *** Thireus has joined #bitcoin-core-dev
397 2015-10-27T23:29:57  *** Thireus3 has quit IRC
398 2015-10-27T23:33:52  *** Thireus has quit IRC
399 2015-10-27T23:34:05  *** Thireus has joined #bitcoin-core-dev
400 2015-10-27T23:39:53  *** Thireus has quit IRC
401 2015-10-27T23:40:13  *** Thireus has joined #bitcoin-core-dev
402 2015-10-27T23:51:34  *** Thireus1 has joined #bitcoin-core-dev
403 2015-10-27T23:54:50  *** Thireus has quit IRC