1 2015-10-26T00:07:18  *** moli has joined #bitcoin-core-dev
  2 2015-10-26T00:10:15  *** molly has quit IRC
  3 2015-10-26T00:25:31  *** CodeShark has quit IRC
  4 2015-10-26T00:25:35  *** CodeShark_ has quit IRC
  5 2015-10-26T00:25:45  *** CodeShark has joined #bitcoin-core-dev
  6 2015-10-26T00:38:45  *** molly has joined #bitcoin-core-dev
  7 2015-10-26T00:40:07  *** evoskuil has quit IRC
  8 2015-10-26T00:41:56  *** moli has quit IRC
  9 2015-10-26T01:10:19  *** evoskuil has joined #bitcoin-core-dev
 10 2015-10-26T01:13:46  *** CodeShark has quit IRC
 11 2015-10-26T01:14:20  *** Ylbam has quit IRC
 12 2015-10-26T01:23:10  *** jgarzik has quit IRC
 13 2015-10-26T02:01:01  *** dgenr8 has quit IRC
 14 2015-10-26T02:02:29  *** dgenr8 has joined #bitcoin-core-dev
 15 2015-10-26T02:40:28  *** d_t has quit IRC
 16 2015-10-26T02:47:45  *** belcher has quit IRC
 17 2015-10-26T02:54:26  <BlueMatt> someone wanna kick travis on #6875
 18 2015-10-26T02:57:49  <sipa> done
 19 2015-10-26T02:58:42  *** dgenr8 has quit IRC
 20 2015-10-26T02:59:37  *** dgenr8 has joined #bitcoin-core-dev
 21 2015-10-26T03:05:37  *** dgenr8 has quit IRC
 22 2015-10-26T03:29:42  *** d_t has joined #bitcoin-core-dev
 23 2015-10-26T04:16:51  *** PaulCape_ has joined #bitcoin-core-dev
 24 2015-10-26T04:19:10  *** PaulCapestany has quit IRC
 25 2015-10-26T04:19:21  *** d_t has quit IRC
 26 2015-10-26T04:20:24  *** JoeLiu has joined #bitcoin-core-dev
 27 2015-10-26T04:24:31  *** d_t has joined #bitcoin-core-dev
 28 2015-10-26T04:30:36  *** dgenr8 has joined #bitcoin-core-dev
 29 2015-10-26T05:08:28  *** go1111111 has joined #bitcoin-core-dev
 30 2015-10-26T05:13:45  *** d_t has quit IRC
 31 2015-10-26T05:32:54  *** d_t has joined #bitcoin-core-dev
 32 2015-10-26T06:25:48  *** JoeLiu has quit IRC
 33 2015-10-26T07:06:33  <GitHub133> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/46f74379b86b...450893769f7a
 34 2015-10-26T07:06:33  <GitHub133> bitcoin/master ceb2a9c Wladimir J. van der Laan: doc: mention BIP65 softfork in bips.md
 35 2015-10-26T07:06:34  <GitHub133> bitcoin/master 4508937 Wladimir J. van der Laan: Merge pull request #6879...
 36 2015-10-26T07:06:43  <GitHub137> [bitcoin] laanwj closed pull request #6879: doc: mention BIP65 softfork in bips.md (master...2015_10_bip65) https://github.com/bitcoin/bitcoin/pull/6879
 37 2015-10-26T07:21:23  <GitHub163> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/450893769f7a...867d6c90b850
 38 2015-10-26T07:21:23  <GitHub163> bitcoin/master dca7bd3 Wladimir J. van der Laan: doc: Add developer notes about gitignore...
 39 2015-10-26T07:21:24  <GitHub163> bitcoin/master 867d6c9 Wladimir J. van der Laan: Merge pull request #6878...
 40 2015-10-26T07:21:28  <GitHub17> [bitcoin] laanwj closed pull request #6878: doc: Add developer notes about gitignore (master...2015_10_ignore_files) https://github.com/bitcoin/bitcoin/pull/6878
 41 2015-10-26T07:35:06  <jonasschnelli> gmaxwell: re: -maxuploadtarget. Will try to complete it today. Need to adapt sdaftuar tests and rebase/squash.
 42 2015-10-26T07:45:17  *** paveljanik has quit IRC
 43 2015-10-26T07:45:36  *** paveljanik has joined #bitcoin-core-dev
 44 2015-10-26T07:45:47  *** paveljanik has quit IRC
 45 2015-10-26T08:06:13  *** Arnavion has quit IRC
 46 2015-10-26T08:07:04  *** Arnavion has joined #bitcoin-core-dev
 47 2015-10-26T08:07:26  *** Ylbam has joined #bitcoin-core-dev
 48 2015-10-26T08:10:04  <GitHub137> [bitcoin] laanwj pushed 7 new commits to master: https://github.com/bitcoin/bitcoin/compare/867d6c90b850...5242bb32c723
 49 2015-10-26T08:10:05  <GitHub137> bitcoin/master 4d2a926 dexX7: Ignore coverage data related and temporary test files
 50 2015-10-26T08:10:05  <GitHub137> bitcoin/master d425877 dexX7: Remove coverage and test related files, when cleaning up...
 51 2015-10-26T08:10:06  <GitHub137> bitcoin/master 8e3a27b dexX7: Require Python for RPC tests, when using lcov...
 52 2015-10-26T08:10:15  <GitHub173> [bitcoin] laanwj closed pull request #6813: Support gathering code coverage data for RPC tests with lcov (master...btc-test-lcov-rpc) https://github.com/bitcoin/bitcoin/pull/6813
 53 2015-10-26T08:34:54  <jonasschnelli> anyone getting the same build errors (osx) for current master: boost: mutex lock failed in pthread_mutex_lock: Invalid argument?
 54 2015-10-26T08:35:06  * jonasschnelli is doing a fresh checkout / build
 55 2015-10-26T08:43:30  <jonasschnelli> hmm.. same issue when doing a fresh checkout / build...
 56 2015-10-26T08:44:03  <wumpus> is that a *build* error?
 57 2015-10-26T08:44:20  <jonasschnelli> libc++abi.dylib: terminating with uncaught exception of type boost::exception_detail::clone_impl<boost::exception_detail::error_info_injector<boost::lock_error> >: boost: mutex lock failed in pthread_mutex_lock: Invalid argument
 58 2015-10-26T08:44:32  <wumpus> looks like a runtime errror to me, something abusing boost?
 59 2015-10-26T08:45:00  <jonasschnelli> i try now to upgarde from 1.57 to 1.58...
 60 2015-10-26T08:45:09  <wumpus> let's first try to debug it
 61 2015-10-26T08:45:15  <wumpus> when does it happen? can you get a traceback?
 62 2015-10-26T08:45:26  <jonasschnelli> stacktrace stops when program tries to call the first LOCK
 63 2015-10-26T08:46:02  <wumpus> ok
 64 2015-10-26T08:46:17  <jonasschnelli> EnterCritical() / push_lock()
 65 2015-10-26T08:46:20  <wumpus> when was the last time it worked? maybe try a git bisect
 66 2015-10-26T08:46:42  <jonasschnelli> that is strange: *pthread_mutex_lock: Invalid argument*
 67 2015-10-26T08:47:10  <jonasschnelli> wumpus: okay... i'll try to find out if it's related to a source change or something on my local machine.
 68 2015-10-26T08:47:12  <wumpus> can be a use-after-free of a lock
 69 2015-10-26T08:48:21  <wumpus> if it happens at startup it may be a unexpected error path that doesn't wind down properly
 70 2015-10-26T09:01:27  *** BashCo has quit IRC
 71 2015-10-26T09:02:03  *** BashCo has joined #bitcoin-core-dev
 72 2015-10-26T09:06:39  *** BashCo has quit IRC
 73 2015-10-26T09:07:22  <GitHub136> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/5242bb32c723...26f5b34e8838
 74 2015-10-26T09:07:23  <GitHub136> bitcoin/master 10e2eae Wladimir J. van der Laan: rpc: Add maxmempool and effective min fee to getmempoolinfo
 75 2015-10-26T09:07:23  <GitHub136> bitcoin/master 26f5b34 Wladimir J. van der Laan: Merge pull request #6877...
 76 2015-10-26T09:07:32  <GitHub170> [bitcoin] laanwj closed pull request #6877: rpc: Add maxmempool and effective min fee to getmempoolinfo (master...2015_10_mempool_effective_fee) https://github.com/bitcoin/bitcoin/pull/6877
 77 2015-10-26T09:14:19  *** dcousens has joined #bitcoin-core-dev
 78 2015-10-26T09:19:18  *** BashCo has joined #bitcoin-core-dev
 79 2015-10-26T09:24:43  *** AtashiCon has quit IRC
 80 2015-10-26T09:27:56  *** rubensayshi has joined #bitcoin-core-dev
 81 2015-10-26T09:28:11  *** AtashiCon has joined #bitcoin-core-dev
 82 2015-10-26T09:33:32  <dcousens> gmaxwell: with the address index comment, you mention it might drive away users, but,  this is an opt-in patch
 83 2015-10-26T09:34:38  <dcousens> I'd much rather use this than my own self-maintained version of the patch,  hell,  several block explorers I know have written their own implementations eating up much more than 50GB of space etc
 84 2015-10-26T09:38:46  <btcdrak> dcousens: FYI, my .bitcoin folder is 67GB with addrindex turned on. What is it without?
 85 2015-10-26T09:39:16  <dcousens> btcdrak: moment, just checking
 86 2015-10-26T09:40:03  <dcousens> 56G
 87 2015-10-26T09:40:14  <dcousens> with -txindex
 88 2015-10-26T09:40:27  <btcdrak> dcousens: but as a maintainer of an addrindex fork, I think this particular implementation is absolutely not the right way to go about it. It's ugly. I'd much prefer to see something like #5048 which maintains a separate index entirely.
 89 2015-10-26T09:40:35  <dcousens> Oh no doubt
 90 2015-10-26T09:40:38  <dcousens> I meant in concept only
 91 2015-10-26T09:40:53  <dcousens> Hence, only my concept ACK
 92 2015-10-26T09:41:53  <dcousens> There might be a way to do this as gmaxwell pointed out, via fast re-scan or something
 93 2015-10-26T09:42:17  <dcousens> It would just have to be super fast in the case of being feasible for block explorers etc, but, I guess you could just throw some caches/LBs in front of it
 94 2015-10-26T09:43:53  <btcdrak> dcousens: yeah, I'm not sure how addrindex would scale in the long term.
 95 2015-10-26T09:45:09  <dcousens> yeah, but I mean, we have that same concern for the blockchain itself haha
 96 2015-10-26T09:45:28  <dcousens> I just figured, in terms of worrying about users losing resources, in the end, their opting-in, so they'd be aware no?
 97 2015-10-26T09:46:39  <btcdrak> dcousens: I'm no longer as passionate because I provide a fork for those that want it. It's easy enough to maintain.
 98 2015-10-26T09:46:46  <wumpus> indexing the whole block chain is generally the wrong thing to do
 99 2015-10-26T09:47:57  <wumpus> and I understand there are forensic or research reasons to do so, but if you end up at that it's generally an indication you're doing something wrong in your design
100 2015-10-26T09:48:35  <dcousens> wumpus: that really depends on your trust model though?
101 2015-10-26T09:48:49  <wumpus> no, it depends on your scaling model
102 2015-10-26T09:49:44  <wumpus> it's best to regard the block chain as transitory data used to update the utxo state
103 2015-10-26T09:50:02  <wumpus> (as well as other things, such as wallets)
104 2015-10-26T09:51:40  <dcousens> agreed, but the UTXO may not be all you care about.  But yes your right, in that sense, its your scaling model
105 2015-10-26T09:52:05  <wumpus> that's why I added (as well as other things) - it's data you want to use to apply changes to your running state, then throw away
106 2015-10-26T09:52:43  <dcousens> wumpus: but that lacks the information in the case of a user wanting to see the 'history' of a certain address
107 2015-10-26T09:52:51  <wumpus> bitcoind's wallet, as one exaple, does this correctly. It only requires the whole beast for rescans. It would be nice to have an external example though that isn't embedded into bitcoind.
108 2015-10-26T09:52:51  <dcousens> the only way to generate that history would be to view all hte transitory data
109 2015-10-26T09:53:10  <wumpus> the way to generate that history would be to store it as it comes along, as metadata
110 2015-10-26T09:53:33  <dcousens> that assumes you know the address from the start
111 2015-10-26T09:53:38  <wumpus> the same way bitcoind's wallet does - store the transactions, or whatever you need to list it later
112 2015-10-26T09:54:29  <dcousens> for example for a wallet I maintain, we very often have users that come along with BIP39 mnemonics with 100's of used addresses, forcing a rescan for each time they introduce a new mnemonic/wallet would be a huge rescan burden for us
113 2015-10-26T09:54:36  <wumpus> history (or the part of history that you want to store for accounting purposes) is part of your running state
114 2015-10-26T09:54:45  <dcousens> Which again, relates to our scaling model, but, that too relates directly to this patch (as a concept)
115 2015-10-26T09:57:03  <wumpus> yes, if you essentially offer rescanning-as-a-service it makes sense to keep an index. I still think it should be something external though, not part of bitcoind. There are tons of differents kinds of indexes one could want, for specific purposes
116 2015-10-26T09:57:48  <wumpus> ideally some external indexing tool would source block data from bitcoind then keep an index, as running state
117 2015-10-26T09:59:27  <wumpus> so my points are: do not encourage people to keep that index, and do not clutter the codebase with all kinds of extra indexes not necessary for node functionaity. It is not "never create an index". sure there may be reasons to do so but it's very custom
118 2015-10-26T10:00:24  <dcousens> wumpus: hmmm, but, maintaing this indexing is not a simple task
119 2015-10-26T10:00:37  <btcdrak> wumpus: I think I finally see your point
120 2015-10-26T10:00:40  <wumpus> I haven't used the word 'simple'
121 2015-10-26T10:00:45  <dcousens> It requires a lot of complex logic in regards to re-orgs etc
122 2015-10-26T10:01:00  <dcousens> wumpus: I know, just putting it out there as to why this is probably a persistent feature request
123 2015-10-26T10:01:08  <wumpus> yes, it requires logic with regard to re-orgs. To be precise, you need to be able to roll back updates from a block.
124 2015-10-26T10:01:33  <wumpus> bitcoind does this by keeping undo files, but your database may have features to do so as well
125 2015-10-26T10:02:08  <wumpus> and once this code is written people can just use it, it's no more complex than using your patch
126 2015-10-26T10:02:38  <dcousens> wumpus: in my mind, it seems more like its just a way to offload the maintenance burden haha
127 2015-10-26T10:03:44  <btcdrak> why dont we use zmq to notify external apps?
128 2015-10-26T10:03:59  <wumpus> there's --enable-zmq?
129 2015-10-26T10:04:41  <dcousens> wumpus: is there a re-org notification?
130 2015-10-26T10:04:48  <dcousens> That would make the roll backs super simple
131 2015-10-26T10:04:53  <btcdrak> it could store whatever it wants, bitcoind would notify about reorgs etc. that would allow to build a sort of indexd. The indexd could always query bitcoind for things like blocks as a passthrough
132 2015-10-26T10:05:46  <wumpus> not sure - there is a notification when there is a new block, you can use that to determine if there has been a reorg. But a more direct notification of what blocks have to be un-done/applies might be useful too. Though be careful, zmq notifications can be lossy, so you shouldn't rely on their content.
133 2015-10-26T10:06:08  <wumpus> btcdrak: right
134 2015-10-26T10:06:18  <wumpus> indexd would pull the data it requires from bitcoind
135 2015-10-26T10:06:20  <btcdrak> wumpus: if we're missing notifications I'm sure they are easy to add.
136 2015-10-26T10:06:36  <dcousens> wumpus: really? Are they consistently lossy?
137 2015-10-26T10:07:02  <wumpus> dcousens: yes, if the queue is full new messages are discarded
138 2015-10-26T10:07:54  <dcousens> wumpus: right, so its just a matter of consumption
139 2015-10-26T10:08:35  <wumpus> not sure if are any guarantees for delivery - that's why I have always been kind of careful there, it's easy to use it in a wrong way. Notifications are "hey, wake up, there is new information". If those get lost there's no big issue, it will pick them up next time.
140 2015-10-26T10:10:21  <dcousens> wumpus: mmm, I guess, in the end, its as btcdrak as suggested.  The underyling problem is that the maintenance of the address index becomes difficult due to re-org detection.  If we can have bitcoind provide us with that information in a clear way, most of this becomes trivial
141 2015-10-26T10:11:49  <wumpus> I'm not sure reorg detection is so difficult? keep the list of hashes up to the current block, then if a new best block comes in, look at its parents, where it intersects with the list of your blocks is the fork/reorg point
142 2015-10-26T10:12:05  <dcousens> wumpus: requires maintaing another list, another source of data
143 2015-10-26T10:12:10  <wumpus> yes it does
144 2015-10-26T10:12:58  <wumpus> that's life, you need to maintain the data structures that you need to do your work. only be keeping this state yourself you can be sure your state is consistent.
145 2015-10-26T10:13:30  <btcdrak> indexd could even use bitcoind for SPV lookups
146 2015-10-26T10:14:04  <wumpus> would be great to have some library or existing software to handle this for you, I'm just talking concepts here not implementation
147 2015-10-26T10:14:33  <dcousens> btcdrak: I was thinking about that
148 2015-10-26T10:14:55  <dcousens> you don't even need to verify since you'd already trust your own node
149 2015-10-26T10:15:10  <wumpus> right - you outsource trust to your node, verification, that's what it's for
150 2015-10-26T10:15:24  <dcousens> just as wumpus said, probably still need to maintain the hash-chain if theres no notification for re-orgs
151 2015-10-26T10:15:43  <wumpus> well you need to know what hash-chain your state is based on
152 2015-10-26T10:16:54  <dcousens> well, not really, all you actually care about is what blocks got reverted, such that you can roll back the relevant transactions in the index
153 2015-10-26T10:17:08  <wumpus> otherwise the couling is dangerously tight, say, your indexd was disconnected from bitcoind for a while and you reconnect and have to catch up
154 2015-10-26T10:17:17  <wumpus> coupling*
155 2015-10-26T10:17:42  <wumpus> yes, but you need to detect that yourself. You can't rely on bitcoind to keep state for you
156 2015-10-26T10:17:59  <dcousens> wumpus: it would know what blocks get reverted in a re-org, no?
157 2015-10-26T10:18:31  <wumpus> but only in realtime - ideally you don't want that kind of tight coupling. What if indexd wasn't connected at the time the reorg happens. Do you want to restart all the way from the beginning?
158 2015-10-26T10:18:58  <dcousens> true
159 2015-10-26T10:19:05  <wumpus> you have a) your own hash chain b) bitcoind's hash chain  , you always want to go from a to b
160 2015-10-26T10:19:42  <wumpus> (given that you trust bitcoind to give you the best chain)
161 2015-10-26T10:20:28  <jonasschnelli> wumpus: i can successfully build up to https://github.com/bitcoin/bitcoin/commit/579b863cd7586b98974484ad55e19be2a54d241d
162 2015-10-26T10:20:48  <jonasschnelli> wumpus: https://github.com/bitcoin/bitcoin/commit/a09297010e171af28a7a3fcad65a4e0aefea53ba seams to break things.. how can this even be possible...
163 2015-10-26T10:20:59  <jonasschnelli> no code changes at all
164 2015-10-26T10:21:02  <dcousens> btcdrak: happy to work with you on indexd concept
165 2015-10-26T10:21:12  <wumpus> would be awesome if someone implemented that
166 2015-10-26T10:21:40  <wumpus> jonasschnelli: huh that just changes my silly dev tools :)
167 2015-10-26T10:22:01  <jonasschnelli> yeah. I know... let me investigate more
168 2015-10-26T10:28:02  <wumpus> so 0fbfc51 works?
169 2015-10-26T10:30:35  <wumpus> (that's the last commit before merging #6854)
170 2015-10-26T10:33:06  <GitHub168> [bitcoin] laanwj pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/26f5b34e8838...ff057f41aa14
171 2015-10-26T10:33:07  <GitHub168> bitcoin/master 9d55050 Mark Friedenbach: Add rules--presently disabled--for using GetMedianTimePast as endpoint for lock-time calculations...
172 2015-10-26T10:33:07  <GitHub168> bitcoin/master dea8d21 Mark Friedenbach: Enable policy enforcing GetMedianTimePast as the end point of lock-time constraints...
173 2015-10-26T10:33:08  <GitHub168> bitcoin/master ff057f4 Wladimir J. van der Laan: Merge pull request #6566...
174 2015-10-26T10:33:11  <GitHub161> [bitcoin] laanwj closed pull request #6566: BIP-113: Mempool-only median time-past as endpoint for lock-time calculations (master...medianpasttimelock) https://github.com/bitcoin/bitcoin/pull/6566
175 2015-10-26T10:36:38  <btcdrak> dcousens: sure.
176 2015-10-26T10:38:13  <dcousens> wumpus: would it better to maintain indexd in repo or as a separate application?
177 2015-10-26T10:39:48  <wumpus> well - better is always to have it completely detached, but I suppose developing it inside the repository is easier for now if you need serialization and certain data structures
178 2015-10-26T10:40:04  <wumpus> we don't have a way to export those as a library yet
179 2015-10-26T10:40:22  <jonasschnelli> wumpus : found the problem: https://github.com/bitcoin/bitcoin/pull/6722/files#diff-ca74c4b28865382863b8fe7633a85cd6R312
180 2015-10-26T10:40:40  <dcousens> wumpus: aye
181 2015-10-26T10:40:50  <jonasschnelli> clear() calls LOCK() in cxx_global_var_init()
182 2015-10-26T10:41:15  <jonasschnelli> i think this should be changed.
183 2015-10-26T10:42:41  *** BashCo_ has joined #bitcoin-core-dev
184 2015-10-26T10:43:19  *** BashCo has quit IRC
185 2015-10-26T10:43:50  *** BashCo has joined #bitcoin-core-dev
186 2015-10-26T10:44:06  <wumpus> common, straightforward to do this is to separate it out into a private function, say clear_(), and a public function clear() that gets the lock and calls clear_()
187 2015-10-26T10:44:35  <wumpus> this way, functions that already have the lock, or don't need it otherwise, can call clear_ instead
188 2015-10-26T10:45:06  <wumpus> in this case it is the constructor so obviously it doesn't need the lock
189 2015-10-26T10:45:28  <jonasschnelli> okay... can try a patch.
190 2015-10-26T10:47:32  *** BashCo_ has quit IRC
191 2015-10-26T10:49:14  <wumpus> not that it should normally hurt to lock a lock that is part of your object inthe constructor... but this looks like some global obscure initialization order thing because the mempool object is global :(
192 2015-10-26T10:51:57  <wumpus> (probably it would be better to use an explicit lifetime for objects such as these, but fixing that is more than you bargained for)
193 2015-10-26T10:54:59  *** dcousens has quit IRC
194 2015-10-26T11:03:50  *** dcousens has joined #bitcoin-core-dev
195 2015-10-26T11:20:35  *** jtimon has quit IRC
196 2015-10-26T11:32:25  <Luke-Jr> huh, the wallet rescan % shown in the splash screen is much higher than the debug.log Progress - block count vs tx count? :/
197 2015-10-26T11:36:13  <wumpus> GUIstd::max(1, std::min(99, (int)((Checkpoints::GuessVerificationProgress(chainParams.Checkpoints(), pindex, false) - dProgressStart) / (dProgressTip - dProgressStart) * 100)))
198 2015-10-26T11:36:23  <wumpus> log: Checkpoints::GuessVerificationProgress(chainParams.Checkpoints(), pindex)
199 2015-10-26T11:37:51  <wumpus> the GUI takes into account the starting point, which you'd expect would result in it showing a *lower* progress % when it makes a difference
200 2015-10-26T11:38:01  <wumpus> but the GUI also has fSigchecks=false for GuessVerificationprogress, maybe tha'ts it
201 2015-10-26T12:42:47  <wumpus> this is curious, anyone else having problems with address parsing on windows 10? https://github.com/bitcoin/bitcoin/issues/6886
202 2015-10-26T12:44:13  <wumpus> looks like the IP parsing (using getaddrinfo) always returns 0:0:0:0:0:0:0:0/128, IPv4 parsing as well as IPv6 parsing fails
203 2015-10-26T12:49:35  *** jtimon has joined #bitcoin-core-dev
204 2015-10-26T12:51:13  *** ParadoxSpiral has joined #bitcoin-core-dev
205 2015-10-26T13:57:56  <GitHub175> [bitcoin] jonasschnelli opened pull request #6889: fix locking issue with new mempool limiting (master...2015/10/fix_mempool_lock) https://github.com/bitcoin/bitcoin/pull/6889
206 2015-10-26T14:07:30  *** Guest25608 has quit IRC
207 2015-10-26T14:07:36  *** treehug88 has joined #bitcoin-core-dev
208 2015-10-26T14:14:00  *** pigeons has joined #bitcoin-core-dev
209 2015-10-26T14:14:24  *** pigeons is now known as Guest20816
210 2015-10-26T14:18:07  *** danielsocials has joined #bitcoin-core-dev
211 2015-10-26T14:23:37  *** danielsocials has quit IRC
212 2015-10-26T14:26:45  *** danielsocials has joined #bitcoin-core-dev
213 2015-10-26T14:46:36  <morcos> Does anybody have any thoughts on storing the number of sigOps in a tx in the CTxMempoolEntry?
214 2015-10-26T14:47:06  <morcos> We calculate this on ATMP, and we need it in CreateNewBlock, but it is on the expensive'ish side to calculate.
215 2015-10-26T14:48:24  <sipa> sounds good
216 2015-10-26T14:48:40  <morcos> Wow, I was prepared to have to fight for that... :)
217 2015-10-26T14:50:29  <morcos> I was naively hoping you could just assume every scriptSig was satisfying a P2SH and you wouldn't over count by too much, but turns out that is NOT the case..
218 2015-10-26T14:55:39  <dgenr8> morcos: any plans to cache ancestor pkg sums the way descendants are?
219 2015-10-26T14:56:59  <morcos> dgenr8: sdaftuar has written something which he might publish soon.  but i don't think there is any point in merging it until we have mining code that can take advantage of it.
220 2015-10-26T14:57:15  <morcos> i'm starting by just trying to make the existing mining code much faster
221 2015-10-26T14:59:03  <gmaxwell> 02:58 < gmaxwell> Is there a reason that for fee purposes we are not using max(size, sigops*BLOCK_MAX_BYTES/BLOCK_MAX_SIGOPS) as the size of a transaction?
222 2015-10-26T14:59:26  <gmaxwell> as in "the size" that we use for mempool/etc. purposes.
223 2015-10-26T15:02:07  <gmaxwell> dcousens: Opt-in isn't sufficient to prevent it from driving users out, I do explain this...
224 2015-10-26T15:02:33  <morcos> gmaxwell: i saw that.  on my list somewhere is to research algorithms for satisfying multiple constraints.  presumably if we know that actual size is usually the limiting factor, then it doesn't seem right to sort something lower b/c it has a lot of sigops even though you're not going to hit that limit
225 2015-10-26T15:02:56  <morcos> i was thnking about it in the terms of some future consensus requirement on utxo deltas as well
226 2015-10-26T15:03:11  *** bsm1175321 has quit IRC
227 2015-10-26T15:05:05  <morcos> the discussion we were having the other day about a separate thread for template generation seems very difficult to me.  its very hard to extract code from needing to lock at least the mempool.
228 2015-10-26T15:05:45  <morcos> i think to do that, we might need 2 mempools, or to just allow the mempool to queue tx's while the mining code is running
229 2015-10-26T15:06:43  <morcos> but its an easy win to make existing createnewblock significantly faster just by adding an index on individual tx feerate, so i figured i'd start with that.
230 2015-10-26T15:12:04  *** bsm1175321 has joined #bitcoin-core-dev
231 2015-10-26T15:13:58  *** BashCo_ has joined #bitcoin-core-dev
232 2015-10-26T15:17:35  *** BashCo has quit IRC
233 2015-10-26T15:19:27  <gmaxwell> morcos: I've generally assumed the multivariable optimization would be much harder and not really worth it-- the sigops limit is high enough that it doesn't normally have an effect. What I suggested at least prevents the case where some silly dos attack transaction with a high per byte fee kills a feerate greedy selection.
234 2015-10-26T15:20:41  <gmaxwell> morcos: hm. So I didn't expect the seperate template thread would avoid locking the mempool. I assumed it would. But that having it would allow the createnewblock to always run without taking any long-held locks.
235 2015-10-26T15:21:33  <gmaxwell> E.g. block creation thread would lock the mempool and build a block. Then take a blocktemplate lock for just an instant to replace the last template.
236 2015-10-26T15:21:45  <morcos> gmaxwell: ah, interesting, so trick the mining code into producing small blocks unecessarily..  maybe that is a good idea.
237 2015-10-26T15:23:33  <morcos> gmaxwell: i see.  i guess i need to understand how getblocktemplate is used in practice.  i was assuming the threading was so we could be optimizing all the time.  i didn't realize it was because you wanted to return a previously generated template to someone calling gbt.  i assumed they had already gotten the previous template
238 2015-10-26T15:24:07  <gmaxwell> and if the cached template is out of date with the chain when gbt runs, you just generate an empty template, which you can do without touching he mempool.
239 2015-10-26T15:25:32  <morcos> yeah, so i did write a threaded version that does what you suggested, but it was bad, because it was basically just busy grabbing cs_main and mempool, and i thought well ok we'll have to make it run only every so often, but then i didn't see why that was better than the existing usage
240 2015-10-26T15:26:02  <morcos> but you want to optimize for the case where the caller currently has nothing valid to work on?
241 2015-10-26T15:26:05  <gmaxwell> When a new block is accepted by the node the miner process learns this (either via p2p inspection or via longpolling) and calls GBT.  You want the lowest latency response possible; because until it returns the mining gear is working on a fork.
242 2015-10-26T15:26:57  <gmaxwell> Then the mining clients will periodically poll to update the transaction list. In this polling you don't care if the transaction data returned is seconds stale.
243 2015-10-26T15:27:04  <morcos> yes, understood that the new block case needs to be optimized, that was my next step, and that seems pretty easy
244 2015-10-26T15:27:21  <gmaxwell> since all that does is deny you the little incremental fee rate from new transactions that came in.
245 2015-10-26T15:28:12  <gmaxwell> We already do createnewblock caching. but it only helps with the second and later requests.
246 2015-10-26T15:28:18  <morcos> so what i was assuming is that when you periodically poll, which is the common use case, you already have valid work, so the existing longpoll functionality is fine, and there is no need for background process
247 2015-10-26T15:28:28  <morcos> yes right
248 2015-10-26T15:29:30  *** jl2012 has quit IRC
249 2015-10-26T15:29:54  *** jl2012 has joined #bitcoin-core-dev
250 2015-10-26T15:30:10  <morcos> so i think what you're saying is that for instance a new block could come in while you happen to not be longpolling, so then it won't be until you call getblocktemplate that it even tries to do anything, and it woudl be better if it started immediately?
251 2015-10-26T15:30:10  <gmaxwell> Ultimately it would be good to avoid the case where if you poll just before a new block is accepted, you don't delay getting work on the new block. Though, indeed, then I think you run into locking issues with the mempool.
252 2015-10-26T15:31:23  <gmaxwell> morcos: yes, when a new block comes in and you have been mining it should immediately compute a new block template, and until thats done, return empty ones so you'll at least be working on extending the longest valid chain.
253 2015-10-26T15:32:24  <gmaxwell> (a failure to make this optimization is part of what contributes to miners mining on other miners work without validating-- they go and 'optimize' at a layer higher than Bitcoin, and as a resut complex details like validation go out the window. :) )
254 2015-10-26T15:32:51  <morcos> it seems like you'd want to be notified of 2 things, when there is a new block template (empty) and when there is the first template with txs based on the new block.   how do you get that notification if you're not longpolling constantly?
255 2015-10-26T15:36:22  <morcos> i was just going to change it so the longpool returns immediately with an empty block if a new block comes in, via a flag to create new block or something.  and then your next long polling call to getblocktemplate also calls CNB immediately which will return with txs.
256 2015-10-26T15:36:42  <gmaxwell> We could trigger the GBT LP's every N seconds or when the fees change by X, I don't actually know what (if anything) triggers the GBT LP now except a new block.  I also don't know how widely GBT LP is used at the moment:  RPC concurrency has been kinda broken for a long time (fixed in git master), and GBT LP was a late addition, so I think most things decide when to poll GBT based on monitoring
257 2015-10-26T15:36:48  <gmaxwell> P2P and then otherwise run on a timer.
258 2015-10-26T15:37:34  <morcos> ok... well doing it the threaded way shouldn't be hard either...
259 2015-10-26T15:38:03  <gmaxwell> morcos: we do want to get people mining transactions quickly too-- as otherwise you're missing out on fees (and the public tends to get really angry when they're waiting for transactions and an empty block shows up. :) )
260 2015-10-26T15:38:36  <morcos> gmaxwell: yes, will be much much faster, so as long as you poll again immediately, shouldn't be more than 100ms
261 2015-10-26T15:39:37  <morcos> i'll probably have a lot more questions when our mining hardware shows up.. :)
262 2015-10-26T15:43:07  <gmaxwell> There is some ... potential for "can't win" here, so there is some hardware that really dislikes being longpolled often. E.g. for some dumb reason (usually a complete lack of understanding about all the reasons you might LP) and they flush the pipeline completely.  Now, GBT LPing doesn't necessarily mean the hardware gets disrupted, as some higher layer thing can ignore an early long poll and pi
263 2015-10-26T15:43:12  <gmaxwell> ck up the non-empty template on a later timed run.
264 2015-10-26T15:44:37  <gmaxwell> So it might be the case that we expose breakage if we start LPing 100ms after the last time we LPed, just because we now have transactions. But if so, we can probably add a flag to delay that second LP event.
265 2015-10-26T15:44:52  <gmaxwell> (that same hardware is the stuff that cannot be used with P2Pool)
266 2015-10-26T15:52:37  *** danielsocials has quit IRC
267 2015-10-26T16:25:33  <GitHub52> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/ff057f41aa14...c8322ff7f754
268 2015-10-26T16:25:33  <GitHub52> bitcoin/master 143d173 Eric Lombrozo: Use BOOST_CHECK_MESSAGE() rather than BOOST_CHECK() in alerts_tests.cpp and initialize strMiscWarning before calling PartitionCheck()."
269 2015-10-26T16:25:34  <GitHub52> bitcoin/master c8322ff Wladimir J. van der Laan: Merge pull request #6888...
270 2015-10-26T16:25:43  <GitHub96> [bitcoin] laanwj closed pull request #6888: Clear strMiscWarning before running PartitionAlert (master...alert_tests) https://github.com/bitcoin/bitcoin/pull/6888
271 2015-10-26T16:34:07  *** challisto has quit IRC
272 2015-10-26T16:42:19  <GitHub16> [bitcoin] laanwj pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/c8322ff7f754...dbc5ee821ecd
273 2015-10-26T16:42:20  <GitHub16> bitcoin/master d4aa54c Kevin Cooper: added org.bitcoin.bitcoind.plist for launchd (OS X)
274 2015-10-26T16:42:20  <GitHub16> bitcoin/master e04b0b6 Kevin Cooper: added OS X documentation to doc/init.md
275 2015-10-26T16:42:21  <GitHub16> bitcoin/master dbc5ee8 Wladimir J. van der Laan: Merge pull request #6621...
276 2015-10-26T16:42:21  <GitHub118> [bitcoin] laanwj closed pull request #6621: added org.bitcoin.bitcoind.plist for launchd (OS X) (master...master) https://github.com/bitcoin/bitcoin/pull/6621
277 2015-10-26T16:48:13  *** CodeShark has joined #bitcoin-core-dev
278 2015-10-26T16:54:20  <GitHub11> [bitcoin] laanwj pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/dbc5ee821ecd...7939164d8985
279 2015-10-26T16:54:21  <GitHub69> [bitcoin] laanwj closed pull request #6622: Introduce -maxuploadtarget (master...2015/09/maxuploadtarget) https://github.com/bitcoin/bitcoin/pull/6622
280 2015-10-26T16:54:21  <GitHub11> bitcoin/master 872fee3 Jonas Schnelli: Introduce -maxuploadtarget...
281 2015-10-26T16:54:22  <GitHub11> bitcoin/master 17a073a Suhas Daftuar: Add RPC test for -maxuploadtarget
282 2015-10-26T16:54:22  <GitHub11> bitcoin/master 7939164 Wladimir J. van der Laan: Merge pull request #6622...
283 2015-10-26T16:54:36  *** MarcoFalke has joined #bitcoin-core-dev
284 2015-10-26T17:06:19  *** d_t has joined #bitcoin-core-dev
285 2015-10-26T17:13:12  *** BashCo_ has quit IRC
286 2015-10-26T17:14:16  *** instagibbs_ is now known as instagibbs
287 2015-10-26T17:21:23  *** PaulCape_ has quit IRC
288 2015-10-26T17:22:52  *** PaulCapestany has joined #bitcoin-core-dev
289 2015-10-26T17:32:16  *** BashCo has joined #bitcoin-core-dev
290 2015-10-26T17:47:53  *** rubensayshi has quit IRC
291 2015-10-26T18:33:55  <sipa> morcos, sdaftuar, BlueMatt: perhaps we need to talk a bit about what the expectation of the coincache + mempool size limiting is
292 2015-10-26T18:34:44  <sipa> do we want a "all of the mempool's dependencies are guaranteed to always fit in memory" ?
293 2015-10-26T18:36:29  <morcos> sipa: hmm, i don't think i was trying to argue for that.  in fact i don't think it would be at all reasonable to do that.
294 2015-10-26T18:37:04  <morcos> i think we should set the default sizes so that is true in general though
295 2015-10-26T18:37:22  <dgenr8> Was reversing the feerate sort just an expressiveness change, or did it fix something? 78b82f4
296 2015-10-26T18:37:55  <sdaftuar> dgenr8: expressiveness mostly, avoids an issue with using project
297 2015-10-26T18:38:00  <sipa> morcos: one way to accomplish that is to just have a combined coincache+mempool memory limit size, and depending on whether the dependencies are large or small, effectively have a smaller resp. larger mempool limit as a result
298 2015-10-26T18:38:36  <sipa> morcos: but if we don't have that, we _must_ enforce the coincache limit
299 2015-10-26T18:38:38  <dgenr8> sdaftuar: thx!
300 2015-10-26T18:38:41  <sdaftuar> i think that might be a little dangerous though because that has an effect on relay policy
301 2015-10-26T18:39:01  <sipa> morcos: if the coincache exceeds its size, it will be wiped anyway when the next block comes
302 2015-10-26T18:39:33  <sipa> calling flush in the tx code just makes it happen a bit earlier to avoid an OOM
303 2015-10-26T18:39:36  <sdaftuar> sipa: i think we should revisit that behavior.  ideally the coinscache would have in it the set of things likely to be needed when the next block arrives
304 2015-10-26T18:39:40  <morcos> sipa: you realize that we can't always enforce the coincache size?  when we're connecting a block it could temporarily exceed for example
305 2015-10-26T18:40:15  <sipa> morcos: yeah...
306 2015-10-26T18:40:18  <morcos> i agree that we should _eventually_ put in something that enforces limitation of the coincache size between blocks
307 2015-10-26T18:40:26  <morcos> but that might as well wait for something smarter than just flushing
308 2015-10-26T18:40:34  <sipa> i think the only real solution is to switching to a per-txout cache instead of a per-tx cache
309 2015-10-26T18:40:45  <sipa> so the ratio is more or less fixed
310 2015-10-26T18:40:45  <sdaftuar> ah i was wondering how feasible that would be
311 2015-10-26T18:40:54  <morcos> and in the meantime we set the ratio of the defaults to be reasonable so that we don't usually blow through the cache limit
312 2015-10-26T18:41:26  <morcos> given that you have to pay for relay to increase the cache size there is some cost to the attack after all
313 2015-10-26T18:41:47  <morcos> and how big are we worried the cache size could get?
314 2015-10-26T18:41:48  <sipa> define usually
315 2015-10-26T18:42:57  <morcos> i guess i'm thinking that if we set mempool to 300M then it'd be rare than the coinsview cache was over 600M or something
316 2015-10-26T18:43:13  <morcos> thats just a lot bigger than the current default
317 2015-10-26T18:45:26  <morcos> i just think flushing after txs should be rare, if we want to set some higher limit to protect against OOM and flush after txs there, then thats fine, lets just set all these defaults so that what happens usually is the cache is big enough to contain a good chunk of the mempool.  it doesn't really have to contain the whole thing, because
318 2015-10-26T18:45:33  <morcos> you wont' get to 100% mempool size in between blocks
319 2015-10-26T19:08:18  *** jl2012_ has joined #bitcoin-core-dev
320 2015-10-26T19:08:24  *** jl2012 has quit IRC
321 2015-10-26T19:33:22  <phantomcircuit> indeed why are we keeping the already accepted mempool dependencies in the cache? they're already validated isn't that going to reduce the cache hit % ?
322 2015-10-26T19:34:19  <sipa> phantomcircuit: they're validated again when a block is received
323 2015-10-26T19:34:25  <sipa> which we want to make fast
324 2015-10-26T19:35:01  <phantomcircuit> sipa, ah that's right
325 2015-10-26T19:35:05  <sipa> an approach to avoid that is caching the success, but if you store that cache value in the mempool, you're making the mempool code consensus critical
326 2015-10-26T19:35:21  <phantomcircuit> yeah lets not do that :)
327 2015-10-26T19:35:26  <sipa> eh, and that doesn't even help
328 2015-10-26T19:35:39  <sipa> signature validation is already cached anyway
329 2015-10-26T19:35:49  <phantomcircuit> oh that reminds me
330 2015-10-26T19:35:59  <sipa> and you need to be able to update the spends in the chainstate anyway after the block is validated
331 2015-10-26T19:36:08  <phantomcircuit> the size of the sig cache isn't in the help menu
332 2015-10-26T19:36:24  <sipa> good point
333 2015-10-26T19:36:25  <phantomcircuit> it significantly reduces worst case AcceptNewBlock() latency to increase that
334 2015-10-26T19:36:40  <phantomcircuit> miners should basically all have that be 10-100x the default
335 2015-10-26T19:36:51  <gmaxwell> what?
336 2015-10-26T19:37:21  <gmaxwell> the entries are _tiny_, we should be able to have a "big enough for ~100% hitrate" cache for everyone.
337 2015-10-26T19:37:47  <sipa> i actually have no idea what size the sigcache is
338 2015-10-26T19:39:04  <phantomcircuit> huh it is in the help
339 2015-10-26T19:39:13  <gmaxwell> phantomcircuit: for block acceptance speed it may make sense to seperately cache success and failure.
340 2015-10-26T19:39:16  <phantomcircuit> 50000 entries
341 2015-10-26T19:40:48  <phantomcircuit> gmaxwell, yes map<prevblockhash, set<txid> >
342 2015-10-26T19:41:01  <phantomcircuit> should work nicely under non adversarial conditions
343 2015-10-26T19:41:09  <sipa> that means that an entry "3000 ago" (which is around 1000 tx) only has a 96% hitrate
344 2015-10-26T19:41:16  <sipa> eh, 94%
345 2015-10-26T19:41:17  <gmaxwell> phantomcircuit: what, no.
346 2015-10-26T19:41:26  <sipa> it uses random replacement
347 2015-10-26T19:41:46  <phantomcircuit> sipa, yup 10-100x makes a big difference
348 2015-10-26T19:41:52  <gmaxwell> phantomcircuit: transactions can spend unconfirmed outputs my friend.
349 2015-10-26T19:42:14  <phantomcircuit> gmaxwell, yeah you still have to check that the dependencies
350 2015-10-26T19:43:00  <phantomcircuit> that *does* make the mempool view consensus critical though
351 2015-10-26T19:43:01  <gmaxwell> phantomcircuit: if you only mean to check that the ECDSA passes, than <txid> is enough.
352 2015-10-26T19:43:18  <sipa> the sigcache is per signature, not per transaction
353 2015-10-26T19:43:34  <sipa> per transaction is pretty complicated, as you need to take active softforks into account etc
354 2015-10-26T19:44:23  <phantomcircuit> sipa, hadn't considered that
355 2015-10-26T19:44:47  <sipa> there have been patching before that cached full tx validation
356 2015-10-26T19:44:58  <sipa> but actually just script validation caching is cheaper
357 2015-10-26T19:45:02  <sipa> s/cheaper/easier
358 2015-10-26T19:45:54  <phantomcircuit> hmm
359 2015-10-26T19:46:09  <sipa> i can implement that easily
360 2015-10-26T19:46:13  <phantomcircuit> have we considered that upload limiting disconnecting peers will have an impact on the network topology?
361 2015-10-26T19:47:02  <gmaxwell> phantomcircuit: it only disconnects peers that fetch historic blocks currently; largely avoiding the concern.
362 2015-10-26T19:47:48  <phantomcircuit> that seems safe
363 2015-10-26T19:48:23  <gmaxwell> well safer at least. it doesn't completely escape topology impact.
364 2015-10-26T19:48:41  <gmaxwell> Because a new node will, of course, not remain connected to anything over its limit.
365 2015-10-26T19:49:01  <gmaxwell> so even once its synced up it will end up with a different position in the topology than it would otherwise.
366 2015-10-26T19:49:34  <phantomcircuit> export MALLOC_ARENA_MAX=1
367 2015-10-26T19:49:39  <phantomcircuit> that completely fixes the problem
368 2015-10-26T19:49:42  <phantomcircuit> neat
369 2015-10-26T19:50:08  <sipa> IBD's stall detection already causes significant movement in the topology
370 2015-10-26T19:50:12  <sipa> phantomcircuit: ??
371 2015-10-26T19:50:19  <sipa> to reduce fragmentation?
372 2015-10-26T19:50:23  <phantomcircuit> gmaxwell, right which means new nodes will tend to end up connected to nodes that have no upload limit
373 2015-10-26T19:50:38  <phantomcircuit> sipa, yeah, calling getblocktemplate in a loop and memory usage <1GB
374 2015-10-26T19:50:41  <gmaxwell> phantomcircuit: at least during their first runtime, after restart their topo will be more uniform.
375 2015-10-26T19:51:04  <gmaxwell> phantomcircuit: we really need outbound rotation to flatten topo hotspots.
376 2015-10-26T19:51:36  <phantomcircuit> gmaxwell, for that we need "outbound slots" basically
377 2015-10-26T19:52:11  <phantomcircuit> the easiest way to do that is to just assign the second half of the outbound connections (based on vNodes position) as rotating
378 2015-10-26T19:52:33  <phantomcircuit> definitely dont want to rotate all the outbounds :)
379 2015-10-26T19:52:48  <gmaxwell> right, it's a good way to wander into a sybil attack.
380 2015-10-26T19:53:46  <CodeShark> are you guys submitting proposals for hong kong?
381 2015-10-26T19:53:55  <CodeShark> sorry if off-topic
382 2015-10-26T19:54:53  <phantomcircuit> hmm this is actually kind of annoying to test
383 2015-10-26T19:55:03  <phantomcircuit> have to get the mempool full again
384 2015-10-26T19:55:21  <phantomcircuit> sending the "mempool" command to peers when they connect results in comical performance issues
385 2015-10-26T19:55:38  <sipa> phantomcircuit: i run with that for benchmarking mempool behaviour
386 2015-10-26T19:55:57  *** molly has quit IRC
387 2015-10-26T19:56:14  <phantomcircuit> sipa, have you noticed that you just sit there spinning at 100% cpu processing something (i haven't bothered to figur eout what yet)
388 2015-10-26T19:56:48  <sipa> phantomcircuit: turn on debug=mempool and debug=mempoolrej
389 2015-10-26T19:57:15  <phantomcircuit> i run this node with debug=
390 2015-10-26T20:04:54  <GitHub138> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/7939164d8985...2b625510d374
391 2015-10-26T20:04:55  <GitHub138> bitcoin/master 7bbc7c3 Suhas Daftuar: Add option for microsecond precision in debug.log
392 2015-10-26T20:04:55  <GitHub138> bitcoin/master 2b62551 Wladimir J. van der Laan: Merge pull request #6881...
393 2015-10-26T20:05:03  <GitHub108> [bitcoin] laanwj closed pull request #6881: Debug: Add option for microsecond precision in debug.log (master...add-microsecond-timestamps) https://github.com/bitcoin/bitcoin/pull/6881
394 2015-10-26T20:08:14  <phantomcircuit> BlueMatt, plz2rebase seed
395 2015-10-26T20:14:08  <phantomcircuit> sipa, non-scientific gdb interrupt+ bt says it's spinning on ECDSA_verify
396 2015-10-26T20:14:14  <phantomcircuit> which i guess is to be expected :)
397 2015-10-26T20:16:28  *** Thireus has quit IRC
398 2015-10-26T20:16:39  <wumpus> is it catching up or in initial sync? if so, that's expected
399 2015-10-26T20:16:51  <wumpus> if in steady state it's not normal
400 2015-10-26T20:17:22  <sipa> wumpus: this is with sending out a BIP35 "mempool" command, so at startup it receives zillions of transactions
401 2015-10-26T20:18:00  <wumpus> ok...
402 2015-10-26T20:18:29  <wumpus> yes, then it makes sense too
403 2015-10-26T20:18:38  <sipa> so i do expect you'd be accepting mempool txn at the speed you can handle
404 2015-10-26T20:26:33  <phantomcircuit> wumpus, http://0bin.net/paste/8349G1adfXBgLthd#l6rCtj99ayGFnNdx6-C+bbP1fYFF0EJIvg4qvV4JnD9
405 2015-10-26T20:32:28  <wumpus> you do have commit 5ce43da03dff3ee655949cd53d952dace555e268?
406 2015-10-26T20:33:39  <phantomcircuit> wumpus, yes
407 2015-10-26T20:33:44  <wumpus> if so, this shouldn't happen - unless gdb happens to reenable the signal
408 2015-10-26T20:34:42  <phantomcircuit> it shouldn't
409 2015-10-26T20:34:46  <wumpus> yeah it does, it reports all signals by default. Try: "handle SIGPIPE nostop noprint pass"
410 2015-10-26T20:34:49  <phantomcircuit> but who knows... gmaxwell do you know :)
411 2015-10-26T20:35:12  <phantomcircuit> wumpus, sigh
412 2015-10-26T20:35:17  <phantomcircuit> ok yes that's good
413 2015-10-26T20:36:14  <phantomcircuit> bleh why does it do that?
414 2015-10-26T20:57:29  <phantomcircuit> wumpus, can verify that MALLOC_ARENA_MAX=1 works btw
415 2015-10-26T20:57:52  <sipa> phantomcircuit: what about 2 instead of 1?
416 2015-10-26T20:58:19  <phantomcircuit> sipa, not sure it takes ages to load enough transactions into the mempool for the result to be valid
417 2015-10-26T20:58:21  <wumpus> phantomcircuit: awesome
418 2015-10-26T20:59:20  <sipa> too bad we can't set MALLOC_ARENA_MAX from the program itself, i suppose?
419 2015-10-26T20:59:53  <phantomcircuit> sipa, well we can but...
420 2015-10-26T21:00:00  <sipa> how so?
421 2015-10-26T21:00:18  <wumpus> I think you can
422 2015-10-26T21:00:32  <sipa> we can change the env variable, but i expect that the arenas are created at library load time
423 2015-10-26T21:00:36  <sipa> before we can change it
424 2015-10-26T21:00:54  <phantomcircuit> sipa, the easiest would be to set the env and execve
425 2015-10-26T21:01:03  <phantomcircuit> ie h4x
426 2015-10-26T21:01:04  <wumpus> let me see glibc source...
427 2015-10-26T21:03:11  <wumpus> it indeed makes no sense to set the env var, as it parses it at startup as expected, however you should be able to do mallopt(M_ARENA_MAX, 1)
428 2015-10-26T21:05:50  <wumpus>  /* mallopt options that actually do something */ in /usr/include/malloc.h :-)
429 2015-10-26T21:06:39  <sipa> man mallopt does not list M_ARENA_MAX here
430 2015-10-26T21:07:16  <wumpus> it's an undocumented option
431 2015-10-26T21:08:59  <wumpus> probably should call it before any call to malloc to be effective
432 2015-10-26T21:09:07  <wumpus> or at least before calling it in a thread
433 2015-10-26T21:09:20  <sipa> the execve approach works...
434 2015-10-26T21:09:49  <sipa> it's ugly because it needs to know the name of the binary you just started
435 2015-10-26T21:09:51  <wumpus> that's really ugly, and makes debugging harder
436 2015-10-26T21:10:18  <sipa> agree
437 2015-10-26T21:10:22  <phantomcircuit> ha it's not even in the libc manual
438 2015-10-26T21:10:33  <wumpus> and that (though you should always have that in argv[0] on unix)
439 2015-10-26T21:10:56  <phantomcircuit> it looks like we could set M_MMAP_THRESHOLD which would be slower but seems to guarantee memory is actually free'd
440 2015-10-26T21:11:09  *** ParadoxSpiral has quit IRC
441 2015-10-26T21:12:28  <wumpus> but gdb will show something like "process 28686 is executing new program: /bin/dash" when a program execve's, and it loses all its state
442 2015-10-26T21:13:31  <phantomcircuit> wumpus, yeah i know thus "h4x"
443 2015-10-26T21:13:40  <phantomcircuit> it would probably also break anybody using start-stop-daemon
444 2015-10-26T21:13:52  <phantomcircuit> gtg
445 2015-10-26T21:14:20  <wumpus> but the mallopt should work too (can't check it right now though)
446 2015-10-26T21:14:29  <wumpus> ok later
447 2015-10-26T21:16:18  *** randy-waterhouse has joined #bitcoin-core-dev
448 2015-10-26T21:28:44  *** treehug88 has quit IRC
449 2015-10-26T21:48:13  *** deepcore has joined #bitcoin-core-dev
450 2015-10-26T22:01:54  *** molly has joined #bitcoin-core-dev
451 2015-10-26T22:12:55  *** MarcoFalke has quit IRC
452 2015-10-26T22:20:48  *** d_t has quit IRC
453 2015-10-26T22:21:33  *** d_t has joined #bitcoin-core-dev
454 2015-10-26T22:31:08  *** belcher has joined #bitcoin-core-dev
455 2015-10-26T23:06:07  <phantomcircuit> 41k transaction in the mempool and counting
456 2015-10-26T23:07:06  <sipa> yeah, 50k sigcache entries won't be good in that case...
457 2015-10-26T23:07:53  <phantomcircuit> yeah i changed mine to maxsigcachesize=1000000
458 2015-10-26T23:13:15  <gmaxwell> sipa: sigcache is currently uint256,vector<char>,cpubkey.... bloat bloat bloat.
459 2015-10-26T23:14:02  <gmaxwell> Could be compressed with H(the above).
460 2015-10-26T23:14:31  <gmaxwell> H() need not even be very collission resistant if there is a per node secret nonce.
461 2015-10-26T23:21:44  <phantomcircuit> gmaxwell, yeah that's what i was thinking
462 2015-10-26T23:21:58  <sipa> it's around 220 bytes per entry now
463 2015-10-26T23:23:15  <phantomcircuit> sipa, depends on the size of the pubkey script right?
464 2015-10-26T23:23:28  <phantomcircuit> it would be nice if it didn't
465 2015-10-26T23:23:59  <phantomcircuit> upto 44.5k now
466 2015-10-26T23:24:10  <sipa> phantomcircuit: it's a pubkey; not a script
467 2015-10-26T23:24:15  <sipa> pubkeys are 65 bytes
468 2015-10-26T23:24:17  <sipa> in memory
469 2015-10-26T23:24:20  <sipa> + alignment
470 2015-10-26T23:24:27  <phantomcircuit> oh
471 2015-10-26T23:26:03  <gmaxwell> sipa: plus whatever overhead there is from the std::set.
472 2015-10-26T23:26:31  <sipa> gmaxwell: around 40 bytes, i included that
473 2015-10-26T23:38:49  *** randy-waterhouse has quit IRC
474 2015-10-26T23:39:10  <gmaxwell> sipa: would it be really ugly if accepting the block carried a flag to the checker that told it to remove the entry?  with that alone the hitrate would no longer be low absent attacks because the cache would not be full all the time.
475 2015-10-26T23:52:15  <morcos> sipa: phantomcircuit: i don't think we need to worry about M_ARENA_MAX.  we can drastically reduce the size of the problem.  first we rewrite CreateNewBlock to only pull into the cache 1 blocks worth of txs.  second we make it run in only 1 thread.
476 2015-10-26T23:53:17  <morcos> but yeah that maxsigcachesize will bite you.  i couldn't figure out why my new code was slower and it was that its sigcache was getting blown away