1 2015-10-20T00:15:26  *** molly has joined #bitcoin-core-dev
  2 2015-10-20T00:18:19  *** moli has quit IRC
  3 2015-10-20T00:28:04  <Luke-Jr> How important is memory comparison for #6851 ? I get a std::bad_alloc with my test wallet, so I'll need to find something less spammy and start over :/
  4 2015-10-20T00:34:34  <gmaxwell> Luke-Jr: how much memory does the host you're testing on have?
  5 2015-10-20T00:35:28  <Luke-Jr> gmaxwell: 16 GB, but 32-bit can only alloc <4 GB
  6 2015-10-20T00:36:13  <Luke-Jr> I'm not surprised that a wallet with dice addresses overflows it.
  7 2015-10-20T00:37:47  <gmaxwell> Luke-Jr: did you try turning down caching? ... we're already pretty close to the wire address space wise on 4gb to begin with.
  8 2015-10-20T00:38:48  <Luke-Jr> I didn't. But IIRC dice uses >4 GB of transactions in the blockchain, so really I don't expect anything to help except remaking the wallet with a lower volume address
  9 2015-10-20T00:45:16  *** challisto has joined #bitcoin-core-dev
 10 2015-10-20T00:55:15  *** Jaamg has quit IRC
 11 2015-10-20T01:17:20  *** belcher has quit IRC
 12 2015-10-20T03:01:11  *** d_t has quit IRC
 13 2015-10-20T03:34:59  *** baldur has quit IRC
 14 2015-10-20T03:48:11  *** baldur has joined #bitcoin-core-dev
 15 2015-10-20T04:56:04  <Luke-Jr> ok, so I did the memory comparison on Eligius's mining key
 16 2015-10-20T04:56:14  <Luke-Jr> = 10184 wtxns
 17 2015-10-20T04:57:13  <Luke-Jr> ps showed lower memory usage (in all three of SIZE, VSZ, and SZ) for the optimised/cached bitcoind. Obviously it couldn't have been actually reduced, so I conclude it's unmeasurably small.
 18 2015-10-20T05:04:59  *** phantomcircuit has quit IRC
 19 2015-10-20T05:10:27  <gmaxwell> 10000 isn't that many. :(
 20 2015-10-20T05:12:26  <Luke-Jr> gmaxwell: not enough to at least show a measurable difference if it were a problematic increase?
 21 2015-10-20T05:12:37  * Luke-Jr ponders how he has no block sources with 10 connections :/
 22 2015-10-20T05:14:30  <Luke-Jr> cute, my next block is inflight from an XT node which is apparently not sending it?
 23 2015-10-20T05:20:50  *** Thireus has joined #bitcoin-core-dev
 24 2015-10-20T05:27:41  *** ParadoxSpiral has joined #bitcoin-core-dev
 25 2015-10-20T05:36:32  *** ParadoxSpiral has quit IRC
 26 2015-10-20T06:18:25  *** Ylbam has joined #bitcoin-core-dev
 27 2015-10-20T06:27:41  <wumpus> curious
 28 2015-10-20T06:36:54  *** Thireus has quit IRC
 29 2015-10-20T06:48:12  <wumpus> how openbsd, even with the recent release, still manages to package a BDB that is *older* than the 4.8 required to build bitcoin core
 30 2015-10-20T06:55:39  <Luke-Jr> lol
 31 2015-10-20T06:56:44  <midnightmagic> netcraft confirms it..?
 32 2015-10-20T06:59:42  * Luke-Jr ponders if it would be difficult to gitian-build bitcoind for Android <.<
 33 2015-10-20T07:02:36  <wumpus> not difficult, probably, a matter of pointing the depends at the NDK toolchain, then the type of slog work that cross-building usually is
 34 2015-10-20T07:02:49  <wumpus> I vaguely remember cfields_ has done it at some point
 35 2015-10-20T07:03:24  <wumpus> midnightmagic: netcraft?
 36 2015-10-20T07:06:54  <midnightmagic> wumpus: Ancient meme. BSD is dying, netcraft confirms it. Used to be the repeated mantra on Slashdot, turn of the century.
 37 2015-10-20T07:07:00  <Luke-Jr> http://bsd.slashdot.org/comments.pl?sid=228247&cid=18495137
 38 2015-10-20T07:07:21  *** Thireus has joined #bitcoin-core-dev
 39 2015-10-20T07:09:08  <wumpus> midnightmagic: oh, okay :-) BSD's 'linux desktop' meme
 40 2015-10-20T07:11:49  <wumpus> but it's refreshing for a change compared to the people that complain that 4.8 is too old
 41 2015-10-20T07:33:41  <midnightmagic> good god, I'm so old, smart people don't know what I'm muttering anymore
 42 2015-10-20T07:35:28  <wumpus> heh I know the feeling
 43 2015-10-20T08:14:18  *** baldur has quit IRC
 44 2015-10-20T08:27:43  *** baldur has joined #bitcoin-core-dev
 45 2015-10-20T09:00:35  <btcdrak> Mempool PR https://github.com/bitcoin/bitcoin/pull/6722 is looking good. lots of ACKs now.
 46 2015-10-20T09:06:53  *** rubensayshi has joined #bitcoin-core-dev
 47 2015-10-20T09:20:53  *** randy-waterhouse has joined #bitcoin-core-dev
 48 2015-10-20T09:49:13  <GitHub6> [bitcoin] laanwj opened pull request #6859: http: Restrict maximum size of http + headers (master...2015_10_max_http_headers) https://github.com/bitcoin/bitcoin/pull/6859
 49 2015-10-20T10:04:53  *** dcousens has joined #bitcoin-core-dev
 50 2015-10-20T10:07:10  <GitHub168> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/da7d57fb9501...488f8517a154
 51 2015-10-20T10:07:11  <GitHub168> bitcoin/master 53b86d0 Daniel Kraft: doc: add comment explaining initial header request...
 52 2015-10-20T10:07:11  <GitHub168> bitcoin/master 488f851 Wladimir J. van der Laan: Merge pull request #6829...
 53 2015-10-20T10:07:15  <GitHub46> [bitcoin] laanwj closed pull request #6829: doc: add comment explaining initial header request (master...doc-getheaders) https://github.com/bitcoin/bitcoin/pull/6829
 54 2015-10-20T10:08:00  <GitHub62> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/488f8517a154...c834f568693b
 55 2015-10-20T10:08:01  <GitHub62> bitcoin/master 7801f43 Eric Lombrozo: Added fPowNoRetargeting field to Consensus::Params that disables nBits recalculation.
 56 2015-10-20T10:08:01  <GitHub62> bitcoin/master c834f56 Wladimir J. van der Laan: Merge pull request #6853...
 57 2015-10-20T10:08:08  <GitHub42> [bitcoin] laanwj closed pull request #6853: Added fPowNoRetargeting field to Consensus::Params (master...fNoRetargeting) https://github.com/bitcoin/bitcoin/pull/6853
 58 2015-10-20T10:21:55  <GitHub88> [bitcoin] laanwj pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/c834f568693b...87e5539e9c50
 59 2015-10-20T10:21:56  <GitHub88> bitcoin/master 0d8b175 MarcoFalke: [rpc-tests] fundrawtransaction: Update fee after minRelayTxFee increase
 60 2015-10-20T10:21:56  <GitHub88> bitcoin/master bd4c22e MarcoFalke: [rpc-tests] Check return code
 61 2015-10-20T10:21:57  <GitHub88> bitcoin/master 87e5539 Wladimir J. van der Laan: Merge pull request #6827...
 62 2015-10-20T10:22:01  <GitHub132> [bitcoin] laanwj closed pull request #6828: [rpc-tests] fundrawtransaction: Update fee after minRelayTxFee increase (master...MarcoFalke-2015-fundrawtransactionTestFix) https://github.com/bitcoin/bitcoin/pull/6828
 63 2015-10-20T10:22:01  <GitHub93> [bitcoin] laanwj closed pull request #6827: [rpc-tests] Check return code (master...MarcoFalke-2015-rpcTestsReturnCode) https://github.com/bitcoin/bitcoin/pull/6827
 64 2015-10-20T10:36:14  <GitHub8> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/87e5539e9c50...ae69a75c554f
 65 2015-10-20T10:36:15  <GitHub8> bitcoin/master e76d9e4 fanquake: [depends] Latest config.guess and config.sub
 66 2015-10-20T10:36:15  <GitHub8> bitcoin/master ae69a75 Wladimir J. van der Laan: Merge pull request #6801...
 67 2015-10-20T10:36:22  <GitHub91> [bitcoin] laanwj closed pull request #6801: [depends] Latest config.guess and config.sub (master...update-config-guess-sub) https://github.com/bitcoin/bitcoin/pull/6801
 68 2015-10-20T10:46:32  *** randy-waterhouse has quit IRC
 69 2015-10-20T10:54:04  <GitHub27> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/ae69a75c554f...020c4073a03a
 70 2015-10-20T10:54:04  <GitHub27> bitcoin/master b6d5e32 Alex Morcos: Make fee aware of min relay in pruning.py RPC test
 71 2015-10-20T10:54:05  <GitHub27> bitcoin/master 020c407 Wladimir J. van der Laan: Merge pull request #6841...
 72 2015-10-20T10:54:14  <GitHub193> [bitcoin] laanwj closed pull request #6841: Increase fee in pruning.py RPC test (master...fixPruningRPC) https://github.com/bitcoin/bitcoin/pull/6841
 73 2015-10-20T11:30:14  *** crescendo has quit IRC
 74 2015-10-20T11:36:23  <GitHub43> [bitcoin] laanwj pushed 1 new commit to 0.11: https://github.com/bitcoin/bitcoin/commit/072032448be5263f58cbd6b47f61edc7bb8210e1
 75 2015-10-20T11:36:23  <GitHub43> bitcoin/0.11 0720324 Alex Morcos: Make fee aware of min relay in pruning.py RPC test...
 76 2015-10-20T11:50:21  <GitHub65> [bitcoin] MarcoFalke opened pull request #6860: Drop minRelayTxFee to 1000 (master...MarcoFalke-2015-minRelayTxFeeDrop) https://github.com/bitcoin/bitcoin/pull/6860
 77 2015-10-20T12:00:15  *** randy-waterhouse has joined #bitcoin-core-dev
 78 2015-10-20T12:08:01  *** ParadoxSpiral has joined #bitcoin-core-dev
 79 2015-10-20T12:25:57  <GitHub23> [bitcoin] laanwj closed pull request #6860: Drop minRelayTxFee to 1000 (master...MarcoFalke-2015-minRelayTxFeeDrop) https://github.com/bitcoin/bitcoin/pull/6860
 80 2015-10-20T12:58:24  <wumpus> jgarzik: you can't be SERIOUS that truncating a 64 bit number to a 32 bit number was a good idea!??!  https://github.com/bitcoin/bitcoin/issues/6765#event-440267158
 81 2015-10-20T12:59:02  <jgarzik> wumpus, You are confusing an issue with a PR
 82 2015-10-20T12:59:03  <wumpus> this is completely crazy
 83 2015-10-20T12:59:38  <jgarzik> wumpus, calm down please.  The issue is univalve conversion cause unintended behavior changes that are as yet not documented or audited
 84 2015-10-20T12:59:57  <wumpus> it is simply not a bug
 85 2015-10-20T13:00:00  <jgarzik> wumpus, objecting to a solution that was not proposed as a PR is fine
 86 2015-10-20T13:00:07  <wumpus> EWONTFIX
 87 2015-10-20T13:00:25  <jgarzik> wumpus, it is a user noticeable behavior change that potentially breaks people in the field
 88 2015-10-20T13:00:38  <jgarzik> wumpus, who knows what else breakage was caused, until analysis is complete
 89 2015-10-20T13:00:39  <wumpus> if you want to change documentation, be my guest, I don't think there is any documentation that says yo ucan provide numbers there that exceed 32 bit numeric range
 90 2015-10-20T13:00:45  <wumpus> well at most you'll get an error now
 91 2015-10-20T13:00:51  <wumpus> instead of dumbly ANDing off the upper bits
 92 2015-10-20T13:01:02  <wumpus> which can cause much worse problems, ask the openssl devs
 93 2015-10-20T13:01:17  <jgarzik> wumpus, not disagreeing, please read what I wrote
 94 2015-10-20T13:01:35  <jgarzik> wumpus, user breakage - breakage was noticed - let's be methodical
 95 2015-10-20T13:01:38  <wumpus> well, do you intend on doing analysys thee?
 96 2015-10-20T13:01:46  <wumpus> if not, I want to close the issue and move on
 97 2015-10-20T13:02:06  <jgarzik> wumpus, yes the issue should stay open until this is analyzed
 98 2015-10-20T13:02:20  <wumpus> I disagree that it is so important
 99 2015-10-20T13:02:50  <jgarzik> wumpus, without analysis it is provably impossible to say what is the impact
100 2015-10-20T13:02:58  <wumpus> but anyhow, if you look into this this week, I'm fine with keeping it open
101 2015-10-20T13:04:38  <wumpus> I don't think you'll find anyone that thinks the new behavior doesn't make sense
102 2015-10-20T13:05:22  <wumpus> I can ask a few people if you think jonasschnelli, dcousens and mine opinion isn't enough
103 2015-10-20T13:06:39  <wumpus> arguably if you were using large values in the field it was already broken. You *thought* you were parsing a large value, but you could have been passing a negative value for all you know
104 2015-10-20T13:06:41  <jonasschnelli> i think the out of range exception behavior is okay and expected
105 2015-10-20T13:08:14  <jonasschnelli> But we need to write some univalue release notes anyways and could mention this there
106 2015-10-20T13:09:40  <wumpus> 1000000000000 turns into -727379968, for example
107 2015-10-20T13:10:19  <jgarzik> wumpus, Please consider the wider picture.  You are focusing on one impacted callsite - and I consistently agree with you that the new behavior at that callsite is better - the issue is that that is not the only .get_int() in the entire codebase.  There may be other user visible breakage where the impact is not so fortunate.  Do not blindly assume.
108 2015-10-20T13:10:32  <wumpus> on other  callsites the breakage would be exactly the same
109 2015-10-20T13:10:41  <jgarzik> you hope and assume...
110 2015-10-20T13:10:44  <wumpus> not, I'm sure
111 2015-10-20T13:10:51  <wumpus> get_int() simply does a range check now
112 2015-10-20T13:10:53  <wumpus> that is safer
113 2015-10-20T13:11:15  <wumpus> if you  have one example where this causes dangerous behavior, just say so
114 2015-10-20T13:11:24  <wumpus> if not, please stop this nonsense
115 2015-10-20T13:11:24  <jonasschnelli> release notes -> mention integers will now face a range check ->  done?
116 2015-10-20T13:11:38  <jgarzik> the new behavior is throwing an exception versus returning a range bound value without an exception
117 2015-10-20T13:11:54  <wumpus> the old behavior was dangerous, not the newo ne
118 2015-10-20T13:12:05  <jgarzik> throwing an exception causes different error return messages and different downstream behaviors
119 2015-10-20T13:12:11  <jgarzik> which in turn cause user visible interface changes
120 2015-10-20T13:12:25  <jonasschnelli> jgarzik: i agree: it's and API change. But in my opinion one that is worth to take.
121 2015-10-20T13:12:30  <wumpus> but ony if the input *was already invalid*
122 2015-10-20T13:12:36  <jgarzik> jonasschnelli, yes
123 2015-10-20T13:12:56  <jgarzik> jonasschnelli, and my point has always been it needs to be analyzed and documented
124 2015-10-20T13:12:58  <wumpus> at least now you are told that the input is invalid, instead of replacing it with an effectively random number
125 2015-10-20T13:13:03  <jgarzik> not simply make the assumption it is OK
126 2015-10-20T13:13:12  <jgarzik> that is extremely poor software engineering - hope and pray
127 2015-10-20T13:13:32  <wumpus> well then do something better
128 2015-10-20T13:13:38  <jgarzik> keeping an issue open until it is analyzed fully is not a huge burden
129 2015-10-20T13:13:50  <wumpus> opening lots of issues that no one ever looks at is just doesn't cut it either
130 2015-10-20T13:14:01  <wumpus> it is a burdern, though
131 2015-10-20T13:14:09  <jonasschnelli> Mention it in the release notes is okay, but not mandatory IMO. If a API consumer was relying range bound value he needs to be slapped anyways. :)
132 2015-10-20T13:14:27  <jgarzik> don't make mountains out of molehills.
133 2015-10-20T13:14:35  <wumpus> well possibly they didn't even know the range was not 64 bits. This bug could even be exploitable in some cases
134 2015-10-20T13:14:46  <wumpus> at least now they aretold
135 2015-10-20T13:15:07  <jgarzik> the issue is assigned to me now.  please stop hyperventilating over a minor issue.
136 2015-10-20T13:15:26  <wumpus> there could be cases where *application side* range checks can be bypassed by providing large numbers, unaware that bitcoind cuts them off
137 2015-10-20T13:36:26  *** treehug88 has joined #bitcoin-core-dev
138 2015-10-20T13:45:36  *** helo_ is now known as helo
139 2015-10-20T13:52:10  *** randy-waterhouse has quit IRC
140 2015-10-20T13:58:18  <GitHub186> [bitcoin] laanwj closed pull request #6855: [REST] API versioning (master...restVersioning) https://github.com/bitcoin/bitcoin/pull/6855
141 2015-10-20T14:11:09  *** treehug88 has quit IRC
142 2015-10-20T14:15:22  *** treehug88 has joined #bitcoin-core-dev
143 2015-10-20T14:20:36  <morcos> wumpus: I'm a bit out of my depth in the memory "leak" related to getblocktemplate,  but i have an idea on the cause
144 2015-10-20T14:20:59  <morcos> in CreateNewBlock, we have a vecPriority that is approximately equal to the size of the mempool.
145 2015-10-20T14:21:15  <morcos> the first time you call getblocktemplate your resident usage jumps by the size of your mempool
146 2015-10-20T14:21:35  <morcos> repeated calls to getblocktemplate will continue to cause that to happen, up to the number of rpc threads you are running
147 2015-10-20T14:22:11  <morcos> this is still mostly guesswork at this point, but its not clear to me why that memory used by vecPriority appears to be sticking around (a copy per thread)
148 2015-10-20T14:29:15  *** Thireus has quit IRC
149 2015-10-20T14:30:39  <wumpus> morcos: strange; I don't think we use any thread-local storage at all
150 2015-10-20T14:31:22  <morcos> well, like i said, still guess work, could be wrong.
151 2015-10-20T14:31:59  <morcos> but vecPriority is a local variable, so that would be thread-local storage right, it just should go away when the function exits
152 2015-10-20T14:32:21  <morcos> but maybe its some stl "optimization" to keep it allocated or something
153 2015-10-20T14:33:58  <jgarzik> In some thread systems (not sure about RPC threads), the threads do not die - they are recycled
154 2015-10-20T14:34:29  <jgarzik> so a huge stack would remain
155 2015-10-20T14:34:29  <morcos> but vecPriority is local to the function CreateNewBlock
156 2015-10-20T14:34:44  <jgarzik> indeed, that should go out of scope
157 2015-10-20T14:35:02  <jgarzik> as should CCoinsViewCache
158 2015-10-20T14:36:30  <wumpus> the threads certainly stay around
159 2015-10-20T14:36:54  <jgarzik> also in terms of allocator behavior, large amounts of memory are not necessarily reclaimed at the process level.  In the C library (not sure about C++), lots of free(3) memory is recycled back into allocator, not returned to system via munmap(2)
160 2015-10-20T14:36:59  <wumpus> but the stack can't grow that far, a vector is allocated on the heap not the stack
161 2015-10-20T14:37:36  <wumpus> (also pthreads default stack size is 8mb per thread; hardly possible to cause so much 'leaking' that way)
162 2015-10-20T14:37:54  <jgarzik> So even though the app has released memory to the allocator, the allocator will not necessarily release memory to the system, as shown in top(1)
163 2015-10-20T14:38:03  <wumpus> right, freed memory is not returned to the system immediately
164 2015-10-20T14:38:30  <wumpus> sometimes never, depending on the size of the allocation
165 2015-10-20T14:38:40  <jgarzik> Process virtual memory size therefore just sits at its maximum, even if unused by app
166 2015-10-20T14:38:47  <jgarzik> *sometimes
167 2015-10-20T14:39:02  <jgarzik> (recent libraries do release memory if chunks are large enough)
168 2015-10-20T14:39:05  <wumpus> (but it's reused if the application allocates again, so it should not result in growing process size over time in itself)
169 2015-10-20T14:39:15  <morcos> we're talking about resident memory, not virtual memory, but you're saying it could still reflect there
170 2015-10-20T14:39:16  <jgarzik> nod
171 2015-10-20T14:39:57  <wumpus> do multiple getblocktemplates run at the same time indifferent threads? that'd explain it
172 2015-10-20T14:39:59  <morcos> wumpus: that's why i put "leak" in quotes, it seems entirely possible to me that the memory usage is limited to an excess of #rpcthreads * maxseenmempoolsize
173 2015-10-20T14:40:02  <jgarzik> per-thread heaps are certainly possible in C++
174 2015-10-20T14:40:11  <jgarzik> does memory exceed RPC thread count
175 2015-10-20T14:40:24  <morcos> wumpus: i was running them serially
176 2015-10-20T14:40:34  <jgarzik> seems like the steady state could be reached once all RPC threads are "huge" :)
177 2015-10-20T14:40:36  <wumpus> morcos: ok
178 2015-10-20T14:41:06  <morcos> yeah, so there might be a steady state, but the steady state is particularly inefficient if you have a lot of RPC threads
179 2015-10-20T14:41:10  <wumpus>  the behavior seems consistent with per-thread heap, yes
180 2015-10-20T14:41:52  <wumpus> default rpcthreads is only 4, there shouldn't usually be a reason to increase that
181 2015-10-20T14:42:15  <wumpus> (especially as everything holds the cs_main lock anyway, so there is only very little scope for paralellism)
182 2015-10-20T14:42:20  <morcos> yes but 4 * max_mempool_usage is a lot of extra memory right?
183 2015-10-20T14:42:48  <jgarzik> not if you have a studly machine
184 2015-10-20T14:42:50  * jgarzik runs
185 2015-10-20T14:43:02  <wumpus> would be interesting to use google heap profiler to see if the memory gets released in between the calls
186 2015-10-20T14:43:38  <wumpus> if it doesn't show up in there, it's the matter of memory not being released to the OS
187 2015-10-20T14:44:26  <jgarzik> Amusingly I bet you could actually go faster with fork(2) and a lockless COW mempool copy that disappears upon child exit(2).    (unrealistic due to portability... not a real proposal)
188 2015-10-20T14:45:23  <jgarzik> faster and less memory bloat :)
189 2015-10-20T14:45:35  <wumpus> (using heap profiler is easy as   LD_PRELOAD="/.../libtcmalloc.so"  src/bitcoind , see https://gperftools.googlecode.com/svn/trunk/doc/heapprofile.html   It can also track mmaps but I had less success using it for that)
190 2015-10-20T14:46:53  <wumpus> well if that means forking a child for every http connection, I don't think it'd be faster, but yeah you could use the processes a few times and then have them exit...
191 2015-10-20T14:49:15  <wumpus> I'm fairly sure apache does as much
192 2015-10-20T14:49:24  <jgarzik> nah just fork off for that one RPC call, and have the thread wait for results
193 2015-10-20T14:50:00  <jgarzik> when you're making a complete copy of the process you can do stuff like that :)
194 2015-10-20T14:50:28  <jgarzik> but doesn't work at all on Windows so...
195 2015-10-20T14:50:47  <jgarzik> on Windows, you have to fake it by executing your own .exe again
196 2015-10-20T14:51:32  <jgarzik> And correct - Apache re-uses both processes and threads for a limited time, then closes then - for reasons precisely like the ones we are seeing
197 2015-10-20T14:51:39  <jgarzik> limited recycle prevents memory from building up
198 2015-10-20T14:51:50  <jgarzik> *closes them
199 2015-10-20T14:51:56  *** rubensayshi has quit IRC
200 2015-10-20T14:53:59  <wumpus> morcos: can you try export MALLOC_ARENA_MAX=1 , this is supposed to disable per-thread arenas for newer glibc
201 2015-10-20T14:54:24  <morcos> sure i will try
202 2015-10-20T14:54:57  <wumpus> (from  https://www.ibm.com/developerworks/community/blogs/kevgrig/entry/linux_glibc_2_10_rhel_6_malloc_may_show_excessive_virtual_memory_usage?lang=en  they see similar excessive allocation behavior)
203 2015-10-20T14:57:09  <wumpus> by default it creates an arena for every CPU core
204 2015-10-20T15:01:34  <jgarzik> yep - that's so thread local structures can run lock free simply by accessing arena_vector[my_cpu_core_number]
205 2015-10-20T15:02:12  <wumpus> from a performance point of view it makes a lot of sense
206 2015-10-20T15:02:52  <wumpus> at least when there is actual paralellism :)
207 2015-10-20T15:04:16  <jgarzik> The cs_main lock has amusing historical parallels.  both freebsd and linux kernels had a Big Kernel Lock - a single lock that nearly everything would grab - during the early days of their conversion to SMP/parallel kernels
208 2015-10-20T15:04:44  <wumpus> yes and don't forget python's big lock :)
209 2015-10-20T15:04:52  <jgarzik> that too
210 2015-10-20T15:05:10  <jgarzik> python is so bad at threading you -have- to fork ;p
211 2015-10-20T15:06:07  <wumpus> but I hope bitcoin's is more like the BSD/Linux kernel lock than the python one, at least the former got rid of it at some point
212 2015-10-20T15:06:46  <GitHub73> [bitcoin] laanwj pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/020c4073a03a...e26a3f6713f8
213 2015-10-20T15:06:47  <GitHub73> bitcoin/master f3525e2 Jorge Timón: Chainparams: Replace CBaseChainParams::Network enum with string constants (suggested by Wladimir)
214 2015-10-20T15:06:47  <GitHub73> bitcoin/master 55a8975 Jorge Timón: Chainparams: Translations: DRY: options and error strings...
215 2015-10-20T15:06:48  <GitHub73> bitcoin/master e26a3f6 Wladimir J. van der Laan: Merge pull request #6235...
216 2015-10-20T15:06:52  <GitHub151> [bitcoin] laanwj closed pull request #6235: Chainparams: Translations: DRY: options and error strings (master...chainparams-dry) https://github.com/bitcoin/bitcoin/pull/6235
217 2015-10-20T15:07:07  <wumpus> with python it's such an essential part of the design that it's unrealistic to get rid of it. At least that was in the 2.x days, i don't know about 3.x
218 2015-10-20T15:09:39  <GitHub87> [bitcoin] laanwj pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/e26a3f6713f8...c6de5cc88614
219 2015-10-20T15:09:40  <GitHub87> bitcoin/master e253e83 Matt Corallo: Update debian/changelog and slight tweak to debian/control
220 2015-10-20T15:09:40  <GitHub87> bitcoin/master c7b36cc Matt Corallo: Change URLs to https in debian/control
221 2015-10-20T15:09:41  <GitHub87> bitcoin/master c6de5cc Wladimir J. van der Laan: Merge pull request #6796...
222 2015-10-20T15:09:45  <GitHub160> [bitcoin] laanwj closed pull request #6796: Update debian/changelog and slight tweak to debian/control (master...debian) https://github.com/bitcoin/bitcoin/pull/6796
223 2015-10-20T15:10:22  <jgarzik> yep
224 2015-10-20T15:10:41  <jgarzik> and we're well on our way to peeling off the cs_main (bsd/linux road)
225 2015-10-20T15:10:49  <jgarzik> imo
226 2015-10-20T15:31:47  <morcos> wumpus: sorry its taking me a while, i was trying to get a full mempool but not also spam the rest of the network...  i haven't run the getblock templates yet, but it sure does make bitciond use a lot less memory in the first place (using MALLOC_ARENA_MAX=1)
227 2015-10-20T15:32:19  <morcos> normally i'm using close to 1G shortly after starting, with this, it was about 250M
228 2015-10-20T15:38:31  *** paveljanik has joined #bitcoin-core-dev
229 2015-10-20T15:46:41  <wumpus> morcos: good to know! thanks for testing, I always had this issue where only very little memory usage would show up in the heap profiler, but top showed much more, now we understand why!
230 2015-10-20T15:48:27  <wumpus> and the more cores, the more it will use
231 2015-10-20T15:48:53  <wumpus> should add that one to the use-less-memory tips
232 2015-10-20T15:49:49  <morcos> wumpus: yep, after repeated calls to getblocktemplate, it only jumped in memory usage once
233 2015-10-20T15:50:37  <morcos> i think concentrating on vecPriority was misguided (its not actually that big, just ptrs to txs) but between all the stuff thats stl-allocated in CNB, that probably explains the big jump.  i havent' gone through really to see if it adds up yet
234 2015-10-20T15:50:43  <morcos> it does still seem like a big jump.
235 2015-10-20T15:50:57  <morcos> whether it happens 1x or #rpcthreads times
236 2015-10-20T15:51:37  <morcos> i suppose it might depend on how much the CCoinsViewCache allocates?
237 2015-10-20T15:52:49  <wumpus> that depends; they will cache everything that is queried through them
238 2015-10-20T15:55:45  <jgarzik> wumpus, any remaining merge blockers for #6722 in your view?
239 2015-10-20T15:58:20  <morcos> +1 for merging it sooner rather than later. only reason not to merge i think is if we can coerce more people into review/test.  but beyond that, lets get it out into the wild to find more bugs.
240 2015-10-20T15:58:42  <morcos> however its worth considering merging the lower chain limits soon as well
241 2015-10-20T15:58:50  <jgarzik> yeah this sort of stuff is perfect for master
242 2015-10-20T16:00:56  <GitHub85> [bitcoin] jtimon reopened pull request #6625: BLOCKING: Consensus: Move blocksize and related parameters to consensusparams ...without removing consensus/consensus.h [#6526 alternative] (master...consensus-blocksize-0.12.99) https://github.com/bitcoin/bitcoin/pull/6625
243 2015-10-20T16:01:03  <jgarzik> the Internet is always a better test lab than anything we cook up in private
244 2015-10-20T16:01:10  <jgarzik> merge easy, revert easy
245 2015-10-20T16:01:59  <jgarzik> I would have merged it myself if it were a normal change, given the ACK quotient, but for a change of this notability I want wumpus, gmaxwell, sipa etc. approval of some sort (sipa has already ACK'd FTR)
246 2015-10-20T16:03:30  <wumpus> voila https://gist.github.com/laanwj/efe29c7661ce9b6620a7#linux-specific
247 2015-10-20T16:03:35  <wumpus> jgarzik: will  take a loook
248 2015-10-20T16:03:47  <jgarzik> I've been testing it as well
249 2015-10-20T16:04:11  <jgarzik> wumpus, +1 gist
250 2015-10-20T16:06:07  <GitHub107> [bitcoin] jtimon closed pull request #6672: Consensus: Separate most consensus functions to consensus.cpp (master...consensus-moveonly-0.12.99) https://github.com/bitcoin/bitcoin/pull/6672
251 2015-10-20T16:10:25  *** randy-waterhouse has joined #bitcoin-core-dev
252 2015-10-20T16:10:53  <morcos> wumpus: do you think thats a sufficient "fix" for the memory usage.
253 2015-10-20T16:11:16  <morcos> still not clear exactly how much heap allocation is happening in getblocktemplate
254 2015-10-20T16:11:31  <morcos> but lets say for the sake of argument its on the order of your mempool usage size
255 2015-10-20T16:11:57  <morcos> now you're saying every node out there (that doesn't run with your low mem tip) is going to use 5x maxmempoolsize
256 2015-10-20T16:12:01  <morcos> at least
257 2015-10-20T16:12:21  <morcos> i guess only if they run getblocktemplate
258 2015-10-20T16:13:03  <morcos> but maybe we can do something to try to reduce that.  we know for instance that there is no need for this particular memory usage to worry about trying to allocate at the same time
259 2015-10-20T16:13:50  <morcos> if that was even a static variable that was just cleared between calls, it would persist the memory all the time, but only 1x right?
260 2015-10-20T16:14:43  <morcos> so maybe vecPriority and view and whatevers using memory in createnewblock
261 2015-10-20T16:18:40  <GitHub92> [bitcoin] sdaftuar closed pull request #6557: Mempool limiting with descendant package tracking (master...mempool-packages) https://github.com/bitcoin/bitcoin/pull/6557
262 2015-10-20T16:19:23  <gmaxwell> memory usage I'm seeing is certantly beyond 5x.
263 2015-10-20T16:20:05  <gmaxwell> 3.77GB res compared to   "usage": 90103744
264 2015-10-20T16:26:21  *** Thireus has joined #bitcoin-core-dev
265 2015-10-20T16:29:58  <morcos> gmaxwell: i think it turns out this arena per core thing will affect a lot of bitcoind mem usage...   also, i think its the max ever mempool usage that matters, not current
266 2015-10-20T16:31:38  <gmaxwell> ah! reading more backscroll, so the idea is that the pre-thread (or whatever) arena is ending up allocating but not sbrking data in each thread (or core?) that runs a GBT and thus has a high peak usage because it's copied the mempool?
267 2015-10-20T16:33:03  <gmaxwell> Having control of stuff like this is why firefox uses jemalloc
268 2015-10-20T16:53:27  <morcos> gmaxwell: yeah to the limit of my understanding.  except still not exactly clear what is taking up the memory, b/c the mempool isn't copied.  but could be the CoinsViewCache.
269 2015-10-20T17:26:25  *** CodeShark has joined #bitcoin-core-dev
270 2015-10-20T17:29:56  <wumpus> coinsviewcache has methods to measure the size
271 2015-10-20T17:38:39  <gmaxwell> so, reading the page about https://www.ibm.com/developerworks/community/blogs/kevgrig/entry/linux_glibc_2_10_rhel_6_malloc_may_show_excessive_virtual_memory_usage?lang=en .. it looks like thats about about virtual usage, not RES.
272 2015-10-20T17:39:47  <gmaxwell> (not that it wouldn't also be good to reduce virtual usage, at least on 32bit.. but I'm seeing about 4x resident than what I expect)
273 2015-10-20T17:41:52  *** treehug88 has quit IRC
274 2015-10-20T17:45:58  <morcos> gmaxwell: yeah that confused me as well.  but from my very ad hoc testing it seemed to affect RES
275 2015-10-20T17:51:51  <wumpus> gmaxwell: well the memory is at least temporary resident when it is used
276 2015-10-20T17:53:17  <wumpus> it's not just reserving an arena, it's actually returning memory from it, to be used and later freed
277 2015-10-20T17:53:48  <wumpus> after which it sticks around, no longer touched so it doesn't have to stay resident, but that depends on mem pressure
278 2015-10-20T17:54:22  <wumpus> (also if you keep sending getblocktemplate it probably reuses them in more-or-less round-robin fashion)
279 2015-10-20T17:56:49  <gmaxwell> oh i bet that interacts super well with people who've cranked rpc threads to get around the issues with rpcblocking.
280 2015-10-20T18:04:28  <wumpus> nah the number of threads is not that relevant, just the number of cores
281 2015-10-20T18:05:07  <wumpus> it tries to balance arenas over cores
282 2015-10-20T18:05:25  <wumpus> (which makes a lot of sense for performance, but in our case it's not useful)
283 2015-10-20T18:05:46  *** Thireus has quit IRC
284 2015-10-20T18:06:33  <gmaxwell> I wonder if its ever useful for large allocations. hm. then again the usage is probably made out of zillions of tiny allocations.
285 2015-10-20T18:07:56  <wumpus> we had a workstation with a crazy NUMA motherboard at uni, where each processor had its own memory. I guess that's the only case where it makes sense for large, infrequent allocations.
286 2015-10-20T18:08:16  <wumpus> (but yes in bitcoin we use tons of small allocations)
287 2015-10-20T18:16:32  *** treehug88 has joined #bitcoin-core-dev
288 2015-10-20T18:26:39  *** Thireus has joined #bitcoin-core-dev
289 2015-10-20T18:29:56  *** treehug88 has quit IRC
290 2015-10-20T18:32:43  *** wangchun has joined #bitcoin-core-dev
291 2015-10-20T18:34:34  *** treehug88 has joined #bitcoin-core-dev
292 2015-10-20T18:43:46  <CodeShark> wumpus: it seems test_bitcoin behavior has changed recently. not too long ago, if I ran src/test/test_bitcoin --run_tests=<testname> from the root project directory, it would only run that specific test - but now it seems to run all of them
293 2015-10-20T18:44:08  <CodeShark> am I tripping?
294 2015-10-20T18:52:55  <wumpus> --run-tests= takes a test suite name, not an individual test name
295 2015-10-20T18:53:03  <CodeShark> I mean singular
296 2015-10-20T18:53:05  <CodeShark> --run_test=
297 2015-10-20T18:53:15  <wumpus> if the test suite doesn't exist, it somehow gets them all
298 2015-10-20T18:53:20  <wumpus> that's not possible
299 2015-10-20T18:53:32  <CodeShark> let me verify what I'm saying to make sure it's accurate...
300 2015-10-20T18:53:46  <wumpus> yes it's singular run-test, but it takes a suite name
301 2015-10-20T18:54:31  <CodeShark> if the test suite doesn't exist I get an error
302 2015-10-20T18:54:33  * wumpus test_bitcoin -l test_suite   will show exactly what test suites and tests it's executing, and how long they take
303 2015-10-20T18:55:47  <wumpus> that behavior is part of boost test, so if it changed, you probably changed your boost version
304 2015-10-20T18:56:37  <wumpus> -run-tests= isn't picked up at all here, -t is
305 2015-10-20T18:57:20  <CodeShark> travis is apparently not liking alert_tests on windows https://travis-ci.org/bitcoin/bitcoin/jobs/86404141
306 2015-10-20T18:58:14  <wumpus> seems to be passing fine here?
307 2015-10-20T18:58:49  <CodeShark> I don't get that error at all - I didn't touch alert_tests
308 2015-10-20T18:58:57  <CodeShark> I mean, I don't understand that error
309 2015-10-20T19:01:49  *** belcher has joined #bitcoin-core-dev
310 2015-10-20T19:02:09  <CodeShark> how can I try to reproduce that error locally?
311 2015-10-20T19:02:14  <wumpus> I had a quite crazy test error in OpenBSD with Alert_tests today: http://s28.postimg.org/duxcxvrh9/Schermafdruk_van_2015_10_20_10_08_52.png  (no, probably not related, something is very wrong there)
312 2015-10-20T19:02:43  <wumpus> you can reproduce it in travis?
313 2015-10-20T19:03:30  <CodeShark> I'm not very familiar with travis - can I get it to run the test again remotely?
314 2015-10-20T19:03:42  <CodeShark> do I need to force push?
315 2015-10-20T19:03:57  <wumpus> I can do that, let me see
316 2015-10-20T19:08:21  <wumpus> should be running again now
317 2015-10-20T19:12:28  <CodeShark> thanks, wumpus - I tried doing a force push anyhow
318 2015-10-20T19:12:52  <wumpus> oh then we triggered it twice, that probably just slows it down
319 2015-10-20T19:14:35  <jgarzik> RE malloc arenas,
320 2015-10-20T19:14:36  <jgarzik> http://journal.siddhesh.in/posts/malloc-per-thread-arenas-in-glibc.html
321 2015-10-20T19:14:50  <jgarzik> (apologies if dup; didn't see it in scrollback)
322 2015-10-20T19:16:17  * jgarzik looks for some way to cap the number of arenas besides MALLOC_ARENA_MAX
323 2015-10-20T19:27:39  <wumpus> don't think I saw that one yet
324 2015-10-20T19:28:17  <Luke-Jr> heh, when I wanted to run a specific test, I edited the makefile to not include the rest <.<
325 2015-10-20T19:29:10  *** treehug88 has quit IRC
326 2015-10-20T19:29:42  <wumpus> "Freed blocks near the end of heaps in an arena are given back to the system either by using madvise(MADV_DONTNEED) or by explicitly unmapping." that doesn't seem to happen for us, probably due to heap fragmentation
327 2015-10-20T19:41:26  <jgarzik> nod - lots of little allocations will do that :/
328 2015-10-20T20:02:01  <gmaxwell> Change to jemalloc which should also reduce fragmentation.
329 2015-10-20T20:02:03  <gmaxwell> :)
330 2015-10-20T20:10:59  *** CodeShark has quit IRC
331 2015-10-20T20:11:27  *** phantomcircuit has joined #bitcoin-core-dev
332 2015-10-20T20:19:28  *** paveljanik has quit IRC
333 2015-10-20T20:41:17  *** ParadoxSpiral has quit IRC
334 2015-10-20T20:46:59  *** dcousens has quit IRC
335 2015-10-20T20:47:22  *** belcher has quit IRC
336 2015-10-20T21:06:27  *** CodeShark has joined #bitcoin-core-dev
337 2015-10-20T21:16:09  *** randy-waterhouse has quit IRC
338 2015-10-20T23:42:04  *** CodeShark has quit IRC