12015-10-20T00:15:26 *** molly has joined #bitcoin-core-dev
22015-10-20T00:18:19 *** moli has quit IRC
32015-10-20T00:28:04 <Luke-Jr> How important is memory comparison for #6851 ? I get a std::bad_alloc with my test wallet, so I'll need to find something less spammy and start over :/
42015-10-20T00:34:34 <gmaxwell> Luke-Jr: how much memory does the host you're testing on have?
52015-10-20T00:35:28 <Luke-Jr> gmaxwell: 16 GB, but 32-bit can only alloc <4 GB
62015-10-20T00:36:13 <Luke-Jr> I'm not surprised that a wallet with dice addresses overflows it.
72015-10-20T00:37:47 <gmaxwell> Luke-Jr: did you try turning down caching? ... we're already pretty close to the wire address space wise on 4gb to begin with.
82015-10-20T00:38:48 <Luke-Jr> I didn't. But IIRC dice uses >4 GB of transactions in the blockchain, so really I don't expect anything to help except remaking the wallet with a lower volume address
92015-10-20T00:45:16 *** challisto has joined #bitcoin-core-dev
102015-10-20T00:55:15 *** Jaamg has quit IRC
112015-10-20T01:17:20 *** belcher has quit IRC
122015-10-20T03:01:11 *** d_t has quit IRC
132015-10-20T03:34:59 *** baldur has quit IRC
142015-10-20T03:48:11 *** baldur has joined #bitcoin-core-dev
152015-10-20T04:56:04 <Luke-Jr> ok, so I did the memory comparison on Eligius's mining key
162015-10-20T04:56:14 <Luke-Jr> = 10184 wtxns
172015-10-20T04:57:13 <Luke-Jr> ps showed lower memory usage (in all three of SIZE, VSZ, and SZ) for the optimised/cached bitcoind. Obviously it couldn't have been actually reduced, so I conclude it's unmeasurably small.
182015-10-20T05:04:59 *** phantomcircuit has quit IRC
192015-10-20T05:10:27 <gmaxwell> 10000 isn't that many. :(
202015-10-20T05:12:26 <Luke-Jr> gmaxwell: not enough to at least show a measurable difference if it were a problematic increase?
212015-10-20T05:12:37 * Luke-Jr ponders how he has no block sources with 10 connections :/
222015-10-20T05:14:30 <Luke-Jr> cute, my next block is inflight from an XT node which is apparently not sending it?
232015-10-20T05:20:50 *** Thireus has joined #bitcoin-core-dev
242015-10-20T05:27:41 *** ParadoxSpiral has joined #bitcoin-core-dev
252015-10-20T05:36:32 *** ParadoxSpiral has quit IRC
262015-10-20T06:18:25 *** Ylbam has joined #bitcoin-core-dev
272015-10-20T06:27:41 <wumpus> curious
282015-10-20T06:36:54 *** Thireus has quit IRC
292015-10-20T06:48:12 <wumpus> how openbsd, even with the recent release, still manages to package a BDB that is *older* than the 4.8 required to build bitcoin core
302015-10-20T06:55:39 <Luke-Jr> lol
312015-10-20T06:56:44 <midnightmagic> netcraft confirms it..?
322015-10-20T06:59:42 * Luke-Jr ponders if it would be difficult to gitian-build bitcoind for Android <.<
332015-10-20T07:02:36 <wumpus> not difficult, probably, a matter of pointing the depends at the NDK toolchain, then the type of slog work that cross-building usually is
342015-10-20T07:02:49 <wumpus> I vaguely remember cfields_ has done it at some point
352015-10-20T07:03:24 <wumpus> midnightmagic: netcraft?
362015-10-20T07:06:54 <midnightmagic> wumpus: Ancient meme. BSD is dying, netcraft confirms it. Used to be the repeated mantra on Slashdot, turn of the century.
372015-10-20T07:07:00 <Luke-Jr> http://bsd.slashdot.org/comments.pl?sid=228247&cid=18495137
382015-10-20T07:07:21 *** Thireus has joined #bitcoin-core-dev
392015-10-20T07:09:08 <wumpus> midnightmagic: oh, okay :-) BSD's 'linux desktop' meme
402015-10-20T07:11:49 <wumpus> but it's refreshing for a change compared to the people that complain that 4.8 is too old
412015-10-20T07:33:41 <midnightmagic> good god, I'm so old, smart people don't know what I'm muttering anymore
422015-10-20T07:35:28 <wumpus> heh I know the feeling
432015-10-20T08:14:18 *** baldur has quit IRC
442015-10-20T08:27:43 *** baldur has joined #bitcoin-core-dev
452015-10-20T09:00:35 <btcdrak> Mempool PR https://github.com/bitcoin/bitcoin/pull/6722 is looking good. lots of ACKs now.
462015-10-20T09:06:53 *** rubensayshi has joined #bitcoin-core-dev
472015-10-20T09:20:53 *** randy-waterhouse has joined #bitcoin-core-dev
482015-10-20T09:49:13 <GitHub6> [bitcoin] laanwj opened pull request #6859: http: Restrict maximum size of http + headers (master...2015_10_max_http_headers) https://github.com/bitcoin/bitcoin/pull/6859
492015-10-20T10:04:53 *** dcousens has joined #bitcoin-core-dev
502015-10-20T10:07:10 <GitHub168> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/da7d57fb9501...488f8517a154
512015-10-20T10:07:11 <GitHub168> bitcoin/master 53b86d0 Daniel Kraft: doc: add comment explaining initial header request...
522015-10-20T10:07:11 <GitHub168> bitcoin/master 488f851 Wladimir J. van der Laan: Merge pull request #6829...
532015-10-20T10:07:15 <GitHub46> [bitcoin] laanwj closed pull request #6829: doc: add comment explaining initial header request (master...doc-getheaders) https://github.com/bitcoin/bitcoin/pull/6829
542015-10-20T10:08:00 <GitHub62> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/488f8517a154...c834f568693b
552015-10-20T10:08:01 <GitHub62> bitcoin/master 7801f43 Eric Lombrozo: Added fPowNoRetargeting field to Consensus::Params that disables nBits recalculation.
562015-10-20T10:08:01 <GitHub62> bitcoin/master c834f56 Wladimir J. van der Laan: Merge pull request #6853...
572015-10-20T10:08:08 <GitHub42> [bitcoin] laanwj closed pull request #6853: Added fPowNoRetargeting field to Consensus::Params (master...fNoRetargeting) https://github.com/bitcoin/bitcoin/pull/6853
582015-10-20T10:21:55 <GitHub88> [bitcoin] laanwj pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/c834f568693b...87e5539e9c50
592015-10-20T10:21:56 <GitHub88> bitcoin/master 0d8b175 MarcoFalke: [rpc-tests] fundrawtransaction: Update fee after minRelayTxFee increase
602015-10-20T10:21:56 <GitHub88> bitcoin/master bd4c22e MarcoFalke: [rpc-tests] Check return code
612015-10-20T10:21:57 <GitHub88> bitcoin/master 87e5539 Wladimir J. van der Laan: Merge pull request #6827...
622015-10-20T10:22:01 <GitHub132> [bitcoin] laanwj closed pull request #6828: [rpc-tests] fundrawtransaction: Update fee after minRelayTxFee increase (master...MarcoFalke-2015-fundrawtransactionTestFix) https://github.com/bitcoin/bitcoin/pull/6828
632015-10-20T10:22:01 <GitHub93> [bitcoin] laanwj closed pull request #6827: [rpc-tests] Check return code (master...MarcoFalke-2015-rpcTestsReturnCode) https://github.com/bitcoin/bitcoin/pull/6827
642015-10-20T10:36:14 <GitHub8> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/87e5539e9c50...ae69a75c554f
652015-10-20T10:36:15 <GitHub8> bitcoin/master e76d9e4 fanquake: [depends] Latest config.guess and config.sub
662015-10-20T10:36:15 <GitHub8> bitcoin/master ae69a75 Wladimir J. van der Laan: Merge pull request #6801...
672015-10-20T10:36:22 <GitHub91> [bitcoin] laanwj closed pull request #6801: [depends] Latest config.guess and config.sub (master...update-config-guess-sub) https://github.com/bitcoin/bitcoin/pull/6801
682015-10-20T10:46:32 *** randy-waterhouse has quit IRC
692015-10-20T10:54:04 <GitHub27> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/ae69a75c554f...020c4073a03a
702015-10-20T10:54:04 <GitHub27> bitcoin/master b6d5e32 Alex Morcos: Make fee aware of min relay in pruning.py RPC test
712015-10-20T10:54:05 <GitHub27> bitcoin/master 020c407 Wladimir J. van der Laan: Merge pull request #6841...
722015-10-20T10:54:14 <GitHub193> [bitcoin] laanwj closed pull request #6841: Increase fee in pruning.py RPC test (master...fixPruningRPC) https://github.com/bitcoin/bitcoin/pull/6841
732015-10-20T11:30:14 *** crescendo has quit IRC
742015-10-20T11:36:23 <GitHub43> [bitcoin] laanwj pushed 1 new commit to 0.11: https://github.com/bitcoin/bitcoin/commit/072032448be5263f58cbd6b47f61edc7bb8210e1
752015-10-20T11:36:23 <GitHub43> bitcoin/0.11 0720324 Alex Morcos: Make fee aware of min relay in pruning.py RPC test...
762015-10-20T11:50:21 <GitHub65> [bitcoin] MarcoFalke opened pull request #6860: Drop minRelayTxFee to 1000 (master...MarcoFalke-2015-minRelayTxFeeDrop) https://github.com/bitcoin/bitcoin/pull/6860
772015-10-20T12:00:15 *** randy-waterhouse has joined #bitcoin-core-dev
782015-10-20T12:08:01 *** ParadoxSpiral has joined #bitcoin-core-dev
792015-10-20T12:25:57 <GitHub23> [bitcoin] laanwj closed pull request #6860: Drop minRelayTxFee to 1000 (master...MarcoFalke-2015-minRelayTxFeeDrop) https://github.com/bitcoin/bitcoin/pull/6860
802015-10-20T12:58:24 <wumpus> jgarzik: you can't be SERIOUS that truncating a 64 bit number to a 32 bit number was a good idea!??! https://github.com/bitcoin/bitcoin/issues/6765#event-440267158
812015-10-20T12:59:02 <jgarzik> wumpus, You are confusing an issue with a PR
822015-10-20T12:59:03 <wumpus> this is completely crazy
832015-10-20T12:59:38 <jgarzik> wumpus, calm down please. The issue is univalve conversion cause unintended behavior changes that are as yet not documented or audited
842015-10-20T12:59:57 <wumpus> it is simply not a bug
852015-10-20T13:00:00 <jgarzik> wumpus, objecting to a solution that was not proposed as a PR is fine
862015-10-20T13:00:07 <wumpus> EWONTFIX
872015-10-20T13:00:25 <jgarzik> wumpus, it is a user noticeable behavior change that potentially breaks people in the field
882015-10-20T13:00:38 <jgarzik> wumpus, who knows what else breakage was caused, until analysis is complete
892015-10-20T13:00:39 <wumpus> if you want to change documentation, be my guest, I don't think there is any documentation that says yo ucan provide numbers there that exceed 32 bit numeric range
902015-10-20T13:00:45 <wumpus> well at most you'll get an error now
912015-10-20T13:00:51 <wumpus> instead of dumbly ANDing off the upper bits
922015-10-20T13:01:02 <wumpus> which can cause much worse problems, ask the openssl devs
932015-10-20T13:01:17 <jgarzik> wumpus, not disagreeing, please read what I wrote
942015-10-20T13:01:35 <jgarzik> wumpus, user breakage - breakage was noticed - let's be methodical
952015-10-20T13:01:38 <wumpus> well, do you intend on doing analysys thee?
962015-10-20T13:01:46 <wumpus> if not, I want to close the issue and move on
972015-10-20T13:02:06 <jgarzik> wumpus, yes the issue should stay open until this is analyzed
982015-10-20T13:02:20 <wumpus> I disagree that it is so important
992015-10-20T13:02:50 <jgarzik> wumpus, without analysis it is provably impossible to say what is the impact
1002015-10-20T13:02:58 <wumpus> but anyhow, if you look into this this week, I'm fine with keeping it open
1012015-10-20T13:04:38 <wumpus> I don't think you'll find anyone that thinks the new behavior doesn't make sense
1022015-10-20T13:05:22 <wumpus> I can ask a few people if you think jonasschnelli, dcousens and mine opinion isn't enough
1032015-10-20T13:06:39 <wumpus> arguably if you were using large values in the field it was already broken. You *thought* you were parsing a large value, but you could have been passing a negative value for all you know
1042015-10-20T13:06:41 <jonasschnelli> i think the out of range exception behavior is okay and expected
1052015-10-20T13:08:14 <jonasschnelli> But we need to write some univalue release notes anyways and could mention this there
1062015-10-20T13:09:40 <wumpus> 1000000000000 turns into -727379968, for example
1072015-10-20T13:10:19 <jgarzik> wumpus, Please consider the wider picture. You are focusing on one impacted callsite - and I consistently agree with you that the new behavior at that callsite is better - the issue is that that is not the only .get_int() in the entire codebase. There may be other user visible breakage where the impact is not so fortunate. Do not blindly assume.
1082015-10-20T13:10:32 <wumpus> on other callsites the breakage would be exactly the same
1092015-10-20T13:10:41 <jgarzik> you hope and assume...
1102015-10-20T13:10:44 <wumpus> not, I'm sure
1112015-10-20T13:10:51 <wumpus> get_int() simply does a range check now
1122015-10-20T13:10:53 <wumpus> that is safer
1132015-10-20T13:11:15 <wumpus> if you have one example where this causes dangerous behavior, just say so
1142015-10-20T13:11:24 <wumpus> if not, please stop this nonsense
1152015-10-20T13:11:24 <jonasschnelli> release notes -> mention integers will now face a range check -> done?
1162015-10-20T13:11:38 <jgarzik> the new behavior is throwing an exception versus returning a range bound value without an exception
1172015-10-20T13:11:54 <wumpus> the old behavior was dangerous, not the newo ne
1182015-10-20T13:12:05 <jgarzik> throwing an exception causes different error return messages and different downstream behaviors
1192015-10-20T13:12:11 <jgarzik> which in turn cause user visible interface changes
1202015-10-20T13:12:25 <jonasschnelli> jgarzik: i agree: it's and API change. But in my opinion one that is worth to take.
1212015-10-20T13:12:30 <wumpus> but ony if the input *was already invalid*
1222015-10-20T13:12:36 <jgarzik> jonasschnelli, yes
1232015-10-20T13:12:56 <jgarzik> jonasschnelli, and my point has always been it needs to be analyzed and documented
1242015-10-20T13:12:58 <wumpus> at least now you are told that the input is invalid, instead of replacing it with an effectively random number
1252015-10-20T13:13:03 <jgarzik> not simply make the assumption it is OK
1262015-10-20T13:13:12 <jgarzik> that is extremely poor software engineering - hope and pray
1272015-10-20T13:13:32 <wumpus> well then do something better
1282015-10-20T13:13:38 <jgarzik> keeping an issue open until it is analyzed fully is not a huge burden
1292015-10-20T13:13:50 <wumpus> opening lots of issues that no one ever looks at is just doesn't cut it either
1302015-10-20T13:14:01 <wumpus> it is a burdern, though
1312015-10-20T13:14:09 <jonasschnelli> Mention it in the release notes is okay, but not mandatory IMO. If a API consumer was relying range bound value he needs to be slapped anyways. :)
1322015-10-20T13:14:27 <jgarzik> don't make mountains out of molehills.
1332015-10-20T13:14:35 <wumpus> well possibly they didn't even know the range was not 64 bits. This bug could even be exploitable in some cases
1342015-10-20T13:14:46 <wumpus> at least now they aretold
1352015-10-20T13:15:07 <jgarzik> the issue is assigned to me now. please stop hyperventilating over a minor issue.
1362015-10-20T13:15:26 <wumpus> there could be cases where *application side* range checks can be bypassed by providing large numbers, unaware that bitcoind cuts them off
1372015-10-20T13:36:26 *** treehug88 has joined #bitcoin-core-dev
1382015-10-20T13:45:36 *** helo_ is now known as helo
1392015-10-20T13:52:10 *** randy-waterhouse has quit IRC
1402015-10-20T13:58:18 <GitHub186> [bitcoin] laanwj closed pull request #6855: [REST] API versioning (master...restVersioning) https://github.com/bitcoin/bitcoin/pull/6855
1412015-10-20T14:11:09 *** treehug88 has quit IRC
1422015-10-20T14:15:22 *** treehug88 has joined #bitcoin-core-dev
1432015-10-20T14:20:36 <morcos> wumpus: I'm a bit out of my depth in the memory "leak" related to getblocktemplate, but i have an idea on the cause
1442015-10-20T14:20:59 <morcos> in CreateNewBlock, we have a vecPriority that is approximately equal to the size of the mempool.
1452015-10-20T14:21:15 <morcos> the first time you call getblocktemplate your resident usage jumps by the size of your mempool
1462015-10-20T14:21:35 <morcos> repeated calls to getblocktemplate will continue to cause that to happen, up to the number of rpc threads you are running
1472015-10-20T14:22:11 <morcos> this is still mostly guesswork at this point, but its not clear to me why that memory used by vecPriority appears to be sticking around (a copy per thread)
1482015-10-20T14:29:15 *** Thireus has quit IRC
1492015-10-20T14:30:39 <wumpus> morcos: strange; I don't think we use any thread-local storage at all
1502015-10-20T14:31:22 <morcos> well, like i said, still guess work, could be wrong.
1512015-10-20T14:31:59 <morcos> but vecPriority is a local variable, so that would be thread-local storage right, it just should go away when the function exits
1522015-10-20T14:32:21 <morcos> but maybe its some stl "optimization" to keep it allocated or something
1532015-10-20T14:33:58 <jgarzik> In some thread systems (not sure about RPC threads), the threads do not die - they are recycled
1542015-10-20T14:34:29 <jgarzik> so a huge stack would remain
1552015-10-20T14:34:29 <morcos> but vecPriority is local to the function CreateNewBlock
1562015-10-20T14:34:44 <jgarzik> indeed, that should go out of scope
1572015-10-20T14:35:02 <jgarzik> as should CCoinsViewCache
1582015-10-20T14:36:30 <wumpus> the threads certainly stay around
1592015-10-20T14:36:54 <jgarzik> also in terms of allocator behavior, large amounts of memory are not necessarily reclaimed at the process level. In the C library (not sure about C++), lots of free(3) memory is recycled back into allocator, not returned to system via munmap(2)
1602015-10-20T14:36:59 <wumpus> but the stack can't grow that far, a vector is allocated on the heap not the stack
1612015-10-20T14:37:36 <wumpus> (also pthreads default stack size is 8mb per thread; hardly possible to cause so much 'leaking' that way)
1622015-10-20T14:37:54 <jgarzik> So even though the app has released memory to the allocator, the allocator will not necessarily release memory to the system, as shown in top(1)
1632015-10-20T14:38:03 <wumpus> right, freed memory is not returned to the system immediately
1642015-10-20T14:38:30 <wumpus> sometimes never, depending on the size of the allocation
1652015-10-20T14:38:40 <jgarzik> Process virtual memory size therefore just sits at its maximum, even if unused by app
1662015-10-20T14:38:47 <jgarzik> *sometimes
1672015-10-20T14:39:02 <jgarzik> (recent libraries do release memory if chunks are large enough)
1682015-10-20T14:39:05 <wumpus> (but it's reused if the application allocates again, so it should not result in growing process size over time in itself)
1692015-10-20T14:39:15 <morcos> we're talking about resident memory, not virtual memory, but you're saying it could still reflect there
1702015-10-20T14:39:16 <jgarzik> nod
1712015-10-20T14:39:57 <wumpus> do multiple getblocktemplates run at the same time indifferent threads? that'd explain it
1722015-10-20T14:39:59 <morcos> wumpus: that's why i put "leak" in quotes, it seems entirely possible to me that the memory usage is limited to an excess of #rpcthreads * maxseenmempoolsize
1732015-10-20T14:40:02 <jgarzik> per-thread heaps are certainly possible in C++
1742015-10-20T14:40:11 <jgarzik> does memory exceed RPC thread count
1752015-10-20T14:40:24 <morcos> wumpus: i was running them serially
1762015-10-20T14:40:34 <jgarzik> seems like the steady state could be reached once all RPC threads are "huge" :)
1772015-10-20T14:40:36 <wumpus> morcos: ok
1782015-10-20T14:41:06 <morcos> yeah, so there might be a steady state, but the steady state is particularly inefficient if you have a lot of RPC threads
1792015-10-20T14:41:10 <wumpus> the behavior seems consistent with per-thread heap, yes
1802015-10-20T14:41:52 <wumpus> default rpcthreads is only 4, there shouldn't usually be a reason to increase that
1812015-10-20T14:42:15 <wumpus> (especially as everything holds the cs_main lock anyway, so there is only very little scope for paralellism)
1822015-10-20T14:42:20 <morcos> yes but 4 * max_mempool_usage is a lot of extra memory right?
1832015-10-20T14:42:48 <jgarzik> not if you have a studly machine
1842015-10-20T14:42:50 * jgarzik runs
1852015-10-20T14:43:02 <wumpus> would be interesting to use google heap profiler to see if the memory gets released in between the calls
1862015-10-20T14:43:38 <wumpus> if it doesn't show up in there, it's the matter of memory not being released to the OS
1872015-10-20T14:44:26 <jgarzik> Amusingly I bet you could actually go faster with fork(2) and a lockless COW mempool copy that disappears upon child exit(2). (unrealistic due to portability... not a real proposal)
1882015-10-20T14:45:23 <jgarzik> faster and less memory bloat :)
1892015-10-20T14:45:35 <wumpus> (using heap profiler is easy as LD_PRELOAD="/.../libtcmalloc.so" src/bitcoind , see https://gperftools.googlecode.com/svn/trunk/doc/heapprofile.html It can also track mmaps but I had less success using it for that)
1902015-10-20T14:46:53 <wumpus> well if that means forking a child for every http connection, I don't think it'd be faster, but yeah you could use the processes a few times and then have them exit...
1912015-10-20T14:49:15 <wumpus> I'm fairly sure apache does as much
1922015-10-20T14:49:24 <jgarzik> nah just fork off for that one RPC call, and have the thread wait for results
1932015-10-20T14:50:00 <jgarzik> when you're making a complete copy of the process you can do stuff like that :)
1942015-10-20T14:50:28 <jgarzik> but doesn't work at all on Windows so...
1952015-10-20T14:50:47 <jgarzik> on Windows, you have to fake it by executing your own .exe again
1962015-10-20T14:51:32 <jgarzik> And correct - Apache re-uses both processes and threads for a limited time, then closes then - for reasons precisely like the ones we are seeing
1972015-10-20T14:51:39 <jgarzik> limited recycle prevents memory from building up
1982015-10-20T14:51:50 <jgarzik> *closes them
1992015-10-20T14:51:56 *** rubensayshi has quit IRC
2002015-10-20T14:53:59 <wumpus> morcos: can you try export MALLOC_ARENA_MAX=1 , this is supposed to disable per-thread arenas for newer glibc
2012015-10-20T14:54:24 <morcos> sure i will try
2022015-10-20T14:54:57 <wumpus> (from https://www.ibm.com/developerworks/community/blogs/kevgrig/entry/linux_glibc_2_10_rhel_6_malloc_may_show_excessive_virtual_memory_usage?lang=en they see similar excessive allocation behavior)
2032015-10-20T14:57:09 <wumpus> by default it creates an arena for every CPU core
2042015-10-20T15:01:34 <jgarzik> yep - that's so thread local structures can run lock free simply by accessing arena_vector[my_cpu_core_number]
2052015-10-20T15:02:12 <wumpus> from a performance point of view it makes a lot of sense
2062015-10-20T15:02:52 <wumpus> at least when there is actual paralellism :)
2072015-10-20T15:04:16 <jgarzik> The cs_main lock has amusing historical parallels. both freebsd and linux kernels had a Big Kernel Lock - a single lock that nearly everything would grab - during the early days of their conversion to SMP/parallel kernels
2082015-10-20T15:04:44 <wumpus> yes and don't forget python's big lock :)
2092015-10-20T15:04:52 <jgarzik> that too
2102015-10-20T15:05:10 <jgarzik> python is so bad at threading you -have- to fork ;p
2112015-10-20T15:06:07 <wumpus> but I hope bitcoin's is more like the BSD/Linux kernel lock than the python one, at least the former got rid of it at some point
2122015-10-20T15:06:46 <GitHub73> [bitcoin] laanwj pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/020c4073a03a...e26a3f6713f8
2132015-10-20T15:06:47 <GitHub73> bitcoin/master f3525e2 Jorge Timón: Chainparams: Replace CBaseChainParams::Network enum with string constants (suggested by Wladimir)
2142015-10-20T15:06:47 <GitHub73> bitcoin/master 55a8975 Jorge Timón: Chainparams: Translations: DRY: options and error strings...
2152015-10-20T15:06:48 <GitHub73> bitcoin/master e26a3f6 Wladimir J. van der Laan: Merge pull request #6235...
2162015-10-20T15:06:52 <GitHub151> [bitcoin] laanwj closed pull request #6235: Chainparams: Translations: DRY: options and error strings (master...chainparams-dry) https://github.com/bitcoin/bitcoin/pull/6235
2172015-10-20T15:07:07 <wumpus> with python it's such an essential part of the design that it's unrealistic to get rid of it. At least that was in the 2.x days, i don't know about 3.x
2182015-10-20T15:09:39 <GitHub87> [bitcoin] laanwj pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/e26a3f6713f8...c6de5cc88614
2192015-10-20T15:09:40 <GitHub87> bitcoin/master e253e83 Matt Corallo: Update debian/changelog and slight tweak to debian/control
2202015-10-20T15:09:40 <GitHub87> bitcoin/master c7b36cc Matt Corallo: Change URLs to https in debian/control
2212015-10-20T15:09:41 <GitHub87> bitcoin/master c6de5cc Wladimir J. van der Laan: Merge pull request #6796...
2222015-10-20T15:09:45 <GitHub160> [bitcoin] laanwj closed pull request #6796: Update debian/changelog and slight tweak to debian/control (master...debian) https://github.com/bitcoin/bitcoin/pull/6796
2232015-10-20T15:10:22 <jgarzik> yep
2242015-10-20T15:10:41 <jgarzik> and we're well on our way to peeling off the cs_main (bsd/linux road)
2252015-10-20T15:10:49 <jgarzik> imo
2262015-10-20T15:31:47 <morcos> wumpus: sorry its taking me a while, i was trying to get a full mempool but not also spam the rest of the network... i haven't run the getblock templates yet, but it sure does make bitciond use a lot less memory in the first place (using MALLOC_ARENA_MAX=1)
2272015-10-20T15:32:19 <morcos> normally i'm using close to 1G shortly after starting, with this, it was about 250M
2282015-10-20T15:38:31 *** paveljanik has joined #bitcoin-core-dev
2292015-10-20T15:46:41 <wumpus> morcos: good to know! thanks for testing, I always had this issue where only very little memory usage would show up in the heap profiler, but top showed much more, now we understand why!
2302015-10-20T15:48:27 <wumpus> and the more cores, the more it will use
2312015-10-20T15:48:53 <wumpus> should add that one to the use-less-memory tips
2322015-10-20T15:49:49 <morcos> wumpus: yep, after repeated calls to getblocktemplate, it only jumped in memory usage once
2332015-10-20T15:50:37 <morcos> i think concentrating on vecPriority was misguided (its not actually that big, just ptrs to txs) but between all the stuff thats stl-allocated in CNB, that probably explains the big jump. i havent' gone through really to see if it adds up yet
2342015-10-20T15:50:43 <morcos> it does still seem like a big jump.
2352015-10-20T15:50:57 <morcos> whether it happens 1x or #rpcthreads times
2362015-10-20T15:51:37 <morcos> i suppose it might depend on how much the CCoinsViewCache allocates?
2372015-10-20T15:52:49 <wumpus> that depends; they will cache everything that is queried through them
2382015-10-20T15:55:45 <jgarzik> wumpus, any remaining merge blockers for #6722 in your view?
2392015-10-20T15:58:20 <morcos> +1 for merging it sooner rather than later. only reason not to merge i think is if we can coerce more people into review/test. but beyond that, lets get it out into the wild to find more bugs.
2402015-10-20T15:58:42 <morcos> however its worth considering merging the lower chain limits soon as well
2412015-10-20T15:58:50 <jgarzik> yeah this sort of stuff is perfect for master
2422015-10-20T16:00:56 <GitHub85> [bitcoin] jtimon reopened pull request #6625: BLOCKING: Consensus: Move blocksize and related parameters to consensusparams ...without removing consensus/consensus.h [#6526 alternative] (master...consensus-blocksize-0.12.99) https://github.com/bitcoin/bitcoin/pull/6625
2432015-10-20T16:01:03 <jgarzik> the Internet is always a better test lab than anything we cook up in private
2442015-10-20T16:01:10 <jgarzik> merge easy, revert easy
2452015-10-20T16:01:59 <jgarzik> I would have merged it myself if it were a normal change, given the ACK quotient, but for a change of this notability I want wumpus, gmaxwell, sipa etc. approval of some sort (sipa has already ACK'd FTR)
2462015-10-20T16:03:30 <wumpus> voila https://gist.github.com/laanwj/efe29c7661ce9b6620a7#linux-specific
2472015-10-20T16:03:35 <wumpus> jgarzik: will take a loook
2482015-10-20T16:03:47 <jgarzik> I've been testing it as well
2492015-10-20T16:04:11 <jgarzik> wumpus, +1 gist
2502015-10-20T16:06:07 <GitHub107> [bitcoin] jtimon closed pull request #6672: Consensus: Separate most consensus functions to consensus.cpp (master...consensus-moveonly-0.12.99) https://github.com/bitcoin/bitcoin/pull/6672
2512015-10-20T16:10:25 *** randy-waterhouse has joined #bitcoin-core-dev
2522015-10-20T16:10:53 <morcos> wumpus: do you think thats a sufficient "fix" for the memory usage.
2532015-10-20T16:11:16 <morcos> still not clear exactly how much heap allocation is happening in getblocktemplate
2542015-10-20T16:11:31 <morcos> but lets say for the sake of argument its on the order of your mempool usage size
2552015-10-20T16:11:57 <morcos> now you're saying every node out there (that doesn't run with your low mem tip) is going to use 5x maxmempoolsize
2562015-10-20T16:12:01 <morcos> at least
2572015-10-20T16:12:21 <morcos> i guess only if they run getblocktemplate
2582015-10-20T16:13:03 <morcos> but maybe we can do something to try to reduce that. we know for instance that there is no need for this particular memory usage to worry about trying to allocate at the same time
2592015-10-20T16:13:50 <morcos> if that was even a static variable that was just cleared between calls, it would persist the memory all the time, but only 1x right?
2602015-10-20T16:14:43 <morcos> so maybe vecPriority and view and whatevers using memory in createnewblock
2612015-10-20T16:18:40 <GitHub92> [bitcoin] sdaftuar closed pull request #6557: Mempool limiting with descendant package tracking (master...mempool-packages) https://github.com/bitcoin/bitcoin/pull/6557
2622015-10-20T16:19:23 <gmaxwell> memory usage I'm seeing is certantly beyond 5x.
2632015-10-20T16:20:05 <gmaxwell> 3.77GB res compared to "usage": 90103744
2642015-10-20T16:26:21 *** Thireus has joined #bitcoin-core-dev
2652015-10-20T16:29:58 <morcos> gmaxwell: i think it turns out this arena per core thing will affect a lot of bitcoind mem usage... also, i think its the max ever mempool usage that matters, not current
2662015-10-20T16:31:38 <gmaxwell> ah! reading more backscroll, so the idea is that the pre-thread (or whatever) arena is ending up allocating but not sbrking data in each thread (or core?) that runs a GBT and thus has a high peak usage because it's copied the mempool?
2672015-10-20T16:33:03 <gmaxwell> Having control of stuff like this is why firefox uses jemalloc
2682015-10-20T16:53:27 <morcos> gmaxwell: yeah to the limit of my understanding. except still not exactly clear what is taking up the memory, b/c the mempool isn't copied. but could be the CoinsViewCache.
2692015-10-20T17:26:25 *** CodeShark has joined #bitcoin-core-dev
2702015-10-20T17:29:56 <wumpus> coinsviewcache has methods to measure the size
2712015-10-20T17:38:39 <gmaxwell> so, reading the page about https://www.ibm.com/developerworks/community/blogs/kevgrig/entry/linux_glibc_2_10_rhel_6_malloc_may_show_excessive_virtual_memory_usage?lang=en .. it looks like thats about about virtual usage, not RES.
2722015-10-20T17:39:47 <gmaxwell> (not that it wouldn't also be good to reduce virtual usage, at least on 32bit.. but I'm seeing about 4x resident than what I expect)
2732015-10-20T17:41:52 *** treehug88 has quit IRC
2742015-10-20T17:45:58 <morcos> gmaxwell: yeah that confused me as well. but from my very ad hoc testing it seemed to affect RES
2752015-10-20T17:51:51 <wumpus> gmaxwell: well the memory is at least temporary resident when it is used
2762015-10-20T17:53:17 <wumpus> it's not just reserving an arena, it's actually returning memory from it, to be used and later freed
2772015-10-20T17:53:48 <wumpus> after which it sticks around, no longer touched so it doesn't have to stay resident, but that depends on mem pressure
2782015-10-20T17:54:22 <wumpus> (also if you keep sending getblocktemplate it probably reuses them in more-or-less round-robin fashion)
2792015-10-20T17:56:49 <gmaxwell> oh i bet that interacts super well with people who've cranked rpc threads to get around the issues with rpcblocking.
2802015-10-20T18:04:28 <wumpus> nah the number of threads is not that relevant, just the number of cores
2812015-10-20T18:05:07 <wumpus> it tries to balance arenas over cores
2822015-10-20T18:05:25 <wumpus> (which makes a lot of sense for performance, but in our case it's not useful)
2832015-10-20T18:05:46 *** Thireus has quit IRC
2842015-10-20T18:06:33 <gmaxwell> I wonder if its ever useful for large allocations. hm. then again the usage is probably made out of zillions of tiny allocations.
2852015-10-20T18:07:56 <wumpus> we had a workstation with a crazy NUMA motherboard at uni, where each processor had its own memory. I guess that's the only case where it makes sense for large, infrequent allocations.
2862015-10-20T18:08:16 <wumpus> (but yes in bitcoin we use tons of small allocations)
2872015-10-20T18:16:32 *** treehug88 has joined #bitcoin-core-dev
2882015-10-20T18:26:39 *** Thireus has joined #bitcoin-core-dev
2892015-10-20T18:29:56 *** treehug88 has quit IRC
2902015-10-20T18:32:43 *** wangchun has joined #bitcoin-core-dev
2912015-10-20T18:34:34 *** treehug88 has joined #bitcoin-core-dev
2922015-10-20T18:43:46 <CodeShark> wumpus: it seems test_bitcoin behavior has changed recently. not too long ago, if I ran src/test/test_bitcoin --run_tests=<testname> from the root project directory, it would only run that specific test - but now it seems to run all of them
2932015-10-20T18:44:08 <CodeShark> am I tripping?
2942015-10-20T18:52:55 <wumpus> --run-tests= takes a test suite name, not an individual test name
2952015-10-20T18:53:03 <CodeShark> I mean singular
2962015-10-20T18:53:05 <CodeShark> --run_test=
2972015-10-20T18:53:15 <wumpus> if the test suite doesn't exist, it somehow gets them all
2982015-10-20T18:53:20 <wumpus> that's not possible
2992015-10-20T18:53:32 <CodeShark> let me verify what I'm saying to make sure it's accurate...
3002015-10-20T18:53:46 <wumpus> yes it's singular run-test, but it takes a suite name
3012015-10-20T18:54:31 <CodeShark> if the test suite doesn't exist I get an error
3022015-10-20T18:54:33 * wumpus test_bitcoin -l test_suite will show exactly what test suites and tests it's executing, and how long they take
3032015-10-20T18:55:47 <wumpus> that behavior is part of boost test, so if it changed, you probably changed your boost version
3042015-10-20T18:56:37 <wumpus> -run-tests= isn't picked up at all here, -t is
3052015-10-20T18:57:20 <CodeShark> travis is apparently not liking alert_tests on windows https://travis-ci.org/bitcoin/bitcoin/jobs/86404141
3062015-10-20T18:58:14 <wumpus> seems to be passing fine here?
3072015-10-20T18:58:49 <CodeShark> I don't get that error at all - I didn't touch alert_tests
3082015-10-20T18:58:57 <CodeShark> I mean, I don't understand that error
3092015-10-20T19:01:49 *** belcher has joined #bitcoin-core-dev
3102015-10-20T19:02:09 <CodeShark> how can I try to reproduce that error locally?
3112015-10-20T19:02:14 <wumpus> I had a quite crazy test error in OpenBSD with Alert_tests today: http://s28.postimg.org/duxcxvrh9/Schermafdruk_van_2015_10_20_10_08_52.png (no, probably not related, something is very wrong there)
3122015-10-20T19:02:43 <wumpus> you can reproduce it in travis?
3132015-10-20T19:03:30 <CodeShark> I'm not very familiar with travis - can I get it to run the test again remotely?
3142015-10-20T19:03:42 <CodeShark> do I need to force push?
3152015-10-20T19:03:57 <wumpus> I can do that, let me see
3162015-10-20T19:08:21 <wumpus> should be running again now
3172015-10-20T19:12:28 <CodeShark> thanks, wumpus - I tried doing a force push anyhow
3182015-10-20T19:12:52 <wumpus> oh then we triggered it twice, that probably just slows it down
3192015-10-20T19:14:35 <jgarzik> RE malloc arenas,
3202015-10-20T19:14:36 <jgarzik> http://journal.siddhesh.in/posts/malloc-per-thread-arenas-in-glibc.html
3212015-10-20T19:14:50 <jgarzik> (apologies if dup; didn't see it in scrollback)
3222015-10-20T19:16:17 * jgarzik looks for some way to cap the number of arenas besides MALLOC_ARENA_MAX
3232015-10-20T19:27:39 <wumpus> don't think I saw that one yet
3242015-10-20T19:28:17 <Luke-Jr> heh, when I wanted to run a specific test, I edited the makefile to not include the rest <.<
3252015-10-20T19:29:10 *** treehug88 has quit IRC
3262015-10-20T19:29:42 <wumpus> "Freed blocks near the end of heaps in an arena are given back to the system either by using madvise(MADV_DONTNEED) or by explicitly unmapping." that doesn't seem to happen for us, probably due to heap fragmentation
3272015-10-20T19:41:26 <jgarzik> nod - lots of little allocations will do that :/
3282015-10-20T20:02:01 <gmaxwell> Change to jemalloc which should also reduce fragmentation.
3292015-10-20T20:02:03 <gmaxwell> :)
3302015-10-20T20:10:59 *** CodeShark has quit IRC
3312015-10-20T20:11:27 *** phantomcircuit has joined #bitcoin-core-dev
3322015-10-20T20:19:28 *** paveljanik has quit IRC
3332015-10-20T20:41:17 *** ParadoxSpiral has quit IRC
3342015-10-20T20:46:59 *** dcousens has quit IRC
3352015-10-20T20:47:22 *** belcher has quit IRC
3362015-10-20T21:06:27 *** CodeShark has joined #bitcoin-core-dev
3372015-10-20T21:16:09 *** randy-waterhouse has quit IRC
3382015-10-20T23:42:04 *** CodeShark has quit IRC