1 2018-08-26T00:03:12 *** meshcollider_ has quit IRC 2 2018-08-26T00:09:21 *** plankers has quit IRC 3 2018-08-26T00:15:30 <gmaxwell> sipa: ah that might have been why I didn't bother trying to make it interpolate-- the seeks are in an index. 4 2018-08-26T00:22:58 *** Giszmo has quit IRC 5 2018-08-26T00:25:23 *** plankers has joined #bitcoin-core-dev 6 2018-08-26T00:29:05 *** Giszmo has joined #bitcoin-core-dev 7 2018-08-26T00:30:22 *** Krellan has quit IRC 8 2018-08-26T00:46:27 *** Chris_Stewart_5 has joined #bitcoin-core-dev 9 2018-08-26T00:53:46 *** Chris_Stewart_5 has quit IRC 10 2018-08-26T01:31:27 *** plankers has quit IRC 11 2018-08-26T02:34:15 *** Jmabsd has joined #bitcoin-core-dev 12 2018-08-26T02:38:00 <jamesob> sipa gmaxwell: mem usage increase is for sure the leveldb change (https://i.imgur.com/yMHJI37.png) and sipa's change didn't change the elevated usage 13 2018-08-26T02:39:06 <sipa> jamesob: so it's still elevated with my patch? 14 2018-08-26T02:39:41 <sipa> or it's not elevated anymore? 15 2018-08-26T02:40:07 <jamesob> still elevated :( 16 2018-08-26T02:41:41 <gmaxwell> might want to make a microbenchmar to figure out if its mmap behavior or something else in leveldb. 17 2018-08-26T02:42:10 <gmaxwell> also because it'll be easier to expirement (e.g. see if MAP_NORESERVE does anything) when we don't have to wait 5 hours. 18 2018-08-26T02:42:43 <gmaxwell> just a loop to mmap all the files in a directory then sleep 30 seconds should test the earlier hypothesis. 19 2018-08-26T02:42:43 *** Krellan has joined #bitcoin-core-dev 20 2018-08-26T02:43:13 <jamesob> gmaxwell: yup, agree 21 2018-08-26T02:43:22 <jamesob> will write if I have time tomorrow 22 2018-08-26T02:50:39 *** murrayn has joined #bitcoin-core-dev 23 2018-08-26T03:12:08 *** BashCo has quit IRC 24 2018-08-26T03:13:04 *** BashCo has joined #bitcoin-core-dev 25 2018-08-26T03:16:04 *** phwalkr has joined #bitcoin-core-dev 26 2018-08-26T03:20:04 *** Chris_Stewart_5 has joined #bitcoin-core-dev 27 2018-08-26T03:22:01 *** d9b4bef9 has quit IRC 28 2018-08-26T03:28:07 *** d9b4bef9 has joined #bitcoin-core-dev 29 2018-08-26T03:28:17 *** Rootsudo has joined #bitcoin-core-dev 30 2018-08-26T03:30:54 *** plankers has joined #bitcoin-core-dev 31 2018-08-26T03:38:20 *** plankers has quit IRC 32 2018-08-26T03:46:35 *** vexbuy_ has quit IRC 33 2018-08-26T03:47:13 *** vexbuy has joined #bitcoin-core-dev 34 2018-08-26T03:51:33 *** vexbuy has quit IRC 35 2018-08-26T03:53:57 <ossifrage> It seems like MADV_RANDOM is useful if there is less locality to the reads then the size of the prefetch window, but I was under the impression that the prefetch is triggered by the pagefault and not by the mmap() so you actually have to do reads to get the extra pages 36 2018-08-26T04:06:24 *** Giszmo has quit IRC 37 2018-08-26T04:17:57 *** Giszmo has joined #bitcoin-core-dev 38 2018-08-26T04:29:09 *** murrayn has quit IRC 39 2018-08-26T04:29:59 *** sipa has quit IRC 40 2018-08-26T04:30:12 *** sipa has joined #bitcoin-core-dev 41 2018-08-26T04:31:53 *** Krellan has quit IRC 42 2018-08-26T04:37:40 *** plankers has joined #bitcoin-core-dev 43 2018-08-26T04:42:15 *** vexbuy has joined #bitcoin-core-dev 44 2018-08-26T04:45:14 *** Jmabsd has quit IRC 45 2018-08-26T04:46:38 *** murrayn has joined #bitcoin-core-dev 46 2018-08-26T04:48:25 *** BashCo has quit IRC 47 2018-08-26T04:49:45 *** BashCo has joined #bitcoin-core-dev 48 2018-08-26T04:53:55 *** phwalkr has quit IRC 49 2018-08-26T04:54:42 *** vexbuy has quit IRC 50 2018-08-26T05:14:47 <luke-jr> was there a point to https://github.com/bitcoin-core/leveldb/pull/19 considering we limit leveldb to 1000 open files anyway? 51 2018-08-26T05:17:15 *** vexbuy has joined #bitcoin-core-dev 52 2018-08-26T05:19:18 <sipa> luke-jr: mmap'ed files aren't kept open after creating the mapping 53 2018-08-26T05:19:31 <sipa> so they don't count towards the 1000 files limit iirc 54 2018-08-26T05:24:10 <luke-jr> sipa: but we increased the file limit to 1000 *because* mmaps didn't count toward the fd limit 55 2018-08-26T05:24:28 <luke-jr> the limit I'm talking about is max_open_files, not ulimit 56 2018-08-26T05:32:01 *** Taifta has quit IRC 57 2018-08-26T05:35:50 <sipa> luke-jr: yes, but there is a separate mmap limit it seems 58 2018-08-26T05:51:01 *** TheHoliestRoger has quit IRC 59 2018-08-26T05:53:49 *** TheHoliestRoger has joined #bitcoin-core-dev 60 2018-08-26T05:59:22 *** TheHoliestRoger has quit IRC 61 2018-08-26T06:02:10 *** TheHoliestRoger has joined #bitcoin-core-dev 62 2018-08-26T06:02:14 *** Chris_Stewart_5 has quit IRC 63 2018-08-26T06:02:58 *** ken2812221 has joined #bitcoin-core-dev 64 2018-08-26T06:03:28 *** Krellan has joined #bitcoin-core-dev 65 2018-08-26T06:04:27 *** Krellan_ has joined #bitcoin-core-dev 66 2018-08-26T06:05:22 *** nOgAnOo_ has quit IRC 67 2018-08-26T06:06:39 *** d_t has joined #bitcoin-core-dev 68 2018-08-26T06:07:38 *** Giszmo has quit IRC 69 2018-08-26T06:08:18 *** Krellan has quit IRC 70 2018-08-26T06:13:02 <luke-jr> sipa: max_open_files effectively limits the number of mmaps too 71 2018-08-26T06:13:29 <luke-jr> so it seems like the LevelDB mmap limit change is effectively a no-op since we keep the 1000 max_open_files limit :x 72 2018-08-26T06:14:59 <gmaxwell> luke-jr: it's very much not a no op change. 73 2018-08-26T06:15:16 <luke-jr> gmaxwell: how does it have an effect? 74 2018-08-26T06:15:43 <gmaxwell> it causes more mmaps to be open, and less files. you can see this easily with lsof. 75 2018-08-26T06:16:06 <luke-jr> the mmap limit is enforced by NewRandomAccessFile, which is only ever called by TableCache which enforces the max_open_files limit :/ 76 2018-08-26T06:16:41 <gmaxwell> Well I'm not telling you what the source code says, I'm telling you what it does. :P 77 2018-08-26T06:16:45 <luke-jr> is there a bug in the LRU cache stuff? :/ 78 2018-08-26T06:18:25 <luke-jr> gmaxwell: is there an easy way to reproduce what you observe? 79 2018-08-26T06:19:45 <gmaxwell> yea, start a node up no connect, with and without the change and lsof it and look at the number of mmaped files. you might need to crank checkblocks up a bit. 80 2018-08-26T06:21:20 *** Giszmo has joined #bitcoin-core-dev 81 2018-08-26T06:32:09 *** plankers has quit IRC 82 2018-08-26T06:33:32 *** Giszmo has quit IRC 83 2018-08-26T06:35:52 *** BashCo has quit IRC 84 2018-08-26T06:36:50 *** BashCo has joined #bitcoin-core-dev 85 2018-08-26T06:46:04 *** plankers has joined #bitcoin-core-dev 86 2018-08-26T06:53:42 *** SopaXorzTaker has joined #bitcoin-core-dev 87 2018-08-26T06:57:30 *** luke-jr has quit IRC 88 2018-08-26T06:57:45 *** Victorsueca has quit IRC 89 2018-08-26T06:58:55 *** Victorsueca has joined #bitcoin-core-dev 90 2018-08-26T07:03:31 *** vexbuy has quit IRC 91 2018-08-26T07:05:02 *** d9b4bef9 has quit IRC 92 2018-08-26T07:05:34 *** BashCo has quit IRC 93 2018-08-26T07:06:08 *** d9b4bef9 has joined #bitcoin-core-dev 94 2018-08-26T07:06:19 *** BashCo has joined #bitcoin-core-dev 95 2018-08-26T07:09:30 *** plankers has quit IRC 96 2018-08-26T07:10:43 *** Taifta has joined #bitcoin-core-dev 97 2018-08-26T07:11:33 *** BashCo has quit IRC 98 2018-08-26T07:14:12 *** BashCo has joined #bitcoin-core-dev 99 2018-08-26T07:15:14 *** luke-jr has joined #bitcoin-core-dev 100 2018-08-26T07:15:22 <wumpus> 0.17.0rc2 binaries up https://bitcoincore.org/bin/bitcoin-core-0.17.0/test.rc2/ 101 2018-08-26T07:19:56 *** vexbuy has joined #bitcoin-core-dev 102 2018-08-26T07:23:38 *** d_t has quit IRC 103 2018-08-26T07:37:25 *** vexbuy has quit IRC 104 2018-08-26T07:48:33 *** vexbuy has joined #bitcoin-core-dev 105 2018-08-26T07:50:03 *** BashCo has quit IRC 106 2018-08-26T07:50:16 <wumpus> luke-jr: the only reason that the change to num files was acceptable is because leveldb, at least the version in our tree, doesn't actually keep more files open (on platforms with mmap) see also https://github.com/bitcoin/bitcoin/pull/12495#issuecomment-377228329 107 2018-08-26T07:50:45 *** BashCo has joined #bitcoin-core-dev 108 2018-08-26T07:53:03 <luke-jr> gmaxwell: cannot reproduce with -checkblocks=10000 :/ 109 2018-08-26T07:53:44 <luke-jr> wumpus: I understand that. I just don't understand why increase the mmap-specific limit, without increasing the num files limit that is required for any mmap 110 2018-08-26T07:53:58 <luke-jr> we will never have more than 1001 mmaps because the files limit is 1000 111 2018-08-26T08:04:27 <wumpus> --I don't think mappings into closed files count against the open files ulimit 112 2018-08-26T08:06:39 *** vexbuy has quit IRC 113 2018-08-26T08:07:17 *** vexbuy has joined #bitcoin-core-dev 114 2018-08-26T08:07:51 *** Rootsudo has quit IRC 115 2018-08-26T08:09:58 <sipa> wumpus: luke's point is that internally in leveldb, the mmapped files count towards the open file limit 116 2018-08-26T08:10:17 <wumpus> but we increased leveldb's limit 117 2018-08-26T08:11:11 <wumpus> though to be honest I'm completely confused now with all the different limites, how they interact, and how they're supposed to interact 118 2018-08-26T08:11:34 <sipa> wumpus: yeah... 119 2018-08-26T08:12:09 <wumpus> given all the confusion about this, I'm starting to think it might be better to revert these changes, I like understanding things even if it means not getting every last possible % of performance 120 2018-08-26T08:12:19 <sipa> so there is a configurable leveldb max open files limit (which is 1000), and a mmap limit (which is hardcoded to 1000, but we raised it to 4096) 121 2018-08-26T08:12:47 <wumpus> and seemingly it now relies on all kinds of undocumented behavior 122 2018-08-26T08:12:59 <sipa> luke says (i haven't verified) that mmapped files count towards that limit... which is inconsistent with our observations 123 2018-08-26T08:13:17 <sipa> i'm just clarifying my understanding as there seems to be some confusion 124 2018-08-26T08:13:18 *** Krellan_ has quit IRC 125 2018-08-26T08:13:27 <wumpus> thanks 126 2018-08-26T08:13:45 <gmaxwell> right if we revert though we should revert both changes. 127 2018-08-26T08:13:51 <wumpus> yes 128 2018-08-26T08:13:57 <gmaxwell> I wouldn't oppose doing that for 0.17. 129 2018-08-26T08:13:59 <sipa> both? what other chanfe? 130 2018-08-26T08:14:01 *** Krellan has joined #bitcoin-core-dev 131 2018-08-26T08:14:10 <sipa> i saw the mmap increaee 132 2018-08-26T08:14:11 <gmaxwell> sipa: the fd count change and the mmap limit change. 133 2018-08-26T08:15:30 <gmaxwell> first FD count was increased with an argument that basically it wouldn't use more FDs due to using mmap. But then we found out that actually when it hit the mmap limit, it started actually using more FDs. 134 2018-08-26T08:15:47 <wumpus> #12495 and https://github.com/bitcoin-core/leveldb/pull/19 135 2018-08-26T08:15:50 <gribble> https://github.com/bitcoin/bitcoin/issues/12495 | Increase LevelDB max_open_files by eklitzke Â· Pull Request #12495 Â· bitcoin/bitcoin Â· GitHub 136 2018-08-26T08:16:05 <gmaxwell> Since the lack of mmaps is probably also bad for performance (though I don't think anyone benchmarked specifically for that), it seemed like it made sense to increase mmaps rather than revert the fd count change. 137 2018-08-26T08:16:19 <wumpus> yes 138 2018-08-26T08:16:47 <wumpus> "By default LevelDB will only mmap() up to 1000 ldb files for reading and then fall back 139 2018-08-26T08:16:50 <wumpus> to using file desciptors." 140 2018-08-26T08:17:00 <wumpus> that pull 19 is *supposed* to change that 141 2018-08-26T08:17:29 <wumpus> (from 1000 to 4096, which is still an arbitrary value) -- maybe luke-jr is using a system leveldb? 142 2018-08-26T08:17:45 <sipa> where did we increase the FDs? 143 2018-08-26T08:18:02 <wumpus> didn't that happen in #12495? 144 2018-08-26T08:18:04 <gribble> https://github.com/bitcoin/bitcoin/issues/12495 | Increase LevelDB max_open_files by eklitzke Â· Pull Request #12495 Â· bitcoin/bitcoin Â· GitHub 145 2018-08-26T08:18:14 <wumpus> I don't know anymore, sorry, should probably get coffee 146 2018-08-26T08:18:53 <gmaxwell> wumpus: 12495, as you say. 147 2018-08-26T08:19:05 <sipa> ah yes 148 2018-08-26T08:19:09 <sipa> hmm 149 2018-08-26T08:19:20 <sipa> we were using 64 files only before 150 2018-08-26T08:19:36 <sipa> but using 1000 there seems pretty standard practice; it's leveldb's own default 151 2018-08-26T08:19:51 <luke-jr> wumpus: not for this testing 152 2018-08-26T08:19:55 <wumpus> sipa: correct 153 2018-08-26T08:20:21 <gmaxwell> sipa: after increasing it, we were using a lot more file descriptors. 154 2018-08-26T08:20:23 <luke-jr> wumpus: but in any case, for testing this, I'm changing the 4096 back to 1000, and can't get LDB to use a fd 155 2018-08-26T08:20:39 <gmaxwell> luke-jr: are you using some system leveldb? 156 2018-08-26T08:20:59 <luke-jr> gmaxwell: like I said, not for this; but in any case, it would be the same 1000 mmaps 157 2018-08-26T08:21:47 <gmaxwell> luke-jr: dunno why it didn't reproduce for you, I saw leveldb using FDs, after using 1000 maps... and not anymore after increasing the map limit. 158 2018-08-26T08:21:55 <luke-jr> hmm 159 2018-08-26T08:22:02 <gmaxwell> sipa also saw it using fd's on his node. 160 2018-08-26T08:22:09 <luke-jr> testnet or mainnet? 161 2018-08-26T08:22:13 <gmaxwell> mainnet 162 2018-08-26T08:22:14 <sipa> mainnet 163 2018-08-26T08:22:19 <luke-jr> I wonder if that's why. :x 164 2018-08-26T08:22:21 <gmaxwell> testnet probably doesn't have enough ldb files. 165 2018-08-26T08:22:26 <luke-jr> ah 166 2018-08-26T08:22:49 <sipa> how large is testnet's utxo set? 167 2018-08-26T08:24:53 *** PatBoy has quit IRC 168 2018-08-26T08:26:12 *** PatBoy has joined #bitcoin-core-dev 169 2018-08-26T08:35:52 <wumpus> "disk_size": 1025042445, as of height 1384188 (it is at 1392776 now, don't know how much it changed since then) 170 2018-08-26T08:48:45 <luke-jr> hopefully I won't "damage" my mainnet .bitcoin trying this.. :x 171 2018-08-26T08:49:05 <luke-jr> (I still run 0.13 wallets from time to time) 172 2018-08-26T08:50:34 *** plankers has joined #bitcoin-core-dev 173 2018-08-26T08:50:41 <luke-jr> looks like the UTXO db upgrade isn't triggering it, at least 174 2018-08-26T08:55:24 *** plankers has quit IRC 175 2018-08-26T09:04:44 * luke-jr facepalms. After UTXO upgrade, it says I need to reindex >_< 176 2018-08-26T09:09:22 *** plankers has joined #bitcoin-core-dev 177 2018-08-26T09:10:00 <wumpus> does it say why / give an error message? the whole point of the utxo upgrade would be to not reindex, unless it fails 178 2018-08-26T09:11:54 <provoostenator> If you're traveling and stuck with a slow laptop, the following gist lets you compile remotely and then connect via SSH tunnel: https://gist.github.com/Sjors/fda9b601e9464f61332ac6490f487c0a (improvements welcome) 179 2018-08-26T09:26:16 <wumpus> provoostenator: nit: you could pass NO_UPNP to the depends build instead of --with-miniupnpc=no 180 2018-08-26T09:26:32 <wumpus> not that libminiupnp takes long to compile 181 2018-08-26T09:27:10 <provoostenator> wumpus: thanks, in this case it's probably easier to keep the configure line flexible. 182 2018-08-26T09:27:17 *** murrayn has quit IRC 183 2018-08-26T09:28:07 <provoostenator> I first tried copying bitcoind over and just running it locally but that resulted in a world of pain, probably because the two systems aren't perfectly identical. 184 2018-08-26T09:28:12 *** plankers has quit IRC 185 2018-08-26T09:28:46 <wumpus> yes copying executables between linux systems is fraught with problems 186 2018-08-26T09:29:02 <provoostenator> It's works fine with c-lightning though. 187 2018-08-26T09:29:16 <wumpus> though it should work with depends and the glibc compat enabled 188 2018-08-26T09:29:53 <provoostenator> I tried with depends as well, haven't tried glibc compat, might try that next time. But this also prevents me from accidentally syncing the chain over 4G. 189 2018-08-26T09:30:03 <wumpus> -static-libstdc++ helps as well 190 2018-08-26T09:30:24 <wumpus> (but only if all c++ dependencies are static, which is the case with a depends build but not otherwise) 191 2018-08-26T09:32:17 <wumpus> then you end up with an executable that is as-good-as-static and it should be portable to non-ancient linux versions with the same CPU architecture 192 2018-08-26T09:33:14 <wumpus> of course, that's the theory, there are still some things that can interfere for example if glibc is built with different options (I vaguely remember redhat and some security/hardening flag), or the target platform has a different libc 193 2018-08-26T09:35:17 <wumpus> this is one reason they came up with "snap"s which are executables that essentially carry around part of the system dependencies with them in a container 194 2018-08-26T09:35:39 <wumpus> (along with some security/isolation features) 195 2018-08-26T09:35:45 <provoostenator> Being able to run locally is especially useful for QT, and with the multiprocess stuff it would be nice to run bitcoind remote and QT locally. 196 2018-08-26T09:36:31 <wumpus> aanyhow if your two systems have the same release of the same distro, and are at the same update level, it should work too, that's probably easiest for most... 197 2018-08-26T09:50:05 <luke-jr> wumpus: I reflink'd the blocks/ and chainstate/ dirs to a new datadir. I'm guessing that simply wasn't sufficient somehow 198 2018-08-26T09:50:30 *** plankers has joined #bitcoin-core-dev 199 2018-08-26T09:52:52 *** murrayn has joined #bitcoin-core-dev 200 2018-08-26T09:54:34 *** BashCo has quit IRC 201 2018-08-26T09:54:57 <luke-jr> oh well, I'll let it sync overnight and see if it hits the breakpoint 202 2018-08-26T09:55:12 <luke-jr> env_posix.cc:379 if anyone else who can reproduce it wants to try 203 2018-08-26T09:55:36 <luke-jr> (if you hit it, maybe see if you can work out why it got there..) 204 2018-08-26T09:56:03 *** BashCo has joined #bitcoin-core-dev 205 2018-08-26T10:18:13 *** SopaXorzTaker has quit IRC 206 2018-08-26T10:20:57 *** Krellan has quit IRC 207 2018-08-26T10:21:43 *** Krellan has joined #bitcoin-core-dev 208 2018-08-26T10:33:31 *** as1nc has quit IRC 209 2018-08-26T10:46:30 *** JackH has joined #bitcoin-core-dev 210 2018-08-26T11:01:42 *** vexbuy has quit IRC 211 2018-08-26T11:14:37 *** vexbuy has joined #bitcoin-core-dev 212 2018-08-26T11:23:18 <provoostenator> Any volunteer for making a PR based on this commit? https://github.com/bitcoin/bitcoin/commit/a9db3dada0119c183d16627805e90c4dbca05c6a 213 2018-08-26T11:23:46 <provoostenator> I left it out of my rebase #13937 because I'd rather not touch thread stuff. 214 2018-08-26T11:23:47 <gribble> https://github.com/bitcoin/bitcoin/issues/13937 | Track best-possible-headers (TheBlueMatt) by Sjors Â· Pull Request #13937 Â· bitcoin/bitcoin Â· GitHub 215 2018-08-26T11:32:53 *** belcher_ has joined #bitcoin-core-dev 216 2018-08-26T11:32:57 *** SopaXorzTaker has joined #bitcoin-core-dev 217 2018-08-26T11:37:37 *** str4d has joined #bitcoin-core-dev 218 2018-08-26T11:41:02 *** cdecker has joined #bitcoin-core-dev 219 2018-08-26T11:51:14 *** str4d has quit IRC 220 2018-08-26T11:51:32 *** str4d has joined #bitcoin-core-dev 221 2018-08-26T12:06:26 <provoostenator> Never mind, that's already been done in #13023 222 2018-08-26T12:06:29 <gribble> https://github.com/bitcoin/bitcoin/issues/13023 | Fix some concurrency issues in ActivateBestChain() by skeees Â· Pull Request #13023 Â· bitcoin/bitcoin Â· GitHub 223 2018-08-26T12:13:33 *** Emcy has quit IRC 224 2018-08-26T12:19:12 *** sipa has quit IRC 225 2018-08-26T12:19:32 *** sipa has joined #bitcoin-core-dev 226 2018-08-26T12:20:43 *** sipa has quit IRC 227 2018-08-26T12:28:30 *** Krellan has quit IRC 228 2018-08-26T12:29:20 *** Krellan has joined #bitcoin-core-dev 229 2018-08-26T12:31:23 *** BashCo has quit IRC 230 2018-08-26T12:33:49 *** BashCo has joined #bitcoin-core-dev 231 2018-08-26T12:34:35 *** sipa has joined #bitcoin-core-dev 232 2018-08-26T12:36:19 *** farmerwampum has joined #bitcoin-core-dev 233 2018-08-26T12:36:57 *** farmerwampum has quit IRC 234 2018-08-26T12:37:19 *** farmerwampum has joined #bitcoin-core-dev 235 2018-08-26T12:39:29 *** wxss has quit IRC 236 2018-08-26T12:47:22 *** csknk has joined #bitcoin-core-dev 237 2018-08-26T12:58:43 *** Taifta has quit IRC 238 2018-08-26T13:02:07 *** Guyver2 has joined #bitcoin-core-dev 239 2018-08-26T13:15:00 *** Taifta has joined #bitcoin-core-dev 240 2018-08-26T13:15:01 *** Victorsueca has quit IRC 241 2018-08-26T13:16:14 *** Victorsueca has joined #bitcoin-core-dev 242 2018-08-26T13:28:22 *** Taifta has quit IRC 243 2018-08-26T13:28:23 *** BashCo has quit IRC 244 2018-08-26T13:28:35 *** booyah has quit IRC 245 2018-08-26T13:29:46 *** booyah has joined #bitcoin-core-dev 246 2018-08-26T13:31:06 *** booyah has quit IRC 247 2018-08-26T13:31:09 *** BashCo has joined #bitcoin-core-dev 248 2018-08-26T13:32:46 *** vexbuy has quit IRC 249 2018-08-26T13:36:54 *** vexbuy has joined #bitcoin-core-dev 250 2018-08-26T13:44:48 *** booyah has joined #bitcoin-core-dev 251 2018-08-26T13:50:37 *** Emcy has joined #bitcoin-core-dev 252 2018-08-26T13:55:30 *** Chris_Stewart_5 has joined #bitcoin-core-dev 253 2018-08-26T13:56:38 *** Taifta has joined #bitcoin-core-dev 254 2018-08-26T14:06:19 <luke-jr> wumpus: heh, I was tired last night; the reason it needed a reindex (or I guess re-IBD) seems to be that I named the blocks directory "newblocks" XD 255 2018-08-26T14:06:29 <luke-jr> err, "tmpblocks" 256 2018-08-26T14:25:00 *** Taifta has quit IRC 257 2018-08-26T14:34:04 *** Emcy has quit IRC 258 2018-08-26T14:36:25 *** Krellan has quit IRC 259 2018-08-26T14:37:27 *** Krellan has joined #bitcoin-core-dev 260 2018-08-26T14:39:02 *** elichai2 has joined #bitcoin-core-dev 261 2018-08-26T14:51:35 *** vexbuy has quit IRC 262 2018-08-26T14:54:00 *** vexbuy has joined #bitcoin-core-dev 263 2018-08-26T15:02:51 *** vexbuy has quit IRC 264 2018-08-26T15:03:29 *** vexbuy has joined #bitcoin-core-dev 265 2018-08-26T15:06:51 *** spinza has quit IRC 266 2018-08-26T15:06:57 *** Chris_Stewart_5 has quit IRC 267 2018-08-26T15:20:30 *** spinza has joined #bitcoin-core-dev 268 2018-08-26T15:29:30 *** BashCo has quit IRC 269 2018-08-26T15:30:12 *** BashCo has joined #bitcoin-core-dev 270 2018-08-26T15:34:46 *** Chris_Stewart_5 has joined #bitcoin-core-dev 271 2018-08-26T15:46:47 *** plankers has quit IRC 272 2018-08-26T15:48:38 *** vexbuy_ has joined #bitcoin-core-dev 273 2018-08-26T15:48:59 *** vexbuy has quit IRC 274 2018-08-26T15:49:39 *** Krellan has quit IRC 275 2018-08-26T15:50:12 *** Krellan has joined #bitcoin-core-dev 276 2018-08-26T15:55:09 *** Krellan has quit IRC 277 2018-08-26T15:55:50 *** Emcy has joined #bitcoin-core-dev 278 2018-08-26T15:56:01 *** vexbuy_ has quit IRC 279 2018-08-26T15:56:37 *** vexbuy has joined #bitcoin-core-dev 280 2018-08-26T16:07:48 *** csknk has quit IRC 281 2018-08-26T16:12:15 *** Krellan has joined #bitcoin-core-dev 282 2018-08-26T16:13:23 *** sipa_ has joined #bitcoin-core-dev 283 2018-08-26T16:14:09 *** sipa has quit IRC 284 2018-08-26T16:14:39 <Chris_Stewart_5> BIP157 doesn't currently have an open PR does it? 285 2018-08-26T16:15:27 *** intcat has quit IRC 286 2018-08-26T16:17:14 *** Krellan has quit IRC 287 2018-08-26T16:18:38 <instagibbs> Chris_Stewart_5, https://github.com/bitcoin/bitcoin/pull/13144 288 2018-08-26T16:18:54 *** intcat has joined #bitcoin-core-dev 289 2018-08-26T16:26:05 <Chris_Stewart_5> instagibbs: So that is just a refactor PR to make it easier to implement BIP157 right? 290 2018-08-26T16:29:19 *** BashCo has quit IRC 291 2018-08-26T16:30:15 *** BashCo has joined #bitcoin-core-dev 292 2018-08-26T16:36:03 <instagibbs> doh wrong one 293 2018-08-26T16:36:04 <instagibbs> one sec 294 2018-08-26T16:36:47 <instagibbs> Chris_Stewart_5, merged two hours ago, didnt see that https://github.com/bitcoin/bitcoin/pull/12254 295 2018-08-26T16:37:02 <instagibbs> nothing open after that 296 2018-08-26T16:37:16 <Chris_Stewart_5> Hmm, is that only BIP158 though? I don't think that BIP158 includes any p2p networking stuff 297 2018-08-26T16:37:26 <Chris_Stewart_5> BIP157 seems to have the semantics for that 298 2018-08-26T16:48:12 *** Krellan has joined #bitcoin-core-dev 299 2018-08-26T16:50:51 *** itaseski has joined #bitcoin-core-dev 300 2018-08-26T16:55:44 *** BashCo has quit IRC 301 2018-08-26T16:57:55 *** Taifta has joined #bitcoin-core-dev 302 2018-08-26T16:58:09 *** BashCo has joined #bitcoin-core-dev 303 2018-08-26T17:18:50 *** reallll has joined #bitcoin-core-dev 304 2018-08-26T17:21:11 *** Victorsueca has quit IRC 305 2018-08-26T17:21:11 *** BashCo has quit IRC 306 2018-08-26T17:21:58 *** BashCo has joined #bitcoin-core-dev 307 2018-08-26T17:22:25 *** Victorsueca has joined #bitcoin-core-dev 308 2018-08-26T17:22:43 *** belcher_ has quit IRC 309 2018-08-26T17:34:41 *** promag has joined #bitcoin-core-dev 310 2018-08-26T17:36:23 *** reallll has quit IRC 311 2018-08-26T17:41:35 *** belcher_ has joined #bitcoin-core-dev 312 2018-08-26T17:42:30 *** promag has quit IRC 313 2018-08-26T18:03:44 *** vexbuy_ has joined #bitcoin-core-dev 314 2018-08-26T18:05:25 *** vexbuy has quit IRC 315 2018-08-26T18:10:34 *** sipa_ is now known as sipa 316 2018-08-26T18:15:02 *** BashCo has quit IRC 317 2018-08-26T18:17:18 *** BashCo has joined #bitcoin-core-dev 318 2018-08-26T18:24:45 *** promag has joined #bitcoin-core-dev 319 2018-08-26T18:26:17 *** promag has quit IRC 320 2018-08-26T18:28:29 <jimpo> Chris_Stewart_5: No, I'm working on the next PR towards BIP 157 support 321 2018-08-26T18:28:39 <jimpo> Which is building filter indexes 322 2018-08-26T18:28:45 *** promag has joined #bitcoin-core-dev 323 2018-08-26T18:33:00 *** promag has quit IRC 324 2018-08-26T18:34:16 *** Krellan has quit IRC 325 2018-08-26T18:34:36 *** Krellan has joined #bitcoin-core-dev 326 2018-08-26T18:37:02 *** jb55 has joined #bitcoin-core-dev 327 2018-08-26T18:39:33 <Chris_Stewart_5> gmaxwell: PIR? Without a PIR method of fetching blocks, use of this is still toxic to user privacy 328 2018-08-26T18:43:32 <sipa> Chris_Stewart_5: anything that doesn't fetch all blocks will leak some information, inevitably 329 2018-08-26T18:45:58 <Chris_Stewart_5> What is the acronym? PIR -- i'm guessing it is not Passive Infrared Sensor hah 330 2018-08-26T18:47:03 <sipa> private information retrieval 331 2018-08-26T18:53:00 *** Taifta has quit IRC 332 2018-08-26T19:01:42 *** BashCo has quit IRC 333 2018-08-26T19:03:17 *** BashCo has joined #bitcoin-core-dev 334 2018-08-26T19:03:17 <Chris_Stewart_5> jimpo: These test vectors seem to be a dead link https://github.com/bitcoin/bips/blob/master/bip-0158.mediawiki#appendix-c-test-vectors 335 2018-08-26T19:03:38 <Chris_Stewart_5> at least the first link 336 2018-08-26T19:07:51 *** Salve has joined #bitcoin-core-dev 337 2018-08-26T19:18:44 *** elichai2 has quit IRC 338 2018-08-26T19:25:26 *** BashCo has quit IRC 339 2018-08-26T19:26:21 *** BashCo has joined #bitcoin-core-dev 340 2018-08-26T19:33:42 *** BashCo has quit IRC 341 2018-08-26T19:41:15 *** Salve has quit IRC 342 2018-08-26T19:53:44 *** rex4539 has quit IRC 343 2018-08-26T19:54:21 *** Scrat has joined #bitcoin-core-dev 344 2018-08-26T19:56:50 *** Scrat has quit IRC 345 2018-08-26T20:04:15 *** rex4539 has joined #bitcoin-core-dev 346 2018-08-26T20:05:37 *** rex4539 has joined #bitcoin-core-dev 347 2018-08-26T20:06:31 *** JackH has quit IRC 348 2018-08-26T20:08:54 *** kallisteiros has joined #bitcoin-core-dev 349 2018-08-26T20:09:11 *** hebasto has quit IRC 350 2018-08-26T20:10:08 *** owowo has quit IRC 351 2018-08-26T20:10:40 *** SopaXorzTaker has quit IRC 352 2018-08-26T20:16:36 <jimpo> Oh, thanks, will open a PR to fix that. Need to add a test block anyway. 353 2018-08-26T20:17:33 <Chris_Stewart_5> Thanks! 354 2018-08-26T20:43:44 *** Krellan has quit IRC 355 2018-08-26T20:44:46 *** Krellan has joined #bitcoin-core-dev 356 2018-08-26T21:05:01 *** kallisteiros has quit IRC 357 2018-08-26T21:14:54 *** itaseski has quit IRC 358 2018-08-26T21:15:34 *** chainhead has joined #bitcoin-core-dev 359 2018-08-26T21:41:16 *** promag has joined #bitcoin-core-dev 360 2018-08-26T21:44:26 *** promag has quit IRC 361 2018-08-26T21:44:45 *** promag has joined #bitcoin-core-dev 362 2018-08-26T21:51:02 *** d9b4bef9 has quit IRC 363 2018-08-26T21:52:08 *** d9b4bef9 has joined #bitcoin-core-dev 364 2018-08-26T21:53:07 *** belcher_ has quit IRC 365 2018-08-26T22:03:27 *** plankers has joined #bitcoin-core-dev 366 2018-08-26T22:06:55 *** promag has quit IRC 367 2018-08-26T22:07:30 *** promag has joined #bitcoin-core-dev 368 2018-08-26T22:12:04 *** promag has quit IRC 369 2018-08-26T22:18:05 *** promag has joined #bitcoin-core-dev 370 2018-08-26T22:28:12 <luke-jr> gmaxwell: sipa: wumpus: sync'd up to 341411 and still no non-mmap opens from LDB.. 371 2018-08-26T22:29:10 <sipa> luke-jr: ? 372 2018-08-26T22:29:15 <sipa> what does that mean 373 2018-08-26T22:29:50 <gmaxwell> 341411 is a long time ago. 374 2018-08-26T22:30:29 <gmaxwell> sipa: he's trying to reproduce the behavior ossifrage, you, and I saw where leveldb would hit the mmap limit and start using more FDs. 375 2018-08-26T22:30:36 <sipa> you'll only exceed 1000 files/mmaps once you're above ~2GB UTXO set size 376 2018-08-26T22:30:43 <sipa> as each file is 2MB ish 377 2018-08-26T22:31:55 <luke-jr> hmm 378 2018-08-26T22:32:25 <ossifrage> pmap 24199 | grep \.ldb | wc -l == 1894 (for a node up ~26 days) 379 2018-08-26T22:34:52 <sipa> luke-jr: so you'll need to get to somewhere in 2017 at least 380 2018-08-26T22:35:23 * luke-jr retries the UTXO upgrade, this time with the blocks dir named correctly 381 2018-08-26T22:35:25 <ossifrage> That includes the txindex and the chainstate 382 2018-08-26T22:36:46 <sipa> ossifrage: per database; txindex and chainstate are separate 383 2018-08-26T22:37:28 <gmaxwell> has anyone just tried making a dummy program that mmaps a couple thousand files to see what the RES shows up as? 384 2018-08-26T22:38:14 *** Victorsueca has quit IRC 385 2018-08-26T22:39:25 *** Victorsueca has joined #bitcoin-core-dev 386 2018-08-26T22:40:01 <ossifrage> is having a larger resident size really an issue, you only get those pages if memory is available, the os should evict them right quick under memory pressure 387 2018-08-26T22:40:45 <sipa> presumably, indeed 388 2018-08-26T22:41:23 <sipa> it may be just an methodology problem, what rss doesn't exactly corresponds to what we care about 389 2018-08-26T22:41:45 <gmaxwell> ossifrage: if thats actually the case, thing things are fine. 390 2018-08-26T22:43:52 <gmaxwell> and if so we probably need to figure out how to start tracking "total dirty pages" or something. 391 2018-08-26T22:44:37 <ossifrage> leveldb only uses mmap for reading, so the dirty pages from all the maps should be 0 392 2018-08-26T22:44:45 <gmaxwell> Right. 393 2018-08-26T22:44:55 <luke-jr> personally, I only look at ps's "size" metric 394 2018-08-26T22:45:25 <luke-jr> size SIZE approximate amount of swap space that would be required if the process were to dirty all 395 2018-08-26T22:45:26 <luke-jr> writable pages and then be swapped out. This number is very rough! 396 2018-08-26T22:45:38 <gmaxwell> ossifrage: so to catch you up. We have infrastructure that benchmarks the software regularly and charts performance... so we find out if some commit accidentally hurts performance. 397 2018-08-26T22:46:03 <gmaxwell> ossifrage: we noticed around the time of the 0.17 feature freeze that the claimed memory usage suddenly went up 20%! 398 2018-08-26T22:46:20 <gmaxwell> during the week where a lot of last minute stuff got merged. 399 2018-08-26T22:46:37 <gmaxwell> Bisection turned up your change to increase the leveldb maps count. 400 2018-08-26T22:46:48 <ossifrage> gmaxwell, where does the claimed memory measurement come from? 401 2018-08-26T22:47:44 <gmaxwell> Thats what we're trying to determine. after finding out that mmap automatically pages in some of the file tried madvising DONTNEED all the maps, but that didn't seem to fix it. 402 2018-08-26T22:48:08 <gmaxwell> It might well be that the increased usage is totally fictional... that it's just counting paged-in non-dirty pages.... and they're not real memory pressure and we don't need to worry about them. 403 2018-08-26T22:48:17 <ossifrage> I'm writing a mmap() test case out of morbid curiosity 404 2018-08-26T22:49:05 <gmaxwell> Or maybe, though we haven't spotted it yet, leveldb is using 200kb of god knows what per open map. :P 405 2018-08-26T22:49:40 <gmaxwell> Meanwhile, luke-jr went and read more of the leveldb code and believes that it can't have more mmaps open than the FD limit, and so he thinks the mmap increase should be a noop. 406 2018-08-26T22:50:05 <gmaxwell> (which it clearly isn't, but it would be good if everyone was on the same page about how leveldb works) 407 2018-08-26T22:50:08 *** Chris_Stewart_5 has quit IRC 408 2018-08-26T22:53:50 <ossifrage> I currently have 992 chainstate ldb files mapped and 859 txindex and 43 blocks/indexes ldb files, but it took a fairly long uptime for the original problem to show up 409 2018-08-26T22:53:57 *** Krellan has quit IRC 410 2018-08-26T22:54:42 *** Krellan has joined #bitcoin-core-dev 411 2018-08-26T22:58:22 *** Guyver2 has quit IRC 412 2018-08-26T23:00:35 <ossifrage> I don't see much of a change between the htop reported RES size between mapping 1 txindex ldb file and mmaping all 2041 of them 413 2018-08-26T23:00:47 <luke-jr> gmaxwell: well, I can reproduce it now, but I haven't figure out why yet 414 2018-08-26T23:00:53 <ossifrage> (but I never read from the map) 415 2018-08-26T23:02:04 <gmaxwell> ossifrage: k? try reading like, the last byte of them or something? 416 2018-08-26T23:02:24 <ossifrage> The VIRT line goes up as expected but RES stays fairly small 417 2018-08-26T23:03:24 <luke-jr> hmm, is everyone reproducing this using txindex? 418 2018-08-26T23:04:25 <ossifrage> gmaxwell, yeah that caused the RES size to approach the VIRT size (read the last byte) 419 2018-08-26T23:05:09 <sipa> gmaxwell: "it can't have more mmaps open than the FD limit" -> s/FD limit/max_open_files setting/ 420 2018-08-26T23:05:27 <gmaxwell> now, try MADV_RANDOM ? 421 2018-08-26T23:05:34 <gmaxwell> ossifrage: ^ 422 2018-08-26T23:05:51 <ossifrage> I read the last byte, so I'm not sure what it would prefetch :-) 423 2018-08-26T23:06:09 *** andytoshi has joined #bitcoin-core-dev 424 2018-08-26T23:06:14 <gmaxwell> I'm guessing that it's prefetching the whole thing. 425 2018-08-26T23:06:25 <gmaxwell> andytoshi: HI! 426 2018-08-26T23:07:23 <andytoshi> gmaxwell: hiya, sorry, my server shit out this morning .. i have no idea why, the logs show everything clear, but it was apparently completely locked up even to a local terminal user 427 2018-08-26T23:07:28 <andytoshi> so i called my dad to hard-boot it and it seems ok now 428 2018-08-26T23:07:46 <ossifrage> MADV_RANDOM didn't seem to change the bump in RES caused by reading last byte 429 2018-08-26T23:08:00 *** promag has quit IRC 430 2018-08-26T23:09:02 <gmaxwell> ossifrage: saddness. ... so run pmap on the process, and see how many dirty pages it says you have, presumably like none. 431 2018-08-26T23:11:03 *** Chris_Stewart_5 has joined #bitcoin-core-dev 432 2018-08-26T23:12:38 <sipa> gmaxwell: mmap is only used in leveldb for read-only files; none should ever be dirty 433 2018-08-26T23:12:54 <gmaxwell> Interesting, there are only 41.3 BTC stored in bare multisig outputs, but there are 410k of them in total. 434 2018-08-26T23:13:22 <gmaxwell> sipa: I know but I also would have expected that increasing the mmap limit wouldn't have increased RES. :) 435 2018-08-26T23:13:51 <ossifrage> One sec, I might have screwed up... 436 2018-08-26T23:14:12 <gmaxwell> I'm going to suggest we start charting the total dirty pages from pmap... and not worry about RES... 437 2018-08-26T23:14:13 <luke-jr> it looks like the mmap limit is global 438 2018-08-26T23:18:32 <ossifrage> Okay, it looks like MADV_RANDOM isn't helping, but just reading the last byte does cause the RES size to jump quite a bit (but I'm not sure how much I expect it to go up) 439 2018-08-26T23:18:43 <ossifrage> at least 1 page per file? 440 2018-08-26T23:18:53 <sipa> yes, 1 page per file is expected 441 2018-08-26T23:21:41 <gmaxwell> I expect that 1 page per file is what you should get, which is a modest amount of memory. 442 2018-08-26T23:21:49 <gmaxwell> in bitcoin we're seeing an increase of hundreds of meg. 443 2018-08-26T23:21:56 <gmaxwell> way more than one page per file. 444 2018-08-26T23:23:55 <ossifrage> When reading the first byte, MAD_RANDOM does decrease the RES usage 445 2018-08-26T23:28:33 <ossifrage> Even with MADV_RANDOM, it would depend entirely on how many pages are faulted in by ldb 446 2018-08-26T23:34:20 *** promag has joined #bitcoin-core-dev 447 2018-08-26T23:37:11 <ossifrage> Err, it looks like for me it is prefetching ~64K 448 2018-08-26T23:43:34 <gmaxwell> 64K with MADV_RANDOM? 449 2018-08-26T23:44:05 <ossifrage> I think I might be suffering from previous state problems 450 2018-08-26T23:45:57 <ossifrage> Okay, dropping the cache and madv_random does seem to only read 4K for most of the files mapped (some have more for some odd reason) 451 2018-08-26T23:47:08 <gmaxwell> oh interesting, so, if you cause the files to be paged in, then start a program with madv_random it will reflect higher res becuase you mmaped files that were already in cache? 452 2018-08-26T23:47:10 <ossifrage> Yeah, without MADV_RANDOM all the files have 64K RES when I read the 1st byte 453 2018-08-26T23:47:27 <ossifrage> echo 3 > /proc/sys/vm/drop_caches between runs 454 2018-08-26T23:47:51 <ossifrage> Yeah I was getting confusing results until I realized that 455 2018-08-26T23:47:54 <gmaxwell> Cool so also we might not have seen the effects of MADV_RANDOM when testing with actual leveldb due to that. 456 2018-08-26T23:48:15 <ossifrage> Depends heavily on the specific hardware yeah 457 2018-08-26T23:48:43 <ossifrage> But drop_caches is a mighty big hammer 458 2018-08-26T23:49:14 <gmaxwell> sure, just in terms of "huh why did MADV_RANDOM seem to do nothing"... now we know. 459 2018-08-26T23:50:19 <ossifrage> I suspect the minimum amount read without MADV_RANDOM depends on things like the filesystem and the underlying block device 460 2018-08-26T23:51:26 <ossifrage> Like reading 1 page with a SSD is no big deal vs 1 page with a RAID5 md array 461 2018-08-26T23:55:28 <ossifrage> Any ideas how much leveldb actually reads at a time? 462 2018-08-26T23:55:43 *** Victorsueca has quit IRC 463 2018-08-26T23:56:35 <ossifrage> It would have to touch at least some of the index and then the actual data it wants to read 464 2018-08-26T23:56:55 *** Victorsueca has joined #bitcoin-core-dev 465 2018-08-26T23:57:48 <gmaxwell> it does a bisection search in the index and then I think it only reads the record of interest.