1 2018-08-26T00:03:12  *** meshcollider_ has quit IRC
  2 2018-08-26T00:09:21  *** plankers has quit IRC
  3 2018-08-26T00:15:30  <gmaxwell> sipa: ah that might have been why I didn't bother trying to make it interpolate-- the seeks are in an index.
  4 2018-08-26T00:22:58  *** Giszmo has quit IRC
  5 2018-08-26T00:25:23  *** plankers has joined #bitcoin-core-dev
  6 2018-08-26T00:29:05  *** Giszmo has joined #bitcoin-core-dev
  7 2018-08-26T00:30:22  *** Krellan has quit IRC
  8 2018-08-26T00:46:27  *** Chris_Stewart_5 has joined #bitcoin-core-dev
  9 2018-08-26T00:53:46  *** Chris_Stewart_5 has quit IRC
 10 2018-08-26T01:31:27  *** plankers has quit IRC
 11 2018-08-26T02:34:15  *** Jmabsd has joined #bitcoin-core-dev
 12 2018-08-26T02:38:00  <jamesob> sipa gmaxwell: mem usage increase is for sure the leveldb change (https://i.imgur.com/yMHJI37.png) and sipa's change didn't change the elevated usage
 13 2018-08-26T02:39:06  <sipa> jamesob: so it's still elevated with my patch?
 14 2018-08-26T02:39:41  <sipa> or it's not elevated anymore?
 15 2018-08-26T02:40:07  <jamesob> still elevated :(
 16 2018-08-26T02:41:41  <gmaxwell> might want to make a microbenchmar to figure out if its mmap behavior or something  else in leveldb.
 17 2018-08-26T02:42:10  <gmaxwell> also because it'll be easier to expirement (e.g. see if MAP_NORESERVE does anything) when we don't have to wait 5 hours.
 18 2018-08-26T02:42:43  <gmaxwell> just a loop to mmap all the files in a directory then sleep 30 seconds should test the earlier hypothesis.
 19 2018-08-26T02:42:43  *** Krellan has joined #bitcoin-core-dev
 20 2018-08-26T02:43:13  <jamesob> gmaxwell: yup, agree
 21 2018-08-26T02:43:22  <jamesob> will write if I have time tomorrow
 22 2018-08-26T02:50:39  *** murrayn has joined #bitcoin-core-dev
 23 2018-08-26T03:12:08  *** BashCo has quit IRC
 24 2018-08-26T03:13:04  *** BashCo has joined #bitcoin-core-dev
 25 2018-08-26T03:16:04  *** phwalkr has joined #bitcoin-core-dev
 26 2018-08-26T03:20:04  *** Chris_Stewart_5 has joined #bitcoin-core-dev
 27 2018-08-26T03:22:01  *** d9b4bef9 has quit IRC
 28 2018-08-26T03:28:07  *** d9b4bef9 has joined #bitcoin-core-dev
 29 2018-08-26T03:28:17  *** Rootsudo has joined #bitcoin-core-dev
 30 2018-08-26T03:30:54  *** plankers has joined #bitcoin-core-dev
 31 2018-08-26T03:38:20  *** plankers has quit IRC
 32 2018-08-26T03:46:35  *** vexbuy_ has quit IRC
 33 2018-08-26T03:47:13  *** vexbuy has joined #bitcoin-core-dev
 34 2018-08-26T03:51:33  *** vexbuy has quit IRC
 35 2018-08-26T03:53:57  <ossifrage> It seems like MADV_RANDOM is useful if there is less locality to the reads then the size of the prefetch window, but I was under the impression that the prefetch is triggered by the pagefault and not by the mmap() so you actually have to do reads to get the extra pages
 36 2018-08-26T04:06:24  *** Giszmo has quit IRC
 37 2018-08-26T04:17:57  *** Giszmo has joined #bitcoin-core-dev
 38 2018-08-26T04:29:09  *** murrayn has quit IRC
 39 2018-08-26T04:29:59  *** sipa has quit IRC
 40 2018-08-26T04:30:12  *** sipa has joined #bitcoin-core-dev
 41 2018-08-26T04:31:53  *** Krellan has quit IRC
 42 2018-08-26T04:37:40  *** plankers has joined #bitcoin-core-dev
 43 2018-08-26T04:42:15  *** vexbuy has joined #bitcoin-core-dev
 44 2018-08-26T04:45:14  *** Jmabsd has quit IRC
 45 2018-08-26T04:46:38  *** murrayn has joined #bitcoin-core-dev
 46 2018-08-26T04:48:25  *** BashCo has quit IRC
 47 2018-08-26T04:49:45  *** BashCo has joined #bitcoin-core-dev
 48 2018-08-26T04:53:55  *** phwalkr has quit IRC
 49 2018-08-26T04:54:42  *** vexbuy has quit IRC
 50 2018-08-26T05:14:47  <luke-jr> was there a point to https://github.com/bitcoin-core/leveldb/pull/19 considering we limit leveldb to 1000 open files anyway?
 51 2018-08-26T05:17:15  *** vexbuy has joined #bitcoin-core-dev
 52 2018-08-26T05:19:18  <sipa> luke-jr: mmap'ed files aren't kept open after creating the mapping
 53 2018-08-26T05:19:31  <sipa> so they don't count towards the 1000 files limit iirc
 54 2018-08-26T05:24:10  <luke-jr> sipa: but we increased the file limit to 1000 *because* mmaps didn't count toward the fd limit
 55 2018-08-26T05:24:28  <luke-jr> the limit I'm talking about is max_open_files, not ulimit
 56 2018-08-26T05:32:01  *** Taifta has quit IRC
 57 2018-08-26T05:35:50  <sipa> luke-jr: yes, but there is a separate mmap limit it seems
 58 2018-08-26T05:51:01  *** TheHoliestRoger has quit IRC
 59 2018-08-26T05:53:49  *** TheHoliestRoger has joined #bitcoin-core-dev
 60 2018-08-26T05:59:22  *** TheHoliestRoger has quit IRC
 61 2018-08-26T06:02:10  *** TheHoliestRoger has joined #bitcoin-core-dev
 62 2018-08-26T06:02:14  *** Chris_Stewart_5 has quit IRC
 63 2018-08-26T06:02:58  *** ken2812221 has joined #bitcoin-core-dev
 64 2018-08-26T06:03:28  *** Krellan has joined #bitcoin-core-dev
 65 2018-08-26T06:04:27  *** Krellan_ has joined #bitcoin-core-dev
 66 2018-08-26T06:05:22  *** nOgAnOo_ has quit IRC
 67 2018-08-26T06:06:39  *** d_t has joined #bitcoin-core-dev
 68 2018-08-26T06:07:38  *** Giszmo has quit IRC
 69 2018-08-26T06:08:18  *** Krellan has quit IRC
 70 2018-08-26T06:13:02  <luke-jr> sipa: max_open_files effectively limits the number of mmaps too
 71 2018-08-26T06:13:29  <luke-jr> so it seems like the LevelDB mmap limit change is effectively a no-op since we keep the 1000 max_open_files limit :x
 72 2018-08-26T06:14:59  <gmaxwell> luke-jr: it's very much not a no op change.
 73 2018-08-26T06:15:16  <luke-jr> gmaxwell: how does it have an effect?
 74 2018-08-26T06:15:43  <gmaxwell> it causes more mmaps to be open, and less files. you can see this easily with lsof.
 75 2018-08-26T06:16:06  <luke-jr> the mmap limit is enforced by NewRandomAccessFile, which is only ever called by TableCache which enforces the max_open_files limit :/
 76 2018-08-26T06:16:41  <gmaxwell> Well I'm not telling you what the source code says, I'm telling you what it does. :P
 77 2018-08-26T06:16:45  <luke-jr> is there a bug in the LRU cache stuff? :/
 78 2018-08-26T06:18:25  <luke-jr> gmaxwell: is there an easy way to reproduce what you observe?
 79 2018-08-26T06:19:45  <gmaxwell> yea, start a node up no connect, with and without the change and lsof it and look at the number of mmaped files. you might need to crank checkblocks up a bit.
 80 2018-08-26T06:21:20  *** Giszmo has joined #bitcoin-core-dev
 81 2018-08-26T06:32:09  *** plankers has quit IRC
 82 2018-08-26T06:33:32  *** Giszmo has quit IRC
 83 2018-08-26T06:35:52  *** BashCo has quit IRC
 84 2018-08-26T06:36:50  *** BashCo has joined #bitcoin-core-dev
 85 2018-08-26T06:46:04  *** plankers has joined #bitcoin-core-dev
 86 2018-08-26T06:53:42  *** SopaXorzTaker has joined #bitcoin-core-dev
 87 2018-08-26T06:57:30  *** luke-jr has quit IRC
 88 2018-08-26T06:57:45  *** Victorsueca has quit IRC
 89 2018-08-26T06:58:55  *** Victorsueca has joined #bitcoin-core-dev
 90 2018-08-26T07:03:31  *** vexbuy has quit IRC
 91 2018-08-26T07:05:02  *** d9b4bef9 has quit IRC
 92 2018-08-26T07:05:34  *** BashCo has quit IRC
 93 2018-08-26T07:06:08  *** d9b4bef9 has joined #bitcoin-core-dev
 94 2018-08-26T07:06:19  *** BashCo has joined #bitcoin-core-dev
 95 2018-08-26T07:09:30  *** plankers has quit IRC
 96 2018-08-26T07:10:43  *** Taifta has joined #bitcoin-core-dev
 97 2018-08-26T07:11:33  *** BashCo has quit IRC
 98 2018-08-26T07:14:12  *** BashCo has joined #bitcoin-core-dev
 99 2018-08-26T07:15:14  *** luke-jr has joined #bitcoin-core-dev
100 2018-08-26T07:15:22  <wumpus> 0.17.0rc2 binaries up https://bitcoincore.org/bin/bitcoin-core-0.17.0/test.rc2/
101 2018-08-26T07:19:56  *** vexbuy has joined #bitcoin-core-dev
102 2018-08-26T07:23:38  *** d_t has quit IRC
103 2018-08-26T07:37:25  *** vexbuy has quit IRC
104 2018-08-26T07:48:33  *** vexbuy has joined #bitcoin-core-dev
105 2018-08-26T07:50:03  *** BashCo has quit IRC
106 2018-08-26T07:50:16  <wumpus> luke-jr: the only reason that the change to num files was acceptable is because leveldb, at least the version in our tree, doesn't actually keep more files open (on platforms with mmap) see also https://github.com/bitcoin/bitcoin/pull/12495#issuecomment-377228329
107 2018-08-26T07:50:45  *** BashCo has joined #bitcoin-core-dev
108 2018-08-26T07:53:03  <luke-jr> gmaxwell: cannot reproduce with -checkblocks=10000 :/
109 2018-08-26T07:53:44  <luke-jr> wumpus: I understand that. I just don't understand why increase the mmap-specific limit, without increasing the num files limit that is required for any mmap
110 2018-08-26T07:53:58  <luke-jr> we will never have more than 1001 mmaps because the files limit is 1000
111 2018-08-26T08:04:27  <wumpus> --I don't think mappings into closed files count against the open files ulimit
112 2018-08-26T08:06:39  *** vexbuy has quit IRC
113 2018-08-26T08:07:17  *** vexbuy has joined #bitcoin-core-dev
114 2018-08-26T08:07:51  *** Rootsudo has quit IRC
115 2018-08-26T08:09:58  <sipa> wumpus: luke's point is that internally in leveldb, the mmapped files count towards the open file limit
116 2018-08-26T08:10:17  <wumpus> but we increased leveldb's limit
117 2018-08-26T08:11:11  <wumpus> though to be honest I'm completely confused now with all the different limites, how they interact, and how they're supposed to interact
118 2018-08-26T08:11:34  <sipa> wumpus: yeah...
119 2018-08-26T08:12:09  <wumpus> given all the confusion about this, I'm starting to think it might be better to revert these changes, I like understanding things even if it means not getting every last possible % of performance
120 2018-08-26T08:12:19  <sipa> so there is a configurable leveldb max open files limit (which is 1000), and a mmap limit (which is hardcoded to 1000, but we raised it to 4096)
121 2018-08-26T08:12:47  <wumpus> and seemingly it now relies on all kinds of undocumented behavior
122 2018-08-26T08:12:59  <sipa> luke says (i haven't verified) that mmapped files count towards that limit... which is inconsistent with our observations
123 2018-08-26T08:13:17  <sipa> i'm just clarifying my understanding as there seems to be some confusion
124 2018-08-26T08:13:18  *** Krellan_ has quit IRC
125 2018-08-26T08:13:27  <wumpus> thanks
126 2018-08-26T08:13:45  <gmaxwell> right if we revert though we should revert both changes.
127 2018-08-26T08:13:51  <wumpus> yes
128 2018-08-26T08:13:57  <gmaxwell> I wouldn't oppose doing that for 0.17.
129 2018-08-26T08:13:59  <sipa> both? what other chanfe?
130 2018-08-26T08:14:01  *** Krellan has joined #bitcoin-core-dev
131 2018-08-26T08:14:10  <sipa> i saw the mmap increaee
132 2018-08-26T08:14:11  <gmaxwell> sipa: the fd count change and the mmap limit change.
133 2018-08-26T08:15:30  <gmaxwell> first FD count was increased with an argument that basically it wouldn't use more FDs due to using mmap.  But then we found out that actually when it hit the mmap limit, it started actually using more FDs.
134 2018-08-26T08:15:47  <wumpus> #12495 and https://github.com/bitcoin-core/leveldb/pull/19
135 2018-08-26T08:15:50  <gribble> https://github.com/bitcoin/bitcoin/issues/12495 | Increase LevelDB max_open_files by eklitzke · Pull Request #12495 · bitcoin/bitcoin · GitHub
136 2018-08-26T08:16:05  <gmaxwell> Since the lack of mmaps is probably also bad for performance (though I don't think anyone benchmarked specifically for that), it seemed like it made sense to increase mmaps rather than revert the fd count change.
137 2018-08-26T08:16:19  <wumpus> yes
138 2018-08-26T08:16:47  <wumpus> "By default LevelDB will only mmap() up to 1000 ldb files for reading and then fall back
139 2018-08-26T08:16:50  <wumpus> to using file desciptors."
140 2018-08-26T08:17:00  <wumpus> that pull 19 is *supposed* to change that
141 2018-08-26T08:17:29  <wumpus> (from 1000 to 4096, which is still an arbitrary value) -- maybe luke-jr is using a system leveldb?
142 2018-08-26T08:17:45  <sipa> where did we increase the FDs?
143 2018-08-26T08:18:02  <wumpus> didn't that happen in #12495?
144 2018-08-26T08:18:04  <gribble> https://github.com/bitcoin/bitcoin/issues/12495 | Increase LevelDB max_open_files by eklitzke · Pull Request #12495 · bitcoin/bitcoin · GitHub
145 2018-08-26T08:18:14  <wumpus> I don't know anymore, sorry, should probably get coffee
146 2018-08-26T08:18:53  <gmaxwell> wumpus: 12495, as you say.
147 2018-08-26T08:19:05  <sipa> ah yes
148 2018-08-26T08:19:09  <sipa> hmm
149 2018-08-26T08:19:20  <sipa> we were using 64 files only before
150 2018-08-26T08:19:36  <sipa> but using 1000 there seems pretty standard practice; it's leveldb's own default
151 2018-08-26T08:19:51  <luke-jr> wumpus: not for this testing
152 2018-08-26T08:19:55  <wumpus> sipa: correct
153 2018-08-26T08:20:21  <gmaxwell> sipa: after increasing it, we were using a lot more file descriptors.
154 2018-08-26T08:20:23  <luke-jr> wumpus: but in any case, for testing this, I'm changing the 4096 back to 1000, and can't get LDB to use a fd
155 2018-08-26T08:20:39  <gmaxwell> luke-jr: are you using some system leveldb?
156 2018-08-26T08:20:59  <luke-jr> gmaxwell: like I said, not for this; but in any case, it would be the same 1000 mmaps
157 2018-08-26T08:21:47  <gmaxwell> luke-jr: dunno why it didn't reproduce for you, I saw leveldb using FDs, after using 1000 maps... and not anymore after increasing the map limit.
158 2018-08-26T08:21:55  <luke-jr> hmm
159 2018-08-26T08:22:02  <gmaxwell> sipa also saw it using fd's on his node.
160 2018-08-26T08:22:09  <luke-jr> testnet or mainnet?
161 2018-08-26T08:22:13  <gmaxwell> mainnet
162 2018-08-26T08:22:14  <sipa> mainnet
163 2018-08-26T08:22:19  <luke-jr> I wonder if that's why. :x
164 2018-08-26T08:22:21  <gmaxwell> testnet probably doesn't have enough ldb files.
165 2018-08-26T08:22:26  <luke-jr> ah
166 2018-08-26T08:22:49  <sipa> how large is testnet's utxo set?
167 2018-08-26T08:24:53  *** PatBoy has quit IRC
168 2018-08-26T08:26:12  *** PatBoy has joined #bitcoin-core-dev
169 2018-08-26T08:35:52  <wumpus> "disk_size": 1025042445, as of height 1384188 (it is at 1392776 now, don't know how much it changed since then)
170 2018-08-26T08:48:45  <luke-jr> hopefully I won't "damage" my mainnet .bitcoin trying this.. :x
171 2018-08-26T08:49:05  <luke-jr> (I still run 0.13 wallets from time to time)
172 2018-08-26T08:50:34  *** plankers has joined #bitcoin-core-dev
173 2018-08-26T08:50:41  <luke-jr> looks like the UTXO db upgrade isn't triggering it, at least
174 2018-08-26T08:55:24  *** plankers has quit IRC
175 2018-08-26T09:04:44  * luke-jr facepalms. After UTXO upgrade, it says I need to reindex >_<
176 2018-08-26T09:09:22  *** plankers has joined #bitcoin-core-dev
177 2018-08-26T09:10:00  <wumpus> does it say why / give an error message? the whole point of the utxo upgrade would be to not reindex, unless it fails
178 2018-08-26T09:11:54  <provoostenator> If you're traveling and stuck with a slow laptop, the following gist lets you compile remotely and then connect via SSH tunnel: https://gist.github.com/Sjors/fda9b601e9464f61332ac6490f487c0a (improvements welcome)
179 2018-08-26T09:26:16  <wumpus> provoostenator: nit: you could pass NO_UPNP to the depends build instead of --with-miniupnpc=no
180 2018-08-26T09:26:32  <wumpus> not that libminiupnp takes long to compile
181 2018-08-26T09:27:10  <provoostenator> wumpus: thanks, in this case it's probably easier to keep the configure line flexible.
182 2018-08-26T09:27:17  *** murrayn has quit IRC
183 2018-08-26T09:28:07  <provoostenator> I first tried copying bitcoind over and just running it locally but that resulted in a world of pain, probably because the two systems aren't perfectly identical.
184 2018-08-26T09:28:12  *** plankers has quit IRC
185 2018-08-26T09:28:46  <wumpus> yes copying executables between linux systems is fraught with problems
186 2018-08-26T09:29:02  <provoostenator> It's works fine with c-lightning though.
187 2018-08-26T09:29:16  <wumpus> though it should work with depends and the glibc compat enabled
188 2018-08-26T09:29:53  <provoostenator> I tried with depends as well, haven't tried glibc compat, might try that next time. But this also prevents me from accidentally syncing the chain over 4G.
189 2018-08-26T09:30:03  <wumpus> -static-libstdc++ helps as well
190 2018-08-26T09:30:24  <wumpus> (but only if all c++ dependencies are static, which is the case with a depends build but not otherwise)
191 2018-08-26T09:32:17  <wumpus> then you end up with an executable that is as-good-as-static and it should be portable to non-ancient linux versions with the same CPU architecture
192 2018-08-26T09:33:14  <wumpus> of course, that's the theory, there are still some things that can interfere for example if glibc is built with different options (I vaguely remember redhat and some security/hardening flag), or the target platform has a different libc
193 2018-08-26T09:35:17  <wumpus> this is one reason they came up with "snap"s which are executables that essentially carry around part of the system dependencies with them in a container
194 2018-08-26T09:35:39  <wumpus> (along with some security/isolation features)
195 2018-08-26T09:35:45  <provoostenator> Being able to run locally is especially useful for QT, and with the multiprocess stuff it would be nice to run bitcoind remote and QT locally.
196 2018-08-26T09:36:31  <wumpus> aanyhow if your two systems have the same release of the same distro, and are at the same update level, it should work too, that's probably easiest for most...
197 2018-08-26T09:50:05  <luke-jr> wumpus: I reflink'd the blocks/ and chainstate/ dirs to a new datadir. I'm guessing that simply wasn't sufficient somehow
198 2018-08-26T09:50:30  *** plankers has joined #bitcoin-core-dev
199 2018-08-26T09:52:52  *** murrayn has joined #bitcoin-core-dev
200 2018-08-26T09:54:34  *** BashCo has quit IRC
201 2018-08-26T09:54:57  <luke-jr> oh well, I'll let it sync overnight and see if it hits the breakpoint
202 2018-08-26T09:55:12  <luke-jr> env_posix.cc:379 if anyone else who can reproduce it wants to try
203 2018-08-26T09:55:36  <luke-jr> (if you hit it, maybe see if you can work out why it got there..)
204 2018-08-26T09:56:03  *** BashCo has joined #bitcoin-core-dev
205 2018-08-26T10:18:13  *** SopaXorzTaker has quit IRC
206 2018-08-26T10:20:57  *** Krellan has quit IRC
207 2018-08-26T10:21:43  *** Krellan has joined #bitcoin-core-dev
208 2018-08-26T10:33:31  *** as1nc has quit IRC
209 2018-08-26T10:46:30  *** JackH has joined #bitcoin-core-dev
210 2018-08-26T11:01:42  *** vexbuy has quit IRC
211 2018-08-26T11:14:37  *** vexbuy has joined #bitcoin-core-dev
212 2018-08-26T11:23:18  <provoostenator> Any volunteer for making a PR based on this commit? https://github.com/bitcoin/bitcoin/commit/a9db3dada0119c183d16627805e90c4dbca05c6a
213 2018-08-26T11:23:46  <provoostenator> I left it out of my rebase #13937 because I'd rather not touch thread stuff.
214 2018-08-26T11:23:47  <gribble> https://github.com/bitcoin/bitcoin/issues/13937 | Track best-possible-headers (TheBlueMatt) by Sjors · Pull Request #13937 · bitcoin/bitcoin · GitHub
215 2018-08-26T11:32:53  *** belcher_ has joined #bitcoin-core-dev
216 2018-08-26T11:32:57  *** SopaXorzTaker has joined #bitcoin-core-dev
217 2018-08-26T11:37:37  *** str4d has joined #bitcoin-core-dev
218 2018-08-26T11:41:02  *** cdecker has joined #bitcoin-core-dev
219 2018-08-26T11:51:14  *** str4d has quit IRC
220 2018-08-26T11:51:32  *** str4d has joined #bitcoin-core-dev
221 2018-08-26T12:06:26  <provoostenator> Never mind, that's already been done in #13023
222 2018-08-26T12:06:29  <gribble> https://github.com/bitcoin/bitcoin/issues/13023 | Fix some concurrency issues in ActivateBestChain() by skeees · Pull Request #13023 · bitcoin/bitcoin · GitHub
223 2018-08-26T12:13:33  *** Emcy has quit IRC
224 2018-08-26T12:19:12  *** sipa has quit IRC
225 2018-08-26T12:19:32  *** sipa has joined #bitcoin-core-dev
226 2018-08-26T12:20:43  *** sipa has quit IRC
227 2018-08-26T12:28:30  *** Krellan has quit IRC
228 2018-08-26T12:29:20  *** Krellan has joined #bitcoin-core-dev
229 2018-08-26T12:31:23  *** BashCo has quit IRC
230 2018-08-26T12:33:49  *** BashCo has joined #bitcoin-core-dev
231 2018-08-26T12:34:35  *** sipa has joined #bitcoin-core-dev
232 2018-08-26T12:36:19  *** farmerwampum has joined #bitcoin-core-dev
233 2018-08-26T12:36:57  *** farmerwampum has quit IRC
234 2018-08-26T12:37:19  *** farmerwampum has joined #bitcoin-core-dev
235 2018-08-26T12:39:29  *** wxss has quit IRC
236 2018-08-26T12:47:22  *** csknk has joined #bitcoin-core-dev
237 2018-08-26T12:58:43  *** Taifta has quit IRC
238 2018-08-26T13:02:07  *** Guyver2 has joined #bitcoin-core-dev
239 2018-08-26T13:15:00  *** Taifta has joined #bitcoin-core-dev
240 2018-08-26T13:15:01  *** Victorsueca has quit IRC
241 2018-08-26T13:16:14  *** Victorsueca has joined #bitcoin-core-dev
242 2018-08-26T13:28:22  *** Taifta has quit IRC
243 2018-08-26T13:28:23  *** BashCo has quit IRC
244 2018-08-26T13:28:35  *** booyah has quit IRC
245 2018-08-26T13:29:46  *** booyah has joined #bitcoin-core-dev
246 2018-08-26T13:31:06  *** booyah has quit IRC
247 2018-08-26T13:31:09  *** BashCo has joined #bitcoin-core-dev
248 2018-08-26T13:32:46  *** vexbuy has quit IRC
249 2018-08-26T13:36:54  *** vexbuy has joined #bitcoin-core-dev
250 2018-08-26T13:44:48  *** booyah has joined #bitcoin-core-dev
251 2018-08-26T13:50:37  *** Emcy has joined #bitcoin-core-dev
252 2018-08-26T13:55:30  *** Chris_Stewart_5 has joined #bitcoin-core-dev
253 2018-08-26T13:56:38  *** Taifta has joined #bitcoin-core-dev
254 2018-08-26T14:06:19  <luke-jr> wumpus: heh, I was tired last night; the reason it needed a reindex (or I guess re-IBD) seems to be that I named the blocks directory "newblocks" XD
255 2018-08-26T14:06:29  <luke-jr> err, "tmpblocks"
256 2018-08-26T14:25:00  *** Taifta has quit IRC
257 2018-08-26T14:34:04  *** Emcy has quit IRC
258 2018-08-26T14:36:25  *** Krellan has quit IRC
259 2018-08-26T14:37:27  *** Krellan has joined #bitcoin-core-dev
260 2018-08-26T14:39:02  *** elichai2 has joined #bitcoin-core-dev
261 2018-08-26T14:51:35  *** vexbuy has quit IRC
262 2018-08-26T14:54:00  *** vexbuy has joined #bitcoin-core-dev
263 2018-08-26T15:02:51  *** vexbuy has quit IRC
264 2018-08-26T15:03:29  *** vexbuy has joined #bitcoin-core-dev
265 2018-08-26T15:06:51  *** spinza has quit IRC
266 2018-08-26T15:06:57  *** Chris_Stewart_5 has quit IRC
267 2018-08-26T15:20:30  *** spinza has joined #bitcoin-core-dev
268 2018-08-26T15:29:30  *** BashCo has quit IRC
269 2018-08-26T15:30:12  *** BashCo has joined #bitcoin-core-dev
270 2018-08-26T15:34:46  *** Chris_Stewart_5 has joined #bitcoin-core-dev
271 2018-08-26T15:46:47  *** plankers has quit IRC
272 2018-08-26T15:48:38  *** vexbuy_ has joined #bitcoin-core-dev
273 2018-08-26T15:48:59  *** vexbuy has quit IRC
274 2018-08-26T15:49:39  *** Krellan has quit IRC
275 2018-08-26T15:50:12  *** Krellan has joined #bitcoin-core-dev
276 2018-08-26T15:55:09  *** Krellan has quit IRC
277 2018-08-26T15:55:50  *** Emcy has joined #bitcoin-core-dev
278 2018-08-26T15:56:01  *** vexbuy_ has quit IRC
279 2018-08-26T15:56:37  *** vexbuy has joined #bitcoin-core-dev
280 2018-08-26T16:07:48  *** csknk has quit IRC
281 2018-08-26T16:12:15  *** Krellan has joined #bitcoin-core-dev
282 2018-08-26T16:13:23  *** sipa_ has joined #bitcoin-core-dev
283 2018-08-26T16:14:09  *** sipa has quit IRC
284 2018-08-26T16:14:39  <Chris_Stewart_5> BIP157 doesn't currently have an open PR does it?
285 2018-08-26T16:15:27  *** intcat has quit IRC
286 2018-08-26T16:17:14  *** Krellan has quit IRC
287 2018-08-26T16:18:38  <instagibbs> Chris_Stewart_5, https://github.com/bitcoin/bitcoin/pull/13144
288 2018-08-26T16:18:54  *** intcat has joined #bitcoin-core-dev
289 2018-08-26T16:26:05  <Chris_Stewart_5> instagibbs: So that is just a refactor PR to make it easier to implement BIP157 right?
290 2018-08-26T16:29:19  *** BashCo has quit IRC
291 2018-08-26T16:30:15  *** BashCo has joined #bitcoin-core-dev
292 2018-08-26T16:36:03  <instagibbs> doh wrong one
293 2018-08-26T16:36:04  <instagibbs> one sec
294 2018-08-26T16:36:47  <instagibbs> Chris_Stewart_5, merged two hours ago, didnt see that https://github.com/bitcoin/bitcoin/pull/12254
295 2018-08-26T16:37:02  <instagibbs> nothing open after that
296 2018-08-26T16:37:16  <Chris_Stewart_5> Hmm, is that only BIP158 though? I don't think that BIP158 includes any p2p networking stuff
297 2018-08-26T16:37:26  <Chris_Stewart_5> BIP157 seems to have the semantics for that
298 2018-08-26T16:48:12  *** Krellan has joined #bitcoin-core-dev
299 2018-08-26T16:50:51  *** itaseski has joined #bitcoin-core-dev
300 2018-08-26T16:55:44  *** BashCo has quit IRC
301 2018-08-26T16:57:55  *** Taifta has joined #bitcoin-core-dev
302 2018-08-26T16:58:09  *** BashCo has joined #bitcoin-core-dev
303 2018-08-26T17:18:50  *** reallll has joined #bitcoin-core-dev
304 2018-08-26T17:21:11  *** Victorsueca has quit IRC
305 2018-08-26T17:21:11  *** BashCo has quit IRC
306 2018-08-26T17:21:58  *** BashCo has joined #bitcoin-core-dev
307 2018-08-26T17:22:25  *** Victorsueca has joined #bitcoin-core-dev
308 2018-08-26T17:22:43  *** belcher_ has quit IRC
309 2018-08-26T17:34:41  *** promag has joined #bitcoin-core-dev
310 2018-08-26T17:36:23  *** reallll has quit IRC
311 2018-08-26T17:41:35  *** belcher_ has joined #bitcoin-core-dev
312 2018-08-26T17:42:30  *** promag has quit IRC
313 2018-08-26T18:03:44  *** vexbuy_ has joined #bitcoin-core-dev
314 2018-08-26T18:05:25  *** vexbuy has quit IRC
315 2018-08-26T18:10:34  *** sipa_ is now known as sipa
316 2018-08-26T18:15:02  *** BashCo has quit IRC
317 2018-08-26T18:17:18  *** BashCo has joined #bitcoin-core-dev
318 2018-08-26T18:24:45  *** promag has joined #bitcoin-core-dev
319 2018-08-26T18:26:17  *** promag has quit IRC
320 2018-08-26T18:28:29  <jimpo> Chris_Stewart_5: No, I'm working on the next PR towards BIP 157 support
321 2018-08-26T18:28:39  <jimpo> Which is building filter indexes
322 2018-08-26T18:28:45  *** promag has joined #bitcoin-core-dev
323 2018-08-26T18:33:00  *** promag has quit IRC
324 2018-08-26T18:34:16  *** Krellan has quit IRC
325 2018-08-26T18:34:36  *** Krellan has joined #bitcoin-core-dev
326 2018-08-26T18:37:02  *** jb55 has joined #bitcoin-core-dev
327 2018-08-26T18:39:33  <Chris_Stewart_5> gmaxwell: PIR? Without a PIR method of fetching blocks, use of this is still toxic to user privacy
328 2018-08-26T18:43:32  <sipa> Chris_Stewart_5: anything that doesn't fetch all blocks will leak some information, inevitably
329 2018-08-26T18:45:58  <Chris_Stewart_5> What is the acronym? PIR -- i'm guessing it is not Passive Infrared Sensor hah
330 2018-08-26T18:47:03  <sipa> private information retrieval
331 2018-08-26T18:53:00  *** Taifta has quit IRC
332 2018-08-26T19:01:42  *** BashCo has quit IRC
333 2018-08-26T19:03:17  *** BashCo has joined #bitcoin-core-dev
334 2018-08-26T19:03:17  <Chris_Stewart_5> jimpo: These test vectors seem to be a dead link https://github.com/bitcoin/bips/blob/master/bip-0158.mediawiki#appendix-c-test-vectors
335 2018-08-26T19:03:38  <Chris_Stewart_5> at least the first link
336 2018-08-26T19:07:51  *** Salve has joined #bitcoin-core-dev
337 2018-08-26T19:18:44  *** elichai2 has quit IRC
338 2018-08-26T19:25:26  *** BashCo has quit IRC
339 2018-08-26T19:26:21  *** BashCo has joined #bitcoin-core-dev
340 2018-08-26T19:33:42  *** BashCo has quit IRC
341 2018-08-26T19:41:15  *** Salve has quit IRC
342 2018-08-26T19:53:44  *** rex4539 has quit IRC
343 2018-08-26T19:54:21  *** Scrat has joined #bitcoin-core-dev
344 2018-08-26T19:56:50  *** Scrat has quit IRC
345 2018-08-26T20:04:15  *** rex4539 has joined #bitcoin-core-dev
346 2018-08-26T20:05:37  *** rex4539 has joined #bitcoin-core-dev
347 2018-08-26T20:06:31  *** JackH has quit IRC
348 2018-08-26T20:08:54  *** kallisteiros has joined #bitcoin-core-dev
349 2018-08-26T20:09:11  *** hebasto has quit IRC
350 2018-08-26T20:10:08  *** owowo has quit IRC
351 2018-08-26T20:10:40  *** SopaXorzTaker has quit IRC
352 2018-08-26T20:16:36  <jimpo> Oh, thanks, will open a PR to fix that. Need to add a test block anyway.
353 2018-08-26T20:17:33  <Chris_Stewart_5> Thanks!
354 2018-08-26T20:43:44  *** Krellan has quit IRC
355 2018-08-26T20:44:46  *** Krellan has joined #bitcoin-core-dev
356 2018-08-26T21:05:01  *** kallisteiros has quit IRC
357 2018-08-26T21:14:54  *** itaseski has quit IRC
358 2018-08-26T21:15:34  *** chainhead has joined #bitcoin-core-dev
359 2018-08-26T21:41:16  *** promag has joined #bitcoin-core-dev
360 2018-08-26T21:44:26  *** promag has quit IRC
361 2018-08-26T21:44:45  *** promag has joined #bitcoin-core-dev
362 2018-08-26T21:51:02  *** d9b4bef9 has quit IRC
363 2018-08-26T21:52:08  *** d9b4bef9 has joined #bitcoin-core-dev
364 2018-08-26T21:53:07  *** belcher_ has quit IRC
365 2018-08-26T22:03:27  *** plankers has joined #bitcoin-core-dev
366 2018-08-26T22:06:55  *** promag has quit IRC
367 2018-08-26T22:07:30  *** promag has joined #bitcoin-core-dev
368 2018-08-26T22:12:04  *** promag has quit IRC
369 2018-08-26T22:18:05  *** promag has joined #bitcoin-core-dev
370 2018-08-26T22:28:12  <luke-jr> gmaxwell: sipa: wumpus: sync'd up to 341411 and still no non-mmap opens from LDB..
371 2018-08-26T22:29:10  <sipa> luke-jr: ?
372 2018-08-26T22:29:15  <sipa> what does that mean
373 2018-08-26T22:29:50  <gmaxwell> 341411 is a long time ago.
374 2018-08-26T22:30:29  <gmaxwell> sipa: he's trying to reproduce the behavior ossifrage, you, and I saw where leveldb would hit the mmap limit and start using more FDs.
375 2018-08-26T22:30:36  <sipa> you'll only exceed 1000 files/mmaps once you're above ~2GB UTXO set size
376 2018-08-26T22:30:43  <sipa> as each file is 2MB ish
377 2018-08-26T22:31:55  <luke-jr> hmm
378 2018-08-26T22:32:25  <ossifrage> pmap 24199 | grep \.ldb | wc -l    == 1894    (for a node up ~26 days)
379 2018-08-26T22:34:52  <sipa> luke-jr: so you'll need to get to somewhere in 2017 at least
380 2018-08-26T22:35:23  * luke-jr retries the UTXO upgrade, this time with the blocks dir named correctly
381 2018-08-26T22:35:25  <ossifrage> That includes the txindex and the chainstate
382 2018-08-26T22:36:46  <sipa> ossifrage: per database; txindex and chainstate are separate
383 2018-08-26T22:37:28  <gmaxwell> has anyone just tried making a dummy program that mmaps a couple thousand files to see what the RES shows up as?
384 2018-08-26T22:38:14  *** Victorsueca has quit IRC
385 2018-08-26T22:39:25  *** Victorsueca has joined #bitcoin-core-dev
386 2018-08-26T22:40:01  <ossifrage> is having a larger resident size really an issue, you only get those pages if memory is available, the os should evict them right quick under memory pressure
387 2018-08-26T22:40:45  <sipa> presumably, indeed
388 2018-08-26T22:41:23  <sipa> it may be just an methodology problem, what rss doesn't exactly corresponds to what we care about
389 2018-08-26T22:41:45  <gmaxwell> ossifrage: if thats actually the case, thing things are fine.
390 2018-08-26T22:43:52  <gmaxwell> and if so we probably need to figure out how to start tracking "total dirty pages" or something.
391 2018-08-26T22:44:37  <ossifrage> leveldb only uses mmap for reading, so the dirty pages from all the maps should be 0
392 2018-08-26T22:44:45  <gmaxwell> Right.
393 2018-08-26T22:44:55  <luke-jr> personally, I only look at ps's "size" metric
394 2018-08-26T22:45:25  <luke-jr>        size        SIZE      approximate amount of swap space that would be required if the process were to dirty all
395 2018-08-26T22:45:26  <luke-jr>                              writable pages and then be swapped out.  This number is very rough!
396 2018-08-26T22:45:38  <gmaxwell> ossifrage: so to catch you up. We have infrastructure that benchmarks the software regularly and charts performance... so we find out if some commit accidentally hurts performance.
397 2018-08-26T22:46:03  <gmaxwell> ossifrage: we noticed around the time of the 0.17 feature freeze that the claimed memory usage suddenly went up 20%!
398 2018-08-26T22:46:20  <gmaxwell> during the week where a lot of last minute stuff got merged.
399 2018-08-26T22:46:37  <gmaxwell> Bisection turned up your change to increase the leveldb maps count.
400 2018-08-26T22:46:48  <ossifrage> gmaxwell, where does the claimed memory measurement come from?
401 2018-08-26T22:47:44  <gmaxwell> Thats what we're trying to determine. after finding out that mmap automatically pages in some of the file tried madvising DONTNEED all the maps, but that didn't seem to fix it.
402 2018-08-26T22:48:08  <gmaxwell> It might well be that the increased usage is totally fictional... that it's just counting paged-in non-dirty pages.... and they're not real memory pressure and we don't need to worry about them.
403 2018-08-26T22:48:17  <ossifrage> I'm writing a mmap() test case out of morbid curiosity
404 2018-08-26T22:49:05  <gmaxwell> Or maybe, though we haven't spotted it yet, leveldb is using 200kb of god knows what per open map. :P
405 2018-08-26T22:49:40  <gmaxwell> Meanwhile, luke-jr went and read more of the leveldb code and believes that it can't have more mmaps open than the FD limit, and so he thinks the mmap increase should be a noop.
406 2018-08-26T22:50:05  <gmaxwell> (which it clearly isn't, but it would be good if everyone was on the same page about how leveldb works)
407 2018-08-26T22:50:08  *** Chris_Stewart_5 has quit IRC
408 2018-08-26T22:53:50  <ossifrage> I currently have 992 chainstate ldb files mapped and 859 txindex and 43 blocks/indexes ldb files, but it took a fairly long uptime for the original problem to show up
409 2018-08-26T22:53:57  *** Krellan has quit IRC
410 2018-08-26T22:54:42  *** Krellan has joined #bitcoin-core-dev
411 2018-08-26T22:58:22  *** Guyver2 has quit IRC
412 2018-08-26T23:00:35  <ossifrage> I don't see much of a change between the htop reported RES size between mapping 1 txindex ldb file and mmaping all 2041 of them
413 2018-08-26T23:00:47  <luke-jr> gmaxwell: well, I can reproduce it now, but I haven't figure out why yet
414 2018-08-26T23:00:53  <ossifrage> (but I never read from the map)
415 2018-08-26T23:02:04  <gmaxwell> ossifrage: k? try reading like, the last byte of them or something?
416 2018-08-26T23:02:24  <ossifrage> The VIRT line goes up as expected but RES stays fairly small
417 2018-08-26T23:03:24  <luke-jr> hmm, is everyone reproducing this using txindex?
418 2018-08-26T23:04:25  <ossifrage> gmaxwell, yeah that caused the RES size to approach the VIRT size (read the last byte)
419 2018-08-26T23:05:09  <sipa> gmaxwell: "it can't have more mmaps open than the FD limit" -> s/FD limit/max_open_files setting/
420 2018-08-26T23:05:27  <gmaxwell> now, try MADV_RANDOM ?
421 2018-08-26T23:05:34  <gmaxwell> ossifrage: ^
422 2018-08-26T23:05:51  <ossifrage> I read the last byte, so I'm not sure what it would prefetch :-)
423 2018-08-26T23:06:09  *** andytoshi has joined #bitcoin-core-dev
424 2018-08-26T23:06:14  <gmaxwell> I'm guessing that it's prefetching the whole thing.
425 2018-08-26T23:06:25  <gmaxwell> andytoshi: HI!
426 2018-08-26T23:07:23  <andytoshi> gmaxwell: hiya, sorry, my server shit out this morning .. i have no idea why, the logs show everything clear, but it was apparently completely locked up even to a local terminal user
427 2018-08-26T23:07:28  <andytoshi> so i called my dad to hard-boot it and it seems ok now
428 2018-08-26T23:07:46  <ossifrage> MADV_RANDOM didn't seem to change the bump in RES caused by reading last byte
429 2018-08-26T23:08:00  *** promag has quit IRC
430 2018-08-26T23:09:02  <gmaxwell> ossifrage: saddness.  ... so run pmap on the process, and see how many dirty pages it says you have, presumably like none.
431 2018-08-26T23:11:03  *** Chris_Stewart_5 has joined #bitcoin-core-dev
432 2018-08-26T23:12:38  <sipa> gmaxwell: mmap is only used in leveldb for read-only files; none should ever be dirty
433 2018-08-26T23:12:54  <gmaxwell> Interesting, there are only 41.3 BTC stored in bare multisig outputs, but there are 410k of them in total.
434 2018-08-26T23:13:22  <gmaxwell> sipa: I know but I also would have expected that increasing the mmap limit wouldn't have increased RES. :)
435 2018-08-26T23:13:51  <ossifrage> One sec, I might have screwed up...
436 2018-08-26T23:14:12  <gmaxwell> I'm going to suggest we start charting the total dirty pages from pmap... and not worry about RES...
437 2018-08-26T23:14:13  <luke-jr> it looks like the mmap limit is global
438 2018-08-26T23:18:32  <ossifrage> Okay, it looks like MADV_RANDOM isn't helping, but just reading the last byte does cause the RES size to jump quite a bit (but I'm not sure how much I expect it to go up)
439 2018-08-26T23:18:43  <ossifrage> at least 1 page per file?
440 2018-08-26T23:18:53  <sipa> yes, 1 page per file is expected
441 2018-08-26T23:21:41  <gmaxwell> I expect that 1 page per file is what you should get, which is a modest amount of memory.
442 2018-08-26T23:21:49  <gmaxwell> in bitcoin we're seeing an increase of hundreds of meg.
443 2018-08-26T23:21:56  <gmaxwell> way more than one page per file.
444 2018-08-26T23:23:55  <ossifrage> When reading the first byte, MAD_RANDOM does decrease the RES usage
445 2018-08-26T23:28:33  <ossifrage> Even with MADV_RANDOM, it would depend entirely on how many pages are faulted in by ldb
446 2018-08-26T23:34:20  *** promag has joined #bitcoin-core-dev
447 2018-08-26T23:37:11  <ossifrage> Err, it looks like for me it is prefetching ~64K
448 2018-08-26T23:43:34  <gmaxwell> 64K with MADV_RANDOM?
449 2018-08-26T23:44:05  <ossifrage> I think I might be suffering from previous state problems
450 2018-08-26T23:45:57  <ossifrage> Okay, dropping the cache and madv_random does seem to only read 4K for most of the files mapped (some have more for some odd reason)
451 2018-08-26T23:47:08  <gmaxwell> oh interesting, so, if you cause the files to be paged in, then start a program with madv_random it will reflect higher res becuase you mmaped files that were already in cache?
452 2018-08-26T23:47:10  <ossifrage> Yeah, without MADV_RANDOM all the files have 64K RES when I read the 1st byte
453 2018-08-26T23:47:27  <ossifrage> echo 3 > /proc/sys/vm/drop_caches     between runs
454 2018-08-26T23:47:51  <ossifrage> Yeah I was getting confusing results until I realized that
455 2018-08-26T23:47:54  <gmaxwell> Cool so also we might not have seen the effects of MADV_RANDOM when testing with actual leveldb due to that.
456 2018-08-26T23:48:15  <ossifrage> Depends heavily on the specific hardware yeah
457 2018-08-26T23:48:43  <ossifrage> But drop_caches is a mighty big hammer
458 2018-08-26T23:49:14  <gmaxwell> sure, just in terms of "huh why did MADV_RANDOM seem to do nothing"... now we know.
459 2018-08-26T23:50:19  <ossifrage> I suspect the minimum amount read without MADV_RANDOM depends on things like the filesystem and the underlying block device
460 2018-08-26T23:51:26  <ossifrage> Like reading 1 page with a SSD is no big deal vs 1 page with a RAID5 md array
461 2018-08-26T23:55:28  <ossifrage> Any ideas how much leveldb actually reads at a time?
462 2018-08-26T23:55:43  *** Victorsueca has quit IRC
463 2018-08-26T23:56:35  <ossifrage> It would have to touch at least some of the index and then the actual data it wants to read
464 2018-08-26T23:56:55  *** Victorsueca has joined #bitcoin-core-dev
465 2018-08-26T23:57:48  <gmaxwell> it does a bisection search in the index and then I think it only reads the record of interest.