1 2015-11-10T00:15:12  *** zooko` has joined #bitcoin-core-dev
  2 2015-11-10T00:16:29  *** zooko has quit IRC
  3 2015-11-10T00:17:15  *** zooko` is now known as zooko
  4 2015-11-10T00:28:57  *** zooko has quit IRC
  5 2015-11-10T00:30:02  *** zooko` has joined #bitcoin-core-dev
  6 2015-11-10T00:31:34  *** GAit has joined #bitcoin-core-dev
  7 2015-11-10T00:33:01  *** zooko` is now known as zokoo
  8 2015-11-10T00:33:02  *** zokoo is now known as zooko
  9 2015-11-10T00:53:49  *** Guest51134 is now known as pigeons
 10 2015-11-10T01:08:14  *** GAit has quit IRC
 11 2015-11-10T01:16:09  *** GAit has joined #bitcoin-core-dev
 12 2015-11-10T01:34:24  *** Ylbam has quit IRC
 13 2015-11-10T01:42:39  *** randy-waterhouse has joined #bitcoin-core-dev
 14 2015-11-10T01:56:25  <PRab> no surprise, but running 0.11.2rc1 uneventfully.
 15 2015-11-10T01:56:36  <gmaxwell> Likewise.
 16 2015-11-10T01:58:10  <PRab> If you guys want, I can crash my computer. I have been hit by the DB corruption bug several times and it looks like this release might fix that.
 17 2015-11-10T02:00:37  <btcdrak> PRab, great.
 18 2015-11-10T02:00:57  *** CodeShark_ has joined #bitcoin-core-dev
 19 2015-11-10T02:01:06  <PRab> btcdrak: Great that its running, or great that I can test crashing?
 20 2015-11-10T02:01:19  <btcdrak> that it is running great
 21 2015-11-10T02:01:27  <gmaxwell> PRab: yes, it should fix most of the corruption reports on windows. All except the anti-virus related ones, as far as we know.
 22 2015-11-10T02:01:55  *** CodeShark is now known as Guest30488
 23 2015-11-10T02:02:06  *** CodeShark_ has quit IRC
 24 2015-11-10T02:02:28  *** CodeShark_ has joined #bitcoin-core-dev
 25 2015-11-10T02:02:53  *** CodeShark_ has quit IRC
 26 2015-11-10T02:08:49  *** Guest30488 has quit IRC
 27 2015-11-10T02:09:10  *** CodeShark_ has joined #bitcoin-core-dev
 28 2015-11-10T02:09:46  *** CodeShark_ has quit IRC
 29 2015-11-10T02:09:56  *** CodeShark has joined #bitcoin-core-dev
 30 2015-11-10T02:15:53  *** zooko has quit IRC
 31 2015-11-10T02:21:08  *** zooko has joined #bitcoin-core-dev
 32 2015-11-10T02:25:59  <GitHub106> [bitcoin] theuni opened pull request #6978: Alternative fix for #6248 (qt+fPIE) (master...qt-pie) https://github.com/bitcoin/bitcoin/pull/6978
 33 2015-11-10T02:56:24  *** CodeShark has quit IRC
 34 2015-11-10T03:01:31  *** CodeShark has joined #bitcoin-core-dev
 35 2015-11-10T03:07:12  *** guest234234 has joined #bitcoin-core-dev
 36 2015-11-10T03:09:25  *** zooko` has joined #bitcoin-core-dev
 37 2015-11-10T03:12:45  *** zooko`` has joined #bitcoin-core-dev
 38 2015-11-10T03:13:17  *** zooko has quit IRC
 39 2015-11-10T03:16:18  *** zooko` has quit IRC
 40 2015-11-10T03:21:42  *** zooko`` has quit IRC
 41 2015-11-10T03:50:12  *** zooko has joined #bitcoin-core-dev
 42 2015-11-10T03:56:35  *** zooko has quit IRC
 43 2015-11-10T04:06:27  *** zooko` has joined #bitcoin-core-dev
 44 2015-11-10T04:15:18  *** CodeShark has quit IRC
 45 2015-11-10T04:26:39  *** tulip has quit IRC
 46 2015-11-10T04:31:12  *** guest234234 has quit IRC
 47 2015-11-10T05:02:17  *** skyraider has quit IRC
 48 2015-11-10T05:08:09  *** zooko` has quit IRC
 49 2015-11-10T05:12:41  *** tulip has joined #bitcoin-core-dev
 50 2015-11-10T05:39:26  *** guest234234 has joined #bitcoin-core-dev
 51 2015-11-10T06:23:03  *** Ylbam has joined #bitcoin-core-dev
 52 2015-11-10T06:46:07  *** fkhan has quit IRC
 53 2015-11-10T06:50:42  *** Thireus has quit IRC
 54 2015-11-10T06:51:02  *** Thireus has joined #bitcoin-core-dev
 55 2015-11-10T06:53:31  *** CodeShark has joined #bitcoin-core-dev
 56 2015-11-10T07:05:35  *** guest234234 has quit IRC
 57 2015-11-10T07:09:25  *** tulip has quit IRC
 58 2015-11-10T07:13:12  *** fkhan has joined #bitcoin-core-dev
 59 2015-11-10T07:13:21  *** CodeShark_ has joined #bitcoin-core-dev
 60 2015-11-10T07:14:59  *** CodeShark has quit IRC
 61 2015-11-10T07:15:33  *** CodeShark has joined #bitcoin-core-dev
 62 2015-11-10T07:17:26  *** CodeShark_ has quit IRC
 63 2015-11-10T07:32:15  *** Thireus has quit IRC
 64 2015-11-10T07:33:20  *** CodeShark has quit IRC
 65 2015-11-10T08:11:19  <jonasschnelli> PRab: would be nice if you could test the v0.11.2rc1 on real-windows (non VM).
 66 2015-11-10T08:12:11  <jonasschnelli> I did some VMWare power-off simulations... it did mess up the db even with the fix. But this particular fix is better tested in a non-vm environment.
 67 2015-11-10T08:17:07  *** GAit has joined #bitcoin-core-dev
 68 2015-11-10T08:18:03  *** Thireus has joined #bitcoin-core-dev
 69 2015-11-10T08:28:46  *** fanquake has joined #bitcoin-core-dev
 70 2015-11-10T08:31:21  *** BashCo has quit IRC
 71 2015-11-10T08:44:32  *** fanquake has quit IRC
 72 2015-11-10T08:48:38  *** randy-waterhouse has quit IRC
 73 2015-11-10T08:53:51  *** trippysalmon has joined #bitcoin-core-dev
 74 2015-11-10T08:57:35  *** Guyver2 has joined #bitcoin-core-dev
 75 2015-11-10T09:17:16  *** CodeShark has joined #bitcoin-core-dev
 76 2015-11-10T09:22:00  <wumpus> I tested it on a real windows laptop (with NotMyFault to inject kernel faults) and wasn't able to get any corruption with the new syncing behavior, while the old behavior was to corrupt every single time
 77 2015-11-10T09:22:15  <wumpus> so even if not perfect it's a lot better
 78 2015-11-10T09:58:13  *** rubensayshi has joined #bitcoin-core-dev
 79 2015-11-10T10:03:02  <wumpus> gitian is broken for me :( "Could not download some packages, please run gbuild --upgrade" when trying to sign the mac package
 80 2015-11-10T10:03:39  <dcousens> jonasschnelli: I've had similar nodes get 'stuck' before, so unless its repeatable, I'm not sure that is related to that PR
 81 2015-11-10T10:03:58  <wumpus> nothing wrong in install.log
 82 2015-11-10T10:04:08  <jonasschnelli> dcousens: right. I wrote that in a comment. Probably unrelated to the secp256k1 switch PR
 83 2015-11-10T10:04:28  <jonasschnelli> But a serious bug,...
 84 2015-11-10T10:04:30  <wumpus> faketime is already the newest version. libc6:i386 is already the newest version. 0 upgraded, 0 newly installed, 0 to remove and 3 not upgraded.
 85 2015-11-10T10:04:35  <dcousens> jonasschnelli: I know, just providing an anecdote that might further that
 86 2015-11-10T10:04:45  <dcousens> further support*
 87 2015-11-10T10:05:04  <wumpus> jonasschnelli: I had a similar issue a while back of a node that just stopped catching up with the chain - never been able to repeat it
 88 2015-11-10T10:05:11  <dcousens> also, you wrote F.I.Y, for interested yours? :P
 89 2015-11-10T10:05:34  <jonasschnelli> yeah... it seams to be hard to create clear steps to reproduce...
 90 2015-11-10T10:05:40  <wumpus> (and I didn't build with debug symbols at the time, so was unable to do anything useful to troubleshoot internal state)
 91 2015-11-10T10:05:54  <jonasschnelli> s/F.I.Y/F.Y.I... :)
 92 2015-11-10T10:06:06  <wumpus> if it happens again I'm prepared. But it never happened again
 93 2015-11-10T10:06:16  <dcousens> wumpus: I had it a week ago
 94 2015-11-10T10:06:47  <jonasschnelli> wumpus: while(1) { IBD() } ...
 95 2015-11-10T10:06:56  <wumpus> here: https://github.com/bitcoin/bitcoin/issues/6188
 96 2015-11-10T10:07:28  <wumpus> well it didn't happen during IBD in my case, just with a node that was synced but suddenly stopped without rejecting the chain or anything like that
 97 2015-11-10T10:07:43  <wumpus> (and after four days it magically started again)
 98 2015-11-10T10:07:49  <dcousens> That said, in restarting the node, I found the most prominent factor was the peers it was connected to
 99 2015-11-10T10:08:02  <jonasschnelli> true.
100 2015-11-10T10:08:39  *** CodeShark has quit IRC
101 2015-11-10T10:09:00  <jonasschnelli> the strange thing is, as you can see here (https://bitcoin.jonasschnelli.ch/secp/stuck_debug.log), no block was rejected.
102 2015-11-10T10:09:02  <wumpus> that wasn't the problem in my case. it was a node with many incoming connections, and I did try dropping the network to get new connections
103 2015-11-10T10:09:44  *** BashCo has joined #bitcoin-core-dev
104 2015-11-10T10:09:59  <wumpus> it is likely some race condition, where it forgets about being insistent about requesting the next block it needs
105 2015-11-10T10:10:24  <wumpus> likely caused by one or more uncooprorative peers
106 2015-11-10T10:15:24  *** pmienk has joined #bitcoin-core-dev
107 2015-11-10T10:20:30  *** GAit has quit IRC
108 2015-11-10T10:21:00  *** GAit has joined #bitcoin-core-dev
109 2015-11-10T10:23:55  <wumpus> re: the gitian issue, adding --upgrade to my gbuild line seems to have solved it. I don't understand why this is suddenly needed, but ok, great :)
110 2015-11-10T11:07:45  *** ParadoxSpiral has joined #bitcoin-core-dev
111 2015-11-10T11:48:15  *** MarcoFalke has joined #bitcoin-core-dev
112 2015-11-10T12:43:30  *** jonasschnelli has quit IRC
113 2015-11-10T12:44:19  *** jonasschnelli has joined #bitcoin-core-dev
114 2015-11-10T12:50:54  *** Naphex has joined #bitcoin-core-dev
115 2015-11-10T12:50:57  <GitHub175> [bitcoin] jonasschnelli opened pull request #6979: [Qt] simple mempool info in debug window (master...2015/11/qt_mempool_easyinfo) https://github.com/bitcoin/bitcoin/pull/6979
116 2015-11-10T12:58:42  *** jonasschnelli has quit IRC
117 2015-11-10T13:00:20  *** jonasschnelli has joined #bitcoin-core-dev
118 2015-11-10T13:04:09  *** MarcoFalke has quit IRC
119 2015-11-10T13:14:02  *** GAit has quit IRC
120 2015-11-10T13:14:37  *** GAit has joined #bitcoin-core-dev
121 2015-11-10T13:26:06  *** GAit has quit IRC
122 2015-11-10T13:26:55  *** GAit has joined #bitcoin-core-dev
123 2015-11-10T13:44:41  <dcousens> jonasschnelli: what block did you pause on OOI?
124 2015-11-10T13:45:07  <jonasschnelli> dcousens: OOI?
125 2015-11-10T13:45:14  <dcousens> out of interest
126 2015-11-10T13:45:41  <jonasschnelli> dcousens: 00000000000000001043ada919c3851e17876deca67acc19f365fab4a79bd59d, 382748
127 2015-11-10T13:45:55  <jonasschnelli> dcousens: check: https://bitcoin.jonasschnelli.ch/secp/stuck_debug.log, search after last UpdateTip
128 2015-11-10T13:46:44  <GitHub50> [bitcoin] laanwj pushed 2 new commits to 0.11: https://github.com/bitcoin/bitcoin/compare/3dcb390fe9e2...7e278929df53
129 2015-11-10T13:46:44  <GitHub50> bitcoin/0.11 ab6ff12 David A. Harding: [doc] 0.11.2 release notes: use original pull numbers...
130 2015-11-10T13:46:44  <GitHub175> [bitcoin] laanwj closed pull request #6975: [doc] 0.11.2 release notes: use original pull numbers (0.11...note-0.11.2-orig-prs) https://github.com/bitcoin/bitcoin/pull/6975
131 2015-11-10T13:46:45  <GitHub50> bitcoin/0.11 7e27892 Wladimir J. van der Laan: Merge pull request #6975...
132 2015-11-10T13:47:36  <dcousens> jonasschnelli: interesting
133 2015-11-10T13:47:52  <dcousens> I stopped around 362250
134 2015-11-10T13:48:08  <dcousens> But wumpus' post was around 370k
135 2015-11-10T13:48:45  <jonasschnelli> dcousens: do you see a relation between 362250 and 382748?
136 2015-11-10T13:49:21  <dcousens> jonasschnelli: i haven't checked, the only relation I have so far is that my block parser has exploded around those blocks recently
137 2015-11-10T13:49:31  <dcousens> and I haven't yet debugged it, could be completely unrelated
138 2015-11-10T13:49:57  <jonasschnelli> hmm... yes. Would be nice if we could track down that problem.
139 2015-11-10T13:50:27  <dcousens> anyway, late here, will have a closer look in the morrow :)
140 2015-11-10T13:50:29  <dcousens> night
141 2015-11-10T13:52:52  <wumpus> 0.10.4rc1 executables live https://bitcoin.org/bin/bitcoin-core-0.10.4/test/
142 2015-11-10T13:55:11  *** dcousens has quit IRC
143 2015-11-10T14:05:14  *** zooko has joined #bitcoin-core-dev
144 2015-11-10T14:12:41  <jonasschnelli> wumpus : thanks for the reminder on #6900, just started a gitian build with the new descriptor...
145 2015-11-10T14:12:55  <jonasschnelli> hmm... i need the base images first i assume
146 2015-11-10T14:25:30  *** fanquake has joined #bitcoin-core-dev
147 2015-11-10T14:25:51  <fanquake> wumpus building the gitian pull now
148 2015-11-10T14:32:15  <GitHub179> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/503ff6e1ae69...77beab70deae
149 2015-11-10T14:32:15  <GitHub179> bitcoin/master 87cbdb8 Jorge Timón: Globals: Explicit Consensus::Params arg for main:...
150 2015-11-10T14:32:16  <GitHub179> bitcoin/master 77beab7 Wladimir J. van der Laan: Merge pull request #6163...
151 2015-11-10T14:32:20  <GitHub104> [bitcoin] laanwj closed pull request #6163: Chainparams: Explicit Consensus::Params arg for almost all remaining functions (master...chainparams-remaining-consensus) https://github.com/bitcoin/bitcoin/pull/6163
152 2015-11-10T14:32:30  <GitHub170> [bitcoin] laanwj closed pull request #6248: Fix Qt build on arch by setting -fPIC (master...archbuild) https://github.com/bitcoin/bitcoin/pull/6248
153 2015-11-10T14:35:41  *** ParadoxSpiral_ has joined #bitcoin-core-dev
154 2015-11-10T14:38:19  *** ParadoxSpiral has quit IRC
155 2015-11-10T14:38:35  *** zooko has quit IRC
156 2015-11-10T14:48:45  <GitHub35> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/77beab70deae...755b4ba848bc
157 2015-11-10T14:48:45  <GitHub35> bitcoin/master fd55571 Luke Dashjr: wallet: Expose GUI labels in RPC
158 2015-11-10T14:48:46  <GitHub35> bitcoin/master 755b4ba Wladimir J. van der Laan: Merge pull request #5574...
159 2015-11-10T14:48:46  <GitHub43> [bitcoin] laanwj closed pull request #5574: Expose GUI labels in RPC as comments (master...rpc_labels) https://github.com/bitcoin/bitcoin/pull/5574
160 2015-11-10T14:51:48  <GitHub120> [bitcoin] laanwj closed pull request #6693: Set Windows TCP buffers to 64KB to match OSX and Unix (master...issue_6554) https://github.com/bitcoin/bitcoin/pull/6693
161 2015-11-10T14:58:09  <GitHub78> [bitcoin] laanwj pushed 4 new commits to master: https://github.com/bitcoin/bitcoin/compare/755b4ba848bc...9fa54a1b0c1a
162 2015-11-10T14:58:10  <GitHub78> bitcoin/master 5f46a7d MarcoFalke: transaction_tests: Be more strict checking dust...
163 2015-11-10T14:58:10  <GitHub78> bitcoin/master 536766c MarcoFalke: [trivial] New DEFAULT_MIN_RELAY_TX_FEE = 1000
164 2015-11-10T14:58:11  <GitHub78> bitcoin/master e20d924 MarcoFalke: [trivial] init: Use defaults MIN_RELAY_TX_FEE & TRANSACTION_MAXFEE
165 2015-11-10T14:58:18  <GitHub65> [bitcoin] laanwj closed pull request #6822: [tests] Be more strict checking dust (master...MarcoFalke-2015-minRelayTxFeeCleanup) https://github.com/bitcoin/bitcoin/pull/6822
166 2015-11-10T14:59:01  <wumpus> jonasschnelli: you need one new base image, trusty amd64
167 2015-11-10T15:01:38  *** zooko has joined #bitcoin-core-dev
168 2015-11-10T15:08:53  <fanquake> I'll put up a pull to bump boost, as well as ccache & miniupnpc
169 2015-11-10T15:09:27  <fanquake> ccache because of an osx compile regression & miniupnpc so we get wumpus's string handling/overflow fixes.
170 2015-11-10T15:10:08  *** jgarzik has joined #bitcoin-core-dev
171 2015-11-10T15:11:07  <wumpus> awesome
172 2015-11-10T15:13:05  <jonasschnelli> wumpus: did you face similar issues when building the trusy base image:...
173 2015-11-10T15:13:05  <jonasschnelli> Process (['mkfs.ext4', '-F', '/dev/mapper/loop0p1']) returned 1. stdout: , stderr: mke2fs
174 2015-11-10T15:13:12  <jonasschnelli> The file /dev/mapper/loop0p1 does not exist and no size was specified.
175 2015-11-10T15:13:22  <fanquake> jonasschnelli yes
176 2015-11-10T15:13:34  <jonasschnelli> i have updates gitian to the master git tip
177 2015-11-10T15:13:42  <jonasschnelli> fanquake: ah. How did you solve that?
178 2015-11-10T15:13:51  <wumpus> don't remember that, I did succeed in making the image
179 2015-11-10T15:14:27  <fanquake> jonasschnelli I haven't yet. Got sidetracked looking at depends.
180 2015-11-10T15:15:03  <jonasschnelli> fanquake: okay. Thanks.. then i go down that rabbit hole...
181 2015-11-10T15:16:09  <wumpus> just did `bin/make-base-vm --arch amd64 --suite trusty`
182 2015-11-10T15:18:50  *** gribble has quit IRC
183 2015-11-10T15:26:04  <jonasschnelli> wumpus: i guess your on ubuntu? On debian i always had issues with gitian...
184 2015-11-10T15:26:43  <wumpus> right,ubuntu
185 2015-11-10T15:31:46  *** gribble has joined #bitcoin-core-dev
186 2015-11-10T15:32:18  <GitHub111> [bitcoin] fanquake opened pull request #6980: [Depends] Bump Boost, miniupnpc & ccache (master...depends-bump-boost) https://github.com/bitcoin/bitcoin/pull/6980
187 2015-11-10T15:35:21  <GitHub99> [bitcoin] laanwj closed pull request #6937: Fix Boost 1.58.0 build for mips arch (master...mips-options-fix) https://github.com/bitcoin/bitcoin/pull/6937
188 2015-11-10T15:37:42  *** jtimon has joined #bitcoin-core-dev
189 2015-11-10T15:39:25  <GitHub85> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/9fa54a1b0c1a...32d8b1570cb0
190 2015-11-10T15:39:25  <GitHub85> bitcoin/master 7ca73dc Jonathan Cross: Improving labels for Sent / Received "Bytes"...
191 2015-11-10T15:39:26  <GitHub85> bitcoin/master 32d8b15 Wladimir J. van der Laan: Merge pull request #6940...
192 2015-11-10T15:39:35  <GitHub102> [bitcoin] laanwj closed pull request #6940: Improving labels for Sent / Received "Bytes" (master...patch-1) https://github.com/bitcoin/bitcoin/pull/6940
193 2015-11-10T15:45:13  <GitHub4> [bitcoin] laanwj pushed 1 new commit to master: https://github.com/bitcoin/bitcoin/commit/b56953e9bb5a32bc35365d1f0c5de5528c0650dd
194 2015-11-10T15:45:14  <GitHub4> bitcoin/master b56953e Wladimir J. van der Laan: qt: Periodic translations update
195 2015-11-10T15:53:10  *** bsm1175321 has quit IRC
196 2015-11-10T16:00:24  *** zooko has quit IRC
197 2015-11-10T16:03:48  *** fanquake has left #bitcoin-core-dev
198 2015-11-10T16:14:03  *** Thireus has quit IRC
199 2015-11-10T16:26:22  <wumpus> jonasschnelli: any luck building the image? maybe try updating vmbuilder, if you're building it from source
200 2015-11-10T16:30:27  *** GAit has quit IRC
201 2015-11-10T16:34:07  *** PaulCapestany has quit IRC
202 2015-11-10T16:34:17  *** PaulCape_ has joined #bitcoin-core-dev
203 2015-11-10T16:35:10  *** trippysalmon has quit IRC
204 2015-11-10T16:43:04  *** randy-waterhouse has joined #bitcoin-core-dev
205 2015-11-10T16:43:26  *** Thireus has joined #bitcoin-core-dev
206 2015-11-10T16:45:56  <sipa> wumpus, jonasschnelli: where in the codebase are we using powers-of-1024 based units?
207 2015-11-10T16:46:52  <sipa> bitcoin traditionally uses 1000-based units (feerate is per 1000 bytes, block limit is 1000000 bytes), but I can't say I've payed much attention to it myself
208 2015-11-10T16:46:57  <jgarzik> sipa, it's all over Qt
209 2015-11-10T16:47:42  <sipa> if anything, be consistent (don't use MB for 1048576 bytes, and certainly don't use KB - that would mean kelvin bytes)
210 2015-11-10T16:47:47  <jgarzik> several core uses in default values, usually related to file size
211 2015-11-10T16:52:55  <wumpus> sipa: I'm not sure. But in this case jonasschnelli tried to introduce a use of 1024*1024
212 2015-11-10T16:53:12  <sipa> I know
213 2015-11-10T16:53:13  <wumpus> which was sensibly commented on
214 2015-11-10T16:54:08  <sipa> I think using MB = 1024^2 is a no-go. My question is whether or not we should aim for only MB = 1000000 or only MiB = 1024^2
215 2015-11-10T16:54:10  <wumpus> there's probably a few other cases but in general the idea is to use 1000000/MB
216 2015-11-10T16:55:00  <wumpus> only 1000 *unless* there is a convincing reason to use 1048576/MiB, which I'm not sure exists
217 2015-11-10T16:55:30  *** rubensayshi has quit IRC
218 2015-11-10T16:55:35  <sipa> We clearly need a --si commandline flag
219 2015-11-10T16:55:37  <sipa> *ducks*
220 2015-11-10T16:55:52  <wumpus> e.g. I can understand using MiB when you're selling memory chips which for hardware reasons, only in powers of two
221 2015-11-10T16:56:10  <wumpus> but for bitcoin core which is a high-level application there shoulod be no reason to not just use SI units
222 2015-11-10T16:56:17  <sipa> agree
223 2015-11-10T16:56:50  <wumpus> and it's always been that way, there's not really a reason to discuss or revise this :)
224 2015-11-10T16:57:45  * wumpus likes Kelvin*Bytes though
225 2015-11-10T16:57:45  <sipa> I was just asking whether or not we currently have cases where 1024-based units are exposed to users. Consistency with already existing uses would be a reason.
226 2015-11-10T16:58:13  <sipa> "Our cache is very hot. Many KB."
227 2015-11-10T16:58:20  <wumpus> hehe
228 2015-11-10T16:59:15  * wumpus going to add a note about using SI units to the developer notes
229 2015-11-10T16:59:35  <jgarzik> as noted, we have many cases exposed to users
230 2015-11-10T17:00:53  <sipa> i think -dbcache may be 1024-based
231 2015-11-10T17:01:17  <wumpus> should be changed
232 2015-11-10T17:01:19  <jgarzik> -dblogsize, several Qt UI attributes
233 2015-11-10T17:01:37  <jgarzik> several defaults are pow2
234 2015-11-10T17:02:00  *** GAit has joined #bitcoin-core-dev
235 2015-11-10T17:02:39  <sipa> i'm in favor of making user-exposed amounts consistently 1000-based, if there aren't too many
236 2015-11-10T17:03:25  <wumpus> yes
237 2015-11-10T17:07:42  <jgarzik> +1
238 2015-11-10T17:08:06  <wangchun> Dear core devs, how do you plan to response to the latest actions regarding BIP101?
239 2015-11-10T17:09:40  <wumpus> not at all
240 2015-11-10T17:10:41  <sipa> Bitcoin Core will implement any uncontroversial hard fork.
241 2015-11-10T17:11:16  <jgarzik> wangchun, Scaling Bitcoin Pt 2 is the next step in figuring out uncontroversial hard fork
242 2015-11-10T17:11:49  *** skyraider has joined #bitcoin-core-dev
243 2015-11-10T17:14:30  *** bsm1175321 has joined #bitcoin-core-dev
244 2015-11-10T17:14:47  *** paveljanik has joined #bitcoin-core-dev
245 2015-11-10T17:15:56  *** PaulCape_ has quit IRC
246 2015-11-10T17:16:21  *** PaulCapestany has joined #bitcoin-core-dev
247 2015-11-10T17:21:26  <btcdrak> sipa: I just added some commits which I think address your concerns https://github.com/bitcoin/bitcoin/pull/6312#issuecomment-154652895
248 2015-11-10T17:26:15  *** GAit has quit IRC
249 2015-11-10T17:26:51  *** GAit has joined #bitcoin-core-dev
250 2015-11-10T17:34:04  <GitHub159> [bitcoin] laanwj opened pull request #6981: doc: Add note about SI units to developer notes (master...2015_11_si_units) https://github.com/bitcoin/bitcoin/pull/6981
251 2015-11-10T17:41:08  *** molly has quit IRC
252 2015-11-10T17:53:49  *** BashCo has quit IRC
253 2015-11-10T17:57:17  *** molly has joined #bitcoin-core-dev
254 2015-11-10T18:19:10  <GitHub108> [bitcoin] laanwj closed pull request #6965: Benchmark sanity checks and fork checks in ConnectBlock (master...bench) https://github.com/bitcoin/bitcoin/pull/6965
255 2015-11-10T18:19:24  <GitHub111> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/b56953e9bb5a...de7d4591a7ce
256 2015-11-10T18:19:25  <GitHub111> bitcoin/master 77f1f59 Matt Corallo: Benchmark sanity checks and fork checks in ConnectBlock
257 2015-11-10T18:19:25  <GitHub111> bitcoin/master de7d459 Wladimir J. van der Laan: Merge pull request #6965...
258 2015-11-10T18:21:39  *** gribble has quit IRC
259 2015-11-10T18:31:37  *** gribble has joined #bitcoin-core-dev
260 2015-11-10T18:47:36  *** GAit has quit IRC
261 2015-11-10T19:01:36  <GitHub176> [bitcoin] jtimon opened pull request #6982: Fix #6163 (AcceptBlockHeader) (master...fix-6163) https://github.com/bitcoin/bitcoin/pull/6982
262 2015-11-10T19:06:09  <jonasschnelli> consensus on MB and /1000^2?
263 2015-11-10T19:06:32  <jgarzik> IMO the consensus was already "move to MiB" before today...
264 2015-11-10T19:06:57  * jonasschnelli is confused... :/
265 2015-11-10T19:09:13  * Luke-Jr dislikes "_iB" notation and would avoid other software enforcing it.
266 2015-11-10T19:09:27  <Luke-Jr> I don't suppose Qt has a nice localisation method for these?
267 2015-11-10T19:09:27  <sipa> jgarzik: now i am confused too
268 2015-11-10T19:09:49  <jgarzik> ok, I meant SI units
269 2015-11-10T19:10:05  <jgarzik> whatever the abbrevation is
270 2015-11-10T19:10:13  <Luke-Jr> SI units are 1000-based kB/MB/GB
271 2015-11-10T19:10:15  <jgarzik> Luke-Jr, agree!
272 2015-11-10T19:10:29  <jgarzik> AFAIK the consensus is SI units </correction>
273 2015-11-10T19:10:59  <Luke-Jr> I'd prefer configurable at some level (DE level would be nice), but not worth the time if Qt doesn't make it easy.
274 2015-11-10T19:11:12  <sipa> jgarzik: there are 2 questions? "should we use 1000 or 1024-based units" and "should we use Mi-style prefixes for 1024-based units?"
275 2015-11-10T19:11:42  <Luke-Jr> Right now, the traffic thing uses 1024-based KB/MB/GB
276 2015-11-10T19:11:51  <Luke-Jr> (which is what I prefer)
277 2015-11-10T19:12:03  <sipa> that's the only unacceptable thing for me :)
278 2015-11-10T19:12:21  <sipa> either 1000000/MB, or 1049576/MiB
279 2015-11-10T19:12:34  <sipa> with a preference for the first
280 2015-11-10T19:12:44  <sipa> 1048576 i mean
281 2015-11-10T19:12:51  <gmaxwell> sipa: really you have a prefered for the first?
282 2015-11-10T19:13:18  <sipa> yes, though i have a tendency for the aecond
283 2015-11-10T19:13:32  <jgarzik> 1) 1000-based units,    2) I hate Mi style prefixes -- grew up thinking 1024==KB, 1024*1024==MB, etc.  However I think that's the direction of the world...
284 2015-11-10T19:14:06  <jgarzik> easy enough to ensure we eliminate as many 1024-based units as possible, to minimize Mi prefixes
285 2015-11-10T19:14:07  <Luke-Jr> jgarzik: the direction of the world is defined by people in the world, such as us to a small degree ;)
286 2015-11-10T19:14:41  <gmaxwell> Normally _bandwidth_ is measured in megabits which is a 1000 based unit. Transfer regretably is often in 1024 based units. "MB" should never be used for 1024 based because it's unnecessarily confusing; MiB is unambigious. The purpose of the messages is to communicate, not to be pretty.
287 2015-11-10T19:15:12  <sipa> gmaxwell: i think the world would be a (marginally) better place if everything used 1000-based prefixes, it would make reasoning so much simpler
288 2015-11-10T19:15:17  <jgarzik> problem - so few know WTF MiB is, out in the world
289 2015-11-10T19:15:55  * Luke-Jr did not realise megabits were typically 1000-based
290 2015-11-10T19:16:30  <gmaxwell> Luke-Jr: not just typically but always except when handled by drooling idiots. :)
291 2015-11-10T19:17:12  <gmaxwell> jgarzik: far more common than you think, I think. But even still, if so then their ignorance is apparent to them and a 10 second search will resolve it.
292 2015-11-10T19:17:22  <sipa> Luke-Jr: a 100 Mb/s link can do 11.9 MiB/s :)
293 2015-11-10T19:17:41  *** zooko has joined #bitcoin-core-dev
294 2015-11-10T19:18:44  <Luke-Jr> Well, in that case, /me suggests 1000-based with an unchecked-by-default checkbox that uses 1024 in GUI without the "i" silliness. <.<
295 2015-11-10T19:19:09  <gmaxwell> Luke-Jr: please, no freeing options.
296 2015-11-10T19:19:14  <gmaxwell> er freeking.
297 2015-11-10T19:19:26  <gmaxwell> It'll pepper the code with conditional logic that will never get tested.
298 2015-11-10T19:19:37  <jgarzik> nod
299 2015-11-10T19:19:41  <sipa> agree
300 2015-11-10T19:19:56  <Luke-Jr> gmaxwell: this is an option that different people want to use. so obviously it would get tested.
301 2015-11-10T19:20:06  <sipa> nobody will bother
302 2015-11-10T19:20:20  <sipa> no actual user will bother, rather
303 2015-11-10T19:20:28  <Luke-Jr> at least I will
304 2015-11-10T19:20:45  <gmaxwell> Do you even actually use the GUI except for testing? :)
305 2015-11-10T19:20:48  <Luke-Jr> yes
306 2015-11-10T19:21:05  <Luke-Jr> the only time I use the RPC is for testing.
307 2015-11-10T19:21:37  <jgarzik> Similar to -logtimemicros option -- it's basically an option 1-2 people will use, and the rest of the world will be unaware.
308 2015-11-10T19:21:48  <jgarzik> Should just pick a useful behavior and not make it an option.
309 2015-11-10T19:22:03  <Luke-Jr> let's just drop all non-English languages too :P
310 2015-11-10T19:22:11  <sipa> it doesn't hurt to add more digits
311 2015-11-10T19:22:27  <sipa> and the cases when they're useful is not when you know it in advance
312 2015-11-10T19:22:55  <jgarzik> I would rather remove -logtimemicros and make it unconditionally default-on.
313 2015-11-10T19:22:59  <sipa> agree
314 2015-11-10T19:23:01  <sipa> me too
315 2015-11-10T19:23:01  <wumpus> I disagree
316 2015-11-10T19:23:10  <jgarzik> From OS perspective there is no added cost
317 2015-11-10T19:23:12  <wumpus> seconds are precise enough for logging
318 2015-11-10T19:23:39  <jgarzik> 1) other userland servers log microseconds,  2) it's a pointless option that will be used by no one - yet we must maintain
319 2015-11-10T19:23:43  <sipa> i would love to be able to get bug reports "when did X take 10ms?! it should just be changing a variable!"
320 2015-11-10T19:23:49  <Luke-Jr> let's go with systemd binary logs! /s
321 2015-11-10T19:23:51  <sipa> nobody will even notice now
322 2015-11-10T19:24:08  <Luke-Jr> sipa: lol
323 2015-11-10T19:24:17  <jgarzik> default-off options should be terminated with extreme prejudice.
324 2015-11-10T19:24:23  <zooko> jgarzik: +1
325 2015-11-10T19:24:34  <kanzure> replaced with what?
326 2015-11-10T19:24:38  <jgarzik> (translation: there should be a good argument for keeping and maintaining them, above "it's nice")
327 2015-11-10T19:24:45  <wumpus> bla, this is useless
328 2015-11-10T19:25:03  <sipa> wumpus: what is your reason against microsecond logs?
329 2015-11-10T19:25:04  <kanzure> ah okay. "it's nice" as insufficient. ok.
330 2015-11-10T19:25:30  <kanzure> you could send all logs through zeromq inside the process, then people can subscribe to logs in different ways using zeromq.
331 2015-11-10T19:25:38  <wumpus> sipa: too precise timestamps are a possible security issue
332 2015-11-10T19:25:43  <wumpus> e.g. timing attacks
333 2015-11-10T19:25:48  <sipa> hmm
334 2015-11-10T19:26:01  <wumpus> also it makes correlation easier, to breach privacy
335 2015-11-10T19:26:03  <sipa> i thought we dropped that concern when switching to -logtimestamps default on
336 2015-11-10T19:26:08  <jgarzik> sipa, yep
337 2015-11-10T19:26:09  <wumpus> yes, for seconds
338 2015-11-10T19:26:13  <gmaxwell> I would rather stop logging things that breach privacy.
339 2015-11-10T19:26:33  <Luke-Jr> as long as sipa hates 1024 KB, and I prefer 1024 KB, there's no way to satisfy both without an option. If you all decide to make it 1000-based only, I can just add an option to Bitcoin LJR if I ever care enough (unlikely)
340 2015-11-10T19:26:35  <wumpus> microsconds just goes too far, I see no point in that kind of precision
341 2015-11-10T19:26:44  <sipa> Luke-Jr: kB
342 2015-11-10T19:26:46  <sipa> :p
343 2015-11-10T19:26:50  <Luke-Jr> sipa: kB is 1000-based always
344 2015-11-10T19:26:50  <gmaxwell> We generate a LOT of excess IO via logging. Lets reduce the chattyness, that is a much more robust privacy protection than decreasing timestamp precision.
345 2015-11-10T19:27:02  <jgarzik> There is no useful privacy argument for seconds, which does not also apply to microseconds.  The matter of degrees is tiny.
346 2015-11-10T19:27:04  <jgarzik> +1 gmaxwell
347 2015-11-10T19:27:09  <wumpus> god damnit
348 2015-11-10T19:27:18  <kanzure> timing attacks tho
349 2015-11-10T19:27:40  <kanzure> oh
350 2015-11-10T19:27:47  <wumpus> I agree with reducing chattyness as well, but that's a different concern
351 2015-11-10T19:28:00  <kanzure> right, i think you mean to say "microseconds does not have any useful privacy advantage over seconds".
352 2015-11-10T19:28:54  <Luke-Jr> it does have a screen-space advantage, but I can't believe we're spending time discussing this.
353 2015-11-10T19:29:15  <jgarzik> kanzure, no.  the argument being made is "microseconds goes too far" and "it makes correlation easier, to breach privacy"
354 2015-11-10T19:29:41  <wumpus> Luke-Jr: and it's already possible to enable microseconds if you want to use them
355 2015-11-10T19:29:43  <sipa> Luke-Jr: KB is kelvin byte. I don't understand where your capital K even comes from; IEC uses Ki for 1024 because ki would make it look like a unit rather than a prefix
356 2015-11-10T19:30:03  <jgarzik> kanzure, no privacy advantage in microseconds is being claimed - just the opposite - though the matter of degree is tiny
357 2015-11-10T19:30:10  <wumpus> I don't understand this insistence on enabling it by default
358 2015-11-10T19:30:11  <jgarzik> versus the diagnostic utility and common practice elsewhere
359 2015-11-10T19:30:14  <kanzure> jgarzik: understood
360 2015-11-10T19:30:15  <Luke-Jr> KB is the traditional standard notation for 1024 bytes, defined in at least JEDEC standards
361 2015-11-10T19:31:15  <Luke-Jr> sipa: ^
362 2015-11-10T19:31:17  <kanzure> i believe the BFOH approach was "KiBi" instead of KB or kB or KiB. (not serious here) (but i did see this proposed once by a BOFH).
363 2015-11-10T19:31:53  <sipa> Luke-Jr: interesting :)
364 2015-11-10T19:32:04  <jgarzik> wumpus, Because merging options that nearly-nobody will use is dumb
365 2015-11-10T19:32:14  <sipa> wumpus: being able to benchmark things after the fact
366 2015-11-10T19:32:23  <wumpus> jgarzik: people that want to benchmark things with precision will enable it
367 2015-11-10T19:32:36  <wumpus> if no one wanted it it wouldn't have been merged at all
368 2015-11-10T19:32:49  <kanzure> was this cli option or was this build option?
369 2015-11-10T19:32:55  <sipa> cli
370 2015-11-10T19:32:56  <jgarzik> c.f. "1-2 users"    Most people will not even know about it.
371 2015-11-10T19:32:57  <jgarzik> cli
372 2015-11-10T19:33:04  <kanzure> was build option considered?
373 2015-11-10T19:33:09  <wumpus> why?
374 2015-11-10T19:33:28  <kanzure> because cli options are disagreeable for good reasons
375 2015-11-10T19:33:32  <wumpus> what?
376 2015-11-10T19:33:53  <wumpus> disagreeable for what reason?
377 2015-11-10T19:34:18  <kanzure> 11:24 < jgarzik> default-off options should be terminated with extreme prejudice.
378 2015-11-10T19:34:29  <wumpus> and that doesn't apply to build options?
379 2015-11-10T19:34:32  <jgarzik> build option is even worse - conditional compilation creates code not even built always, but still must be maintained.
380 2015-11-10T19:34:33  *** BashCo has joined #bitcoin-core-dev
381 2015-11-10T19:34:42  <gmaxwell> wumpus: it's more conditional code which will be inadequately tested in one form or another, especially all the combinations will not be tested.
382 2015-11-10T19:34:44  <kanzure> hmm okay.
383 2015-11-10T19:34:45  <wumpus> yeah this is absolutely going the wrong way
384 2015-11-10T19:35:03  <wumpus> gmaxwell: I agree if it's a complex conditional, but come on, a logging option
385 2015-11-10T19:35:24  <gmaxwell> E.g. all of us will turn microseconds on;  and then no-microseconds + disable wallet will end up crashing; and we won't notice until after a release.  ... but yes; in this particular case this argument is not strong. I agree that it's mostly safe.
386 2015-11-10T19:35:37  <wumpus> but anyhow feel free to remove the option, I'm done discussing this
387 2015-11-10T19:36:04  <kanzure> what about zeromq as logging transport, then just use whatever temporal resolution at receive time of each message?
388 2015-11-10T19:36:07  <kanzure> ok nevermind then
389 2015-11-10T19:36:18  <Luke-Jr> kanzure: ZeroMQ is stupidly unreliable.
390 2015-11-10T19:36:27  <wumpus> debug logging is *not* meant as application interface
391 2015-11-10T19:36:36  <kanzure> Luke-Jr: fedpeg.py is just misconfigured :P
392 2015-11-10T19:36:37  <wumpus> it's just for debugging and troubleshooting
393 2015-11-10T19:36:48  <wumpus> not for processing by other applications
394 2015-11-10T19:36:49  <Luke-Jr> kanzure: fedpeg.py is not my only experience with ZeroMQ>
395 2015-11-10T19:37:21  <wumpus> if you insist you can already use -logtoconsole and pipe it to something
396 2015-11-10T19:38:20  <gmaxwell> kanzure: I have other expirence with ZeroMQ too. It does not correctly handle non-lossless networks.
397 2015-11-10T19:40:18  <jgarzik> Makes sense.  zmq was built for LANs, with similar use cases to RDMA.
398 2015-11-10T19:40:41  <Luke-Jr> LANs aren't always lossless (hi wifi)
399 2015-11-10T19:40:42  *** berndj has quit IRC
400 2015-11-10T19:40:43  <morcos> just to be clear, i love and will always use logtimemicros, but i have another annoying question
401 2015-11-10T19:40:46  <kanzure> so complaint is lack of application-level retries? or tcp delivery guarantees too strongly stated in zeromq docs?
402 2015-11-10T19:40:48  <sipa> zmq is explicitly lossy
403 2015-11-10T19:40:51  <sipa> like udp
404 2015-11-10T19:40:54  <Luke-Jr> also, ZeroMQ 4.0 is incompatible with 4.1
405 2015-11-10T19:41:01  <sipa> i thought?
406 2015-11-10T19:41:07  <morcos> wumpus: i have created two new functions EstimateApproxFee and EstimateApproxPriority in 6134
407 2015-11-10T19:41:17  <morcos> my plan was not to expose them via RPC
408 2015-11-10T19:41:29  <morcos> because i think over time the fee estimator still has some more significant evolution
409 2015-11-10T19:41:49  <morcos> and then in the end it might be nice to expose what we think the final interface should be
410 2015-11-10T19:42:12  <morcos> however sdaftuar suggested i might be able to expose them now and comment they might change or keep them hidden or something
411 2015-11-10T19:42:25  <morcos> they would primarily be useful in developing tests for the new functionality i think?
412 2015-11-10T19:43:28  <morcos> in any case, would you like me to keep them unexposed for now?
413 2015-11-10T19:43:49  *** Guest8443 has joined #bitcoin-core-dev
414 2015-11-10T19:44:42  <jgarzik> morcos, Respectfully submitted, wumpus is the release manager not The DecisionMaker - ask the crowd not the one
415 2015-11-10T19:45:12  <jgarzik> There's a reason why I pushed Gavin hard for multiple committers when Satoshi disappeared - Gavin was release manager, not Bitcoin Leader
416 2015-11-10T19:45:36  <morcos> jgarzik: sure, s/wumpus/everyone/ .  although i thought i recalled him expressing an opinion about rpc api churn before
417 2015-11-10T19:50:37  <jgarzik> "release early, release often"  :)     IMO Expose them.   Maybe name the methods "beta.XXXX" to emphasize they are not final.   The goal is the communicate to users that churn will be forthcoming, but also expose them for testing and further field development.
418 2015-11-10T19:52:10  <wumpus> morcos: I don't mind, if it's useful to have them then add them
419 2015-11-10T19:52:30  <wumpus> morcos: if it is an unstable interface mention it in the help
420 2015-11-10T19:53:38  <morcos> ok thanks
421 2015-11-10T19:53:40  <wumpus> morcos: there's less an issue with adding new commands than changing old ones, especially adding arguments to existing calls is messy
422 2015-11-10T19:53:54  <jgarzik> +1
423 2015-11-10T19:53:55  *** zooko has quit IRC
424 2015-11-10T19:54:16  <morcos> wumpus: yes that was my concern with the fact that this one might change or disappear later, but i'll clearly mention it
425 2015-11-10T19:58:20  *** d4de has joined #bitcoin-core-dev
426 2015-11-10T19:59:33  <wumpus> morcos: you could decide to not list them in the overview, like invalidateblock/reconsiderblock
427 2015-11-10T20:00:51  <wumpus> (the "hidden" category)
428 2015-11-10T20:01:27  *** skyraider has quit IRC
429 2015-11-10T20:01:38  *** skyraider has joined #bitcoin-core-dev
430 2015-11-10T20:08:28  <GitHub60> [bitcoin] laanwj closed pull request #6981: doc: Add note about SI units to developer notes (master...2015_11_si_units) https://github.com/bitcoin/bitcoin/pull/6981
431 2015-11-10T20:11:03  <morcos> sipa: question re: in memory sizes..   i was thinking of making a quick pull that defaults the dbcache to 2 * maxmempool and the maxsigcachesize to 1/5 * maxmempool.  or do you think there is a better way to control total mem usage?
432 2015-11-10T20:11:53  <morcos> i'd like to test out a couple of different configurations just to make sure there aren't any regressions with a particularly large or particularly small mempool, and i was hoping to have an idea of what a typical large configuration and typical small configuration might look like
433 2015-11-10T20:13:08  <morcos> if those are the default ratios, then i'd just try out 50M mempool , 300M mempool and 2G mempool  (hmm  maybe 4G is too much for dbcache in that case?)  seems like it woudl be nice to just control 1 number
434 2015-11-10T20:14:09  <jgarzik> nod - in general users should not need to fiddle N settings just to get a usable configuration
435 2015-11-10T20:14:39  *** fkhan has quit IRC
436 2015-11-10T20:14:43  <sipa> morcos: where do you 2G and 4G numbers from from?
437 2015-11-10T20:16:09  <morcos> i didn't get it from anywhere, i was just goign to try out somethign thats pretty big.  i think 2G is about as big as mempools got in the recent spam attack, maybe between 2-3...  also easily enough for a weeks worth of backlog in txs which we had previously discussed aiming for
438 2015-11-10T20:17:21  <morcos> mostly i want to see if any of the code that traverses the whole mempool gets a bit slow then or if the resorts from multindex changing get slow.
439 2015-11-10T20:18:16  <sipa> I think 2G is too much..
440 2015-11-10T20:18:49  <morcos> maybe i explained wrong
441 2015-11-10T20:18:58  <morcos> i was going to leave maxmempool default at 300
442 2015-11-10T20:19:17  <morcos> and change dbache default to 2x maxmempool and maxsigcachesize to 1/5
443 2015-11-10T20:19:47  <morcos> and then i was going to test what happens if someone runs a particularly small or big node.
444 2015-11-10T20:20:08  <morcos> but i think it makes sense as jgarzik says for their other defaults to scale with setting one value (unless they explicitly set them otherwise)
445 2015-11-10T20:20:54  <morcos> so yeah 2G is huge, but thats sure what i'd do if i was a miner
446 2015-11-10T20:22:24  <jgarzik> We're well away from miners caring about maximizing fees to an Nth degree...   Reliability, orphaning and other factors drive miner conservatism in settings like this...
447 2015-11-10T20:23:35  <gmaxwell> The benchmarks w/ the libsecp256k1 pull really highlight the impact of increasing the size of the cache on IBD time.
448 2015-11-10T20:24:09  <morcos> gmaxwell, if you're in IBD, we should steal maxmempool size for dbacache . :)
449 2015-11-10T20:24:34  <sdaftuar> +1
450 2015-11-10T20:25:29  <morcos> so the question is what should the defaults for each of these 3 things be.  and if you want to turn 1 knob to change to big mem footprint or small, how should that work?
451 2015-11-10T20:25:50  <jgarzik> Related tech note - if one option sets another option, it becomes order-dependent
452 2015-11-10T20:26:07  <jgarzik> i.e. set dbcache before mempoolsize, and dbcache gets stomped.
453 2015-11-10T20:26:25  <jgarzik> Still, I think the base argument holds - users do not want to set N settings to achieve a useful config.
454 2015-11-10T20:26:41  <morcos> jgarzik, sure if you change the order of lines of code things get messed up
455 2015-11-10T20:26:56  <jgarzik> morcos, no, order of configuration file defines
456 2015-11-10T20:27:04  <morcos> no i dont' think so
457 2015-11-10T20:27:10  <sipa> ideally there is just a single setting for "amount of cache memory" which is sum of dbcache and mempool, and the flush criteron becomes "too large percentage of the memory is dirty cache entries"
458 2015-11-10T20:27:32  <sipa> jgarzik: no, the logic for options settings other options is earlier in init.cpp
459 2015-11-10T20:27:42  <jgarzik> ok, good
460 2015-11-10T20:27:57  <wumpus> jgarzik: in case of bitcoin core that's noto the case, only still-unset settings get overridden
461 2015-11-10T20:28:54  <morcos> sipa: so you still need a limit for the mempool right, which makes sense to be a fraction of your total
462 2015-11-10T20:29:13  <morcos> you cant just go and trim your mempool a bunch when a whole new set of txins get cached
463 2015-11-10T20:29:22  *** GAit has joined #bitcoin-core-dev
464 2015-11-10T20:29:39  <wumpus> I don't think utxo cache and mempool are very related
465 2015-11-10T20:30:01  <morcos> yeah i mostly agree with wumpus there
466 2015-11-10T20:30:14  <sipa> wumpus: well they're both essentially caches of information that we'll except to need for validating the next block
467 2015-11-10T20:30:21  <wumpus> enough reasons to increase one and not the other. I'm fine with setting some sane defaults, but they shouldn't be linked in general
468 2015-11-10T20:30:21  <sipa> *expect
469 2015-11-10T20:30:41  <sipa> except the utxo cache is *also* a buffer of unwritten changes
470 2015-11-10T20:30:53  <wumpus> well a smaller mempool doens't make your node slower, a smaller dbcache does
471 2015-11-10T20:30:55  <morcos> sipa: the mempool doesn't serve as a cache does it
472 2015-11-10T20:30:58  <morcos> right
473 2015-11-10T20:31:16  *** GAit has quit IRC
474 2015-11-10T20:31:17  <wumpus> so even though they're 'caches' they have a widely different reason
475 2015-11-10T20:31:19  *** fkhan has joined #bitcoin-core-dev
476 2015-11-10T20:31:26  <wumpus> mempool is necessary for correct functioning, dbcache is just for performance
477 2015-11-10T20:31:57  <morcos> wumpus, the idea of linking though is that why make the user figure out how to divide up his available N gigs of memory
478 2015-11-10T20:32:03  <jgarzik> User experience.  What does the user need to do to achieve a useful config on both large & small boxes?
479 2015-11-10T20:32:07  <morcos> we should just do soemthing vaguely smart by default
480 2015-11-10T20:32:09  *** GAit has joined #bitcoin-core-dev
481 2015-11-10T20:32:09  <jgarzik> nod
482 2015-11-10T20:32:43  <wumpus> morcos: yes as said by default you could do that, make a -memoryquota parameter or such that automatically allocates them, like automatic partitioning in an OS instlal
483 2015-11-10T20:32:55  <wumpus> morcos: but it should certainly be possible to set them separately
484 2015-11-10T20:32:59  <dhill> getrlimit and base initial memory sizes on that?
485 2015-11-10T20:33:02  <morcos> ok agreed
486 2015-11-10T20:33:17  <wumpus> dhill: that assumes everyone wants to give all their memory to bitcoind
487 2015-11-10T20:33:58  <dhill> naw
488 2015-11-10T20:34:01  <wumpus> and no, just because I have 32GB of memory now doesn't mean I want to allocate it all to my node, and have compiles crash again...
489 2015-11-10T20:34:02  *** jgarzik has quit IRC
490 2015-11-10T20:34:10  <dhill> i didn't suggest that
491 2015-11-10T20:34:19  <morcos> so -maxmempool default is some fraction of memoryquota which also has a default.  , but then if you individually set your maxmempool,sigcahe,or dbcache, you are not necessarily going to respect your memoryquota
492 2015-11-10T20:34:51  <sipa> i think it makes sense for bitcoin-qt to do something smartish to guess how much memory to use for caches
493 2015-11-10T20:35:15  *** jgarzik has joined #bitcoin-core-dev
494 2015-11-10T20:35:31  <morcos> well if we do this memory quota idea, then we can always have smart code that sets that as a later addition if we want
495 2015-11-10T20:35:45  <wumpus> agreed, doing something smart is good, but basing it (trivially) on the available memory does't make too much sense imo
496 2015-11-10T20:36:08  <wumpus> people generally want to run their node inthe background
497 2015-11-10T20:36:12  <morcos> so back to my original quesiton  if i set memoryquota=N.   what should the 3 components each be
498 2015-11-10T20:36:30  <wumpus> having applications use more memory just because it is available, and no othe reason is...weird
499 2015-11-10T20:36:42  *** jgarzik has quit IRC
500 2015-11-10T20:37:00  <morcos> .4 N for maxmempol, .5N for dbacache , .1N for maxsigcachesize ?
501 2015-11-10T20:37:04  <wumpus> I would be angry if firefox used more memory per tab just because I have more, that's so self defeating
502 2015-11-10T20:37:15  <morcos> sdaftuar was trying to convince me 2 * is too much for dbcache
503 2015-11-10T20:38:42  <morcos> and default memory quota to 800M which should be around the 300M mempool we had previously been planning for, and keep total usage to under a gig (ugh, i hope, what else uses memory)
504 2015-11-10T20:38:44  <wumpus> dbcache is nice but apart from during IBD ,not too important
505 2015-11-10T20:39:32  <morcos> wumpus: i like the idea that block relay over p2p is very fast.  i think that depending on the relay network for efficient block relay is a bad idea, even if p2p will never be quite as fast
506 2015-11-10T20:39:34  <wumpus> I sort of like the idea to use the mempool quota for UTXO cache during IBD, after all the mempool will be almost empty at that time
507 2015-11-10T20:39:56  <wumpus> though making them linked is less nice for seperation of concerns
508 2015-11-10T20:40:04  <morcos> for block relay to be fast, you need decent dbcache size
509 2015-11-10T20:43:09  *** paveljanik has quit IRC
510 2015-11-10T20:43:18  <wumpus> a better eviction policy would help too there :)
511 2015-11-10T20:44:44  *** jgarzik has joined #bitcoin-core-dev
512 2015-11-10T20:44:44  *** jgarzik has joined #bitcoin-core-dev
513 2015-11-10T20:44:59  <morcos> i have a PR for that.  :)  but it requires still a decent size dbcache.  a 1MB block might need over 50MB of dbcache just for itself, if you knew exactly what was goign to be confirmed in that block
514 2015-11-10T20:46:49  <wumpus> ok cool
515 2015-11-10T20:47:01  *** zooko has joined #bitcoin-core-dev
516 2015-11-10T20:48:04  <jgarzik> "having applications use more memory just because it is available, and no othe reason is...weird"   <--  That's pretty much exactly how OS page cache works
517 2015-11-10T20:48:28  <jgarzik> And I'm looking at how memory usage changes once mempool is on disk, and nearly everything goes through OS page cache
518 2015-11-10T20:49:37  <wumpus> yes at the OS level you can get away wit hit
519 2015-11-10T20:50:21  <wumpus> e.g. page cache isn't considered 'used' memory in most computations, it can be evicted when an application really needs it
520 2015-11-10T20:52:23  <jgarzik> quite true if you s/application/OS/   -- though "working set" is part of the app.  If working set size gets below a threshold, performance falls due to increased I/O.   As such, it is part of the app, and one that scales when more memory is available.
521 2015-11-10T20:52:27  <wumpus> you really can't do that in a well-behaving application - yes I know it's possible to use some madvise trick to map application pages as volatile, but supporting that cross platform will be a nightmare, and it will probably still look to the user as if it uses a lot of memory unconditionally
522 2015-11-10T20:55:27  *** Guest8443 is now known as berndj
523 2015-11-10T20:57:14  *** zooko has quit IRC
524 2015-11-10T20:58:18  *** CodeShark has joined #bitcoin-core-dev
525 2015-11-10T20:58:36  <wumpus> yes I wasn't talking about the page cache, but a hypothetical application that would ask the OS how much memory is available and claim a fixed part of that. It would make the egotistical assumption that a user buys more memory just to give your application more space :)
526 2015-11-10T20:59:10  <jgarzik> hehe
527 2015-11-10T21:00:19  *** zooko has joined #bitcoin-core-dev
528 2015-11-10T21:05:23  <wumpus> it would be very nice if we could rely on the page cache though for caching the db, this whole utxo cache is mostly a workaround because there is a lot of retrieve overhead even if the data is cached in RAM (do we even know why? deserialization/allocation overhead?)
529 2015-11-10T21:06:41  <gmaxwell> has nothing to do with storage vs ram, you get basically the same speedups from big utxo cache even if the datadir is in tmpfs.
530 2015-11-10T21:07:49  <morcos> gmaxwell: oh really?? why is that
531 2015-11-10T21:08:27  <wumpus> that's what I mean with overhead even if the data is already in RAM
532 2015-11-10T21:09:19  <wumpus> storing the datadir in tmpfs is an extreme example of that, but normally the page cache would make sure at least a part of the files are still cached
533 2015-11-10T21:10:10  <wumpus> morcos: that's what I wonder too, probably deserialization/allocation overhead, going from storage representation to internal data structure
534 2015-11-10T21:11:29  *** belcher has joined #bitcoin-core-dev
535 2015-11-10T21:12:21  <gmaxwell> I had a prior belief but pieter tells me my understanding of leveldb's behavior was not correct. (I thought leveldb did a sequential scan of the log to find any records there)
536 2015-11-10T21:12:22  <wumpus> another possibility is that leveldb queries are really slow even if there is no seek/io overhead - maybe because of the all-pervasive checksum verification
537 2015-11-10T21:13:10  <sipa> gmaxwell: the log is sequential, but is never read
538 2015-11-10T21:13:31  <wumpus> anecdotally when I broke off block verification a few times in gdb it always ended up in leveldb checksum verification. That's no serious way to do profiling though :-)
539 2015-11-10T21:13:35  <sipa> gmaxwell: it's compacted into the database at startup (which is why startup after shutdown with high dbcache is slow, it has large log to process)
540 2015-11-10T21:14:14  <sipa> the database however has several levels, and several (non-overlapping) files for every level
541 2015-11-10T21:14:26  <sipa> which means it may need to scan through multiple levels to find data
542 2015-11-10T21:14:50  <sipa> every file has a bloom filter, so it will quickly skip the ones that don't have the requested amount
543 2015-11-10T21:14:58  <wumpus> right
544 2015-11-10T21:15:05  <sipa> within every file, it uses a bisection search to find the data, i think
545 2015-11-10T21:15:21  <wumpus> yes IIRC too
546 2015-11-10T21:15:24  <sipa> or a bisection search in its index
547 2015-11-10T21:15:27  <sipa> which points to the data
548 2015-11-10T21:18:14  *** GAit has quit IRC
549 2015-11-10T21:18:21  *** GAit has joined #bitcoin-core-dev
550 2015-11-10T21:26:45  *** GAit has quit IRC
551 2015-11-10T21:28:39  *** GAit has joined #bitcoin-core-dev
552 2015-11-10T21:32:03  *** dcousens has joined #bitcoin-core-dev
553 2015-11-10T21:34:17  *** molly has quit IRC
554 2015-11-10T21:40:21  *** ParadoxSpiral_ has quit IRC
555 2015-11-10T21:56:34  *** CodeShark_ has joined #bitcoin-core-dev
556 2015-11-10T21:56:45  *** CodeShark_ has quit IRC
557 2015-11-10T21:57:54  *** CodeShark_ has joined #bitcoin-core-dev
558 2015-11-10T21:58:19  *** CodeShark has quit IRC
559 2015-11-10T22:04:54  *** Guyver2 has quit IRC
560 2015-11-10T22:32:30  *** GAit has quit IRC
561 2015-11-10T22:33:17  *** GAit has joined #bitcoin-core-dev
562 2015-11-10T22:44:58  *** CodeShark_ has quit IRC
563 2015-11-10T22:45:11  *** CodeShark has joined #bitcoin-core-dev
564 2015-11-10T22:52:01  *** GAit has quit IRC
565 2015-11-10T22:53:16  *** GAit has joined #bitcoin-core-dev
566 2015-11-10T22:57:02  *** GAit has quit IRC
567 2015-11-10T23:15:38  *** molly has joined #bitcoin-core-dev
568 2015-11-10T23:58:31  <gmaxwell> Would having a protocol message that says "please don't INV loose transactions to me, I'm mostly only interested in transactions in blocks"  be a bad idea?  I want to have bandwidth limited nodes that don't care about unconfirmed txn not recieve anything but confirmed transactions.