1 2018-09-27T00:07:01  *** Murch has joined #bitcoin-core-dev
  2 2018-09-27T00:11:36  *** Krellan has quit IRC
  3 2018-09-27T00:12:42  *** Krellan has joined #bitcoin-core-dev
  4 2018-09-27T00:16:01  *** irc_viewer_test has quit IRC
  5 2018-09-27T00:26:39  *** murrayn has quit IRC
  6 2018-09-27T00:28:42  *** murrayn has joined #bitcoin-core-dev
  7 2018-09-27T00:36:46  *** promag has quit IRC
  8 2018-09-27T00:56:19  *** Randolf has joined #bitcoin-core-dev
  9 2018-09-27T01:05:56  *** jarthur has joined #bitcoin-core-dev
 10 2018-09-27T01:06:25  <jarthur> Any of you pretty familiar with the functional tests?
 11 2018-09-27T01:07:02  <jarthur> In the ones that spin up a couple peers, I'm curious how deterministic the peer numbers are when asserting log text.
 12 2018-09-27T01:14:26  <sipa> the peer numbers are sequential
 13 2018-09-27T01:14:31  <sipa> time of connectio
 14 2018-09-27T01:19:26  <jarthur> Makes sense. Do the multi-peer test usually connect the nodes in sequence to avoid non-deterministic output?
 15 2018-09-27T01:19:31  <jarthur> s/test/tests/
 16 2018-09-27T01:25:46  *** Emcy has quit IRC
 17 2018-09-27T01:28:36  *** Emcy has joined #bitcoin-core-dev
 18 2018-09-27T02:01:06  <phantomcircuit> anybody know why the linter is failing on #14336
 19 2018-09-27T02:01:08  <gribble> https://github.com/bitcoin/bitcoin/issues/14336 | net: implement poll by pstratem · Pull Request #14336 · bitcoin/bitcoin · GitHub
 20 2018-09-27T02:06:22  <phantomcircuit> ah i see nvm
 21 2018-09-27T02:06:39  <gmaxwell> jarthur: I think they are in practice, but as a general rule a test should try to avoid being sensitive to things other than what they're testing, otherwise it makes them brittle.
 22 2018-09-27T02:06:57  <phantomcircuit> the white space linter is confusing when run against multiple commits the corresponding message are nonsensical
 23 2018-09-27T02:21:21  *** Randolf has quit IRC
 24 2018-09-27T02:26:54  *** Murch has quit IRC
 25 2018-09-27T02:32:03  <jarthur> gmaxwell: thanks
 26 2018-09-27T02:45:03  *** charley has joined #bitcoin-core-dev
 27 2018-09-27T03:02:27  <phantomcircuit> aight i have no idea why it's failing that test and cannot reproduce locally
 28 2018-09-27T03:03:00  <gmaxwell> phantomcircuit: looking
 29 2018-09-27T03:03:17  <phantomcircuit> https://travis-ci.org/bitcoin/bitcoin/jobs/433861290
 30 2018-09-27T03:03:27  <phantomcircuit> bitcoind exiting with code -6 during initialization
 31 2018-09-27T03:03:39  <gmaxwell> phantomcircuit: so thats an enable debug build.
 32 2018-09-27T03:03:46  <phantomcircuit> yes
 33 2018-09-27T03:04:44  <gmaxwell> so with enable debug, rpc bind functional test passes for you?
 34 2018-09-27T03:05:28  <phantomcircuit> yes
 35 2018-09-27T03:05:32  <phantomcircuit> all tests pass actually
 36 2018-09-27T03:06:00  <phantomcircuit>  ./configure --with-debug --with-incompatible-bdb --enable-zmq --with-gui=qt5 --enable-glibc-back-compat --enable-reduce-exports --enable-debug
 37 2018-09-27T03:08:39  <phantomcircuit> the only -6 is see as a constant is the addrman check but that should just write to the log
 38 2018-09-27T03:08:51  <gmaxwell> I dunno what -6 means there... like if it died on due to an unhandled signal it would one hundred and something.
 39 2018-09-27T03:11:19  <gmaxwell> perhaps, $ errno 6
 40 2018-09-27T03:11:19  <gmaxwell> ENXIO 6 No such device or address
 41 2018-09-27T03:11:41  <gmaxwell> there might be a case where the test triggers an error that the old code ignores and the new code propagates.
 42 2018-09-27T03:12:00  <gmaxwell> like what happens if you try to bind to 127.0.0.1 but there isn't a 127.0.0.1?
 43 2018-09-27T03:13:15  *** prometheus_falli has quit IRC
 44 2018-09-27T03:16:42  *** prometheus_falli has joined #bitcoin-core-dev
 45 2018-09-27T03:18:18  <phantomcircuit> gmaxwell, should fail in the same ways im pretty sure
 46 2018-09-27T03:18:51  *** irc_viewer_test has joined #bitcoin-core-dev
 47 2018-09-27T03:21:10  <gmaxwell> in any case a reason that test may fail on travis and work for you, is that the travis enviroment probably has pretty different networking.
 48 2018-09-27T03:21:25  <phantomcircuit> yeah im sure it does
 49 2018-09-27T03:22:52  <gmaxwell> And the reason it would matter is if under that case, you handle errors differently, e.g. the actual bug may be elsewhere in bitcoin or in the test but were hidden by the old behavior.  The fact that you can't reproduce it locally is kind of annoying. you could try to figure out which test case is failing, by adding commits to change the test.
 50 2018-09-27T03:25:18  *** asoltys has joined #bitcoin-core-dev
 51 2018-09-27T03:30:38  *** irc_viewer_test has quit IRC
 52 2018-09-27T03:45:30  <phantomcircuit> gmaxwell, indeed
 53 2018-09-27T04:06:38  *** jarthur has quit IRC
 54 2018-09-27T04:07:16  *** jarthur has joined #bitcoin-core-dev
 55 2018-09-27T04:09:49  <phantomcircuit> gmaxwell, i guess travis doesn't have ipv4?
 56 2018-09-27T04:09:57  <phantomcircuit> it's specifically the ipv4 rpc bind that's failing
 57 2018-09-27T04:11:35  <phantomcircuit> hmm it doesn't , not specifically 127.0.0.1
 58 2018-09-27T04:11:50  <phantomcircuit> which is what the test tries to bind to
 59 2018-09-27T04:22:40  *** jarthur has quit IRC
 60 2018-09-27T04:29:06  <phantomcircuit> gmaxwell, blargh there's still the select() calls in netbase
 61 2018-09-27T04:33:21  *** molz has joined #bitcoin-core-dev
 62 2018-09-27T04:34:32  *** mol has quit IRC
 63 2018-09-27T04:55:36  *** escrivner has joined #bitcoin-core-dev
 64 2018-09-27T05:05:04  *** qrestlove has quit IRC
 65 2018-09-27T05:13:53  <kallewoof> wumpus: noticed that NicolasDorier is not in the Credits list despite him being listed for 9991 at https://github.com/bitcoin-core/bitcoin-devwiki/wiki/0.17.0-Release-notes
 66 2018-09-27T05:14:07  <kallewoof> we just edit? I thought credits were automagic
 67 2018-09-27T05:19:56  *** itaseski has joined #bitcoin-core-dev
 68 2018-09-27T05:39:10  *** qrestlove has joined #bitcoin-core-dev
 69 2018-09-27T05:44:38  *** Victorsueca has quit IRC
 70 2018-09-27T05:45:46  *** Victorsueca has joined #bitcoin-core-dev
 71 2018-09-27T06:08:16  *** itaseski has quit IRC
 72 2018-09-27T06:20:25  *** go1111111 has quit IRC
 73 2018-09-27T06:33:15  *** emzy has quit IRC
 74 2018-09-27T06:36:26  *** go1111111 has joined #bitcoin-core-dev
 75 2018-09-27T07:02:36  *** setpill has joined #bitcoin-core-dev
 76 2018-09-27T07:34:36  <jonasschnelli> Bitcoin Qt looks a bit strange in OSX 10.14 (Mojaves) dark mode. :)
 77 2018-09-27T07:35:07  *** t0adst00l has joined #bitcoin-core-dev
 78 2018-09-27T07:35:29  *** nullptr| has quit IRC
 79 2018-09-27T07:36:25  *** prometheus_falli has quit IRC
 80 2018-09-27T07:45:06  *** nullptr| has joined #bitcoin-core-dev
 81 2018-09-27T08:12:16  *** Krellan has quit IRC
 82 2018-09-27T08:12:38  *** Krellan has joined #bitcoin-core-dev
 83 2018-09-27T08:13:30  *** emzy has joined #bitcoin-core-dev
 84 2018-09-27T08:16:55  *** timothy has joined #bitcoin-core-dev
 85 2018-09-27T08:25:41  *** jb55 has joined #bitcoin-core-dev
 86 2018-09-27T08:37:02  <wumpus> kallewoof: the lists are generated automatically but always need human editing
 87 2018-09-27T08:37:44  <wumpus> kallewoof: the list of PRs is generated from github API data, the authors list from git commit authors—this might be what causes the divergence in this case
 88 2018-09-27T08:38:17  <wumpus> jonasschnelli: that's curious, I always run it with dark gtk themes and that seems to go well
 89 2018-09-27T08:38:55  <jonasschnelli> wumpus: Could be because of the different platformstyles... investigating now
 90 2018-09-27T08:39:19  <jonasschnelli> black icons on black background are not ideal.. but at least you can see them
 91 2018-09-27T08:39:33  <jonasschnelli> correction: black icons on dark-gray background
 92 2018-09-27T08:40:46  <wumpus> jonasschnelli: yes that's it, I think on UNIX it does an attempt to grab the icon color from the theme, on MacOS is defaults to black
 93 2018-09-27T08:41:06  <jonasschnelli> jup...
 94 2018-09-27T08:42:40  <wumpus> kallewoof: that's it; the commits in 9991 are by JeremyRubin and apparently my script lists authors only, not committers
 95 2018-09-27T08:43:11  <wumpus> kallewoof: I'll try to fix the script, otherwise I'll just edit him in manually
 96 2018-09-27T08:57:02  *** arubi has quit IRC
 97 2018-09-27T08:59:27  *** promag has joined #bitcoin-core-dev
 98 2018-09-27T09:01:01  <wumpus> kallewoof: added in committers to my list-authors script; looks like Dorier was the only one missed to this
 99 2018-09-27T09:04:23  <wumpus> phantomcircuit: travis does have IPv4, but no IPv6 IIRC
100 2018-09-27T09:04:29  <wumpus> (*not even localhost*)
101 2018-09-27T09:10:25  <echeveria> I fixed testnet, for what it's worth. the most work is now in a valid chain.
102 2018-09-27T09:24:49  *** hebasto has joined #bitcoin-core-dev
103 2018-09-27T09:28:11  <jonasschnelli> wumpus: using the dark arc theme in Ubuntu Bionic, the background of Bitcoin Qt is still light gray/whiteish? Is that expected?
104 2018-09-27T09:30:27  *** e4xit has quit IRC
105 2018-09-27T09:32:44  <wumpus> jonasschnelli: I don't think so; the bitcoin-qt background should be the same as other gtk applications, say "charmap"
106 2018-09-27T09:33:21  <wumpus> jonasschnelli: though that might only work if you build from source, theme integration is not available with the shipped binaries
107 2018-09-27T09:33:54  <jonasschnelli> aha... I see.
108 2018-09-27T09:34:41  *** e4xit has joined #bitcoin-core-dev
109 2018-09-27T09:34:48  <wumpus> (the latter because it's based on plugins, which are not available when building qt statically)
110 2018-09-27T09:35:47  *** SopaXorzTaker has joined #bitcoin-core-dev
111 2018-09-27T09:36:12  <jonasschnelli> wumpus: I would have expected that the OS provides some sort of color scheme for background, etc. (macOS Mojave) handles it that way). Qt could use OsBackgroundColor(), etc. for certain things.
112 2018-09-27T09:36:20  <wumpus> huh—looks broken on ubuntu 18.04, I get a light grey background now too w/ Adwaita-dark theme
113 2018-09-27T09:36:33  <wumpus> built against system qt
114 2018-09-27T09:36:42  <wumpus> AHH when did this happen
115 2018-09-27T09:36:42  <jonasschnelli> hmm...
116 2018-09-27T09:36:54  <jonasschnelli> Must be a Qt upstream issue
117 2018-09-27T09:36:59  <jonasschnelli> (or a flag)
118 2018-09-27T09:37:14  <wumpus> yes, indeed
119 2018-09-27T09:37:25  <jonasschnelli> https://askubuntu.com/questions/910012/how-can-i-get-qt5-applications-to-use-the-gtk-theme-in-ubuntu-17-04
120 2018-09-27T09:37:31  <wumpus> I see the same in Quasselclient
121 2018-09-27T09:37:37  <wumpus> which is also a qt (5 AFAIK) app
122 2018-09-27T09:37:54  <jonasschnelli> «The problem has occurred since Qt5.7. In this release, the GTK2 platform theme and style was removed and replaced with the GTK3 platform theme»
123 2018-09-27T09:38:29  <wumpus> but how! I'm sure my theme also covers gtk3
124 2018-09-27T09:38:59  <wumpus> 'charmap' is gtk3 and has a dark grey background
125 2018-09-27T09:39:53  * jonasschnelli loving QT
126 2018-09-27T09:42:47  *** t0adst00l has quit IRC
127 2018-09-27T09:44:55  *** belcher has joined #bitcoin-core-dev
128 2018-09-27T09:52:45  <wumpus> I guess... it is the least of evils, with regard to cross-platform GUIs. Though losing theme integration on by far most linux distributions (except KDE-based ones?) is a pity.
129 2018-09-27T09:56:23  <wumpus> installing qt5-style-plugins and then doing 'export QT_QPA_PLATFORMTHEME=gtk2' works here, but that's helluva awkward
130 2018-09-27T09:57:20  <hebasto> wumpus: what is the way to build bitcoin-qt against system qt?
131 2018-09-27T10:06:24  <wumpus> hebasto: it does that by default if you don't do a depends build
132 2018-09-27T10:06:33  <wumpus> just follow the steps in build-unix.md
133 2018-09-27T10:09:16  <wumpus> that way, bitcoin won't magically install its own qt; so it will build against system qt if available, and not build against qt at all if not available
134 2018-09-27T10:12:00  <wumpus> this is the preferred way, we even used to ship that way for qt4, but unfortunately the range of portability of the executables is much smaller then so it's not an option anymore (then again, nothing of this matters anymore, if qt upstream broke desktop integration...)
135 2018-09-27T10:17:29  *** t0adst00l has joined #bitcoin-core-dev
136 2018-09-27T10:23:30  <hebasto> wumpus: thank you
137 2018-09-27T10:28:24  *** Krellan has quit IRC
138 2018-09-27T10:30:44  *** Krellan has joined #bitcoin-core-dev
139 2018-09-27T10:43:06  *** SopaXorzTaker has quit IRC
140 2018-09-27T10:43:37  *** SopaXorzTaker has joined #bitcoin-core-dev
141 2018-09-27T10:57:32  *** Victor_sueca has joined #bitcoin-core-dev
142 2018-09-27T10:57:54  *** Victorsueca has quit IRC
143 2018-09-27T11:00:47  *** Victor_sueca has quit IRC
144 2018-09-27T11:01:38  *** Victor_sueca has joined #bitcoin-core-dev
145 2018-09-27T11:10:07  <provoostenator> wumpus: "great" to hear macOS isn't the only platform having QT headaches now.
146 2018-09-27T11:17:54  *** setpill has quit IRC
147 2018-09-27T11:20:01  *** Kvaciral has quit IRC
148 2018-09-27T11:25:06  *** Krellan has quit IRC
149 2018-09-27T11:26:05  *** Krellan has joined #bitcoin-core-dev
150 2018-09-27T11:56:51  *** promag has quit IRC
151 2018-09-27T12:25:03  *** adiabat has quit IRC
152 2018-09-27T12:26:06  <wumpus> GUIs were a mistake
153 2018-09-27T12:30:19  <Sentineo> an audio interface would have been betterperhaps :)
154 2018-09-27T12:33:26  *** schnerchi has joined #bitcoin-core-dev
155 2018-09-27T12:41:43  <wumpus> direct brain interface ftw :)
156 2018-09-27T12:42:22  <Sentineo> yeah :)
157 2018-09-27T12:56:11  *** Chris_Stewart_5 has joined #bitcoin-core-dev
158 2018-09-27T13:10:41  *** elichai2 has joined #bitcoin-core-dev
159 2018-09-27T13:22:03  *** scroll2 has joined #bitcoin-core-dev
160 2018-09-27T13:24:38  *** csknk has joined #bitcoin-core-dev
161 2018-09-27T13:30:24  *** irc_viewer_test has joined #bitcoin-core-dev
162 2018-09-27T13:37:29  *** promag has joined #bitcoin-core-dev
163 2018-09-27T13:48:50  *** irc_viewer_test1 has joined #bitcoin-core-dev
164 2018-09-27T13:50:47  <promag> hebasto: hi
165 2018-09-27T13:51:14  <hebasto> promag: hi
166 2018-09-27T13:51:23  <promag> hebasto: I don't have space to install fedora :/ but I hope to get around that today/tomorrow
167 2018-09-27T13:51:30  *** irc_viewer_test has quit IRC
168 2018-09-27T13:51:45  <hebasto> promag: I see
169 2018-09-27T13:51:56  <promag> hebasto: can only test by then
170 2018-09-27T13:53:10  <hebasto> promag: don't be hurry, I'm working on your suggestions right now
171 2018-09-27T13:53:39  <promag> so in fact it's necessary to not map?
172 2018-09-27T13:54:17  <hebasto> yes, it is. I've tested on fedora.
173 2018-09-27T14:00:08  <hebasto> promag: would you mind to look into #14222?
174 2018-09-27T14:00:09  <gribble> https://github.com/bitcoin/bitcoin/issues/14222 | Qt: Fix restoration of minimized to tray window by hebasto · Pull Request #14222 · bitcoin/bitcoin · GitHub
175 2018-09-27T14:01:37  *** Emcy has quit IRC
176 2018-09-27T14:04:53  *** Emcy has joined #bitcoin-core-dev
177 2018-09-27T14:16:03  <promag> hebasto: will fo
178 2018-09-27T14:16:08  <promag> *do
179 2018-09-27T14:28:11  *** profmac has joined #bitcoin-core-dev
180 2018-09-27T14:34:57  *** AaronvanW has quit IRC
181 2018-09-27T14:37:32  *** csknk has quit IRC
182 2018-09-27T14:39:15  *** promag has quit IRC
183 2018-09-27T14:39:50  *** AaronvanW has joined #bitcoin-core-dev
184 2018-09-27T14:50:14  *** michaelsdunn1 has joined #bitcoin-core-dev
185 2018-09-27T14:55:29  *** michaelsdunn1 has quit IRC
186 2018-09-27T15:01:46  *** michaelsdunn1 has joined #bitcoin-core-dev
187 2018-09-27T15:02:23  *** irc_viewer_test1 has quit IRC
188 2018-09-27T15:16:43  *** jarthur has joined #bitcoin-core-dev
189 2018-09-27T15:24:28  *** Murch has joined #bitcoin-core-dev
190 2018-09-27T15:28:10  *** TheCharlatan has joined #bitcoin-core-dev
191 2018-09-27T15:34:00  *** irc_viewer_test has joined #bitcoin-core-dev
192 2018-09-27T15:39:54  *** Victor_sueca is now known as Victorsueca
193 2018-09-27T15:40:24  *** emilengler has joined #bitcoin-core-dev
194 2018-09-27T15:43:01  *** irc_viewer_test has quit IRC
195 2018-09-27T16:09:54  *** Krellan has quit IRC
196 2018-09-27T16:10:46  *** emilengler has quit IRC
197 2018-09-27T16:10:59  *** Krellan has joined #bitcoin-core-dev
198 2018-09-27T16:12:01  *** emilengler has joined #bitcoin-core-dev
199 2018-09-27T16:13:18  *** emilengler has quit IRC
200 2018-09-27T16:19:46  *** adiabat has joined #bitcoin-core-dev
201 2018-09-27T16:28:06  <luke-jr> 57b59260952742aa51dca79a37849429a456496292d3e4f28cdf7de3eef516f3
202 2018-09-27T16:29:12  *** Krellan has quit IRC
203 2018-09-27T16:32:06  *** jarthur has quit IRC
204 2018-09-27T16:32:16  *** promag has joined #bitcoin-core-dev
205 2018-09-27T16:34:26  *** Krellan has joined #bitcoin-core-dev
206 2018-09-27T16:36:36  *** jarthur has joined #bitcoin-core-dev
207 2018-09-27T16:40:37  <promag> luke-jr: heh
208 2018-09-27T16:54:04  *** jungly has quit IRC
209 2018-09-27T16:55:51  *** rex4539 has quit IRC
210 2018-09-27T16:58:47  *** tryphe has joined #bitcoin-core-dev
211 2018-09-27T17:06:18  *** tryphe has quit IRC
212 2018-09-27T17:14:11  *** promag has quit IRC
213 2018-09-27T17:42:45  *** t0adst00l has quit IRC
214 2018-09-27T17:44:21  *** Victorsueca has quit IRC
215 2018-09-27T17:45:39  *** Victorsueca has joined #bitcoin-core-dev
216 2018-09-27T17:54:03  *** escrivner has left #bitcoin-core-dev
217 2018-09-27T18:15:52  *** tryphe has joined #bitcoin-core-dev
218 2018-09-27T18:19:42  *** timothy has quit IRC
219 2018-09-27T18:24:09  *** irc_viewer_test has joined #bitcoin-core-dev
220 2018-09-27T18:28:14  *** Krellan has quit IRC
221 2018-09-27T18:48:16  *** irc_viewer_test has quit IRC
222 2018-09-27T18:49:31  *** promag has joined #bitcoin-core-dev
223 2018-09-27T18:55:39  *** clarkmoody has joined #bitcoin-core-dev
224 2018-09-27T19:00:33  <wumpus> meeting time?
225 2018-09-27T19:00:36  <jonasschnelli> hiyes
226 2018-09-27T19:00:52  <wumpus> #startmeeting
227 2018-09-27T19:00:52  <lightningbot> Meeting started Thu Sep 27 19:00:52 2018 UTC.  The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
228 2018-09-27T19:00:52  <lightningbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
229 2018-09-27T19:01:02  <promag> hi
230 2018-09-27T19:01:19  <wumpus> #bitcoin-core-dev Meeting: wumpus sipa gmaxwell jonasschnelli morcos luke-jr btcdrak sdaftuar jtimon cfields petertodd kanzure bluematt instagibbs phantomcircu
231 2018-09-27T19:01:21  <provoostenator> hi
232 2018-09-27T19:01:22  <wumpus> it codeshark michagogo marcofalke paveljanik NicolasDorier jl2012 achow101 meshcollider jnewbery maaku fanquake promag provoostenator
233 2018-09-27T19:01:26  <cfields> hi
234 2018-09-27T19:01:29  <achow101> hi
235 2018-09-27T19:02:01  <meshcollider> Hi
236 2018-09-27T19:02:24  <jonasschnelli> Status of 0.15.2 and 0.14.3?
237 2018-09-27T19:02:45  <phantomcircuit> hi
238 2018-09-27T19:03:27  <wumpus> jonasschnelli: good question; are there enough gitian sigs to upload binaries?
239 2018-09-27T19:03:42  <jonasschnelli> I think so... 5 or 6
240 2018-09-27T19:03:47  <luke-jr> yeah
241 2018-09-27T19:04:15  <jamesob> hi
242 2018-09-27T19:04:19  <achow101> 0.14.3 has 6, 0.15.2 has 5
243 2018-09-27T19:04:19  <jonasschnelli> 6 for 0.14.3 and 5 for 0.15.2 AFAIK
244 2018-09-27T19:04:34  <wumpus> ok, will do that then, I'm back from Riga so I'm able to sign/upload binaries again
245 2018-09-27T19:04:36  <jonasschnelli> due to my shitty setup, I can't compile trusty stuff on Gitian anymore
246 2018-09-27T19:04:46  <jonasschnelli> thanks wumpus
247 2018-09-27T19:05:05  <wumpus> 0.14/0.15 build still seems to work here
248 2018-09-27T19:05:31  <provoostenator> It took some pain for me as well, but I still keep an old Gitian VM for backports.
249 2018-09-27T19:05:32  <wumpus> #topic 0.17.0 release
250 2018-09-27T19:06:24  <jonasschnelli> Not sure if #14339 is something we want to address for 0.17... probably not
251 2018-09-27T19:06:24  <gribble> https://github.com/bitcoin/bitcoin/issues/14339 | Qt 0.17.0rc4 (and master) not running on Ubuntu 14.04.5 LTS · Issue #14339 · bitcoin/bitcoin · GitHub
252 2018-09-27T19:06:43  <provoostenator> So macOS GUI compilation seems completely broken: #14327, but that wouldn't stop cross compling a release I suppose.
253 2018-09-27T19:06:44  <gribble> https://github.com/bitcoin/bitcoin/issues/14327 | macOS Mojave QT 5.11 compilation fails · Issue #14327 · bitcoin/bitcoin · GitHub
254 2018-09-27T19:06:49  <wumpus> so the blocker for 0.17 is #14289
255 2018-09-27T19:06:50  <gribble> https://github.com/bitcoin/bitcoin/issues/14289 | Unbounded growth of scheduler queue · Issue #14289 · bitcoin/bitcoin · GitHub
256 2018-09-27T19:07:25  <wumpus> the GUI problems are annoying and need to be fixed but are not blocking the release at this stage, IMO
257 2018-09-27T19:07:25  <instagibbs> hi
258 2018-09-27T19:07:32  <jonasschnelli> provoostenator: hmm.. compiled master yesterday on a fresh Mojave installation
259 2018-09-27T19:07:36  <jonasschnelli> (no problems)
260 2018-09-27T19:07:50  <jonasschnelli> wumpus: agree
261 2018-09-27T19:08:18  <provoostenator> jonasschnelli: ok, maybe it's machine specific? Can you and others comment on that issue?
262 2018-09-27T19:08:26  <jonasschnelli> (will do later)
263 2018-09-27T19:09:15  *** jimmysong has joined #bitcoin-core-dev
264 2018-09-27T19:09:23  <provoostenator> (I'm trying now on my Macbook as well, maybe it's just my iMac being a pain)
265 2018-09-27T19:09:27  *** SopaXorzTaker has quit IRC
266 2018-09-27T19:09:42  <jonasschnelli> What about #14289 and #14104?
267 2018-09-27T19:09:43  <gribble> https://github.com/bitcoin/bitcoin/issues/14289 | Unbounded growth of scheduler queue · Issue #14289 · bitcoin/bitcoin · GitHub
268 2018-09-27T19:09:44  <gribble> https://github.com/bitcoin/bitcoin/issues/14104 | 0.17.2rc issue (standardness change for bare multisig) · Issue #14104 · bitcoin/bitcoin · GitHub
269 2018-09-27T19:09:50  <wumpus> although, it's likely that #14289 is not a regression for 0.17 it's still something that needs to be addressed in some way
270 2018-09-27T19:09:51  <gribble> https://github.com/bitcoin/bitcoin/issues/14289 | Unbounded growth of scheduler queue · Issue #14289 · bitcoin/bitcoin · GitHub
271 2018-09-27T19:10:05  <wumpus> jonasschnelli: I don't think #14104 is a blocker, but the other one is nasty
272 2018-09-27T19:10:06  <gribble> https://github.com/bitcoin/bitcoin/issues/14104 | 0.17.2rc issue (standardness change for bare multisig) · Issue #14104 · bitcoin/bitcoin · GitHub
273 2018-09-27T19:10:23  <provoostenator> 14289: one "solution" could be to recommend people to resync if they're still on v0.13, since it's impractically slow anyway even without the memory problem.
274 2018-09-27T19:10:37  <jonasschnelli> We just need to make sure to communicate the standardness change in 0.17.0
275 2018-09-27T19:10:55  <provoostenator> Or they can install 0.15 first, wait for that process to finish and then install 0.17
276 2018-09-27T19:11:05  <jonasschnelli> meh
277 2018-09-27T19:11:10  <sipa> hi, i'm sortof here - ping me if needed
278 2018-09-27T19:11:17  <wumpus> provoostenator: agree; though putting a message in the code itself if people are upgrading from 0.13.0 would make sense, for those not carefully reading the release notes, but anyhow
279 2018-09-27T19:11:57  <gmaxwell> I think even though 0.16 appears to have added the replay bloat, 0.17 makes bloat worse because it adds an additional place where they're queued.  (this doesn't change that I think notices are probably the best move for now)
280 2018-09-27T19:12:00  <wumpus> but yes for the most common case (0.13.0 upgrade rollback), adding a message to the release notes would be acceptable solution
281 2018-09-27T19:12:02  <luke-jr> couldn't we detect the reorg needed, and just prompt for user action instead of trying to upgrade it?
282 2018-09-27T19:12:12  <provoostenator> If it can be done safely, having the upgrade simply refuse and throw an error message seems safer than just a release note.
283 2018-09-27T19:12:24  <sipa> gmaxwell: i'm not sure anything was added in 0.17
284 2018-09-27T19:12:40  <sipa> i blamed the txindex change, but the asynchronous processing was added earlier
285 2018-09-27T19:12:48  <wumpus> so I guess there isn't really a hurry to release 0.17.0 at this point
286 2018-09-27T19:12:51  <gmaxwell> sipa: txindex also schedulers queues block connections and disconnection, no?
287 2018-09-27T19:13:28  <gmaxwell> regardless... We don't yet have a proper fix for the issue, and I don't think we're still learning much about 0.17rc.
288 2018-09-27T19:13:52  <sipa> gmaxwell: i think (sorry no time to look now) is that 0.17 just added the txindex as a listener for those blockconnected events; the issue is not that, but the events in the first place
289 2018-09-27T19:14:05  <gmaxwell> ah.
290 2018-09-27T19:14:35  <sipa> my suggestion is to just point out in release notes that upgrading from 0.13.0 or before is a known memory bloat issue, which can be worked around with -reindex
291 2018-09-27T19:14:42  <gmaxwell> I didn't walk through the patches so it wasn't clear to me that the events existed before. Got it.
292 2018-09-27T19:14:45  <luke-jr> (ActivateBestChain actually checks for the queue length and blocks on it getting caught up)
293 2018-09-27T19:14:59  <sipa> luke-jr: it does; but RewindBlock and InvalidateBlock don't
294 2018-09-27T19:15:13  <luke-jr> sipa: would it be so bad if they did?
295 2018-09-27T19:15:31  <sipa> luke-jr: they need to release cs_main for that, which would be a major refactor for those functions
296 2018-09-27T19:15:33  <gmaxwell> luke-jr: that can be tricky to do safely as car has to be taken to make sure they don't wait while holding any locks.
297 2018-09-27T19:15:39  <gmaxwell> care*
298 2018-09-27T19:15:42  <luke-jr> hmm
299 2018-09-27T19:15:57  <sipa> but we can probably remove the upgrading logic from pre-segwit blocks anyway - as provoostenator says, it's already pretty unwieldy even without the memory bloat issue
300 2018-09-27T19:16:12  <gmaxwell> yea, reindex might already actually be faster.
301 2018-09-27T19:16:19  <sipa> i'm pretty sure it is
302 2018-09-27T19:16:25  <gmaxwell> but I assumed we'd want to use the rewind for future softforks.
303 2018-09-27T19:16:30  <provoostenator> Does reindex just toss out block files it doesn't need?
304 2018-09-27T19:16:37  <sipa> i don't care so much that invalidateblock takes a lot of memory - it would be a nice to have if we could actually make it revert to genesis, but it's not a priority
305 2018-09-27T19:16:42  <sipa> provoostenator: yes
306 2018-09-27T19:16:58  <gmaxwell> sipa: uh it's a little worse than that.
307 2018-09-27T19:17:25  <gmaxwell> I mean it hits 64+GB usage rewinding only a couple months of blocks.
308 2018-09-27T19:17:45  <sipa> yeah, ok
309 2018-09-27T19:17:46  <provoostenator> And it doesn't stop once it's going.
310 2018-09-27T19:18:03  <gmaxwell> indeed, and it doesn't return the memory, ever.
311 2018-09-27T19:18:11  <sipa> we'll need to refactor InvalidateBlock to deal with that; doesn't sound impossible, but probably for 0.17.1?
312 2018-09-27T19:18:30  <gmaxwell> Not a reason to hold 0.17, but it's not an irrelevant inefficiency.
313 2018-09-27T19:18:38  <sipa> fair
314 2018-09-27T19:18:42  <gmaxwell> sipa: sounds fine to me.
315 2018-09-27T19:18:56  <luke-jr> <2% of the network (<2000 nodes) run non-segwit software at this point; throwing an error that we can't upgrade them anymore seems reasonable
316 2018-09-27T19:19:27  <wumpus> yes
317 2018-09-27T19:19:50  <sipa> luke-jr: that's a useful statistics
318 2018-09-27T19:19:52  <gmaxwell> I still think we shouldn't just can the rewinding code though.
319 2018-09-27T19:19:55  <provoostenator> Maybe also throw an error if the user tries to invalidate more than 10K blocks? They can always do it in increments.
320 2018-09-27T19:20:06  <gmaxwell> ugh.
321 2018-09-27T19:20:33  <gmaxwell> just release note it, then we'll fix it later, please don't add additional falure cases to the function.
322 2018-09-27T19:20:40  <wumpus> agree with gmaxwell
323 2018-09-27T19:20:45  <wumpus> please don't overdesign temporary code
324 2018-09-27T19:20:58  <sipa> the refactor will effectively just do that - split it up in batches of X blocks to disconnect at once
325 2018-09-27T19:21:00  *** michaelsdunn1 has quit IRC
326 2018-09-27T19:21:12  <wumpus> this needs to be fixed properly, in master, and in the future we should be careful to review anything that adds a queue or global data structure for this possiblity
327 2018-09-27T19:21:20  <sipa> agree
328 2018-09-27T19:21:22  <wumpus> but don't spend too much time on the workaround
329 2018-09-27T19:21:49  <provoostenator> I guess anyone upgrading all the way from 0.13  will probably pay more than average attention to the Upgrade Notice section in bold at the top...
330 2018-09-27T19:22:32  <wumpus> I think most people still running 0.13.x do so intentionally, and won't be upgrading to 0.17.x any time soon
331 2018-09-27T19:22:43  <wumpus> (not those nodes, at least!)
332 2018-09-27T19:22:44  *** michaelsdunn1 has joined #bitcoin-core-dev
333 2018-09-27T19:22:56  <luke-jr> most 0.13 nodes are 0.13.2 anyway
334 2018-09-27T19:23:04  *** elichai2 has quit IRC
335 2018-09-27T19:23:07  <sipa> yes; 0.13 is not the issue; 0.13.0 is
336 2018-09-27T19:23:12  <luke-jr> sipa: https://luke.dashjr.org/programs/bitcoin/files/charts/services.html fwiw
337 2018-09-27T19:23:23  <wumpus> e.g. some users want to run old node software to cross-validate
338 2018-09-27T19:25:01  <wumpus> so: for 0.17.0, we should add a mention to the release notes for users upgrading from 0.13.0. This would require no new rc, so the way to final can continue as normal.
339 2018-09-27T19:25:42  <achow101> there's a pr for backports, will that be fore 0.17.1, or are those going in for .0?
340 2018-09-27T19:26:04  <wumpus> I haven't seen it, but unless they solve critical problems, they are not going in .0
341 2018-09-27T19:26:18  <achow101> #14328
342 2018-09-27T19:26:20  <gribble> https://github.com/bitcoin/bitcoin/issues/14328 | [0.17] Backports by MarcoFalke · Pull Request #14328 · bitcoin/bitcoin · GitHub
343 2018-09-27T19:27:35  <wumpus> I don't think any of those are very serious
344 2018-09-27T19:27:59  <wumpus> potential unititialized value sounds dangerous, but looking at the PR, it's impossible to trigger in practice
345 2018-09-27T19:28:08  <wumpus> I hate that kind of PR naming
346 2018-09-27T19:28:39  <provoostenator> PR bait? :-)
347 2018-09-27T19:28:42  <gmaxwell> I've complained about that a number of times in the past.
348 2018-09-27T19:28:57  <wumpus> me too, so many times, the guy doesn't listen to me
349 2018-09-27T19:29:02  <achow101> so does that mean 0.17.0 final after the meeting?
350 2018-09-27T19:29:11  <wumpus> (or gal, I don't really know)
351 2018-09-27T19:29:19  <provoostenator> Works for me.
352 2018-09-27T19:29:40  <wumpus> I guess so? would be nice to have the release notes change in
353 2018-09-27T19:29:47  <wumpus> before tagging
354 2018-09-27T19:31:05  <gmaxwell> just needs a one liner, no?  "If upgrading from 0.13 or prior, you must start with -reindex"
355 2018-09-27T19:31:33  <sipa> yah
356 2018-09-27T19:31:58  <wumpus> I've noted it here https://github.com/bitcoin/bitcoin/issues/12391#issuecomment-425211080
357 2018-09-27T19:32:01  <gmaxwell> k
358 2018-09-27T19:32:08  <achow101> we also need to add a known issues section
359 2018-09-27T19:33:32  <wumpus> yepp
360 2018-09-27T19:33:35  <wumpus> any other topics?
361 2018-09-27T19:33:56  <phantomcircuit> can anybody reproduce the travis error on #14336
362 2018-09-27T19:33:58  <gribble> https://github.com/bitcoin/bitcoin/issues/14336 | net: implement poll by pstratem · Pull Request #14336 · bitcoin/bitcoin · GitHub
363 2018-09-27T19:34:02  <phantomcircuit> i cannot reproduce it locally
364 2018-09-27T19:34:25  <wumpus> #topic Travis error on poll() PR
365 2018-09-27T19:34:32  *** jarthur has quit IRC
366 2018-09-27T19:34:49  <jonasschnelli> IMO after the meeting
367 2018-09-27T19:34:58  <wumpus> I guess this is a action item, that people shoudl try the PR locally?
368 2018-09-27T19:34:58  <gmaxwell> instagibbs had a related question, where are the bitcoind exit codes coming from.  phantomcircuit's travis failure bitcoind has a return value of -6
369 2018-09-27T19:35:12  <wumpus> after the meeting, yes, doesn't make sense to do a real-time debugging session I think :)
370 2018-09-27T19:35:32  <instagibbs> I shall wait
371 2018-09-27T19:35:38  <jonasschnelli> would be fun... but yes. Better later.
372 2018-09-27T19:35:42  <promag> wumpus: topic suggestion: multiwallet
373 2018-09-27T19:35:43  <wumpus> #action try to run tests on #14336 on different environments to see if it reproduces travis error
374 2018-09-27T19:35:45  <jonasschnelli> High-Prio list?
375 2018-09-27T19:35:45  <gribble> https://github.com/bitcoin/bitcoin/issues/14336 | net: implement poll by pstratem · Pull Request #14336 · bitcoin/bitcoin · GitHub
376 2018-09-27T19:36:02  <wumpus> #topic multiwallet (promag)
377 2018-09-27T19:36:12  <promag> cc jnewbery
378 2018-09-27T19:36:25  <promag> just want some feedback regarding https://github.com/bitcoin/bitcoin/pull/13100#issuecomment-424342813
379 2018-09-27T19:36:34  <promag> also, regarding listwalletdir
380 2018-09-27T19:36:49  <wumpus> jonasschnelli: I haven't paid attention to the high-prio list at all this week, with the CVE issue and Riga travel so I'm not sure there's much to do there, but sure we can discuss it
381 2018-09-27T19:37:12  <jonasschnelli> I think Concept ack on both (promag)! Will test more soon.
382 2018-09-27T19:37:20  <promag> and I'm going to submit a refactor PR introducing WalletsModel
383 2018-09-27T19:37:37  <promag> that combines loaded wallets and existing wallets
384 2018-09-27T19:37:53  <jonasschnelli> wumpus: Yeah. I only wanted to ask for a review on #14046 from gmaxwell and sipa since they both commented already on it (fixed stuff)
385 2018-09-27T19:37:55  <gribble> https://github.com/bitcoin/bitcoin/issues/14046 | net: Refactor message parsing (CNetMessage), adds flexibility by jonasschnelli · Pull Request #14046 · bitcoin/bitcoin · GitHub
386 2018-09-27T19:38:01  <promag> otherwise #13100 looks like junk code..
387 2018-09-27T19:38:03  <gribble> https://github.com/bitcoin/bitcoin/issues/13100 | gui: Add dynamic wallets support by promag · Pull Request #13100 · bitcoin/bitcoin · GitHub
388 2018-09-27T19:39:53  <promag> any objections to WalletsModel? IIRC it was previously suggested
389 2018-09-27T19:40:56  <gmaxwell> I'll look at 14046 again, thanks for the ping.
390 2018-09-27T19:42:54  <jonasschnelli> topic proposal: factor out berkeley-db
391 2018-09-27T19:43:06  <wumpus> #topic factor out berekey-db
392 2018-09-27T19:43:11  <wumpus> (jonasschnelli)
393 2018-09-27T19:43:12  *** Krellan has joined #bitcoin-core-dev
394 2018-09-27T19:43:15  <jonasschnelli> I tried this serval times but seems more complex then initially though..
395 2018-09-27T19:43:32  <jonasschnelli> But I think we should slowly consider alternative database systems (internal) for wallet files
396 2018-09-27T19:43:46  <jonasschnelli> And factroing out BDB should be done sooner then later
397 2018-09-27T19:43:58  <jamesob> where did the complexity come from?
398 2018-09-27T19:44:06  <provoostenator> Could be combined with  jnewbery's standalone wallet tool, which can hold  on to the direct dependnecy a bit longer.
399 2018-09-27T19:44:20  <jonasschnelli> jamesob: I think mostly due to the complex code layering...
400 2018-09-27T19:44:47  <jonasschnelli> I think switching the database backend on runtime should be possible....
401 2018-09-27T19:45:08  <promag> switching the database backend on runtime should be possible <- why?
402 2018-09-27T19:45:11  <jonasschnelli> I hope someone working on the wallet can initiate that refactor: jamesob, jnewbery, ryanofsky
403 2018-09-27T19:45:23  <provoostenator> I assume it makes most sense to move it to leveldb since we're using that for most other things?
404 2018-09-27T19:45:33  <sipa> leveldb is very annoying
405 2018-09-27T19:45:34  <gmaxwell> what gah no
406 2018-09-27T19:45:42  <jamesob> I don't think so; leveldb can't use a single .dat-ish file
407 2018-09-27T19:45:43  <jonasschnelli> promag: we must assume there will be a pretty long "transition" phase from the old to the new database layer
408 2018-09-27T19:45:45  <jonasschnelli> Not leveldb...
409 2018-09-27T19:45:48  <sipa> it needs whole directories, and provides way too many features
410 2018-09-27T19:45:50  <gmaxwell> and not a particularly good fit for what its used for here.
411 2018-09-27T19:45:56  <jonasschnelli> sipa one wrote a simple append only database...
412 2018-09-27T19:46:00  <jamesob> sqlite IMO seems like a good bet
413 2018-09-27T19:46:13  <gmaxwell> jonasschnelli: do we need to assume there is a long transistion instead of a standalone conversion util?
414 2018-09-27T19:46:16  <wumpus> yeah, though FWIW for the berkekeydb we also suggest a whole directory per wallet at the moment
415 2018-09-27T19:46:21  <jonasschnelli> (logdb),... there is a 2-4 year old PR somewhere (search after logdb)
416 2018-09-27T19:46:25  <sipa> sqlite is fine, though we also don't need any of its features, apart from safe updating
417 2018-09-27T19:46:29  <gmaxwell> wumpus: yes, but we know we don't like to do that. :)
418 2018-09-27T19:46:35  <jamesob> or honestly just a raw format of our own creation
419 2018-09-27T19:46:47  <wumpus> but the dangerous thing here is that anything you choose for the wallet will need to be supported more or less forever, so it's not an easy choice
420 2018-09-27T19:46:49  <jonasschnelli> gmaxwell: could also be a conversion utility,.. but at least the utility must be capable to run both database systems,... so won't change that much for the refactroing)
421 2018-09-27T19:46:52  <provoostenator> If we add another dependency, maybe pick one that's potentially useful for block and index storage?
422 2018-09-27T19:46:54  <gmaxwell> esp since we just end up loading the whole thing into memory, a fancy database is kinda overkill.
423 2018-09-27T19:47:13  <jonasschnelli> gmaxwell: agree
424 2018-09-27T19:47:24  <sipa> provoostenator: those things actually need efficient querying, atomic batch updates, ...
425 2018-09-27T19:47:27  <wumpus> we don't *need* to load the whole thing in memroy, that's a current design choice
426 2018-09-27T19:47:36  <sipa> provoostenator: for the wallet we just need a key-value store with some append only records :)
427 2018-09-27T19:47:37  <gmaxwell> (and also makes the files much more fragile and harder to work with than they would be otherwise)
428 2018-09-27T19:47:38  <cfields> sqlite is also nice in that they provide a monolithic source file and encourage direct integration.
429 2018-09-27T19:47:47  <wumpus> if there's proper indexing, loading every single transaction into memory could be avoided
430 2018-09-27T19:47:50  <sipa> yeah, i'm not opposed to sqlite
431 2018-09-27T19:47:54  <jonasschnelli> logdb (#5686) would be a simple append only hard to corrupt "database"... everything is hold in memory
432 2018-09-27T19:47:57  <gribble> https://github.com/bitcoin/bitcoin/issues/5686 | [Wallet] replace BDB with internal append only (logdb) backend by jonasschnelli · Pull Request #5686 · bitcoin/bitcoin · GitHub
433 2018-09-27T19:47:57  <sipa> it has very thorough tests
434 2018-09-27T19:48:00  <luke-jr> cfields: that's not a good thing. -.-
435 2018-09-27T19:48:00  <provoostenator> Someone once complained that the wallet didn't have atomicity guarantees.
436 2018-09-27T19:48:02  <gmaxwell> wumpus: indeed. but that decision should be made jointly.
437 2018-09-27T19:48:03  <jonasschnelli> Or sqlite... yeah
438 2018-09-27T19:48:11  <wumpus> I like sqlite, especially with deterministic wallets it wouldn't need to store all the keys
439 2018-09-27T19:48:20  <jamesob> sqlite seems like a pretty safe bet
440 2018-09-27T19:48:22  <cfields> luke-jr: the alternative is the reason we're switching away...
441 2018-09-27T19:48:24  <wumpus> so it's pretty much a metadata database, and sqlite is great for metadata and querying metadata
442 2018-09-27T19:48:32  <sipa> wumpus: with descriptor based wallets we don't need that anyway :)
443 2018-09-27T19:48:35  <luke-jr> cfields: what?
444 2018-09-27T19:48:41  <wumpus> sipa: even beter
445 2018-09-27T19:48:42  <gmaxwell> I don't think sqlite makes much sense unless the intent is also to move away from pulling everything into memory.
446 2018-09-27T19:48:50  <jonasschnelli> Is it guaranteed that sqlite databases are interoperatable between platforms and versions of sqlite?
447 2018-09-27T19:48:59  <jamesob> gmaxwell: what would you propose in lieu?
448 2018-09-27T19:49:01  <wumpus> please move away from loading everyting into memory in the long run
449 2018-09-27T19:49:02  <gmaxwell> And if we're going to do that, the scheme in the database matters a lot, so that change should probably be made at the same time.
450 2018-09-27T19:49:11  <wumpus> on the short term it's not a priority
451 2018-09-27T19:49:14  <wumpus> but don't make it impossible
452 2018-09-27T19:49:27  <wumpus> (in a new format)
453 2018-09-27T19:49:35  <gmaxwell> jamesob: if we're loading the whole thing into memory, a simple serialized format like logdb is I think vastly superior.
454 2018-09-27T19:49:37  <jonasschnelli> I agree with gmaxwell: sqlite makes most sense if we want to one active handling of merchant size wallets
455 2018-09-27T19:49:46  <jonasschnelli> and not sure if we want that
456 2018-09-27T19:49:56  <promag> does it have to be an embedded database?
457 2018-09-27T19:49:58  <wumpus> some large users of the wallet run into memory issues, and have to remake a new wallet perioidically because of this limitation
458 2018-09-27T19:50:06  <sipa> promag: no
459 2018-09-27T19:50:19  <gmaxwell> if we just use sqllite but then just treat it like a blob holder, then the whole schema will need to change to avoid memory loading it in any case.
460 2018-09-27T19:50:20  <phantomcircuit> jonasschnelli, think it makes most sense to have a tool which is a separate binary to convert from bdb to "new" wallet format
461 2018-09-27T19:50:21  <jonasschnelli> I don't think it has to be a "database" at all
462 2018-09-27T19:50:21  <wumpus> (due to storing all the transactions in memory, and also the time overhead of loading the whopping thing at startup)
463 2018-09-27T19:50:22  <instagibbs> wumpus, or abusing rpc calls to whiddle it down
464 2018-09-27T19:50:27  <phantomcircuit> and for the new wallet format to simply be a flat file
465 2018-09-27T19:50:30  <wumpus> instagibbs: oh :-)
466 2018-09-27T19:50:38  <jonasschnelli> phantomcircuit: I agree. But that tools would require the refactoring also
467 2018-09-27T19:50:40  <cfields> luke-jr: eh, not worth getting into it and muddying the conversation
468 2018-09-27T19:50:43  <wumpus> instagibbs: I mean, :-(
469 2018-09-27T19:50:47  <gmaxwell> wumpus: More than memory issues, they run into problems that many of our rpc operations iterate over all txn in the wallet and then become super slow.
470 2018-09-27T19:50:55  <instagibbs> gmaxwell, yeah that one
471 2018-09-27T19:51:07  <phantomcircuit> jonasschnelli, yes but has the advantage that you can write the conversion tool and then just rip out a ton of the walletdb logic entirely
472 2018-09-27T19:51:08  <wumpus> gmaxwell: right - another lmitation of not having indexing, either in memory or on disk
473 2018-09-27T19:51:08  <instagibbs> i know people who delete completely spent tx(plus 100 confs or something) to speed it wallets
474 2018-09-27T19:51:21  <phantomcircuit> which makes refactoring much easier, cause you dont have to support both simultaneously
475 2018-09-27T19:51:31  <jonasschnelli> phantomcircuit: the logic must still be available somewhere,... could be in a tool source only. yeah
476 2018-09-27T19:51:49  <phantomcircuit> cfields, sqlite doesn't actually provide a monolithic file in the same way bdb doesn't
477 2018-09-27T19:51:50  <wumpus> instagibbs: ah yes, the "wallet only needs a view of current utxos, not all of history" view
478 2018-09-27T19:52:05  <gmaxwell> "pruned wallet"
479 2018-09-27T19:52:06  <instagibbs> wumpus, either that or listunspent takes forever :(
480 2018-09-27T19:52:07  <cfields> phantomcircuit: eh? They for sure used to.
481 2018-09-27T19:52:08  <phantomcircuit> to operate in the fast safe mode it needs a separate write ahead log file
482 2018-09-27T19:52:09  <wumpus> gmaxwell: right
483 2018-09-27T19:52:15  <luke-jr> well, at least we don't need to keep the history in memory
484 2018-09-27T19:52:23  <wumpus> luke-jr: indeed
485 2018-09-27T19:52:24  <cfields> phantomcircuit: oh, I was talking about source file, not the database format.
486 2018-09-27T19:52:24  <phantomcircuit> cfields, you cant have a single file but it's amazingly slow
487 2018-09-27T19:52:28  <phantomcircuit> oh
488 2018-09-27T19:52:34  <phantomcircuit> yes it does have that but like
489 2018-09-27T19:52:35  <phantomcircuit> meh
490 2018-09-27T19:52:40  <wumpus> that's where something like sqlite would be, more or less, useful, I like how clightning uses it
491 2018-09-27T19:52:50  <gmaxwell> going back to the prior point. ... if the history isn't in memory, then the database storing the wallet needs to be structured in a way that suports that
492 2018-09-27T19:53:15  <cfields> phantomcircuit: that makes integration into our build a no-brainer. That's a signifacant feature imo.
493 2018-09-27T19:53:49  <phantomcircuit> gmaxwell, and needs to be quite fast actually
494 2018-09-27T19:54:05  <wumpus> integrating sqlite into a project is trivial, indeed can be done as a single .cpp file if that's desirable
495 2018-09-27T19:54:16  <jamesob> if we're thinking longterm (esp. about not loading everything into memory simultaneously), I think it makes sense to come up with a normalized, relational schema for the wallet and use sqlite. shouldn't be hard to come up with something non-controversial (famous last words)
496 2018-09-27T19:54:33  <promag> any reason to not consider postgres for instance?
497 2018-09-27T19:54:38  <wumpus> AHHHH
498 2018-09-27T19:54:38  <sipa> gmaxwell: i don't think the choice of container format and the choice of database layout need to be made at the same time
499 2018-09-27T19:54:41  <sipa> promag: god why
500 2018-09-27T19:54:42  <jamesob> wat
501 2018-09-27T19:54:43  <luke-jr> promag: uh, lots?
502 2018-09-27T19:55:04  <jonasschnelli> Oracle?
503 2018-09-27T19:55:09  <cfields> haha
504 2018-09-27T19:55:11  <wumpus> jonasschnelli: :-) <3
505 2018-09-27T19:55:19  <sipa> Oracle BDB?
506 2018-09-27T19:55:22  <promag> luke-jr: name one
507 2018-09-27T19:55:28  <jonasschnelli> I think however we proceed (sqlite, logdb, etc.), factoring out BDB in a nice layered way will be require (even helps if we keep BDB forever)
508 2018-09-27T19:55:32  <luke-jr> I mean, if we're using sqlite, the queries could be compatible with multiple backends, but expecting regular users to set up Postgres is crazy..
509 2018-09-27T19:55:35  <jonasschnelli> I hope someone picks that up
510 2018-09-27T19:55:35  <sipa> promag: let's do that outside this meeting
511 2018-09-27T19:55:39  <cfields> this might work better in terms of concrete proposals rather than rounds of "how about xyz?"
512 2018-09-27T19:55:45  <jonasschnelli> Also BDB is a compile pitfall
513 2018-09-27T19:55:48  <promag> sure
514 2018-09-27T19:56:03  <wumpus> cfields: good point
515 2018-09-27T19:56:12  <wumpus> 'what about mongodb?' :')
516 2018-09-27T19:56:28  <wumpus> any other topics? 4 minutes left
517 2018-09-27T19:56:29  <cfields> haha
518 2018-09-27T19:57:04  <provoostenator> We should just store it on a blockchain.
519 2018-09-27T19:57:21  <luke-jr> provoostenator: it would be nice if it was possible to commit to it in such a way
520 2018-09-27T19:57:40  <luke-jr> eg, if you could get a historical hash of the wallet state for commitments
521 2018-09-27T19:57:45  <wumpus> luke-jr: right, optional support for a large-scale DBM like postgres would be useful for really big users, but that's really a long term goal I suppose, if at all
522 2018-09-27T19:57:51  <gmaxwell> I'd prefer it if just ban anyone that ever directly uses the name of any database system from the channel.
523 2018-09-27T19:58:09  <jonasschnelli> but leveldb!
524 2018-09-27T19:58:11  <wumpus> except oracle, of course *ducks*
525 2018-09-27T19:58:17  <gmaxwell> sipa: "container" is basically my point, if we're just using it as a "container" a simple log would be a lot better.
526 2018-09-27T19:58:19  <jonasschnelli> heh
527 2018-09-27T19:58:52  <wumpus> #endmeering
528 2018-09-27T19:58:55  <wumpus> #endmeeting
529 2018-09-27T19:58:55  <lightningbot> Meeting ended Thu Sep 27 19:58:55 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
530 2018-09-27T19:58:55  <lightningbot> Minutes:        http://www.erisian.com.au/meetbot/bitcoin-core-dev/2018/bitcoin-core-dev.2018-09-27-19.00.html
531 2018-09-27T19:58:55  <lightningbot> Minutes (text): http://www.erisian.com.au/meetbot/bitcoin-core-dev/2018/bitcoin-core-dev.2018-09-27-19.00.txt
532 2018-09-27T19:58:55  <lightningbot> Log:            http://www.erisian.com.au/meetbot/bitcoin-core-dev/2018/bitcoin-core-dev.2018-09-27-19.00.log.html
533 2018-09-27T19:58:57  <sipa> gmaxwell: yes, but sqlite is essentially two layers; a well-tested safe updatable blob storing mechanism, and then a database layer on top
534 2018-09-27T19:58:57  <gmaxwell> sipa: if you just dump transactions/keys into a container and then have to load them into memory and index them thats fine.. if you want to avoid having to do that, you can't just use the storage as a container, it needs to be used as a database.
535 2018-09-27T19:59:12  <jonasschnelli> The append-only easyness got a bit lost since we added abandontransaction
536 2018-09-27T19:59:25  <wumpus> jonasschnelli: yes
537 2018-09-27T19:59:26  <jonasschnelli> Its still possible but comes at high disk-mem cost
538 2018-09-27T19:59:27  <gmaxwell> jonasschnelli: why?
539 2018-09-27T19:59:27  <promag> gmaxwell: how about aws s3? o/
540 2018-09-27T19:59:36  <gmaxwell> jonasschnelli: abandon transaction just logs the txn as abandoned.
541 2018-09-27T19:59:40  <luke-jr> ^
542 2018-09-27T19:59:41  <sipa> gmaxwell: by switching to sqlite as a container and only using the low level format, you get pretty much what we need (apart from the extra code for the database stuff), but you also get the ability to later switch the db schema
543 2018-09-27T19:59:52  <gmaxwell> So we could start using sqllite as a container, but then later actually have a schema, thats a thing that could be done, but really its just two incompatible wallet formats.
544 2018-09-27T20:00:00  <jamesob> ripping out bdb in favor of sqlite sounds fun... I might take a crack if no one gets around to it in the next few months. sounds like a good christmas break project :)
545 2018-09-27T20:00:02  <promag> gmaxwell: and support log compaction?
546 2018-09-27T20:00:09  <jonasschnelli> gmaxwell: yeah.. but if you abandon a 10k bytes transaction you have 10k unused bytes (where only 32bytes change)
547 2018-09-27T20:00:18  <gmaxwell> promag: why would that be needed?
548 2018-09-27T20:00:20  <jonasschnelli> log compactation is of course possible
549 2018-09-27T20:00:39  <wumpus> jamesob: it's not too hard to replace the database, I replaced berkeleydb with leveldb once in a day or so
550 2018-09-27T20:00:50  <gmaxwell> jonasschnelli: yes so? no one cares about 10k extra data in their wallet, if they did any of these database based solutions would be unacceptable from the getgo as they're all fairly space inefficent.
551 2018-09-27T20:00:53  <phantomcircuit> jonasschnelli, nobody is using abandon transaction because of 10kb of disk space being used lol
552 2018-09-27T20:00:55  <wumpus> jamesob: (that's in a very naive way, though, proper schemas will be more work)
553 2018-09-27T20:00:56  <sipa> i don't see how abandomtransaction changed anything more than a transaction confirming does
554 2018-09-27T20:01:02  <jamesob> wumpus: what's the tricky part? providing a migration path?
555 2018-09-27T20:01:05  <sipa> it's just an update "this tx is now abandoned"
556 2018-09-27T20:01:07  <instagibbs> so, what does exit status "1" mean in the functional test suite, when shutting down a node?
557 2018-09-27T20:01:10  <jonasschnelli> wumpus, jamesob: I think the hard part of the refactoring is to make two database work (which is ultimatively required for a conversion tool)
558 2018-09-27T20:01:17  <promag> why not? for systems that use bitcoin wallet in a couple of months it can be pretty large
559 2018-09-27T20:01:23  <sipa> jonasschnelli: i think that's the easiest part
560 2018-09-27T20:01:24  <phantomcircuit> what does -6 mean?
561 2018-09-27T20:01:30  <wumpus> jamesob: the "don't load everything into memory and use proper indexes" part, and yes ,migration is also an issue
562 2018-09-27T20:01:33  <sipa> jonasschnelli: agreeing on a new database schema will be impossible, however :)
563 2018-09-27T20:01:34  <achow101> instagibbs: 1 is EXIT_FAILURE
564 2018-09-27T20:01:48  <achow101> phantomcircuit: where do you get -6?
565 2018-09-27T20:01:49  <luke-jr> append a "drop the tx" command, and punch a hole where the original data was
566 2018-09-27T20:01:49  <jonasschnelli> gmaxwell: Yes. I just meant the easyness (or the effectiviness) got a bit lost with state changes on transactions like abandontx
567 2018-09-27T20:01:56  <phantomcircuit> jonasschnelli, that is for sure the principle issue, supporting two formats at the same time is difficult
568 2018-09-27T20:02:01  <sipa> jonasschnelli: i really don't see where that comes from
569 2018-09-27T20:02:02  <luke-jr> then it can be saved append-only in backups, and also frees the disk space (potentially)
570 2018-09-27T20:02:07  <sipa> jonasschnelli: a tx confirming is equally a state change
571 2018-09-27T20:02:20  <instagibbs> achow101, ah ok, i shall hunt for this error
572 2018-09-27T20:02:30  <gmaxwell> sipa: in any case, my point is that just swapping the backing store doesn't seem to produce any real benefit, but it creates incompatiblity. I might be missing it, but it sounds like replacing a black box with a more trendy black box.  Actually making use of it to to keep stuff out of memory would potentially be a big gain, but that isn't the discussion, and if it were done would create
573 2018-09-27T20:02:30  <gmaxwell> effectively another new wallet format.
574 2018-09-27T20:02:32  <phantomcircuit> achow101, https://travis-ci.org/bitcoin/bitcoin/jobs/433891452
575 2018-09-27T20:02:33  <phantomcircuit> test_framework.test_node.FailedToStartError: [node 0] bitcoind exited with status -6 during initialization
576 2018-09-27T20:02:35  <jamesob> wumpus: as you say earlier, though, we can gradually build in better use of the database; I think it'd be nice just to shed the bdb dependency asap
577 2018-09-27T20:02:36  <wumpus> luke-jr: fallocate(FALLOC_FL_PUNCH_HOLE, ...)?
578 2018-09-27T20:02:50  <phantomcircuit> have literally no idea what would cause that
579 2018-09-27T20:03:03  <sipa> gmaxwell: well the low-level part of sqlite looks like a well-tested implementation of what we'd want actually
580 2018-09-27T20:03:08  <gmaxwell> jonasschnelli: I'm still not seeing it, we always made changes to tx.. adding labels, or as sipa notes, confirming them.. spending their outputs, etc.
581 2018-09-27T20:03:09  <sipa> gmaxwell: of course, needs investigation
582 2018-09-27T20:03:25  <luke-jr> wumpus: I don't have the exact APIs memorised, but probably something like that
583 2018-09-27T20:03:43  <luke-jr> wumpus: so long as the loader ignores null bytes, it should be fairly trivial
584 2018-09-27T20:04:11  <wumpus> luke-jr: I remembered because it sounds so senselessly aggressive, it replaces a page in a file with a hole (all-zeros) page
585 2018-09-27T20:04:33  <luke-jr> and frees the space used by it
586 2018-09-27T20:04:34  <jonasschnelli> gmaxwell: I think confirmation is stateless from the wallet perspective (only in conjunction with the blockchain/header state)... but yes. Its non crucial.
587 2018-09-27T20:04:38  <luke-jr> aggressive in what sense?
588 2018-09-27T20:05:25  <gmaxwell> does windows do sparse files?
589 2018-09-27T20:05:34  <wumpus> luke-jr: I mean, as in violent
590 2018-09-27T20:05:48  <luke-jr> gmaxwell: yes, but not sure if it has an easy punch-hole
591 2018-09-27T20:05:50  <sipa> jonasschnelli: ... no, we add the block hash that contains the tx to the CWalletTx record
592 2018-09-27T20:06:04  <wumpus> gmaxwell: modern versions of windows do
593 2018-09-27T20:06:12  <achow101> phantomcircuit: I think maybe that's a SIGABRT
594 2018-09-27T20:06:30  <jonasschnelli> uh. yes. Right. There is then at least one update from mempool to in-block,... right (and eventually some reorg writes)
595 2018-09-27T20:06:51  <luke-jr> we don't support versions that don't I think :p
596 2018-09-27T20:08:19  <wumpus> so does MacOS support sparse files?
597 2018-09-27T20:08:23  <gmaxwell> achow101: how would sigabrt show up that way? signals are normally very high exit codes
598 2018-09-27T20:08:32  *** ott0disk has joined #bitcoin-core-dev
599 2018-09-27T20:08:33  <sipa> promag: short answer to why not postgres; we don't need a multi-user networked database, and we don't need sql
600 2018-09-27T20:08:45  <sipa> we need a file on disk
601 2018-09-27T20:08:55  <sipa> that's safe to interact with and move around
602 2018-09-27T20:09:17  <wumpus> requireing postgres would be really evil, though not as evil as requiring oracledb
603 2018-09-27T20:10:06  <wumpus> besides being unnecessary it has lots of extra setup steps to get started in the first place
604 2018-09-27T20:10:06  <achow101> gmaxwell: exit codes are signed, so anything above 128 would be negative, right?
605 2018-09-27T20:10:37  <gmaxwell> exit codes are a char?
606 2018-09-27T20:10:44  <sipa> i think so yes
607 2018-09-27T20:10:46  <wumpus> on most UNIX, yes
608 2018-09-27T20:10:55  <gmaxwell> ha, if I knew that it was long since lost!
609 2018-09-27T20:11:01  <provoostenator> There are of course some other cryptocurrency projects dealing with lots of I/O. Anything useful we can use e.g. from libbitcoin, or LMDB that Monero uses?
610 2018-09-27T20:11:09  <gmaxwell> okay well SIGABRT would mean he was hitting an assertion most likely.
611 2018-09-27T20:11:21  <wumpus> and negative values are generally only used when the process was killed with a signal
612 2018-09-27T20:11:22  <provoostenator> (not that the wallet is I/O heavy, so maybe that's not the thing to worry about)
613 2018-09-27T20:11:29  <gmaxwell> provoostenator: I have detected you mentioned a database. You are a bad person
614 2018-09-27T20:11:30  <jamesob> would it be an acceptable user experience for us to completely strip bdb out in some major release, provide an upgrade tool, and throw an error if users try to start bitcoin with bdb-format wallets?
615 2018-09-27T20:11:59  <gmaxwell> jamesob: I had assumed that we would eventually do exactly that.
616 2018-09-27T20:12:03  <wumpus> provoostenator: monero doesn't use LMDB for the wallet ,as far as I know, only for the block chain
617 2018-09-27T20:12:04  <sipa> provoostenator: we are not talking about a database. we're talking about the wallet
618 2018-09-27T20:12:09  <achow101> gmaxwell: according to http://tldp.org/LDP/abs/html/exitcodes.html, the exit codes are 128 + n where n is the signal number
619 2018-09-27T20:12:19  <wumpus> monero has some really simple format for the "simplewallet" IIRC
620 2018-09-27T20:12:27  <achow101> so I think that means -6 becomes just signal number 6, which is sigabrt
621 2018-09-27T20:12:38  <wumpus> achow101: yes
622 2018-09-27T20:12:55  <sipa> provoostenator: there are very different trade-offs relevant if we're talking about the UTXO set etc, but that's not the topic now
623 2018-09-27T20:12:58  <jamesob> gmaxwell: makes sense - didn't know if anyone was intent on having some kind of runtime migration
624 2018-09-27T20:13:05  <promag> sipa: I understand, just playing devil's advocate
625 2018-09-27T20:13:58  <provoostenator> I just think it's suboptimal to pull in a whole new dependency _only_ for the wallet.
626 2018-09-27T20:14:23  <provoostenator> So might as well think ahead a bit, unless we're sure we never want to move away from leveldb for everything else.
627 2018-09-27T20:14:42  <gmaxwell> have you all considered rewritting the software in javascript? it has webscale. it's very popular now, I hear nodecoin uses it.
628 2018-09-27T20:14:59  <gmaxwell> provoostenator: the wallets needs are almost but not entirely unlike the needs of the system state.
629 2018-09-27T20:15:16  <gmaxwell> It's basically an entirely different task, with very different requirements.
630 2018-09-27T20:15:25  <wumpus> yes
631 2018-09-27T20:15:58  <wumpus> and sqlite would be a trivial dependency at least
632 2018-09-27T20:16:53  <gmaxwell> (as an aside, people have also tested lmdb and sqllite for what we use leveldb fore. lmdb seemed to be about the same or slightly worse, depending on hardware in use... and the report was on sqllite that no one had enough patience to let it finish syncing after the Nth day.. :) )
633 2018-09-27T20:17:00  <jamesob> leveldb strikes me as totally fine for the utxo set and uprooting it seems very dangerous (not sure if I'm attacking a strawman here)
634 2018-09-27T20:17:15  <gmaxwell> s/fore/for/
635 2018-09-27T20:17:30  <provoostenator> "SQLite does not use Git" that's a great start...
636 2018-09-27T20:17:40  <provoostenator> (but nothing a checksum can't handle)
637 2018-09-27T20:17:46  <sipa> heh
638 2018-09-27T20:17:53  <sipa> what does that have to do with anything
639 2018-09-27T20:17:53  <luke-jr> SQLite isn't consensus-critical.
640 2018-09-27T20:18:19  <luke-jr> just pkgconfig sqlite3, done
641 2018-09-27T20:18:23  <wumpus> yes
642 2018-09-27T20:18:40  <gmaxwell> phantomcircuit was saying above that the safe mode of sqllite requires a seperate log file. :(
643 2018-09-27T20:18:47  <gmaxwell> if so, thats a bummer.
644 2018-09-27T20:18:49  <BlueMatt> wait, huh, what happened to append-only wallet?
645 2018-09-27T20:18:53  <wumpus> anyhow I don't think we're ever going to agree on this
646 2018-09-27T20:18:58  <luke-jr> gmaxwell: do we need the safe mode?
647 2018-09-27T20:19:00  <BlueMatt> seems like a really, really bad idea to move from a design like that to some standard db env
648 2018-09-27T20:19:02  <sipa> BlueMatt: great idea, just discussing alternaives :)
649 2018-09-27T20:19:14  <BlueMatt> heh, we havent had enough fun with bdb?
650 2018-09-27T20:19:25  <sipa> sqlite is nothing like bdb :)
651 2018-09-27T20:19:41  <wumpus> at least sqlite supports proper indexing and queries...
652 2018-09-27T20:19:49  <sipa> (but if what gmaxwell says is true, that makes it mch less attractive)
653 2018-09-27T20:19:50  <BlueMatt> true, but still
654 2018-09-27T20:19:58  <wumpus> which is great for finding transactions matching a certain criterion, for example
655 2018-09-27T20:20:15  <jamesob> wumpus: isn't the sqlite logfile just some optional performance tweak?
656 2018-09-27T20:20:22  <gmaxwell> wumpus: unless our way of using it is just throwing in a lot of binary blobs like we pretty much use bdb. :P
657 2018-09-27T20:20:30  <wumpus> with keyvalue databses all that indexing has to be implemented manually
658 2018-09-27T20:20:44  <wumpus> gmaxwell: as I said, I like how clightning uses it
659 2018-09-27T20:21:08  <wumpus> it's really sensible and well-designed
660 2018-09-27T20:21:49  <provoostenator> Problem solved then, a SQL lite binary blob inside bdb.
661 2018-09-27T20:21:59  <wumpus> somehow I really dislike how we ended up with a wallet embeded in our project and being unable to move away from it
662 2018-09-27T20:22:03  <gmaxwell> wumpus: to be clear, I'm +1 on using it with a schema, -1 on using it in a blobby way,  neutral on blobby in general, but if we were going to stay blobby a simple thing that just writes out a change log for in memory state would probably be better (more reliable, simpler, etc.).
663 2018-09-27T20:22:24  <wumpus> I wish it was just some external thing, but that's too much to wish for at this point
664 2018-09-27T20:22:41  <jamesob> wumpus: huge agree
665 2018-09-27T20:22:55  <wumpus> everyone is always going to disagree how to move on from here, it's like a really bad local maximum in evolution
666 2018-09-27T20:23:11  <gmaxwell> if it were, I think bitcoin would be nearly dead... there would be very little reason for people to bother even trying to start a node.
667 2018-09-27T20:23:30  <gmaxwell> Altcoins where node operations are totally distinct from using a wallet basically just don't haver users using nodes.
668 2018-09-27T20:23:59  <wumpus> the node could still be shipped with some wallet
669 2018-09-27T20:24:02  <luke-jr> on that point, we do need to make it easier to pair nodes with other wallets
670 2018-09-27T20:24:29  <luke-jr> the hard part is the port forwarding stuff :/
671 2018-09-27T20:24:43  <jamesob> luke-jr: in your opinion is that an issue of making the rpc interface more granular/complete or something else?
672 2018-09-27T20:24:48  <wumpus> or like monero simply ship with the most simple wallet possible, with no ambitions of anything more
673 2018-09-27T20:24:50  <gmaxwell> luke-jr: because there is so much development surplus that it would be good to spend it on facilitating duplication of efforts that aren't even being duplicated yet. :P
674 2018-09-27T20:25:07  <luke-jr> jamesob: RPC interface isn't really used much for other wallets
675 2018-09-27T20:25:38  <jamesob> anecdotally, if someone just wants to transact in bitcoin they often just go to electrum/trezor/bread/etc.
676 2018-09-27T20:25:40  <luke-jr> gmaxwell: people are using other wallets whether we facilitate it or not
677 2018-09-27T20:25:51  <jamesob> I think few people who _just_ want a wallet go to the bother of syncing bitcoind
678 2018-09-27T20:26:17  <gmaxwell> luke-jr: I don't see how that connects to your comment.
679 2018-09-27T20:26:23  <jamesob> though I am sympathetic to that reasoning... wish there was a way of quantifying it
680 2018-09-27T20:26:32  <provoostenator> jamesob: that's probably because they don't understand the privacy tradeoff
681 2018-09-27T20:26:39  <gmaxwell> luke-jr: how someing using a hosted web wallet doesn't really have anything to do with bitcoind.
682 2018-09-27T20:26:51  <luke-jr> gmaxwell: I mean electrum/trezor/bread/etc
683 2018-09-27T20:26:55  <wumpus> in any case, this comes back every time
684 2018-09-27T20:27:00  <jamesob> of course - but even a user like me doesn't actually use the core wallet because it doesn't, e.g., support hardware devices
685 2018-09-27T20:27:09  <wumpus> the same discussions since 2012
686 2018-09-27T20:27:11  <luke-jr> ideally, they could just scan a QR code from Core, and have it use their own node
687 2018-09-27T20:27:18  <luke-jr> the trouble is getting past firewalls
688 2018-09-27T20:27:21  <gmaxwell> luke-jr: and what does that hae to do with bitcoind?
689 2018-09-27T20:27:31  <jamesob> provoostenator: oops ^
690 2018-09-27T20:27:33  <gmaxwell> It's a lot more than that.
691 2018-09-27T20:27:49  <wumpus> even the "let's use a SQL database, oh why not postgres" I'm fairly sure we alrady had back in 2012
692 2018-09-27T20:27:57  <gmaxwell> Trezor, for example, is effectively a hosted web wallet, though with signing using your hardware fob.
693 2018-09-27T20:28:26  <jamesob> gmaxwell: right - which I hate but nonetheless use
694 2018-09-27T20:28:59  <gmaxwell> I am really frustrated by the fact that any time anyone mentions storing anything on disk there is a tiresome discussion of the trendy database things. Someone above joked mongo, but in the 2013 (IIRC) flavor of these rehashings, it would actually get suggested seriously.
695 2018-09-27T20:29:02  <provoostenator> And because people are physically holding it, they're less aware of that problem.
696 2018-09-27T20:29:28  <provoostenator> Everyone always saying "don't use web wallets, use a hardware wallet"..
697 2018-09-27T20:29:30  <wumpus> the other wallets aren't better in every regard, that's for sure, the core wallet is pretty good, I'm just tired of some kinds of discussions
698 2018-09-27T20:29:47  <gmaxwell> jamesob: my point in responding to luke there, that no amount of changed to bitcoind is going to make the 'trezor wallet' work with it.
699 2018-09-27T20:29:52  *** Murch has quit IRC
700 2018-09-27T20:29:56  <booyah> hashtag nosql
701 2018-09-27T20:30:00  <wumpus> gmaxwell: exactly...
702 2018-09-27T20:30:02  <provoostenator> jamesob: at least there's hope: https://gist.github.com/achow101/a9cf757d45df56753fae9d65db4d6e1d
703 2018-09-27T20:30:05  <gmaxwell> (getting support for hardware wallets, OTOH, would enable people to not use the trezor wallet, and thats another matter)
704 2018-09-27T20:30:13  <booyah> gmaxwell: as for trezor, well with electrum it's a signing engine, not webwallet at all
705 2018-09-27T20:30:37  *** Murch has joined #bitcoin-core-dev
706 2018-09-27T20:30:54  <luke-jr> gmaxwell: Electrum already supports Trezors (altohugh to support Electrum, we need a lot more than just port forwarding)
707 2018-09-27T20:31:00  <gmaxwell> booyah: right, and PSBT was a first step in getting thins so we could use devices like that.
708 2018-09-27T20:31:01  <wumpus> I'm happy that we managed to depreacte accounts at least, to simplify things, that was also an issue running almost since satoshi left
709 2018-09-27T20:31:10  <booyah> gmaxwell: freaking finally ;)
710 2018-09-27T20:31:11  <wumpus> yes hardware wallet support is something worth working on!
711 2018-09-27T20:31:15  <jamesob> gmaxwell provoostenator: I think even now you can do some bitcoind -> electrum personal server -> electrum -> trezor thing, but I haven't tried it because it sounds like a ton of work
712 2018-09-27T20:31:18  <wumpus> that would be actually nice
713 2018-09-27T20:31:28  <wumpus> in contrast to the eternal database discussion
714 2018-09-27T20:31:41  <gmaxwell> booyah: unfortunately its hard going because the vendors of these things have not been helping, instead mostly just making their own custom stuff for time to market reasons.
715 2018-09-27T20:31:47  <jonasschnelli> jamesob: its working and setup is easy
716 2018-09-27T20:31:56  <booyah> gmaxwell: satoshi labs was not helpful?
717 2018-09-27T20:31:56  <provoostenator> I've tried the electrum personal server approach. Works reasonably well. I've also tried the achow101 approach, which works great once you're used to it, but too manual for now.
718 2018-09-27T20:32:07  <jamesob> jonasschnelli: oh cool, I'll try that then.
719 2018-09-27T20:32:19  <wumpus> https://github.com/romanz/electrs looks really neat
720 2018-09-27T20:32:20  <jonasschnelli> jamesob: Bitbox has a native desktop app that can connect to EPS <-> Core
721 2018-09-27T20:32:32  <wumpus> (haven't tried it yet though)
722 2018-09-27T20:32:35  <gmaxwell> jamesob: and it involves using electrum... which has plenty of its own downsides.
723 2018-09-27T20:32:41  <jamesob> right
724 2018-09-27T20:33:15  <gmaxwell> I'm certantly not faulting the HW wallet vendors for focusing on time to market, it's just what it is.
725 2018-09-27T20:33:26  <provoostenator> I have some WIP stuff where I try to turn achow101's HWI scripts into an RPC server, which could eventually be a universal "sign something" service, not necessarily restricted to hardware wallets.
726 2018-09-27T20:33:59  <wumpus> me neither, though they have been kind of shooting themselves in the foot by not thinking about a common standard, but maybe that's something taht can only arise after experimentation... happy about PSBT
727 2018-09-27T20:34:04  <provoostenator> In that case all Core needs to do is make a bunch of RPC calls to a user configured URI (or a file socket maybe).
728 2018-09-27T20:34:39  <gmaxwell> provoostenator: I'd suggested previously that we exeute a configured command and shove crap at it over stdin.
729 2018-09-27T20:34:42  <provoostenator> And then it's up to hardware wallet makers to listen for those calls.
730 2018-09-27T20:35:01  <gmaxwell> (I'm less of a fan of URIs because of the security problems, e.g. when some local JS starts connecting to it...)
731 2018-09-27T20:35:08  <sipa> provoostenator: or PSBT and invoke a binary to sign it
732 2018-09-27T20:35:08  <provoostenator> Executing commands sounds scary too
733 2018-09-27T20:35:20  <gmaxwell> provoostenator: hows it scary? it's configured explicitly.
734 2018-09-27T20:35:58  <provoostenator> True, malware gonna malware, no matter how you do it.
735 2018-09-27T20:36:06  <gmaxwell> provoostenator: the URI thing in general has a challenge that browsers can make connections to things that listen to the network. There have been a LOT of thefts in eth land, due to 'wallets' that take commands from local sockets.
736 2018-09-27T20:36:32  <clarkmoody> Apply UNIX philosophy to wallet: text is the universal interface
737 2018-09-27T20:36:36  <gmaxwell> provoostenator: I'm not talking about malware running unconstrained on your host, I'm talking about just stuff in your webbrowser.
738 2018-09-27T20:36:38  <wumpus> in any case I think it makes sense to, in meetings, discuss the things each participant plans on working on, not wild far future plans
739 2018-09-27T20:36:39  <provoostenator> gmaxwell: c-lightning uses file sockets which at least prevent hat.
740 2018-09-27T20:36:56  <luke-jr> provoostenator: on Windows? O.o
741 2018-09-27T20:36:57  <wumpus> otherwise we end up discussing the same things from 2012 every time
742 2018-09-27T20:37:04  <provoostenator> luke-jr: there's that...
743 2018-09-27T20:37:05  <gmaxwell> provoostenator: okay, thats essentially the same security model as what I was talking about. didn't know you could do that on windows.
744 2018-09-27T20:37:55  *** michaelsdunn1 has quit IRC
745 2018-09-27T20:38:03  <provoostenator> I wish browsers didn't let websites talk to any URL in the universe, but it seems they do.
746 2018-09-27T20:38:10  <gmaxwell> wumpus: The way it seems to go though is.. "I plan on working on something something something database" "Excuse me sir, but have you considered Monodb? It has webscale"  then goto 2012;
747 2018-09-27T20:38:44  *** rex4539 has joined #bitcoin-core-dev
748 2018-09-27T20:38:44  <gmaxwell> maybe I just need to write a webpage to link to, with links to every other time we've gone through this loop.
749 2018-09-27T20:38:55  <wumpus> windows also has various local RPC methods
750 2018-09-27T20:39:49  <wumpus> gmaxwell: agree, there seems to be a lack of memory in these kind of things, although it seems to slightly improve, back then we had the literal same discussion within weeks of each other with new people
751 2018-09-27T20:39:51  <provoostenator> gmaxwell: a nice use case for RPC calls would be a web based multisig service.
752 2018-09-27T20:40:21  *** michaelsdunn1 has joined #bitcoin-core-dev
753 2018-09-27T20:40:23  <gmaxwell> provoostenator: in any case the model of "you tell your wallet to talk to some other local program that has the task of signing things" sounds great to me. It also could be used "talk to some other local program that has the task of collecting other signatures for multisig" easily.
754 2018-09-27T20:40:29  <wumpus> (I think recent windows 10 even has UNIX sockets)
755 2018-09-27T20:40:47  <gmaxwell> provoostenator: I'd say jinx except this was all pointed out last time I was lobbying in that direction. ;)
756 2018-09-27T20:40:58  <provoostenator> Yes, it just means those services have to ship an executable, but that trade-off might be worth it.
757 2018-09-27T20:41:18  <wumpus> but in any case, if you want a *secure* RPC method, you will have to determine for every OS what is the best choice
758 2018-09-27T20:41:22  <gmaxwell> provoostenator: "Driver", it might well be the case that eventually we ship drivers for common cases ourselves.
759 2018-09-27T20:41:34  <jamesob> gmaxwell: you know we're trending upwards because no one *seriously* recommended mongodb this time around
760 2018-09-27T20:41:43  <wumpus> just deciding for 'UNIX sockets' is very UNIX-centric by default
761 2018-09-27T20:41:56  <gmaxwell> jamesob: I think thats just because the world at large no longer thinks mongodb is the new hotness. :)
762 2018-09-27T20:42:04  <jamesob> heh, true
763 2018-09-27T20:42:09  <provoostenator> gmaxwell: especially if someone comes up with a proper USB protocol standard for talking with devices, so we wouldn't have to merge device specific stuff?
764 2018-09-27T20:42:12  <gmaxwell> wumpus: well thats why I liked stdio, as it does exist everwhere. :P
765 2018-09-27T20:42:25  <wumpus> mongodb suggestion was a reductio ad absurdum by me this time
766 2018-09-27T20:42:30  *** ott0disk has quit IRC
767 2018-09-27T20:42:54  <gmaxwell> provoostenator: right, but it's not just the usb, there are UI considerations. e.g. many hardware wallets have some pin entry on the host computer or similar.
768 2018-09-27T20:43:26  <gmaxwell> Basically the need to control enough of the host probably contributes to making your hardware wallet supported by creating a whole web wallet for it.
769 2018-09-27T20:43:27  <wumpus> gmaxwell: yes, communicating through a pipe over stdin/stdout seems to be something every OS has
770 2018-09-27T20:44:01  <provoostenator> At least Ledger and Trezor these days do pin entry on device. So in that case it's just a matter of waiting.
771 2018-09-27T20:44:38  <wumpus> (though on windows it's different; stdio only exists for console applications, not for winmain ones)
772 2018-09-27T20:45:06  <jamesob> the Trezor One requires on-host software for pin entry
773 2018-09-27T20:45:17  <wumpus> it's such a crazy labyrinth people have created, nothing is straightforward
774 2018-09-27T20:45:21  *** michaelsdunn1 has quit IRC
775 2018-09-27T20:46:23  <gmaxwell> jamesob: okay that one was the one that I was thinking about.
776 2018-09-27T20:46:35  *** irc_viewer_test has joined #bitcoin-core-dev
777 2018-09-27T20:48:25  *** clarkmoody has quit IRC
778 2018-09-27T20:49:41  <provoostenator> There's a bunch of ports that some browsers ban, but it's quite narrow, and those are already used by the system (that's why they're blocked). I also don't think there's a standard.
779 2018-09-27T20:53:00  <provoostenator> You can add authentication, maybe some tofu mechanism. But browers are extremely powerful these days.
780 2018-09-27T20:53:14  *** hebasto has quit IRC
781 2018-09-27T20:53:45  <provoostenator> Even USB is no longer off limits: https://developers.google.com/web/updates/2016/03/access-usb-devices-on-the-web
782 2018-09-27T20:53:54  <phantomcircuit> gmaxwell, sqlite has a safe *but slow* single file mode
783 2018-09-27T20:54:03  <phantomcircuit> like really slow
784 2018-09-27T20:54:15  <phantomcircuit> cause it's using fsync() multiple times per write
785 2018-09-27T20:54:23  *** elichai2 has joined #bitcoin-core-dev
786 2018-09-27T20:55:43  <achow101> provoostenator: the whole HWI thing is intended for the "execute a command and shove crap at it over stdin"
787 2018-09-27T20:59:23  <wumpus> phantomcircuit: writes to the wallet are more or less rare, though, so it might still be ok
788 2018-09-27T20:59:37  <gmaxwell> phantomcircuit: that sounds like it would probably be fine
789 2018-09-27T21:01:42  *** jimmysong has quit IRC
790 2018-09-27T21:07:22  <wumpus> by far and far, most queries will be reads
791 2018-09-27T21:08:15  <wumpus> this is not like the utxo set where every read tends to imply a write/update
792 2018-09-27T21:09:07  *** Krellan has quit IRC
793 2018-09-27T21:12:07  *** Murch has quit IRC
794 2018-09-27T21:12:18  *** Krellan has joined #bitcoin-core-dev
795 2018-09-27T21:28:11  *** irc_viewer_test has quit IRC
796 2018-09-27T21:38:16  *** irc_viewer_test has joined #bitcoin-core-dev
797 2018-09-27T21:38:48  *** Murch has joined #bitcoin-core-dev
798 2018-09-27T21:39:38  *** reallll has joined #bitcoin-core-dev
799 2018-09-27T21:39:41  *** belcher has quit IRC
800 2018-09-27T21:48:08  *** irc_viewer_test has quit IRC
801 2018-09-27T21:50:03  *** Chris_Stewart_5 has quit IRC
802 2018-09-27T22:02:00  *** jarthur has joined #bitcoin-core-dev
803 2018-09-27T22:04:05  *** jimmysong has joined #bitcoin-core-dev
804 2018-09-27T22:15:58  *** ghost43 has quit IRC
805 2018-09-27T22:16:16  *** ghost43 has joined #bitcoin-core-dev
806 2018-09-27T22:31:32  *** jarthur has quit IRC
807 2018-09-27T22:41:58  *** spinza has quit IRC
808 2018-09-27T22:51:47  *** Chris_Stewart_5 has joined #bitcoin-core-dev
809 2018-09-27T22:57:35  <phantomcircuit> wumpus, reads are fast... for a sql database
810 2018-09-27T22:57:48  <phantomcircuit> but are significantly slower than the in memory stuff is now
811 2018-09-27T22:58:21  <phantomcircuit> gmaxwell, seems the issue is in the commit just before the poll() implementation actually
812 2018-09-27T22:59:50  *** promag has quit IRC
813 2018-09-27T23:04:06  *** Chris_Stewart_5 has quit IRC
814 2018-09-27T23:07:28  *** Chris_Stewart_5 has joined #bitcoin-core-dev
815 2018-09-27T23:14:58  *** dqx has joined #bitcoin-core-dev
816 2018-09-27T23:22:30  *** paracyst_ has quit IRC
817 2018-09-27T23:23:08  *** Chris_Stewart_5 has quit IRC
818 2018-09-27T23:26:56  *** grubles has quit IRC
819 2018-09-27T23:31:26  *** paracyst has joined #bitcoin-core-dev
820 2018-09-27T23:32:08  *** dqx has quit IRC
821 2018-09-27T23:36:33  *** Chris_Stewart_5 has joined #bitcoin-core-dev
822 2018-09-27T23:36:48  *** gribble has quit IRC
823 2018-09-27T23:49:44  *** gribble has joined #bitcoin-core-dev
824 2018-09-27T23:50:47  *** promag has joined #bitcoin-core-dev
825 2018-09-27T23:58:56  *** Giszmo has quit IRC