1 2018-09-21T00:01:11  *** AaronvanW has quit IRC
  2 2018-09-21T00:01:48  *** AaronvanW has joined #bitcoin-core-dev
  3 2018-09-21T00:02:31  *** pipp8 has quit IRC
  4 2018-09-21T00:03:56  *** Murch has quit IRC
  5 2018-09-21T00:06:10  *** AaronvanW has quit IRC
  6 2018-09-21T00:11:51  *** AaronvanW has joined #bitcoin-core-dev
  7 2018-09-21T00:16:29  *** VanCo has joined #bitcoin-core-dev
  8 2018-09-21T00:16:38  *** AaronvanW has quit IRC
  9 2018-09-21T00:16:41  *** lnostdal has joined #bitcoin-core-dev
 10 2018-09-21T00:23:28  <gmaxwell> https://bitcoincore.org/en/2018/09/20/notice/
 11 2018-09-21T00:25:12  <luke-jr> this seems way too premature; only 2% of the network's upgraded :/
 12 2018-09-21T00:27:19  <gmaxwell> unfortunately, the issue was made public.
 13 2018-09-21T00:27:30  <luke-jr> :/
 14 2018-09-21T00:28:06  <gmaxwell> it wasn't clear if it would propagate or die out, but since it was moderatly easy to discover on your own, even just the rumors of it risked someone exploiting it.
 15 2018-09-21T00:30:12  *** gribble has quit IRC
 16 2018-09-21T00:31:29  <achow101> gmaxwell: luke-jr: there's an r/btc thread about someone finding the bug
 17 2018-09-21T00:32:12  <gmaxwell> achow101: it's referenced in the message. 'circulating'
 18 2018-09-21T00:34:08  <jamesob> Wow, that was fast
 19 2018-09-21T00:36:35  *** jarthur has joined #bitcoin-core-dev
 20 2018-09-21T00:40:52  <promag> jnewbery: fyi #14283
 21 2018-09-21T00:41:05  *** gribble has joined #bitcoin-core-dev
 22 2018-09-21T00:43:22  *** VanCo has quit IRC
 23 2018-09-21T00:45:16  *** tryphe has quit IRC
 24 2018-09-21T00:45:41  *** tryphe has joined #bitcoin-core-dev
 25 2018-09-21T00:46:34  *** intcat has quit IRC
 26 2018-09-21T00:46:34  *** VanCo has joined #bitcoin-core-dev
 27 2018-09-21T00:48:16  *** intcat has joined #bitcoin-core-dev
 28 2018-09-21T01:05:00  *** promag has quit IRC
 29 2018-09-21T01:05:13  *** drexl has quit IRC
 30 2018-09-21T01:10:43  *** Murch has joined #bitcoin-core-dev
 31 2018-09-21T01:11:16  <gmaxwell> theymos has a new announcement up https://www.reddit.com/r/Bitcoin/comments/9hkoo6/new_info_escalates_importance_upgrading_to_0163/
 32 2018-09-21T01:12:31  *** AaronvanW has joined #bitcoin-core-dev
 33 2018-09-21T01:17:38  *** AaronvanW has quit IRC
 34 2018-09-21T01:20:13  <gmaxwell> should I PR the extended test case?
 35 2018-09-21T01:35:35  *** promag has joined #bitcoin-core-dev
 36 2018-09-21T01:38:05  <luke-jr> gmaxwell: I'd wait
 37 2018-09-21T01:39:59  <gmaxwell> OK.
 38 2018-09-21T01:41:06  *** promag has quit IRC
 39 2018-09-21T01:44:04  *** Chris_Stewart_5 has joined #bitcoin-core-dev
 40 2018-09-21T01:45:18  *** tryphe has quit IRC
 41 2018-09-21T01:45:40  *** tryphe has joined #bitcoin-core-dev
 42 2018-09-21T01:50:09  *** VanCo has quit IRC
 43 2018-09-21T01:56:05  <luke-jr> did anyone mail the announcement ML about 0.16.3? I don't think I saw one..
 44 2018-09-21T01:56:54  *** Victorsueca has quit IRC
 45 2018-09-21T01:56:57  *** Krellan has quit IRC
 46 2018-09-21T01:58:03  <achow101> luke-jr: there was an announcement for it
 47 2018-09-21T02:00:38  <aj> is bitcoin-core-dev the announcement list? https://lists.linuxfoundation.org/pipermail/bitcoin-core-dev/2018-September/000060.html
 48 2018-09-21T02:00:51  <luke-jr> no
 49 2018-09-21T02:00:57  <luke-jr> this one https://bitcoincore.org/en/list/announcements/join/
 50 2018-09-21T02:03:34  <nanotube> would it make sense to propose a 'contact' page on bitcoin.org  similar to the one on bitcoincore.org? it appears it is non-trivial to find where to privately report security issues unless one knows to go to bitcoincore.org, since earlz had to come asking on #bitcoin-dev and -core-dev for a way to report.
 51 2018-09-21T02:03:54  *** Krellan has joined #bitcoin-core-dev
 52 2018-09-21T02:04:21  *** Chris_Stewart_5 has quit IRC
 53 2018-09-21T02:05:48  <gmaxwell> nanotube: :( I feel really uncomfortable with people going to bitcoin.org for that kind of information.
 54 2018-09-21T02:05:49  <harding> nanotube: any page on Bitcoin.org, top menu, Participate, Development, "to report an issue, please see the Bug Reporting page", Responsible Disclosure.
 55 2018-09-21T02:05:56  <gmaxwell> but its there
 56 2018-09-21T02:05:57  <gmaxwell> yea
 57 2018-09-21T02:08:18  *** Krellan has quit IRC
 58 2018-09-21T02:11:26  <nanotube> yes not ideal, but people probably still go there unless they know to check bitcoincore.org... so, good that it has the bug reporting page in there somewhere.
 59 2018-09-21T02:35:57  *** RubenSomsen has joined #bitcoin-core-dev
 60 2018-09-21T02:45:16  *** tryphe has quit IRC
 61 2018-09-21T02:45:40  *** tryphe has joined #bitcoin-core-dev
 62 2018-09-21T02:55:07  *** promag has joined #bitcoin-core-dev
 63 2018-09-21T02:58:39  <echeveria> there used to be bitcoin-security, but that was handled sort of poorly.
 64 2018-09-21T02:59:07  <echeveria> the contents of it ended up being published when someone stole the old satoshi email address.
 65 2018-09-21T02:59:27  *** promag has quit IRC
 66 2018-09-21T03:11:30  *** promag has joined #bitcoin-core-dev
 67 2018-09-21T03:13:20  *** AaronvanW has joined #bitcoin-core-dev
 68 2018-09-21T03:15:47  *** promag has quit IRC
 69 2018-09-21T03:18:41  *** AaronvanW has quit IRC
 70 2018-09-21T03:37:29  *** promag has joined #bitcoin-core-dev
 71 2018-09-21T03:41:43  *** promag has quit IRC
 72 2018-09-21T03:45:16  *** tryphe has quit IRC
 73 2018-09-21T03:45:41  *** tryphe has joined #bitcoin-core-dev
 74 2018-09-21T03:45:51  *** shinohai has quit IRC
 75 2018-09-21T04:00:08  *** gribble has quit IRC
 76 2018-09-21T04:08:45  *** BlueMatt has quit IRC
 77 2018-09-21T04:09:16  *** jnewbery has quit IRC
 78 2018-09-21T04:13:45  *** promag has joined #bitcoin-core-dev
 79 2018-09-21T04:14:08  *** BlueMatt has joined #bitcoin-core-dev
 80 2018-09-21T04:17:04  <kanzure> hi. mailing list admin has hit a bug and is unusable at the moment.
 81 2018-09-21T04:18:32  <kanzure> We're sorry, we hit a bug!
 82 2018-09-21T04:18:33  <kanzure> Please inform the webmaster for this site of this problem. Printing of traceback and other system information has been explicitly inhibited, but the webmaster can find this information in the Mailman error logs.
 83 2018-09-21T04:21:31  *** promag has quit IRC
 84 2018-09-21T04:22:27  <sipa> great.
 85 2018-09-21T04:22:38  *** gribble has joined #bitcoin-core-dev
 86 2018-09-21T04:32:35  *** paracyst_ has joined #bitcoin-core-dev
 87 2018-09-21T04:32:58  *** paracyst has quit IRC
 88 2018-09-21T04:33:41  <echeveria> I was looking at one of the public internet mapping tools for bitcoin core versions. there's a pretty disturbing number of hosts that have 8332 open.
 89 2018-09-21T04:35:27  *** baldur has quit IRC
 90 2018-09-21T04:36:56  <echeveria> is there some tool or setup guide that is telling people to open this port? I thought it was pretty difficult (not a single switch) to get Bitcoin Core to bind the RPC interface to 0.0.0.0.
 91 2018-09-21T04:45:16  *** tryphe has quit IRC
 92 2018-09-21T04:45:41  *** tryphe has joined #bitcoin-core-dev
 93 2018-09-21T04:46:26  <echeveria> of 8000 IPv4 nodes, 1142 have a RPC port 8332 that respond to a SYN.
 94 2018-09-21T04:48:00  *** bsm117532 has joined #bitcoin-core-dev
 95 2018-09-21T04:48:16  *** baldur has joined #bitcoin-core-dev
 96 2018-09-21T04:49:44  *** promag has joined #bitcoin-core-dev
 97 2018-09-21T04:54:23  *** promag has quit IRC
 98 2018-09-21T04:54:52  *** CubicEarth has quit IRC
 99 2018-09-21T04:56:05  *** CubicEarth has joined #bitcoin-core-dev
100 2018-09-21T04:59:56  *** jarthur has quit IRC
101 2018-09-21T05:09:24  *** promag has joined #bitcoin-core-dev
102 2018-09-21T05:10:17  <gmaxwell> maybe honeypots?
103 2018-09-21T05:10:34  <gmaxwell> as you note, you must take extra steps to bind..
104 2018-09-21T05:13:49  *** promag has quit IRC
105 2018-09-21T05:45:16  *** tryphe has quit IRC
106 2018-09-21T05:45:42  *** tryphe has joined #bitcoin-core-dev
107 2018-09-21T05:52:03  <echeveria> I don't think so. they're over a huge number of different hosts, old and new.
108 2018-09-21T05:53:45  *** promag has joined #bitcoin-core-dev
109 2018-09-21T05:58:21  *** promag has quit IRC
110 2018-09-21T06:03:46  *** promag has joined #bitcoin-core-dev
111 2018-09-21T06:08:04  *** promag has quit IRC
112 2018-09-21T06:14:40  *** promag has joined #bitcoin-core-dev
113 2018-09-21T06:17:42  *** promag has quit IRC
114 2018-09-21T06:20:22  *** StopAndDecrypt has quit IRC
115 2018-09-21T06:22:11  *** StopAndDecrypt has joined #bitcoin-core-dev
116 2018-09-21T06:25:15  *** Urgo has joined #bitcoin-core-dev
117 2018-09-21T06:45:16  *** tryphe has quit IRC
118 2018-09-21T06:45:41  *** tryphe has joined #bitcoin-core-dev
119 2018-09-21T06:52:16  *** hebasto has joined #bitcoin-core-dev
120 2018-09-21T07:01:34  *** Murch has quit IRC
121 2018-09-21T07:04:45  *** ken2812221_ has joined #bitcoin-core-dev
122 2018-09-21T07:06:25  <ken2812221_> MarcoFalke: Your gpg signing key has expired
123 2018-09-21T07:07:33  *** nullptr| has quit IRC
124 2018-09-21T07:15:02  *** nullptr| has joined #bitcoin-core-dev
125 2018-09-21T07:16:02  *** AaronvanW has joined #bitcoin-core-dev
126 2018-09-21T07:18:29  *** harrymm_ has quit IRC
127 2018-09-21T07:20:36  *** AaronvanW has quit IRC
128 2018-09-21T07:21:33  *** go1111111 has joined #bitcoin-core-dev
129 2018-09-21T07:45:16  *** tryphe has quit IRC
130 2018-09-21T07:45:41  *** tryphe has joined #bitcoin-core-dev
131 2018-09-21T07:46:29  *** hebasto has quit IRC
132 2018-09-21T07:48:21  *** ishvarts has joined #bitcoin-core-dev
133 2018-09-21T07:55:13  *** Guyver2 has joined #bitcoin-core-dev
134 2018-09-21T08:02:45  *** setpill has joined #bitcoin-core-dev
135 2018-09-21T08:07:40  *** games_ has quit IRC
136 2018-09-21T08:13:57  *** Victorsueca has joined #bitcoin-core-dev
137 2018-09-21T08:33:05  *** promag has joined #bitcoin-core-dev
138 2018-09-21T08:45:16  *** tryphe has quit IRC
139 2018-09-21T08:45:29  *** promag has quit IRC
140 2018-09-21T08:45:41  *** tryphe has joined #bitcoin-core-dev
141 2018-09-21T08:46:27  *** AaronvanW has joined #bitcoin-core-dev
142 2018-09-21T08:49:25  *** bralyclow has joined #bitcoin-core-dev
143 2018-09-21T08:49:40  *** Guyver2 has quit IRC
144 2018-09-21T08:51:34  *** justan0theruser has quit IRC
145 2018-09-21T08:52:40  *** promag has joined #bitcoin-core-dev
146 2018-09-21T09:22:08  *** rex4539 has quit IRC
147 2018-09-21T09:26:04  *** rex4539 has joined #bitcoin-core-dev
148 2018-09-21T09:42:11  *** timothy has joined #bitcoin-core-dev
149 2018-09-21T09:45:16  *** tryphe has quit IRC
150 2018-09-21T09:45:41  *** tryphe has joined #bitcoin-core-dev
151 2018-09-21T09:56:21  *** promag has quit IRC
152 2018-09-21T10:14:11  *** harrymm has joined #bitcoin-core-dev
153 2018-09-21T10:45:16  *** tryphe has quit IRC
154 2018-09-21T10:45:41  *** tryphe has joined #bitcoin-core-dev
155 2018-09-21T11:01:41  *** drexl has joined #bitcoin-core-dev
156 2018-09-21T11:05:24  *** rex4539 has quit IRC
157 2018-09-21T11:15:03  *** ishvarts has quit IRC
158 2018-09-21T11:22:02  *** rh0nj has quit IRC
159 2018-09-21T11:23:07  *** rh0nj has joined #bitcoin-core-dev
160 2018-09-21T11:45:16  *** tryphe has quit IRC
161 2018-09-21T11:45:41  *** tryphe has joined #bitcoin-core-dev
162 2018-09-21T11:50:54  *** Chris_Stewart_5 has joined #bitcoin-core-dev
163 2018-09-21T12:01:01  *** SopaXorzTaker has joined #bitcoin-core-dev
164 2018-09-21T12:04:47  *** Jmabsd has joined #bitcoin-core-dev
165 2018-09-21T12:05:19  <Jmabsd> the code that qualifies and validates a segwit transaction starts on what code locations?
166 2018-09-21T12:10:26  <Jmabsd> mostly validation.cpp's AcceptToMemoryPoolWorker
167 2018-09-21T12:21:43  <Jmabsd> Where is the code that checks the witness merkle root in the coinbase transaction?
168 2018-09-21T12:23:51  *** SopaXorzTaker has quit IRC
169 2018-09-21T12:34:03  *** SopaXorzTaker has joined #bitcoin-core-dev
170 2018-09-21T12:36:34  *** setpill has quit IRC
171 2018-09-21T12:36:43  *** rex4539 has joined #bitcoin-core-dev
172 2018-09-21T12:45:16  *** tryphe has quit IRC
173 2018-09-21T12:45:28  *** owowo has quit IRC
174 2018-09-21T12:45:40  *** tryphe has joined #bitcoin-core-dev
175 2018-09-21T12:48:44  *** rex4539 has quit IRC
176 2018-09-21T12:50:40  *** owowo has joined #bitcoin-core-dev
177 2018-09-21T13:04:27  *** promag has joined #bitcoin-core-dev
178 2018-09-21T13:07:44  *** csknk has joined #bitcoin-core-dev
179 2018-09-21T13:12:07  *** games_ has joined #bitcoin-core-dev
180 2018-09-21T13:13:24  *** Sentineo has joined #bitcoin-core-dev
181 2018-09-21T13:24:47  *** grubles_ has joined #bitcoin-core-dev
182 2018-09-21T13:24:58  *** promag has quit IRC
183 2018-09-21T13:27:13  *** grubles has quit IRC
184 2018-09-21T13:45:19  *** tryphe has quit IRC
185 2018-09-21T13:45:53  *** tryphe has joined #bitcoin-core-dev
186 2018-09-21T13:48:52  *** voltron has joined #bitcoin-core-dev
187 2018-09-21T13:54:04  *** hebasto has joined #bitcoin-core-dev
188 2018-09-21T13:56:43  *** owowo has quit IRC
189 2018-09-21T14:00:34  *** Krellan has joined #bitcoin-core-dev
190 2018-09-21T14:01:48  *** owowo has joined #bitcoin-core-dev
191 2018-09-21T14:02:27  *** bralyclow has quit IRC
192 2018-09-21T14:03:06  *** bralyclow has joined #bitcoin-core-dev
193 2018-09-21T14:05:04  *** Krellan has quit IRC
194 2018-09-21T14:06:44  *** promag has joined #bitcoin-core-dev
195 2018-09-21T14:07:27  *** rafalcpp has quit IRC
196 2018-09-21T14:13:09  *** qinfengling has joined #bitcoin-core-dev
197 2018-09-21T14:19:46  *** jarthur has joined #bitcoin-core-dev
198 2018-09-21T14:21:47  *** rafalcpp has joined #bitcoin-core-dev
199 2018-09-21T14:25:01  *** rh0nj has quit IRC
200 2018-09-21T14:26:07  *** rh0nj has joined #bitcoin-core-dev
201 2018-09-21T14:38:00  *** grubles_ has quit IRC
202 2018-09-21T14:45:17  *** tryphe has quit IRC
203 2018-09-21T14:45:43  *** tryphe has joined #bitcoin-core-dev
204 2018-09-21T14:49:12  *** grubles has joined #bitcoin-core-dev
205 2018-09-21T14:56:46  *** owowo has quit IRC
206 2018-09-21T15:00:53  *** michaelsdunn1 has joined #bitcoin-core-dev
207 2018-09-21T15:05:35  *** owowo has joined #bitcoin-core-dev
208 2018-09-21T15:09:25  *** rex4539 has joined #bitcoin-core-dev
209 2018-09-21T15:18:32  *** grubles has quit IRC
210 2018-09-21T15:18:52  *** grubles has joined #bitcoin-core-dev
211 2018-09-21T15:29:27  *** owowo has quit IRC
212 2018-09-21T15:31:45  *** emilengler has joined #bitcoin-core-dev
213 2018-09-21T15:33:13  <emilengler> If I download the linux tarball, will I be able to select a download path for the blockchain ?
214 2018-09-21T15:33:59  <luke-jr> yes, if you know how or use the GUI
215 2018-09-21T15:34:09  <luke-jr> better topic for #Bitcoin
216 2018-09-21T15:34:31  <emilengler> Ok I will keep this in mind excuse me
217 2018-09-21T15:34:57  *** owowo has joined #bitcoin-core-dev
218 2018-09-21T15:40:12  *** emilengler has quit IRC
219 2018-09-21T15:40:45  *** ExtraCrispy has joined #bitcoin-core-dev
220 2018-09-21T15:45:17  *** tryphe has quit IRC
221 2018-09-21T15:45:41  *** tryphe has joined #bitcoin-core-dev
222 2018-09-21T16:04:07  *** Jmabsd has quit IRC
223 2018-09-21T16:11:14  *** Murch has joined #bitcoin-core-dev
224 2018-09-21T16:14:19  *** rex4539 has quit IRC
225 2018-09-21T16:16:41  *** BashCo has joined #bitcoin-core-dev
226 2018-09-21T16:18:27  *** mr_paz has joined #bitcoin-core-dev
227 2018-09-21T16:23:15  *** jarthur has quit IRC
228 2018-09-21T16:27:44  *** mr_paz_ has joined #bitcoin-core-dev
229 2018-09-21T16:28:22  *** mr_paz has quit IRC
230 2018-09-21T16:29:25  *** emilengler has joined #bitcoin-core-dev
231 2018-09-21T16:35:15  *** Victorsueca has quit IRC
232 2018-09-21T16:36:26  *** Victorsueca has joined #bitcoin-core-dev
233 2018-09-21T16:45:17  *** tryphe has quit IRC
234 2018-09-21T16:45:41  *** tryphe has joined #bitcoin-core-dev
235 2018-09-21T17:18:04  *** voltron has quit IRC
236 2018-09-21T17:29:59  *** timothy has quit IRC
237 2018-09-21T17:31:52  <provoostenator> I used invalidateblock on a remote node to go back to ~ 475000, but  lost the connection after a few hours. The last debug message is from an hour ago, an updatetip down to 508954. It's in a weird state.
238 2018-09-21T17:32:32  <provoostenator> Memory usage was swinging between 5GB and 8GB. I was able to shut it down via rpc, though the last message was "net thread exit" which sounds like an unclean exit.
239 2018-09-21T17:32:36  *** treebeardd has joined #bitcoin-core-dev
240 2018-09-21T17:34:37  <provoostenator> Restarting the node, now it's "Replaying blocks", "Rolling back ... 542229" and down from there.
241 2018-09-21T17:34:57  *** Sinclair_ has joined #bitcoin-core-dev
242 2018-09-21T17:34:59  *** Sinclair6 has quit IRC
243 2018-09-21T17:38:24  *** jnewbery has joined #bitcoin-core-dev
244 2018-09-21T17:45:17  *** tryphe has quit IRC
245 2018-09-21T17:45:41  *** tryphe has joined #bitcoin-core-dev
246 2018-09-21T17:47:46  *** hebasto has quit IRC
247 2018-09-21T17:59:13  <sdaftuar> provoostenator: what version bitcoind was it?
248 2018-09-21T18:00:05  <provoostenator> sdaftuar: v0.17.0rc2 (I was actually dumb enough to not upgrade it before doing this)
249 2018-09-21T18:01:38  <provoostenator> I also don't know if invalidateblock is supposed to work for such a huge rollback. Though if not, then perhaps the documentation should warn against that.
250 2018-09-21T18:02:00  <sdaftuar> well, i think we do want it to work
251 2018-09-21T18:02:22  <provoostenator> Also, the RPC call is blocking. Does getting disconnected have any bearing on that?
252 2018-09-21T18:02:51  <sdaftuar> no, the invalidateblock function should continue even after the rpc client disconnects, i think
253 2018-09-21T18:03:01  <sdaftuar> i believe if you had waited long enough, it probably would have finished?
254 2018-09-21T18:03:07  <sdaftuar> but it might be several hours
255 2018-09-21T18:03:10  <provoostenator> The logs also suggest it continued for about 30 minutes after disconnecting.
256 2018-09-21T18:04:12  <provoostenator> It got about halfway in 2-3 hours, so indeed it looked like it would have made it.
257 2018-09-21T18:04:12  <sdaftuar> disconnecting blocks is heavily disk-bound. when i last looked at it (on different hardware than i use today), i think i noticed i could disconnect on the order of 3-5 blocks/second, on average
258 2018-09-21T18:04:47  <provoostenator> This is an iMac with SSD and plenty of memory.
259 2018-09-21T18:05:00  <sdaftuar> we used to have an issue where the memory usage could grow sort of unbounded, as disconnected blocks would have their transactions added to the mempool
260 2018-09-21T18:05:08  <sdaftuar> but that was fixed
261 2018-09-21T18:05:23  <provoostenator> Yeah the weird thing I noticed is how dbcache kept growing as it was disconnecting.
262 2018-09-21T18:05:25  <sdaftuar> but your comment about 5-8GB of memory has me slightly concerned
263 2018-09-21T18:05:42  <provoostenator> The machine has 64 GB so it didn't run out.
264 2018-09-21T18:05:51  <sdaftuar> what is -dbcache set to?
265 2018-09-21T18:06:16  <provoostenator> 5000, so that's bad
266 2018-09-21T18:06:49  *** itaseski has joined #bitcoin-core-dev
267 2018-09-21T18:07:02  <provoostenator> The mempool is just the default, so that shouldn't have grown so much, right?
268 2018-09-21T18:07:29  <sdaftuar> yeah assuming the code works correctly, the mempool's memory usage would have been bounded pretty well
269 2018-09-21T18:08:56  <provoostenator> Last log entry had cache=4650.7MiB
270 2018-09-21T18:09:18  <sdaftuar> oh so that seems good then
271 2018-09-21T18:10:13  <provoostenator> The 5-8 GB RAM usage was an hour after the last log entry, when I reconnected, found through "top".
272 2018-09-21T18:14:11  <sdaftuar> alright well maybe this is all expected (crappy) behavior.  i don't know of any clever ideas to speed up block disconnection, unfortunately.
273 2018-09-21T18:14:37  <sdaftuar> maybe someone could implement https://github.com/bitcoin/bitcoin/issues/8037
274 2018-09-21T18:17:35  <provoostenator> Maybe invalidateblock could have a "don't bother adding to the mempool" option?
275 2018-09-21T18:19:09  <provoostenator> I just noticed I have txindex=1, so that could be another issue.
276 2018-09-21T18:19:34  <sdaftuar> provoostenator: yeah that's fair but i suspect it would still take hours
277 2018-09-21T18:19:43  <gmaxwell> provoostenator: it's not clear to me what you're saying you saw
278 2018-09-21T18:19:54  <gmaxwell> provoostenator: was it still rolling back when you stopped it?
279 2018-09-21T18:20:34  <gmaxwell> if it was then it just sounds like expected behavior.
280 2018-09-21T18:20:46  <provoostenator> gmaxwell: rolling back was after I restarted (it's still doing that now).
281 2018-09-21T18:20:48  <gmaxwell> I've rolled back all the way to block 0 many times, though not recently.
282 2018-09-21T18:21:02  <gmaxwell> provoostenator: yea, it'll keep going until it finishes.
283 2018-09-21T18:21:14  <kanzure> mailing list bug has been reoslved; can someone send the post mortem link to the mailing list subscribers plzkthx?  like https://bitcoincore.org/en/2018/09/20/notice/
284 2018-09-21T18:21:30  <sipa> kanzure: didn't BlueMatt send one?
285 2018-09-21T18:21:45  <provoostenator> Before I disconnected (a few hours ago) it was doing "UpdateTip: new best ..." in reverse order, as expcted from doing invalidateblock
286 2018-09-21T18:21:55  <kanzure> i don't see one in the mod queue
287 2018-09-21T18:22:03  <kanzure> and i don't see it on https://lists.linuxfoundation.org/pipermail/bitcoin-dev/2018-September/thread.html
288 2018-09-21T18:22:15  <provoostenator> The logs shows it kept doing that for 30 mins after I disconnected from the machine.
289 2018-09-21T18:23:01  <provoostenator> When I logged back into the machine, bitcoind was still running, using 5-8 GB of RAM (it was actually going up and down in the space of minutes), but log wasn't updating. I then stopped it via rpc and restarted.
290 2018-09-21T18:23:25  <provoostenator> So it seems it was still doing _something_, despite not logging.
291 2018-09-21T18:23:57  <luke-jr> FYI, I still didn't get anything for 0.6.3/CVE from https://bitcoincore.org/en/list/announcements/join/ yet
292 2018-09-21T18:24:02  <gmaxwell> provoostenator: what you're seeing without the logs is the atomic flush roll forward probably.
293 2018-09-21T18:24:07  <luke-jr> sipa: ^ since you are one of the 3 who can apparently send those
294 2018-09-21T18:24:16  <sdaftuar> provoostenator: gmaxwell: it does seem surprising i guess that an unclean shutdown happened?
295 2018-09-21T18:24:27  <sdaftuar> how would that be possible if you just use rpc to stop the node?
296 2018-09-21T18:25:25  <provoostenator> gmaxwell: what is a "atomic flush roll forward"?
297 2018-09-21T18:25:55  <sdaftuar> provoostenator: on startup, we detect if the utxo state wasn't finished being written as of what we think our tip is.
298 2018-09-21T18:26:09  <sdaftuar> in that situation, we have a rollback / rollforward mechanism to fix the utxo
299 2018-09-21T18:26:39  <sdaftuar> by disconnecting blocks that are no longer on our chain, and replaying the blocks that might need applying to the utxo state
300 2018-09-21T18:27:18  <sdaftuar> that should only happen after an unclean shutdown though
301 2018-09-21T18:27:45  <gmaxwell> not to change subject but anyone know what this is? https://www.reddit.com/r/Bitcoin/comments/9hrusk/orhpan_blocks/e6e4zhk/?context=3
302 2018-09-21T18:28:14  <gmaxwell> sdaftuar: indeed. I missed that the shutdown was supposted to be clean.
303 2018-09-21T18:33:57  *** Guyver2 has joined #bitcoin-core-dev
304 2018-09-21T18:38:23  <provoostenator> gmaxwell: that warning is triggered here: https://github.com/bitcoinj/bitcoinj/blob/master/core/src/main/java/org/bitcoinj/core/AbstractBlockChain.java#L584
305 2018-09-21T18:45:17  *** tryphe has quit IRC
306 2018-09-21T18:45:42  *** tryphe has joined #bitcoin-core-dev
307 2018-09-21T18:45:42  *** Victorsueca has quit IRC
308 2018-09-21T18:45:50  <provoostenator> "duplicate block" seems to mean that it already processed it, nothing to do with orphans.
309 2018-09-21T18:46:26  *** rafalcpp has quit IRC
310 2018-09-21T18:46:56  *** Victorsueca has joined #bitcoin-core-dev
311 2018-09-21T18:48:44  *** rafalcpp has joined #bitcoin-core-dev
312 2018-09-21T18:53:35  <kanzure> *poke* postmortem email plzkthx
313 2018-09-21T19:01:24  *** BashCo has quit IRC
314 2018-09-21T19:06:24  *** rex4539 has joined #bitcoin-core-dev
315 2018-09-21T19:09:06  *** BashCo has joined #bitcoin-core-dev
316 2018-09-21T19:12:24  *** jarthur has joined #bitcoin-core-dev
317 2018-09-21T19:12:42  <provoostenator> Ok, so roll back made it to 509650 and all seems well. Except the node seems to have forgotten I invalidated block 485000, because it jumped right into IBD and is moving forward again.
318 2018-09-21T19:16:09  <provoostenator> I guess that's because the node doesn't check the full block index at launch.
319 2018-09-21T19:19:05  <provoostenator> Restarted, now using v0.17.0rc4, doing nanother invalidateblock. Memory usage is almost 2GB higher than cache= shown in the logs, and seems to outpace it.
320 2018-09-21T19:19:18  <provoostenator> I've also turned off the index.
321 2018-09-21T19:21:20  <gmaxwell> what are the actual dirty page counts?
322 2018-09-21T19:21:51  <gmaxwell> we recently realized that OS cached pages in mmaped files show up in res.
323 2018-09-21T19:22:15  <provoostenator> Also, it's still going even though bitcoin-cli stop said it would stop. I'll look at the dirty page counts...
324 2018-09-21T19:22:46  <provoostenator> (note to self: do not google "top dirty pages")
325 2018-09-21T19:23:09  <sipa> lol
326 2018-09-21T19:23:47  <gmaxwell> oh sorry, pmap -x $(pidof bitcoind) | tail -n 1 | tr -s ' ' | cut -d' ' -f 4
327 2018-09-21T19:26:48  <provoostenator> macOS doesn't have pmap, but vmmap gives me this summary: https://gist.github.com/Sjors/6b01711ccd0f96128c7db5230c85ae8f
328 2018-09-21T19:27:10  *** SopaXorzTaker has quit IRC
329 2018-09-21T19:28:37  <provoostenator> Also a long list of "mapped file", e.g. many "locks/index/*.ldb"
330 2018-09-21T19:29:03  <gmaxwell> k, so ~2GB of your resident size is mapped files.
331 2018-09-21T19:29:31  <gmaxwell> whats your dbcache setting?
332 2018-09-21T19:30:00  <provoostenator> dbcache=5000 MB, the log currently says cache=2300 MiB, so that part makes sense?
333 2018-09-21T19:30:55  <provoostenator> It's the just the other 6 GB that needs explaining. Memory usage is now 10 GB. 38 more and the machine is going to OOM, which I'm not going to allow.
334 2018-09-21T19:32:31  <provoostenator> MALLOC_TINY is now at 8.7, so that seems to be the thing that's mooning.
335 2018-09-21T19:36:00  <provoostenator> (actually this is still v0.17.0rc2, sorry, though hopefully that doesn't matter here)
336 2018-09-21T19:36:11  *** owowo has quit IRC
337 2018-09-21T19:36:53  <provoostenator> (no, it is v0.17.0rc4)
338 2018-09-21T19:37:58  <provoostenator> kill has no effect, kill -9 did
339 2018-09-21T19:45:17  *** tryphe has quit IRC
340 2018-09-21T19:45:44  *** tryphe has joined #bitcoin-core-dev
341 2018-09-21T19:46:50  <provoostenator> Getting fairly consistent behavior now, even with disablewallet=1.  bitcoin-cli stop seems to stop the RPC server, but not the invalidation process. Curious if anyone can reproduce. I'll let it sync to the tip before trying again.
342 2018-09-21T19:47:48  <sipa> if invalidateblock does not succeed, its state isn't writte
343 2018-09-21T19:48:05  <sipa> it first disconnects the blocks, and then marks them as invalid
344 2018-09-21T19:48:21  <sipa> so at startup they will be connected again if bitcoind was killed in the middle
345 2018-09-21T19:50:13  <provoostenator> That makes sense. I wonder if it matter that I was essentially interrupting IBD with that invalidateblock call. Memory usage seemed way worse than what I saw earlier today.
346 2018-09-21T19:50:49  *** owowo has joined #bitcoin-core-dev
347 2018-09-21T19:50:59  <sipa> invalidateblock also keeps a list of transactions to re-add to the mempool after the invalidation completes
348 2018-09-21T19:51:09  <sipa> i assume that's the memory usage you see
349 2018-09-21T19:51:56  <provoostenator> Is it also not abortable once in progress?
350 2018-09-21T19:52:02  <sipa> no
351 2018-09-21T19:53:21  <provoostenator> Ok, so in that case the way to roll back a long way would be to do it in smaller increments.
352 2018-09-21T19:54:36  <sipa> right
353 2018-09-21T19:54:39  <sipa> that should work
354 2018-09-21T19:55:09  <gmaxwell> sipa: the mempool usage is limited.
355 2018-09-21T19:55:49  <sipa> gmaxwell: how?
356 2018-09-21T19:58:49  <gmaxwell> https://github.com/bitcoin/bitcoin/pull/9208
357 2018-09-21T20:01:11  <sipa> gmaxwell: it doesn't look like DisconnectedBlockTransactions enforces any memory limits
358 2018-09-21T20:01:41  <gmaxwell> MAX_DISCONNECTED_TX_POOL_SIZE
359 2018-09-21T20:02:04  <sipa> oh
360 2018-09-21T20:02:14  <provoostenator> There's also this open issue: #9027
361 2018-09-21T20:02:16  <gribble> https://github.com/bitcoin/bitcoin/issues/9027 | Unbounded reorg memory usage · Issue #9027 · bitcoin/bitcoin · GitHub
362 2018-09-21T20:02:51  <sipa> yup
363 2018-09-21T20:03:01  <sipa> i was expecting the code to be elsewhere, my bad
364 2018-09-21T20:05:46  <provoostenator> I take great pride in doing stupid things that lead to a new release candidate, so hopefully you'll find something :-)
365 2018-09-21T20:08:06  <gmaxwell> provoostenator: are you running txindex?
366 2018-09-21T20:08:36  <provoostenator> No, I did the first time today, but turned that off in more recent attempts.
367 2018-09-21T20:10:11  *** rex4539 has quit IRC
368 2018-09-21T20:11:57  *** rex4539 has joined #bitcoin-core-dev
369 2018-09-21T20:13:49  <provoostenator> Re incremental approach: I rolled back ~10,000 blocks using about 13GB of RAM, cache=2200 at the peak. Sounds like it's holding all transactions in memory.
370 2018-09-21T20:16:47  <provoostenator> But then it gets weird. ERROR: AcceptToMemoryPoolWorker: Consensus::CheckTxInputs: ... bad-txns-premature-spend-of-coinbase, tried to spend coinbase at depth 92
371 2018-09-21T20:17:11  <provoostenator> InvalidChainFound: invalid block [the block I invalidated]
372 2018-09-21T20:17:15  <gmaxwell> thats normal.
373 2018-09-21T20:17:31  <provoostenator> Yeah, but then it starts syncing again.
374 2018-09-21T20:19:03  *** rex4539 has quit IRC
375 2018-09-21T20:19:43  *** rex4539 has joined #bitcoin-core-dev
376 2018-09-21T20:22:30  <provoostenator> Ok, now I think I destroyed my chain :-)  At boot: "assertion failed: (!setBlockIndexCandidates.empty()), function PruneBlockIndexCandidates, file validation.cpp, line 2547"
377 2018-09-21T20:32:59  <sipa> provoostenator: i found the issue
378 2018-09-21T20:33:05  <sipa> it's specific to InvalidateBlock
379 2018-09-21T20:33:50  <provoostenator> sipa: nice!
380 2018-09-21T20:38:25  *** emilengler has quit IRC
381 2018-09-21T20:41:21  <provoostenator> sipa: is it because disconnectpool holds on to transactions which reference a shared_ptr<CBlock> pblock, so those don't get deallocated?
382 2018-09-21T20:41:29  <sipa> provoostenator: no
383 2018-09-21T20:42:51  *** JackH has joined #bitcoin-core-dev
384 2018-09-21T20:44:34  <sipa> the event queue holds on to the shared_ptr<CBlock> objects in callbacks to DisconnectedBlock
385 2018-09-21T20:44:52  <sipa> and InvalidateBlock doesn't limit the size of the queue
386 2018-09-21T20:46:39  <sipa> provoostenator: could you check whether this issue also occurs when Rewinding?
387 2018-09-21T20:47:00  <sipa> create a 0.13.0 node, sync it to tip, and then upgrade to 0.17+
388 2018-09-21T20:47:15  <sipa> i suspect it is, and if that's the case, i would consider it a release blocker
389 2018-09-21T20:48:10  <provoostenator> That's the rewind that happens if you upgrade a non-segwit node to a segwit node?
390 2018-09-21T20:48:15  <sipa> yup
391 2018-09-21T20:49:10  <provoostenator> I'll give it a try this weekend or early next week. Getting a bit late here. Maybe someone else gets to it first.
392 2018-09-21T20:49:21  <sipa> thanks!
393 2018-09-21T20:50:14  <provoostenator> I'm not looking forward to doing another release notes for 0.14 and 0.15 backports :-)
394 2018-09-21T21:03:49  *** treebeardd has quit IRC
395 2018-09-21T21:23:36  *** rafaeltokyo has joined #bitcoin-core-dev
396 2018-09-21T21:24:29  *** rafaeltokyo has left #bitcoin-core-dev
397 2018-09-21T21:28:00  *** rafaeltokyo has joined #bitcoin-core-dev
398 2018-09-21T21:28:08  *** rafaeltokyo has left #bitcoin-core-dev
399 2018-09-21T21:40:10  *** justan0theruser has joined #bitcoin-core-dev
400 2018-09-21T21:42:30  <MarcoFalke> About #14289, was it ever supported to call invalidateblock on a block very far back?
401 2018-09-21T21:42:31  <gribble> https://github.com/bitcoin/bitcoin/issues/14289 | Unbounded growth of scheduler queue · Issue #14289 · bitcoin/bitcoin · GitHub
402 2018-09-21T21:43:02  <sipa> MarcoFalke: i would say no, but it'd be a nice-to-have if it worked
403 2018-09-21T21:43:21  <sipa> having invalidateblock 100000 blocks deep use a massive amount of memory is not a blocker, i think
404 2018-09-21T21:43:54  <MarcoFalke> Ok, that was my impression because every time I tried that it would deadlock the node a bit until I got impatient and CTRL+C out
405 2018-09-21T21:44:22  <MarcoFalke> If that is supported with reasonable memory gurantees, we should add a test/benchmark so it doesn't randomly regress
406 2018-09-21T21:45:06  <MarcoFalke> Also my key is de-expired, but I am having issues uploading it to keyservers.
407 2018-09-21T21:45:17  *** tryphe has quit IRC
408 2018-09-21T21:45:37  <MarcoFalke> All of them return some obscure proxy error or timeout or ...
409 2018-09-21T21:45:43  *** tryphe has joined #bitcoin-core-dev
410 2018-09-21T21:46:30  <gmaxwell> MarcoFalke: yes worked fine since 9208.
411 2018-09-21T21:46:52  <gmaxwell> the rpc will disconnect, because the rpc timeout isn't long enough for it to finish, but a node will happly work its way back to block 0.
412 2018-09-21T21:48:55  *** Victorsueca has quit IRC
413 2018-09-21T21:50:11  *** Victorsueca has joined #bitcoin-core-dev
414 2018-09-21T22:02:32  <Murch> MarcoFalke: Luckily Keyservers may soonish be a thing of the past: https://wiki.gnupg.org/WKD
415 2018-09-21T22:27:07  *** michaelsdunn1 has quit IRC
416 2018-09-21T22:37:49  *** Guyver2 has quit IRC
417 2018-09-21T22:45:17  *** tryphe has quit IRC
418 2018-09-21T22:45:43  *** tryphe has joined #bitcoin-core-dev
419 2018-09-21T22:47:18  *** lnostdal has quit IRC
420 2018-09-21T22:49:08  *** JackH has quit IRC
421 2018-09-21T22:55:23  *** csknk has quit IRC
422 2018-09-21T23:14:51  *** jarthur has quit IRC
423 2018-09-21T23:32:19  *** ken2812221__ has joined #bitcoin-core-dev
424 2018-09-21T23:33:28  *** gribble has quit IRC
425 2018-09-21T23:33:39  *** adam3us has quit IRC
426 2018-09-21T23:35:18  *** rafalcpp has quit IRC
427 2018-09-21T23:35:18  *** promag has quit IRC
428 2018-09-21T23:35:18  *** ken2812221_ has quit IRC
429 2018-09-21T23:35:39  *** rafalcpp has joined #bitcoin-core-dev
430 2018-09-21T23:36:33  *** adam3us has joined #bitcoin-core-dev
431 2018-09-21T23:39:31  *** gribble has joined #bitcoin-core-dev
432 2018-09-21T23:45:17  *** tryphe has quit IRC
433 2018-09-21T23:45:41  *** tryphe has joined #bitcoin-core-dev
434 2018-09-21T23:45:41  *** r8921039 has joined #bitcoin-core-dev
435 2018-09-21T23:49:39  *** r8921039 has quit IRC