1 2018-09-09T00:24:35  *** miknotauro has joined #bitcoin-core-dev
  2 2018-09-09T00:29:00  *** echonaut has quit IRC
  3 2018-09-09T00:29:07  *** echonaut10 has joined #bitcoin-core-dev
  4 2018-09-09T01:36:02  *** drexl has quit IRC
  5 2018-09-09T01:38:58  *** polydin has joined #bitcoin-core-dev
  6 2018-09-09T02:19:51  *** saransh_ has joined #bitcoin-core-dev
  7 2018-09-09T02:31:39  *** saransh_ has quit IRC
  8 2018-09-09T02:53:48  *** ken2812221_ has joined #bitcoin-core-dev
  9 2018-09-09T02:54:47  *** ken2812221 has quit IRC
 10 2018-09-09T03:06:31  *** unholymachine has quit IRC
 11 2018-09-09T03:07:27  *** unholymachine has joined #bitcoin-core-dev
 12 2018-09-09T03:16:15  *** polydin has quit IRC
 13 2018-09-09T03:23:19  *** Krellan has quit IRC
 14 2018-09-09T03:24:17  *** Krellan has joined #bitcoin-core-dev
 15 2018-09-09T04:12:40  *** RubenSomsen has quit IRC
 16 2018-09-09T04:22:07  *** polydin has joined #bitcoin-core-dev
 17 2018-09-09T04:43:15  *** jhfrontz has joined #bitcoin-core-dev
 18 2018-09-09T04:50:09  *** adiabat has joined #bitcoin-core-dev
 19 2018-09-09T05:02:08  *** jhfrontz has quit IRC
 20 2018-09-09T05:05:27  *** jhfrontz has joined #bitcoin-core-dev
 21 2018-09-09T05:29:13  *** Krellan has quit IRC
 22 2018-09-09T05:33:40  *** Krellan has joined #bitcoin-core-dev
 23 2018-09-09T05:38:32  *** jhfrontz has quit IRC
 24 2018-09-09T05:40:56  *** pkx1 has joined #bitcoin-core-dev
 25 2018-09-09T05:41:25  *** jhfrontz has joined #bitcoin-core-dev
 26 2018-09-09T05:45:20  *** pkx1 has quit IRC
 27 2018-09-09T05:45:54  *** pkx1 has joined #bitcoin-core-dev
 28 2018-09-09T05:51:01  *** TheHoliestRoger has quit IRC
 29 2018-09-09T05:53:43  *** pkx1 has quit IRC
 30 2018-09-09T05:53:50  *** TheHoliestRoger has joined #bitcoin-core-dev
 31 2018-09-09T05:54:22  *** hebasto has joined #bitcoin-core-dev
 32 2018-09-09T05:59:19  *** TheHoliestRoger has quit IRC
 33 2018-09-09T06:00:52  *** jhfrontz has quit IRC
 34 2018-09-09T06:01:04  *** jhfrontz has joined #bitcoin-core-dev
 35 2018-09-09T06:02:07  *** TheHoliestRoger has joined #bitcoin-core-dev
 36 2018-09-09T06:17:07  *** plankers has joined #bitcoin-core-dev
 37 2018-09-09T06:18:06  *** jhfrontz has quit IRC
 38 2018-09-09T06:27:57  *** hebasto has quit IRC
 39 2018-09-09T06:33:43  *** pkx1 has joined #bitcoin-core-dev
 40 2018-09-09T07:09:49  *** pkx1 has quit IRC
 41 2018-09-09T07:19:32  *** phwalkr has joined #bitcoin-core-dev
 42 2018-09-09T07:28:13  *** phwalkr has quit IRC
 43 2018-09-09T07:35:18  *** Krellan has quit IRC
 44 2018-09-09T07:36:22  *** Krellan has joined #bitcoin-core-dev
 45 2018-09-09T07:43:32  *** plankers has quit IRC
 46 2018-09-09T07:52:27  *** echonaut has joined #bitcoin-core-dev
 47 2018-09-09T07:53:52  *** echonaut10 has quit IRC
 48 2018-09-09T08:16:12  <wumpus> there is a feature for manual pruning, isn't  there?
 49 2018-09-09T08:16:43  <wumpus> I remember something like that was merged at some point, a command line flag to prevent automatic pruning as well as a RPC to prune
 50 2018-09-09T08:17:47  <wumpus> client software could use this to prune only what they no longer require
 51 2018-09-09T08:21:53  <wumpus> (I don't think any lightning daemons do this, at the moment, but would be useful)
 52 2018-09-09T08:37:53  <CubicEarth> wumpus: yeah, sipa was also saying there is manual prune option
 53 2018-09-09T08:39:53  <CubicEarth> so that should be able to be used by lightning nodes to ensure the only blocks pruned are ones they don't need anymore
 54 2018-09-09T08:42:01  *** hebasto has joined #bitcoin-core-dev
 55 2018-09-09T08:42:57  <gmaxwell> the design of manual prune is kinda not ideal if you have multiple potential things that need data. :(
 56 2018-09-09T08:44:47  <CubicEarth> if there was a DoNotPruneAbove, it could be set by different things, with the lowest one controlling
 57 2018-09-09T08:45:30  <gmaxwell> indeed but then you'd need interface elements to remove zombie DNPAs.
 58 2018-09-09T08:45:55  <CubicEarth> Yeah :)
 59 2018-09-09T08:46:09  <CubicEarth> If too many things want the data, not pruning at all might be favorable
 60 2018-09-09T08:50:43  <gmaxwell> it might be more useful if there were just a way of setting it up so you won't process blocks unless your lightning is running, or you've removed the binding.
 61 2018-09-09T08:51:00  <CubicEarth> In either case though, if blocks are to be retained until the pruning command is given by another service, it seems useful to have bitcoin have an option to not download more than some MBs or GBs worth of blocks
 62 2018-09-09T08:51:12  <CubicEarth> ahead
 63 2018-09-09T08:53:23  <CubicEarth> gmaxwell: are we saying the same thing?
 64 2018-09-09T08:59:21  <wumpus> it's not ideal, but nothing is even using the current functionality as far as I know, so I'm kind of reluctant to discuss adding even more
 65 2018-09-09T09:00:12  <wumpus> once it works for the 1 bitcoind - 1 lightningd case, it'd be more fruitful to discuss extending it to other cases
 66 2018-09-09T09:01:16  <wumpus> instead of adding a lot of logic for something that no one will be using in practice
 67 2018-09-09T09:02:43  <wumpus> FWIW "DNPA" could be implemented by an external process using the current RPC interface
 68 2018-09-09T09:02:57  <wumpus> have it manage the list of requirements and manage manual pruning
 69 2018-09-09T09:03:01  <wumpus> accordingly
 70 2018-09-09T09:05:01  <CubicEarth> Yeah. DNPA is mostly the same as manual pruning it seems.
 71 2018-09-09T09:05:13  <CubicEarth> And that makes sense about not adding complexity.
 72 2018-09-09T09:05:55  <CubicEarth> Maybe it would be simple to add a DoNotValidateBeyond ?
 73 2018-09-09T09:06:21  <wumpus> what I mean is that manual pruning provides the required low-level interface, DNPA would be one strategy to manage it at a higher level with multiple data consumers, but likely not the only way (at some point you might want to kill off consumers if they're too slow, for example, depending on disk space etc)
 74 2018-09-09T09:08:15  <CubicEarth> I understood as such ^^ even if my reply suggested otherwise :)
 75 2018-09-09T09:08:23  <wumpus> something like that, though validation is not the problem in this case, block download is - you'd want to tell it to pause downloading new blocks
 76 2018-09-09T09:08:58  <wumpus> I think there's a command that more or less does that, pause networking, but might be GUI-only I don't remember exactly
 77 2018-09-09T09:10:04  *** SopaXorzTaker has joined #bitcoin-core-dev
 78 2018-09-09T09:10:55  <wumpus> "setnetworkactive"
 79 2018-09-09T09:11:04  *** Krellan has quit IRC
 80 2018-09-09T09:12:35  <wumpus> it can continue validating the blocks it already has, that won't fill up disk space
 81 2018-09-09T09:13:13  <wumpus> with manual pruning, it's not the validation that triggers pruning anyhow
 82 2018-09-09T09:15:09  <CubicEarth> hmmm, but block are not downloaded sequentially anymore, right?
 83 2018-09-09T09:16:08  <CubicEarth> or non-sequentially within a sliding window?
 84 2018-09-09T09:16:13  <wumpus> I don't see how that is relevant to this, stopping block download is stopping block download, whether they're downloaded sequentially or not, if you want it to stop filling up disk you want to pause block download
 85 2018-09-09T09:17:49  <wumpus> the blocks it downloads can be non-sequential in a consensus sense, but on disk, they're all simply to the block files in a first come first serve basis
 86 2018-09-09T09:17:56  <wumpus> +appended
 87 2018-09-09T09:19:13  <CubicEarth> I was thinking it wouldn't be as helpful to have downloaded and storing a bunch of blocks, and yet be missing an earlier one, at the point networking was shut off
 88 2018-09-09T09:19:36  <wumpus> why not?
 89 2018-09-09T09:20:02  <CubicEarth> validation couldn't proceed
 90 2018-09-09T09:20:03  <wumpus> that stops filling the disk; once it's turned on again, it will download the rest and connect them
 91 2018-09-09T09:20:09  <wumpus> of course validation cannot proceed
 92 2018-09-09T09:20:20  <wumpus> you're essentially shutting down your node without really shutting it down
 93 2018-09-09T09:20:53  <wumpus> so it will respond to RPC queries to get the blocks and other data already there
 94 2018-09-09T09:21:01  <wumpus> and to pruning commands
 95 2018-09-09T09:21:44  <wumpus> once the clients have caught up, and a pruning command has been issues, the networking can be waken up again, but maybe I'm missing something I don't know...
 96 2018-09-09T09:22:44  <wumpus> and validation will continue as far as it can with the blocks that it had at the time of network shutdown, not further
 97 2018-09-09T09:24:30  <CubicEarth> I need to think about it more (not from the lightning perspective ... I am no lighting expert). What you are describing might be totally workable. I was just thinking if blocks are not downloaded sequentially, and networking was turned off, there will be storage being taken up with blocks that can't be validated due to gaps. I wasn't thinking practically so much. Just in theory, it would be nice to only be storing
 98 2018-09-09T09:24:30  <CubicEarth> what could be validated
 99 2018-09-09T09:25:09  <CubicEarth> Yes, those gaps would be filled as soon as networking was turned back on, and it would proceed.
100 2018-09-09T09:25:46  <wumpus> yes I think it makes sense to get a clear idea of what you practically need, otherwise you'll by definition drag unrelated concerns into it
101 2018-09-09T09:25:59  <wumpus> exactly
102 2018-09-09T09:29:37  <wumpus> from the perspective of the lightning clients they will see the block number stop increasing when validation is blocked, which will negatively affect their usefulness as lightning nodes, channel status updates will no longer be detected
103 2018-09-09T09:31:55  <CubicEarth> the block download window is 1024, btw
104 2018-09-09T09:32:52  <gmaxwell> CubicEarth: no, above I meant a something much simpler. A setting in bitcoind where it just won't run unless your lightning daemon is connected.
105 2018-09-09T09:33:55  <gmaxwell> but as wumpus was saying, they aren't even bothering to use the existing functionality.
106 2018-09-09T09:37:30  <CubicEarth> those lazy lighting devs!
107 2018-09-09T09:39:51  <gmaxwell> more like they only recently graduated to not requiring -txindex...
108 2018-09-09T09:41:55  <jl2012> I have a question about this: https://github.com/bitcoin/bitcoin/blob/master/src/secp256k1/src/scalar_8x32_impl.h#L78
109 2018-09-09T09:42:34  <jl2012> if no becomes true in line 78 to 81, yes must be false
110 2018-09-09T09:42:46  <jl2012> why don't it return the result earlier?
111 2018-09-09T09:44:09  <gmaxwell> Because there is no need to, it might well even be slower to do so.
112 2018-09-09T09:47:25  <gmaxwell> it would also make the function non-constant-time if it terminates early.
113 2018-09-09T09:47:47  <jl2012> oh, I think this is the real reason?
114 2018-09-09T09:48:52  <gmaxwell> if that were the only reason we'd have two versions.
115 2018-09-09T09:49:13  <gmaxwell> But as I said, I wouldn't be surprised if branching out earlier was slower.
116 2018-09-09T09:52:51  <jl2012> thanks!
117 2018-09-09T09:59:28  <wumpus> CubicEarth: please, don't call anyone lazy just because they don't focus on your specific issue, everyone is more like overworked
118 2018-09-09T10:00:36  <wumpus> btw, wow, lightiningd's API to get the last log messages for a specific node is very nice
119 2018-09-09T10:01:50  <wumpus> even works to get log messages at a lower logging level (say, debug) than are written to disk
120 2018-09-09T10:02:29  <gmaxwell> I had aspirations of doing all our logging into a ringbuffer... with disk just taking a feed off it.
121 2018-09-09T10:02:49  <gmaxwell> which would allow us to do things like no logs normally, but dump the ring on crash.
122 2018-09-09T10:02:59  <wumpus> that's what they do, I think, well I don't know if it's implemented as a ringbuffer but it acts that way on the outside
123 2018-09-09T10:03:54  <wumpus> but yes that would be neat to have
124 2018-09-09T10:04:03  *** rex4539 has quit IRC
125 2018-09-09T10:04:10  <wumpus> at least for log messages that don't cost a lot of overhead to format unnecessarily
126 2018-09-09T10:04:23  <CubicEarth> wumpus: you are right. I meant it not seriously, but things don't always come across as intended on irc. Nothing but respect for lightning devs from me!
127 2018-09-09T10:04:37  <wumpus> CubicEarth: okay, sorry for misreading you then
128 2018-09-09T10:05:32  <gmaxwell> wumpus: yea, I wondered about that but IIRC other than the format strings themselves there are only ~two places in the code where we totally skip some processing according the the log flags.
129 2018-09-09T10:05:57  <gmaxwell> and I seem to recall thinking that those could probably just go away.
130 2018-09-09T10:06:24  <wumpus> gmaxwell: agree, my concern i not relevant to many places and there we already skip the processin explicitly, so the ring-buffer thing would just work
131 2018-09-09T10:06:37  <wumpus> (except for those messages, obviously)
132 2018-09-09T10:07:45  <gmaxwell> I think in general, for most users, the logging provides no real value, just wastes disk space even becomes an attack vector (e.g. via spam up the disk attacks). So as a result we've made default logging pretty conservative, but then useful stuff isn't there when something goes wrong.  perhaps more interesting is how any of this would interact with concurrency.
133 2018-09-09T10:12:47  <wumpus> it'd have to synchronize around writes to the ringbuffer, instead of around disk writes like now
134 2018-09-09T10:13:54  <wumpus> there is a PR that moves part of logging to a thread I suddenly remember, #13200, should look into it in more detail
135 2018-09-09T10:13:56  <gribble> https://github.com/bitcoin/bitcoin/issues/13200 | Process logs in a separate thread by jamesob · Pull Request #13200 · bitcoin/bitcoin · GitHub
136 2018-09-09T10:14:10  <gmaxwell> right but if the ringbuffer is getting more logs into it because it logs more things than disk, that might be annoying. Of course we could have a seperate buffer per thread(group) and only take locks for all of them to merge output for display.
137 2018-09-09T10:14:41  *** Krellan has joined #bitcoin-core-dev
138 2018-09-09T10:15:14  <wumpus> yes, fair enough
139 2018-09-09T10:42:16  *** rex4539 has joined #bitcoin-core-dev
140 2018-09-09T11:09:57  *** SopaXorzTaker has quit IRC
141 2018-09-09T11:24:36  *** SopaXorzTaker has joined #bitcoin-core-dev
142 2018-09-09T11:32:24  *** drexl has joined #bitcoin-core-dev
143 2018-09-09T12:17:10  *** timothy has quit IRC
144 2018-09-09T12:19:45  *** Krellan has quit IRC
145 2018-09-09T12:20:36  *** Krellan has joined #bitcoin-core-dev
146 2018-09-09T12:54:06  *** owowo has quit IRC
147 2018-09-09T13:15:56  *** polydin has quit IRC
148 2018-09-09T13:21:57  *** drexl has quit IRC
149 2018-09-09T13:53:37  *** owowo has joined #bitcoin-core-dev
150 2018-09-09T13:57:11  *** jcorgan has joined #bitcoin-core-dev
151 2018-09-09T14:04:16  *** Guyver2 has joined #bitcoin-core-dev
152 2018-09-09T14:23:24  *** belcher_ has joined #bitcoin-core-dev
153 2018-09-09T14:56:57  *** emilengler has joined #bitcoin-core-dev
154 2018-09-09T15:25:47  *** nullptr| has quit IRC
155 2018-09-09T15:31:04  *** nullptr| has joined #bitcoin-core-dev
156 2018-09-09T15:32:31  *** lnostdal has quit IRC
157 2018-09-09T15:34:03  *** lnostdal has joined #bitcoin-core-dev
158 2018-09-09T15:40:21  *** jarthur has joined #bitcoin-core-dev
159 2018-09-09T15:43:17  *** grubles has quit IRC
160 2018-09-09T15:47:29  *** miknotauro has quit IRC
161 2018-09-09T15:56:41  *** Chris_Stewart_5 has joined #bitcoin-core-dev
162 2018-09-09T15:59:39  *** Cheeseo has joined #bitcoin-core-dev
163 2018-09-09T16:03:51  *** emilengler has quit IRC
164 2018-09-09T16:07:06  *** jhfrontz has joined #bitcoin-core-dev
165 2018-09-09T16:07:44  *** hebasto has quit IRC
166 2018-09-09T16:15:56  *** jhfrontz has quit IRC
167 2018-09-09T16:16:43  *** Chris_Stewart_5 has quit IRC
168 2018-09-09T16:49:59  *** promag has joined #bitcoin-core-dev
169 2018-09-09T16:50:07  *** jhfrontz has joined #bitcoin-core-dev
170 2018-09-09T16:50:40  *** Cheeseo has quit IRC
171 2018-09-09T16:52:59  *** jhfrontz has quit IRC
172 2018-09-09T16:56:01  *** jhfrontz has joined #bitcoin-core-dev
173 2018-09-09T16:58:27  *** Guyver2 has quit IRC
174 2018-09-09T17:07:28  *** itaseski has joined #bitcoin-core-dev
175 2018-09-09T17:11:32  *** jhfrontz has quit IRC
176 2018-09-09T17:31:56  *** grubles has joined #bitcoin-core-dev
177 2018-09-09T17:46:50  *** AaronvanW has joined #bitcoin-core-dev
178 2018-09-09T17:48:16  *** jarthur has quit IRC
179 2018-09-09T17:51:31  *** DougieBot5000 has joined #bitcoin-core-dev
180 2018-09-09T17:51:43  *** DougieBot5000_ has quit IRC
181 2018-09-09T17:56:53  *** promag has quit IRC
182 2018-09-09T17:57:27  *** promag has joined #bitcoin-core-dev
183 2018-09-09T17:59:34  *** ExtraCrispy has quit IRC
184 2018-09-09T18:01:45  *** promag has quit IRC
185 2018-09-09T18:10:58  *** owowo has quit IRC
186 2018-09-09T18:11:39  *** DougieBot5000 has quit IRC
187 2018-09-09T18:16:27  *** DougieBot5000 has joined #bitcoin-core-dev
188 2018-09-09T18:29:04  *** owowo has joined #bitcoin-core-dev
189 2018-09-09T18:49:55  *** SopaXorzTaker has quit IRC
190 2018-09-09T20:17:09  *** promag has joined #bitcoin-core-dev
191 2018-09-09T20:29:09  *** Linrono has joined #bitcoin-core-dev
192 2018-09-09T20:30:09  *** promag has quit IRC
193 2018-09-09T21:28:32  <phantomcircuit> it seems kind of awkward to me to use the new naming convention in the same (small) function as the old one
194 2018-09-09T21:30:47  *** promag has joined #bitcoin-core-dev
195 2018-09-09T21:31:18  *** Linrono has quit IRC
196 2018-09-09T21:36:24  <sipa> phantomcircuit: if it's a small function, just change it all
197 2018-09-09T21:38:46  <phantomcircuit> sipa, im looking at https://github.com/bitcoin/bitcoin/blob/718843aed5e60e42ebbe822457511b4fbbf318eb/src/net.cpp#L1225
198 2018-09-09T21:39:01  <phantomcircuit> which is mostly moveonly, changing it would change that
199 2018-09-09T21:39:09  <phantomcircuit> but the nMicroTime variable is new
200 2018-09-09T21:46:35  *** DigiByteDev has joined #bitcoin-core-dev
201 2018-09-09T22:04:39  *** drexl has joined #bitcoin-core-dev
202 2018-09-09T22:17:54  *** spinza has quit IRC
203 2018-09-09T22:26:25  *** spinza has joined #bitcoin-core-dev
204 2018-09-09T22:35:55  *** Zenton has quit IRC
205 2018-09-09T22:43:11  *** AaronvanW has quit IRC
206 2018-09-09T22:43:43  *** AaronvanW has joined #bitcoin-core-dev
207 2018-09-09T22:48:09  *** miknotauro has joined #bitcoin-core-dev
208 2018-09-09T23:13:42  *** Chris_Stewart_5 has joined #bitcoin-core-dev
209 2018-09-09T23:15:42  *** belcher_ has quit IRC
210 2018-09-09T23:19:06  *** profmac has quit IRC
211 2018-09-09T23:33:24  *** profmac has joined #bitcoin-core-dev
212 2018-09-09T23:34:11  *** Aaronvan_ has joined #bitcoin-core-dev
213 2018-09-09T23:37:08  *** AaronvanW has quit IRC
214 2018-09-09T23:38:54  *** profmac has quit IRC
215 2018-09-09T23:41:08  *** Aaronvan_ has quit IRC
216 2018-09-09T23:41:43  *** AaronvanW has joined #bitcoin-core-dev
217 2018-09-09T23:43:51  *** Chris_Stewart_5 has quit IRC
218 2018-09-09T23:45:47  *** AaronvanW has quit IRC
219 2018-09-09T23:48:51  *** promag has quit IRC
220 2018-09-09T23:49:25  *** promag has joined #bitcoin-core-dev
221 2018-09-09T23:53:27  *** promag has quit IRC