1 2015-10-19T00:10:39  <Luke-Jr> gmaxwell: perhaps. optimizing the sorted list would help in multiple places though
  2 2015-10-19T00:13:49  <gmaxwell> as far as optimizing, how about adding one of the satoshi dice addreses to a wallet and rescanning?
  3 2015-10-19T00:14:05  <Luke-Jr> (just the last item will actually change the algorithm btw)
  4 2015-10-19T00:14:30  <gmaxwell> Luke-Jr: sure other optimizations are good too, but AFAIR all that needs to do is track a the extrema.
  5 2015-10-19T00:16:03  <Luke-Jr> gmaxwell: it ignores any transactions timestamped >5m in the future from the local time, which means such txns would have to be reconsidered over time
  6 2015-10-19T00:18:54  <gmaxwell> Luke-Jr: uh. how are they ever going to get learned then?
  7 2015-10-19T00:19:08  <Luke-Jr> gmaxwell: ?
  8 2015-10-19T00:19:24  <Luke-Jr> ignores in the context of looking at other txns timestamps to decide the timestamp for this one
  9 2015-10-19T00:20:09  <phantomcircuit> what's the relative cost of checking an address vs pattern matching the output to start with?
 10 2015-10-19T00:20:11  <gmaxwell> remind me precisely what it does?
 11 2015-10-19T00:20:23  <gmaxwell> phantomcircuit: go read the pr
 12 2015-10-19T00:20:28  <gmaxwell> (the text on the opening)
 13 2015-10-19T00:21:23  <Luke-Jr> gmaxwell: it's complex enough that I'd just be rewriting the code in pseudo-code to explain it :/
 14 2015-10-19T00:21:45  <gmaxwell> Luke-Jr: good well that means it can change behavior and not violate anyone's expectations because apparently no one can understand it. :P
 15 2015-10-19T00:21:59  <Luke-Jr> it's not hard to understand, it's just complex :p
 16 2015-10-19T00:22:10  <phantomcircuit> gmaxwell, i never understood why we even bothered to guess when an unconfirmed transaction happened
 17 2015-10-19T00:22:19  <Luke-Jr> the point is that it meets expectations
 18 2015-10-19T00:22:27  <gmaxwell> phantomcircuit: because we show them in listtransactions
 19 2015-10-19T00:22:56  <phantomcircuit> gmaxwell, "this transaction happened now" seems reasonable for all unconfirmed transactions
 20 2015-10-19T00:23:13  <gmaxwell> phantomcircuit: but then its inconsistent with what you get when it confirms.
 21 2015-10-19T00:23:21  <Luke-Jr> phantomcircuit: no, definitely not when you're syncing a node that was off
 22 2015-10-19T00:23:48  <phantomcircuit> Luke-Jr, how about null
 23 2015-10-19T00:23:50  <gmaxwell> (the listtransaction output gives several different times when you care about details... and a smart time)
 24 2015-10-19T00:24:03  <gmaxwell> phantomcircuit: bet peoples software will handle that super awesomely well.
 25 2015-10-19T00:24:06  <Luke-Jr> phantomcircuit: … that is not useful
 26 2015-10-19T00:24:24  <phantomcircuit> the timestamp we're returning now isn't very useful either
 27 2015-10-19T00:24:31  <gmaxwell> phantomcircuit: It's usually pretty good.
 28 2015-10-19T00:24:58  <gmaxwell> Luke-Jr: I think you might need to write out the algorithim again, we cannot continue with this being slow like this.
 29 2015-10-19T00:25:33  <phantomcircuit> i guess...
 30 2015-10-19T00:26:06  <Luke-Jr> gmaxwell: that's why I'm working on optimizing it.. if we want accounting to keep working, I can deal with that, just would have saved time to omit it
 31 2015-10-19T00:26:55  <Luke-Jr> (this code actually seems to be somewhat broken with accounts, fwiw)
 32 2015-10-19T00:27:37  <gmaxwell> Luke-Jr: I thought smarttime just returned blocktime for confirmed transactions that you didn't observe first?
 33 2015-10-19T00:29:08  <Luke-Jr> not necessarily
 34 2015-10-19T00:29:12  *** Thireus1 has quit IRC
 35 2015-10-19T00:29:50  <gmaxwell> unless the blocktime is in the future, in which case?
 36 2015-10-19T00:30:04  *** Thireus has joined #bitcoin-core-dev
 37 2015-10-19T00:32:27  <phantomcircuit> this seems much smarter than it needs to be...
 38 2015-10-19T00:32:33  <Luke-Jr> it's approximately max(last_txn_time, min(block_time, max(last_txn_time, now)))
 39 2015-10-19T00:32:38  <phantomcircuit> also it seems like all of these values should be cachable
 40 2015-10-19T00:33:17  <gmaxwell> Luke-Jr: than can be simplified to refer only to blocktimes in the context of a rescan.
 41 2015-10-19T00:33:37  <Luke-Jr> gmaxwell: half the goal here is to avoid transactions ever having timestamps out of order
 42 2015-10-19T00:34:03  <gmaxwell> which they cannot, in a rescan.
 43 2015-10-19T00:34:08  <Luke-Jr> uh, why not?
 44 2015-10-19T00:34:19  <Luke-Jr> block times are out of order all the time
 45 2015-10-19T00:34:40  <phantomcircuit> what's the rational behind not setting the transaction timestamp to the block timestamp?
 46 2015-10-19T00:34:52  <gmaxwell> Luke-Jr: as I just said, you can apply that filter to blocktimes only!
 47 2015-10-19T00:35:14  <phantomcircuit> oh you want to avoid the timestamps disagreeing with the nOrdPos
 48 2015-10-19T00:35:15  <gmaxwell> phantomcircuit: becuase they end up out of order which really screws with people.
 49 2015-10-19T00:35:48  <Luke-Jr> gmaxwell: that sounds a lot more complex, and doesn't consider that we can ignore it when blocks don't have any txns of ours
 50 2015-10-19T00:36:16  <gmaxwell> Luke-Jr: this code takes _9.5 hours of processing_ in the requesting users example.
 51 2015-10-19T00:36:23  <gmaxwell> And you are going to complain about complex?
 52 2015-10-19T00:38:14  <gmaxwell> Luke-Jr: having it use some really old time because your last transaction was 50 blocks ago isn't good.
 53 2015-10-19T00:38:22  <Luke-Jr> I find it simpler to just optimise the ordered tx list
 54 2015-10-19T00:38:51  <gmaxwell> e.g. if the latest block is in the future, okay you don't use the time. Then it should use the time of the prior block, even if there was no transaction there, not the time of your last transaction.
 55 2015-10-19T00:39:09  <gmaxwell> at least in the context of a rescan where you do not learn times anywhere except blocks.
 56 2015-10-19T00:39:17  <gmaxwell> I think the current behavior is just busted there.
 57 2015-10-19T00:39:38  <Luke-Jr> gmaxwell: it would use `now` in that case
 58 2015-10-19T00:40:07  <Luke-Jr> in a rescan, block times will basically never be in the future.
 59 2015-10-19T00:40:16  <gmaxwell> yea, unlikely, indeed.
 60 2015-10-19T00:42:05  <Luke-Jr> gmaxwell: will you NACK an optimization that preserves existing behaviour?
 61 2015-10-19T00:42:49  <gmaxwell> if it's measurably slower, yes.  This should not take any time at all. as I said, I would have NAKED smarttimes if I knew it caused a measurable slowdown.
 62 2015-10-19T00:44:18  *** Ylbam has quit IRC
 63 2015-10-19T00:54:33  *** equil_ has joined #bitcoin-core-dev
 64 2015-10-19T01:16:13  *** CodeShark has joined #bitcoin-core-dev
 65 2015-10-19T01:16:31  *** belcher has quit IRC
 66 2015-10-19T01:32:49  *** maaku has quit IRC
 67 2015-10-19T01:33:19  <Luke-Jr> gmaxwell: I *think* it is possible to optimise so it's reasonably close, but in case not, there is a possibility of rescan just skipping smart-time assignment (which currently causes "time" to fall back to received time = rescan time, but maybe that should change?)
 68 2015-10-19T01:34:23  <Luke-Jr> side note: hmm, sync of my node confused BFGMiner as it got notified of old blocks O.o
 69 2015-10-19T01:34:41  <gmaxwell> Luke-Jr: of course, if this function is slow during rescan its also slow during runtime... so I do support speeding it up generally too.
 70 2015-10-19T01:35:00  <gmaxwell> Luke-Jr: would rather fall back to monotoneized blocktime than rescan time.
 71 2015-10-19T01:35:08  <Luke-Jr> well, part of why it was unoptimised is probably because runtime is not time sensitive
 72 2015-10-19T01:36:06  <gmaxwell> Luke-Jr: hope you're not mining from a node with a wallet...
 73 2015-10-19T01:36:34  <gmaxwell> but really having a very big wallet (Even bigger than the ones there) would eventually be disruptive.
 74 2015-10-19T01:36:52  *** maaku has joined #bitcoin-core-dev
 75 2015-10-19T01:37:16  *** maaku is now known as Guest57960
 76 2015-10-19T01:37:30  <Luke-Jr> eh? it's just BFGMiner (running for testing) automatically picking up my hot wallet
 77 2015-10-19T01:38:06  <Luke-Jr> it's using Eligius mainly, but the local node's block disagreements somewhat confused it
 78 2015-10-19T01:38:43  <Luke-Jr> no big deal if mining is negatively impacted by it; maybe a slightly bigger deal if it were to compromise my hot wallet
 79 2015-10-19T01:39:46  <Luke-Jr> but if that latter case is a risk, we should fix it. I'm not aware of such a problem.
 80 2015-10-19T01:43:23  <gmaxwell> I wasn't commenting on your bfgminer comment.
 81 2015-10-19T01:44:28  <Luke-Jr> oh
 82 2015-10-19T01:44:37  <gmaxwell> I'm saying that you shouldn't dismiss that the performance of sorting the whole wallet on an add as being important. As an example, someone who is mining with a wallet cares about a 100ms delay, (which is the ballpark of what that user was reporting).. or very big wallets.
 83 2015-10-19T01:45:34  *** Guest57960 is now known as maaku
 84 2015-10-19T01:48:46  <phantomcircuit> Luke-Jr, cant you just cache the oldest record timestamp?
 85 2015-10-19T01:49:24  <Luke-Jr> phantomcircuit: newest*, but no, because of the 5 minute future rule
 86 2015-10-19T01:50:06  <Luke-Jr> optimising OrderedTxList looks like an easy big win. let me finish that and we can see where we are and if we need more improvements?
 87 2015-10-19T01:59:56  <gmaxwell> Luke-Jr: sounds like something to try.
 88 2015-10-19T02:08:25  <phantomcircuit> bleh the real problem is the rpc api is wrong
 89 2015-10-19T02:08:36  <phantomcircuit> probably cant reasonably fix that though
 90 2015-10-19T02:11:36  <gmaxwell> on my not very huge wallet, the PR on github is a ~1% speedup.
 91 2015-10-19T02:12:14  <gmaxwell> really we're doing something dumb if every wallet isn't approximately the same speed for rescan.
 92 2015-10-19T02:27:51  *** d_t has quit IRC
 93 2015-10-19T02:29:52  *** d_t has joined #bitcoin-core-dev
 94 2015-10-19T02:56:34  <kanzure> rescan was killing all my rpcthreads a few weeks ago and i didn't have any insight into what was going on :-( maybe i'll submit an rpcthread status info command.
 95 2015-10-19T02:56:42  <kanzure> s/killed/pegged? what's the right term
 96 2015-10-19T02:57:30  *** jgarzik has joined #bitcoin-core-dev
 97 2015-10-19T02:57:30  *** jgarzik has joined #bitcoin-core-dev
 98 2015-10-19T03:01:19  *** maaku has quit IRC
 99 2015-10-19T03:02:34  *** maaku has joined #bitcoin-core-dev
100 2015-10-19T03:02:58  *** maaku is now known as Guest67646
101 2015-10-19T03:11:23  *** d_t has quit IRC
102 2015-10-19T03:13:36  *** Guest67646 is now known as maaku
103 2015-10-19T03:13:43  *** d_t has joined #bitcoin-core-dev
104 2015-10-19T04:13:57  *** Thireus has quit IRC
105 2015-10-19T04:16:48  *** Thireus has joined #bitcoin-core-dev
106 2015-10-19T04:30:15  *** Thireus has quit IRC
107 2015-10-19T04:30:48  *** Thireus has joined #bitcoin-core-dev
108 2015-10-19T05:10:23  *** Thireus has quit IRC
109 2015-10-19T05:12:29  *** equil_ has quit IRC
110 2015-10-19T05:35:34  *** Thireus has joined #bitcoin-core-dev
111 2015-10-19T05:38:24  *** dcousens has joined #bitcoin-core-dev
112 2015-10-19T05:38:37  <dcousens> jgarzik: not sure what users would use it tbh
113 2015-10-19T05:38:42  <dcousens> I know I would for an API perspective
114 2015-10-19T05:38:46  <dcousens> So, in my case, automated?
115 2015-10-19T05:39:07  <dcousens> And I guess for sites like webbtc, they lightly dress it up in some readable HTML
116 2015-10-19T05:40:02  <dcousens> hell, if an addrindex existed, a block explorer would solely be a HTML dress up of the bitcoind REST API ha
117 2015-10-19T05:40:27  <jgarzik> dcousens, yeah the 'GET /foo' case is more likely to be used manually, versus PUT/POST
118 2015-10-19T05:40:52  <dcousens> jgarzik: probably a more important question is,  what is the default
119 2015-10-19T05:40:58  <dcousens> that will dictate how easy it is for curl users
120 2015-10-19T05:41:17  <dcousens> And even then, content type isn't that hard for curl users?
121 2015-10-19T05:47:27  <dcousens> I feel like most users might do something along the lines of     'bitcoin-tx ... | curl 'http://somepopularnode.com/tx' --data-raw -H "Content-Type:application/octet-stream"
122 2015-10-19T05:47:52  *** challisto has quit IRC
123 2015-10-19T05:47:54  <dcousens> or application/json if `-json` was used
124 2015-10-19T05:48:24  <dcousens> Probably easiest to just specify the format to whatever the default is, and have `-json` or whatever on bitcoin-tx though
125 2015-10-19T05:52:29  *** paveljanik has joined #bitcoin-core-dev
126 2015-10-19T06:30:37  *** Thireus has quit IRC
127 2015-10-19T06:35:35  *** baldur has quit IRC
128 2015-10-19T06:48:31  *** baldur has joined #bitcoin-core-dev
129 2015-10-19T06:49:28  *** Ylbam has joined #bitcoin-core-dev
130 2015-10-19T06:52:01  <wumpus> kanzure: rescan blocks all your RPC threads, on aquiring cs_main
131 2015-10-19T06:53:10  <wumpus> kanzure: long-running RPC commands tend to block the rest, apart from gettxoutsetinfo which I changed at some point to not hold locks (as it runs on a snapshot of the database)
132 2015-10-19T06:54:34  <wumpus> probably in rescan the lock could be moved down too, and only aquired when block database structures are accessed
133 2015-10-19T06:56:56  *** d_t has quit IRC
134 2015-10-19T07:03:21  *** Thireus has joined #bitcoin-core-dev
135 2015-10-19T07:35:55  *** equil_ has joined #bitcoin-core-dev
136 2015-10-19T08:02:20  *** paveljanik has quit IRC
137 2015-10-19T08:15:51  <Luke-Jr> gmaxwell: caching ordered tx list is faster than removing smart time. good enough? :x
138 2015-10-19T08:16:52  <gmaxwell> Luke-Jr: so what happens when the wallet is huge?
139 2015-10-19T08:17:20  <gmaxwell> then we end up with this gigantic tx list cache in ram. :(
140 2015-10-19T08:17:41  <Luke-Jr> eh, right now we store all the tx anyway..
141 2015-10-19T08:17:53  <Luke-Jr> is storing a list of pointers all that much more?
142 2015-10-19T08:19:48  <gmaxwell> I guess not, probably only another three gigabytes of ram with the whole blockchain in your wallet.
143 2015-10-19T08:20:09  <gmaxwell> I'm not super keen on this whole issue existing though. :-/
144 2015-10-19T08:25:57  <gmaxwell> Re: 6844 I am twitching a bit about more things being added to the rest api, while little effort has been performed to gain strong confidence that it's not a remote code execution vector. IIRC, we were able to crash the node via it when it went in and we addressed that by disabling it by default. I absolutely don't want to crap on 6844 itself, as it looks like a great feature. But we are making
145 2015-10-19T08:26:03  <gmaxwell> an error if we continue to offer a potentially insecure external interface and make it more and more attractive to turn it on and expose it.
146 2015-10-19T08:29:21  <gmaxwell> (keep in mind that even if it is localhost only; a local webbrowser has almost free reign over it.)
147 2015-10-19T08:32:44  <phantomcircuit> whoa what?
148 2015-10-19T08:32:49  <phantomcircuit> you can remotely crash the rest api?
149 2015-10-19T08:35:06  <gmaxwell> phantomcircuit: Could initially. I'm unsure if it's fixed, it's a huge attack surface in any case-- we don't have any fuzzing harness for it either.
150 2015-10-19T08:35:34  <gmaxwell> (the prior crash I was aware of was just send it an infinityish sized request...)
151 2015-10-19T08:55:25  <wumpus> it's relatively safe as long as the JSON parser is not used for input data, and if there is an input argument it is just a simple linear string directly part of the URI, with straightforward validation
152 2015-10-19T08:56:08  <wumpus> the crash was a case where JSON spirit was used to parse input data query, which had a known out of memory crash if you make structures too deep
153 2015-10-19T08:56:51  <gmaxwell> heh. I found a (local) buffer overflow bug in wget while trying to crash it.
154 2015-10-19T08:56:54  <wumpus> IMO REST is for 'trivial, quickly answered, traightforward queries', not where you need to pass in JSON encoded monstriosities
155 2015-10-19T08:57:30  <jonasschnelli> wumpus, yes. Probably a good idea. What about maximum length of URLs in case of submit tx?
156 2015-10-19T08:57:32  <wumpus> gmaxwell: that could still be interesting; some scripts are bound to pass remote input to wget's parameters
157 2015-10-19T08:57:48  <wumpus> jonasschnelli: there definitely should be a limit!
158 2015-10-19T08:58:23  <jonasschnelli> wumpus: right. But it might be, that http clients or proxys are using limits lower then the user might expect.
159 2015-10-19T08:58:35  <jonasschnelli> I think for large tx /GET is not suitable.
160 2015-10-19T08:58:43  * wumpus wishes he had more time to hunt for vulnerabilities in bitcoind's codebase
161 2015-10-19T08:58:49  <jonasschnelli> POST with hex/bin should be supported IMO
162 2015-10-19T08:59:38  <wumpus> using GET to provide input for tx submission is very un-restful
163 2015-10-19T08:59:39  <jonasschnelli> most webservers (and probably proxys) have a limit of 8KB for URL (HTTP GET) length
164 2015-10-19T08:59:54  <wumpus> should be PUT?
165 2015-10-19T09:00:09  <wumpus> or yes, POST
166 2015-10-19T09:00:14  <jonasschnelli> wumpus: agree. PUT or POST would be better. But there comes in the question of supporting JSON... which is a security risk.
167 2015-10-19T09:00:19  <wumpus> no, no JSON
168 2015-10-19T09:00:19  <jonasschnelli> Maybe just support BIN/HEX
169 2015-10-19T09:00:22  <wumpus> just raw POST data
170 2015-10-19T09:00:23  <jonasschnelli> (or only BIN)
171 2015-10-19T09:00:32  <jonasschnelli> wumpus: agree.
172 2015-10-19T09:01:51  <wumpus> JSON  is ok for returning data in some cases, but should not be *parsed* in REST
173 2015-10-19T09:02:06  <Luke-Jr> can we parse plain English language instead?
174 2015-10-19T09:02:07  * Luke-Jr hides
175 2015-10-19T09:02:36  <wumpus> Luke-Jr: http://inform7.com/ ? :p
176 2015-10-19T09:05:16  <gmaxwell> please don't do the rest submittx at all if its only bin, if it won't take hex it will be massively less used, and we'll just have more code to hang around and bloat the codebase; while people will continue to eschew bitcoin core and use some web api that actually gives an interface people expect. :-/
177 2015-10-19T09:05:50  <gmaxwell> okay, I feel better now-- I have a whole bunchof fuzzers running against it at the moment and it's not immediately dying.
178 2015-10-19T09:05:55  <gmaxwell> \O/
179 2015-10-19T09:05:59  <jonasschnelli> gmaxwell: agree. Hex should be supported in the first place.
180 2015-10-19T09:06:52  <wumpus> hex is fine, just don't add a zillion kinds of input formats...
181 2015-10-19T09:07:04  <wumpus> each one adds more attack surface
182 2015-10-19T09:07:05  <Luke-Jr> tonal pls
183 2015-10-19T09:07:13  <wumpus> hah...
184 2015-10-19T09:07:22  <Luke-Jr> (no, tonal isn't used for binary data..)
185 2015-10-19T09:08:29  <Luke-Jr> is -connect=0 the best way to avoid any external interactions with p2p?
186 2015-10-19T09:08:38  <wumpus> -connect=
187 2015-10-19T09:08:46  <Luke-Jr> k
188 2015-10-19T09:09:05  <wumpus> -connect=0 will work too, but will spam your debug.log with errors
189 2015-10-19T09:10:05  <wumpus> but yes I feel bad about adding more and more to the REST interface as well. Initially it was meant to just request data, making it possible to submit transactions unauthenticated is a different thing.
190 2015-10-19T09:10:52  <wumpus> I understand why it's useful of course...
191 2015-10-19T09:12:01  <Luke-Jr> IMO it will get horridly abused
192 2015-10-19T09:12:14  <Luke-Jr> when Eligius offered such a thing, it got hard-coded into spam scripts
193 2015-10-19T09:13:29  <wumpus> you're not *supposed* to offer REST interface to the open internet, but lacking auth it may be easy to do accidentally (same as zillions of other critical systems are accidentally on the internet)
194 2015-10-19T09:13:49  <gmaxwell> I dunno if there is any real reason to do anything but hex.
195 2015-10-19T09:13:49  <gmaxwell> I suppose if block submission was also supported then the hex overhead would be pretty unwelcome.
196 2015-10-19T09:13:49  <gmaxwell> and for that we'd want binary data. so there might be a uniformity argument.
197 2015-10-19T09:13:49  *** gmaxwell has quit IRC
198 2015-10-19T09:13:57  *** gmaxwell has joined #bitcoin-core-dev
199 2015-10-19T09:14:18  *** Guest52664 has quit IRC
200 2015-10-19T09:14:21  *** gmaxwell is now known as Guest89849
201 2015-10-19T09:14:34  *** petertodd has joined #bitcoin-core-dev
202 2015-10-19T09:14:57  *** petertodd is now known as Guest82934
203 2015-10-19T09:15:13  <wumpus> binary is the least risky to accept, I don't think there's anything to be won by not accepting that
204 2015-10-19T09:15:49  <wumpus> then again I don't really care, if you want to make it hex-only, be my guest, just don't add JSON parsing
205 2015-10-19T09:16:18  *** BlueMatt has quit IRC
206 2015-10-19T09:16:47  <wumpus> (or any encoding more involved than straightforward encode-two-characters-to-one like hex)
207 2015-10-19T09:20:51  *** BlueMatt has joined #bitcoin-core-dev
208 2015-10-19T09:24:17  <GitHub21> [bitcoin] luke-jr opened pull request #6851: Optimisation: Store transaction list order in memory rather than compute it every need (master...opti_txorder) https://github.com/bitcoin/bitcoin/pull/6851
209 2015-10-19T09:26:43  <wumpus> Luke-Jr: why is -zapwallettxes performance a goal?
210 2015-10-19T09:27:20  <wumpus> isn't that a troubleshooting option that should be quite rarely used?
211 2015-10-19T09:28:43  <Luke-Jr> wumpus: I don't know, gmaxwell was annoyed it was taking some guy with a big wallet 9 hours. Apparently it's more common because we don't handle malleability right still.
212 2015-10-19T09:29:15  <Luke-Jr> wumpus: but right now, it sorts the entire wallet's transactions every time it adds a new transaction to the wallet. so it's pretty bad.
213 2015-10-19T09:29:47  <GitHub48> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/d78a880900c1...5888046323dd
214 2015-10-19T09:29:47  <GitHub48> bitcoin/master d2e3fbc Thomas Kerin: Mention PHP bindings to libbitcoinconsensus
215 2015-10-19T09:30:19  <GitHub48> bitcoin/master 5888046 Wladimir J. van der Laan: Merge pull request #6849...
216 2015-10-19T09:30:19  <GitHub130> [bitcoin] laanwj closed pull request #6849: Mention PHP bindings to libbitcoinconsensus (master...bitcoinconsensus-php-bindings) https://github.com/bitcoin/bitcoin/pull/6849
217 2015-10-19T09:30:53  <wumpus> we had a RPC pull request at some point where it was possible to remove conflicted transactions one-by-one, I greatly preferred that to the hammer that is zapwallettxes
218 2015-10-19T09:31:55  <wumpus> it's pretty bad but does it warrant using more memory even in the cases where people don't use zapwallettxes?
219 2015-10-19T09:32:23  <wumpus> I don't think so - memory usage of bitcoind is already famously bad
220 2015-10-19T09:32:49  <Luke-Jr> for the wallet? I've only heard complaints when people fail to configure their mempool policy
221 2015-10-19T09:33:09  <wumpus> sorry to be grumpy about this but there's so many people complaining about that
222 2015-10-19T09:33:15  <wumpus> well the entire wallet is kept in memory at alll times
223 2015-10-19T09:34:33  <Luke-Jr> yes, that's why I didn't think the overhead from this would be significant ;)
224 2015-10-19T09:35:32  <Luke-Jr> (my hot wallet has been in regularish use since 2009 and hasn't reached 40 MB yet..)
225 2015-10-19T09:35:39  <wumpus> i don't like "it's already bad, so making it any worse doesn't matter" arguments, but ok...
226 2015-10-19T09:35:39  <Luke-Jr> 2010*
227 2015-10-19T09:36:33  <dcousens> IMHO, I'd argue the REST API shouldn't exist at all
228 2015-10-19T09:36:39  <dcousens> Its trivial to expose it via the RPC if you want to do that
229 2015-10-19T09:37:56  <wumpus> that was my argument too; it could just as well have been an external script that wraps RPC and proides a new API
230 2015-10-19T09:37:56  <dcousens> For the same reasons Guest89849 (gmaxwell) mentioned before, its just a huge surface vector for limited gain
231 2015-10-19T09:38:07  <wumpus> but who listens to me...
232 2015-10-19T09:38:12  <Luke-Jr> wumpus: the benefits IMO significantly outweigh the costs. someone would need to be running some major service provider to use a lot of memory for the wallet, and that use case is just out of reach for bitcoind's wallet without any kind of major rewrite
233 2015-10-19T09:38:36  <wumpus> not everything has to be in the monolithic bitcoind executable
234 2015-10-19T09:38:36  <Luke-Jr> even aside from zap, it improves performance for listtransactions RPC etc
235 2015-10-19T09:42:00  <wumpus> Luke-Jr: oh if it helps perfomance in other places too you should definitely mention that!
236 2015-10-19T09:42:17  <dcousens> wumpus: Luke-Jr has mentioned a few times now people writing their own mempool policy, which is fine, and maybe including stricter default policies might be slightly controversial (too restrictive etc),  but,  maybe some pre-written policies that users can easily enable that have some sane defaults would be beneficial?
237 2015-10-19T09:42:18  <wumpus> Luke-Jr: I wasn't aware, and just 'it improves zapwallettx' is not a big selling point to me
238 2015-10-19T09:43:36  <Luke-Jr> k, added it to the summary. looks like it's *just* listtransactions
239 2015-10-19T09:43:44  <wumpus> dcousens: if someone wants to write and test them, I don't see why not, there are some pulls that factor out the mempool policy stuff which makes it easier to create custom ones, at least in theory
240 2015-10-19T09:44:02  <dcousens> wumpus: I mean, I guess that would be the ideal case right?
241 2015-10-19T09:44:13  <dcousens> Everyone configuring their policy to what they are happy with
242 2015-10-19T09:44:29  <wumpus> dcousens: the ideal case - from the network view - is that everyone runs the same, predictable policy
243 2015-10-19T09:44:41  *** paveljanik has joined #bitcoin-core-dev
244 2015-10-19T09:44:57  <dcousens> wumpus: true, but varying hardware requirements mean that is difficult to cater for
245 2015-10-19T09:45:13  <dcousens> So, I guess, you could go the highly configurable policy route
246 2015-10-19T09:45:30  <wumpus> dcousens: if people insist on using their own mempool policy on their miner there's no way to prevent that, and arguably it shouldn't be made more difficult than necessary to prevent stupid issues from patching
247 2015-10-19T09:45:42  <dcousens> Or what seems to be the current direction, scalable policy which matches your hardware (if configured)
248 2015-10-19T09:45:52  <wumpus> dcousens: but I'm not sure it makes sense to talk about the ideal case there
249 2015-10-19T09:46:04  <wumpus> how does hardware determine the policy? just how much memory available?
250 2015-10-19T09:46:14  <dcousens> wumpus: yes
251 2015-10-19T09:46:27  <wumpus> if so, adding a parameter to set the amount of memory used (with as little impact otherwise) would be preferable
252 2015-10-19T09:46:43  <Luke-Jr> wumpus: everyone running the same predictable policy necessarily implies centralisation and censorship risk
253 2015-10-19T09:46:46  <wumpus> that's the whole point of the mempool limiting
254 2015-10-19T09:47:05  <wumpus> add limiting but don't mess up the network too much or introduce DDoSes
255 2015-10-19T09:47:38  <wumpus> I don't see the point of 'highly configurable policy' just to work around hardware limitations
256 2015-10-19T09:47:43  <dcousens> Luke-Jr: only if the default policy is restrictive right?
257 2015-10-19T09:48:30  <Luke-Jr> dcousens: which it must be
258 2015-10-19T09:48:43  <dcousens> Luke-Jr: or do you mean, encouraging people just to use the bundled policy is the centralizatoin risk?
259 2015-10-19T09:48:53  <dcousens> (as it seemed you implied to me last night)
260 2015-10-19T09:49:00  <wumpus> Luke-Jr: censorship risk? that's a strawman, I'm talking a about a *sane* default policy, not some blacklist hellscape
261 2015-10-19T09:49:01  <Luke-Jr> dcousens: I was responding to wumpus saying ideal is everyone on the same policy.
262 2015-10-19T09:49:52  <wumpus> if bitcoin core starts to include code that censors specific transactions/senders, start running very fast
263 2015-10-19T09:49:53  <Luke-Jr> wumpus: a unified policy means there is someone making the decisions
264 2015-10-19T09:50:18  <dcousens> Luke-Jr: you could say the same thing about a unified protocol
265 2015-10-19T09:51:34  *** zveda has joined #bitcoin-core-dev
266 2015-10-19T09:51:49  *** zveda has left #bitcoin-core-dev
267 2015-10-19T09:56:35  <wumpus> right, it is about encouraging people to use the same or similar policy for relaying, there is no mechanism to force nodes to use a certain policy. And if someone powerful wanted to censor it'd be much more effective to approach the (heavily centralized) miners than trying to get it merged upstream.
268 2015-10-19T10:01:27  <GitHub166> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/5888046323dd...fc41278e9689
269 2015-10-19T10:01:27  <GitHub166> bitcoin/master 99963b9 Mitchell Cash: Correct spelling mistakes in doc folder...
270 2015-10-19T10:01:28  <GitHub166> bitcoin/master fc41278 Wladimir J. van der Laan: Merge pull request #6843...
271 2015-10-19T10:01:37  <GitHub177> [bitcoin] laanwj closed pull request #6843: Trivial: Correct spelling mistakes in doc folder (master...spelling_fixes) https://github.com/bitcoin/bitcoin/pull/6843
272 2015-10-19T10:13:26  <GitHub163> [bitcoin] laanwj opened pull request #6852: build: make sure OpenSSL heeds noexecstack (master...2015_10_openssl_noexecstack) https://github.com/bitcoin/bitcoin/pull/6852
273 2015-10-19T10:15:50  <GitHub98> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/fc41278e9689...5f5ee0f3ce6b
274 2015-10-19T10:15:51  <GitHub98> bitcoin/master 0658d0b Alex Morcos: temporarily comment out rpcbind.py...
275 2015-10-19T10:15:51  <GitHub98> bitcoin/master 5f5ee0f Wladimir J. van der Laan: Merge pull request #6837...
276 2015-10-19T10:16:00  <GitHub134> [bitcoin] laanwj closed pull request #6837: Cleanup rpc-tests.py (master...fixupRPCtests) https://github.com/bitcoin/bitcoin/pull/6837
277 2015-10-19T10:44:08  <GitHub199> [bitcoin] laanwj closed pull request #6162: Use 512-bit arithmetic for difficulty retarget calculation. (master...regtest_consensus_fix) https://github.com/bitcoin/bitcoin/pull/6162
278 2015-10-19T10:48:02  *** murr4y has joined #bitcoin-core-dev
279 2015-10-19T10:54:11  *** equil_ has quit IRC
280 2015-10-19T10:56:42  *** Thireus has quit IRC
281 2015-10-19T10:56:56  *** Thireus has joined #bitcoin-core-dev
282 2015-10-19T11:20:30  *** randy-waterhouse has joined #bitcoin-core-dev
283 2015-10-19T11:58:56  *** ParadoxSpiral has joined #bitcoin-core-dev
284 2015-10-19T12:13:53  *** randy-waterhouse has quit IRC
285 2015-10-19T12:26:36  <GitHub124> [bitcoin] CodeShark opened pull request #6853: Added fNoRetargeting field to Consensus::Params (master...fNoRetargeting) https://github.com/bitcoin/bitcoin/pull/6853
286 2015-10-19T12:39:20  <CodeShark> why are the travis tests failing on windows? https://travis-ci.org/bitcoin/bitcoin/jobs/86161511
287 2015-10-19T12:40:31  <btcdrak> Trolling for reviews/ACKs please https://github.com/bitcoin/bitcoin/pull/6566 (Median past locktime)
288 2015-10-19T12:44:53  *** malte has quit IRC
289 2015-10-19T12:47:09  *** malte has joined #bitcoin-core-dev
290 2015-10-19T13:01:56  <GitHub52> [bitcoin] laanwj opened pull request #6854: devtools: Add security-check.py (master...2015_10_security_checks) https://github.com/bitcoin/bitcoin/pull/6854
291 2015-10-19T13:03:21  <GitHub166> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/5f5ee0f3ce6b...da7d57fb9501
292 2015-10-19T13:03:21  <GitHub166> bitcoin/master bfcdc21 Wladimir J. van der Laan: build: make sure OpenSSL heeds noexecstack...
293 2015-10-19T13:03:22  <GitHub166> bitcoin/master da7d57f Wladimir J. van der Laan: Merge pull request #6852...
294 2015-10-19T13:03:27  <GitHub39> [bitcoin] laanwj closed pull request #6852: build: make sure OpenSSL heeds noexecstack (master...2015_10_openssl_noexecstack) https://github.com/bitcoin/bitcoin/pull/6852
295 2015-10-19T13:05:11  <GitHub136> [bitcoin] laanwj pushed 1 new commit to 0.11: https://github.com/bitcoin/bitcoin/commit/21e58b8e3544987db0bdaa91c23bb398e655bd21
296 2015-10-19T13:05:12  <GitHub136> bitcoin/0.11 21e58b8 Wladimir J. van der Laan: build: make sure OpenSSL heeds noexecstack...
297 2015-10-19T13:05:49  <GitHub70> [bitcoin] laanwj pushed 1 new commit to 0.10: https://github.com/bitcoin/bitcoin/commit/0b3fd07fd24a96b205d4f85d00157d5f115fad9e
298 2015-10-19T13:05:50  <GitHub70> bitcoin/0.10 0b3fd07 Wladimir J. van der Laan: build: make sure OpenSSL heeds noexecstack...
299 2015-10-19T13:09:46  <kanzure> wumpus: for rescan locking, would be nice to show lock status in another rpc command.
300 2015-10-19T13:09:53  <kanzure> lock statuses per thread or something
301 2015-10-19T13:20:24  <wumpus> meh, I don't think debugging information belongs on RPC
302 2015-10-19T13:20:54  <wumpus> at least not something that low level - can't gdb show locking debug information?
303 2015-10-19T13:22:24  <wumpus> I guess it could be something explicitly enabled with --enable-debug like DEBUG_LOGCONTENTION
304 2015-10-19T13:22:32  <wumpus> LOCK*
305 2015-10-19T13:27:41  <GitHub47> [bitcoin] laanwj closed pull request #6836: Bring historical release notes up to date (0.10...0.10-add-release-notes) https://github.com/bitcoin/bitcoin/pull/6836
306 2015-10-19T13:27:41  <GitHub169> [bitcoin] laanwj pushed 2 new commits to 0.10: https://github.com/bitcoin/bitcoin/compare/0b3fd07fd24a...7e9a9874f3cc
307 2015-10-19T13:27:41  <GitHub169> bitcoin/0.10 fb818b6 Micha: Bring historical release notes up to date...
308 2015-10-19T13:27:42  <GitHub169> bitcoin/0.10 7e9a987 Wladimir J. van der Laan: Merge pull request #6836...
309 2015-10-19T13:29:35  *** dcousens has quit IRC
310 2015-10-19T14:00:23  *** jgarzik has quit IRC
311 2015-10-19T14:00:38  *** jgarzik has joined #bitcoin-core-dev
312 2015-10-19T14:00:38  *** jgarzik has joined #bitcoin-core-dev
313 2015-10-19T14:10:08  *** Guest82934 is now known as petertodd
314 2015-10-19T14:20:45  *** randy-waterhouse has joined #bitcoin-core-dev
315 2015-10-19T14:23:27  <wumpus> the more I think about it the more I think we should have implemented REST as a separate script that wraps the RPC interface... I really don't like how another interface to the same things is converging more and more on RPC's functionality
316 2015-10-19T14:24:51  <btcdrak> ^
317 2015-10-19T14:25:25  <btcdrak> wumpus: could still be changed that way now
318 2015-10-19T14:26:05  <wumpus> yes
319 2015-10-19T14:28:58  <wumpus> with libevent's http, and univalue-as-as-library it shouldn't be that difficult at all
320 2015-10-19T14:31:04  <wumpus> (even to do it in C wouldn't, I mean, in python it's obviously easy)
321 2015-10-19T14:32:53  *** treehug88 has joined #bitcoin-core-dev
322 2015-10-19T14:37:01  <btcdrak> wumpus: so really, if you want this to be done you'll have to refuse to merge new duplicate REST methods until the wrapper is done.
323 2015-10-19T14:37:32  <wumpus> if I want it to be done I'll have to do it myself...
324 2015-10-19T15:05:06  *** Thireus has quit IRC
325 2015-10-19T15:14:02  *** paveljanik has quit IRC
326 2015-10-19T15:14:19  *** d_t has joined #bitcoin-core-dev
327 2015-10-19T15:15:35  *** malte has quit IRC
328 2015-10-19T15:21:41  *** malte has joined #bitcoin-core-dev
329 2015-10-19T15:22:18  *** Thireus has joined #bitcoin-core-dev
330 2015-10-19T16:00:01  *** d_t has quit IRC
331 2015-10-19T16:06:14  *** d_t has joined #bitcoin-core-dev
332 2015-10-19T16:18:32  *** Guest89849 is now known as gmaxwell
333 2015-10-19T16:18:33  *** gmaxwell has joined #bitcoin-core-dev
334 2015-10-19T16:23:26  <GitHub73> [bitcoin] lclc opened pull request #6855: [REST] API versioning (master...restVersioning) https://github.com/bitcoin/bitcoin/pull/6855
335 2015-10-19T16:28:24  *** sdaftuar has quit IRC
336 2015-10-19T16:29:01  *** jcorgan has quit IRC
337 2015-10-19T16:29:02  *** isis has quit IRC
338 2015-10-19T16:29:37  *** petertodd has quit IRC
339 2015-10-19T16:29:37  *** kanzure has quit IRC
340 2015-10-19T16:30:08  *** sdaftuar has joined #bitcoin-core-dev
341 2015-10-19T16:30:53  *** petertodd has joined #bitcoin-core-dev
342 2015-10-19T16:31:03  *** kanzure has joined #bitcoin-core-dev
343 2015-10-19T16:31:17  *** petertodd is now known as Guest98736
344 2015-10-19T16:35:01  *** jcorgan has joined #bitcoin-core-dev
345 2015-10-19T16:35:02  *** jcorgan has joined #bitcoin-core-dev
346 2015-10-19T16:37:52  *** isis has joined #bitcoin-core-dev
347 2015-10-19T16:55:19  *** maaku has quit IRC
348 2015-10-19T16:56:01  <gmaxwell> wumpus: I like that too.
349 2015-10-19T16:56:44  *** maaku has joined #bitcoin-core-dev
350 2015-10-19T16:57:07  *** maaku is now known as Guest17159
351 2015-10-19T17:33:31  *** paveljanik has joined #bitcoin-core-dev
352 2015-10-19T17:33:33  *** paveljanik has joined #bitcoin-core-dev
353 2015-10-19T17:53:58  *** Guest17159 is now known as maaku
354 2015-10-19T18:29:09  *** maaku has quit IRC
355 2015-10-19T18:30:32  *** maaku has joined #bitcoin-core-dev
356 2015-10-19T18:30:52  *** maaku is now known as Guest13301
357 2015-10-19T18:31:21  *** Guest13301 is now known as maaku
358 2015-10-19T19:28:59  *** CodeShark has quit IRC
359 2015-10-19T19:32:09  *** belcher has joined #bitcoin-core-dev
360 2015-10-19T19:38:40  <GitHub130> [bitcoin] morcos opened pull request #6856: Do not allow block file pruning during reindex. (master...noPruneDuringReindex) https://github.com/bitcoin/bitcoin/pull/6856
361 2015-10-19T19:41:10  *** maaku__ has joined #bitcoin-core-dev
362 2015-10-19T19:44:23  <GitHub123> [bitcoin] morcos opened pull request #6857: Require nLastBlockFile to be the highest numbered file. (master...nLastBlockFile) https://github.com/bitcoin/bitcoin/pull/6857
363 2015-10-19T20:08:24  *** randy-waterhouse has quit IRC
364 2015-10-19T20:16:12  *** ParadoxSpiral has quit IRC
365 2015-10-19T20:49:00  *** paveljanik has quit IRC
366 2015-10-19T21:07:05  *** midnightmagic has quit IRC
367 2015-10-19T21:10:48  *** Thireus has quit IRC
368 2015-10-19T21:12:58  *** midnightmagic has joined #bitcoin-core-dev
369 2015-10-19T21:31:07  *** treehug88 has quit IRC
370 2015-10-19T21:40:46  *** midnightmagic_ has joined #bitcoin-core-dev
371 2015-10-19T21:43:48  *** midnightmagic has quit IRC
372 2015-10-19T21:44:06  *** midnightmagic_ is now known as midnightmagic
373 2015-10-19T23:54:18  *** Ylbam has quit IRC