1 2015-11-14T00:06:51  *** squidicuz has quit IRC
  2 2015-11-14T00:24:22  <gmaxwell> wumpus: jonasschnelli:  Care to give me a quick design sniff test?  I'm thinking of having someone implement functionality for external signers in Bitcoin core along these lines, http://0bin.net/paste/7HCkLWNJBDqJB7Up#hPgdwfs06m7FJw-WQZUCBNTNfCZnrix2K0Mo++6jpbH
  3 2015-11-14T00:25:05  <gmaxwell> I realize that some might want a more extensive wallet redesign, but I think this would be good incremental progress and would enable a _lot_ of flexibility for minimal code impact.
  4 2015-11-14T00:29:14  *** Squidicuz has joined #bitcoin-core-dev
  5 2015-11-14T01:13:20  *** JackH has quit IRC
  6 2015-11-14T01:45:32  *** d_t has quit IRC
  7 2015-11-14T01:53:33  *** ParadoxSpiral has quit IRC
  8 2015-11-14T01:54:27  *** Ylbam has quit IRC
  9 2015-11-14T02:16:52  *** dgenr8 has quit IRC
 10 2015-11-14T02:21:39  *** randy-waterhouse has joined #bitcoin-core-dev
 11 2015-11-14T03:52:53  *** PaulCape_ has joined #bitcoin-core-dev
 12 2015-11-14T03:55:51  *** PaulCapestany has quit IRC
 13 2015-11-14T04:00:26  *** randy-waterhouse has quit IRC
 14 2015-11-14T04:00:26  *** dcousens has joined #bitcoin-core-dev
 15 2015-11-14T04:33:29  *** PaulCape_ has quit IRC
 16 2015-11-14T04:33:50  *** PaulCapestany has joined #bitcoin-core-dev
 17 2015-11-14T04:37:57  *** d_t has joined #bitcoin-core-dev
 18 2015-11-14T05:00:14  *** tulip has joined #bitcoin-core-dev
 19 2015-11-14T05:09:43  *** kanzure_ has joined #bitcoin-core-dev
 20 2015-11-14T05:09:45  *** nkuttler_ has joined #bitcoin-core-dev
 21 2015-11-14T05:10:26  *** petertod1 has joined #bitcoin-core-dev
 22 2015-11-14T05:10:52  *** tripleslash_l has joined #bitcoin-core-dev
 23 2015-11-14T05:14:25  *** kanzure has quit IRC
 24 2015-11-14T05:14:25  *** tripleslash has quit IRC
 25 2015-11-14T05:14:26  *** nkuttler has quit IRC
 26 2015-11-14T05:14:26  *** bsm117532 has quit IRC
 27 2015-11-14T05:14:27  *** petertodd has quit IRC
 28 2015-11-14T05:14:30  *** nkuttler_ is now known as nkuttler
 29 2015-11-14T05:15:37  *** bsm117532 has joined #bitcoin-core-dev
 30 2015-11-14T05:17:50  <GitHub147> [bitcoin] gmaxwell pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/dbd2c135ddb9...e0a5ef84272b
 31 2015-11-14T05:17:51  <GitHub147> bitcoin/master 61e1eb2 Peter Todd: Actually use includeWatching value in fundrawtransaction...
 32 2015-11-14T05:17:51  <GitHub147> bitcoin/master 10953a7 Peter Todd: Better error message for fundrawtransaction w/ empty vout...
 33 2015-11-14T05:17:52  <GitHub147> bitcoin/master e0a5ef8 Gregory Maxwell: Merge pull request #7010...
 34 2015-11-14T05:18:00  <GitHub67> [bitcoin] gmaxwell closed pull request #7010: Fix fundrawtransaction handling of includeWatching (master...2015-11-fix-fundrawtransaction-bugs) https://github.com/bitcoin/bitcoin/pull/7010
 35 2015-11-14T05:31:59  *** jtimon has quit IRC
 36 2015-11-14T05:35:21  *** dcousens has quit IRC
 37 2015-11-14T05:48:08  *** luke-jr_ has joined #bitcoin-core-dev
 38 2015-11-14T05:49:15  *** pmienk_ has joined #bitcoin-core-dev
 39 2015-11-14T05:50:51  *** michagogo_ has joined #bitcoin-core-dev
 40 2015-11-14T05:51:02  *** nkuttler_ has joined #bitcoin-core-dev
 41 2015-11-14T05:51:03  *** michagogo has quit IRC
 42 2015-11-14T05:51:04  *** lclc has quit IRC
 43 2015-11-14T05:51:05  *** teward has quit IRC
 44 2015-11-14T05:51:06  *** berndj has quit IRC
 45 2015-11-14T05:51:06  *** baldur has quit IRC
 46 2015-11-14T05:51:08  *** phantomcircuit has quit IRC
 47 2015-11-14T05:51:08  *** Luke-Jr has quit IRC
 48 2015-11-14T05:51:09  *** pmienk has quit IRC
 49 2015-11-14T05:51:11  *** zmanian_ has quit IRC
 50 2015-11-14T05:51:11  *** warren has quit IRC
 51 2015-11-14T05:51:14  *** midnightmagic has quit IRC
 52 2015-11-14T05:51:15  *** nkuttler has quit IRC
 53 2015-11-14T05:51:16  *** nkuttler_ is now known as nkuttler
 54 2015-11-14T05:51:43  *** berndj has joined #bitcoin-core-dev
 55 2015-11-14T05:52:14  *** zmanian_ has joined #bitcoin-core-dev
 56 2015-11-14T05:52:45  *** midnightmagic has joined #bitcoin-core-dev
 57 2015-11-14T05:52:47  *** warren has joined #bitcoin-core-dev
 58 2015-11-14T05:52:58  *** michagogo_ is now known as michagogo
 59 2015-11-14T05:53:08  *** lclc has joined #bitcoin-core-dev
 60 2015-11-14T05:53:08  *** baldur has joined #bitcoin-core-dev
 61 2015-11-14T05:54:39  *** phantomcircuit has joined #bitcoin-core-dev
 62 2015-11-14T05:57:17  *** teward has joined #bitcoin-core-dev
 63 2015-11-14T05:59:38  <phantomcircuit> gmaxwell, thoughts on a "-miner" mode that does things like reduce (or even remove) the sleep calls in some of the networking threads?
 64 2015-11-14T05:59:56  <phantomcircuit> (ie please do everything possible to reduce latency even if you spin my cpu at 100% constantly)
 65 2015-11-14T06:07:53  <gmaxwell> phantomcircuit: what sleep calls are there at all in networking? (other than in e.g. connect loops, which are needed to avoid dos attacking peers)
 66 2015-11-14T06:08:41  *** luke-jr_ is now known as Luke-Jr
 67 2015-11-14T06:09:22  <Luke-Jr> we probably should have a miner mode regardless. relay nodes have no real need for the mempool.
 68 2015-11-14T06:09:38  <Luke-Jr> (they just need to relay and forget, plus some txid record so they can't be DoS'd)
 69 2015-11-14T06:10:42  <phantomcircuit> Luke-Jr, please define relay node more specifically
 70 2015-11-14T06:12:00  *** d_t has quit IRC
 71 2015-11-14T06:12:06  <Luke-Jr> phantomcircuit: a node that wishes to participate (non-leech) in the p2p network, but has no miners
 72 2015-11-14T06:12:37  <phantomcircuit> Luke-Jr, they require a mempool if they are going to forward transactions otherwise they become a DoS amplifier
 73 2015-11-14T06:12:42  <phantomcircuit> gmaxwell, there's a sleep on select() error (im not even sure why that's there actually)
 74 2015-11-14T06:12:58  <Luke-Jr> phantomcircuit: why can't they just remember txids?
 75 2015-11-14T06:13:36  <phantomcircuit> and there's messageHandlerCondition which can be replaced by a spin lock
 76 2015-11-14T06:14:09  <phantomcircuit> there's also a bunch of other things like changing the defaults for various things to be much much larger
 77 2015-11-14T06:14:39  <gmaxwell> Luke-Jr: forgetting things is not so useful for avoiding the 2x++ redundant transmission when the same data shows up again in a block.
 78 2015-11-14T06:15:55  <phantomcircuit> gmaxwell, i'd also add a background thread that just calls CreateNewBlock in a loop trying to improve on the current cached block template
 79 2015-11-14T06:16:16  <Luke-Jr> gmaxwell: hm, that's a point
 80 2015-11-14T06:16:39  <Luke-Jr> phantomcircuit: merely calling CNB won't use/update the cache
 81 2015-11-14T06:16:53  <phantomcircuit> Luke-Jr, yes i know
 82 2015-11-14T06:17:09  * phantomcircuit hand waves details
 83 2015-11-14T06:24:41  <phantomcircuit> Luke-Jr, it'll update every 5 seconds if the memory pool was modified, correct?
 84 2015-11-14T06:24:52  <phantomcircuit> (or the tip changes obviously)
 85 2015-11-14T06:25:27  <Luke-Jr> phantomcircuit: the cache is in the RPC code
 86 2015-11-14T06:25:35  <phantomcircuit> yes i know
 87 2015-11-14T06:25:55  <phantomcircuit> that's the logic though right? (just verifying i read it correctly)
 88 2015-11-14T06:27:01  <gmaxwell> phantomcircuit: I was recommending before that we change the gbt behavior to _never_ run createnewblock at the time the rpc is called.
 89 2015-11-14T06:27:30  <phantomcircuit> gmaxwell, yes that's what i would be implementing, a background thread only
 90 2015-11-14T06:29:07  <phantomcircuit> there's some interesting optimizations that can be done if we're willing to spend a bunch of cpu time on it
 91 2015-11-14T06:30:02  <phantomcircuit> the obvious one is optimizing for the block that pays the highest fee
 92 2015-11-14T06:30:10  <phantomcircuit> (or whatever)
 93 2015-11-14T06:44:01  <wumpus> gmaxwell: concept ACK
 94 2015-11-14T06:44:35  <wumpus> gmaxwell:  I'd personally prefer not adding more 'invoke external programs', I know we already have walletnotify etc but it makes sandboxing the thing harder because we can't disable fork/exec
 95 2015-11-14T06:45:24  <wumpus> gmaxwell: I know there is walletnotify, blocknotify etc but at least we have an alternative notification mechanism for that now, so they may be deprecated in some future version
 96 2015-11-14T06:51:17  <wumpus> gmaxwell: anyhow so I'm not strongly against that, just wary of it, also with regard to argument shell injection. Although at least that an be easily avoided by executing a program directly instead of using ::system
 97 2015-11-14T06:58:29  <wumpus> an alternative would be to use a little protocol intead for communicating with the signer
 98 2015-11-14T07:00:09  <gmaxwell> wumpus: hm. we can limit to what we can exec (via seccomp) .. interesting I was thinking the seperate process there was a sandboxing benefit, e.g. because it can have totally distinct selinux / seccomp settings.
 99 2015-11-14T07:00:26  <wumpus> sure, just be really careful
100 2015-11-14T07:00:46  <gmaxwell> But the biggest reason for the seperate process is to tear down development barriers. E.g. so someone can build a module without the scarryness in bitcoin core. :)
101 2015-11-14T07:00:46  <wumpus> walletnotify/blocknotify use ::system that has always bugged me
102 2015-11-14T07:01:12  <wumpus> sure that's why I dont give "integrate it into bitcoin core" as an alternative, but say communicate through a pipe or socket
103 2015-11-14T07:01:50  <gmaxwell> wumpus: still need a way to start the thing that speaks the protocol.
104 2015-11-14T07:02:05  <wumpus> could be similar to e.g. torcontrol.cpp which speaks a simple protocol with Tor
105 2015-11-14T07:02:10  <gmaxwell> But okay, reasonable.
106 2015-11-14T07:02:12  <wumpus> true
107 2015-11-14T07:02:18  <wumpus> ok, let's leave that isue for later
108 2015-11-14T07:02:47  *** CodeShark_ has joined #bitcoin-core-dev
109 2015-11-14T07:02:55  <wumpus> for initial version invoking a process is fine
110 2015-11-14T07:03:11  <wumpus> just don't use ::system :D
111 2015-11-14T07:03:48  *** pigeons has joined #bitcoin-core-dev
112 2015-11-14T07:03:55  *** Guest78100 has quit IRC
113 2015-11-14T07:03:59  *** zxzzt has quit IRC
114 2015-11-14T07:04:01  *** da2ce7 has quit IRC
115 2015-11-14T07:04:12  *** pigeons is now known as Guest25458
116 2015-11-14T07:04:39  *** zxzzt has joined #bitcoin-core-dev
117 2015-11-14T07:05:44  *** da2ce7 has joined #bitcoin-core-dev
118 2015-11-14T07:24:50  <GitHub73> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/e0a5ef84272b...36baa9f47587
119 2015-11-14T07:24:51  <GitHub73> bitcoin/master b3ae384 Peter Todd: Remove LOCK(cs_main) from decodescript...
120 2015-11-14T07:24:51  <GitHub73> bitcoin/master 36baa9f Wladimir J. van der Laan: Merge pull request #7013...
121 2015-11-14T07:24:55  <GitHub81> [bitcoin] laanwj closed pull request #7013: Remove LOCK(cs_main) from decodescript (master...2015-11-remove-cs-main-from-decodescript) https://github.com/bitcoin/bitcoin/pull/7013
122 2015-11-14T07:26:15  <GitHub151> [bitcoin] laanwj pushed 1 new commit to master: https://github.com/bitcoin/bitcoin/commit/44ac42e50d567b08d3cb3f3c5766588468ce5bbf
123 2015-11-14T07:26:16  <GitHub151> bitcoin/master 44ac42e Wladimir J. van der Laan: Merge pull request #7004...
124 2015-11-14T07:26:20  <GitHub79> [bitcoin] laanwj closed pull request #7004: update jonasschnellis gpg key (master...2015/11/signing_key_jonasschnelli) https://github.com/bitcoin/bitcoin/pull/7004
125 2015-11-14T07:47:52  *** PaulCape_ has joined #bitcoin-core-dev
126 2015-11-14T07:50:42  *** PaulCapestany has quit IRC
127 2015-11-14T07:55:04  <GitHub18> [bitcoin] gmaxwell pushed 4 new commits to master: https://github.com/bitcoin/bitcoin/compare/44ac42e50d56...9ffc687288dd
128 2015-11-14T07:55:05  <GitHub18> bitcoin/master d61fcff Jonas Schnelli: don't enforce maxuploadtargets disconnect for whitelisted peers
129 2015-11-14T07:55:05  <GitHub18> bitcoin/master 5760749 Jonas Schnelli: [docs] rename reducetraffic.md to reduce-traffic.md
130 2015-11-14T07:55:06  <GitHub18> bitcoin/master e495ed5 Jonas Schnelli: add documentation for exluding whitelistes peer from maxuploadtarget
131 2015-11-14T07:55:09  <GitHub44> [bitcoin] gmaxwell closed pull request #6984: don't enforce maxuploadtarget's disconnect for whitelisted peers (master...2015/11/maxupload_whitebind) https://github.com/bitcoin/bitcoin/pull/6984
132 2015-11-14T08:01:37  *** Ylbam has joined #bitcoin-core-dev
133 2015-11-14T08:05:56  *** paveljanik has joined #bitcoin-core-dev
134 2015-11-14T08:05:56  *** paveljanik has joined #bitcoin-core-dev
135 2015-11-14T08:07:45  <gmaxwell> wumpus: do you have any advice for phantomcircuit on my DEFAULT_BLOCKSONLY complaint?  My response to him was not terribly constructive... in part because I don't have any good advice.
136 2015-11-14T08:08:22  <gmaxwell> I'd have merged it already but for that issue.
137 2015-11-14T08:10:30  <wumpus> what is the issue exactly? can't find it in the comments
138 2015-11-14T08:11:01  <wumpus> I do see DEFAULT_BLOCKSONLY is not actually used (besides documentation)
139 2015-11-14T08:11:14  <wumpus> but what prevents him from doing that?
140 2015-11-14T08:11:17  <gmaxwell> oh he adds a DEFAULT_BLOCKSONLY, but then doesn't use it in the getargs because some of them are in net.cpp.
141 2015-11-14T08:11:38  <wumpus> move the constant to net.h?
142 2015-11-14T08:11:46  <gmaxwell> And he would pass it as an argument but apparently pushversion is called in a blinking constructor.
143 2015-11-14T08:12:00  * gmaxwell checks if he can do that easily
144 2015-11-14T08:12:21  <wumpus> in general the constants should be defined in the header of the cpp they are used in
145 2015-11-14T08:12:52  <wumpus> if they are used in multiple places that could be annoying, but at least init.cpp already inclused net.h so that's not a problem
146 2015-11-14T08:12:58  <gmaxwell> :) I'd assumed there was a reason he couldn't do that, but looking now.
147 2015-11-14T08:13:51  <gmaxwell> yup he can. one is in main.cpp but it also pulls in net.h.
148 2015-11-14T08:13:52  <wumpus> otherwise define a global bool and assign that in init.cpp, instead of calling GetBoolArg every time, we do that for more settings (especially those used in inner loops where the overhead of argument-parsing-every-time hurts)
149 2015-11-14T08:13:57  <wumpus> ok!
150 2015-11-14T08:14:09  <gmaxwell> okay thanks for the cluestick. I'd assumed that he used them someplace that didn't have net.h but I didn't look.
151 2015-11-14T08:25:35  <gmaxwell> sipa: someone was seeking feedback from you here: https://github.com/bitcoin/bitcoin/pull/6844
152 2015-11-14T08:35:39  *** dcousens has joined #bitcoin-core-dev
153 2015-11-14T08:46:04  *** PaulCape_ has quit IRC
154 2015-11-14T08:46:30  *** PaulCapestany has joined #bitcoin-core-dev
155 2015-11-14T09:22:46  *** moli has joined #bitcoin-core-dev
156 2015-11-14T09:26:41  *** molly has quit IRC
157 2015-11-14T09:39:57  *** molly has joined #bitcoin-core-dev
158 2015-11-14T09:44:14  *** moli has quit IRC
159 2015-11-14T09:49:01  <wumpus> interesting, so this seccomp filter is a little program (in a restricted instruction set) that is uploaded to the kernel that checks system calls and arguments and returns whether allowed or not
160 2015-11-14T09:50:01  <wumpus> never expected it to be so dynamic
161 2015-11-14T09:51:48  <wumpus> and it should be possible to set it as non-privileged user, given that PR_SET_NO_NEW_PRIVS is set first to avoid tricking set-uid executables with it
162 2015-11-14T09:54:07  <wumpus> sounds ActuallyUseful, should do some experiments with it some time...
163 2015-11-14T09:55:45  <gmaxwell> Yes. It is. I tried to do it for bitcoin core eons ago, but the utility library was GPLed... they since changed the licensing.
164 2015-11-14T09:55:47  <wumpus> (somehow always avoided use of seccomp because I assumed you'd need root to set them, like chroot etc)
165 2015-11-14T09:56:10  <wumpus> cjdns uses it without utility library
166 2015-11-14T09:56:55  <gmaxwell> nope. The thing that makes me most sad right now is walletbackup and the import thing, since they require arbritary file access. But I'd like to replace them with little utility programs, which we'd let ourselves exec and talk to over pipes.
167 2015-11-14T09:57:01  <wumpus> (but cjdns does recommend starting it as root, which was one of the things that put the idea in my head, but they need it for tun setup)
168 2015-11-14T09:57:29  <wumpus> (as well as other jail functionality, which, unfortunately, does need root)
169 2015-11-14T09:58:26  <wumpus> well my idea at least at first would be to have a restricted, secure mode, that disabled calls like that (and also walletnotify/blocknotify and everything that calls external processes)
170 2015-11-14T09:58:45  <gmaxwell> ACK
171 2015-11-14T10:00:25  <wumpus> but you're right about walletbackup/import, I would prefer if they just dumped/retrieved their data over HTTP instead of making files
172 2015-11-14T10:01:21  <wumpus> within the JSON RPC framework this is difficult to solve, but outside that w/ using the http server directly it'd be pretty easy to stream instead
173 2015-11-14T10:05:25  *** Madars has joined #bitcoin-core-dev
174 2015-11-14T10:06:59  <gmaxwell> I think there is some utility for triggering local backups, at least. But instead of arbritary file access it could simply be making it write out dated files; without giving the rpc caller huge freedom.
175 2015-11-14T10:17:11  *** btcdrak_ has joined #bitcoin-core-dev
176 2015-11-14T10:17:15  *** Arnavion has quit IRC
177 2015-11-14T10:17:19  *** Arnavion3 has joined #bitcoin-core-dev
178 2015-11-14T10:17:23  *** Arnavion3 is now known as Arnavion
179 2015-11-14T10:17:50  *** jgarzik_ has joined #bitcoin-core-dev
180 2015-11-14T10:19:10  *** tripleslash has joined #bitcoin-core-dev
181 2015-11-14T10:20:08  *** Thireus1 has joined #bitcoin-core-dev
182 2015-11-14T10:20:11  *** midnightmagic_ has joined #bitcoin-core-dev
183 2015-11-14T10:20:12  *** sipa_ has joined #bitcoin-core-dev
184 2015-11-14T10:20:29  *** tripleslash_l has quit IRC
185 2015-11-14T10:20:29  *** btcdrak has quit IRC
186 2015-11-14T10:20:32  *** midnightmagic has quit IRC
187 2015-11-14T10:20:35  *** zarathustra has quit IRC
188 2015-11-14T10:20:38  *** Thireus has quit IRC
189 2015-11-14T10:20:38  *** jgarzik has quit IRC
190 2015-11-14T10:20:39  *** sipa has quit IRC
191 2015-11-14T10:20:47  *** btcdrak_ is now known as btcdrak
192 2015-11-14T10:30:12  <wumpus> one thing that would make sense is to write to the datadir only
193 2015-11-14T10:30:29  <wumpus> it also proves that the user using the calls has access to it
194 2015-11-14T10:31:01  <wumpus> on the other hand, of course people want to backup to mounted filesystems etc
195 2015-11-14T10:31:44  *** ParadoxSpiral has joined #bitcoin-core-dev
196 2015-11-14T10:32:08  <wumpus> hence I'd really prefer it to stream from/to http so that the client can decide where to put it or take it from, anywhere *they* have access to
197 2015-11-14T10:33:20  <wumpus> this avoids using bitcoind in a confused deputy attack (e.g have it write /etc/passwd :-) )
198 2015-11-14T10:34:18  <wumpus> (or read, for that matter, although you'd be hard pressed to find a file that it likes to read and interpret)
199 2015-11-14T10:34:33  <wumpus> (except maybe *other* backups it happens to have access to)
200 2015-11-14T10:43:32  *** midnightmagic_ is now known as midnightmagic
201 2015-11-14T10:50:30  *** tulip has quit IRC
202 2015-11-14T10:53:07  *** zarathustra has joined #bitcoin-core-dev
203 2015-11-14T10:58:01  <gmaxwell> IMO if you want to backup to a mounted FS you can either symlink it into the datadir, or have something external copy there. but ::shrugs::
204 2015-11-14T11:08:06  *** tulip has joined #bitcoin-core-dev
205 2015-11-14T11:26:09  *** jtimon has joined #bitcoin-core-dev
206 2015-11-14T11:32:53  <morcos> phantomcircuit: I wrote a prototype separate thread for running CreateNewBlock, but the naive approach turned out much worse than the existing implementation.
207 2015-11-14T11:33:37  <morcos> The issue was that the template creation code kept needing to hold cs_main, so you ended up holding that lock all the time and holding up things like block relay.
208 2015-11-14T11:34:14  <morcos> Actually it makes a lot of sense to be sure you are not holding that lock at all when a new block comes in so you can validate it as quickly as possilbe and decide whether you should now be building off that
209 2015-11-14T11:34:58  <morcos> Of course redesigning the template generation code to not need all the locking is probably what we need to do, but it turned out to be a much easier win just to speed up how long it takes
210 2015-11-14T11:35:39  <morcos> It's really a mempool lock it needs but thats unfortunately cs_main partly.
211 2015-11-14T11:36:32  <morcos> sipa and I discussed some ideas on having a transaction storage object, which the mempool and the mining code could separately refer to.
212 2015-11-14T11:36:41  <gmaxwell> a while back matt was also just suggesting that CNB could terminate early if there was contention for the lock from a new block coming in.
213 2015-11-14T11:37:03  <gmaxwell> which sounds a little hackish, but probably the kind of prudent hack that would work pretty well.
214 2015-11-14T11:37:12  <gmaxwell> "Look, we've invented cooperative multitasking!"
215 2015-11-14T11:37:14  <morcos> unfortunately everything uses cs_main though
216 2015-11-14T11:37:31  <morcos> the hold createnewblock would have constantly been holding up ATMP by 3 secs or more
217 2015-11-14T11:37:42  <morcos> old
218 2015-11-14T11:37:57  <gmaxwell> yea, not at all a replacement for making it not embarassingly slow.
219 2015-11-14T11:38:05  <morcos> i'm not saying its not fixable, but its a bit of work
220 2015-11-14T11:38:19  *** CodeShark_ has quit IRC
221 2015-11-14T11:38:27  <gmaxwell> ::nods::
222 2015-11-14T11:38:34  <morcos> also the other issue is all the package stucture that sdaftuar put into the mempool is potentially really important for mining
223 2015-11-14T11:38:47  <morcos> and so we have to decide whether it makes sense to duplicate all that code
224 2015-11-14T11:38:54  <morcos> should their be two mempools?
225 2015-11-14T11:39:03  <morcos> that are each dynamically updated
226 2015-11-14T11:39:17  <morcos> should the mining code just copy all it cares about from the real mempool and then do its work
227 2015-11-14T11:39:46  <morcos> or perhaps my favorite approach is to fine grain the locks a little better, and then have a mode where txs can queue up to enter the mempool
228 2015-11-14T11:40:04  <morcos> so you can manipulate the actual mempool in the mining code (i think it'll be relatively fast)
229 2015-11-14T11:40:27  <morcos> but anyway, properly sorting out the locks is a precursor to any of it
230 2015-11-14T11:40:55  <morcos> btw, i'm a bit confused by your 30 secs of the same work for mining hardware
231 2015-11-14T11:41:14  <morcos> i didn't analyze too closely, but the antminer seemed to be sending new work much more often than that, or at least considering doing so
232 2015-11-14T11:43:52  <gmaxwell> All modern devices have internal queing and pipelines so they don't stall, some are unfortunately pretty deep.
233 2015-11-14T11:45:43  <morcos> will the pool software that phantomcircuit suggested use the longpool feature of gbt, or is that not what most people do
234 2015-11-14T11:46:14  <morcos> constantly polling getblockcount and getblockhash seems pretty silly (what cgminer does)
235 2015-11-14T11:46:21  <gmaxwell> the 30 second rule of thumb is an amalgmation of total delays, in pratice;-- including most pool server software being fairly slow. as far as hardware goes I'm not sure what the antminer s7 is like; S1 was very good (I beat on them pre-release); S2 was pretty terrible (and completely unusable with p2pool which has 30 second blocks); I think I heard S5 was better than S2, I haven't heard about S7
236 2015-11-14T11:46:27  <gmaxwell> .  KNC devices have quite high latency (seconds to respond to new work in my expirence),  SP10/SP20 have quite low latency.
237 2015-11-14T11:47:03  <gmaxwell> ckpool uses blocknotify, many things (including p2pool) talk to bitcoind on the p2p port.  Eloipool will use GBT longpool I think, as well as p2p port.
238 2015-11-14T11:47:48  <morcos> shocking that there are so many different mining software implementations, when its essentially consensus critical too
239 2015-11-14T11:47:53  <gmaxwell> Even when things long poll they can take a really long time to get it out to the devices due to 'reasons'. (such as almost no one ever actually measuring)
240 2015-11-14T11:48:09  <morcos> and do the big miners use the existing implementations, or have custom code.   and who maintains all those things
241 2015-11-14T11:48:14  <morcos> sorry for the barrage of ?'s
242 2015-11-14T11:48:16  <gmaxwell> morcos: well it's armored by bitcoin nodes in and out; not that there haven't been problems.
243 2015-11-14T11:49:39  <gmaxwell> morcos: so historically mining pools have mostly been custom code; sometimes hacked on top of parts openly available.  Historically, public pools spend a lot of their development brainpower on dealing with DOS attacks; then data management for payout computation..
244 2015-11-14T11:50:32  <morcos> are "pools" still what dominate mining or is it mostly single entities using the pool infrastructure b/c thats what exists
245 2015-11-14T11:51:34  <gmaxwell> So there are still big public pools but many of them have their own hashpower (either directly or in commonly owned partner companies). The exact breakdown of things is unclear.
246 2015-11-14T11:53:03  <gmaxwell> There are, of course, big entities now and they have simpler mining challenges (e.g. not so much having to worry about DOS).  They sometimes use existing public pools. (I mean some large entities are always on public pools; some only use them sometimes for various reasons)
247 2015-11-14T11:54:02  <gmaxwell> Mining ecosystem is weird; as you observe there are a bazillion seperate implementations of this or that... and then no one goes and makes createnewblock fast.
248 2015-11-14T11:54:17  <morcos> yeah, thats what i was wondering about
249 2015-11-14T11:55:00  <morcos> but i suppose for our purposes (not that it would lead to a different design goal) what we should envision is i guess a p2pool implementation
250 2015-11-14T11:55:09  <gmaxwell> Only 'mining pool' people who've really been active in core are luke-jr and phantomcircuit (who used to run the gear for a big-mining entitiy before working at BS).  And mostly, even for these guys the ideal strategy is to work around limitations; because doing so eas easier/safer.
251 2015-11-14T11:55:14  <morcos> i mean the whole point is to make the small miner able to compete?
252 2015-11-14T11:56:20  <gmaxwell> E.g. eloipool (the mining pool software luke-jr wrote that runs eligius) compensates for CNB latency by constructing empty blocks on its own, until CNB responds. And most mining operations just run multiple bitcoinds: one for GBT and several others that they submit through.
253 2015-11-14T11:56:28  <tulip> morcos: many of the larger pools appear to run completely custom software, for the most part their quirks are completely unique. many of the smaller ones now use ckpool, which seems to be for the most part significantly faster than everybody else.
254 2015-11-14T11:57:37  <morcos> tulip: thanks.  i'm trying to narrow the universe of pool/mining software to explore
255 2015-11-14T11:59:03  <tulip> morcos: ckpool, eloipool, p2pool, NOMP
256 2015-11-14T11:59:18  <gmaxwell> P2pool is nice idea with cute features, but has always struggled with the high startup cost of having to run a bitcoin node with it; and then high latency asics showed up (in particular, some of the early BFL products had 10%+ hashrate loss from 10 second retasking)
257 2015-11-14T12:00:14  <gmaxwell> and that pretty much killed it there, and when it wasn't yet dead enough; somewhat later antpool announced that they'd be converting to p2pool based (but with a friendly front end where they run it for you) and lots of p2pool users moved over, then they didn't actually do it.
258 2015-11-14T12:00:52  <gmaxwell> so now p2pool is too low of a hashrate for anyone who is variance twitchy to use. :(
259 2015-11-14T12:01:03  <morcos> oh no, thats sad
260 2015-11-14T12:01:51  <gmaxwell> Also through this time the developer burned out on Bitcoin, and only does life support maintaince on p2pool now.
261 2015-11-14T12:02:16  <gmaxwell> at the moment P2Pool's hashrate is 918 TH, says my local p2pool node.
262 2015-11-14T12:02:31  <gmaxwell> (it's pretty neat, has a built in webserver and graphs and such)
263 2015-11-14T12:03:58  <gmaxwell> (Forrestv seems to have flamed out due to a mixture of bitcoin drama, and then donations to support p2pool being relatively low (compared to life changing amounts of income centeralized pool operators were getting), and then he got ripped off ... twice, I think, by mining hardware makers.)
264 2015-11-14T12:07:40  <tulip> morcos: there's also stratum proxies like bfgminer and stratum-mining-proxy which present work to downstream miners, based on either an upstream work server or a Bitcoin node. the latter was meant for use with early hardware miners which supported getwork but not stratum.
265 2015-11-14T12:10:09  <gmaxwell> then of course there is all these mining devices with embedded mips and arm linux systems that run usually very old versions of cgminer (or sometimes bfgminer)
266 2015-11-14T12:24:50  <gmaxwell> morcos: while mentioning p2pool I was looking at my stats, and the 1 year view on GBT latency is amusing: https://people.xiph.org/~greg/p2pool-gbt.png
267 2015-11-14T12:43:11  *** tulip has quit IRC
268 2015-11-14T13:06:06  <midnightmagic> sigh
269 2015-11-14T13:23:27  <GitHub86> [bitcoin] gmaxwell pushed 9 new commits to master: https://github.com/bitcoin/bitcoin/compare/9ffc687288dd...b632145edeb3
270 2015-11-14T13:23:27  <GitHub86> bitcoin/master 4044f07 Patick Strateman: Add blocksonly mode
271 2015-11-14T13:23:28  <GitHub86> bitcoin/master 420fa81 Patick Strateman: Do not process tx inv's in blocksonly mode
272 2015-11-14T13:23:29  <GitHub86> bitcoin/master 3a96497 Patick Strateman: Add whitelistalwaysrelay option
273 2015-11-14T13:23:37  <GitHub168> [bitcoin] gmaxwell closed pull request #6993: Add -blocksonly option (master...blocksonly) https://github.com/bitcoin/bitcoin/pull/6993
274 2015-11-14T13:28:27  <phantomcircuit> morcos, generally speaking.. yes i dont care that cnb is slow because it's mostly irrelevant
275 2015-11-14T13:28:52  <phantomcircuit> and i really dont care that it holds a lock for ages since blocks are submitted to nodes dedicated for that
276 2015-11-14T13:29:01  <gmaxwell> it's much less irrelevant if you're not assuming a big miner farm.
277 2015-11-14T13:29:14  <phantomcircuit> gmaxwell, true
278 2015-11-14T13:30:21  <morcos> phantomcircuit: i thinking having getblocktemplate and createnewblock quickly return a valid block after a new tip is important to everyone
279 2015-11-14T13:30:37  <morcos> even better if it is with txs, but not critical
280 2015-11-14T13:31:00  <phantomcircuit> morcos, well making it return *something* immediately afterwards would certainly be nice
281 2015-11-14T13:31:14  <phantomcircuit> that would reduce the need for some of the comical hacks
282 2015-11-14T13:31:41  <phantomcircuit> but as gmaxwell said most of the hardware has queues that are very long and dont flush
283 2015-11-14T13:31:54  <phantomcircuit> (there's good reason for this but i wont get into that)
284 2015-11-14T13:32:07  <morcos> right so the way i look at it is assuming we don't want to code up validationless mining (which i am not yet convinced of)  then we need 100ish ms to validate the new tip and about 100ish ms to generate a new block with txs (with the new code)
285 2015-11-14T13:32:48  <morcos> so at most we could carve out the second 100ms, but at this point we've gotten 95% of the improvement, so maybe not worth doing that unless we are going all the way
286 2015-11-14T13:32:52  <phantomcircuit> morcos, 100ms to return an empty but validated block template would be better :P
287 2015-11-14T13:33:38  <morcos> and before gmaxwell reaches through the tubes to strangle me, my primary argument for considering doing validationless mining is that if people are going to implement it anyway, we might as well make it as safe as possible
288 2015-11-14T13:34:37  <morcos> but i agree that if there are other bottlenecks to switching work, then its not something we can do...   (validationless block header then 100-200ms later the validated version, if you can't switch after 200ms, then its not safe to start on the unvalidated version)
289 2015-11-14T13:35:44  <morcos> but that same argument applies much less importantly to empty blocks.  if you're not going to switch after 100ms to the block with txs...  then you probably dont' want to start with the empty block...   depends on the amount of fees and the minimum switching delay i guess
290 2015-11-14T13:35:52  <gmaxwell> If we want to implement validationless mining, then we need to do it right; which would mean doing something like signaling in the block header what we haven't verified prev, and so SPV clients should ignore the block for conf count purposes.
291 2015-11-14T13:36:04  <gmaxwell> But I'd rather get validation lower first. :)
292 2015-11-14T13:36:30  <phantomcircuit> morcos, there's basically no bottleneck in the validation
293 2015-11-14T13:37:11  <morcos> gmaxwell: sure agreed.   phantomcircuit: no bottleneck? i'm talking about latency in sending new work to hardware outside of bitcoind
294 2015-11-14T13:37:43  <morcos> i'd guess that a significant chunk of the remaining 100ms is allocation (in the non degenerate block case)
295 2015-11-14T13:38:15  <phantomcircuit> morcos, yes... the issue is almost entirely not in validation but in selecting transactions
296 2015-11-14T13:38:20  <gmaxwell> it's an interesting point that the extra 100ms doesn't matter. But rather, I think getting the 100ms upfront matters, and then it doesn't really matter if CNB is slow (except for lock holding reasons)
297 2015-11-14T13:38:30  <phantomcircuit> the mempool indexing + limiting mostly fixes that
298 2015-11-14T13:38:37  <morcos> phantomcircuit: huh? no thats backwards.  selecting txs is blazingly fast.  (well until cpfp)
299 2015-11-14T13:39:01  <morcos> oh you're talking about in master
300 2015-11-14T13:39:59  <morcos> yeah sure.. i'm talking about the remaining 100ms...  which is 10ms tx selection and 90 ms validation.   tx selection is still only 20ms if you scan an entire 300mb mempool to also sort by priority (with a fast priority calc)
301 2015-11-14T13:40:46  <phantomcircuit> morcos, im not (and i dont think anybody paying attention) is worried about validation times of 100ms
302 2015-11-14T13:41:14  <morcos> i see 100 ms, and i think the units still need to change.  :)
303 2015-11-14T13:41:19  <phantomcircuit> that's like 0.01% orphan rate
304 2015-11-14T13:41:42  <phantomcircuit> yes well faster is always better
305 2015-11-14T13:42:03  <gmaxwell> hah. I think 100ms is really slow, but people clearly don't mind that much. But they don't mind in part because they perform hacks; some of which are harmful and we'd like them to stop. :)
306 2015-11-14T13:42:17  <morcos> actually we are forgetting to measure some things there
307 2015-11-14T13:42:36  <morcos> i'm measuring from receive block to new tip = 100ms  and new tip to new template = 100ms
308 2015-11-14T13:42:36  <gmaxwell> when I say they don't mind that much; a year ago there were two places in the network handling code that just interted 100ms _sleeps_.
309 2015-11-14T13:42:48  <morcos> but we also need to worry about receive most work header to receive block
310 2015-11-14T13:43:05  <morcos> that's probably what we need to optimize now
311 2015-11-14T13:43:42  <phantomcircuit> yes the push head instead of inv patch is a huge win for that
312 2015-11-14T13:43:50  <morcos> or even someone else genarates new tip to receiving most work header.  direct headers will help with that
313 2015-11-14T13:44:22  <gmaxwell> yes, well we can remove a one way delay by nominating your favorite peer to relay uninvited; beyond that needs something like matt's relay protocol. (or better)
314 2015-11-14T13:44:40  <morcos> yeah i need to look into how the relay client works...  if you generate a new tip... how do you communicate that to the relay network, same as p2p
315 2015-11-14T13:44:48  * phantomcircuit looks at brain clock *wow so late* goes to sleep
316 2015-11-14T13:45:13  <gmaxwell> morcos: the relay network has its own protocol, and a local proxy you run that speaks bitcoin p2p.
317 2015-11-14T13:45:32  <morcos> oh i suppose if you submitted to a node thats not mining, then it doesn't have the problem of that immediate cs_main lock from gbt after you generate a new tim
318 2015-11-14T13:45:42  <morcos> tip
319 2015-11-14T13:45:54  <gmaxwell> yea, thats why the earlier mentioned architecture of multiple nodes.
320 2015-11-14T13:46:57  <gmaxwell> The relay protocol is super simple. It can send lose transactions, and keeps a history of the X sent. When you want to send a block it sends the header and a list of two byte indexes into that transaction history, as well as any history-miss transactions.
321 2015-11-14T13:47:22  <gmaxwell> other end reconstructs the reblock and passes it on to bitcond unsolicited.
322 2015-11-14T13:48:28  <gmaxwell> his server side, passes all transactions through a local bitcoind, and only relays what it spits out.. but for blocks it checks work and relays them on without more then minimal verification.
323 2015-11-14T13:50:07  <gmaxwell> This really simple approach frequently managed to turn 1MB blocks into 3.6KB. http://bitcoinrelaynetwork.org/stats.html  and leaves matt either fighting sha256 speed ... or mempool spam that breaks his hitrates.
324 2015-11-14T13:57:49  <GitHub101> [bitcoin] gmaxwell opened pull request #7016: Avoid a compile error on hosts with libevent too old for EVENT_LOG_WARN. (master...without_EVENT_LOG_WARN) https://github.com/bitcoin/bitcoin/pull/7016
325 2015-11-14T14:19:27  *** zarathustra has quit IRC
326 2015-11-14T15:13:33  *** zarathustra has joined #bitcoin-core-dev
327 2015-11-14T15:25:16  *** Guyver2 has joined #bitcoin-core-dev
328 2015-11-14T15:26:21  *** kanzure_ is now known as kanzure
329 2015-11-14T16:23:27  *** bsm117532 has quit IRC
330 2015-11-14T16:27:11  *** jgarzik_ has quit IRC
331 2015-11-14T16:27:11  *** jgarzik_ has joined #bitcoin-core-dev
332 2015-11-14T16:27:14  *** jgarzik_ is now known as jgarzik
333 2015-11-14T16:30:34  *** sipa_ is now known as sipa
334 2015-11-14T16:34:10  *** vev__ has joined #bitcoin-core-dev
335 2015-11-14T16:34:13  <vev__> we need people to support us #libreidea
336 2015-11-14T16:39:17  *** vev__ has left #bitcoin-core-dev
337 2015-11-14T16:52:06  *** jl2012 has quit IRC
338 2015-11-14T16:52:28  *** jl2012 has joined #bitcoin-core-dev
339 2015-11-14T17:31:17  *** zooko has joined #bitcoin-core-dev
340 2015-11-14T18:25:40  *** lightningbot has joined #bitcoin-core-dev
341 2015-11-14T18:28:55  *** Thireus1 has quit IRC
342 2015-11-14T18:31:13  *** Thireus has joined #bitcoin-core-dev
343 2015-11-14T18:34:29  *** d_t has joined #bitcoin-core-dev
344 2015-11-14T18:40:31  *** jl2012 has quit IRC
345 2015-11-14T18:58:19  *** zooko has quit IRC
346 2015-11-14T19:03:22  <GitHub50> [bitcoin] morcos closed pull request #6292: Rename and comment priority calculation in TxMemPoolEntry (master...PriorityComment) https://github.com/bitcoin/bitcoin/pull/6292
347 2015-11-14T19:06:04  <morcos> Can someone tag #6134 with 0.12 milestone.
348 2015-11-14T19:13:14  <morcos> I also think #6915 should go in for 0.12.
349 2015-11-14T19:17:43  *** jl2012 has joined #bitcoin-core-dev
350 2015-11-14T19:21:40  *** zooko has joined #bitcoin-core-dev
351 2015-11-14T19:42:26  *** dgenr8 has joined #bitcoin-core-dev
352 2015-11-14T19:44:38  *** zooko has quit IRC
353 2015-11-14T19:50:36  *** zooko has joined #bitcoin-core-dev
354 2015-11-14T20:26:17  *** d_t has quit IRC
355 2015-11-14T20:27:42  *** zarathustra has quit IRC
356 2015-11-14T20:27:42  *** zarathustra has joined #bitcoin-core-dev
357 2015-11-14T21:39:22  *** PRab has quit IRC
358 2015-11-14T21:42:59  *** andytoshi has quit IRC
359 2015-11-14T21:42:59  *** andytoshi has joined #bitcoin-core-dev
360 2015-11-14T21:43:46  <GitHub136> [bitcoin] MarcoFalke opened pull request #7019: [trivial] travis: cover *receivedby* rpcs (master...MarcoFalke-2015-receivedby) https://github.com/bitcoin/bitcoin/pull/7019
361 2015-11-14T21:46:51  *** PaulCape_ has joined #bitcoin-core-dev
362 2015-11-14T21:49:26  *** PaulCapestany has quit IRC
363 2015-11-14T21:57:30  *** zooko has quit IRC
364 2015-11-14T22:03:22  *** Guyver2 has quit IRC
365 2015-11-14T22:05:25  *** petertod1 is now known as petertodd
366 2015-11-14T22:05:55  *** petertodd is now known as Guest19775
367 2015-11-14T22:14:35  *** MarcoFalke has joined #bitcoin-core-dev
368 2015-11-14T22:32:08  *** PaulCape_ has quit IRC
369 2015-11-14T22:32:32  *** PaulCapestany has joined #bitcoin-core-dev
370 2015-11-14T22:35:05  *** zooko has joined #bitcoin-core-dev
371 2015-11-14T23:10:16  *** CodeShark_ has joined #bitcoin-core-dev
372 2015-11-14T23:11:16  *** MarcoFalke has quit IRC
373 2015-11-14T23:17:46  *** Guest19775 is now known as petertodd
374 2015-11-14T23:25:08  *** alpalp has joined #bitcoin-core-dev
375 2015-11-14T23:25:49  *** alpalp has quit IRC
376 2015-11-14T23:35:33  *** MarcoFalke has joined #bitcoin-core-dev
377 2015-11-14T23:44:44  *** moli has joined #bitcoin-core-dev
378 2015-11-14T23:48:35  *** molly has quit IRC
379 2015-11-14T23:55:52  *** molly has joined #bitcoin-core-dev
380 2015-11-14T23:59:38  *** moli has quit IRC