1 2017-12-05T00:00:18  <BlueMatt> achow101: give 11824 a spin?
  2 2017-12-05T00:00:22  <bitcoin-git> [bitcoin] TheBlueMatt opened pull request #11824:  Block ActivateBestChain to empty validationinterface queue (master...2017-12-11822-debug) https://github.com/bitcoin/bitcoin/pull/11824
  3 2017-12-05T00:00:22  <BlueMatt> seems to be working for me
  4 2017-12-05T00:00:47  <achow101> pulling now
  5 2017-12-05T00:01:17  <achow101> why's it so big?
  6 2017-12-05T00:02:23  *** dpg has joined #bitcoin-core-dev
  7 2017-12-05T00:02:41  *** jb55 has quit IRC
  8 2017-12-05T00:04:37  *** intcat has quit IRC
  9 2017-12-05T00:05:03  <BlueMatt> cause you cant call the callbacks themselves with cs_main held...I mean the first commit is just a nice-to-have that makes the queue drain faster, the third commit adds DEBUG_LOCKORDER annotations and removes spurious locks in tests, and the last two should probably be squashed
 10 2017-12-05T00:05:04  <achow101> BlueMatt: build error ---> validation.cpp:2480:18: error: ‘promise’ is not a member of ‘std’
 11 2017-12-05T00:05:06  <BlueMatt> its not as bad as it looks
 12 2017-12-05T00:05:37  <BlueMatt> achow101: oops, try now, forgot to commit that line
 13 2017-12-05T00:05:52  *** intcat has joined #bitcoin-core-dev
 14 2017-12-05T00:07:36  <achow101> what's the queue depth set to?
 15 2017-12-05T00:08:10  <BlueMatt> max is 10
 16 2017-12-05T00:08:14  <BlueMatt> feel free to play with it
 17 2017-12-05T00:09:34  <achow101> well it compiled and it's running
 18 2017-12-05T00:10:00  <sipa> ship it.
 19 2017-12-05T00:10:57  <BlueMatt> someone should tag 11822 as 0.16
 20 2017-12-05T00:11:16  <achow101> I'll let you know how it goes, so far so good.
 21 2017-12-05T00:11:47  <BlueMatt> I mean it definitely fixes the obvious infite-queue issue, but there could be other gremlins hiding
 22 2017-12-05T00:11:50  <BlueMatt> I doubt it
 23 2017-12-05T00:12:07  <BlueMatt> my bigger worry with that patch is future lockorder violations, but not a ton to be done about it, honestly
 24 2017-12-05T00:12:11  <sipa> tagged
 25 2017-12-05T00:15:42  <cfields> BlueMatt: i think it'd make for a smoother transition if a fence was inserted after each block's callbacks had been queued, rather than stalling at an arbitrary depth
 26 2017-12-05T00:16:04  <cfields> that way we could still work with the same assumptions as before, while we work out any remaining out-of-sync issues
 27 2017-12-05T00:16:30  <BlueMatt> cfields: ugh, we started the fence-removal process two releases ago, I'd rather not add it back if we can avoid it :(
 28 2017-12-05T00:16:44  *** intcat has quit IRC
 29 2017-12-05T00:18:19  <cfields> BlueMatt: the PR is the same thing, I'm just suggesting using a block-callback-depth (depth == 1 initially) rather than an individual callback depth.
 30 2017-12-05T00:18:27  <BlueMatt> hmm, that patch may not be sufficient :/ achow101 you see it staying low?
 31 2017-12-05T00:18:57  <achow101> BlueMatt: it's reasonable given my current dbcache of 8000
 32 2017-12-05T00:19:19  *** intcat has joined #bitcoin-core-dev
 33 2017-12-05T00:19:56  <BlueMatt> cfields: well if you want to do that we should move the blocking to the end and not the beginning and set it to 1 right before release/on the backport branch, not on master, please
 34 2017-12-05T00:20:08  <BlueMatt> cfields: 90% of the reason to merge this stuff sooner rather than later is to get testing cycles on it
 35 2017-12-05T00:20:19  <BlueMatt> otherwise its never gonna happen
 36 2017-12-05T00:21:41  <achow101> BlueMatt: I'd like this to be merged so I can run my fucking node :p
 37 2017-12-05T00:22:08  <BlueMatt> achow101: lol, you dont carry patches on your node?
 38 2017-12-05T00:23:25  *** jb55 has joined #bitcoin-core-dev
 39 2017-12-05T00:23:28  <achow101> BlueMatt: my node is also what I test and develop on and I would like to be able to test the other stuff I'm working on and PR it without extra patches
 40 2017-12-05T00:23:30  *** wxss has quit IRC
 41 2017-12-05T00:24:11  *** microapple has quit IRC
 42 2017-12-05T00:24:33  <BlueMatt> heh, well feel free to review
 43 2017-12-05T00:26:32  <cfields> BlueMatt: to be clear, i was suggesting that we let it run a full block ahead (depth of single block's callbacks). I wasn't suggesting that we use an actual depth of 1.
 44 2017-12-05T00:26:36  <BlueMatt> well res size seems to have stabilized on my reindex-chainstate node
 45 2017-12-05T00:26:55  <BlueMatt> cfields: I dont know how much that'd do aside from hide bugs?
 46 2017-12-05T00:27:20  <BlueMatt> if we want to neuter it cause we're worried, we should make ProcessNewBlock/ActivateBestChain block until callbacks finish for the latest action
 47 2017-12-05T00:27:38  <BlueMatt> letting it run ahead but heavily limiting how much it can run ahead just sounds like it'll hide more bugs than anything, no?
 48 2017-12-05T00:29:40  <cfields> well yes, but it'd ensure that we're only introducing one class of bugs at a time
 49 2017-12-05T00:33:16  <achow101> for each extra item in the queue, how much extra memory does that use
 50 2017-12-05T00:34:27  *** microapple has joined #bitcoin-core-dev
 51 2017-12-05T00:35:27  <sipa> a few MB, i expect
 52 2017-12-05T00:36:54  *** jb55 has quit IRC
 53 2017-12-05T00:36:57  *** ensign has quit IRC
 54 2017-12-05T00:37:29  <BlueMatt> achow101: (or not reindex)
 55 2017-12-05T00:37:52  <BlueMatt> achow101: its (during IBD, mostly just) a shared_ptr<const CBlock>
 56 2017-12-05T00:38:05  <achow101> BlueMatt: I'm not sure if the problem I saw was limited to reindex only though
 57 2017-12-05T00:38:06  <BlueMatt> cfields: I'm not sure which class of bugs is resolved by your suggestion, tbh?
 58 2017-12-05T00:38:18  <BlueMatt> achow101: it should almost entirely be
 59 2017-12-05T00:38:34  <BlueMatt> your wallet shouldnt take 10 minutes to process a new block....
 60 2017-12-05T00:38:44  <achow101> BlueMatt: the node crashed at some point before I had to reindex, and I think that was because of an OOM to (had to do the whole reboot computer stuff as I did when it crashed during reindex)
 61 2017-12-05T00:38:48  <BlueMatt> cfields: blocking before return may fix some stuff, but....
 62 2017-12-05T00:39:03  <achow101> that crash caused me to need to reindex it now
 63 2017-12-05T00:39:13  <BlueMatt> achow101: thats.....weird
 64 2017-12-05T00:39:58  <achow101> it's possible that that was unrelated, but I'm not sure
 65 2017-12-05T00:39:59  *** ensign has joined #bitcoin-core-dev
 66 2017-12-05T00:41:11  <BlueMatt> was that like right after a restart where you needed to catch up a few days blocks?
 67 2017-12-05T00:41:12  <BlueMatt> though even that shouldnt use all that much memory....
 68 2017-12-05T00:41:25  *** dcousens has joined #bitcoin-core-dev
 69 2017-12-05T00:41:31  <BlueMatt> i mean this has been on master a while...
 70 2017-12-05T00:42:01  <achow101> no, it had been up for a few days at that point, and I had it running for several weeks with a build of master prior to that
 71 2017-12-05T00:42:19  <achow101> although it is possible that the build I was running before did not have your PR merged
 72 2017-12-05T00:44:48  <achow101> I can grep through my log file to see what was running before I started on this build. although it may take a while since that log file is 42 GB because I forgot to turn of debug=net
 73 2017-12-05T00:45:10  <BlueMatt> hmm, I mean I did a read-through of the queue logic and didnt see any obvious races
 74 2017-12-05T00:45:42  <BlueMatt> ls -lh ~/.bitcoin/debug.log
 75 2017-12-05T00:45:42  <BlueMatt> -rw------- 1 matt matt 6.5T Dec  4 19:45 /home/matt/.bitcoin/debug.log
 76 2017-12-05T00:45:59  <BlueMatt> errr, s/races/leaks/
 77 2017-12-05T00:46:28  <achow101> BlueMatt: do you have your node patched to not shrink the debug.log at each restart?
 78 2017-12-05T00:46:32  *** DvdKhl has quit IRC
 79 2017-12-05T00:46:44  <BlueMatt> i believe it does not shrink by default if -debug? or something like that
 80 2017-12-05T00:46:53  *** microapple has quit IRC
 81 2017-12-05T00:47:26  * BlueMatt -> out, if bug is still there, do a heap profile
 82 2017-12-05T00:47:45  *** microapple has joined #bitcoin-core-dev
 83 2017-12-05T00:52:17  *** microapple has quit IRC
 84 2017-12-05T00:53:09  *** jb55 has joined #bitcoin-core-dev
 85 2017-12-05T00:55:45  *** dabura667 has joined #bitcoin-core-dev
 86 2017-12-05T00:59:13  *** jb55 has quit IRC
 87 2017-12-05T01:03:09  <jamesob> going to test #11824 with -reindex
 88 2017-12-05T01:03:10  <gribble> https://github.com/bitcoin/bitcoin/issues/11824 | Block ActivateBestChain to empty validationinterface queue by TheBlueMatt · Pull Request #11824 · bitcoin/bitcoin · GitHub
 89 2017-12-05T01:05:46  *** Aaronvan_ has joined #bitcoin-core-dev
 90 2017-12-05T01:06:02  *** Aaronvan_ has quit IRC
 91 2017-12-05T01:09:21  *** AaronvanW has quit IRC
 92 2017-12-05T01:18:35  *** Ylbam has quit IRC
 93 2017-12-05T01:19:51  *** Deacyde has joined #bitcoin-core-dev
 94 2017-12-05T01:22:39  *** intcat has quit IRC
 95 2017-12-05T01:23:32  *** intcat has joined #bitcoin-core-dev
 96 2017-12-05T01:24:58  *** Randolf has joined #bitcoin-core-dev
 97 2017-12-05T01:40:54  *** microapple has joined #bitcoin-core-dev
 98 2017-12-05T01:56:41  *** cdecker has quit IRC
 99 2017-12-05T01:57:07  *** cdecker has joined #bitcoin-core-dev
100 2017-12-05T01:57:20  *** lifeofguenter has quit IRC
101 2017-12-05T01:57:33  *** arubi has quit IRC
102 2017-12-05T01:59:21  *** lifeofguenter_ has joined #bitcoin-core-dev
103 2017-12-05T01:59:51  *** arubi has joined #bitcoin-core-dev
104 2017-12-05T02:01:29  *** microapple has quit IRC
105 2017-12-05T02:03:34  *** rhavar has quit IRC
106 2017-12-05T02:09:41  *** intcat has quit IRC
107 2017-12-05T02:10:34  *** intcat has joined #bitcoin-core-dev
108 2017-12-05T02:14:29  *** Murch has quit IRC
109 2017-12-05T02:14:56  *** intcat has quit IRC
110 2017-12-05T02:15:02  <promag> please give feedback to #11826, ty
111 2017-12-05T02:15:04  <gribble> https://github.com/bitcoin/bitcoin/issues/11826 | RFC: Activity feature · Issue #11826 · bitcoin/bitcoin · GitHub
112 2017-12-05T02:15:45  *** intcat has joined #bitcoin-core-dev
113 2017-12-05T02:19:50  *** microapple has joined #bitcoin-core-dev
114 2017-12-05T02:21:59  *** Chris_Stewart_5 has joined #bitcoin-core-dev
115 2017-12-05T02:26:35  *** microapple has quit IRC
116 2017-12-05T02:31:20  *** Randolf has quit IRC
117 2017-12-05T02:32:35  *** shesek has quit IRC
118 2017-12-05T02:42:01  *** Emcy has joined #bitcoin-core-dev
119 2017-12-05T02:44:17  *** Emcy_ has quit IRC
120 2017-12-05T02:47:29  *** jamesob has quit IRC
121 2017-12-05T02:48:42  *** shesek has joined #bitcoin-core-dev
122 2017-12-05T02:49:18  *** Emcy has quit IRC
123 2017-12-05T02:52:25  *** promag has quit IRC
124 2017-12-05T03:18:50  *** intcat has quit IRC
125 2017-12-05T03:20:58  *** intcat has joined #bitcoin-core-dev
126 2017-12-05T03:23:12  *** Emcy has joined #bitcoin-core-dev
127 2017-12-05T03:23:30  *** Chris_Stewart_5 has quit IRC
128 2017-12-05T03:26:58  *** jb55 has joined #bitcoin-core-dev
129 2017-12-05T03:29:15  *** microapple has joined #bitcoin-core-dev
130 2017-12-05T03:53:05  *** promag has joined #bitcoin-core-dev
131 2017-12-05T04:02:26  *** jtimon has quit IRC
132 2017-12-05T04:08:21  *** dgenr8 has quit IRC
133 2017-12-05T04:09:06  *** dgenr8 has joined #bitcoin-core-dev
134 2017-12-05T04:10:44  *** intcat has quit IRC
135 2017-12-05T04:11:38  *** intcat has joined #bitcoin-core-dev
136 2017-12-05T04:27:09  *** Victorsueca has quit IRC
137 2017-12-05T04:28:24  *** Victorsueca has joined #bitcoin-core-dev
138 2017-12-05T04:38:41  *** microapple has quit IRC
139 2017-12-05T04:44:15  <bitcoin-git> [bitcoin] kallewoof closed pull request #11489: [wallet] sendtoaddress style argument (master...201709_segwitwallet2_sendtoaddress) https://github.com/bitcoin/bitcoin/pull/11489
140 2017-12-05T04:44:15  *** Randolf has joined #bitcoin-core-dev
141 2017-12-05T04:46:20  *** brianhoffman has quit IRC
142 2017-12-05T04:46:39  *** brianhoffman has joined #bitcoin-core-dev
143 2017-12-05T04:48:01  *** bule has joined #bitcoin-core-dev
144 2017-12-05T04:48:56  *** bule2 has quit IRC
145 2017-12-05T04:59:54  *** roidster has joined #bitcoin-core-dev
146 2017-12-05T05:00:16  *** roidster is now known as Guest70224
147 2017-12-05T05:00:42  *** Guest70224 is now known as roidster
148 2017-12-05T05:07:40  *** intcat has quit IRC
149 2017-12-05T05:08:35  *** intcat has joined #bitcoin-core-dev
150 2017-12-05T05:11:34  *** musalbas has joined #bitcoin-core-dev
151 2017-12-05T05:16:24  *** bule has quit IRC
152 2017-12-05T05:16:48  *** bule has joined #bitcoin-core-dev
153 2017-12-05T05:22:49  <kallewoof> I've been asked about address index on top of bitcoin core (to replace the unmaintained fork that exists somewhere else). What are people's opinions on this? I know some people prefer a minimal code base, but having an address index option seems like it would alleviate a lot of problems in a lot of places.
154 2017-12-05T05:27:15  <gmaxwell> a history address index is unsustainable; anyone who uses it is eventually going to be forced to use trust a third party service.
155 2017-12-05T05:27:45  <kallewoof> Why?
156 2017-12-05T05:28:00  <kallewoof> Too much space, or..?
157 2017-12-05T05:28:05  <gmaxwell> I don't see any problems with having hooks to support such things, but directly incorporating an index of history wouldn't just be a waste of our review and maintaince resources, it would be actively harmful to the ecosystem.
158 2017-12-05T05:28:28  <gmaxwell> Yes too much space, too much seek load, and incompatible with pruning; or any kind of pruned sync.
159 2017-12-05T05:29:02  * kallewoof nods
160 2017-12-05T05:29:17  <gmaxwell> we've also already seen that behavior where people using such indexes constructed via abe or other tools that created them, failed back to blockchain.info and similar.
161 2017-12-05T05:29:57  <gmaxwell> They're also not required except for unusual cases (like making an explorer, which also requires spent-position indexes).   An index of unspent outputs would be far more reasonable.
162 2017-12-05T05:30:43  <kallewoof> Yeah, this would be used by explorers, I would expect. I think some (lots?) of them use the unmaintained fork I mentioned earlier.
163 2017-12-05T05:30:59  <gmaxwell> I sure hope not.
164 2017-12-05T05:31:16  <kallewoof> Rumor says it is so. Part of why I was asked to look into it.
165 2017-12-05T05:31:23  <gmaxwell> I know with absolute certanty that that old index patch could lose data, I just don't know _how_.
166 2017-12-05T05:31:41  *** intcat has quit IRC
167 2017-12-05T05:32:04  <kallewoof> I think having hooks in bitcoin core that let such a system exist separately is a good plan. It would only rely on API breakage, which bitcoin core is very careful about.
168 2017-12-05T05:32:28  *** intcat has joined #bitcoin-core-dev
169 2017-12-05T05:32:32  <gmaxwell> As in I had a node with it where it was faliing to actual spendable outputs connected to an address, and I tested that it was incorrect quite throughly. I assumed the bad data was some gotcha with reorgs or likewise, but no one investigated.
170 2017-12-05T05:32:41  <gmaxwell> Yes, hooks for such things seem fine to me.
171 2017-12-05T05:32:59  <kallewoof> Ah.. yeah, reorgs is definitely a challenge.
172 2017-12-05T05:35:34  <sipa> if i had infinite time i'd write a separate efficient indexing service, which just speaks the p2p protocol
173 2017-12-05T05:36:18  <sipa> which you then can connect to bitcoind
174 2017-12-05T05:38:00  <kallewoof> sipa: I guess that would have the obvious benefit of not actually needing a node at all if it could just sync from the network?
175 2017-12-05T05:38:38  <sipa> right
176 2017-12-05T05:39:25  <sipa> it would be nice to reuse some of the headers sync logic in core, but if you assume you're connected to a trusted peer anyway, i don't think it needs to be allthat complicated
177 2017-12-05T05:39:29  <sipa> just deal with reorgs
178 2017-12-05T05:40:09  *** xRavenheart has joined #bitcoin-core-dev
179 2017-12-05T05:40:53  <gmaxwell> from an arch perspective you'd even want it to just keep a SQL database in sync, unfortunately the performance would be horrible except on gold plated system. (massive nvem raid 1 arrays and gobs of ram or whatever)
180 2017-12-05T05:42:06  <kallewoof> SQL db sounds like it would be kinda slow yeah.
181 2017-12-05T05:42:39  <gmaxwell> thats how abe worked.
182 2017-12-05T05:42:58  <kallewoof> I was thinking of using redis, but I think I'm gonna have to do profiling to find a good solution.
183 2017-12-05T05:43:07  <gmaxwell> last I heard of anyone successfully running it, it required well over 1TB state and would take a month to sync or something.
184 2017-12-05T05:43:18  <kallewoof> Wow..
185 2017-12-05T05:43:54  <sipa> i think it's been unusably slow for the better part of 5 years now
186 2017-12-05T05:44:10  <gmaxwell> this was years ago, it would presumably be some multiple of that now.
187 2017-12-05T05:49:19  <kallewoof> It would be address -> tx list, tx -> address list, block -> tx list.. tx'es could be compressed as block height + index (so ~8b). Not sure what else neat stuff you could do. sipa's work on tx format should be inspiring.
188 2017-12-05T05:50:12  <esotericnonsense> blockheight + index doesn't work well at all, what about reorgs
189 2017-12-05T05:51:29  <kallewoof> Maybe keep track of last 1000 entries and undo all based on shared fork point height?
190 2017-12-05T05:51:30  <gmaxwell> works fine, so long as you're only tracking a single chain at a time.
191 2017-12-05T05:51:36  <gmaxwell> you have to back out the index and reapply.
192 2017-12-05T05:51:48  <kallewoof> Could just look at block again and undo, yeah.
193 2017-12-05T05:53:10  <esotericnonsense> eh I suppose that's right. SQL is a bit abstracted for me i'm not sure how it stores repeated data internally. f.ex. if you have a ton of tables with foreign keys do they store the keys repeatedly or use a more space efficient method.
194 2017-12-05T05:55:45  <esotericnonsense> for example on chain you might have a 256bit txid used hundreds of times if it has a bunch of outputs but you can probably just store that as a 32bit(?) incrementing integer on every tx you've seen since genesis.
195 2017-12-05T05:55:47  <kallewoof> I really doubt SQL would work well for this, but I could be wrong.
196 2017-12-05T05:56:45  <Randolf> esotericnonsense:  It depends on the SQL server software being used.  As far as I know, whatever you choose to store in the columns is what's stored there.
197 2017-12-05T05:57:18  <Randolf> esotericnonsense:  Usually it's up to the DBA to choose to set up smaller IDs that reference more data in a separate row.
198 2017-12-05T05:58:40  <kallewoof> We've used up 6.5% of a uint32 (278236762/4294967295), but it could work for awhile at least.
199 2017-12-05T05:58:51  <Randolf> esotericnonsense:  For Bitcoin stuff, I'd create custom datatypes that save the data in binary format.
200 2017-12-05T05:59:21  <esotericnonsense> basically what i'm thinking is that these indexes can easily multiply the blockchain size by five  and possibly more depending on how you do it if you store the full data.
201 2017-12-05T05:59:48  <Randolf> It's possible.
202 2017-12-05T06:00:29  <esotericnonsense> address -> txlist, block -> txlist, and tx (unless you don't store tx but pull it out of a stored blockchain) is 3x txid
203 2017-12-05T06:00:40  <Randolf> An index normally refers to an internal reference that's usually smaller than the column data it's indexing.
204 2017-12-05T06:01:23  <Randolf> Can arrays help?  In PostgreSQL a column type can be an array.
205 2017-12-05T06:01:31  <Randolf> These can also be indexed efficiently.
206 2017-12-05T06:02:35  <Randolf> While you'd still be storing TX data, you could cut down on the number of rows.
207 2017-12-05T06:08:05  <gmaxwell> this sort of speculation isn't so useful.
208 2017-12-05T06:09:06  <gmaxwell> IIRC the abe schema wasn't too offensively inefficienct, but there are overheads and billions of records add up... but nothing is really learned from this discussion.  just "I think this would be not efficienct" and "I think it would be, maybe" :P
209 2017-12-05T06:09:30  <esotericnonsense> yeah agree.
210 2017-12-05T06:09:38  <kallewoof> I'll just try a bunch of stuff I guess.
211 2017-12-05T06:10:51  <esotericnonsense> it sounds to me like what might be useful is to have a seperate service that only maintains the set of things core doesn't
212 2017-12-05T06:11:22  <esotericnonsense> so basically addrindex but as an external thing. all it needs is addr->[txids]. if you want the transactions just ask core rpc with txindex on
213 2017-12-05T06:12:23  <esotericnonsense> that'd only need the core rpc api to remain (relatively) static and not have to be maintained as a thing on an ongoing basis if internals shuffle around.
214 2017-12-05T06:12:43  <kallewoof> If txs are stored as height+index you don't even need txindex, at least if #10275 is merged.
215 2017-12-05T06:12:47  <gribble> https://github.com/bitcoin/bitcoin/issues/10275 | [rpc] Allow fetching tx directly from specified block in getrawtransaction by kallewoof · Pull Request #10275 · bitcoin/bitcoin · GitHub
216 2017-12-05T06:13:36  <kallewoof> And yeah, we agreed this should be an external thing that either uses hooks in bitcoin or implements the p2p protocol and talks to the network directly.
217 2017-12-05T06:15:22  <esotericnonsense> i guess i was commenting on "address -> tx list, tx -> address list, block -> tx list" and how i don't think you need the latter two because bitcoind does it
218 2017-12-05T06:15:45  * kallewoof nods
219 2017-12-05T06:16:09  <kallewoof> I wasn't assuming the person running had a local synced node. Maybe I should.
220 2017-12-05T06:17:00  <esotericnonsense> eh? addrindex implies that no? (well that or that you have rpc access anyway).
221 2017-12-05T06:18:07  <kallewoof> sipa had the idea of an efficient indexing service which speaks the p2p protocol.
222 2017-12-05T06:19:12  <gmaxwell> that assumption saves you basically 1x copy of the blockchain at most, but then means you can't run the indexer on the seperate hardware from the node or fail over between multiple nodes, etc.
223 2017-12-05T06:19:28  <gmaxwell> or upgrade the node independantly from the indexer
224 2017-12-05T06:21:46  *** roidster has quit IRC
225 2017-12-05T06:22:24  <kallewoof> Makes sense
226 2017-12-05T06:22:35  <kallewoof> (to not assume local node)
227 2017-12-05T06:23:36  <esotericnonsense> i'm confused. you can run the indexer on seperate hardware just fine. failover works in the same way anyway (you can have multiple bitcoinds if you want)
228 2017-12-05T06:24:48  <esotericnonsense> height+index storing of transactions becomes complicated in that case sure.
229 2017-12-05T06:25:19  *** intcat has quit IRC
230 2017-12-05T06:25:40  <kallewoof> Ultimately it doesn't matter to the indexer if you have a local node or not. It won't use it directly, it will use the p2p network. The only time it needs the RPC is when it wants to convert e.g. block height+index into a tx, but it might just as well give you the height+index pairs and let you deal yourself.
231 2017-12-05T06:26:12  *** intcat has joined #bitcoin-core-dev
232 2017-12-05T06:26:57  *** intcat has quit IRC
233 2017-12-05T06:29:15  <gmaxwell> also if the node the indexer is behind is pruned then the overhead of a redundant copy mostly goes away.
234 2017-12-05T06:30:34  *** intcat has joined #bitcoin-core-dev
235 2017-12-05T06:33:52  *** indistylo has joined #bitcoin-core-dev
236 2017-12-05T06:38:16  <kallewoof> If the node is pruned, the indexer will not be able to use it for any lookups of block height <= pruned height
237 2017-12-05T06:38:42  * esotericnonsense is in a state of befuddlement
238 2017-12-05T06:39:11  <esotericnonsense> if it has p2p logic (basically it's a node) then what is the seperate bitcoind node being used for at all
239 2017-12-05T06:39:16  *** dcousens has quit IRC
240 2017-12-05T06:39:41  <esotericnonsense> trusted verifier i guess?
241 2017-12-05T06:39:47  *** dcousens has joined #bitcoin-core-dev
242 2017-12-05T06:39:59  <kallewoof> The indexer doesn't store anything but the indexing parts, so it needs a node to resolve the 8 byte (block height + tx index) into actual transactions. It could store the entire block chain while at it, but that seems a bit redundant.
243 2017-12-05T06:40:25  <sipa> kallewoof: that seems inefficient
244 2017-12-05T06:40:46  <sipa> relying on a node for storage won't gove the kind of performance a blockexplorer needs
245 2017-12-05T06:41:30  <kallewoof> I thought you were suggesting a p2p indexer that doesn't depend on a local node. Why not use RPC then?
246 2017-12-05T06:42:41  <sipa> rpc is slow
247 2017-12-05T06:42:43  <kallewoof> Or are you saying store the entire transaction in the indexer db?
248 2017-12-05T06:42:47  <sipa> yes
249 2017-12-05T06:42:51  <kallewoof> Ah
250 2017-12-05T07:07:00  *** DvdKhl has joined #bitcoin-core-dev
251 2017-12-05T07:14:51  *** intcat has quit IRC
252 2017-12-05T07:15:13  *** jb55 has quit IRC
253 2017-12-05T07:15:45  *** intcat has joined #bitcoin-core-dev
254 2017-12-05T07:17:21  *** promag has quit IRC
255 2017-12-05T07:17:35  *** promag has joined #bitcoin-core-dev
256 2017-12-05T07:17:36  *** intcat has quit IRC
257 2017-12-05T07:19:27  *** promag has quit IRC
258 2017-12-05T07:20:17  *** Ylbam has joined #bitcoin-core-dev
259 2017-12-05T07:22:26  *** intcat has joined #bitcoin-core-dev
260 2017-12-05T07:33:38  *** shesek has quit IRC
261 2017-12-05T07:39:44  *** xRavenheart has quit IRC
262 2017-12-05T07:41:09  *** Cogito_Ergo_Sum has joined #bitcoin-core-dev
263 2017-12-05T07:41:09  *** Cogito_Ergo_Sum has joined #bitcoin-core-dev
264 2017-12-05T07:44:38  *** promag has joined #bitcoin-core-dev
265 2017-12-05T07:53:06  *** unholymachine has quit IRC
266 2017-12-05T07:54:37  *** unholymachine has joined #bitcoin-core-dev
267 2017-12-05T07:57:23  *** promag has quit IRC
268 2017-12-05T08:01:42  *** sanjeev has joined #bitcoin-core-dev
269 2017-12-05T08:04:48  *** Deacyde has quit IRC
270 2017-12-05T08:05:17  *** Deacyde has joined #bitcoin-core-dev
271 2017-12-05T08:07:45  *** promag has joined #bitcoin-core-dev
272 2017-12-05T08:10:37  *** Deacyde has quit IRC
273 2017-12-05T08:10:43  *** Deacyde has joined #bitcoin-core-dev
274 2017-12-05T08:19:24  *** promag has quit IRC
275 2017-12-05T08:20:30  *** meshcollider has quit IRC
276 2017-12-05T08:22:53  *** intcat has quit IRC
277 2017-12-05T08:23:59  *** intcat has joined #bitcoin-core-dev
278 2017-12-05T08:25:06  *** JDRacer has joined #bitcoin-core-dev
279 2017-12-05T08:25:30  *** laurentmt has joined #bitcoin-core-dev
280 2017-12-05T08:25:30  *** JDRacer has quit IRC
281 2017-12-05T08:34:52  *** bule has quit IRC
282 2017-12-05T08:39:44  *** rhavar has joined #bitcoin-core-dev
283 2017-12-05T08:49:23  *** Deacyde has quit IRC
284 2017-12-05T08:49:56  *** Deacyde has joined #bitcoin-core-dev
285 2017-12-05T09:04:10  *** indistylo has quit IRC
286 2017-12-05T09:10:42  *** DvdKhl has quit IRC
287 2017-12-05T09:13:37  *** Randolf has quit IRC
288 2017-12-05T09:20:39  *** zshlyk has joined #bitcoin-core-dev
289 2017-12-05T09:21:26  <kallewoof> Hum. I could avoid a lot of reinventing wheels by simply requiring libbitcoin_[x].a in the indexer to link p2p stuff in. Potential minefield, but would be super easy to keep up to date with bitcoin core if it worked.
290 2017-12-05T09:22:05  *** timothy has joined #bitcoin-core-dev
291 2017-12-05T09:23:20  <kallewoof> Rewriting a minimal p2p codebase could be useful for lots of cases, but would need to be maintained to a greater degree.
292 2017-12-05T09:23:56  *** intcat has quit IRC
293 2017-12-05T09:24:48  <gmaxwell> also less important if its out of date... p2p protocol intentionally does not change frequently.
294 2017-12-05T09:25:16  <gmaxwell> an indexer written against 0.2.10 would likely still work (though wouldn't index p2sh/segwit, but that may not matter if you're not using those things)
295 2017-12-05T09:25:51  <kallewoof> That's a good point.
296 2017-12-05T09:27:08  *** zshlyk has quit IRC
297 2017-12-05T09:27:19  <kallewoof> I guess I'll try linking directly and see how far that gets me. I suspect there will be a lot of issues where the bitcoin code expects a fully fledged bitcoin instance running.
298 2017-12-05T09:28:09  <kallewoof> If not, not regularly maintained would not be a critical issue as you noted.
299 2017-12-05T09:31:15  *** Babozor_ has quit IRC
300 2017-12-05T09:32:07  *** zshlyk has joined #bitcoin-core-dev
301 2017-12-05T09:36:29  *** promag has joined #bitcoin-core-dev
302 2017-12-05T09:45:54  *** tiagotrs has joined #bitcoin-core-dev
303 2017-12-05T09:48:07  *** AaronvanW has joined #bitcoin-core-dev
304 2017-12-05T09:53:10  *** Ylbam has quit IRC
305 2017-12-05T09:53:45  *** zshlyk has quit IRC
306 2017-12-05T09:53:56  *** rafalcpp has quit IRC
307 2017-12-05T09:54:38  *** zshlyk has joined #bitcoin-core-dev
308 2017-12-05T09:54:50  *** Guyver2 has joined #bitcoin-core-dev
309 2017-12-05T09:57:33  *** Victorsueca has quit IRC
310 2017-12-05T09:58:44  *** Victorsueca has joined #bitcoin-core-dev
311 2017-12-05T10:00:35  *** rafalcpp has joined #bitcoin-core-dev
312 2017-12-05T10:03:42  *** zshlyk has quit IRC
313 2017-12-05T10:05:10  *** zshlyk has joined #bitcoin-core-dev
314 2017-12-05T10:06:10  *** meshcollider has joined #bitcoin-core-dev
315 2017-12-05T10:06:29  *** AaronvanW has quit IRC
316 2017-12-05T10:07:45  *** zshlyk has quit IRC
317 2017-12-05T10:08:38  *** zshlyk has joined #bitcoin-core-dev
318 2017-12-05T10:28:43  *** belcher has joined #bitcoin-core-dev
319 2017-12-05T10:55:29  *** esotericnonsense has quit IRC
320 2017-12-05T11:04:46  *** zshlyk has quit IRC
321 2017-12-05T11:05:15  *** Babozor has joined #bitcoin-core-dev
322 2017-12-05T11:05:52  *** zshlyk has joined #bitcoin-core-dev
323 2017-12-05T11:23:22  *** Victorsueca has quit IRC
324 2017-12-05T11:23:27  *** Victor_sueca has joined #bitcoin-core-dev
325 2017-12-05T11:26:25  *** tiagotrs has quit IRC
326 2017-12-05T11:28:55  <bitcoin-git> [bitcoin] MeshCollider opened pull request #11829: Test datadir specified in conf file exists (master...201712_datadir_crash) https://github.com/bitcoin/bitcoin/pull/11829
327 2017-12-05T11:30:20  *** luihuidui has joined #bitcoin-core-dev
328 2017-12-05T11:30:25  <luihuidui> Hi core devs
329 2017-12-05T11:32:57  *** zshlyk has quit IRC
330 2017-12-05T11:33:40  *** zshlyk has joined #bitcoin-core-dev
331 2017-12-05T11:51:25  *** BashCo_ has joined #bitcoin-core-dev
332 2017-12-05T11:52:33  *** BashCo has quit IRC
333 2017-12-05T11:54:56  *** hellxcode has quit IRC
334 2017-12-05T11:57:13  *** Victor_sueca is now known as Victorsueca
335 2017-12-05T12:06:43  *** zshlyk has quit IRC
336 2017-12-05T12:07:33  *** zshlyk has joined #bitcoin-core-dev
337 2017-12-05T12:09:01  *** tiagotrs has joined #bitcoin-core-dev
338 2017-12-05T12:10:22  *** tiagotrs has quit IRC
339 2017-12-05T12:10:22  *** tiagotrs has joined #bitcoin-core-dev
340 2017-12-05T12:15:58  *** meshcollider has quit IRC
341 2017-12-05T12:16:57  *** hellxcode has joined #bitcoin-core-dev
342 2017-12-05T12:17:00  *** dabura667 has quit IRC
343 2017-12-05T12:19:27  *** luihuidui has quit IRC
344 2017-12-05T12:24:22  *** SopaXorzTaker has joined #bitcoin-core-dev
345 2017-12-05T12:35:35  *** microapple has joined #bitcoin-core-dev
346 2017-12-05T12:37:14  *** harrymm has joined #bitcoin-core-dev
347 2017-12-05T12:41:00  *** checksauce has joined #bitcoin-core-dev
348 2017-12-05T12:42:59  *** checksauce has quit IRC
349 2017-12-05T12:53:45  *** zshlyk has quit IRC
350 2017-12-05T12:54:38  *** zshlyk has joined #bitcoin-core-dev
351 2017-12-05T12:56:42  *** hellxcode has quit IRC
352 2017-12-05T12:57:34  *** JackH has joined #bitcoin-core-dev
353 2017-12-05T13:00:01  *** promag has quit IRC
354 2017-12-05T13:05:02  *** zshlyk has quit IRC
355 2017-12-05T13:05:55  *** zshlyk has joined #bitcoin-core-dev
356 2017-12-05T13:31:50  *** zshlyk has quit IRC
357 2017-12-05T13:32:47  *** zshlyk has joined #bitcoin-core-dev
358 2017-12-05T13:38:04  *** Chris_Stewart_5 has joined #bitcoin-core-dev
359 2017-12-05T13:57:07  *** microapple has quit IRC
360 2017-12-05T14:02:37  *** tiagotrs has quit IRC
361 2017-12-05T14:02:44  *** mypopydev has joined #bitcoin-core-dev
362 2017-12-05T14:04:59  *** harrymm has quit IRC
363 2017-12-05T14:08:11  *** promag has joined #bitcoin-core-dev
364 2017-12-05T14:12:06  *** esotericnonsense has joined #bitcoin-core-dev
365 2017-12-05T14:12:32  *** Chris_Stewart_5 has quit IRC
366 2017-12-05T14:15:44  *** zshlyk has quit IRC
367 2017-12-05T14:16:38  *** tiagotrs has joined #bitcoin-core-dev
368 2017-12-05T14:16:40  *** zshlyk has joined #bitcoin-core-dev
369 2017-12-05T14:19:01  *** harrymm has joined #bitcoin-core-dev
370 2017-12-05T14:23:53  <promag> ping jnewbery
371 2017-12-05T14:24:27  <promag> is it possible to test .conf file?
372 2017-12-05T14:24:52  <promag> ref #11829
373 2017-12-05T14:24:54  <gribble> https://github.com/bitcoin/bitcoin/issues/11829 | Test datadir specified in conf file exists by MeshCollider · Pull Request #11829 · bitcoin/bitcoin · GitHub
374 2017-12-05T14:25:14  *** esotericnonsense has quit IRC
375 2017-12-05T14:27:52  *** zshlyk has quit IRC
376 2017-12-05T14:28:46  *** zshlyk has joined #bitcoin-core-dev
377 2017-12-05T14:29:24  <aj> promag: do stuff like https://github.com/bitcoin/bitcoin/pull/10996/files#diff-7814a1916d2f12f31509d33370931974 ?
378 2017-12-05T14:29:59  <promag> aj: \m/
379 2017-12-05T14:35:04  *** roidster has joined #bitcoin-core-dev
380 2017-12-05T14:35:27  *** roidster is now known as Guest24280
381 2017-12-05T14:37:33  *** Guest24280 is now known as [_smitty]
382 2017-12-05T14:40:30  *** Victorsueca has quit IRC
383 2017-12-05T14:41:44  *** Victorsueca has joined #bitcoin-core-dev
384 2017-12-05T14:41:48  *** ghost43 has quit IRC
385 2017-12-05T14:42:54  *** mypopydev has left #bitcoin-core-dev
386 2017-12-05T14:44:35  *** [_smitty] is now known as roidster
387 2017-12-05T14:45:57  *** ekrok_ has joined #bitcoin-core-dev
388 2017-12-05T14:46:58  *** ghost43 has joined #bitcoin-core-dev
389 2017-12-05T14:48:31  *** ekrok has quit IRC
390 2017-12-05T14:51:41  *** Chris_Stewart_5 has joined #bitcoin-core-dev
391 2017-12-05T14:54:03  <bitcoin-git> [bitcoin] hkjn opened pull request #11830: rpcuser.py: Use 'python' not 'python2' (master...rpcuser-py) https://github.com/bitcoin/bitcoin/pull/11830
392 2017-12-05T14:54:57  *** sanjeev has quit IRC
393 2017-12-05T14:57:11  *** Chris_Stewart_5 has quit IRC
394 2017-12-05T15:10:58  *** jtimon has joined #bitcoin-core-dev
395 2017-12-05T15:23:56  *** microapple has joined #bitcoin-core-dev
396 2017-12-05T15:27:44  *** zshlyk has quit IRC
397 2017-12-05T15:29:01  *** zshlyk has joined #bitcoin-core-dev
398 2017-12-05T15:30:11  *** ghost43 has quit IRC
399 2017-12-05T15:30:36  *** ghost43 has joined #bitcoin-core-dev
400 2017-12-05T15:31:02  *** tiagotrs has quit IRC
401 2017-12-05T15:32:25  *** tiagotrs has joined #bitcoin-core-dev
402 2017-12-05T15:35:36  *** ghost43 has quit IRC
403 2017-12-05T15:35:51  *** ghost43 has joined #bitcoin-core-dev
404 2017-12-05T15:36:48  *** zshlyk has quit IRC
405 2017-12-05T15:37:35  *** zshlyk has joined #bitcoin-core-dev
406 2017-12-05T15:41:16  <bitcoin-git> [bitcoin] TheBlueMatt opened pull request #11831: Always return true if AppInitMain got to the end (master...2017-12-startup-exit-return-code-race) https://github.com/bitcoin/bitcoin/pull/11831
407 2017-12-05T15:44:12  *** dermoth has quit IRC
408 2017-12-05T15:44:42  *** dermoth has joined #bitcoin-core-dev
409 2017-12-05T15:45:18  *** sanjeev has joined #bitcoin-core-dev
410 2017-12-05T15:53:57  *** Chris_Stewart_5 has joined #bitcoin-core-dev
411 2017-12-05T15:56:35  <promag> BlueMatt: I didn't realise main thread was sitting there in WaitForShutdown
412 2017-12-05T15:56:57  <BlueMatt> hmm?
413 2017-12-05T15:57:08  <BlueMatt> ohoh, yea, we have a global thread that sits there and does nothing :(
414 2017-12-05T15:57:52  <promag> actually it does.. are we there yet? are we there yet? are we there yet? ..
415 2017-12-05T15:58:27  <BlueMatt> heh
416 2017-12-05T15:58:29  <BlueMatt> yea...that
417 2017-12-05T15:59:28  <promag> so zapwallettxes causes the process to exit with non-zero
418 2017-12-05T15:59:55  <BlueMatt> Ive never seen that failure before, but I think its the race with ReacceptWalletTransactions
419 2017-12-05T16:00:03  <BlueMatt> if nothing else that pr should be an improvement
420 2017-12-05T16:00:08  <BlueMatt> its crazy to have that race
421 2017-12-05T16:11:55  *** ExtraCrispy has joined #bitcoin-core-dev
422 2017-12-05T16:12:49  *** tiagotrs has quit IRC
423 2017-12-05T16:13:45  *** zshlyk has quit IRC
424 2017-12-05T16:14:37  *** zshlyk has joined #bitcoin-core-dev
425 2017-12-05T16:16:08  *** AaronvanW has joined #bitcoin-core-dev
426 2017-12-05T16:18:15  *** Aaronvan_ has joined #bitcoin-core-dev
427 2017-12-05T16:20:16  *** ExtraCrispy has quit IRC
428 2017-12-05T16:20:20  *** wraithm has joined #bitcoin-core-dev
429 2017-12-05T16:20:45  *** AaronvanW has quit IRC
430 2017-12-05T16:25:52  *** DvdKhl has joined #bitcoin-core-dev
431 2017-12-05T16:27:00  *** tiagotrs has joined #bitcoin-core-dev
432 2017-12-05T16:34:35  *** tiagotrs has quit IRC
433 2017-12-05T16:40:01  *** JackH has quit IRC
434 2017-12-05T16:41:10  *** promag has quit IRC
435 2017-12-05T16:41:22  *** wxss has joined #bitcoin-core-dev
436 2017-12-05T16:48:44  *** zshlyk has quit IRC
437 2017-12-05T16:49:31  *** zshlyk has joined #bitcoin-core-dev
438 2017-12-05T16:50:03  *** microapple has quit IRC
439 2017-12-05T16:53:41  *** JackH has joined #bitcoin-core-dev
440 2017-12-05T16:53:46  *** earlz is now known as WowSuchEarlz
441 2017-12-05T16:58:38  *** microapple has joined #bitcoin-core-dev
442 2017-12-05T17:11:08  *** zshlyk has quit IRC
443 2017-12-05T17:12:01  *** zshlyk has joined #bitcoin-core-dev
444 2017-12-05T17:18:41  *** Aaronvan_ is now known as AaronvanW
445 2017-12-05T17:32:14  *** zshlyk has quit IRC
446 2017-12-05T17:33:06  *** zshlyk has joined #bitcoin-core-dev
447 2017-12-05T17:33:17  *** spudowiar has joined #bitcoin-core-dev
448 2017-12-05T17:33:43  <sipa> off topic for at least 5 years: c++17 is now officia
449 2017-12-05T17:37:58  *** jamesob has joined #bitcoin-core-dev
450 2017-12-05T17:41:08  *** jb55 has joined #bitcoin-core-dev
451 2017-12-05T17:42:32  *** Murch has joined #bitcoin-core-dev
452 2017-12-05T17:42:48  *** BGL has quit IRC
453 2017-12-05T17:43:27  *** twistedline has quit IRC
454 2017-12-05T17:47:04  *** twistedline has joined #bitcoin-core-dev
455 2017-12-05T17:47:17  *** goatpig has joined #bitcoin-core-dev
456 2017-12-05T17:47:44  <spudowiar> sipa: With these new C++ standards I'll soon playing fast and loose with the word "know" when I say that I know C++
457 2017-12-05T17:47:48  <spudowiar> *soon be
458 2017-12-05T17:49:20  *** Ylbam has joined #bitcoin-core-dev
459 2017-12-05T17:50:15  <sipa> haha
460 2017-12-05T17:58:31  *** WowSuchEarlz is now known as eWowSuchEarlz
461 2017-12-05T17:58:49  *** eWowSuchEarlz is now known as WowSuchEarlz
462 2017-12-05T18:00:01  *** timothy has quit IRC
463 2017-12-05T18:05:12  *** ghost43 has quit IRC
464 2017-12-05T18:05:25  *** ghost43 has joined #bitcoin-core-dev
465 2017-12-05T18:07:59  *** microapple has quit IRC
466 2017-12-05T18:10:12  *** zshlyk has quit IRC
467 2017-12-05T18:11:02  *** zshlyk has joined #bitcoin-core-dev
468 2017-12-05T18:14:32  *** rafalcpp has quit IRC
469 2017-12-05T18:15:20  *** ekrok_ has quit IRC
470 2017-12-05T18:15:32  *** Victorsueca has quit IRC
471 2017-12-05T18:16:45  *** Victorsueca has joined #bitcoin-core-dev
472 2017-12-05T18:18:28  <phantomcircuit> BlueMatt, wait doesn't WaitForShutdown wait in the kernel?
473 2017-12-05T18:19:17  *** ekrok has joined #bitcoin-core-dev
474 2017-12-05T18:20:03  <BlueMatt> phantomcircuit: because signal handlers cant do "things"
475 2017-12-05T18:20:26  <BlueMatt> so eg we cant do a cv notify in the SIGTERM signal handler
476 2017-12-05T18:20:32  <BlueMatt> which is what we'd prefer to do
477 2017-12-05T18:21:04  <BlueMatt> so *something* has to check for fShutdownRequested regularly
478 2017-12-05T18:22:00  *** goatpig has quit IRC
479 2017-12-05T18:22:46  *** microapple has joined #bitcoin-core-dev
480 2017-12-05T18:24:45  *** zshlyk has quit IRC
481 2017-12-05T18:25:36  *** zshlyk has joined #bitcoin-core-dev
482 2017-12-05T18:38:37  *** SopaXorzTaker has quit IRC
483 2017-12-05T18:38:48  *** goatpig has joined #bitcoin-core-dev
484 2017-12-05T18:39:42  *** owowo has quit IRC
485 2017-12-05T18:43:18  *** BGL has joined #bitcoin-core-dev
486 2017-12-05T18:45:26  *** owowo has joined #bitcoin-core-dev
487 2017-12-05T18:57:44  *** microapple has quit IRC
488 2017-12-05T18:58:09  *** microapple has joined #bitcoin-core-dev
489 2017-12-05T18:58:25  *** microapple has quit IRC
490 2017-12-05T19:04:40  *** sanjeev has quit IRC
491 2017-12-05T19:07:45  *** zshlyk has quit IRC
492 2017-12-05T19:08:50  *** yoctopede has joined #bitcoin-core-dev
493 2017-12-05T19:12:36  *** ekrok has quit IRC
494 2017-12-05T19:14:23  <sipa> BlueMatt: did i do something wrong when merging #10773? All travis master builds since report 'HEAD was not signed with a trusted key!'
495 2017-12-05T19:14:25  <gribble> https://github.com/bitcoin/bitcoin/issues/10773 | Shell script cleanups by practicalswift · Pull Request #10773 · bitcoin/bitcoin · GitHub
496 2017-12-05T19:14:39  <BlueMatt> almost certainly, then
497 2017-12-05T19:14:40  <BlueMatt> :p
498 2017-12-05T19:14:52  <BlueMatt> sipa: this is why you're supposed to use the verify-commits pre-push-hook :(
499 2017-12-05T19:15:11  *** roidster has quit IRC
500 2017-12-05T19:15:54  <sipa> well where is the error?
501 2017-12-05T19:16:06  <sipa> it's signed with the same key as all my previous merges
502 2017-12-05T19:17:14  *** Randolf has joined #bitcoin-core-dev
503 2017-12-05T19:18:27  <sipa> BlueMatt: also, please document the pre-push things in developer-notes.md or so
504 2017-12-05T19:18:52  <BlueMatt> sipa: well the pre-push-hook is only for the committers...and I could have sworn I've yelled at all of y'all about it
505 2017-12-05T19:19:19  <sipa> then create a committers.md or so
506 2017-12-05T19:20:23  <BlueMatt> cp contrib/verify-commits/pre-push-hook.sh .git/hooks/pre-push
507 2017-12-05T19:20:43  <BlueMatt> or, if you prefer, echo 'cp contrib/verify-commits/pre-push-hook.sh .git/hooks/pre-push' > committers.md
508 2017-12-05T19:21:10  <sipa> please document it somewhere more permanently than IRC :)
509 2017-12-05T19:21:32  <sipa> haha
510 2017-12-05T19:21:53  <BlueMatt> as for travis, cant reproduce, so need to dig more, hold on
511 2017-12-05T19:24:34  <sipa> thanks!
512 2017-12-05T19:25:32  *** Babozor has quit IRC
513 2017-12-05T19:25:47  *** yoctopede has quit IRC
514 2017-12-05T19:26:37  *** yoctopede has joined #bitcoin-core-dev
515 2017-12-05T19:29:05  *** Dizzle has joined #bitcoin-core-dev
516 2017-12-05T19:29:36  *** laurentmt has quit IRC
517 2017-12-05T19:55:28  *** helloworld has joined #bitcoin-core-dev
518 2017-12-05T19:58:01  <aj> jnewbery: ping?
519 2017-12-05T19:58:12  <BlueMatt> sipa: did you extend your key's (or subkey's) expiry period?
520 2017-12-05T19:58:42  <BlueMatt> sipa: for some reason travis keeps getting a copy where the A636E97631F767E0 subkey expires on oct 19, 2017
521 2017-12-05T19:58:42  <sipa> BlueMatt: no
522 2017-12-05T19:58:47  <sipa> heh
523 2017-12-05T19:59:07  <sipa> i can't remember changing that, at least
524 2017-12-05T19:59:27  *** helloworld has quit IRC
525 2017-12-05T19:59:27  *** Victorsueca has quit IRC
526 2017-12-05T20:00:39  *** Victorsueca has joined #bitcoin-core-dev
527 2017-12-05T20:09:55  *** yoctopede has quit IRC
528 2017-12-05T20:10:42  *** yoctopede has joined #bitcoin-core-dev
529 2017-12-05T20:13:23  *** yoctopede has quit IRC
530 2017-12-05T20:14:02  *** yoctopede has joined #bitcoin-core-dev
531 2017-12-05T20:17:30  <BlueMatt> sipa: lol, did you know POSIX echo is poorly-defined?
532 2017-12-05T20:17:44  <sipa> ?
533 2017-12-05T20:18:01  <BlueMatt> sec
534 2017-12-05T20:18:58  *** JackH has quit IRC
535 2017-12-05T20:19:35  <aj> BlueMatt: "echo -e hi" ?
536 2017-12-05T20:19:57  <BlueMatt> aj: -e is implied for (some) \ things in dash, and you cant turn it off
537 2017-12-05T20:21:21  <aj> BlueMatt: yeah. use printf or printf "%s" "what you actually want to print" ...
538 2017-12-05T20:21:28  <spudowiar> BlueMatt: That is why I always use python -c "print(*__import__(\"sys\").argv[1:])" in my shell scripts
539 2017-12-05T20:21:29  <BlueMatt> yes
540 2017-12-05T20:21:38  <BlueMatt> spudowiar: lolwat
541 2017-12-05T20:21:55  *** JackH has joined #bitcoin-core-dev
542 2017-12-05T20:22:41  <spudowiar> BlueMatt: :)
543 2017-12-05T20:22:57  *** jb55 has quit IRC
544 2017-12-05T20:23:13  <spudowiar> I wonder if we should discuss #11833 here as it's turned into a conversation that might be easier to have on IRC :P
545 2017-12-05T20:23:14  <gribble> https://github.com/bitcoin/bitcoin/issues/11833 | [Net] WebSocket support · Issue #11833 · bitcoin/bitcoin · GitHub
546 2017-12-05T20:23:26  <bitcoin-git> [bitcoin] TheBlueMatt opened pull request #11834: [verify-commits] Fix gpg.sh's echoing for commits with '\n' (master...2017-12-verify-commits-fix) https://github.com/bitcoin/bitcoin/pull/11834
547 2017-12-05T20:27:06  <BlueMatt> spudowiar: lots of folks seem  like the "bitcoin has a p2p network, lets shove more features in it so that *everything* can use it!" approach, cause its easy, but it also doesnt scale....ultimately the bitcoin p2p network is full of garbage evil nodes anyway, its not magic...
548 2017-12-05T20:27:26  <bitcoin-git> [bitcoin] practicalswift opened pull request #11835: Add Travis check for unused Python imports (master...lint-python) https://github.com/bitcoin/bitcoin/pull/11835
549 2017-12-05T20:28:43  <spudowiar> BlueMatt: Awww, but I was trying to run bitcoind in the browser though :P
550 2017-12-05T20:29:30  <BlueMatt> spudowiar: please build second network of servers which provide blockchain data, it'd be useful to have redundancy for bitcoind to sync from them, even
551 2017-12-05T20:29:46  <BlueMatt> (or...whats the wallet for that again? electrum?)
552 2017-12-05T20:30:23  <spudowiar> huh?
553 2017-12-05T20:32:06  <spudowiar> Does --with-sysroot= work with Bitcoin Core?
554 2017-12-05T20:32:14  <achow101> spudowiar: why the hell would you want to run bitcoind in the browser
555 2017-12-05T20:32:23  <spudowiar> achow101: Because I can, I guess?
556 2017-12-05T20:32:30  <spudowiar> I'm a weird guy
557 2017-12-05T20:33:05  <sipa> do you mean... compile with emscripten?
558 2017-12-05T20:34:03  <spudowiar> sipa: Yeah, I'm using Emscripten and I have Bitcoin-Qt mostly working (except networking is disabled)
559 2017-12-05T20:34:07  <spudowiar> And some graphics are acting wierd
560 2017-12-05T20:34:10  <spudowiar> *weird
561 2017-12-05T20:34:23  <instagibbs> neveraskedifheshould.gif
562 2017-12-05T20:34:46  <sipa> spudowiar: that's pretty cool :)
563 2017-12-05T20:39:05  *** goatpig has quit IRC
564 2017-12-05T20:43:02  <jonasschnelli> BlueMatt: https://github.com/bitcoin/bitcoin/pull/11740/commits/f9cc6dcc65a9cd9ce72c7fbb6bde9d2efb71a8f7#r154686982
565 2017-12-05T20:43:09  <jonasschnelli> can you elaborate? Can't follow
566 2017-12-05T20:43:38  *** goatpig has joined #bitcoin-core-dev
567 2017-12-05T20:44:16  <BlueMatt> jonasschnelli: the buffer comment?
568 2017-12-05T20:44:20  <jonasschnelli> Yeah
569 2017-12-05T20:44:46  <jonasschnelli> So accepting blocks up to NODE_NETWORK_LIMITED_MIN_BLOCKS(288)+2?`
570 2017-12-05T20:45:08  <BlueMatt> yea, that
571 2017-12-05T20:45:19  *** spudowiar has quit IRC
572 2017-12-05T20:47:08  <jonasschnelli> BlueMatt: current impl.: if you tip is 1000 and someone requests height 713, you will disconnect because (1000-713 == 289)
573 2017-12-05T20:47:23  <jonasschnelli> But you want disconnect 712 (==288 == within the treshold)
574 2017-12-05T20:47:35  <jonasschnelli> BlueMatt: I don't see whats wrong with it
575 2017-12-05T20:48:21  <jonasschnelli> BlueMatt: We already have a buffer of 144 (we have 144 additional blocks for the reorg buffer)
576 2017-12-05T20:48:44  <jonasschnelli> (goal is 144 [one day], +144 reorg buffer = 288)
577 2017-12-05T20:48:45  <BlueMatt> the service bit seems to imply that you can/will serve the top 288 blocks, but if you request a block, and then they get a new block before they respond to you, you get disconnected
578 2017-12-05T20:48:49  <BlueMatt> even though you didnt violate the bip
579 2017-12-05T20:49:42  <jonasschnelli> I guess if a client detects that he will request at the very bottom of NODE_NETWORK_LIMITED_MIN_BLOCKS he needs to expect a disconnect
580 2017-12-05T20:50:01  <BlueMatt> I'd call the current implementation a violation of the BIP
581 2017-12-05T20:50:04  <jonasschnelli> BlueMatt: what you propose would essentially increase the 144 block buffer leading to a similar event just two blocks higher?
582 2017-12-05T20:50:07  <BlueMatt> "If signaled, the peer MUST be capable of serving at least the last 288 blocks "
583 2017-12-05T20:50:14  <BlueMatt> the current implementation is clearly not capable of doign so
584 2017-12-05T20:50:37  <jonasschnelli> BlueMatt: hmm? why? I can serve the last 288 blocks?
585 2017-12-05T20:50:40  <BlueMatt> if I request block at 288 deep, I do not expect to get disconnected for it, even if there is a race and you get another block at that same time
586 2017-12-05T20:50:59  <jonasschnelli> hmm..
587 2017-12-05T20:51:17  <BlueMatt> while we're on the bip (cause I just pulled it up): "A safety buffer of additional 144 blocks to handle chain reorganizations SHOULD be taken into account when connecting to a peer signaling the NODE_NETWORK_LIMITED service bit." <-- I have absolutely no idea what in the hell that means
588 2017-12-05T20:51:26  <jonasschnelli> https://github.com/bitcoin/bips/blob/master/bip-0159.mediawiki#counter-measures-for-peer-fingerprinting
589 2017-12-05T20:51:41  <jonasschnelli> Okay... your right. Got your point!
590 2017-12-05T20:52:05  <BlueMatt> I mean you're not *really* serving deeper, maybe one or two, but its certainly not a fingerprinting attack if everyone does it
591 2017-12-05T20:52:05  <jonasschnelli> Though...
592 2017-12-05T20:52:28  <jonasschnelli> But what if we use two blocks and other implementation would use a single block as additional buffer?
593 2017-12-05T20:52:41  <jonasschnelli> But... meh,.. doesn't matter
594 2017-12-05T20:52:47  <BlueMatt> then they'd already be fingerprintable because they have a different subver :p
595 2017-12-05T20:52:48  <jonasschnelli> let me just add two blocks
596 2017-12-05T20:52:53  <jonasschnelli> BlueMatt: indeed
597 2017-12-05T20:53:10  <BlueMatt> I'm still confused as to what you meant by that sentence in the BIP
598 2017-12-05T20:53:18  <BlueMatt> "A safety buffer of additional 144 blocks to handle chain reorganizations SHOULD be taken into account when connecting to a peer signaling the NODE_NETWORK_LIMITED service bit."
599 2017-12-05T20:53:23  <BlueMatt> I have no idea what that means
600 2017-12-05T20:53:45  <gmaxwell> 144? I thought that was supposted to advise 12.
601 2017-12-05T20:53:46  <sipa> i think it means you should only connect out to a LIMITED peer if you don't think you need more than 144 blocks (rather than 288)
602 2017-12-05T20:53:47  <jonasschnelli> BlueMatt: assume you are a light client and what to fetch a block within the last 144
603 2017-12-05T20:54:01  <jonasschnelli> what sipa said
604 2017-12-05T20:54:25  <BlueMatt> sipa: elaborate? I'm still lost...
605 2017-12-05T20:54:31  <BlueMatt> gmaxwell: wat?
606 2017-12-05T20:54:39  <gmaxwell> oh jesus people.
607 2017-12-05T20:54:57  <sipa> BlueMatt: i think it means that you should only signal LIMITED if you keep the last 288 blocks, but only fetch from a peer if you think you don't need more than 144
608 2017-12-05T20:55:16  <BlueMatt> sipa: yea, sorry, that half I got, the *why* part I'm still missing...
609 2017-12-05T20:55:25  <sipa> i don't know!
610 2017-12-05T20:55:25  <BlueMatt> gmaxwell: what'd we blow up?
611 2017-12-05T20:56:05  <jonasschnelli> BlueMatt: It's a kindly reminder for a reorg buffer...
612 2017-12-05T20:56:30  <jonasschnelli> BlueMatt: if you have a light client app, you need tip-150, don't connect to LIMITED
613 2017-12-05T20:56:37  <gmaxwell> BlueMatt: please read the prior discussions.
614 2017-12-05T20:56:44  <jonasschnelli> BlueMatt: but do if you need block TIP-144
615 2017-12-05T20:57:03  <jonasschnelli> (it's pure informative)
616 2017-12-05T20:57:12  <sipa> it shouldn't be just informative
617 2017-12-05T20:57:18  <jonasschnelli> Technically, it's always 288
618 2017-12-05T20:57:25  <gmaxwell> gah
619 2017-12-05T20:57:52  <sipa> gmaxwell: i vaguely remember discussions about shorter windows, but i wouldn't know where to find them
620 2017-12-05T20:57:55  <gmaxwell> look, the BIP requires you keep your last 288.  Thats it, the rest is advice.
621 2017-12-05T20:58:02  <jonasschnelli> "A safety buffer of additional 144 blocks to handle chain reorganizations SHOULD be taken into account when connecting to a peer signaling the NODE_NETWORK_LIMITED service bit."
622 2017-12-05T20:58:10  <jonasschnelli> Is purely advice... you can't set a MUST there IMO
623 2017-12-05T20:58:23  <sipa> it can be a SHOULD
624 2017-12-05T20:58:53  <gmaxwell> If a client _guesses_ it's going to need blocks 285 ago, then a LIMITED peer is quite possibly useless to it: one, it's estimate of how many it will need will not be accurate (there could have been more blocks) and even it if is, they could be on a fork from where the peer is.
625 2017-12-05T20:59:22  <BlueMatt> yes, that much I get, I'm just really confused as to where the N/2 came from there
626 2017-12-05T20:59:35  <BlueMatt> why does the buffer need to be 2x what you want?
627 2017-12-05T20:59:57  <gmaxwell> The actual correct behavior is complicated; if you're estimating your need based on nothing but time you should be pretty conservative (n/2 for example).  If you have headers and know what you need, that doesn't exist.
628 2017-12-05T21:00:27  <BlueMatt> ok, it just seemed to be weird to be so specific...
629 2017-12-05T21:02:36  <gmaxwell> I think the 144 number comes more out of "if we want to be able to reliably serve 144, then our target will need to be higher, e.g. 288)
630 2017-12-05T21:02:49  *** yoctopede has quit IRC
631 2017-12-05T21:03:42  *** yoctopede has joined #bitcoin-core-dev
632 2017-12-05T21:03:52  <BlueMatt> ok, fair
633 2017-12-05T21:06:33  <BlueMatt> it just reads funny, was my point :p
634 2017-12-05T21:06:57  <BlueMatt> jonasschnelli: wanna just squash and I'll ack
635 2017-12-05T21:07:19  <jonasschnelli> BlueMatt: Yeah. Just squashing (had to rebase anyways)
636 2017-12-05T21:07:23  <jonasschnelli> Give me a min
637 2017-12-05T21:11:46  <BlueMatt> suresure
638 2017-12-05T21:21:36  <jonasschnelli> BlueMatt: done
639 2017-12-05T21:26:00  <BlueMatt> lol, how's this for an idea: try to load libsensors and if it loads write cpu temperateure to debug.log on a regular basis
640 2017-12-05T21:26:44  *** yoctopede has quit IRC
641 2017-12-05T21:27:36  *** yoctopede has joined #bitcoin-core-dev
642 2017-12-05T21:29:10  *** dermoth has quit IRC
643 2017-12-05T21:32:38  *** jb55 has joined #bitcoin-core-dev
644 2017-12-05T21:33:10  <bitcoin-git> [bitcoin] sipa pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/91eeaa03354b...5bea05bc1d17
645 2017-12-05T21:33:10  <bitcoin-git> bitcoin/master a38686c Matt Corallo: [verify-commits] Fix gpg.sh's echoing for commits with '\n'
646 2017-12-05T21:33:10  <bitcoin-git> bitcoin/master 5bea05b Pieter Wuille: Merge #11834: [verify-commits] Fix gpg.sh's echoing for commits with '\n'...
647 2017-12-05T21:33:20  *** wraithm has quit IRC
648 2017-12-05T21:33:45  <bitcoin-git> [bitcoin] sipa closed pull request #11834: [verify-commits] Fix gpg.sh's echoing for commits with '\n' (master...2017-12-verify-commits-fix) https://github.com/bitcoin/bitcoin/pull/11834
649 2017-12-05T21:33:47  *** Murch has quit IRC
650 2017-12-05T21:34:02  <gmaxwell> BlueMatt: oh no, moar dependencies.
651 2017-12-05T21:34:13  <jimpo> Is there a preferred implementation of union/variant types? I know about boost::variant, but I've heard Core is trying to use boost less.
652 2017-12-05T21:34:22  <BlueMatt> gmaxwell: well you could do a if-you-cant-find-dep-dont-print-temp
653 2017-12-05T21:34:48  <BlueMatt> gmaxwell: but it'd be so nice in debugging peoples' rejecting-chain issues to see debug.log entries that say "WARNING: My hardware is overheating garbage, dont trust output"
654 2017-12-05T21:34:50  <sipa> jimpo: std::variant in c++17 ? :)
655 2017-12-05T21:35:06  <jimpo> Oh, is C++17 allowed/enabled?
656 2017-12-05T21:35:11  <BlueMatt> lol no
657 2017-12-05T21:35:16  <jimpo> derp
658 2017-12-05T21:35:16  <sipa> no
659 2017-12-05T21:35:28  <sipa> it was just officially published... today
660 2017-12-05T21:35:31  <BlueMatt> is it even in debian-testing?
661 2017-12-05T21:35:42  <BlueMatt> i think clang supports it
662 2017-12-05T21:35:45  <sipa> needs GCC 7
663 2017-12-05T21:36:23  <sipa> and Clang 5
664 2017-12-05T21:36:30  <jimpo> Ah, that explains why std::variant came up in searches today and I didn't see it a month ago
665 2017-12-05T21:37:02  <sipa> to be fair, the last month or so was just formally for ISO to publish it
666 2017-12-05T21:37:11  <sipa> it was exactly known what the spec would be
667 2017-12-05T21:37:17  <jimpo> So is a ghetto struct with an enum + union preferred to boost::variant?
668 2017-12-05T21:38:27  <sipa> i think you can use boost::variant, as we're already using it anyway
669 2017-12-05T21:38:32  <jimpo> k
670 2017-12-05T21:38:48  <BlueMatt> if you can keep it in a cpp, use boost::variant, if its in an infection header....ugh, but maybe ok
671 2017-12-05T21:51:58  *** rabidus has quit IRC
672 2017-12-05T21:54:55  *** dqx has joined #bitcoin-core-dev
673 2017-12-05T21:56:35  *** Chris_Stewart_5 has quit IRC
674 2017-12-05T22:00:35  *** rabidus has joined #bitcoin-core-dev
675 2017-12-05T22:07:08  *** Murch has joined #bitcoin-core-dev
676 2017-12-05T22:07:33  *** Victorsueca has quit IRC
677 2017-12-05T22:08:39  *** Murch has quit IRC
678 2017-12-05T22:08:45  *** Victorsueca has joined #bitcoin-core-dev
679 2017-12-05T22:13:33  *** Murch has joined #bitcoin-core-dev
680 2017-12-05T22:14:46  *** yoctopede has quit IRC
681 2017-12-05T22:15:22  *** Murch has quit IRC
682 2017-12-05T22:16:53  *** Murch has joined #bitcoin-core-dev
683 2017-12-05T22:16:54  *** yoctopede has joined #bitcoin-core-dev
684 2017-12-05T22:17:51  *** rhavar has quit IRC
685 2017-12-05T22:27:35  *** DvdKhl has quit IRC
686 2017-12-05T22:31:08  *** arubi has quit IRC
687 2017-12-05T22:35:43  *** Guyver2 has quit IRC
688 2017-12-05T22:35:51  *** belcher_ has joined #bitcoin-core-dev
689 2017-12-05T22:35:52  *** arubi has joined #bitcoin-core-dev
690 2017-12-05T22:36:23  *** belcher_ has quit IRC
691 2017-12-05T22:37:32  <luke-jr> jimpo: boost is fine outside of libconsensus I think
692 2017-12-05T22:38:54  <GAit> patches to get core to build with the ndk (working on android shell with no deps or glibc/LD_LIBRARY_PATH hacks) if anyone is interested can be reviewed at https://github.com/greenaddress/bitcoin_ndk  - warning, just like abcore, these patches are also very alpha and currently I don't think there's anything worth making an upstream PR for. and fwiw later ndk releases seem to fail.
693 2017-12-05T22:39:20  *** belcher has quit IRC
694 2017-12-05T22:40:12  <GAit> mostly deals with setting up the toolchain and patching some different headers or things missing in android (ifaddrs for example)
695 2017-12-05T22:55:40  *** arubi has quit IRC
696 2017-12-05T22:56:26  *** arubi has joined #bitcoin-core-dev
697 2017-12-05T22:56:42  *** roadcrap has quit IRC
698 2017-12-05T22:57:06  *** dqx has quit IRC
699 2017-12-05T23:01:45  *** Cogito_Ergo_Sum has quit IRC
700 2017-12-05T23:02:06  *** Randolf has quit IRC
701 2017-12-05T23:07:30  *** meshcollider has joined #bitcoin-core-dev
702 2017-12-05T23:07:55  *** yoctopede has quit IRC
703 2017-12-05T23:08:46  *** yoctopede has joined #bitcoin-core-dev
704 2017-12-05T23:09:51  *** shesek has joined #bitcoin-core-dev
705 2017-12-05T23:09:51  *** shesek has joined #bitcoin-core-dev
706 2017-12-05T23:10:44  <TD-Linux> BlueMatt, I think a built-in memtest would be a better use of time :)
707 2017-12-05T23:11:08  <BlueMatt> TD-Linux: that is almost certainly true
708 2017-12-05T23:16:47  *** yoctopede has quit IRC
709 2017-12-05T23:17:38  *** yoctopede has joined #bitcoin-core-dev
710 2017-12-05T23:25:11  *** ghost43 has quit IRC
711 2017-12-05T23:25:45  *** ghost43 has joined #bitcoin-core-dev
712 2017-12-05T23:28:24  <phantomcircuit> iirc there are user space memtest utilities but i dont see how they can actually be effective unless they use all available memory
713 2017-12-05T23:30:21  <phantomcircuit> BlueMatt, ^
714 2017-12-05T23:31:18  <BlueMatt> I think, or at least have a suspicion, that half our user-memory-corruption errors are due to cpu overheating and writing garbage
715 2017-12-05T23:31:57  <BlueMatt> but that isnt based on much and may be very bogus
716 2017-12-05T23:32:19  <phantomcircuit> hmm
717 2017-12-05T23:32:31  <phantomcircuit> indeed that would be caught with a pretty simple test
718 2017-12-05T23:32:47  <phantomcircuit> just load some test vectors and check siphash results
719 2017-12-05T23:33:36  <BlueMatt> yea, I mean its a shame we have assumevalid now...otherwise I'd suggest just busywaiting one or two of the script check threads with something like that
720 2017-12-05T23:36:27  *** Giszmo has quit IRC
721 2017-12-05T23:37:50  *** jb55 has quit IRC
722 2017-12-05T23:39:47  *** Randolf has joined #bitcoin-core-dev
723 2017-12-05T23:46:24  *** treeuntor has joined #bitcoin-core-dev
724 2017-12-05T23:51:12  *** Giszmo has joined #bitcoin-core-dev
725 2017-12-05T23:52:33  *** gonzazoid has joined #bitcoin-core-dev
726 2017-12-05T23:53:31  *** Evel-Knievel has quit IRC
727 2017-12-05T23:53:54  *** str4d has joined #bitcoin-core-dev
728 2017-12-05T23:54:29  *** shesek has quit IRC
729 2017-12-05T23:56:02  <gmaxwell> we could make block validation fails sleep for a second then try again... and log&crash if successful the second time.
730 2017-12-05T23:56:13  *** jb55 has joined #bitcoin-core-dev
731 2017-12-05T23:58:24  *** shesek has joined #bitcoin-core-dev
732 2017-12-05T23:58:24  *** shesek has joined #bitcoin-core-dev
733 2017-12-05T23:59:20  *** gonzazoid has quit IRC