1 2018-02-13T00:06:23  *** lnostdal has quit IRC
  2 2018-02-13T00:14:44  *** mirese_ has joined #bitcoin-core-dev
  3 2018-02-13T00:16:37  *** lnostdal has joined #bitcoin-core-dev
  4 2018-02-13T00:17:46  *** mirese has quit IRC
  5 2018-02-13T00:20:34  *** meshcollider has quit IRC
  6 2018-02-13T00:26:21  *** rex_4539 has quit IRC
  7 2018-02-13T00:33:21  *** Randolf has joined #bitcoin-core-dev
  8 2018-02-13T00:50:17  *** StopAndDecrypt has joined #bitcoin-core-dev
  9 2018-02-13T00:57:21  *** Randolf has quit IRC
 10 2018-02-13T00:58:19  *** dabura667 has joined #bitcoin-core-dev
 11 2018-02-13T01:02:24  *** meshcollider has joined #bitcoin-core-dev
 12 2018-02-13T01:09:56  *** Jacinthe3Hermist has quit IRC
 13 2018-02-13T01:13:27  *** grafcaps has quit IRC
 14 2018-02-13T01:15:53  *** RezusMinus has joined #bitcoin-core-dev
 15 2018-02-13T01:19:05  *** jonasschnelli has quit IRC
 16 2018-02-13T01:19:47  <ProfMac> was there ever a time when GetBlockWork used the actual hash instead of nBits to calculate work?
 17 2018-02-13T01:20:05  *** wolfspraul has quit IRC
 18 2018-02-13T01:20:12  *** wolfspraul has joined #bitcoin-core-dev
 19 2018-02-13T01:20:16  <bitcoin-git> [bitcoin] promag opened pull request #12419: wallet: Force distinct destinations in CWallet::CreateTransaction (master...2018-02-distinct-destinations) https://github.com/bitcoin/bitcoin/pull/12419
 20 2018-02-13T01:20:16  <gmaxwell> no, that wouldn't be correct.
 21 2018-02-13T01:20:29  <ProfMac> tell me more.
 22 2018-02-13T01:20:37  *** jonasschnelli has joined #bitcoin-core-dev
 23 2018-02-13T01:21:48  *** dabura667_ has joined #bitcoin-core-dev
 24 2018-02-13T01:21:54  *** dabura667 has quit IRC
 25 2018-02-13T01:22:13  <gmaxwell> the hash reflects more than actual work, by a factor of two on average.  It also has very high variance. meaning that if you got a block with ten times that 'work' expected (which you'd get in 1/10) blocks you could just delay announcing it until a couple blocks had passed, then announce it and be very likely to reorg them out.
 26 2018-02-13T01:22:48  <promag> gmaxwell: regarding duplicate outputs #12419
 27 2018-02-13T01:22:49  <gribble> https://github.com/bitcoin/bitcoin/issues/12419 | Force distinct destinations in CWallet::CreateTransaction by promag · Pull Request #12419 · bitcoin/bitcoin · GitHub
 28 2018-02-13T01:23:40  <ProfMac> Yes, gmaxwell, I agree with all that.
 29 2018-02-13T01:23:51  <ProfMac> Thanks.
 30 2018-02-13T01:24:23  <ProfMac> Not unlike "curling" or "corn hole"
 31 2018-02-13T01:29:27  *** grafcaps has joined #bitcoin-core-dev
 32 2018-02-13T01:36:17  *** wacawacawaca has joined #bitcoin-core-dev
 33 2018-02-13T01:37:35  *** justanotheruser has joined #bitcoin-core-dev
 34 2018-02-13T01:38:36  *** contrapumpkin is now known as Whale
 35 2018-02-13T01:38:53  *** Whale is now known as contrapumpkin
 36 2018-02-13T01:39:32  *** contrapumpkin is now known as Whale
 37 2018-02-13T01:39:39  *** Whale is now known as contrapumpkin
 38 2018-02-13T01:53:21  *** belcher has quit IRC
 39 2018-02-13T02:34:26  <gmaxwell> n1bor: conversation continued in #bitcoin he posted bench logs which indicated to me that he was IO bound very badly. Today he followed up with,
 40 2018-02-13T02:34:29  <gmaxwell> n1bor: 18:28:24 < murrayn> gmaxwell, just to follow up from last night: 1) i shutdown gracefully and restarted, and the reindex started from the beginning.  From what you said, it shouldn't do that? and 2) it was definitely disk IO bound. doing it on an SSD this time. much better
 41 2018-02-13T02:34:48  <gmaxwell> (1) was something sipa warned him about in here, that restarting with -reindex-chainstate would restart the reindex.
 42 2018-02-13T02:36:34  *** TheRec_ has joined #bitcoin-core-dev
 43 2018-02-13T02:36:47  *** TheRec has quit IRC
 44 2018-02-13T02:38:51  *** AaronvanW has quit IRC
 45 2018-02-13T02:39:21  <midnightmagic> doh
 46 2018-02-13T02:40:44  *** lnostdal has quit IRC
 47 2018-02-13T02:43:55  *** Randolf has joined #bitcoin-core-dev
 48 2018-02-13T02:55:50  *** jamesob has joined #bitcoin-core-dev
 49 2018-02-13T03:05:03  *** mandric has quit IRC
 50 2018-02-13T03:10:44  <meshcollider> looks like contrib/bitcoin-cli.bash-completion needs an update, hasn't been updated for ages
 51 2018-02-13T03:24:10  <esotericnonsense> k, so i'm looking at the JSON-RPC 2.0 spec stuff now. 'strict compliance' requires two main changes as far as i can see. one of them I've made gated behind a flag 'strictjsonrpcspec' (better names accepted)
 52 2018-02-13T03:24:29  *** StopAndDecrypt has quit IRC
 53 2018-02-13T03:24:29  *** StopAndDecrypt has joined #bitcoin-core-dev
 54 2018-02-13T03:24:41  <esotericnonsense> that's the 'client must send jsonrpc=2.0, server must send jsonrpc=2.0, only one of 'error' or 'result' should be returned'
 55 2018-02-13T03:25:32  <esotericnonsense> the other bit is that if the client does not send an id with a request, the server should send a blank response. within a batch it should just be skipped. so if you send [a id=1, b, c id=3] you should get back [aresult, cresult].
 56 2018-02-13T03:26:14  *** StopAndDecrypt has quit IRC
 57 2018-02-13T03:26:54  <esotericnonsense> i think that's enough to make it fully spec compliant. the former one is enough to get it to work with libjsonrpccpp, could probably write some unit tests based on the jsonrpc spec as well, manual tests i've done look fine
 58 2018-02-13T03:27:15  *** StopAndDecrypt has joined #bitcoin-core-dev
 59 2018-02-13T03:27:15  *** StopAndDecrypt has quit IRC
 60 2018-02-13T03:27:15  *** StopAndDecrypt has joined #bitcoin-core-dev
 61 2018-02-13T03:28:10  *** Giszmo has quit IRC
 62 2018-02-13T03:29:52  *** StopAndDecrypt has quit IRC
 63 2018-02-13T03:30:26  *** StopAndDecrypt has joined #bitcoin-core-dev
 64 2018-02-13T03:30:39  *** StopAndDecrypt has quit IRC
 65 2018-02-13T03:31:46  *** StopAndDecrypt has joined #bitcoin-core-dev
 66 2018-02-13T03:34:40  *** dongcarl has joined #bitcoin-core-dev
 67 2018-02-13T03:37:42  *** jamesob has quit IRC
 68 2018-02-13T03:39:32  *** zautomata1 has joined #bitcoin-core-dev
 69 2018-02-13T03:41:05  *** zautomata has quit IRC
 70 2018-02-13T03:54:28  <bitcoin-git> [bitcoin] MarcoFalke reopened pull request #12349: shutdown: fix crash on shutdown with reindex-chainstate (master...fix-qt-shutdown) https://github.com/bitcoin/bitcoin/pull/12349
 71 2018-02-13T04:11:45  *** shesek has quit IRC
 72 2018-02-13T04:23:51  *** dongcarl_ has joined #bitcoin-core-dev
 73 2018-02-13T04:24:33  *** dongcarl has quit IRC
 74 2018-02-13T04:41:28  *** shesek has joined #bitcoin-core-dev
 75 2018-02-13T04:41:28  *** shesek has joined #bitcoin-core-dev
 76 2018-02-13T04:54:49  <kallewoof> What's going on here? Either that comment is wrong, or the check is wrong... https://github.com/bitcoin/bitcoin/blame/master/src/script/interpreter.cpp#L1233 (it says nOut out of range, but it checks nIn against txTo.vout.size()...)
 77 2018-02-13T04:57:12  <kallewoof> I guess it's because single requires the in to sign the same index out. Just weird looking.
 78 2018-02-13T05:06:28  <aj> kallewoof: yeah, "no vout corresponding to nIn" or something would make more sense to me as a comment
 79 2018-02-13T05:06:38  <aj> kallewoof: code seems right at least
 80 2018-02-13T05:07:18  *** RezusMinus has quit IRC
 81 2018-02-13T05:09:39  <aj> kallewoof: wow, that comment used to be a log output saying nOut out of range
 82 2018-02-13T05:13:29  *** cryptojanitor has quit IRC
 83 2018-02-13T05:14:37  <kallewoof> aj: yeah agreed!
 84 2018-02-13T05:15:42  *** goatpig has joined #bitcoin-core-dev
 85 2018-02-13T05:18:46  *** rex_4539 has joined #bitcoin-core-dev
 86 2018-02-13T05:36:21  *** mmgen has joined #bitcoin-core-dev
 87 2018-02-13T05:53:22  *** Sinclair_ has joined #bitcoin-core-dev
 88 2018-02-13T05:55:12  *** Sinclair6 has quit IRC
 89 2018-02-13T06:15:37  *** sengehest has joined #bitcoin-core-dev
 90 2018-02-13T06:22:10  *** dongcarl_ has quit IRC
 91 2018-02-13T06:23:23  *** murrayn has quit IRC
 92 2018-02-13T06:32:14  *** sengehest has quit IRC
 93 2018-02-13T06:32:59  *** rex_4539 has quit IRC
 94 2018-02-13T07:00:27  *** kallewoof has quit IRC
 95 2018-02-13T07:04:35  *** kallewoof has joined #bitcoin-core-dev
 96 2018-02-13T07:15:05  *** tripleslash has quit IRC
 97 2018-02-13T07:15:33  *** mmgen has quit IRC
 98 2018-02-13T07:16:39  *** Varunram has quit IRC
 99 2018-02-13T07:16:39  *** Varunram has joined #bitcoin-core-dev
100 2018-02-13T07:16:39  *** Varunram has joined #bitcoin-core-dev
101 2018-02-13T07:20:05  *** tripleslash has joined #bitcoin-core-dev
102 2018-02-13T07:30:10  *** bsm117532 has quit IRC
103 2018-02-13T07:43:44  *** dongcarl has joined #bitcoin-core-dev
104 2018-02-13T07:53:23  *** mmgen has joined #bitcoin-core-dev
105 2018-02-13T08:01:30  *** jonasschnelli has quit IRC
106 2018-02-13T08:01:30  *** jonasschnelli has joined #bitcoin-core-dev
107 2018-02-13T08:04:35  *** murrayn has joined #bitcoin-core-dev
108 2018-02-13T08:04:35  *** murrayn has joined #bitcoin-core-dev
109 2018-02-13T08:05:05  *** merehap has quit IRC
110 2018-02-13T08:09:23  *** merehap has joined #bitcoin-core-dev
111 2018-02-13T08:25:50  *** dongcarl has quit IRC
112 2018-02-13T08:29:21  *** baldur has quit IRC
113 2018-02-13T08:49:37  *** n1bor has quit IRC
114 2018-02-13T08:50:01  *** n1bor has joined #bitcoin-core-dev
115 2018-02-13T09:00:09  *** timothy has joined #bitcoin-core-dev
116 2018-02-13T09:18:19  *** timothy has quit IRC
117 2018-02-13T09:24:49  *** timothy has joined #bitcoin-core-dev
118 2018-02-13T09:32:07  <bitcoin-git> [bitcoin] laanwj pushed 5 new commits to master: https://github.com/bitcoin/bitcoin/compare/c997f8808256...2dbc4a4740cd
119 2018-02-13T09:32:08  <bitcoin-git> bitcoin/master cc046f6 John Newbery: [tests] Reduce NodeConn connection logging from info to debug
120 2018-02-13T09:32:08  <bitcoin-git> bitcoin/master c32cf9f John Newbery: [tests] Add P2PDataStore class...
121 2018-02-13T09:32:09  <bitcoin-git> bitcoin/master 359d067 John Newbery: [tests] Fix flake8 warnings in invalidtxrequest
122 2018-02-13T09:32:37  <bitcoin-git> [bitcoin] laanwj closed pull request #11771:  [tests] Change invalidtxrequest to use BitcoinTestFramework (master...refactor_invalidtxrequest) https://github.com/bitcoin/bitcoin/pull/11771
123 2018-02-13T09:59:26  <bitcoin-git> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/2dbc4a4740cd...f4f4f51f1a94
124 2018-02-13T09:59:26  <bitcoin-git> bitcoin/master a71c56a Luke Dashjr: clientversion: Use full commit hash for commit-based version descriptions...
125 2018-02-13T09:59:27  <bitcoin-git> bitcoin/master f4f4f51 Wladimir J. van der Laan: Merge #11966: clientversion: Use full commit hash for commit-based version descriptions...
126 2018-02-13T10:00:04  <bitcoin-git> [bitcoin] laanwj closed pull request #11966: clientversion: Use full commit hash for commit-based version descriptions (master...ver_full_commit_hash) https://github.com/bitcoin/bitcoin/pull/11966
127 2018-02-13T10:02:45  *** timothy has quit IRC
128 2018-02-13T10:17:13  *** timothy has joined #bitcoin-core-dev
129 2018-02-13T10:28:05  *** dabura667_ has quit IRC
130 2018-02-13T10:37:30  *** Scrat has joined #bitcoin-core-dev
131 2018-02-13T10:37:50  *** baldur has joined #bitcoin-core-dev
132 2018-02-13T10:39:15  *** laurentmt has joined #bitcoin-core-dev
133 2018-02-13T10:40:18  <wumpus> hmmm
134 2018-02-13T10:40:19  <wumpus> util.cpp:384:121: note: in instantiation of template class 'std::__1::pair<const std::__1::basic_string<char>, boost::interprocess::file_lock>' requested here
135 2018-02-13T10:40:19  <wumpus>         boost::interprocess::file_lock& lock = locks.emplace(pathLockFile.string(), pathLockFile.string().c_str()).first->second;
136 2018-02-13T10:41:22  <wumpus> full error here: https://github.com/bitcoin/bitcoin/issues/12413
137 2018-02-13T10:41:37  <wumpus> anyone have an idea what can cause this c++ error monstriosity?
138 2018-02-13T10:46:38  *** larafale has joined #bitcoin-core-dev
139 2018-02-13T10:53:14  *** Victorsueca has quit IRC
140 2018-02-13T10:54:49  *** Victorsueca has joined #bitcoin-core-dev
141 2018-02-13T11:03:25  <wumpus> so it looks like "static std::map<std::string, boost::interprocess::file_lock> locks;" is illegal with openbsd's clang + boost version combo
142 2018-02-13T11:03:40  <bitcoin-git> [bitcoin] Sjors opened pull request #12421: [qt] navigate to  transaction history page after send (master...2018/02/qt-goto-transactions-after-send) https://github.com/bitcoin/bitcoin/pull/12421
143 2018-02-13T11:04:40  <Lauda> Will there be rc4?
144 2018-02-13T11:05:13  *** epic has quit IRC
145 2018-02-13T11:05:36  *** epic has joined #bitcoin-core-dev
146 2018-02-13T11:08:47  *** belcher has joined #bitcoin-core-dev
147 2018-02-13T11:11:29  <provoostenator> Lauda: there's two open PR's which I suspect need to be fixed before rc4: https://github.com/bitcoin/bitcoin/milestone/30
148 2018-02-13T11:12:29  <Lauda> Alright, thanks. I'll wait for those.
149 2018-02-13T11:17:31  <bitcoin-git> [bitcoin] laanwj opened pull request #12422: util: Use unique_ptr in locks map (master...2018_01_openbsd_util_fix) https://github.com/bitcoin/bitcoin/pull/12422
150 2018-02-13T11:18:14  *** Guyver2 has joined #bitcoin-core-dev
151 2018-02-13T11:24:45  <gmaxwell> wumpus: can you explain why it works inside a unique ptr? (I am just curious)
152 2018-02-13T11:26:03  <wumpus> gmaxwell: no, I have no idea why it fails in the first place
153 2018-02-13T11:26:28  <wumpus> tbh this is really over my limit of understanding C++
154 2018-02-13T11:27:32  <wumpus> well, the point of having pointers in a map instead of whole objects makes it 'simpler' to work with layout in memory I suppose
155 2018-02-13T11:28:03  <wumpus> as for some reason the whole lock objects can't be moved
156 2018-02-13T11:28:12  <wumpus> and pointers, obviously, can
157 2018-02-13T11:28:59  <gmaxwell> yea, I was wondering if values in maps had to be relocatable, but quick googling didn't answer the question for me.
158 2018-02-13T11:29:56  <wumpus> it must be some edge case that was solved in either a newer version of clang, or in a different version of boost, as I've not seen the issue on other platforms
159 2018-02-13T11:30:53  <wumpus> the 'map' here is used to avoid locking a directory multiple times I guess? this structure is write-only, no lookups are ever done in this map, so there should be no performance difference whatsoever
160 2018-02-13T11:31:00  <wumpus> gah
161 2018-02-13T11:31:20  <goatpig> from the look of the error log
162 2018-02-13T11:31:24  <goatpig> this has to do with the ctor
163 2018-02-13T11:31:27  <goatpig> not the map itself
164 2018-02-13T11:31:41  <wumpus> so emplace returns the current record if one already exists, right?
165 2018-02-13T11:31:46  *** hsmiths has quit IRC
166 2018-02-13T11:32:02  *** hsmiths has joined #bitcoin-core-dev
167 2018-02-13T11:32:14  <wumpus> I dont' think this construction even accomplishes what it is meant to accomplish
168 2018-02-13T11:32:59  <goatpig> emplace is like insert afaik, returns a pair of <iter, bool>
169 2018-02-13T11:33:20  <goatpig> the iterator points at the inserted element or the already existing one, the bool tells you if it was added (true) or fetched (false)
170 2018-02-13T11:34:18  <wumpus> ok that depends on what does lock->try_lock does on a lock that is already locked
171 2018-02-13T11:34:29  <goatpig> you mean within the same thread?
172 2018-02-13T11:34:40  <goatpig> or another one?
173 2018-02-13T11:34:40  <wumpus> within the same process
174 2018-02-13T11:34:45  <goatpig> i think it throws
175 2018-02-13T11:34:46  <goatpig> not sure
176 2018-02-13T11:34:48  <wumpus> this is a per-process lock, not per-thread
177 2018-02-13T11:35:01  *** timothy has quit IRC
178 2018-02-13T11:35:14  <wumpus> I wonder what the intent is for LockDirectory being called on the same directory multiple times
179 2018-02-13T11:35:25  <goatpig> but that wouldnt have anything to do compile, that would just be runtime error
180 2018-02-13T11:35:31  <wumpus> I know.
181 2018-02-13T11:35:40  <wumpus> I'm just doubting that this code does what it is meant to do, at all
182 2018-02-13T11:35:53  <goatpig> typically that's to make sure you got the lock
183 2018-02-13T11:35:56  <wumpus> separate from what it takes to compile it
184 2018-02-13T11:36:04  <goatpig> now if the same threads is trying to get a lock multiple times
185 2018-02-13T11:36:25  <goatpig> it presupposes the methods that are accessed can be used on their own
186 2018-02-13T11:36:33  <wumpus> it should probably at least check the boolean it gets back from emplace
187 2018-02-13T11:36:40  <wumpus> whether the lock already exists
188 2018-02-13T11:36:54  <wumpus> otherwise it will lock an old lock another time
189 2018-02-13T11:36:56  <goatpig> that part is like
190 2018-02-13T11:36:59  <goatpig> idk id have to look at the code
191 2018-02-13T11:37:02  <goatpig> but it seems pointless
192 2018-02-13T11:37:14  <goatpig> why hold a reference to the lock in the scope? that's usless, the map perpetuates it
193 2018-02-13T11:37:16  <wumpus> either that's a no-op, or it throws, but it's pointless
194 2018-02-13T11:37:33  <goatpig> and that could be your compile issue
195 2018-02-13T11:37:37  <goatpig> that these locks are not copiable
196 2018-02-13T11:37:45  <goatpig> ie just get rid of the referencing altgother
197 2018-02-13T11:37:46  <wumpus> defining the map itself causes the compile issue
198 2018-02-13T11:37:53  <wumpus> referencing is not the problem
199 2018-02-13T11:37:56  <goatpig> boost::interprocess::file_lock& lock = locks.emplace(pathLockFile.string(), pathLockFile.string().c_str()).first->second;
200 2018-02-13T11:38:03  <goatpig> get rid of the reference and try
201 2018-02-13T11:38:06  <gmaxwell> in general I wouldn't expect a lock to be copyable.
202 2018-02-13T11:38:07  <goatpig> if that doesnt bork other stuff
203 2018-02-13T11:38:16  <wumpus> goatpig: I tried, see discussion in https://github.com/bitcoin/bitcoin/issues/12413
204 2018-02-13T11:38:19  <goatpig> i.e. just do locks.emplace(pathLockFile.string(), pathLockFile.string().c_str());
205 2018-02-13T11:38:31  <gmaxwell> what the heck is this doing?
206 2018-02-13T11:38:33  <gmaxwell> anyways?
207 2018-02-13T11:38:43  <wumpus> goatpig: as I said above, defining the map itself already causes the error, even if you don't access it at all!
208 2018-02-13T11:39:10  <wumpus> gmaxwell: that was also my question, what is the semantics supposed to be
209 2018-02-13T11:39:27  <goatpig> you could just ninja that my making it a map of shared_ptr<lock>
210 2018-02-13T11:39:33  <goatpig> by*
211 2018-02-13T11:39:47  <gmaxwell> maybe we can remove it all entirely? thats always the best fix when possible. :P
212 2018-02-13T11:39:51  <wumpus> goatpig: that's what I did in #12322 (but with unique_lock)
213 2018-02-13T11:39:52  <gribble> https://github.com/bitcoin/bitcoin/issues/12322 | Docs: Remove step making cloned repository world-writable for Windows build. by murrayn · Pull Request #12322 · bitcoin/bitcoin · GitHub
214 2018-02-13T11:40:00  <wumpus> eh #12422
215 2018-02-13T11:40:02  <gribble> https://github.com/bitcoin/bitcoin/issues/12422 | util: Use unique_ptr in locks map (fix OpenBSD 6.2 build) by laanwj · Pull Request #12422 · bitcoin/bitcoin · GitHub
216 2018-02-13T11:40:08  *** jl2012 has quit IRC
217 2018-02-13T11:40:19  * gmaxwell channels some contributor and considers just submitting a PR to remove stuff he doesn't understand
218 2018-02-13T11:40:25  *** jl2012 has joined #bitcoin-core-dev
219 2018-02-13T11:40:48  <goatpig> wumpus: sure, dunno what the code really intents. i'd prefer unique_ptr when possible
220 2018-02-13T11:40:59  <goatpig> but if it's trying to reference the lock, then it should be using shared_ptrs
221 2018-02-13T11:41:08  <wumpus> no, shared_ptre is not necessary
222 2018-02-13T11:41:11  *** wacawacawaca has quit IRC
223 2018-02-13T11:41:14  <wumpus> the lock does not leave the scope at all
224 2018-02-13T11:41:25  <goatpig> the lock exists within the map
225 2018-02-13T11:41:29  <wumpus> yes
226 2018-02-13T11:41:47  <wumpus> and after the change, it exists inside a unique pointer
227 2018-02-13T11:41:57  <goatpig> if you kill the map, if there is a thread with the ptr/reference, it will blow up
228 2018-02-13T11:41:59  <wumpus> which is unique, inside the map
229 2018-02-13T11:42:10  <wumpus> no one has a reference to it
230 2018-02-13T11:42:11  *** timothy has joined #bitcoin-core-dev
231 2018-02-13T11:42:14  <wumpus> please read the code
232 2018-02-13T11:42:21  <wumpus> no reference leaves the function
233 2018-02-13T11:43:00  <goatpig> checking
234 2018-02-13T11:43:02  <wumpus> the destructor will be called when the process has shut down, after the main function, by automatic cleanup
235 2018-02-13T11:43:12  <wumpus> by that time no one will be using the data dir anymore
236 2018-02-13T11:43:49  <wumpus> the map exists to hold on to the objects until the process shuts down
237 2018-02-13T11:45:57  <goatpig> so the locks persist the process?
238 2018-02-13T11:46:01  <goatpig> or rather
239 2018-02-13T11:46:11  <goatpig> no method tries to clean them up
240 2018-02-13T11:46:37  <wumpus> as I said, they'll be automatically cleaned up by the runtime framework
241 2018-02-13T11:47:34  <wumpus> <gmaxwell> in general I wouldn't expect a lock to be copyable. <- they are; it's only moving them at most, not copying
242 2018-02-13T11:50:11  <goatpig> wumpus: is it assumed that code gets accessed by only a single thread?
243 2018-02-13T11:50:21  <gmaxwell> goatpig: this isn't an ordinary lock
244 2018-02-13T11:50:26  <gmaxwell> it's a file locking thing
245 2018-02-13T11:50:36  <wumpus> goatpig: yes, it's only used during initialization, by the init thread
246 2018-02-13T11:50:42  <gmaxwell> used to keep seperate processes from muckign about with the same data directories.
247 2018-02-13T11:50:50  <wumpus> goatpig: oh... I'm actually not sure anymore
248 2018-02-13T11:50:51  <goatpig> i understand that much, but it's not thread safe, so im assuming this is every hit once
249 2018-02-13T11:51:01  <goatpig> err
250 2018-02-13T11:51:02  *** dgy_ has quit IRC
251 2018-02-13T11:51:02  <wumpus> wallet directories complicated this
252 2018-02-13T11:51:03  <goatpig> hit ever once
253 2018-02-13T11:51:23  <goatpig> i mean the semantics of the method suggest this is accessed just to check the lock is held at all
254 2018-02-13T11:51:23  <wumpus> as soon as dynamic opening of wallets is supported, this can be called from the RPC thread
255 2018-02-13T11:51:33  <wumpus> but right now it's only used from init
256 2018-02-13T11:52:18  <wumpus> it's a good question, the function as it is is certainly not thread safe
257 2018-02-13T11:52:22  <goatpig> you are adding elements to a map. ideally it should check the map has the element, otherwise grab a mutex, add the element, then go back to regular execution
258 2018-02-13T11:52:23  *** timothy has quit IRC
259 2018-02-13T11:53:08  <wumpus> it should grab a mutex before accessing the map at all
260 2018-02-13T11:53:12  <gmaxwell> ^
261 2018-02-13T11:53:14  <wumpus> *if* it's supposed to be thread safe
262 2018-02-13T11:53:19  <goatpig> ok, sure
263 2018-02-13T11:53:38  <gmaxwell> and its trivial here in any case.. since the purpose of the map is just lifetime management for the contained objects.
264 2018-02-13T11:53:45  <wumpus> yes
265 2018-02-13T11:54:09  <goatpig> that would be if the method was not meant to return state of the lock
266 2018-02-13T11:54:22  <goatpig> the semantics just suggest to me that it could be accessed by another thread
267 2018-02-13T11:54:46  <wumpus> I think the intent is to return true if the process has a lock on the directory, whether it's new or not. Not sure the code matches that intent, at the moment.
268 2018-02-13T11:54:52  <gmaxwell> nah its just init only now. though wumpus is right that dynamic wallet opening/closing will cause accesses from rpc.
269 2018-02-13T11:55:38  <wumpus> I've asked meshcollider (who wrote the code) in the issue just in case
270 2018-02-13T11:55:59  <gmaxwell> for wallet opening we probably want it to return true if it takes a lock and false if it already has it or can't take it.
271 2018-02-13T11:56:37  <meshcollider> wumpus: I think ryanofsky suggested the currently used approach as a comment on the PR
272 2018-02-13T11:56:41  <wumpus> gmaxwell:  it's a lock on a directory, not on the wallet, bdb takes care of not opening the same wallet multiple times
273 2018-02-13T11:56:53  <wumpus> gmaxwell: there can be multiple wallets in a directory
274 2018-02-13T11:57:13  <gmaxwell> wumpus: ah point.
275 2018-02-13T11:57:20  <gmaxwell> Carry on then.
276 2018-02-13T11:59:48  *** Chris_Stewart_5 has joined #bitcoin-core-dev
277 2018-02-13T12:06:10  <wumpus> we could certainly use unit tests for that function
278 2018-02-13T12:08:01  *** Chris_Stewart_5 has quit IRC
279 2018-02-13T12:08:50  *** rodarmor has quit IRC
280 2018-02-13T12:09:25  *** rodarmor has joined #bitcoin-core-dev
281 2018-02-13T12:10:01  <wumpus> meshcollider: ok adding jnewbery to reviewers
282 2018-02-13T12:12:55  <meshcollider> jnewbery or ryanofsky?
283 2018-02-13T12:13:11  <meshcollider> wumpus: your PR looks good to me
284 2018-02-13T12:13:41  <wumpus> eh, sorry, yes ryanofsky, added the wrong one
285 2018-02-13T12:13:51  *** jamesob has joined #bitcoin-core-dev
286 2018-02-13T12:15:02  <meshcollider> wumpus: didn't even think about the dynamic loading/unloading case / thread safety when I wrote it, apologies
287 2018-02-13T12:15:17  <meshcollider> Was just trying to fix the issue for 0.16
288 2018-02-13T12:15:43  <wumpus> meshcollider: no worries, we found that issue before it became a problem at least
289 2018-02-13T12:21:35  *** cryptojanitor has joined #bitcoin-core-dev
290 2018-02-13T12:32:44  * wumpus working on a unit test for LockDataDirectory - part of it will only work on UNIX because fork()
291 2018-02-13T12:46:09  *** Chris_Stewart_5 has joined #bitcoin-core-dev
292 2018-02-13T12:50:57  *** Chris_Stewart_5 has quit IRC
293 2018-02-13T13:05:11  *** Chris_Stewart_5 has joined #bitcoin-core-dev
294 2018-02-13T13:11:22  <provoostenator> You have got to be kidding... anyone had to jump through these hoops before? http://htmlpreview.github.io/?https://github.com/JKSH/QtSdkRepoChooser/blob/master/doc/index.html
295 2018-02-13T13:14:58  *** Pavle has joined #bitcoin-core-dev
296 2018-02-13T13:15:11  <wumpus> no, not really, though I've had to build qt for embedded devices, the source code is huge, it takes a long time and it's a pain to configure for cross compile
297 2018-02-13T13:15:35  <wumpus> it pretty much regards itself as an operating system in itself
298 2018-02-13T13:19:29  *** lnostdal has joined #bitcoin-core-dev
299 2018-02-13T13:21:03  *** Pavle has quit IRC
300 2018-02-13T13:22:29  <wumpus> we can be really happy that linux distributions simply package that stuff pre-built
301 2018-02-13T13:28:02  *** jtimon has quit IRC
302 2018-02-13T13:34:13  *** timothy has joined #bitcoin-core-dev
303 2018-02-13T13:56:17  *** jamesob has quit IRC
304 2018-02-13T13:58:38  *** SopaXorzTaker has joined #bitcoin-core-dev
305 2018-02-13T14:09:49  *** jamesob has joined #bitcoin-core-dev
306 2018-02-13T14:18:18  <midnightmagic> /w/w 59
307 2018-02-13T14:18:51  *** dermoth has quit IRC
308 2018-02-13T14:19:41  *** dermoth has joined #bitcoin-core-dev
309 2018-02-13T14:20:01  <cfields> wumpus: seems it's moot now, but I think std::piecewise_construct might be what's missing for the emplace
310 2018-02-13T14:20:52  *** laurentmt has quit IRC
311 2018-02-13T14:22:23  *** Chris_Stewart_5 has quit IRC
312 2018-02-13T14:23:05  <cfields> as usual, spoke too soon. Red herring. Nevermind.
313 2018-02-13T14:25:20  <wumpus> I'll look at that one, never heard of it
314 2018-02-13T14:26:52  <cfields> wumpus: see for example: https://github.com/bitcoin/bitcoin/blob/master/src/net_processing.cpp#L574
315 2018-02-13T14:27:37  *** timothy has quit IRC
316 2018-02-13T14:31:57  *** rex_4539 has joined #bitcoin-core-dev
317 2018-02-13T14:36:27  *** Victorsueca has quit IRC
318 2018-02-13T14:37:48  *** Victorsueca has joined #bitcoin-core-dev
319 2018-02-13T14:38:31  *** laurentmt has joined #bitcoin-core-dev
320 2018-02-13T14:39:33  *** timothy has joined #bitcoin-core-dev
321 2018-02-13T14:40:54  *** Chris_Stewart_5 has joined #bitcoin-core-dev
322 2018-02-13T14:42:07  *** meshcollider has quit IRC
323 2018-02-13T14:49:49  *** jamesob has quit IRC
324 2018-02-13T14:50:14  *** ken2812221 has quit IRC
325 2018-02-13T14:50:39  *** ken2812221 has joined #bitcoin-core-dev
326 2018-02-13T14:50:46  *** ossifrage has quit IRC
327 2018-02-13T14:51:20  *** ossifrage has joined #bitcoin-core-dev
328 2018-02-13T14:56:33  *** dermoth has quit IRC
329 2018-02-13T14:56:33  *** mmgen has quit IRC
330 2018-02-13T14:56:33  *** ghost43 has quit IRC
331 2018-02-13T14:56:54  *** ghost43 has joined #bitcoin-core-dev
332 2018-02-13T14:57:07  *** dermoth has joined #bitcoin-core-dev
333 2018-02-13T14:57:16  *** mmgen has joined #bitcoin-core-dev
334 2018-02-13T14:58:55  *** rex_4539 has quit IRC
335 2018-02-13T15:01:49  *** rex_4539 has joined #bitcoin-core-dev
336 2018-02-13T15:02:21  <hkjn0> a bit lazy q: am I reading depends/ docs right that there's currently no way to use other platforms than x86_64 to build e.g. bdb?
337 2018-02-13T15:02:27  *** alfa has joined #bitcoin-core-dev
338 2018-02-13T15:03:15  <hkjn0> so if I'm compiling code on armv7l and want wallet support, best thing to do is to get on x86_64 instead, and crosscompile towards arm as necessary?
339 2018-02-13T15:03:57  *** ghost43 has quit IRC
340 2018-02-13T15:03:59  <hkjn0> (I did get bdb4.8 compiled from source on armv7l outside of the depends system, but had trouble getting it to be noticed by the build system..)
341 2018-02-13T15:04:09  *** ghost43 has joined #bitcoin-core-dev
342 2018-02-13T15:06:13  <wumpus> hkjn0: the depends system assumes it's building on x86_64, however if you want to build db4 outside the depends system there's the script  contrib/install_db4.sh and the flags  BDB_LIBS="-L${BDB_PREFIX}/lib -ldb_cxx-4.8" BDB_CFLAGS="-I${BDB_PREFIX}/include"
343 2018-02-13T15:07:41  <wumpus> would be nice if the depends system also supported building *from* other architectures, though
344 2018-02-13T15:10:02  *** Giszmo has joined #bitcoin-core-dev
345 2018-02-13T15:10:18  <hkjn0> ah, great, I bet I was just misisng BDB_LIBS or somesuch.. will try out the contrib/ script
346 2018-02-13T15:10:33  <wumpus> esp. at some point we'd like to do gitian builds from other architectures
347 2018-02-13T15:11:59  <hkjn0> right. I've been setting up these weird build environments already, would be happy to try to contribute to gitian work or otherwise support other build platforms, just need to understand more of how current system hangs together first.. :)
348 2018-02-13T15:18:27  *** Chris_Stewart_5 has quit IRC
349 2018-02-13T15:44:01  *** jamesob has joined #bitcoin-core-dev
350 2018-02-13T15:46:47  *** owowo has quit IRC
351 2018-02-13T15:49:22  *** dermoth has quit IRC
352 2018-02-13T15:51:36  *** owowo has joined #bitcoin-core-dev
353 2018-02-13T15:51:40  *** Chris_Stewart_5 has joined #bitcoin-core-dev
354 2018-02-13T15:55:30  *** grafcaps has quit IRC
355 2018-02-13T16:05:21  *** jamesob has quit IRC
356 2018-02-13T16:07:11  *** Chris_Stewart_5 has quit IRC
357 2018-02-13T16:10:17  *** lnostdal has quit IRC
358 2018-02-13T16:11:55  *** grafcaps has joined #bitcoin-core-dev
359 2018-02-13T16:22:40  <cfields> depends doesn't assume build is x86_64. It's the only arch that really gets tested, though
360 2018-02-13T16:23:09  <cfields> but, all of the machinery is there to auto-detect the build platform and try to cope.
361 2018-02-13T16:24:46  <cfields> hkjn0: if you try building native on arm and run into issues, feel free to ping me
362 2018-02-13T16:27:03  *** AaronvanW has joined #bitcoin-core-dev
363 2018-02-13T16:28:25  <hkjn0> cfield, thanks! building with contrib/install_db4.sh + BDB_LIBS it suggested for ./configure as wumpus suggested seems to work fine so far
364 2018-02-13T16:29:44  <hkjn0> (before that I tried a plain `make` in depends/ and it failed on armv7l, but it is possible I am missing some packages)
365 2018-02-13T16:34:01  <cfields> hkjn0: well if you'd be willing, I'd be happy to step through the depends with you and try to get it patched to build
366 2018-02-13T16:35:10  <cfields> like wumpus said, it'd be nice to have several working build arches
367 2018-02-13T16:44:42  <hkjn0> cfields: sure, let me know how you'd like to proceed.. do we take it off the main channel to not clutter it up too much?
368 2018-02-13T16:45:08  <cfields> hkjn0: sure, #bitcoin-build
369 2018-02-13T16:52:33  *** Randolf has quit IRC
370 2018-02-13T16:54:59  <michagogo> wumpus/cfields: Do you remember (on a describe-in-a-couple-sentences-off-the-top-of-your-head level) what the actual issue is with Xenial and why it's broken?
371 2018-02-13T16:55:27  <michagogo> And does Artful(/Bionic) just work?
372 2018-02-13T16:55:42  <michagogo> (with cross-compilation for Windows, I mean)
373 2018-02-13T16:56:36  <michagogo> I tried looking through the issues on GitHub and got confused by the various different threads and things people tried and experienced
374 2018-02-13T16:56:37  <cfields> michagogo: by default, the toolchain is configured for win threads rather than posix, which don't exist (at least with that gcc version)
375 2018-02-13T16:56:41  *** Murch has joined #bitcoin-core-dev
376 2018-02-13T16:56:52  <cfields> further, the ssp is busted
377 2018-02-13T16:57:30  <michagogo> And were those fixed in later versions, e.g. Artful?
378 2018-02-13T16:58:13  <cfields> I believe so. But we should be switching to our own toolchains for 0.17 anyway. Or if not a full switch, probably at least for mingw.
379 2018-02-13T16:59:10  <michagogo> That sounds like it would be a good thing
380 2018-02-13T17:00:24  <michagogo> I ask because last night and today I spent a bunch of hours trying to see how hard it would be to just backport the version of mingw-w64 that's in artful/bionic to xenial
381 2018-02-13T17:01:16  <michagogo> I tried the straightforward "`backportpackage` into a PPA", but ran into chains of missing dependencies
382 2018-02-13T17:02:00  <michagogo> I won't bore you with everything I tried and looked at, but at this point I've come to the realization that I don't know enough about Ubuntu packaging and how all that works, and beyond that, more important stuff
383 2018-02-13T17:02:17  <cfields> michagogo: I can upload a stand-alone mingw toolchain soon, if it would help
384 2018-02-13T17:02:21  <michagogo> Like, what is mingw-w64, vs what is gcc, vs what is gcc-mingw-w64
385 2018-02-13T17:02:23  <cfields> heh
386 2018-02-13T17:02:27  <michagogo> and how do all those interact
387 2018-02-13T17:02:43  <cfields> right
388 2018-02-13T17:02:56  <cfields> well, in case it helps...
389 2018-02-13T17:03:25  <michagogo> In #ubuntu-devel someone said: 18:34:22 <rbasak> michagogo: I would step back and consider your approach. I don't know what problem you're solving, but there might be an easier way.
390 2018-02-13T17:04:10  <michagogo> I explained that the version in xenial was broken, and they said this: 18:39:39 <rbasak> Also, if the reason it doesn't work in Xenial is due to a bug, we're quite happy in general to cherry-pick the fix for all Xenial users to benefit.
391 2018-02-13T17:04:27  <michagogo> But somehow I suspect if it were that simple it would have already happened...
392 2018-02-13T17:04:38  <cfields> to build a mingw-w64 toolchain, you need: mingw-w64 headers (think kernel headers), mingw-64 runtime (think glibc), gcc, and binutils. And obviously they need to be built in a way that lets them play nicely.
393 2018-02-13T17:05:04  <michagogo> Right. So the thing is, I don't actually know the difference between mingw-w64 and gcc
394 2018-02-13T17:05:09  <michagogo> Well, that's not entirely true
395 2018-02-13T17:05:14  <michagogo> I sort of know what gcc is
396 2018-02-13T17:05:34  <michagogo> What I don't know is, what is mingw-w64, and how does it interact with/based on/use gcc
397 2018-02-13T17:06:23  *** dgenr8 has joined #bitcoin-core-dev
398 2018-02-13T17:07:08  <cfields> mingw-w64 is a shim that allows for posix-y code to work on windows.
399 2018-02-13T17:08:50  <michagogo> So what is gcc-mingw-w64?
400 2018-02-13T17:09:20  <cfields> so you can build a compiler (gcc) that's able to produce win32 code, but it wouldn't do much good if you couldn't use standard unixy apis.
401 2018-02-13T17:10:20  <cfields> so, for example, mingw-w64 can use either win or pthread threading models. Configured without pthread support, you'll have trouble with lots of applications. That's one of our issues.
402 2018-02-13T17:13:11  <michagogo> So gcc-mingw-64, say, is a version of GCC that builds the mingw-w64 layer into the binaries it produces?
403 2018-02-13T17:13:40  <cfields> it's probably easiest to just think of mingw-w64 as a libc for windows.
404 2018-02-13T17:14:17  <michagogo> I guess at this point the knowledge that I'm missing is: 1) what _exactly_ the term "toolchain" refers to and how you build one
405 2018-02-13T17:14:37  <michagogo> 2) How Ubuntu packages work together to make up a toolchain
406 2018-02-13T17:15:09  <michagogo> And all kinds of other things related to those, like how exactly Ubuntu packaging works
407 2018-02-13T17:15:22  <cfields> well, the packaging is an entirely different beast
408 2018-02-13T17:16:02  <michagogo> I'm thinking I'll drop the issue for now, since I don't really have a ton of time and I suspect it would take me at least a few days to sort all these things out
409 2018-02-13T17:16:46  <michagogo> I mean, regarding what rbasak said over in #ubuntu-devel, are the problems we have simple bugs with focused fixes that can be cherry-picked, or more fundamental problems that can't really be fixed without a wholesale upgrade to a newer version of MinGW?
410 2018-02-13T17:17:27  <cfields> a toolchain is the group of progs that are required to produce a working binary. Typically that's target kernel/os headers in some form, target libc, host compiler, other host apps (in linux, binutils provides ar/ranlib/etc.), and a linker (also provided by binutils on linux)
411 2018-02-13T17:17:37  <jcorgan> michagogo: it's been a little while since i've used gcc-mingw-w64 but i recall in ubuntu that installing that package pulls in all the accessories (runtime, binutils, etc.) that cfields mentioned
412 2018-02-13T17:17:59  <michagogo> And actually, is the issue just with the MinGW component? Meaning, say one were to rebuild the gcc-mingw-w64 packages with the same GCC version and just an upgraded MinGW, would that work?
413 2018-02-13T17:19:04  <jcorgan> in late to the conversation here, so not sure you're exact goal
414 2018-02-13T17:19:27  <cfields> michagogo: the thread thing is just a config option. There's also the stack issue, which I haven't looked into for a while, but afaik that's no longer an issue in newer ubuntu versions. So it was either an upstream bug that's been fixed, or a config problem that ubuntu has fixed
415 2018-02-13T17:19:52  <michagogo> jcorgan: Well, at the end of the day, here's my big-picture goal
416 2018-02-13T17:20:02  <michagogo> Make https://github.com/bitcoin/bitcoin/blob/master/doc/build-windows.md#building-for-64-bit-windows die in a fire
417 2018-02-13T17:20:30  <michagogo> Specifically, the lines telling you to sudo add-apt-repository "deb http://archive.ubuntu.com/ubuntu zesty universe"; sudo apt update;sudo apt upgrade
418 2018-02-13T17:20:54  <michagogo> Because besides that not working, because zesty is EOL, that's just going to hose your system
419 2018-02-13T17:21:28  <jcorgan> ah, i only have experience going from LTS to LTS, never use the ones in between
420 2018-02-13T17:21:56  <michagogo> Even doing that, you never ever just add the newer repository like that
421 2018-02-13T17:21:59  <michagogo> There's an upgrader
422 2018-02-13T17:22:37  <michagogo> So my thought was, well, can I get the packages from the newer version brought back to xenial?
423 2018-02-13T17:23:03  <jcorgan> it might be possible, but that way lies madness
424 2018-02-13T17:23:25  <michagogo> So it seems.
425 2018-02-13T17:23:51  <michagogo> I tried just naively `backportpackage`ing into a PPA
426 2018-02-13T17:24:09  <jcorgan> you're on zesty and need packages that only exist later than that?
427 2018-02-13T17:24:22  <cfields> michagogo: yea, I'm not sure it's worth the trouble. We already have a plan for resolving this, and it means that we don't have to rely on Ubuntu anymore
428 2018-02-13T17:24:24  <michagogo> I’m not on zesty
429 2018-02-13T17:25:11  <michagogo> I’m actually not an active Ubuntu (or Linux) user day to day
430 2018-02-13T17:25:13  <jcorgan> you could see if the packages you need are in the 'backports' repo
431 2018-02-13T17:25:18  <michagogo> Nah
432 2018-02-13T17:25:36  <michagogo> My idea was to *get* a newer mingw-w64 into -backports
433 2018-02-13T17:26:25  <michagogo> I filed the “please backport” bugs
434 2018-02-13T17:26:28  <cfields> michagogo: well, that certainly wouldn't be a bad thing. It's definitely not just us.
435 2018-02-13T17:26:44  <jcorgan> i expect that would require specialized knowledge of how gcc/binutils/etc are packaged that a typical user wouldn't normally need to know
436 2018-02-13T17:27:07  <jcorgan> and personally i'd look for an alternative way to solve whatever problem
437 2018-02-13T17:27:32  <michagogo> I had the naive thought that, well, it should be easy to backport, because it’s a cross-compiler, not anything that the system uses
438 2018-02-13T17:28:26  <michagogo> jcorgan: AIUI “whatever problem” is “mingw-w64 on Xenial is very broken”
439 2018-02-13T17:30:38  *** Pavle has joined #bitcoin-core-dev
440 2018-02-13T17:31:03  <jcorgan> personally i'd see if the must-be-xenial constraint can be relaxed. but if you wanted a challenge you could learn enough about how it works to help fix the issue. that may not be worth your time, but it could also be fun and very educational.
441 2018-02-13T17:31:21  <jcorgan> (depends on you)
442 2018-02-13T17:31:21  *** mrannanay has joined #bitcoin-core-dev
443 2018-02-13T17:31:25  <michagogo> A no-changes backport failed, firstly, because it build-depends on gcc-7
444 2018-02-13T17:32:08  <michagogo> Which must-be-Xenial constraint?
445 2018-02-13T17:32:23  <michagogo> The thing is, 1) that’s the latest LTS
446 2018-02-13T17:32:51  <jcorgan> right, i use that extensively
447 2018-02-13T17:32:54  <michagogo> And 2) that’s the version that you get with WSL, and as far as I know trying to upgrade just breaks it
448 2018-02-13T17:33:31  <jcorgan> ah, another piece of the picture i was unaware of
449 2018-02-13T17:34:07  <michagogo> So the “constraint” is that we really want to let users on Xenial cross-compile is
450 2018-02-13T17:34:12  <jcorgan> you're in a maze of twisty passages, all alike
451 2018-02-13T17:34:15  <michagogo> cross-compile us*
452 2018-02-13T17:36:01  <michagogo> I mean, it’s possible that if I knew Ubuntu packaging and had enough time I could try just reverting the changes that made it build with gcc 7
453 2018-02-13T17:36:05  <jcorgan> so, getting a backport made of mingw does seem like the right way to go, but the number of people who are likely to know how to do that is probably small
454 2018-02-13T17:36:13  <michagogo> And bring it back to 5.whatever
455 2018-02-13T17:36:58  <michagogo> For all I know it might be just changing the build-depends line back to point at the gcc version that’s in Xenial
456 2018-02-13T17:37:06  <jcorgan> sure
457 2018-02-13T17:38:00  <jcorgan> but there could also be some specific bug in gcc5 that caused gcc7 to be needed.
458 2018-02-13T17:38:08  <michagogo> I doubt it
459 2018-02-13T17:38:23  <michagogo> I mean, I shouldn’t t say that
460 2018-02-13T17:38:49  <jcorgan> that does sound like a (relatively) easy first thing to try, and you might get lucky
461 2018-02-13T17:39:04  <michagogo> But I’m pretty sure that the upgrade to gcc 6 in... yakkety or zesty, I think? And to 7 in bionic, was just part of a wholesale upgrade to those versions
462 2018-02-13T17:39:52  <jcorgan> i'm going to be in pain when i start looking at migrating everything to 18.04, aren't I?
463 2018-02-13T17:40:39  <michagogo> But besides not knowing even the basics of Ubuntu packaging, I’d need the ability to go over all the other changes to the gcc-mingw-w64 source package and figure out which ones are related to the gcc 5/7 stuff, vs which ones should be there anyway
464 2018-02-13T17:40:56  *** AaronvanW has quit IRC
465 2018-02-13T17:40:56  <bitcoin-git> [bitcoin] ryanofsky opened pull request #12424: Fix rescan test failure due to unset g_address_type, g_change_type (master...pr/scang) https://github.com/bitcoin/bitcoin/pull/12424
466 2018-02-13T17:41:01  <michagogo> jcorgan: define “everything”
467 2018-02-13T17:41:33  <jcorgan> most of the systems i use/support are 16.04, some still 14.04
468 2018-02-13T17:41:33  *** AaronvanW has joined #bitcoin-core-dev
469 2018-02-13T17:41:47  <jcorgan> it sounds like a lot of changes are coming in 18.04
470 2018-02-13T17:41:53  <michagogo> No idea
471 2018-02-13T17:42:06  <jcorgan> anyway, this is straying a bit from coredev
472 2018-02-13T17:42:10  <michagogo> I mean, it’s 2 years of progress in the overall Linux ecosystem
473 2018-02-13T17:42:17  <michagogo> Bringing everything up to date
474 2018-02-13T17:42:38  <michagogo> So yeah, some things may break…
475 2018-02-13T17:42:47  <jcorgan> well, the older LTS versions get new packages for everything that isn't a fundamental system change
476 2018-02-13T17:43:24  <jcorgan> and the backports repo even lets you run newer kernels and video drivers
477 2018-02-13T17:44:29  <jcorgan> it's usually the lower-level stuff like sysvinit/upstart/systemd that causes me pain
478 2018-02-13T17:45:40  *** AaronvanW has quit IRC
479 2018-02-13T17:47:43  <michagogo> Get new packages? What do you mean?
480 2018-02-13T17:48:21  <michagogo> My understanding is that -updates only has bug fixes, ~never new wholesale versions of software that have features
481 2018-02-13T17:48:47  *** farmerwampum has quit IRC
482 2018-02-13T17:49:51  <jcorgan> maybe i'm thinking of PPAs, where authors make recent versions of their software for several different ubuntu releases
483 2018-02-13T17:49:58  <michagogo> Yep
484 2018-02-13T17:50:17  <michagogo> With a PPA you can give people who decide to use you whatever you want, whenever you want
485 2018-02-13T17:50:51  <michagogo> (Including installing a backdoor on their systems. It’s important that you trust the owners of PPAs you use.)
486 2018-02-13T17:51:08  <jcorgan> well, i hope you figure out how to get teh mingw/WSL stuff working
487 2018-02-13T17:51:18  *** Pavle has quit IRC
488 2018-02-13T17:51:31  *** AaronvanW has joined #bitcoin-core-dev
489 2018-02-13T17:51:32  <michagogo> Yeah, I think I'm just going to drop the issue at this point
490 2018-02-13T17:52:08  <michagogo> cfields said that we're going to be using our own toolchain at some point (soon? ish?)
491 2018-02-13T17:52:10  *** alfa has quit IRC
492 2018-02-13T17:52:59  <cfields> michagogo: yes. Ubuntu versions/packages won't matter for gitian/travis after that.
493 2018-02-13T17:53:26  <michagogo> cfields: I'm thinking mostly about the case of someone wanting to cross-build binaries for themselves
494 2018-02-13T17:53:56  <michagogo> Either just not wanting to go through the hassle of setting up gitian, or with something like WSL
495 2018-02-13T17:54:15  <michagogo> I really don't like the fact that we're giving users broken instructions, especially ones that (were they to work) have them completely mess up their system
496 2018-02-13T17:54:15  <cfields> michagogo: sure. None of the new stuff will be required, ofc.
497 2018-02-13T17:54:39  <michagogo> (https://github.com/bitcoin/bitcoin/blob/master/doc/build-windows.md#building-for-64-bit-windows)
498 2018-02-13T17:54:48  *** ProfMac has quit IRC
499 2018-02-13T17:55:28  <michagogo> And of course, there's the fact that mingw is broken for everyone else
500 2018-02-13T17:56:26  *** AaronvanW has quit IRC
501 2018-02-13T17:56:30  *** S3RVO has joined #bitcoin-core-dev
502 2018-02-13T18:04:08  *** timothy has quit IRC
503 2018-02-13T18:04:56  *** harrymm has quit IRC
504 2018-02-13T18:05:13  *** Randolf has joined #bitcoin-core-dev
505 2018-02-13T18:07:00  *** PaulCapestany has quit IRC
506 2018-02-13T18:10:21  *** PaulCapestany has joined #bitcoin-core-dev
507 2018-02-13T18:18:02  *** harrymm has joined #bitcoin-core-dev
508 2018-02-13T18:21:34  *** shesek has quit IRC
509 2018-02-13T18:25:33  *** Murch has quit IRC
510 2018-02-13T18:26:12  *** Murch has joined #bitcoin-core-dev
511 2018-02-13T18:27:24  *** ProfMac has joined #bitcoin-core-dev
512 2018-02-13T18:28:13  *** sengehest has joined #bitcoin-core-dev
513 2018-02-13T18:29:19  *** Chris_Stewart_5 has joined #bitcoin-core-dev
514 2018-02-13T18:29:39  *** S3RVO has quit IRC
515 2018-02-13T18:32:06  *** ProfMac has quit IRC
516 2018-02-13T18:33:32  *** PaulCape_ has joined #bitcoin-core-dev
517 2018-02-13T18:34:16  *** PaulCapestany has quit IRC
518 2018-02-13T18:35:02  *** shesek has joined #bitcoin-core-dev
519 2018-02-13T18:35:02  *** shesek has joined #bitcoin-core-dev
520 2018-02-13T18:39:36  *** Murch has quit IRC
521 2018-02-13T18:42:20  *** Murch has joined #bitcoin-core-dev
522 2018-02-13T18:45:35  <bitcoin-git> [bitcoin] richardkiss opened pull request #12425: Add some script tests (master...feature/bool_tests) https://github.com/bitcoin/bitcoin/pull/12425
523 2018-02-13T18:47:38  *** promag has quit IRC
524 2018-02-13T18:49:38  *** promag has joined #bitcoin-core-dev
525 2018-02-13T18:50:42  *** promag has quit IRC
526 2018-02-13T18:51:16  *** Murch has quit IRC
527 2018-02-13T18:53:56  *** Murch has joined #bitcoin-core-dev
528 2018-02-13T19:10:38  *** Victorsueca has quit IRC
529 2018-02-13T19:11:48  *** Victorsueca has joined #bitcoin-core-dev
530 2018-02-13T19:11:59  *** AaronvanW has joined #bitcoin-core-dev
531 2018-02-13T19:16:33  *** sengehest has quit IRC
532 2018-02-13T19:19:13  *** mirese has joined #bitcoin-core-dev
533 2018-02-13T19:22:58  *** mirese_ has quit IRC
534 2018-02-13T19:23:17  *** ProfMac has joined #bitcoin-core-dev
535 2018-02-13T19:27:35  *** AaronvanW has quit IRC
536 2018-02-13T19:31:48  *** Murch has quit IRC
537 2018-02-13T19:36:17  *** Julie73Donnelly has quit IRC
538 2018-02-13T19:37:03  *** mirese_ has joined #bitcoin-core-dev
539 2018-02-13T19:37:33  *** mirese__ has joined #bitcoin-core-dev
540 2018-02-13T19:40:03  *** Murch has joined #bitcoin-core-dev
541 2018-02-13T19:40:11  *** mirese___ has joined #bitcoin-core-dev
542 2018-02-13T19:40:34  *** mirese has quit IRC
543 2018-02-13T19:41:30  *** mirese has joined #bitcoin-core-dev
544 2018-02-13T19:41:50  *** mirese_ has quit IRC
545 2018-02-13T19:43:51  *** dcousens has quit IRC
546 2018-02-13T19:44:06  *** mirese__ has quit IRC
547 2018-02-13T19:44:09  *** dcousens has joined #bitcoin-core-dev
548 2018-02-13T19:44:40  *** mirese___ has quit IRC
549 2018-02-13T19:46:05  *** jtimon has joined #bitcoin-core-dev
550 2018-02-13T19:48:27  *** dcousens has quit IRC
551 2018-02-13T19:49:05  *** dcousens has joined #bitcoin-core-dev
552 2018-02-13T19:50:29  *** AaronvanW has joined #bitcoin-core-dev
553 2018-02-13T19:52:42  *** Tobin19Abernathy has joined #bitcoin-core-dev
554 2018-02-13T19:55:10  *** Dizzle has joined #bitcoin-core-dev
555 2018-02-13T19:55:49  *** Murch has quit IRC
556 2018-02-13T19:57:49  *** Randolf has quit IRC
557 2018-02-13T19:58:14  *** promag has joined #bitcoin-core-dev
558 2018-02-13T20:01:56  *** AaronvanW has quit IRC
559 2018-02-13T20:01:57  *** SopaXorzTaker has quit IRC
560 2018-02-13T20:04:05  *** tli has joined #bitcoin-core-dev
561 2018-02-13T20:08:20  *** promag has quit IRC
562 2018-02-13T20:08:54  *** promag has joined #bitcoin-core-dev
563 2018-02-13T20:08:54  *** neha has quit IRC
564 2018-02-13T20:10:27  *** Tobin19Abernathy has quit IRC
565 2018-02-13T20:13:21  *** promag has quit IRC
566 2018-02-13T20:15:53  *** neha has joined #bitcoin-core-dev
567 2018-02-13T20:22:35  *** promag has joined #bitcoin-core-dev
568 2018-02-13T20:23:16  *** tli has quit IRC
569 2018-02-13T20:24:37  <gmaxwell> Note: if anyone here uses bitmessage, it has a remote code execution bug. (deserializing using eval)
570 2018-02-13T20:25:01  <sipa> what is it written in?
571 2018-02-13T20:26:06  <gmaxwell> Python.
572 2018-02-13T20:26:32  <contrapumpkin> is today national facepalm security day or something
573 2018-02-13T20:29:56  <esotericnonsense> deserializing using eval ???
574 2018-02-13T20:30:03  <esotericnonsense> -_-.
575 2018-02-13T20:31:23  *** promag has quit IRC
576 2018-02-13T20:35:10  <instagibbs> the vulnerability is being exploited in the wild as well
577 2018-02-13T20:35:38  <instagibbs> not sure about success, but trying to run windows executables, grab electrum wallets, the like
578 2018-02-13T20:37:27  <gmaxwell> esotericnonsense: deserializing using eval is a thing I've seen many times w/ javascript, php, and python.
579 2018-02-13T20:37:49  <contrapumpkin> java does its own version of that too, if you use their native serialization
580 2018-02-13T20:38:45  *** tli has joined #bitcoin-core-dev
581 2018-02-13T20:38:58  *** jamesob has joined #bitcoin-core-dev
582 2018-02-13T20:39:14  <esotericnonsense> gmaxwell: oh sure it's a thing. i just feel like it's one of the most famous 'don't do this' footguns
583 2018-02-13T20:39:31  <esotericnonsense> evidently not
584 2018-02-13T20:44:41  *** jamesob_ has joined #bitcoin-core-dev
585 2018-02-13T20:47:00  *** jamesob has quit IRC
586 2018-02-13T20:48:19  <mmgen> wouldn't use of the json module protect against this?
587 2018-02-13T20:49:40  <mmgen> first json.loads(), then deserialize?
588 2018-02-13T20:52:27  <reardencode> mmgen: yes.
589 2018-02-13T20:52:40  <mmgen> strange they didn't do that
590 2018-02-13T20:52:46  *** Marcia15Hilpert has joined #bitcoin-core-dev
591 2018-02-13T20:55:53  *** Randolf has joined #bitcoin-core-dev
592 2018-02-13T20:59:07  <mmgen> the fix: https://github.com/Bitmessage/PyBitmessage/commit/3a8016d31f517775d226aa8b902480f4a3a148a9
593 2018-02-13T21:00:17  *** Randolf has quit IRC
594 2018-02-13T21:00:31  *** tli has quit IRC
595 2018-02-13T21:02:31  <contrapumpkin> that still seems like an iffy fix, even though the scope for mischief is far more limited
596 2018-02-13T21:03:14  <contrapumpkin> it'll basically call an arbitrary method specified by untrusted data on the specified class
597 2018-02-13T21:03:14  *** meshcollider has joined #bitcoin-core-dev
598 2018-02-13T21:03:27  <bitcoin-git> [bitcoin] MarcoFalke opened pull request #12426: qt: Initialize members in WalletModel (master...Mf1802-qtInitializeMembersWalletModel) https://github.com/bitcoin/bitcoin/pull/12426
599 2018-02-13T21:05:17  *** promag has joined #bitcoin-core-dev
600 2018-02-13T21:07:09  <mmgen> contrapumpkin: hopefully just an emergency fix. far better than nothing in any case
601 2018-02-13T21:07:41  <contrapumpkin> yup :)
602 2018-02-13T21:13:45  *** harrymm has quit IRC
603 2018-02-13T21:17:33  <mmgen> seems like try: assert data[""
604 2018-02-13T21:18:07  <mmgen> seems like try: assert data[""] in ('message','vote') would have been safer
605 2018-02-13T21:19:10  <contrapumpkin> the hoops people go through to avoid writing a proper parser... 😢
606 2018-02-13T21:19:42  <mmgen> since 'message' and 'vote' are the only permissible values
607 2018-02-13T21:19:53  <sipa> it's getting a bit off topic for this channel
608 2018-02-13T21:19:59  <contrapumpkin> sorry :) I'll shut up
609 2018-02-13T21:21:03  *** Victorsueca has quit IRC
610 2018-02-13T21:21:07  <mmgen> sipa: sorry
611 2018-02-13T21:21:47  <sipa> np!
612 2018-02-13T21:22:18  *** Victorsueca has joined #bitcoin-core-dev
613 2018-02-13T21:22:37  <mmgen> sipa: though it **was** gmaxwell who brought up the topic
614 2018-02-13T21:23:46  <sipa> i'm aware, and i participated too; no blame here
615 2018-02-13T21:23:57  <sipa> just noting that the discussion is getting too far removed from the topic here
616 2018-02-13T21:24:46  <mmgen> mmgen: i know, i'll really shut up now ;)
617 2018-02-13T21:27:07  *** harrymm has joined #bitcoin-core-dev
618 2018-02-13T21:27:13  *** AaronvanW has joined #bitcoin-core-dev
619 2018-02-13T21:29:12  <bitcoin-git> [bitcoin] sipa opened pull request #12427: Make signrawtransaction accept P2SH-P2WSH redeemscripts (master...201802_signrawp2shp2wsh) https://github.com/bitcoin/bitcoin/pull/12427
620 2018-02-13T21:30:55  <gmaxwell> doh, how'd we miss that
621 2018-02-13T21:32:19  <grafcaps> hah
622 2018-02-13T21:38:25  *** mmgen has quit IRC
623 2018-02-13T21:38:32  <sipa> gmaxwell: it was actually partially being addressed in #11708
624 2018-02-13T21:38:36  <gribble> https://github.com/bitcoin/bitcoin/issues/11708 | Add P2SH-P2WSH support to signrawtransaction and listunspent RPC by MeshCollider · Pull Request #11708 · bitcoin/bitcoin · GitHub
625 2018-02-13T21:39:39  *** tsunami_86 has joined #bitcoin-core-dev
626 2018-02-13T21:41:05  <promag> sipa: is it possible to add a test for #12427?
627 2018-02-13T21:41:07  <gribble> https://github.com/bitcoin/bitcoin/issues/12427 | Make signrawtransaction accept P2SH-P2WSH redeemscripts by sipa · Pull Request #12427 · bitcoin/bitcoin · GitHub
628 2018-02-13T21:41:21  <sipa> promag: yes, will look into that
629 2018-02-13T21:41:29  <sipa> just wanted to have the PR visible as soon as possible
630 2018-02-13T21:42:37  *** tsunami_86 has quit IRC
631 2018-02-13T21:46:44  *** Cogito_Ergo_Sum has joined #bitcoin-core-dev
632 2018-02-13T21:46:44  *** Cogito_Ergo_Sum has joined #bitcoin-core-dev
633 2018-02-13T21:51:09  <Chris_Stewart_5> sipa: Is the basic idea that if we can sign the p2sh redeem script, we should be able to sign P2SH(P2WSH())?
634 2018-02-13T21:51:10  *** laurentmt has quit IRC
635 2018-02-13T21:54:52  *** Murch has joined #bitcoin-core-dev
636 2018-02-13T21:58:00  *** Murch has quit IRC
637 2018-02-13T22:00:55  *** Murch has joined #bitcoin-core-dev
638 2018-02-13T22:01:44  *** promag has quit IRC
639 2018-02-13T22:03:23  *** cheese_ has joined #bitcoin-core-dev
640 2018-02-13T22:04:03  *** harrymm has quit IRC
641 2018-02-13T22:05:48  *** Cheeseo has quit IRC
642 2018-02-13T22:10:27  *** Murch has quit IRC
643 2018-02-13T22:10:49  *** Cogito_Ergo_Sum has quit IRC
644 2018-02-13T22:11:26  *** Guyver2 has quit IRC
645 2018-02-13T22:13:06  *** Murch has joined #bitcoin-core-dev
646 2018-02-13T22:14:23  *** mrannanay has quit IRC
647 2018-02-13T22:15:31  *** jamesob_ has quit IRC
648 2018-02-13T22:15:53  *** Randolf has joined #bitcoin-core-dev
649 2018-02-13T22:17:46  *** promag has joined #bitcoin-core-dev
650 2018-02-13T22:19:01  *** Randolf has quit IRC
651 2018-02-13T22:21:03  *** Murch has quit IRC
652 2018-02-13T22:25:29  *** tryphe has quit IRC
653 2018-02-13T22:25:54  *** tryphe has joined #bitcoin-core-dev
654 2018-02-13T22:27:25  *** Murch has joined #bitcoin-core-dev
655 2018-02-13T22:48:57  *** owowo has quit IRC
656 2018-02-13T22:51:41  <eklitzke> this is kind of cool, i got a build of bitcoind working with systemtap markers (in this case I'm marking CCoinsViewCache flushes): https://gist.github.com/eklitzke/8bf6957fe886ddec36cde737d69ac6f5
657 2018-02-13T22:56:32  *** PaulCape_ has quit IRC
658 2018-02-13T22:59:03  *** PaulCapestany has joined #bitcoin-core-dev
659 2018-02-13T22:59:11  *** dongcarl has joined #bitcoin-core-dev
660 2018-02-13T23:00:52  <gmaxwell> eklitzke: whats the advantage of that?
661 2018-02-13T23:04:29  *** Murch has quit IRC
662 2018-02-13T23:05:30  *** Murch has joined #bitcoin-core-dev
663 2018-02-13T23:13:33  *** Emcy has quit IRC
664 2018-02-13T23:13:45  <eklitzke> i'm going to add more probes and then use that to measure some work i'm trying to do to the ccoinscacheview, e.g. i want to make the cache into a real writeback cache and get cache hit statistics out of it
665 2018-02-13T23:14:42  <eklitzke> today if you wanted to get cache hit statistics from CCoinCacheView::GetCoin you'd have to hack up some existing rpc to dump the information (or dump it to a log), with stap/dtrace probes that would be much easier and less invasive
666 2018-02-13T23:16:15  <warren> Does anyone have working gitian with qemu-kvm instead of lxc? I want to compare notes.
667 2018-02-13T23:18:35  <gmaxwell> eklitzke: you'll defeat its purpose entirely
668 2018-02-13T23:20:13  *** larafale has quit IRC
669 2018-02-13T23:20:28  <gmaxwell> eklitzke: the cache's purpose for existing is not to act as a cache, it's purpose is to act as a buffer to prevent writes from hitting the database entirely. And also to allow atomic updates to keep the database consistent with the chain at a block level (this is not as important since we now use a replay for consistency).  It also satisfies reads, but if you defeat that it doesn't make a huge p
670 2018-02-13T23:20:34  <gmaxwell> erformance difference in IBD.
671 2018-02-13T23:20:56  *** larafale has joined #bitcoin-core-dev
672 2018-02-13T23:20:57  *** Emcy has joined #bitcoin-core-dev
673 2018-02-13T23:21:46  <gmaxwell> okay maybe on a non-SSD the read cache matters somewhat more, to be fair I haven't benchmarked it with defeated read caching on a spinning disk.
674 2018-02-13T23:21:47  <eklitzke> i have thought through these issues, you can increase the amount of coins in memory during ibd in a way that's safe and still batches disk writes
675 2018-02-13T23:21:57  <eklitzke> yes this is primarily for spinning disks
676 2018-02-13T23:22:04  <eklitzke> or worse, people's cloud vm instances
677 2018-02-13T23:22:45  <sipa> eklitzke: i've experimented with various cache flushing strategies before which try to predict which entries will be more useful in the near future, and keep them longer
678 2018-02-13T23:22:52  <gmaxwell> eklitzke: the purpose of the replay stuff we did in 0.15.x was in part to enable incremental flushing, so yes, I also see advantages there.
679 2018-02-13T23:23:04  <sipa> in practice, anything that reduced the ability to avoid writes slowed things down
680 2018-02-13T23:23:14  <gmaxwell> but what we did find is that trying to achieve a higher read hit rate is mostly pointless.
681 2018-02-13T23:23:31  <gmaxwell> (on a SSD at least)
682 2018-02-13T23:24:08  <gmaxwell> I fear we've more or less given up on performance on non-SSDs :(  esp since many large spinning disks are shingled now and have horrifying performance.
683 2018-02-13T23:24:51  <gmaxwell> Incremenal flushing would also give the advantage of getting rid of flushing latency spikes and perhaps better overlap IO and validation.
684 2018-02-13T23:24:57  *** larafale has quit IRC
685 2018-02-13T23:24:59  <eklitzke> the flushing code right now is pretty inefficient, anyway databases like mysql do this by having a dirty buffer pool watermark that causes them to flush (potentially a large amount of data), and then they retaing data in memory that was flushed to disk if it's not dirty
686 2018-02-13T23:25:54  <eklitzke> if you kept exactly the same semantics wrt flushing a large amount of data but didn't automatically expire the old coins from memory that would be an improvement (in my opinion)
687 2018-02-13T23:25:58  *** AaronvanW has quit IRC
688 2018-02-13T23:26:20  <eklitzke> would love to chat with you about it next week, as i'm participating in the chaincode labs hacker residency program
689 2018-02-13T23:26:25  <gmaxwell> eklitzke: We've measured that, it doesn't help.
690 2018-02-13T23:26:34  *** AaronvanW has joined #bitcoin-core-dev
691 2018-02-13T23:26:47  <eklitzke> i've also measured it and shown that it does
692 2018-02-13T23:26:48  <gmaxwell> Thats the whole reason I brought the subject up-- so you don't assume read caching is important.
693 2018-02-13T23:27:18  <gmaxwell> eklitzke: Interesting, I'd love to see those results.
694 2018-02-13T23:27:19  <eklitzke> if there are any writeups or mailing list posts about experiments in this area i'd be interested to read them
695 2018-02-13T23:27:39  <gmaxwell> eklitzke: probably in the PR introducing the replay on start there are some of them.
696 2018-02-13T23:27:49  <eklitzke> ok, i'll take a look
697 2018-02-13T23:28:57  *** AaronvanW has quit IRC
698 2018-02-13T23:29:20  <sipa> eklitzke: i'll be at the residency on 22nd and 23rd
699 2018-02-13T23:29:24  <gmaxwell> I'd offer you the specific code I benchmarked but I don't have access to the system with it anymore.
700 2018-02-13T23:29:36  *** AaronvanW has joined #bitcoin-core-dev
701 2018-02-13T23:30:08  *** jojeyh has quit IRC
702 2018-02-13T23:31:02  <sipa> eklitzke: the bitcoin utxo set is pretty unique as a data set in that a very large number of newly written entries are almost immediately after deleted again
703 2018-02-13T23:31:16  *** AaronvanW has quit IRC
704 2018-02-13T23:31:26  *** Chris_Stewart_5 has quit IRC
705 2018-02-13T23:31:35  <gmaxwell> We tested many configurations.  For an example, one of the specific cases we tested was having flush leave the entries in but marking them as non-dirty and existing in the backing store, and seperately removing non-dirty entries when full.  What appeared to be the case is that avoiding the read hit seemed basically irrelevant, because if i got read, that means it is getting spent, which means th
706 2018-02-13T23:31:41  <gmaxwell> ere is going to be an erasure which must hit the disk.
707 2018-02-13T23:31:51  *** AaronvanW has joined #bitcoin-core-dev
708 2018-02-13T23:32:04  <TD-Linux> since many large spinning disks are shingled now and have horrifying performance. <- these are still really uncommon
709 2018-02-13T23:32:33  <gmaxwell> TD-Linux: they seem to be 100% of people showing up and talking about their node performance on spinning disks, but now that I say that outloud...
710 2018-02-13T23:32:34  <sipa> eklitzke: but we should talk more about this
711 2018-02-13T23:33:03  <eklitzke> i would like to change some of the leveldb settings too, but that's another matter entirely; for instance leveldb is configured to only open 64 files at once, so during ibd or a reindex you see the process thrashing a lot in a pattern where it opens an lbd file, mmaps it, munmaps it, and closes it
712 2018-02-13T23:33:09  <eklitzke> that's another matter entirely though
713 2018-02-13T23:33:20  <gmaxwell> eklitzke: by all means if you have results, great.
714 2018-02-13T23:33:38  *** TheRec_ is now known as TheRec
715 2018-02-13T23:33:52  <gmaxwell> eklitzke: we suffer from FD exhaustion issues currently. :(
716 2018-02-13T23:34:35  <eklitzke> yeah I talked to BlueMatt about it a bit yesterday, the thing is you can mmap a file, close the fd, and the mmap mapping is still valid
717 2018-02-13T23:34:56  <gmaxwell> ah, cute.
718 2018-02-13T23:34:57  <eklitzke> so if you didn't mind the extra memory overhead from the PTEs, you could potentially mmap all of the ldb files at once and not actually have 1000+ ldb files open
719 2018-02-13T23:35:05  <eklitzke> doing *all* of them is probably too agressive
720 2018-02-13T23:35:12  <gmaxwell> leveldb doesn't use mmap on 32 bit hosts.
721 2018-02-13T23:35:27  <gmaxwell> and on 64-bit the only harm is the PTE as you note, which is pretty minor.
722 2018-02-13T23:36:34  *** AaronvanW has quit IRC
723 2018-02-13T23:36:42  *** rex_4539 has quit IRC
724 2018-02-13T23:37:21  <gmaxwell> eklitzke: if you want to get into leveldb internals there are probably a lot of other gains. E.g. our keys are almost uniformly random, and so bisections in leveldb can probably be replaced with a secant search.  (and if our keys aren't random enough, we could fix that)   leveldb block size is tunable too
725 2018-02-13T23:37:45  <gmaxwell> I think I found larger blocks to be a loss, but that might not be as true with a secant search.
726 2018-02-13T23:38:19  <eklitzke> yeah I was actually surprised that leveldb itself wasn't doing the mmap trick I just described (or at least have it as a tuneable parameter), I want to dig deeper into their code
727 2018-02-13T23:39:00  <promag> ryanofsky: friendly ping https://github.com/bitcoin/bitcoin/pull/11687#pullrequestreview-94272991
728 2018-02-13T23:40:50  <gmaxwell> eklitzke: I have a believe which I've not been able to validate yet, that a lot of the performance limitations of the current dbcache is all the malloc activity... that often doesn't show up too clearly in profiles. We'd talked about changing the main dbcache to an open hashtable with fixed size entries (and some special indirection to an exception map for unusually large entries).. but it's a l
729 2018-02-13T23:40:56  <gmaxwell> ot of work to just try it and see if it helps.
730 2018-02-13T23:42:10  <eklitzke> interesting because glibc malloc already has systemtap probes bulitin, since 2.24 i think, so that would go well with my systemtap branch because you could write stap scripts that correlated activity in malloc with events in bitcoind
731 2018-02-13T23:42:42  *** meshcollider has quit IRC
732 2018-02-13T23:43:43  *** AaronvanW has joined #bitcoin-core-dev
733 2018-02-13T23:55:13  *** vicenteH has quit IRC
734 2018-02-13T23:58:02  *** Madscotslad has joined #bitcoin-core-dev
735 2018-02-13T23:58:24  <Madscotslad> Wow nice to see a full IRC room its been a while :) hey folks.
736 2018-02-13T23:58:51  <luke-jr> if you think this is full, you should see #bitcoin