1 2017-10-05T00:01:41  *** Alina-malina has joined #bitcoin-core-dev
  2 2017-10-05T00:12:21  *** Alina-malina has quit IRC
  3 2017-10-05T00:13:16  *** Alina-malina has joined #bitcoin-core-dev
  4 2017-10-05T00:20:59  *** Ylbam has quit IRC
  5 2017-10-05T00:21:33  *** Alina-malina has quit IRC
  6 2017-10-05T00:23:16  *** Alina-malina has joined #bitcoin-core-dev
  7 2017-10-05T00:25:25  *** AaronvanW has quit IRC
  8 2017-10-05T00:32:12  *** arubi has quit IRC
  9 2017-10-05T00:37:03  *** arubi has joined #bitcoin-core-dev
 10 2017-10-05T00:43:24  *** Alina-malina has quit IRC
 11 2017-10-05T00:45:48  *** Alina-malina has joined #bitcoin-core-dev
 12 2017-10-05T00:53:52  *** Alina-malina has quit IRC
 13 2017-10-05T00:56:19  *** Alina-malina has joined #bitcoin-core-dev
 14 2017-10-05T01:02:05  *** Alina-malina has quit IRC
 15 2017-10-05T01:05:16  *** Alina-malina has joined #bitcoin-core-dev
 16 2017-10-05T01:07:06  *** jb55 has joined #bitcoin-core-dev
 17 2017-10-05T01:11:23  *** Alina-malina has quit IRC
 18 2017-10-05T01:12:43  *** Alina-malina has joined #bitcoin-core-dev
 19 2017-10-05T01:31:02  *** Chris_Stewart_5 has joined #bitcoin-core-dev
 20 2017-10-05T01:39:22  *** Murch has quit IRC
 21 2017-10-05T01:48:01  *** d9b4bef9 has quit IRC
 22 2017-10-05T01:49:08  *** d9b4bef9 has joined #bitcoin-core-dev
 23 2017-10-05T02:00:47  *** berndj has quit IRC
 24 2017-10-05T02:01:34  *** ybit has quit IRC
 25 2017-10-05T02:02:03  *** berndj has joined #bitcoin-core-dev
 26 2017-10-05T02:03:35  *** ybit has joined #bitcoin-core-dev
 27 2017-10-05T02:13:52  *** Alina-malina has quit IRC
 28 2017-10-05T02:15:51  *** Alina-malina has joined #bitcoin-core-dev
 29 2017-10-05T02:21:24  *** ula has quit IRC
 30 2017-10-05T02:22:32  *** Chris_Stewart_5 has quit IRC
 31 2017-10-05T02:27:46  *** wxxs has quit IRC
 32 2017-10-05T02:35:43  *** Alina-malina has quit IRC
 33 2017-10-05T02:38:52  *** Alina-malina has joined #bitcoin-core-dev
 34 2017-10-05T02:51:48  *** atroxes has quit IRC
 35 2017-10-05T02:52:32  *** atroxes has joined #bitcoin-core-dev
 36 2017-10-05T03:05:57  *** [Author] has quit IRC
 37 2017-10-05T03:24:45  *** [Author] has joined #bitcoin-core-dev
 38 2017-10-05T03:29:14  *** wraithm has joined #bitcoin-core-dev
 39 2017-10-05T03:29:20  *** meshcollider has quit IRC
 40 2017-10-05T03:33:33  *** wraithm has quit IRC
 41 2017-10-05T03:36:36  <bitcoin-git> [bitcoin] jonasschnelli closed pull request #10517: Factor out CCoinsView based AreInputsStandard/IsWitnessStandard (master...2017/06/policy_compile) https://github.com/bitcoin/bitcoin/pull/10517
 42 2017-10-05T03:37:42  <bitcoin-git> [bitcoin] jonasschnelli closed pull request #10238: Change setKeyPool to hold flexible entries (master...2017/04/keypool_fix_a) https://github.com/bitcoin/bitcoin/pull/10238
 43 2017-10-05T04:05:30  <bitcoin-git> [bitcoin] jonasschnelli closed pull request #10251: Add balances cache / GUI: use a signal instead of a poll thread (master...2017/04/gui_rm_locks) https://github.com/bitcoin/bitcoin/pull/10251
 44 2017-10-05T04:09:48  *** arubi has quit IRC
 45 2017-10-05T05:29:34  *** Ylbam has joined #bitcoin-core-dev
 46 2017-10-05T05:41:33  *** jb55 has quit IRC
 47 2017-10-05T06:03:01  *** d9b4bef9 has quit IRC
 48 2017-10-05T06:04:07  *** d9b4bef9 has joined #bitcoin-core-dev
 49 2017-10-05T06:28:19  *** intcat has quit IRC
 50 2017-10-05T06:38:45  *** Emcy has quit IRC
 51 2017-10-05T07:06:15  *** protomar has joined #bitcoin-core-dev
 52 2017-10-05T07:06:18  *** BashCo has quit IRC
 53 2017-10-05T07:21:24  *** SopaXorzTaker has joined #bitcoin-core-dev
 54 2017-10-05T07:25:38  *** BashCo has joined #bitcoin-core-dev
 55 2017-10-05T07:44:19  *** meshcollider has joined #bitcoin-core-dev
 56 2017-10-05T07:46:09  *** JackH has joined #bitcoin-core-dev
 57 2017-10-05T07:54:59  *** AaronvanW has joined #bitcoin-core-dev
 58 2017-10-05T08:01:45  *** DrOlmer has joined #bitcoin-core-dev
 59 2017-10-05T08:17:18  *** Guyver2 has joined #bitcoin-core-dev
 60 2017-10-05T08:31:30  *** timothy has joined #bitcoin-core-dev
 61 2017-10-05T08:39:46  *** AaronvanW has quit IRC
 62 2017-10-05T08:41:05  *** AaronvanW has joined #bitcoin-core-dev
 63 2017-10-05T08:45:07  *** Ylbam has quit IRC
 64 2017-10-05T08:46:07  *** arubi has joined #bitcoin-core-dev
 65 2017-10-05T08:57:01  *** promag has joined #bitcoin-core-dev
 66 2017-10-05T09:07:33  *** ghost43 has quit IRC
 67 2017-10-05T09:08:36  *** promag has quit IRC
 68 2017-10-05T09:08:53  *** promag has joined #bitcoin-core-dev
 69 2017-10-05T09:15:15  *** pigeons has quit IRC
 70 2017-10-05T09:16:08  *** rabidus has quit IRC
 71 2017-10-05T09:18:40  *** promag has quit IRC
 72 2017-10-05T09:21:40  *** promag has joined #bitcoin-core-dev
 73 2017-10-05T09:43:42  *** AaronvanW has quit IRC
 74 2017-10-05T09:44:27  *** AaronvanW has joined #bitcoin-core-dev
 75 2017-10-05T09:44:54  *** promag has quit IRC
 76 2017-10-05T09:46:17  *** promag has joined #bitcoin-core-dev
 77 2017-10-05T09:48:12  *** AaronvanW has quit IRC
 78 2017-10-05T09:49:55  *** AaronvanW has joined #bitcoin-core-dev
 79 2017-10-05T09:51:42  *** AaronvanW has joined #bitcoin-core-dev
 80 2017-10-05T09:53:00  *** wxxs has joined #bitcoin-core-dev
 81 2017-10-05T09:54:44  *** promag has quit IRC
 82 2017-10-05T09:55:32  *** Lesley has joined #bitcoin-core-dev
 83 2017-10-05T10:21:17  *** promag has joined #bitcoin-core-dev
 84 2017-10-05T10:22:56  *** promag has quit IRC
 85 2017-10-05T10:26:11  *** W4RL0RD has joined #bitcoin-core-dev
 86 2017-10-05T10:27:53  *** Aaronvan_ has joined #bitcoin-core-dev
 87 2017-10-05T10:31:03  *** AaronvanW has quit IRC
 88 2017-10-05T11:16:38  *** Aaronvan_ has quit IRC
 89 2017-10-05T11:17:14  *** AaronvanW has joined #bitcoin-core-dev
 90 2017-10-05T11:18:36  *** AaronvanW has joined #bitcoin-core-dev
 91 2017-10-05T11:22:39  <sturles> I have a problem with boost after upgrading to Debian 9.  I did an apt-get --reinstall install libboost-all-dev to be sure I have everything, but it still fails: https://0bin.net/paste/NjYBdILEZBChNmbI#rsDCuSOm7iKfR-UZj7ytJdxoAYMLRdB3szd43qbZmvH
 92 2017-10-05T11:23:24  *** promag has joined #bitcoin-core-dev
 93 2017-10-05T11:24:02  <sturles> Any ideas?
 94 2017-10-05T11:28:33  *** promag has quit IRC
 95 2017-10-05T11:40:14  *** protomar has quit IRC
 96 2017-10-05T11:52:33  *** Alina-malina has quit IRC
 97 2017-10-05T11:53:45  *** Alina-malina has joined #bitcoin-core-dev
 98 2017-10-05T12:00:00  *** wxxs has quit IRC
 99 2017-10-05T12:00:01  *** Alina-malina has quit IRC
100 2017-10-05T12:01:27  *** wxxs has joined #bitcoin-core-dev
101 2017-10-05T12:03:45  *** Alina-malina has joined #bitcoin-core-dev
102 2017-10-05T12:08:40  *** rabidus has joined #bitcoin-core-dev
103 2017-10-05T12:10:20  *** SopaXorzTaker has quit IRC
104 2017-10-05T12:15:40  *** afk11 has quit IRC
105 2017-10-05T12:16:01  *** d9b4bef9 has quit IRC
106 2017-10-05T12:17:09  *** d9b4bef9 has joined #bitcoin-core-dev
107 2017-10-05T12:22:53  *** SopaXorzTaker has joined #bitcoin-core-dev
108 2017-10-05T12:34:08  *** meshcollider has quit IRC
109 2017-10-05T12:40:38  *** SopaXorzTaker has quit IRC
110 2017-10-05T12:42:09  *** SopaXorzTaker has joined #bitcoin-core-dev
111 2017-10-05T13:09:15  <bitcoin-git> [bitcoin] MarcoFalke pushed 5 new commits to master: https://github.com/bitcoin/bitcoin/compare/167cef8082e2...e93fff1463ae
112 2017-10-05T13:09:16  <bitcoin-git> bitcoin/master 58d91af MeshCollider: Fix race for mapBlockIndex in AppInitMain
113 2017-10-05T13:09:16  <bitcoin-git> bitcoin/master 35aeabe MeshCollider: Make fReindex atomic to avoid race
114 2017-10-05T13:09:17  <bitcoin-git> bitcoin/master 731065b MeshCollider: Consistent parameter names in txdb.h
115 2017-10-05T13:09:49  <bitcoin-git> [bitcoin] MarcoFalke closed pull request #11107: Fix races in AppInitMain and others with lock and atomic bools (master...fix_mapBlockIndex_race) https://github.com/bitcoin/bitcoin/pull/11107
116 2017-10-05T13:36:28  *** Chris_Stewart_5 has joined #bitcoin-core-dev
117 2017-10-05T13:48:40  *** pigeons has joined #bitcoin-core-dev
118 2017-10-05T13:51:03  *** Alina-malina has quit IRC
119 2017-10-05T13:52:51  *** Alina-malina has joined #bitcoin-core-dev
120 2017-10-05T14:03:26  *** Alina-malina has quit IRC
121 2017-10-05T14:05:53  *** Alina-malina has joined #bitcoin-core-dev
122 2017-10-05T14:05:53  *** Emcy has joined #bitcoin-core-dev
123 2017-10-05T14:30:33  *** Chris_Stewart_5 has quit IRC
124 2017-10-05T14:38:56  *** rafalcpp has quit IRC
125 2017-10-05T14:39:58  *** rafalcpp has joined #bitcoin-core-dev
126 2017-10-05T14:44:12  *** wraithm has joined #bitcoin-core-dev
127 2017-10-05T14:49:25  *** arubi has quit IRC
128 2017-10-05T14:50:03  *** Chris_Stewart_5 has joined #bitcoin-core-dev
129 2017-10-05T15:00:20  *** deltaT has joined #bitcoin-core-dev
130 2017-10-05T15:01:19  <deltaT> simple question i hope, when i created my bitcoin core wallet i didnt write down the private key, how do i find it now?
131 2017-10-05T15:02:27  *** Chris_Stewart_5 has quit IRC
132 2017-10-05T15:06:43  <jonasschnelli> deltaT: which Bitcoin Core version?
133 2017-10-05T15:07:16  <jonasschnelli> deltaT: did you lost you wallet.dat? file.... also, this question should be asked in #bitcoin (this here is the development channel)
134 2017-10-05T15:07:26  <deltaT> 0.14.2
135 2017-10-05T15:07:27  <BlueMatt> wumpus: #9572 looks relatively merge-able...it is consensus so if you want to ack first that'd also be nice, but its simple and has 4 already
136 2017-10-05T15:07:29  <gribble> https://github.com/bitcoin/bitcoin/issues/9572 | Skip witness sighash cache for non-segwit transactions by jl2012 · Pull Request #9572 · bitcoin/bitcoin · GitHub
137 2017-10-05T15:08:56  *** deltaT has quit IRC
138 2017-10-05T15:11:57  *** jb55 has joined #bitcoin-core-dev
139 2017-10-05T15:13:39  <bitcoin-git> [bitcoin] mess110 opened pull request #11455: CTxMemPool::GetMinFee should not return CFeeRate(0) (master...fix_mempool_GetMinFee_bug_returning_below_minRelayTxFee) https://github.com/bitcoin/bitcoin/pull/11455
140 2017-10-05T15:16:06  *** rafalcpp_ has joined #bitcoin-core-dev
141 2017-10-05T15:17:01  *** rafalcpp has quit IRC
142 2017-10-05T15:18:38  <bitcoin-git> [bitcoin] TheBlueMatt opened pull request #11456: Replace relevant services logic with a function suite. (master...2017-09-service-flags-cleanups) https://github.com/bitcoin/bitcoin/pull/11456
143 2017-10-05T15:20:19  <bitcoin-git> [bitcoin] mess110 closed pull request #11410: [rpc] [tests] mempoolminfee should not drop below minRelayTxFee (master...add_minrelaytxfee_to_getmempoolinfo) https://github.com/bitcoin/bitcoin/pull/11410
144 2017-10-05T15:26:13  *** jb55 has quit IRC
145 2017-10-05T15:28:28  *** tloriato has joined #bitcoin-core-dev
146 2017-10-05T15:31:17  <tloriato> Good afternoon. I was looking into BIP45, that tries to standardize the Structure for Deterministic P2SH Multisignature Wallets, but I've read the emails discussing it and seems to have a little or disagreement between the need for this particular BIP. I'm wondering if there is another standard way to generate a Deterministic Multisig Wallet? I was inclined to generate a standard Mnemonic (BIP39) and split it using Shamir
147 2017-10-05T15:33:10  *** arubi has joined #bitcoin-core-dev
148 2017-10-05T15:36:28  <BlueMatt> sipa: did you find time to write up the segwit wallet tradeoffs between the various ways of doing it and making an argument for why your pr is your preferred version?
149 2017-10-05T15:36:36  <BlueMatt> sipa: I believe you said you were gonna do that last meeting
150 2017-10-05T15:48:27  *** Chris_Stewart_5 has joined #bitcoin-core-dev
151 2017-10-05T15:50:19  <jonasschnelli> tloriato: AFAIK BIP45 is the only "standard" to create multisig addresses with a set of given pubkeys.
152 2017-10-05T15:50:27  <jonasschnelli> *extended pubkeys
153 2017-10-05T15:53:50  <tloriato> jonasschnelli: thanks
154 2017-10-05T15:55:55  *** rafalcpp_ has quit IRC
155 2017-10-05T15:56:15  *** rafalcpp has joined #bitcoin-core-dev
156 2017-10-05T16:00:30  *** Murch has joined #bitcoin-core-dev
157 2017-10-05T16:00:45  *** tloriato has quit IRC
158 2017-10-05T16:00:47  * BlueMatt has a super interesting C++ performance puzzle if someone wants it: https://github.com/bitcoinfibre/bitcoinfibre/blob/matts-servers/src/udprelay.cpp#L501 sometimes (about once every 2nd or 3rd day's worth of blocks) takes on the rorder of 30ms on dedicated hardware (!!!), but all it does is allocate
159 2017-10-05T16:00:51  <BlueMatt> nothing else does anything close
160 2017-10-05T16:01:41  *** DrOlmer has quit IRC
161 2017-10-05T16:02:38  *** DrOlmer has joined #bitcoin-core-dev
162 2017-10-05T16:05:08  *** Emcy has quit IRC
163 2017-10-05T16:05:18  *** BashCo has quit IRC
164 2017-10-05T16:05:35  *** Emcy has joined #bitcoin-core-dev
165 2017-10-05T16:05:52  *** BashCo has joined #bitcoin-core-dev
166 2017-10-05T16:06:40  <bitcoin-git> [bitcoin] laanwj pushed 3 new commits to master: https://github.com/bitcoin/bitcoin/compare/e93fff1463ae...becbd71b0c16
167 2017-10-05T16:06:41  <bitcoin-git> bitcoin/master 4f890ba Donal OConnor: Add new step to clean $PATH var by removing /mnt specific Window's %PATH% paths that cause issues with the make system
168 2017-10-05T16:06:41  <bitcoin-git> bitcoin/master 696ce46 fanquake: [Docs] Update Windows build instructions for using WSL and Ubuntu 17.04
169 2017-10-05T16:06:42  <bitcoin-git> bitcoin/master becbd71 Wladimir J. van der Laan: Merge #11437: [Docs] Update Windows build instructions for using WSL and Ubuntu 17.04...
170 2017-10-05T16:07:25  <bitcoin-git> [bitcoin] laanwj closed pull request #11437: [Docs] Update Windows build instructions for using WSL and Ubuntu 17.04 (master...windows-build-1704) https://github.com/bitcoin/bitcoin/pull/11437
171 2017-10-05T16:08:15  <bitcoin-git> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/becbd71b0c16...9e8ef9d99179
172 2017-10-05T16:08:16  <bitcoin-git> bitcoin/master f3ba869 practicalswift: [tests] Add libFuzzer support....
173 2017-10-05T16:08:16  <bitcoin-git> bitcoin/master 9e8ef9d Wladimir J. van der Laan: Merge #10440: [tests] Add libFuzzer support...
174 2017-10-05T16:08:30  <bitcoin-git> [bitcoin] laanwj closed pull request #10440: [tests] Add libFuzzer support (master...libfuzzer) https://github.com/bitcoin/bitcoin/pull/10440
175 2017-10-05T16:10:05  *** BashCo has quit IRC
176 2017-10-05T16:17:12  <sipa> BlueMatt: i've been distracted by something else
177 2017-10-05T16:18:05  *** jb55 has joined #bitcoin-core-dev
178 2017-10-05T16:35:53  *** Aaronvan_ has joined #bitcoin-core-dev
179 2017-10-05T16:39:22  *** AaronvanW has quit IRC
180 2017-10-05T16:40:12  *** Aaronvan_ has quit IRC
181 2017-10-05T16:42:50  *** BashCo has joined #bitcoin-core-dev
182 2017-10-05T16:45:38  *** abpa has joined #bitcoin-core-dev
183 2017-10-05T16:53:56  *** AaronvanW has joined #bitcoin-core-dev
184 2017-10-05T17:02:54  <bitcoin-git> [bitcoin] sdaftuar closed pull request #10200: Mining: Skip recent transactions if fee difference is small (master...2017-04-dont-mine-recent-tx) https://github.com/bitcoin/bitcoin/pull/10200
185 2017-10-05T17:06:03  *** Emcy has quit IRC
186 2017-10-05T17:06:47  *** Emcy has joined #bitcoin-core-dev
187 2017-10-05T17:12:30  *** Aaronvan_ has joined #bitcoin-core-dev
188 2017-10-05T17:12:37  *** promag has joined #bitcoin-core-dev
189 2017-10-05T17:13:37  *** timothy has quit IRC
190 2017-10-05T17:14:31  *** Emcy has quit IRC
191 2017-10-05T17:14:44  *** Emcy has joined #bitcoin-core-dev
192 2017-10-05T17:14:59  *** kingjockey has joined #bitcoin-core-dev
193 2017-10-05T17:15:28  *** Aaronvan_ has quit IRC
194 2017-10-05T17:16:07  *** Aaronvan_ has joined #bitcoin-core-dev
195 2017-10-05T17:16:29  *** AaronvanW has quit IRC
196 2017-10-05T17:19:48  *** promag has quit IRC
197 2017-10-05T17:25:01  *** Ylbam has joined #bitcoin-core-dev
198 2017-10-05T17:26:27  *** Emcy has quit IRC
199 2017-10-05T17:27:50  *** Emcy has joined #bitcoin-core-dev
200 2017-10-05T17:30:12  *** Chris_Stewart_5 has quit IRC
201 2017-10-05T17:44:12  *** Guyver2 has quit IRC
202 2017-10-05T17:48:01  *** RealM9 has joined #bitcoin-core-dev
203 2017-10-05T17:48:40  <RealM9> Hello, how long until meeting starts? At what time it starts?
204 2017-10-05T17:49:30  <sipa> 11 minutes from now
205 2017-10-05T17:49:44  <sipa> sorry, 71 minutes
206 2017-10-05T17:50:11  <sipa> it's at 7pm UTC
207 2017-10-05T17:50:16  <bitcoin-git> [bitcoin] laanwj pushed 2 new commits to master: https://github.com/bitcoin/bitcoin/compare/9e8ef9d99179...17f2acedbe07
208 2017-10-05T17:50:16  <bitcoin-git> bitcoin/master 0da49b5 Johnson Lau: Skip precompute sighash for transactions without witness
209 2017-10-05T17:50:17  <bitcoin-git> bitcoin/master 17f2ace Wladimir J. van der Laan: Merge #9572: Skip witness sighash cache for non-segwit transactions...
210 2017-10-05T17:50:23  <bitcoin-git> [bitcoin] laanwj closed pull request #9572: Skip witness sighash cache for non-segwit transactions (master...nocache) https://github.com/bitcoin/bitcoin/pull/9572
211 2017-10-05T17:51:09  *** promag has joined #bitcoin-core-dev
212 2017-10-05T17:51:16  <RealM9> Ok, thnx sipa
213 2017-10-05T17:57:05  *** Aaronvan_ has quit IRC
214 2017-10-05T17:57:16  *** AaronvanW has joined #bitcoin-core-dev
215 2017-10-05T17:58:35  *** kingjockey has quit IRC
216 2017-10-05T17:58:51  *** promag has quit IRC
217 2017-10-05T17:59:31  <jl2012> why do we need a vector of PrecomputedTransactionData here? https://github.com/bitcoin/bitcoin/blob/master/src/validation.cpp#L1755
218 2017-10-05T17:59:36  *** laurentmt has joined #bitcoin-core-dev
219 2017-10-05T18:00:28  <sipa> jl2012: what alternative do you suggest?
220 2017-10-05T18:00:29  *** AaronvanW has quit IRC
221 2017-10-05T18:00:38  <jl2012> why couldn't we just have a PrecomputedTransactionData for each transaction?
222 2017-10-05T18:00:48  <sipa> we do?
223 2017-10-05T18:00:57  <sipa> that's just the vector that holds them
224 2017-10-05T18:01:06  *** AaronvanW has joined #bitcoin-core-dev
225 2017-10-05T18:01:34  <jl2012> yes, but why need to hold them all?
226 2017-10-05T18:01:59  <sipa> variables need to exist somewhere...
227 2017-10-05T18:03:23  <jl2012> yes, I mean, why not just have a line PrecomputedTransactionData txdata(tx); inside the loop "for (unsigned int i = 0; i < block.vtx.size(); i++)"?
228 2017-10-05T18:03:59  <sipa> jl2012: oh, i read "why need to them *at* all", sorry
229 2017-10-05T18:04:18  <sipa> the reason is that they're used from other threads during parallel validation
230 2017-10-05T18:05:41  <jl2012> ok...so without the vector, each thread will re-compute the hashes once?
231 2017-10-05T18:06:07  <sipa> depends how you do it
232 2017-10-05T18:06:26  <sipa> if you do what you suggest, you'd get undefined behaviour
233 2017-10-05T18:06:49  <jl2012> why?
234 2017-10-05T18:06:53  <sipa> as the precomputation data would be out of scope before it's accessed from the validation threads
235 2017-10-05T18:07:39  *** Emcy has quit IRC
236 2017-10-05T18:08:09  *** W4RL0RD has quit IRC
237 2017-10-05T18:08:15  <sipa> most of the validation happens after that loop exits, and before the queue's wait call returns
238 2017-10-05T18:08:18  *** Emcy has joined #bitcoin-core-dev
239 2017-10-05T18:10:35  <JeremyRubin> jl2012: I have a patch that makes them shared_ptrs
240 2017-10-05T18:10:45  <BlueMatt> sipa: not anymore (tx script caching) :p
241 2017-10-05T18:10:46  <JeremyRubin> conceptually, that's what you want
242 2017-10-05T18:10:55  <JeremyRubin> performance wise, a vector is better
243 2017-10-05T18:12:49  <jonasschnelli> gmaxwell: for the notification system, you had concerns about the reliability.. would a long poll queue that only removes elements from the queue on an ack from the client be something that would defeat your concerns?
244 2017-10-05T18:13:00  <jonasschnelli> With that concept, loosing notifications would be very unlikely
245 2017-10-05T18:13:04  <sipa> BlueMatt: ok, i reformulate - most of the validation, if it happens, happens after that loop exits
246 2017-10-05T18:15:34  <jl2012> oh, I thought all script validation is done with CheckInputs?
247 2017-10-05T18:15:55  <sipa> jl2012: yes and no
248 2017-10-05T18:16:17  <sipa> CheckInputs returns a vector of CScriptCheck objects, which represent validation that will happen on another thread
249 2017-10-05T18:16:33  <sipa> those get pushed to the checkqueue, where validation threads pick them up
250 2017-10-05T18:16:35  *** Emcy has quit IRC
251 2017-10-05T18:16:51  <jl2012> ontrol.Add(vChecks); ?
252 2017-10-05T18:16:58  <jl2012> control.Add(vChecks); ?
253 2017-10-05T18:16:59  <JeremyRubin> yes
254 2017-10-05T18:17:20  <gmaxwell> jonasschnelli: it would but wumpus pointed out that that queue would potentially grow without bound if the client stops acking entirely.
255 2017-10-05T18:17:58  <sipa> jl2012: and the CScriptCheck objects have a pointer to the precomputation data
256 2017-10-05T18:18:05  <JeremyRubin> control.Add returns immediately
257 2017-10-05T18:18:12  <wumpus> yes, 100% reliable notification given any behavior of the receiver is not possible given finite space
258 2017-10-05T18:18:24  <jonasschnelli> gmaxwell: a queue limit is unavoidable... if you poll to lazy and the queue limit is to little, the only way to detect would be via the sequence number
259 2017-10-05T18:18:31  <sipa> jl2012: which means we must guarantee that that precomputation data remains alive as long as other threads may dereference the pointer
260 2017-10-05T18:18:40  <wumpus> some notification queue middleware logs events to disk in that case
261 2017-10-05T18:18:43  *** abpa has quit IRC
262 2017-10-05T18:18:58  <jonasschnelli> wumpus: in long polling, the queue must be finite because we don't know when the client polls next,...
263 2017-10-05T18:19:14  <wumpus> e.g. rabbitmq etc, not by any means suggesting we take that up
264 2017-10-05T18:19:21  *** AaronvanW has quit IRC
265 2017-10-05T18:19:53  <wumpus> jonasschnelli: I agree there needs to be a limit realistically, it's enough if a client can detect missed events to resync
266 2017-10-05T18:19:58  *** AaronvanW has joined #bitcoin-core-dev
267 2017-10-05T18:20:08  <jonasschnelli> I personally think acking notifications is unnecessary,.. making it configurable seems also an overkill... so unsure.
268 2017-10-05T18:20:32  <JeremyRubin> jl2012: this is why I say a shared_ptr is what you really want, because the lifetime is automatically handled for you, whereas the vector is only by careful programming. However, that careful programming is currently correct :)
269 2017-10-05T18:21:18  <sipa> jonasschnelli: if you don't ask notifications, then your clients must manually deal with restarts... which implies having a way to ask for all events since some time... which, if you have it, removes the need for an event log entirely, as you can just use that all the time
270 2017-10-05T18:21:28  <jl2012> JeremyRubin, sipa: thanks. I'm thinking how I could do this: https://github.com/bitcoin/bitcoin/pull/9572#issuecomment-334282408
271 2017-10-05T18:21:47  <wumpus> what's important is to document the limitations of the notification system in that regard
272 2017-10-05T18:22:10  <sipa> jl2012: do what?
273 2017-10-05T18:22:41  <jl2012> "I'd be a little more comfortable with this if PrecomputedTransactionData were passed around as const, so nobody's tempted to mess with "ready" for some crazy reason. But that can always be done later."
274 2017-10-05T18:22:54  <gmaxwell> shared_ptrs are a way to produce a software engineering disaster, because they let you be sloppy with ownership. There are cases of frestanding objects that don't have organized ownership but they are relatively rare. They also have non-trivial overhad.
275 2017-10-05T18:23:25  <sipa> jl2012: seems trivial?
276 2017-10-05T18:23:54  *** Emcy has joined #bitcoin-core-dev
277 2017-10-05T18:23:55  <sipa> i think you're overthinking it
278 2017-10-05T18:24:01  *** AaronvanW has quit IRC
279 2017-10-05T18:24:28  <jl2012> I'm thinking something like "const PrecomputedTransactionData txdata(tx);"
280 2017-10-05T18:25:30  <sipa> jl2012: cfields is only asking to pass it around as const; not to make the entire object const
281 2017-10-05T18:26:35  <cfields> ^^ yep
282 2017-10-05T18:27:17  <cfields> though i think making the members const would be trivial too
283 2017-10-05T18:27:32  <cfields> either way, it's not really important for that PR. just an observation.
284 2017-10-05T18:27:44  *** Emcy_ has joined #bitcoin-core-dev
285 2017-10-05T18:27:48  <jl2012> it's even better if we could make the object const? There is no reason to make any modification
286 2017-10-05T18:28:20  <JeremyRubin> gmaxwell: am aware, but strictly speaking that is the point here: they are designed to track the actual needed lifetime, and are a 'tighter fit' to the problem in this situation. That they have problems is why I didn't PR my patch.
287 2017-10-05T18:28:32  <cfields> jl2012: just make the members const, then. use initializers rather than setting them in the function body
288 2017-10-05T18:29:03  *** chjj has quit IRC
289 2017-10-05T18:29:04  <JeremyRubin> jl2012: I think that we shouldn't make it const, because there will be in the future per-tx state that we'll modify
290 2017-10-05T18:29:38  <JeremyRubin> e.g., if at some point in the future signature agg ends up needing a per-tx mutable struct that each input modifies
291 2017-10-05T18:29:52  <gmaxwell> JeremyRubin: in the case of the validation the ownership is just a straght line though, the creating code owns it, then the ownership is passed to the queue, then the ownership passes to the verifier thread..
292 2017-10-05T18:30:37  <JeremyRubin> Maybe it should be a different struct, but I think (to me), changing PreComputedTxData to PerTxData is straightforward.
293 2017-10-05T18:30:56  <sipa> JeremyRubin: i think that should be separate
294 2017-10-05T18:31:07  <sipa> if you want to avoid locking on the precomputed data
295 2017-10-05T18:31:29  *** Emcy has quit IRC
296 2017-10-05T18:31:32  <JeremyRubin> gmaxwell: yes, and then they are kept alive for longer than strictly needed. After the last scriptcheck executes for that tx, they can be freed.
297 2017-10-05T18:31:33  <cfields> agree. It'd be nice to keep the factual data separate from what's aggregated/stateful
298 2017-10-05T18:31:58  <sipa> JeremyRubin: yes, so? it doesn't change the worst case resource consumption
299 2017-10-05T18:32:04  <JeremyRubin> cfields: all the data should be factual in consensus?
300 2017-10-05T18:33:12  <jonasschnelli> sipa: persist the notification queue (with it's sequence numbers) to make it restart-safe?
301 2017-10-05T18:33:12  <JeremyRubin> sipa: I'm really only trying to talk about lifetimes here, which is the core of what jl2012 is discussing
302 2017-10-05T18:33:24  <JeremyRubin> sipa: asking
303 2017-10-05T18:35:21  <sipa> JeremyRubin: i'm just trying to point out that minimizing the time an object lives is not always wanted
304 2017-10-05T18:35:26  <cfields> JeremyRubin: i just meant that it can be safely computed ahead of time and seen as immutible after that.
305 2017-10-05T18:35:45  <cfields> factual was the wrong word, i suppose.
306 2017-10-05T18:36:09  *** Chris_Stewart_5 has joined #bitcoin-core-dev
307 2017-10-05T18:37:02  <JeremyRubin> sipa: of course
308 2017-10-05T18:38:31  <sipa> jonasschnelli: that's one way (but it has pretty bad costs, if you want to make it durable - now you need synchronization across all files/databases)
309 2017-10-05T18:39:10  <sipa> jonasschnelli: another model is that you just have RPCs like listsinceblock, where the client tells the server effectively what it already knows
310 2017-10-05T18:39:31  <sipa> jonasschnelli: and then all you need is a notification like "there's something for you to look at"
311 2017-10-05T18:40:06  <jonasschnelli> sipa: by a rolling hash?
312 2017-10-05T18:40:12  <sipa> jonasschnelli: wut?
313 2017-10-05T18:40:29  <sipa> no, just like listsinceblock
314 2017-10-05T18:40:30  <jonasschnelli> how would the client tell the server what notfications it has... just the sequence number?
315 2017-10-05T18:40:36  <jonasschnelli> (need to check that)
316 2017-10-05T18:40:43  <jonasschnelli> thanks
317 2017-10-05T18:40:43  <sipa> no, i'm saying there are no notification
318 2017-10-05T18:40:53  <sipa> there are just state changes
319 2017-10-05T18:41:23  <sipa> and the client can ask "i've synced up to block X" or "i've seen transactions up to timestamp Y", or "the last balance update i saw was Z"
320 2017-10-05T18:42:00  <sipa> because most data has inherent sequencing anyway already
321 2017-10-05T18:42:22  <jonasschnelli> could it also do long polling? ... because IMO that is what users want (not const. polling)
322 2017-10-05T18:42:41  <sipa> yes, but the long poll just returns "there is something you should look at", not what
323 2017-10-05T18:43:01  <jonasschnelli> ah. And then the state update call.
324 2017-10-05T18:43:05  *** promag has joined #bitcoin-core-dev
325 2017-10-05T18:43:08  <sipa> ... no
326 2017-10-05T18:43:28  <sipa> i mean *exactly* like listsinceblock
327 2017-10-05T18:43:41  <jonasschnelli> heh.. okay let me dive into there first
328 2017-10-05T18:43:50  <sipa> it has no sequence numbers, or notifications
329 2017-10-05T18:44:15  <sipa> it's a call where the client tells the server "hey, i've seen all transactions up to block X. what new transactions are there"
330 2017-10-05T18:44:38  <sipa> the next time the client calls, it uses the block hash at the time of its previous call
331 2017-10-05T18:44:54  <sipa> the server doesn't need to keep track of anything
332 2017-10-05T18:45:50  <jonasschnelli> would that also work with non block/tx data? Like new wallet txns (locally injected) or bandwith watermarks?
333 2017-10-05T18:46:08  <jonasschnelli> although not sure if there is a need for that
334 2017-10-05T18:46:10  <sipa> perhaps - maybe it's not possible for everything
335 2017-10-05T18:46:41  *** jcorgan has joined #bitcoin-core-dev
336 2017-10-05T18:46:45  <sipa> bandwidth watermarks are easy... the client doesn't care if he misses one
337 2017-10-05T18:47:06  *** AaronvanW has joined #bitcoin-core-dev
338 2017-10-05T18:47:29  <sipa> for wallet txn you can use the txid of the last seen tx
339 2017-10-05T18:47:37  *** promag has quit IRC
340 2017-10-05T18:48:20  <sipa> jonasschnelli: my point is that you probably need something like that anyway, at least for when a new client starts up or went offline for a while
341 2017-10-05T18:49:01  <jonasschnelli> sipa: Yes. Your idea make sense.
342 2017-10-05T18:49:12  <sipa> and when you do, why bother with a separate event queue - all events could just be "hey something happened, you should check what"
343 2017-10-05T18:49:37  <sipa> that's a big concern i had with ZMQ... as it passes the actual new tx and blocks, but without guarantees that it delivers all
344 2017-10-05T18:49:49  <sipa> it was fixed with sequence numbers
345 2017-10-05T18:49:50  <jonasschnelli> maybe you should be able to not get such notifications on new mempool txns
346 2017-10-05T18:50:10  <sipa> but it could also have been fixed by not having it at all, and making clients query for the data they didn't have after every notifications"
347 2017-10-05T18:50:57  <sipa> anyway, just an idea
348 2017-10-05T18:51:10  <jonasschnelli> Thanks for sharing... need to think about it a bit more
349 2017-10-05T18:51:14  <sipa> having events you can subscribe to individually that persist etc... is certainly more convenient for clients
350 2017-10-05T18:51:50  <sipa> but it makes bitcoin core responsible for tracking who knows what, instead of leaving that up to clients (who are arguably in a better position to know what they already know)
351 2017-10-05T18:52:08  <jonasschnelli> Indeed. And also the ressources.
352 2017-10-05T18:52:53  *** promag has joined #bitcoin-core-dev
353 2017-10-05T18:55:34  <sipa> it would be pretty nice if there was a way to subscribe to "hey, let me know when txid X gets Y confirmations or gets reorged", but perhaps that could be done in a python script that just uses a simpler interface with bitcoidn
354 2017-10-05T19:00:07  <achow101> meeting?
355 2017-10-05T19:00:39  *** meshcollider has joined #bitcoin-core-dev
356 2017-10-05T19:00:44  <instagibbs> yes
357 2017-10-05T19:01:12  <sipa> yes
358 2017-10-05T19:02:18  <jonasschnelli> #startmeeting
359 2017-10-05T19:02:18  <lightningbot> Meeting started Thu Oct  5 19:02:18 2017 UTC.  The chair is jonasschnelli. Information about MeetBot at http://wiki.debian.org/MeetBot.
360 2017-10-05T19:02:18  <lightningbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
361 2017-10-05T19:02:22  <wumpus> yes
362 2017-10-05T19:02:26  <jonasschnelli> Meeting: wumpus sipa gmaxwell jonasschnelli morcos luke-jr btcdrak sdaftuar jtimon cfields petertodd kanzure bluematt instagibbs phantomcircuit codeshark michagogo marcofalke paveljanik NicolasDorier
363 2017-10-05T19:02:32  <kanzure> hi.
364 2017-10-05T19:02:34  <jonasschnelli> wumpus: sry: though your where OL
365 2017-10-05T19:02:42  <cfields> hi
366 2017-10-05T19:02:43  <wumpus> #bitcoin-core-dev Meeting: wumpus sipa gmaxwell jonasschnelli morcos luke-jr btcdrak sdaftuar jtimon cfields petertodd kanzure bluematt instagibbs phantomcircuit codeshark michagogo marcofalke paveljanik NicolasDorier jl2012 achow101
367 2017-10-05T19:02:43  <meshcollider> Hello
368 2017-10-05T19:02:45  <achow101> hi
369 2017-10-05T19:02:49  <luke-jr> hi
370 2017-10-05T19:03:10  <wumpus> jonasschnelli: yes I was a bit late, sorry
371 2017-10-05T19:03:21  <wumpus> #topic high-priority for review
372 2017-10-05T19:03:43  <michagogo> Huh?
373 2017-10-05T19:03:43  <jonasschnelli> #chair wumpus
374 2017-10-05T19:03:43  <lightningbot> Current chairs: jonasschnelli wumpus
375 2017-10-05T19:03:44  <achow101> #10637 please?
376 2017-10-05T19:03:47  <gribble> https://github.com/bitcoin/bitcoin/issues/10637 | Coin Selection with Murchs algorithm by achow101 · Pull Request #10637 · bitcoin/bitcoin · GitHub
377 2017-10-05T19:04:15  <michagogo> Oh, right
378 2017-10-05T19:04:19  <michagogo> It's actually Thursday
379 2017-10-05T19:04:34  *** jb55 has quit IRC
380 2017-10-05T19:04:34  <wumpus> achow101: ok
381 2017-10-05T19:05:16  <wumpus> I also added #11389 today as it's blocking segwit wallet support
382 2017-10-05T19:05:16  * jtimon locks at #8498 and hides for the rest of the meeting
383 2017-10-05T19:05:17  <gribble> https://github.com/bitcoin/bitcoin/issues/11389 | Support having SegWit always active in regtest by sipa · Pull Request #11389 · bitcoin/bitcoin · GitHub
384 2017-10-05T19:05:20  <gribble> https://github.com/bitcoin/bitcoin/issues/8498 | Near-Bugfix: Optimization: Minimize the number of times it is checked that no money... by jtimon · Pull Request #8498 · bitcoin/bitcoin · GitHub
385 2017-10-05T19:05:47  <meshcollider> #11403 itself should be in there too probably?
386 2017-10-05T19:05:50  <gribble> https://github.com/bitcoin/bitcoin/issues/11403 | SegWit wallet support by sipa · Pull Request #11403 · bitcoin/bitcoin · GitHub
387 2017-10-05T19:06:07  <sipa> wumpus: i haven't had the time to work further on 11403, though concept review is certainly welcome
388 2017-10-05T19:06:16  <jnewbery> I've been reviewing 11389 this afternoon. It looks generally good, but breaks assumevalid.py, which I'm trying to fix now
389 2017-10-05T19:06:28  <jtimon> s/locks/looks/
390 2017-10-05T19:06:47  <BlueMatt> sipa: I think we need a document on the various possible approaches, tbh
391 2017-10-05T19:06:58  <sipa> BlueMatt: yes, i'll work on that soon
392 2017-10-05T19:07:00  <BlueMatt> there are a few and talking through all of them is going to need something more formal
393 2017-10-05T19:07:02  <BlueMatt> thanks
394 2017-10-05T19:07:21  <morcos> achow101: does 10637 implement all the coin selection logic we discussed in SF or only BnB?  Is there a high level description somewhere of what the PR is purporting to accomplish and what else will need to be done before 0.16?
395 2017-10-05T19:07:32  <achow101> morcos: only BnB
396 2017-10-05T19:07:48  <achow101> morcos: IIRC Murch is working on all of the coin selection stuff that we discussed
397 2017-10-05T19:07:52  <wumpus> btw I posted a proposed release schedule for 0.16.0 yesterday
398 2017-10-05T19:07:55  <wumpus> #link https://github.com/bitcoin/bitcoin/issues/11449
399 2017-10-05T19:08:28  <morcos> achow101: ok.. i've already forgotten what that is, so might be nice to have that written up in an issue or something so we remember the goal and can think about how this BnB implementation is going to fit into the big picture
400 2017-10-05T19:09:16  <achow101> morcos: the description of what 10637 does is in the first comment.
401 2017-10-05T19:09:35  <achow101> I can make an issue for coin selection changes in general
402 2017-10-05T19:09:42  <achow101> *to keep track of
403 2017-10-05T19:10:48  *** jtimon has quit IRC
404 2017-10-05T19:10:53  <wumpus> ok, I think that concludes high priority for review proposals
405 2017-10-05T19:10:57  <wumpus> any other topics?
406 2017-10-05T19:11:20  <achow101> topic suggestion: bad block interrogation/invalid block peer banning
407 2017-10-05T19:11:25  <wumpus> #action achow101 make an issue for coin selection changes in general
408 2017-10-05T19:11:38  <wumpus> #topic bad block interrogation/invalid block peer banning
409 2017-10-05T19:11:53  <achow101> relevant PR is #11446 (I did this in class so it kinda sucks)
410 2017-10-05T19:11:54  <gribble> https://github.com/bitcoin/bitcoin/issues/11446 | [WIP] Bad block interrogation by achow101 · Pull Request #11446 · bitcoin/bitcoin · GitHub
411 2017-10-05T19:12:06  <Murch> hey, sorry, was still in a meeting
412 2017-10-05T19:12:16  *** abpa has joined #bitcoin-core-dev
413 2017-10-05T19:12:34  <achow101> basically the idea is gmaxwell's. when we receive an invalid block, we want to make sure that all of our peers would also reject that block as invalid. If they don't ban them
414 2017-10-05T19:12:37  <Murch> I've been working on it, but since I do that in my free time in the evenings, it's been rather slow.
415 2017-10-05T19:12:53  <luke-jr> wumpus: this feels delayed?
416 2017-10-05T19:13:07  <gmaxwell> The general idea is that we aren't sufficiently agressive about punting peers on different consensus rules, so they can DOS attack us by sucking up slots, potentially hours per peer leaving us isolated... So there are number of things we can to do seek and destroy to speed up up.
417 2017-10-05T19:13:09  <wumpus> luke-jr: what feels delayed?
418 2017-10-05T19:13:14  <luke-jr> wumpus: 0.16
419 2017-10-05T19:13:17  <Murch> @achow101: If you want to collaborate on a write-up, I'd make myself available for that.
420 2017-10-05T19:13:20  <gmaxwell> luke-jr: release schedule is delayed because of 0.15.1
421 2017-10-05T19:13:23  <achow101> Murch: ok
422 2017-10-05T19:13:24  <luke-jr> i c
423 2017-10-05T19:13:31  <wumpus> luke-jr: yes, two months extra added, I mention that in the issue
424 2017-10-05T19:13:53  <achow101> what I wanted to discuss was the way to actually go about determining who to ban
425 2017-10-05T19:14:20  <Murch> @achow101: Gonna be traveling the next three weeks, so I might actually have more time. ;)
426 2017-10-05T19:14:22  <sipa> what is the issue with just looking at headers?
427 2017-10-05T19:14:23  <gmaxwell> achow101: I was kinda hoping we could implement something just from the messages we already get, it's my belief (could be wrong) that effectively we always learn the peers best header chain-- so we can begin kicking off peers based on that, as a first pass.
428 2017-10-05T19:14:34  <luke-jr> achow101: this contradicts the fixes in #10593
429 2017-10-05T19:14:36  <gribble> https://github.com/bitcoin/bitcoin/issues/10593 | Relax punishment for peers relaying invalid blocks and headers by luke-jr · Pull Request #10593 · bitcoin/bitcoin · GitHub
430 2017-10-05T19:14:58  <gmaxwell> achow101: I think we should be also drawing a distinction between inbound and outbound: the issue is what if we have a peer that accepts a broader set of blocks but would switch to our chain after learning of it.
431 2017-10-05T19:15:15  <achow101> gmaxwell: that's what I am not sure about. I don't think we necessarily know our peer's best header chain. suppose both us and them are fully synced, how do we know their best header chain until a new block appears?
432 2017-10-05T19:15:47  <wumpus> luke-jr: maybe two months is too much, but we'll see...
433 2017-10-05T19:16:03  <luke-jr> wumpus: nah, that sounds reasonable
434 2017-10-05T19:16:06  <sipa> achow101: when a new block appears, assuming it's PoW-valid to us, we'll learn about it through inv/headers/cb/...
435 2017-10-05T19:16:14  <jonasschnelli> Yes. +2 M seems okay to me
436 2017-10-05T19:16:36  <gmaxwell> sipa: but I believe he's right, we would have to wait for a new block, which is among the situations we're trying to resolve.
437 2017-10-05T19:17:07  <achow101> sipa: right, but I'm concerned about before a new block appears. we just connected to them or they just connected to us. we want to know then if we should ban them or not
438 2017-10-05T19:17:20  <luke-jr> IMO the desirable logic would be: for outbound connections, disconnect (don't ban) peers that aren't on the same chain; for inbound, tolerate it unless they reject a known-valid block
439 2017-10-05T19:17:22  <sdaftuar> we send getheaders messages on connect, typically
440 2017-10-05T19:17:25  <gmaxwell> For example say we are surrounded by ForkCoin peers, they are rejecting all bitcoin blocks.  There are few forkcoin miners so they only get blocks once per day.
441 2017-10-05T19:18:01  <gmaxwell> We don't want to wait for them to get a new block just to figure out our current batch of peers are already on a chain we reject.
442 2017-10-05T19:18:02  <achow101> sdaftuar: are you sure? all I could find is that we sometimes send getheaders, not all the time
443 2017-10-05T19:18:29  <sdaftuar> achow101: we send getheaders messages to all our peers at some point after startup, but they might ignore them
444 2017-10-05T19:18:34  <cfields> sdaftuar: not to incoming light clients, i think?
445 2017-10-05T19:18:35  <sdaftuar> eg if they are doing ibd themselves or something
446 2017-10-05T19:18:44  <sdaftuar> not to light clients, correct
447 2017-10-05T19:18:52  <sipa> light clients don't matter here
448 2017-10-05T19:18:53  <sdaftuar> but to inbound node_network ndoes we do
449 2017-10-05T19:19:07  <gmaxwell> If we _always_ sent getheaders and then kicked outbound peers whos chain has a block we've rejected, then I think that is the best we can do per that concern (still not a perfect fix, since you're isolated until forkcoin finds at least one block)
450 2017-10-05T19:19:18  <achow101> sdaftuar: if we are sending getheaders, if they are on a different chain, we still wouldn't necessarily know because our start block may not be on their best chain
451 2017-10-05T19:19:23  <gmaxwell> oh hm. then perhaps we already do where it matters.
452 2017-10-05T19:19:42  <sdaftuar> gmaxwell: the difficult part might be that you don't know the chain they're on is invalid
453 2017-10-05T19:19:47  <sdaftuar> if it's got less work than yours
454 2017-10-05T19:19:53  <RealM9> Topic suggestion: s2x
455 2017-10-05T19:20:04  <jonasschnelli> RealM9: no
456 2017-10-05T19:20:06  <luke-jr> sdaftuar: do you care?
457 2017-10-05T19:20:28  <luke-jr> if they're rejecting your better chain, you want to disconnect them anyway
458 2017-10-05T19:20:33  <gmaxwell> sdaftuar: seems like a seperate concern, we should also be kicking outbound peers that have less work than us, I think.
459 2017-10-05T19:20:38  <RealM9> Ok, but community is pretty interested. Are you going to change POW?
460 2017-10-05T19:20:43  <sdaftuar> gmaxwell: i think that would be a good idea, yeah
461 2017-10-05T19:20:43  <gmaxwell> But it would be silly to be overly agressive.
462 2017-10-05T19:20:48  <sipa> RealM9: us?
463 2017-10-05T19:20:58  <achow101> sdaftuar: gmaxwell what I propose is that we send a getheaders for our earliest known invalid block (within a certain time frame) and see if they respond with invalid blocks
464 2017-10-05T19:21:18  <luke-jr> RealM9: that's a decision for the community, not for developers. anyhow, ask on #bitcoin if you really want to discuss it
465 2017-10-05T19:21:34  <gmaxwell> achow101: I don't think we need to do that: for sync purposes any outbound peer we should be makign sure we learn their headers chain period (they may have a better chain than us and we should sync up ASAP)
466 2017-10-05T19:21:36  <sdaftuar> achow101: i'm not sure that's necessary?
467 2017-10-05T19:21:45  <morcos> RealM9: we're in a meeting , but please see: https://bitcoincore.org/en/2017/08/18/btc1-misleading-statements/
468 2017-10-05T19:21:49  <gmaxwell> achow101: if we're already doing that we'll notice known invalid block in their header chaip (well we will once we have code for that)
469 2017-10-05T19:21:52  <sdaftuar> i think if we do gmaxwell's suggestion of booting inbound peers who are on less work chains, then we'd be in good shape
470 2017-10-05T19:21:58  <sdaftuar> s/inbound/outbound/
471 2017-10-05T19:22:14  <luke-jr> I think we may actually want to track the headers of invalid chains..
472 2017-10-05T19:22:21  <achow101> what about inbound peers?
473 2017-10-05T19:22:31  <gmaxwell> For _inbound_ I think we should be setting a flag that they're consensus inconsistent which excludes them from the inbound peer management connection reservation.
474 2017-10-05T19:22:35  <sdaftuar> achow101: i think we should more aggerssively evict inbound peers if they appear to be on invalid chains
475 2017-10-05T19:22:43  <gmaxwell> so they'll get kicked off in favor of other inbound peers.
476 2017-10-05T19:22:52  <meshcollider> Agreed
477 2017-10-05T19:23:02  <gmaxwell> so we don't need to be agressive: they'll just get pushed out by other inbound peers.
478 2017-10-05T19:23:02  <luke-jr> consider: if an invalid chain has higher hashrate than the real chain, and then suddenly the invalid chain's hashrate drops off, without an equivalent increase on the main chain, we should consider that a possible attack and hold back on confirming transactions until it is resolved
479 2017-10-05T19:23:16  <sdaftuar> gmaxwell: yes i agree with you
480 2017-10-05T19:23:49  <achow101> my point is how do we know that an inbound peer is on an invalid chain?
481 2017-10-05T19:24:03  <gmaxwell> luke-jr: I think there is some need for smarter wallet confirmation logic but I think thats a seperate matter. (there was a paper 6-ish months ago that also points out the the reorg probablity math in the whitepaper is somewhat incomplete)
482 2017-10-05T19:24:19  <sdaftuar> achow101: set a flag if they relay an invalid block/blockheader
483 2017-10-05T19:24:32  <luke-jr> gmaxwell: right, but this is relevant because we can't assume "relays invalid headers" means the other node *accepts* the invalid block
484 2017-10-05T19:24:44  <gmaxwell> and we still interogate their headers if they're NODE_NETWORK/NODE_LIMITED
485 2017-10-05T19:24:46  *** RoyceX has joined #bitcoin-core-dev
486 2017-10-05T19:24:58  <luke-jr> sdaftuar: we intentionally relay blocks before checking validity now
487 2017-10-05T19:25:03  <achow101> sdaftuar: that requires them to have a block to relay to us, which could take hours or days
488 2017-10-05T19:25:18  <gmaxwell> luke-jr: the protocol does not have you realying a header of a block you haven't accepted. If you do that you risk dos attacking peers already.
489 2017-10-05T19:25:19  <sdaftuar> achow101: i don't think we need to worry as much about inbound peers
490 2017-10-05T19:25:34  <sdaftuar> achow101: for instance an attacker can already try to use all your inbound slots and not send you anything
491 2017-10-05T19:25:40  <gmaxwell> luke-jr: the only place that happens in the protocol is HB BIP152 messages.
492 2017-10-05T19:25:51  <achow101> sdaftuar: right, ok
493 2017-10-05T19:25:58  <luke-jr> gmaxwell: which may be all you see from CB peers
494 2017-10-05T19:26:18  <gmaxwell> sdaftuar: yes, for inbound we can just deprive them of reservations.
495 2017-10-05T19:26:49  <sdaftuar> luke-jr: even with bip152 the headers need to be valid
496 2017-10-05T19:27:04  <luke-jr> sdaftuar: yes, the header itself; but it can be a valid header for an invalid block
497 2017-10-05T19:27:16  <gmaxwell> yes, though we'd catch it on the _next_ block.
498 2017-10-05T19:27:19  <sdaftuar> luke-jr: if it builds on an invalid chain, i believe the header would be invalid
499 2017-10-05T19:27:21  *** laurentmt has quit IRC
500 2017-10-05T19:27:26  <achow101> so when we connect to an outbound peer, we will send them a getheaders so we know their best headers chain and ban accordingly
501 2017-10-05T19:27:29  <luke-jr> (note I tried to keep track of peer bestblocks in #10512 and basically gave up)
502 2017-10-05T19:27:31  <gribble> https://github.com/bitcoin/bitcoin/issues/10512 | Rework same-chain from abusing DoS banning, to explicit checks by luke-jr · Pull Request #10512 · bitcoin/bitcoin · GitHub
503 2017-10-05T19:27:41  <gmaxwell> when they relay a CB message for a child of an invalid block.
504 2017-10-05T19:27:51  *** Cheeseo has quit IRC
505 2017-10-05T19:28:04  <gmaxwell> achow101: yes, but based on the above I believe we already always send it.
506 2017-10-05T19:28:11  <achow101> the other part of 11446 is to ban other peers for relaying us an invalid block for which we already know is invalid
507 2017-10-05T19:28:20  <gmaxwell> achow101: because we send it to nodenetwork peers and outbound always are (or or disconnected)
508 2017-10-05T19:28:20  <achow101> but I'm not sure how that interacts with compact blocks
509 2017-10-05T19:28:33  <gmaxwell> achow101: FWIW, I think we should probably be just disconnecting and not banning.
510 2017-10-05T19:28:37  <sdaftuar> achow101: oh that interaction might be tricky
511 2017-10-05T19:28:54  <achow101> gmaxwell: why not ban?
512 2017-10-05T19:29:07  <gmaxwell> I think the interaction isn't too bad, for this purpose a BIP152 CB HB block is relaying you the header of its parent.
513 2017-10-05T19:29:20  *** chjj has joined #bitcoin-core-dev
514 2017-10-05T19:29:22  <luke-jr> achow101: in a softfork, old nodes will send invalid blocks
515 2017-10-05T19:29:27  <luke-jr> potentially
516 2017-10-05T19:29:42  <gmaxwell> achow101: because it's hardly any better and it means that when some dimbulb tries running forkcoin it results in him being unable to run Bitcoin (perhaps concurrently) on the same host.
517 2017-10-05T19:30:03  <achow101> gmaxwell: ok
518 2017-10-05T19:30:04  <gmaxwell> it also blocks inbound from that peer, which we'd be find allowing.
519 2017-10-05T19:30:09  <gmaxwell> s/find/fine/
520 2017-10-05T19:30:34  <gmaxwell> In general we should be moving away from bans except when the thing we banned for was expensive for us.
521 2017-10-05T19:31:03  <achow101> so 11446 can really just be reduced to an ~1 line change to disconnect on a header for a block we already know is invalid
522 2017-10-05T19:31:11  <BlueMatt> yea, that
523 2017-10-05T19:31:18  <sdaftuar> achow101: agree, though we have to be careful about compact blocks i think
524 2017-10-05T19:31:20  <luke-jr> achow101: aka #10593 …
525 2017-10-05T19:31:22  <gribble> https://github.com/bitcoin/bitcoin/issues/10593 | Relax punishment for peers relaying invalid blocks and headers by luke-jr · Pull Request #10593 · bitcoin/bitcoin · GitHub
526 2017-10-05T19:31:48  <gmaxwell> achow101: yes, but for compact block interactions (HB mode will relay us blocks that are invalid).
527 2017-10-05T19:32:27  <achow101> gmaxwell: so we would have to check the specific type of invalidness and whether it was a CB?
528 2017-10-05T19:32:31  <gmaxwell> luke-jr: IIRC when you proposed that before you got squaked at that it would undermine our protection against isolation...
529 2017-10-05T19:33:04  <gmaxwell> achow101: or just don't do it for the peers maked for HB CBs for now
530 2017-10-05T19:33:17  <achow101> gmaxwell: isn't that likely to be most peers though
531 2017-10-05T19:33:19  <achow101> ?
532 2017-10-05T19:33:21  *** SopaXorzTaker has quit IRC
533 2017-10-05T19:33:23  <gmaxwell> No, it's at most three.
534 2017-10-05T19:33:33  <luke-jr> gmaxwell: I don't see how. It's literally what achow101 was just describing.
535 2017-10-05T19:34:32  <luke-jr> gmaxwell: maybe you're thinking of the predecessor 10512?
536 2017-10-05T19:34:35  <achow101> luke-jr: it doesn't look like you are handling invalid duplicates?
537 2017-10-05T19:34:40  <gmaxwell> probably.
538 2017-10-05T19:34:57  *** promag has quit IRC
539 2017-10-05T19:34:58  <gmaxwell> achow101: in any case, we can pick this up on a PR and later discussion.
540 2017-10-05T19:35:09  <achow101> gmaxwell: ok
541 2017-10-05T19:36:01  *** wraithm has quit IRC
542 2017-10-05T19:36:08  <achow101> next topic then?
543 2017-10-05T19:36:09  <wumpus> any other topics?
544 2017-10-05T19:36:10  <luke-jr> achow101: IIRC it does, we can go over it later if you like
545 2017-10-05T19:36:14  <achow101> luke-jr: ok
546 2017-10-05T19:37:09  <jnewbery> luke-jr: any progress on multiwallet GUI without the rpcauth parts?
547 2017-10-05T19:37:12  *** promag has joined #bitcoin-core-dev
548 2017-10-05T19:37:21  <wumpus> #topic multiwallet ui
549 2017-10-05T19:37:32  <luke-jr> jnewbery: not yet, I'll plan to push the update later today
550 2017-10-05T19:38:04  <jnewbery> great! I had a look myself, and I think it's just a one-line change to the debug console commit
551 2017-10-05T19:38:05  <jonasschnelli> https://github.com/bitcoin/bitcoin/pull/11383
552 2017-10-05T19:38:12  <sipa> topic suggestion: dealing with platform-specific code
553 2017-10-05T19:38:24  <jonasschnelli> luke-jr: I can continue to work on 11383 if you want?
554 2017-10-05T19:38:32  <jonasschnelli> (remove the auth stuff :P)
555 2017-10-05T19:38:50  <luke-jr> jnewbery: certainly not that simple.. still need to resolve wallet name to CWallet earlier
556 2017-10-05T19:39:13  <jnewbery> ok, well I've got a branch that works with just that change. Happy to share with you
557 2017-10-05T19:39:47  <gmaxwell> Sounds good.
558 2017-10-05T19:40:00  <luke-jr> jnewbery: push it and I'll take a look
559 2017-10-05T19:40:17  <jnewbery> thanks
560 2017-10-05T19:40:48  *** AaronvanW has quit IRC
561 2017-10-05T19:41:03  <wumpus> #topic dealing with platform-specific code (sipa)
562 2017-10-05T19:41:19  <sipa> i've recently been looking into faster parallel hashing code
563 2017-10-05T19:41:24  *** AaronvanW has joined #bitcoin-core-dev
564 2017-10-05T19:41:37  <wumpus> hashing as in sha256?
565 2017-10-05T19:41:50  <sipa> in particular, for 8-way parallel SHA256 (which would be useful in merkle root computation and block deserialization), a 5x speedup is doable with AVX2
566 2017-10-05T19:42:04  <sipa> and maybe 2.5x with SSE2
567 2017-10-05T19:42:22  <wumpus> and parallel in this case means computing multiple hashes of multiple pieces of data at once?
568 2017-10-05T19:42:27  <sipa> correct
569 2017-10-05T19:42:28  <luke-jr> how much speedup is this for the entire IBD?
570 2017-10-05T19:42:57  <gmaxwell> luke-jr: It saves something like 10 minutes on IBD.  But the greater impact is in block relay.
571 2017-10-05T19:43:01  <wumpus> (I guess there are constraints there, do all the inputs need to be the same size?)
572 2017-10-05T19:43:05  <luke-jr> I imagine merkle root is a tiny fraction of the overall process, but otoh it's also possibly a blocker on parallelization
573 2017-10-05T19:43:05  <gmaxwell> Where hash tree computation is most of the time.
574 2017-10-05T19:43:25  <luke-jr> gmaxwell: it is? :O
575 2017-10-05T19:43:32  <sipa> wumpus: yes and no; for now, it's just a primitive that you give a pointer to N 64-byte inputs, and produces 32-byte outputs
576 2017-10-05T19:43:33  <luke-jr> oh, because the signature checks are cached?
577 2017-10-05T19:43:37  <BlueMatt> in terms of compact block relay, merkle root calculation and deserialize are about the only big timesinks before you can relay
578 2017-10-05T19:43:46  <sipa> wumpus: which is specific for merkle root computation
579 2017-10-05T19:43:48  <gmaxwell> luke-jr: yes for HB BIP152 we don't to validation except hashing!
580 2017-10-05T19:43:50  <sipa> but it can certainly be adapted
581 2017-10-05T19:44:37  <wumpus> sipa: ok
582 2017-10-05T19:44:38  <cfields> sipa: i had a scare when reviewing some new leveldb crc32 changes that i thought (at first glance) could be a consensus issue. I was very angry at myself at that point for not adding a fallback un-optimized verification of the optimized path.
583 2017-10-05T19:44:43  <sipa> anyway, there are multiple ways to integrate this: separate asm code, inline asm blocks, or code using intrinsics (my preference, it's much more easy to review, and has no OS-specific warts like the L label prefix...)
584 2017-10-05T19:44:56  <gmaxwell> sipa has actually implemented the 8-way AVX2 sha2 and a hash tree computation that uses it... along with specialized implementation of 64-byte input double sha2.. which affords an addition 20%-ish speedup over generic sha2.
585 2017-10-05T19:45:09  <cfields> very cool :)
586 2017-10-05T19:45:12  <wumpus> I prefer intrinsics
587 2017-10-05T19:45:21  <wumpus> (except for arm32 whre they suck)
588 2017-10-05T19:45:48  <sipa> so, for intrinsics... do we want to have a separate LIBCRYPTO_AVX2 LIBCRYPTO_SSE2 LIBCRYPTO_... with different compile flags each?
589 2017-10-05T19:46:05  <sipa> or could we rely on __attribute__((target("avx2")))
590 2017-10-05T19:46:05  <gmaxwell> Historically, For some code you cannot achieve equivilent performance w/ intrinsics because you must manage register allocation precisely for things to work, but that isn't the case here....
591 2017-10-05T19:46:08  <wumpus> but 64 bit platforms the SIMD instructions have been specially tweaked to work well with compilers and intrinsics
592 2017-10-05T19:46:12  <sipa> (which works on both clang and gcc)
593 2017-10-05T19:46:23  *** EricCartman has quit IRC
594 2017-10-05T19:46:36  <cfields> sipa: i think we should test for the target attribute and use it if possible, but not completely rely on it
595 2017-10-05T19:46:44  <cfields> iirc that improves dispatching time as well?
596 2017-10-05T19:46:49  <sipa> cfields: no
597 2017-10-05T19:46:57  <wumpus> different compile flags for different compile units, that's the only portable way
598 2017-10-05T19:47:01  <luke-jr> sipa: I think we can't assume intrinsics for all platforms, so we want the separate lib route
599 2017-10-05T19:47:07  <gmaxwell> dispatching is via a function pointer ultimately in all those cases.
600 2017-10-05T19:47:14  <wumpus> luke-jr: you're confusing, that's not about intrinsics
601 2017-10-05T19:47:30  <sipa> the only difference is avoiding the need for build system complication
602 2017-10-05T19:47:35  <wumpus> intrinsics inthis case are headers like xmmintr.h which provides functions that work on vector types
603 2017-10-05T19:47:42  <sipa> exactly
604 2017-10-05T19:47:52  <luke-jr> wumpus: __attribute__((target("avx2"))) isn't an option for separate asm code, though, right?
605 2017-10-05T19:48:00  <sipa> luke-jr: it also isn't needed for asm code
606 2017-10-05T19:48:02  <cfields> gmaxwell: isn't there elf data that allows them to be setup at load time?
607 2017-10-05T19:48:14  <wumpus> oh no no ELF magic please
608 2017-10-05T19:48:24  <luke-jr> hmm
609 2017-10-05T19:48:36  <cfields> not by hand, i thought __attribute__(target) did that behind the scenes
610 2017-10-05T19:48:41  <sipa> target("avx2") just means "this function is compiled as if -mavx2 was passed on the command line
611 2017-10-05T19:49:02  <sipa> cfields: GCC also has target("default"), where you can have multiple versions of the same function... which causes automatic dispatch to be added
612 2017-10-05T19:49:08  <sipa> that's non-portable and has other issues
613 2017-10-05T19:49:09  <gmaxwell> cfields: they're setup at load time, yes-- but they're still just a function pointer, which we could also have setup at load time.  Though it is nice that the function override trick can make them run before main.
614 2017-10-05T19:49:11  <wumpus> I'm normally not scared of low level ELF hacking, but for bitcoin, let's try to keep it safe and portable
615 2017-10-05T19:49:12  <luke-jr> sipa: how does it behave if I have an explicit -mno-avx2?
616 2017-10-05T19:49:15  <sipa> (in particular clang doesn't have that)
617 2017-10-05T19:49:17  <cfields> sipa: ah yes, that's what i was thinking of.
618 2017-10-05T19:49:33  <sipa> cfields: so i'm not suggesting using that
619 2017-10-05T19:49:39  <sipa> luke-jr: i assume it overrides it
620 2017-10-05T19:49:43  <cfields> ok
621 2017-10-05T19:50:02  <luke-jr> I suppose I can test it
622 2017-10-05T19:50:05  <wumpus> yes, gcc can do it automatically on some platforms, but I'm afraid the only portable way is to make our own dispatch logic
623 2017-10-05T19:50:15  <sipa> yes, we'll want our own dispatch logic anyway
624 2017-10-05T19:50:18  <wumpus> we already have some CPUID bits checking
625 2017-10-05T19:50:20  <sipa> so we can test things
626 2017-10-05T19:50:21  <wumpus> so it's nothing new erally
627 2017-10-05T19:50:23  *** Chris_Stewart_5 has quit IRC
628 2017-10-05T19:50:25  <sipa> and report which version is being chosen
629 2017-10-05T19:50:29  <cfields> np, i wasn't suggesting. just trying to understand the advantages of one vs the other.
630 2017-10-05T19:50:30  <wumpus> yes, exactly
631 2017-10-05T19:51:01  <sipa> but if possible i'd like to avoid the overhead of needing half a dozen libcrypto_XXX.a things that need to be linked in everywhere
632 2017-10-05T19:51:08  <sipa> though that's really the only advantage
633 2017-10-05T19:51:38  <wumpus> so I'd say: yes, use intrinsics instead of inline/offline asm,  and use our own dispatching, and compile units compiled with appropriate compiler flags
634 2017-10-05T19:51:55  <sipa> okay.
635 2017-10-05T19:52:28  <gmaxwell> can we say prefer intrinsics instead of use? :) I don't think we'd eschew inline asm if we thought it was better in a particular case.
636 2017-10-05T19:52:29  <wumpus> yes regarding build system it's just verbose, not really complex
637 2017-10-05T19:52:41  <cfields> sipa: see my point above about a fallback, though. In the case of mismatch hashes, i think it's worthwhile to re-check with a generic implementation before deciding it's failed.
638 2017-10-05T19:53:12  <gmaxwell> cfields: we should be testing these things in startup tests.
639 2017-10-05T19:53:12  <luke-jr> (yes, it seems to override -mno-*
640 2017-10-05T19:53:17  <wumpus> gmaxwell: well if there's a case you can do much better than the compiler, sure...
641 2017-10-05T19:53:25  *** promag has quit IRC
642 2017-10-05T19:53:37  * BlueMatt has a weak preference for compile units, but only cause I'd use them in FIBRE for my FEC stuff, too, but thats not much of a reasoning
643 2017-10-05T19:54:26  * luke-jr hopes we can have POWER9 asm in 0.16 <.<
644 2017-10-05T19:54:32  * BlueMatt agrees
645 2017-10-05T19:54:38  <cfields> gmaxwell: an implementation bug in some branch of one optimized path is scary...
646 2017-10-05T19:54:56  <gmaxwell> cfields: try differently if it fails is just not reasonable in a lot of cases; and often would add a lot of complexity (now you have to not cache hashes, but instead only use hash-verify methods) ... and we don't have expected values for thigns like single transaction hashes, just hash roots.
647 2017-10-05T19:54:57  <cfields> gmaxwell: in particular, the crc issue had to do with incoming data alignment on x86_64
648 2017-10-05T19:55:24  <wumpus> cfields: I agree
649 2017-10-05T19:55:51  <wumpus> cfields: I think we should only do asm optimization in cases where it really makes a lot of difference, for that reason, ther risk has to be worth it
650 2017-10-05T19:56:01  <gmaxwell> cfields: yes, thats something that always needs careful review and we should have unit tests that also stress alignment.
651 2017-10-05T19:56:33  <wumpus> special-casing everything makes things a lot harder to review, and test, especially when it starts to need different kinds of hardware
652 2017-10-05T19:56:54  <wumpus> but for testable low-level primitives like SHA256 I'd say it's ok
653 2017-10-05T19:57:11  <gmaxwell> good thing no one is talking about special casing everything. :)
654 2017-10-05T19:57:29  <gmaxwell> yea, sha2 etc have simple testable interfaces.
655 2017-10-05T19:57:37  <wumpus> no, that's just one extreme, I've seen soome graphics drivers which are scary in that regard :)
656 2017-10-05T19:58:09  <gmaxwell> But benchmarks!
657 2017-10-05T19:58:11  <wumpus> oh let's special case 4x4 tiles, 4x5 tiles, 4x6 tiles, ... for 3 different architectures
658 2017-10-05T19:58:20  <wumpus> right :)
659 2017-10-05T19:58:27  <cfields> mmm. I don't see the harm in doing a quick re-check in a few certain cases (merkle mismatch is a good example)
660 2017-10-05T19:58:51  <wumpus> special-casing benchmarks is a curious form of over-learning
661 2017-10-05T19:58:55  <cfields> anyway, i've made my case
662 2017-10-05T19:59:01  <gmaxwell> cfields: because it requires restructing the code to not return hashes but instead only have uncachable hash_Verify methods.
663 2017-10-05T19:59:06  <wumpus> cfields: re-check in what case?
664 2017-10-05T19:59:24  <luke-jr> although someone did manage to screw up xpub serialisation at one point IIRC
665 2017-10-05T19:59:25  <wumpus> cfields: you mean re-run the validation w/ different implementations if an  incoming block fails?
666 2017-10-05T19:59:55  <wumpus> (what about false positives?)
667 2017-10-05T20:00:00  <cfields> wumpus: that's a big hammer, but yes-ish
668 2017-10-05T20:00:11  <gmaxwell> cfields: and for small functions like a hash a check in an innerloop will measurably lower performance. ... and you also create the opposite problem, what if the alternative function is the wrong one?
669 2017-10-05T20:00:26  <gmaxwell> (I'd actually consider whole block level more reasonable)
670 2017-10-05T20:00:29  <wumpus> gmaxwell: I think he means on a high level
671 2017-10-05T20:01:02  <wumpus> on the inner level it's just NASA-level crazy, let's run three implementations and see which ones agree
672 2017-10-05T20:01:06  <sipa> i think re-checking a block if it fails is reasonable... but why switch hash functions? it's massively more likely your CPU is fried than that the hash function implementation is wrong all along and you never noticed
673 2017-10-05T20:01:13  <gmaxwell> but then the dispatch is mutable not just set once at init. :(
674 2017-10-05T20:01:28  <wumpus> yeah ... I think we're overdesigning this
675 2017-10-05T20:01:30  <gmaxwell> right we have a constant slow stream of complaints from users whos hosts have falsely rejected the blockchain.
676 2017-10-05T20:01:37  <wumpus> just continue with what you were doing sipa :)
677 2017-10-05T20:01:40  <cfields> gmaxwell: just have a generic non-dispatchable one
678 2017-10-05T20:01:41  <gmaxwell> I would like to see that improved somehow.
679 2017-10-05T20:01:43  <wumpus> any other topics?
680 2017-10-05T20:01:46  <wumpus> oh wait, it's time
681 2017-10-05T20:01:57  <wumpus> #endmeeting
682 2017-10-05T20:01:57  <lightningbot> Meeting ended Thu Oct  5 20:01:57 2017 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
683 2017-10-05T20:01:57  <lightningbot> Minutes:        http://www.erisian.com.au/meetbot/bitcoin-core-dev/2017/bitcoin-core-dev.2017-10-05-19.02.html
684 2017-10-05T20:01:57  <lightningbot> Minutes (text): http://www.erisian.com.au/meetbot/bitcoin-core-dev/2017/bitcoin-core-dev.2017-10-05-19.02.txt
685 2017-10-05T20:01:57  <lightningbot> Log:            http://www.erisian.com.au/meetbot/bitcoin-core-dev/2017/bitcoin-core-dev.2017-10-05-19.02.log.html
686 2017-10-05T20:02:02  <sipa> #lunch
687 2017-10-05T20:02:51  <meshcollider> And for anyone who doesn't know yet, hacktoberfest = free t-shirt for 4 PRs in october
688 2017-10-05T20:02:56  <meshcollider> https://hacktoberfest.digitalocean.com
689 2017-10-05T20:03:12  <sipa> if someone is interested: https://github.com/sipa/bitcoin/commits/201709_dsha256_64 <- 64-byte specialzed SHA256 and AVX2 code
690 2017-10-05T20:03:14  <gmaxwell> sneaky way to learn developers mailing addresses.
691 2017-10-05T20:03:20  <sipa> (way too WIP to PR)
692 2017-10-05T20:03:28  <achow101> ooh free t-shirts
693 2017-10-05T20:03:48  * sipa found the student
694 2017-10-05T20:03:52  <meshcollider> gmaxwell: Heh true
695 2017-10-05T20:03:54  <cfields> sipa: very cool. didn't mean to rain on your parade.
696 2017-10-05T20:03:56  <luke-jr> gmaxwell: never got any spam that I can tell came from last year's
697 2017-10-05T20:04:08  <luke-jr> just don't use your home address for mailing address ;)
698 2017-10-05T20:04:23  <wumpus> sipa: nice
699 2017-10-05T20:04:55  *** jb55 has joined #bitcoin-core-dev
700 2017-10-05T20:05:08  <gmaxwell> cfields: so imagine that sha2 implementation isn't alignment safe. you get a block and miscompute one of the hashes due to alignment. ...  I don't see any way of efficiently accomplishing your 'try another function' approach that would stop the false rejection.
701 2017-10-05T20:05:56  <sipa> wumpus: too bad the 64-byte specialized generic-x86 code is slower than the generic-data-size SSE4-specialized version
702 2017-10-05T20:06:07  <sipa> wumpus: otherwise i'd straight up PR the 64-byte specialized code
703 2017-10-05T20:06:15  <wumpus> the avx code looks very recognizable
704 2017-10-05T20:06:26  <sipa> wumpus: it's pretty much search replace on SSE4 code
705 2017-10-05T20:06:35  <wumpus> heh
706 2017-10-05T20:06:47  <gmaxwell> it's only detectable at the hash root check at the end a long computation pipeline... when that fails do we just go back and re-deseralize the entire block with different code to compute new hashes in the CTransactions?
707 2017-10-05T20:07:43  <gmaxwell> (and if we do, we magnify a DOS vector for someone sending us invalid blocks, though perhaps not enough to worry about)
708 2017-10-05T20:07:58  <wumpus> that's another advantage of intrinsics, it's usually easier to review than straight up asm
709 2017-10-05T20:08:06  <sipa> wumpus: absolutely
710 2017-10-05T20:08:16  <wumpus> no need to keep track in your head where all the registers go
711 2017-10-05T20:08:30  <cfields> gmaxwell: addmittedly that's ugly, but yes, i think that's worth considering
712 2017-10-05T20:08:36  <sipa> which may be a good thing or a bad thing
713 2017-10-05T20:08:47  <sipa> 1) no need to keep in your head where the registers go!
714 2017-10-05T20:08:57  <sipa> 2) no way to tell the compiler to keep a certainly value in a register!
715 2017-10-05T20:09:39  <wumpus> well combined with the pipelining/interleaving that optimized code needs, and the large number of registers that SIMD architectures have, that can be quite difficult
716 2017-10-05T20:09:40  <gmaxwell> I wish you could assign variables to registers, and have the compiler yell at you if you tried to assign two live variables to the same register.
717 2017-10-05T20:10:39  <wumpus> usually there's so many registers that it would be good to be able to tell that something is *not* worth storing in a register
718 2017-10-05T20:10:49  *** goatpig has joined #bitcoin-core-dev
719 2017-10-05T20:10:53  <sipa> i believe there is actually a way (in GCC) to force a particular variable into a particular register
720 2017-10-05T20:11:04  <sipa> int x asm("%edx");
721 2017-10-05T20:11:25  <gmaxwell> cfields: and still doesn't help with accepting something we shouldn't, which is usually a more serious issue.
722 2017-10-05T20:12:32  <gmaxwell> cfields: if we had some generic infrastructure to retry a failed validation (e.g. to cope with lossy hardware) then perhaps what you're thinking about could be dropped into it.
723 2017-10-05T20:12:47  <cfields> gmaxwell: i'm not saying it's something we have to do, or something that wouldn't be ugly. I'm moreso coming from a place where I was in full-out panic for a few hours because I thought newer x86_64 machines were about to start diverging...
724 2017-10-05T20:12:53  <sipa> https://gcc.gnu.org/onlinedocs/gcc/Local-Register-Variables.html#Local-Register-Variables
725 2017-10-05T20:13:11  <cfields> so you're right, (good!) testing should negate those worries.
726 2017-10-05T20:13:37  <gmaxwell> Well, in particular runtime tests... since they'll catch failures on the actual hardware the user has.
727 2017-10-05T20:13:55  <cfields> gmaxwell: for reference: https://github.com/google/leveldb/commit/2964b803b857932ff7499d7bebb61dc5514dab7c
728 2017-10-05T20:13:58  <cfields> yes
729 2017-10-05T20:14:19  *** promag has joined #bitcoin-core-dev
730 2017-10-05T20:15:01  *** Emcy_ has quit IRC
731 2017-10-05T20:15:17  *** Emcy_ has joined #bitcoin-core-dev
732 2017-10-05T20:16:00  *** promag has quit IRC
733 2017-10-05T20:17:41  *** wumpus has quit IRC
734 2017-10-05T20:19:22  <morcos> I also don't want to rain on anyone's parade, and this is OSS so people work on what they find interesting, but I think we shoudl be careful to think about what are priorities for the project
735 2017-10-05T20:19:53  <morcos> More importantly however, we shoudl be careful about making changes that are not easily reviewable by more than 1 or 2 people unless they are really warranted
736 2017-10-05T20:20:59  <morcos> I've been thinking more about this since bech32.  I think bech32 is completely awesome, I'm super happy we're doing it and imo its a good priority project. But it essentially got no review.  Did anyone other than sdaftuar review it?
737 2017-10-05T20:21:45  <morcos> Sometimes things will have to be like that, but it shoudl be a tradeoff we consider carefully...  how tricky are we trying to be vs how much is it warranted
738 2017-10-05T20:22:03  * BlueMatt tends to agree, though noting that a part of my agreement is my different priorities from some others - performance is maybe much more of a concern for many others in the project more than I
739 2017-10-05T20:22:14  *** wumpus has joined #bitcoin-core-dev
740 2017-10-05T20:22:14  <morcos> I haven't evaluated that in the context of the parallelized hashing, but its something I think we should
741 2017-10-05T20:25:12  <gmaxwell> morcos: bech32 had more review then it appeared because we solicited extensive review before publishing the bip.. what didn't get review was the checksum itself other than myself and pieter, until sdaftuar.  ... but who reviewed the prior address checksum?
742 2017-10-05T20:25:30  <gmaxwell> Clearly no one, because it needlessly sucked. :P
743 2017-10-05T20:25:55  <sipa> well it was too late to review it by the time any of us were around
744 2017-10-05T20:26:27  <luke-jr> I didn't review Bech32 because I figured it was over my head (especially magic checksum stuff).
745 2017-10-05T20:26:41  <gmaxwell> There were many design changes to the earliest Bech32 proposal that arose out of review, e.g. the delimiter character.
746 2017-10-05T20:27:27  <morcos> Yes and of course the 2 people that can't be blamed are you and pieter since you did the work.  But I don't want us to fall into a trap of just assuming if something is too difficult or outside of our field to review properly that someone else must be doign a good job with it
747 2017-10-05T20:28:04  <morcos> gmaxwell: and yes i was referring to the checksum
748 2017-10-05T20:28:15  <morcos> but thats a good point...
749 2017-10-05T20:29:38  <gmaxwell> We deal with this for libsecp256k1 too, that fact that it's in a different repo is just enabling you to ignore it. :)
750 2017-10-05T20:29:54  <gmaxwell> Though we do have effective review there too.
751 2017-10-05T20:30:13  <gmaxwell> Though not as much as I'd like.
752 2017-10-05T20:30:41  <sipa> but at least for secp256k1 it's clear what the goal is (implement secp256k1 EC correctly), so someone could review e.g. just the test and judge that they're sufficient for that goal
753 2017-10-05T20:30:54  <cfields> morcos: yes, well said. I think that's why I find the asm changes scary. I spent a full day trying to learn and understand the sse42 optimized sha2, and only because it was failing some tests. At best, I agree that it looks right, but I could never say it with any degree of certainty. I deferred to "it passes all the tests".
754 2017-10-05T20:31:10  <sipa> with bech32, the effort is in the design, not the implementation, and there is relatively little proof that the design actually has the properties it claims to have
755 2017-10-05T20:31:28  <gmaxwell> But thats the same thing as reviewing the bech32 checksum. and fwiw, if feedback on bech32 we got was we needed more reviewers for the checksum, I would have gone and asked a libsecp256k1 contributor to do it.
756 2017-10-05T20:32:07  <gmaxwell> because I know e.g. andrew (or peter dettman) aren't frightened off by math.
757 2017-10-05T20:32:14  <morcos> I think my feedback is one meta level up.  There should have been more people that questioned how much reivew it got
758 2017-10-05T20:32:21  <gmaxwell> yes, agreed.
759 2017-10-05T20:32:38  <gmaxwell> well I was thinking that before you commented. Started thinking it as soon as I saw other bitcoin software had merged it.
760 2017-10-05T20:32:51  <morcos> so i know i'm not going to review the hashing stuff, just want to make sure other peopel are going to ensure it is properly reviewed
761 2017-10-05T20:33:13  <morcos> i'm frightened off by code. :)
762 2017-10-05T20:34:19  <gmaxwell> had someone commented on that earlier, I would have agreed.  I reviewed the checksum work too, but pieter and I worked a lot togeather on it, and if he screwed up he probably would have tained me.
763 2017-10-05T20:34:24  <sipa> the cool thing about hash functions is that they have essentially no branches... so it's very hard (though not impossible, see the alignment issue) to make it be incorrect only for a hard-to-detect small subset of inputs
764 2017-10-05T20:35:19  <gmaxwell> yes on hash function correctness, but that doesn't help with BECH32 design, which I guess as your point above.  It's easy to be confident a new implementation of it is conformant... not so much that the design is good.
765 2017-10-05T20:35:50  <sipa> right
766 2017-10-05T20:36:24  <gmaxwell> and still no one has basically reviewed our decision to use a cyclic code -- perhaps if we were call coding theorists someone would have known an even better tool... but there is a limit to how far down a rabbit hole we can go.
767 2017-10-05T20:36:31  <morcos> gmaxwell: but part of my point was the tradeoff on how important the feature is.  i am definitely +1 on bech32.  and not negative on parallel hashing just raising points for us to think about
768 2017-10-05T20:36:35  <morcos> anyway, got to run
769 2017-10-05T20:52:29  *** promag has joined #bitcoin-core-dev
770 2017-10-05T21:04:58  *** promag has quit IRC
771 2017-10-05T21:07:13  *** ybit has quit IRC
772 2017-10-05T21:07:50  *** ybit has joined #bitcoin-core-dev
773 2017-10-05T21:26:11  *** wraithm has joined #bitcoin-core-dev
774 2017-10-05T21:26:21  *** jtimon has joined #bitcoin-core-dev
775 2017-10-05T21:29:15  *** RoyceX has quit IRC
776 2017-10-05T21:33:37  <BlueMatt> someone wanna close 11454?
777 2017-10-05T21:52:53  *** Cheeseo has joined #bitcoin-core-dev
778 2017-10-05T21:56:03  <gmaxwell> morcos: I think you can decompose your concerns into two folks-- people will work on whatever they like, but if it's going to get merged it needs to deserve the required review attention, since that isn't just an indivigual decision.
779 2017-10-05T21:56:25  <gmaxwell> forks*
780 2017-10-05T21:56:59  <gmaxwell> morcos: and then unrelated, we shouldn't be in a state where people don't review or even provide meta review of things which are two mathy or too low level and so they assume that they won't be of use.
781 2017-10-05T21:57:15  *** Chris_Stewart_5 has joined #bitcoin-core-dev
782 2017-10-05T21:57:32  <gmaxwell> And the thing to encourage there is that even if its over your head you can still ask some of the right meta questions.
783 2017-10-05T22:12:57  *** Cheeseo has quit IRC
784 2017-10-05T22:13:22  *** Cheeseo has joined #bitcoin-core-dev
785 2017-10-05T22:14:36  *** promag has joined #bitcoin-core-dev
786 2017-10-05T22:19:01  *** wraithm has quit IRC
787 2017-10-05T22:29:12  *** Cheeseo has quit IRC
788 2017-10-05T22:29:37  *** Cheeseo has joined #bitcoin-core-dev
789 2017-10-05T22:30:34  *** echonaut1 has joined #bitcoin-core-dev
790 2017-10-05T22:31:07  *** echonaut has quit IRC
791 2017-10-05T22:36:32  *** Chris_Stewart_5 has quit IRC
792 2017-10-05T23:08:31  *** Cheeseo has quit IRC
793 2017-10-05T23:15:00  <bitcoin-git> [bitcoin] theuni opened pull request #11457: Introduce BanMan (master...move-bandb) https://github.com/bitcoin/bitcoin/pull/11457
794 2017-10-05T23:20:47  *** vicenteH has quit IRC
795 2017-10-05T23:29:29  <luke-jr> jnewbery: your version seems to pass the wallet by name instead of CWalletRef
796 2017-10-05T23:36:08  *** stevenroose has quit IRC
797 2017-10-05T23:38:21  *** stevenroose has joined #bitcoin-core-dev
798 2017-10-05T23:49:27  *** Emcy_ has quit IRC
799 2017-10-05T23:49:54  *** Emcy_ has joined #bitcoin-core-dev
800 2017-10-05T23:58:26  *** abpa has quit IRC