1 2018-08-23T00:05:53  *** promag has joined #bitcoin-core-dev
  2 2018-08-23T00:19:11  *** Victorsueca has quit IRC
  3 2018-08-23T00:22:35  *** Victorsueca has joined #bitcoin-core-dev
  4 2018-08-23T00:39:01  *** d9b4bef9 has quit IRC
  5 2018-08-23T00:40:14  *** d9b4bef9 has joined #bitcoin-core-dev
  6 2018-08-23T00:59:11  *** AaronvanW has quit IRC
  7 2018-08-23T00:59:24  *** Zenton has joined #bitcoin-core-dev
  8 2018-08-23T00:59:49  *** AaronvanW has joined #bitcoin-core-dev
  9 2018-08-23T01:03:57  *** AaronvanW has quit IRC
 10 2018-08-23T01:17:10  *** Krellan has quit IRC
 11 2018-08-23T01:17:45  *** Krellan has joined #bitcoin-core-dev
 12 2018-08-23T01:21:07  *** peevsie has joined #bitcoin-core-dev
 13 2018-08-23T01:26:05  *** promag has quit IRC
 14 2018-08-23T01:26:59  *** unholymachine has joined #bitcoin-core-dev
 15 2018-08-23T02:03:25  *** Jmabsd has joined #bitcoin-core-dev
 16 2018-08-23T02:04:16  <Jmabsd> sipa, re "uint256's ToString() method", what's the exact code location? it's not in uint256.h/c
 17 2018-08-23T02:04:49  <Jmabsd> ah dear, wait, it's there - "class uint256 : public base_blob<256>", and then base_blob has a toString, obviously.
 18 2018-08-23T02:05:50  <Jmabsd> ToString just calls GetHex, and it's defined as "return HexStr(std::reverse_iterator<.." - there we have it. thx!
 19 2018-08-23T02:12:54  <sipa> exactly.
 20 2018-08-23T02:16:12  *** Jmabsd has quit IRC
 21 2018-08-23T02:22:46  *** Lightsword_ has joined #bitcoin-core-dev
 22 2018-08-23T02:28:58  *** achow101_ has joined #bitcoin-core-dev
 23 2018-08-23T02:29:31  *** rex4539 has quit IRC
 24 2018-08-23T02:29:31  *** [b__b] has quit IRC
 25 2018-08-23T02:29:31  *** dcousens has quit IRC
 26 2018-08-23T02:29:32  *** e4xit has quit IRC
 27 2018-08-23T02:29:33  *** jonasschnelli has quit IRC
 28 2018-08-23T02:29:33  *** sipa has quit IRC
 29 2018-08-23T02:29:33  *** pierre_rochard has quit IRC
 30 2018-08-23T02:29:33  *** Lightsword has quit IRC
 31 2018-08-23T02:29:33  *** achow101 has quit IRC
 32 2018-08-23T02:29:33  *** AdrianG has quit IRC
 33 2018-08-23T02:29:39  *** Lightsword_ is now known as Lightsword
 34 2018-08-23T02:29:58  *** achow101_ is now known as achow101
 35 2018-08-23T02:30:33  *** dcousens has joined #bitcoin-core-dev
 36 2018-08-23T02:36:32  *** Jmabsd has joined #bitcoin-core-dev
 37 2018-08-23T02:37:05  *** keymone has joined #bitcoin-core-dev
 38 2018-08-23T02:41:00  <Jmabsd> oh, the HD wallet stuff uses reverse byte order too: https://github.com/bitcoin/bitcoin/blob/master/src/wallet/rpcwallet.cpp#L3059
 39 2018-08-23T02:43:12  <Jmabsd> and the merkle root is printed in reverse order.
 40 2018-08-23T02:46:34  <gmaxwell> 21:29:12 < gmaxwell> Jmabsd: the point is that bitcoin's UI uses the same stuff to print all these 256 bit hashes.
 41 2018-08-23T02:46:49  <Jmabsd> gmaxwell, yeap and also 160bit.
 42 2018-08-23T02:47:09  <Jmabsd> (this one https://github.com/bitcoin/bitcoin/blob/master/src/wallet/rpcwallet.cpp#L3059 is 160bit)
 43 2018-08-23T02:47:28  <Jmabsd> gmaxwell, is there any 256bit hash that is *not* printed in this order? also,
 44 2018-08-23T02:47:56  <gmaxwell> I don't know what code would do that printing...
 45 2018-08-23T02:47:57  <Jmabsd> i see the stronger, material, original reason for this ordering was to get the zero bits in block hashes have the zeroes in the beginning
 46 2018-08-23T02:48:31  <gmaxwell> 21:30:21 < gmaxwell> block hash is interpreted as a 256-bit little endian number for target comparison as luke says, so then bitcoin needed to print  it that way too, otherwise the 0s would have been on the right which would have been confusing... and the same code was then  used to print other hashes.
 47 2018-08-23T02:48:43  <Jmabsd> however may there be some more general, common-sense reason too to print hashes in reverse order, or it's all just about symmetry with the block hash representation?
 48 2018-08-23T02:48:45  *** profmac has quit IRC
 49 2018-08-23T02:49:05  <Jmabsd> ahaa, to get a proper ordering, interesting
 50 2018-08-23T02:50:08  <Jmabsd> that makes sense yes.   what code in Bitcoin Core compares two uint256:es, which would be used for sorting?
 51 2018-08-23T02:52:30  *** plankers has quit IRC
 52 2018-08-23T02:56:20  <Jmabsd> HEX STRING -----(funny reverse order deserialization)----> uint256 structure which is actually a 32B byte vector  ------(sort function, compares the uint256:es by normal < > logic) ---->  sorted list ----- (serialize with funny reverse order again) ----> alphanumeric order sorted list
 53 2018-08-23T02:56:27  <Jmabsd>  that only makes sense if the <  >  operator is little endian :)
 54 2018-08-23T02:57:12  <Jmabsd> ahh i see base_blob's < operator and Compare method
 55 2018-08-23T02:59:03  <Jmabsd> the < logics are in int256.h's base_blob, and implemented as "friend inline bool operator<(const base_blob& a, const base_blob& b) { return a.Compare(b) < 0; }", "inline int Compare(const base_blob& other) const { return memcmp(data, other.data, sizeof(data)); }".
 56 2018-08-23T02:59:22  <Jmabsd> memcmp here would react at the first byte that's different. this comparison logics are suitable for a *big endian* representation of a number right -
 57 2018-08-23T02:59:25  <dcousens> Jmabsd at this point,  we're simply stuck with it
 58 2018-08-23T02:59:46  <dcousens> You can reason about it in a thousand ways,  network byte ordering,  etc,  but,  its merely what was done,  and now its stuck
 59 2018-08-23T03:00:09  <Jmabsd> i compare "0x12 0x54 0xfd 0x55" with "0x12 0x54 0xf0 0x55", in this case memcmp will give a negative value due to the third slot's difference
 60 2018-08-23T03:00:58  <Jmabsd> dcousens: yeap i see. i'm trying to make just a bit of sense here, so that, if you make a Dcousens Protocol to do some Bitcoin-related stuff, and you have some structure in your protocol that you hash and then give to people, in what order should it be.
 61 2018-08-23T03:01:16  <Jmabsd> for instance, for symmetry with Bitcoin, if you make a Dcousens Bitcoin Transaction format, then obviously its hash should be printed in reverse order.
 62 2018-08-23T03:01:34  <Jmabsd> (which is some Bitcoin transaction for some purpose)
 63 2018-08-23T03:01:55  <Jmabsd> e.g. Lightning channels certainly represent the channel open/close transactions as txid:s with reverse order, and maybe some other hashes too.
 64 2018-08-23T03:04:07  <Jmabsd> dcousens: so the uint256 < operator is a big endian thing.
 65 2018-08-23T03:04:24  <Jmabsd> wait where's the PoW bits counting done again - i guess it should count the *last* bits (at the last byte positions of the uint256 structure)?
 66 2018-08-23T03:04:35  <Jmabsd> otherwise the PoW counting and the uint256 < operator are not symmetrical
 67 2018-08-23T03:04:38  <luke-jr> it doesn't count bits at all
 68 2018-08-23T03:04:39  <Jmabsd> or, wait
 69 2018-08-23T03:04:40  <Jmabsd> they are
 70 2018-08-23T03:04:51  <Jmabsd> er.
 71 2018-08-23T03:05:08  <Jmabsd> luke-jr: where is the zero counting done again
 72 2018-08-23T03:05:29  <luke-jr> it's not
 73 2018-08-23T03:05:47  <luke-jr> "zero counting" is a simplification for end users
 74 2018-08-23T03:05:56  <luke-jr> the protocol itself simply does a <
 75 2018-08-23T03:07:16  <dcousens> personally,    I use the terms  `txhash` for the normal byte ordering,  and `txid` for reverse byte ordering
 76 2018-08-23T03:07:55  <Jmabsd> luke-jr: https://github.com/bitcoin/bitcoin/blob/master/src/uint256.h#L49 is this the < operator used, or is the logics somewhere else?
 77 2018-08-23T03:08:01  <dcousens> if they need access to the raw data (aka, hashing),  they typically deal with the hash,   if they are doing lookups,  I give the reverse byte order hex string
 78 2018-08-23T03:08:11  <Jmabsd> dcousens: and for blocks you call it what?
 79 2018-08-23T03:08:24  <dcousens> blockhash blockid
 80 2018-08-23T03:08:30  <dcousens> respectively
 81 2018-08-23T03:08:30  *** Victorsueca has quit IRC
 82 2018-08-23T03:08:55  <Jmabsd> dcousens: and if you ever print a 256bit or 160bit hash in a pubkeyscript, then you print it in *byte order* right
 83 2018-08-23T03:09:04  <Jmabsd> dcousens: merkle root "id" versus "hash" too? :-))
 84 2018-08-23T03:09:05  <dcousens> define print
 85 2018-08-23T03:09:18  <Jmabsd> dcousens: "show to a user in a web page or console"
 86 2018-08-23T03:09:26  *** jb55 has quit IRC
 87 2018-08-23T03:09:30  <dcousens> yeah,  in normal byte order for that
 88 2018-08-23T03:09:42  <dcousens> again, I only make the distinction for block/tx id's
 89 2018-08-23T03:09:44  *** Victorsueca has joined #bitcoin-core-dev
 90 2018-08-23T03:09:58  <Jmabsd> dcousens: aha. Bitcoin Core would print the merkle root in reverse order. anyhow right.
 91 2018-08-23T03:10:05  <Jmabsd> dcousens: i think your approach makes sense.
 92 2018-08-23T03:11:54  *** peevsie has quit IRC
 93 2018-08-23T03:11:57  <dcousens> and merkleroots*
 94 2018-08-23T03:15:29  *** cyber55 has joined #bitcoin-core-dev
 95 2018-08-23T03:15:31  <Jmabsd> dcousens: you reverse-order your merkle roots too ???
 96 2018-08-23T03:15:59  <Jmabsd> wait, hashes are such are binary/byte concepts so the whole idea of relating to them as comparable numbers, is a concept that Bitcoin adds on top of them, right?  the inventor of SHA256 never related to hashing as   SHA256(byte string) => 256bit integer which may be little or big endian, right?
 97 2018-08-23T03:18:51  <dcousens> for display
 98 2018-08-23T03:18:53  <dcousens> IIRC
 99 2018-08-23T03:19:12  <dcousens> Bitcoin core reverse orders any 256-bit hex string, basically
100 2018-08-23T03:19:14  <Jmabsd> dcousens: aha. so "blockid", "txid" and "merkle root" you do reverse order. any more?  HD wallet seeds?
101 2018-08-23T03:19:32  <dcousens> I'd say that is all
102 2018-08-23T03:19:36  <Jmabsd> (y)
103 2018-08-23T03:19:38  *** lone3lf has joined #bitcoin-core-dev
104 2018-08-23T03:19:50  <dcousens> "blockid", "txid", and merkle roots... if I remember correctly.
105 2018-08-23T03:20:23  <dcousens> Typically, I refer to it as "merkleRootId"
106 2018-08-23T03:21:06  <achow101> Jmabsd: Bitcoin Core prints everything that is a byte array in reverse byte order
107 2018-08-23T03:21:15  <achow101> Jmabsd: except for byte arrays in scripts. those are printed as-is
108 2018-08-23T03:21:35  <achow101> literally everything else (block hash, merkle root, txid, tx hash, whatever) is reverse byte order
109 2018-08-23T03:21:48  <Jmabsd> achow101: if you print a transaction's body, that's in normal byte order no ??
110 2018-08-23T03:22:18  <Jmabsd> sorry for being spammy about this byte order question, in a way it's trivial however as soon as a user is trying to use this, if the ordering is wrong then things break.
111 2018-08-23T03:22:19  <achow101> Jmabsd: a transaction is an object, not a byte array
112 2018-08-23T03:22:36  <dcousens> achow101 except private keys
113 2018-08-23T03:22:48  <Jmabsd> achow101: right so Bitcoin Core defines both 256bit and 160bit hashes as subclasses of "base_blob", is this what you mean by byte array?
114 2018-08-23T03:22:55  <achow101> Jmabsd: yes
115 2018-08-23T03:23:08  <Jmabsd> achow101: is anything else than 256bit and 160bit hashes represented as "base_blob"??
116 2018-08-23T03:23:13  <achow101> no
117 2018-08-23T03:23:35  <Jmabsd> achow101: right. (what you say i get confirmed by "grep -r base_blob *" too - it's only mentioned in uint256.*.)
118 2018-08-23T03:24:16  <achow101> dcousens: private keys are printed as WIF :)
119 2018-08-23T03:24:16  <Jmabsd> just for reference, where is the comparison of PoW, that is against the difficulty target, and in a reorg?
120 2018-08-23T03:24:18  *** lone3lf has quit IRC
121 2018-08-23T03:24:25  <Jmabsd> that should be a "<" operator on the uint256_t type no?
122 2018-08-23T03:24:36  <achow101> Jmabsd: src/pow.*
123 2018-08-23T03:24:48  <dcousens> achow101: my point was,  in terms of a community standard
124 2018-08-23T03:25:24  <achow101> Jmabsd: https://github.com/bitcoin/bitcoin/blob/master/src/pow.cpp#L87
125 2018-08-23T03:25:34  <dcousens> the standard is reverse byte order    for   blockid, txid, merklerootid ... I don't think there is anything else you would want to adhere to in reverse byte order.
126 2018-08-23T03:25:50  *** lone3lf has joined #bitcoin-core-dev
127 2018-08-23T03:26:12  <dcousens> And therefore,  even when displaying,  I would _only_ do reverse byte order for those things,  e.g,  if you are printing a private key,  you wouldn't print it in reveres byte order
128 2018-08-23T03:26:34  <achow101> dcousens: it's actually anything that can be interpreted as an integer. e.g. version numbers are little endian which is a form of "reverse byte order"
129 2018-08-23T03:26:53  <dcousens> I don't think I've seen the locktime printed as reverse byte
130 2018-08-23T03:27:04  <dcousens> But maybe!
131 2018-08-23T03:27:17  <achow101> dcousens: is it not reversed from whatever it is stored as?
132 2018-08-23T03:27:59  <dcousens> it is stored in LE
133 2018-08-23T03:28:37  <dcousens> so not reversed afaik
134 2018-08-23T03:28:38  <achow101> right, so it is printed as a big endian decimal which is reverse byte order of little endian
135 2018-08-23T03:29:17  <dcousens> ha
136 2018-08-23T03:29:22  <dcousens> yeah I suppose
137 2018-08-23T03:29:38  <dcousens> but if I saw it in hex
138 2018-08-23T03:29:43  <dcousens> I'd probably assume LE,  not BE
139 2018-08-23T03:29:56  <dcousens> unless I saw `0x...` haha, then I'd assume BE
140 2018-08-23T03:29:58  <dcousens> what a mess
141 2018-08-23T03:31:43  *** justanotheruser has quit IRC
142 2018-08-23T03:36:46  *** rls has quit IRC
143 2018-08-23T03:46:23  *** phwalkr has joined #bitcoin-core-dev
144 2018-08-23T04:06:56  <Jmabsd> wait, for mainnet, configuration includes: consensus.powLimit = uint256S("00000000ffffffffffffffffffffffffffffffffffffffffffffffffffffffff");
145 2018-08-23T04:07:05  <Jmabsd> this means that a single proof of work may not.. what?
146 2018-08-23T04:09:15  *** fanquake has joined #bitcoin-core-dev
147 2018-08-23T04:09:52  <fanquake> cfields are you doing detached sigs now?
148 2018-08-23T04:13:40  <Jmabsd> dcousens,achow101: facepalm lol.
149 2018-08-23T04:14:38  <Jmabsd> dcousens,achow101: i like dcousens' way of relating to it that only blockid, transactionid and merklerootid should be reverse byte order, that limits the possible damage from ambiguous order conventions too ha.
150 2018-08-23T04:17:04  *** lone3lf has quit IRC
151 2018-08-23T04:17:52  <Jmabsd> achow101: the "nBits" value, which is copied from the block header and is applied in PoW here https://github.com/bitcoin/bitcoin/blob/master/src/pow.cpp#L80 : "arith_uint256 bnTarget;bnTarget.SetCompact(nBits, &fNegative, &fOverflow);"  , it's in the Bitcoin-internal 32bit floating point format right??
152 2018-08-23T04:18:10  *** fanquake has quit IRC
153 2018-08-23T04:18:14  <achow101> Jmabsd: yes
154 2018-08-23T04:18:37  <Jmabsd> achow101: actually what code generates that bits value??
155 2018-08-23T04:19:08  <Jmabsd> achow101: at the genesis block, the bits value is *very low* and then it's getting *higher and higher*, slowly approaching 2^256 right?
156 2018-08-23T04:19:39  *** vexbuy_ has quit IRC
157 2018-08-23T04:19:41  <Jmabsd> achow101: each PoW generated out there, has the block hash proof of work 256bit-integer-compared with this target value right?
158 2018-08-23T04:19:56  *** lone3lf has joined #bitcoin-core-dev
159 2018-08-23T04:20:25  <gmaxwell> it goes up and down, whatever is required to keep the block rate on target.
160 2018-08-23T04:20:44  <Jmabsd> so, Bitcoin Core does "if (UintToArith256(hash) > bnTarget)" - so you're checking that the block hash is "more work" than the target value, and meanwhile LibBitcoin does "to_uint256(hash()) <= target;", https://github.com/libbitcoin/libbitcoin/blob/master/src/chain/header.cpp#L464
161 2018-08-23T04:21:02  <Jmabsd> so LibBitcoin is checking that... err.. the target is.. BIGGER than the PoW?
162 2018-08-23T04:21:11  <Jmabsd> gmaxwell ah yes.
163 2018-08-23T04:21:52  <Jmabsd> LibBitcoin comments "Ensure actual work is at least claimed amount (smaller is more work)." - i wonder how they got this inverse definition of work, he. thoughts?
164 2018-08-23T04:22:38  <achow101> Jmabsd: Bitcoin Core checks that if the block hash is bigger than the target, it should fail, hence the return false that follows
165 2018-08-23T04:22:54  <Jmabsd> Ah. yes.
166 2018-08-23T04:23:01  <achow101> libbitcoin checks that the hash is less than the target, and if it is, it continues
167 2018-08-23T04:23:03  <achow101> they mean the same thing
168 2018-08-23T04:23:12  <Jmabsd> achow101: noted.   and... what's the point here?
169 2018-08-23T04:23:27  <gmaxwell> I've been asking myself that.
170 2018-08-23T04:23:30  <Jmabsd> you want the work to be SMALLER than the target?  so the "bits" value is going to zero??
171 2018-08-23T04:23:50  <Jmabsd> so zero means 256 leading zero bits to qualify? :-}
172 2018-08-23T04:24:12  <achow101> Jmabsd: yes. the hash should be less than the target
173 2018-08-23T04:24:39  <Jmabsd> achow101: this makes sense if "leading zero bits" means "the most significant bits in the block hash viewed as a 256bit unsigned integer" -
174 2018-08-23T04:24:46  <Jmabsd> then obviously zero bits means a SMALLER integer
175 2018-08-23T04:25:02  <Jmabsd> ok interesting, this is a very curious piece of endianness juggling, ha.
176 2018-08-23T04:25:16  <achow101> Jmabsd: the whole target checking works like this:
177 2018-08-23T04:25:30  <achow101> interpret the hash (as output by the hash function) as a 256 bit little endian integer
178 2018-08-23T04:26:07  <achow101> take nBits, decompress it and interpret it as a 256 bit little endian integer
179 2018-08-23T04:26:31  <achow101> if the hash is less than or equal to the decompressed nBits, then it is valid.
180 2018-08-23T04:27:07  <achow101> The whole reverse byte order thing is to get "256 bit little endian integer" into "256 big endian integer" because humans are used to looking at things in big endian
181 2018-08-23T04:27:21  <Jmabsd> right. (and, nBits is a custom value format so "little endian integer" here doesn't make sense as such. point is just, "make a 256bit integer type so you can compare it, and don't mess up!" he.)
182 2018-08-23T04:27:33  <achow101> yep
183 2018-08-23T04:28:13  <Jmabsd> achow101: and then for sorting blockid/txid/merklerootid:s, you want the internal 256bit integer sorting to correspond to sorting the string serialization in big endian representation form lol
184 2018-08-23T04:29:05  <gmaxwell> there isn't any normative sorting in the protocol.
185 2018-08-23T04:30:02  *** d9b4bef9 has quit IRC
186 2018-08-23T04:30:36  <Jmabsd> mhm.
187 2018-08-23T04:30:55  <Jmabsd> so we can only understand that transaction id:s and merkle root id:s (and in Bitcoin Core's case also hd wallet seeds) got this reverse byte order,
188 2018-08-23T04:31:00  *** vexbuy has joined #bitcoin-core-dev
189 2018-08-23T04:31:04  <Jmabsd> was just a coincidence of how Satoshi happened to write the code
190 2018-08-23T04:31:17  *** d9b4bef9 has joined #bitcoin-core-dev
191 2018-08-23T04:31:25  <gmaxwell> they didn't "get" any reverse order.
192 2018-08-23T04:31:28  <Jmabsd> at least we can't see more sense in it today.
193 2018-08-23T04:31:45  <Jmabsd> gmaxwell, well in the human readable string representation, they do, right
194 2018-08-23T04:31:46  <gmaxwell> everywhere inside the system there isn't any reversing. It's just printed that way.
195 2018-08-23T04:31:56  <gmaxwell> yes, as was explained to you a day ago.
196 2018-08-23T04:32:27  <gmaxwell> Because blockhashes make sense to print in that order, every other 'big number' just used the same printing code.
197 2018-08-23T04:32:31  <Jmabsd> yeap exactly - the convention of printing transaction id:s and merkle root id:s in reverse order, we can only understand it to be a coincidental design choice that originated from printing block id:s in reverse order, which does have sense in that you get the zeroes in the beginning that way, which is educative.
198 2018-08-23T04:33:14  <Jmabsd> exactly. ok i think i got it now.     regarding dcousens' terminology of "block id" vs "block hash", "tx id" vs "tx hash", and "merkle root id" vs "merkle root hash", is this a widely accepted convention, or is it just his thing??
199 2018-08-23T04:33:27  <Jmabsd> where "id" would mean reverse order and "hash" would be normal order
200 2018-08-23T04:34:52  <gmaxwell> I'm not familar with it.
201 2018-08-23T04:35:14  <gmaxwell> Really, if byte orders are tripping you up you're going to be really confused by every other detail of the system. :)
202 2018-08-23T04:35:47  <Jmabsd> lolol. no i just look at this with particular attention as any user who would get his data trashed would get really confused, ha
203 2018-08-23T04:36:00  <Jmabsd> i mean, say someone makes a program asking a user to pass a "block hash" as an argument
204 2018-08-23T04:36:10  <Jmabsd> and the guy gives it in reverse order, and then the program says "no such block exists, bye"
205 2018-08-23T04:36:38  <Jmabsd> that kind of consequental malbehavior
206 2018-08-23T04:36:50  <Jmabsd> what about "id" versus "hash", is this an established convention re block/tx/merkleroot?
207 2018-08-23T04:36:50  <achow101> Jmabsd: fun fact, bitcoin core does exactly that
208 2018-08-23T04:36:57  <Jmabsd> achow101: :o where?
209 2018-08-23T04:37:11  <achow101> if you give it the actual block hash, it will not find the block
210 2018-08-23T04:37:26  <gmaxwell> achow just means that the input is in the same order as the output.
211 2018-08-23T04:37:43  <gmaxwell> 21:36:50 < Jmabsd> what about "id" versus "hash",  > 21:34:51 < gmaxwell> I'm not familar with it.
212 2018-08-23T04:38:03  <Jmabsd> achow101: derp derp. exactly.
213 2018-08-23T04:38:10  <Jmabsd> > achow just means that the input is in the same order as the output.
214 2018-08-23T04:38:12  <Jmabsd> what do you mean?
215 2018-08-23T04:38:45  <gmaxwell> what was was trying to say is that the order bytes are in are the most trivial of 101 things you have to get right for software to work. I don't really spend any time thinking about it. Sometimes I'll do something and it won't work because I needed to reorder something, so I do, and I move on.
216 2018-08-23T04:38:47  <achow101> Jmabsd: you give it the block hash in the way that it is printed, aka reverse byte order. if you give it how it actually is, it will not find the block
217 2018-08-23T04:39:53  <Jmabsd> achow101: exactly. so all over the whole Bitcoin ecosystem, those three values (block HASH, transaction HASH and merkle root HASH aka ID whatever), when you're dealing with hex representation, it must ALWAYS be in reverse order bc otherwise you'll screw up
218 2018-08-23T04:39:55  <gmaxwell> achow101: there is no spoon^w"how it is". I mean, it "is" electrical charges, but you enter it via moving buttons. Everything we deal with is an arbritary representation.
219 2018-08-23T04:40:14  <Jmabsd> so if you have a tool / app / web site whatever that tells you which block, then you give it in reverse order
220 2018-08-23T04:40:25  <Jmabsd> if you have a CLI tool / app / whatever that asks which block or tx, then you input it in reverse order, etc.
221 2018-08-23T04:40:34  <achow101> Jmabsd: yes
222 2018-08-23T04:40:53  <Jmabsd> however aside from block hashes, transaction hashes and merkle roots, you're free to use the normal order.
223 2018-08-23T04:41:08  <achow101> Jmabsd: no
224 2018-08-23T04:41:14  <Jmabsd> achow101: no? :o
225 2018-08-23T04:41:27  <achow101> Jmabsd: those still use reverse byte order
226 2018-08-23T04:41:33  <Jmabsd> yea that's what i said:
227 2018-08-23T04:41:48  <achow101> oh, I misread, nvm
228 2018-08-23T04:42:05  <Jmabsd> achow101: if anywhere in this world you show a user a block hash, tx hash or merkle root hash, or ask him for it, then you do it in reverse order - but, for other values, such as for instance if you make a HD wallet and print him a hex dump of a seed, or,
229 2018-08-23T04:42:17  <gmaxwell> hash160s are probably also in that order in the interface, if any show up in the interface anywhere.
230 2018-08-23T04:42:20  <Jmabsd> some other hash, even if it relates to the bitcoin protocol, then normal order is fine
231 2018-08-23T04:42:33  <Jmabsd> for instance, the 32B and 20B hashes in pubkeyscripts! those would be normal order
232 2018-08-23T04:42:45  <achow101> gmaxwell: it depends on where the hash160 is, apparently
233 2018-08-23T04:42:53  <gmaxwell> pubkeyscripts is not an arith/hash_n, it's just bytes.
234 2018-08-23T04:42:54  <Jmabsd> gmaxwell: exactly, Bitcoin Core's HDwallet seeds are 160bit and presented in reverse order, yes
235 2018-08-23T04:43:12  *** justanotheruser has joined #bitcoin-core-dev
236 2018-08-23T04:43:15  <gmaxwell> in any case, trying to come up with a greater theme is probably a waste of time. You do whatever works.
237 2018-08-23T04:43:16  <achow101> if it is being output in, e.g. getaddressinfo it will be in reverse byte order. but in decodescript, it is not
238 2018-08-23T04:43:30  <gmaxwell> There is a historical reason that it works the way it does, sure, but all that matters is that you do what works.
239 2018-08-23T04:43:44  <Jmabsd> achow101,gmaxwell: maybe to help the world a bit, it's better actually to talk about "block id", "transaction id" and "merkle root id", and by "id" you mean "reverse order", even though it's not universally established nomenclature - tihs was a nice suggestion by dcousens.
240 2018-08-23T04:44:21  <gmaxwell> if you want to predict what something will be (why?) then I believe "is it stored as a big uint type in the system" is the necessary and sufficient predictor.
241 2018-08-23T04:44:47  <gmaxwell> I think if you're spening enough time thinking about this that you need names for it, you're probably doomed to never do anything useful.
242 2018-08-23T04:44:57  <Jmabsd> achow101: actually when do you ever show a user the merkle root ID...
243 2018-08-23T04:45:03  <Jmabsd> the merkle root 256bit value is soo internal
244 2018-08-23T04:45:09  <Jmabsd> you never compare it with anything else too
245 2018-08-23T04:45:12  <achow101> Jmabsd: in getblock
246 2018-08-23T04:45:22  <gmaxwell> I can tell you that it's possible to work for years on the system and never think about any of this for more than a moment.
247 2018-08-23T04:45:26  <Jmabsd> achow101: you make a lookup of a block based on its merkle root or what :o
248 2018-08-23T04:45:42  <achow101> Jmabsd: nope, just as output for a decoded block
249 2018-08-23T04:45:54  <Jmabsd> right. and it's reverse order. derp derp. yeap.
250 2018-08-23T04:45:58  <gmaxwell> and I've seen from years in this channel that people that come in asking lots of questions about byte order almost all just get frustrated and give up.
251 2018-08-23T04:46:30  <Jmabsd> okok i got it. anyhow all clear now. thx.
252 2018-08-23T04:47:07  <gmaxwell> the order it prints big uints is arbritary, but no more or less arbritary than the decision to print them in hex instead of base2 or base64 or something else.
253 2018-08-23T04:47:47  <gmaxwell> the order made sense for one thing, and then everything else just did the same because they share printing code and the order was irrelevant for everything else.
254 2018-08-23T04:48:22  <Jmabsd> achow101: if you ever printed a merkle *node* to a user, would you do it in reverse order, i guess *not*.
255 2018-08-23T04:48:37  <gmaxwell> of course.
256 2018-08-23T04:49:06  <gmaxwell> if its represented as a uint in the software, it gets printed in that order.
257 2018-08-23T04:49:28  <gmaxwell> so it would be 'revese' if you mean printing an interior node hash.
258 2018-08-23T04:49:35  <gmaxwell> er. reverse.
259 2018-08-23T04:50:34  <achow101> Jmabsd: it would be in reverse because I would use a uint256 and that prints in reverse :p
260 2018-08-23T04:50:46  <achow101> (in a modification of Bitcoin Core)
261 2018-08-23T04:50:56  <gmaxwell> there are already a zillion random fields that show up in the UI in reverse order that litterally no one ever thought about. They just printed a uint and thats how the uints print method works.
262 2018-08-23T04:51:07  <gmaxwell> So, for example, there is something that prints a hash of the utxo set.
263 2018-08-23T04:51:16  <gmaxwell> that was added last year.
264 2018-08-23T04:51:45  <gmaxwell> Without looking I'm sure it's in "reverse" order. No one ever even thought about it. Because the order is irrelevant, so they just get the default behavior.
265 2018-08-23T04:52:09  <Jmabsd> gmaxwell, a hash of all utxo:s in the blockchain at a given block height?
266 2018-08-23T04:52:30  <gmaxwell> at the current height.
267 2018-08-23T04:52:45  <gmaxwell> it's useful for checking for corruption in the utxo set.
268 2018-08-23T04:52:52  <Jmabsd> derp derp, ok Bitcoin Core gonna kick in more reverse order hex conventions in the ecosystem over time, he he. however users won't care really about those.
269 2018-08-23T04:52:59  <Jmabsd> aha yep.
270 2018-08-23T04:53:42  <Jmabsd> one could call it "utxo set id" to subtly emphasise that it's reverse order.
271 2018-08-23T04:53:54  <gmaxwell> or, one could not care because it's a total waste of time.
272 2018-08-23T04:54:13  <gmaxwell> no one ever had to think about it at all.
273 2018-08-23T04:54:42  <gmaxwell> if you get too busy thinking about minutia you'll probably find your mind becomes too crowded to think about more complicated things. :)
274 2018-08-23T05:02:53  *** profmac has joined #bitcoin-core-dev
275 2018-08-23T05:13:20  <gmaxwell> achow101: https://bitcoin.stackexchange.com/questions/78292/insanely-high-fee lol I wonder if we should make it _harder_ to bypass that...
276 2018-08-23T05:13:39  <gmaxwell> I wonder how many people hit that and then frantically work around it without realizing that its saving them from doom.
277 2018-08-23T05:14:54  <gmaxwell> like maybe to bypass it the node should present you with a randomly generated nonogram puzzle. :P
278 2018-08-23T05:15:55  <luke-jr> insanely high fee isn't enforced as a relay policy, is it? O.o
279 2018-08-23T05:16:23  <gmaxwell> no.
280 2018-08-23T05:16:39  <gmaxwell> it's just enforced at the user interfaces.
281 2018-08-23T05:16:43  <gmaxwell> and its easy to bypass.
282 2018-08-23T05:17:06  <gmaxwell> but I've several times seen people asking about how they could bypass it and it turns out it was saving them.
283 2018-08-23T05:17:10  <gmaxwell> (as is the case in that link)
284 2018-08-23T05:30:00  *** Krellan has quit IRC
285 2018-08-23T05:30:52  *** Krellan has joined #bitcoin-core-dev
286 2018-08-23T06:13:54  *** plankers has joined #bitcoin-core-dev
287 2018-08-23T06:16:03  *** vexbuy has quit IRC
288 2018-08-23T06:35:05  *** schmidty has quit IRC
289 2018-08-23T06:40:27  *** vexbuy has joined #bitcoin-core-dev
290 2018-08-23T06:51:49  *** jonasschnelli_ has joined #bitcoin-core-dev
291 2018-08-23T06:52:02  *** jonasschnelli_ is now known as jonasschnelli
292 2018-08-23T06:53:33  *** Jmabsd has quit IRC
293 2018-08-23T06:56:30  *** vexbuy has quit IRC
294 2018-08-23T06:57:53  <jonasschnelli> cfields: I wonder why our OSX SDKs have a different hashs: https://github.com/bitcoin-core/gitian.sigs/commit/2ed8b5fba57623c0160eebc0c77ee4f9f2fe041a#diff-4bb2ae50ac744a39b2feaf294eb21d8eR9. https://github.com/bitcoin-core/gitian.sigs/pull/781/files#diff-0016af707809da5b52923d223d1943fcR9
295 2018-08-23T06:58:00  <jonasschnelli> It's not a problem,... just wondering
296 2018-08-23T07:05:36  *** [b__b] has joined #bitcoin-core-dev
297 2018-08-23T07:06:18  <provoostenator> They're all different? https://github.com/bitcoin-core/gitian.sigs/pull/780/files#diff-32221b5f40db6596d8213cb93b29c705R9
298 2018-08-23T07:07:53  <provoostenator> The SDK file isn't distributed, right?
299 2018-08-23T07:08:58  <jonasschnelli> looks like... I though we have all shared the same tarball? But maybe not
300 2018-08-23T07:13:55  <provoostenator> I used OSX to extract the relevant stuff from the thing I downloaded from Apple. On Linux you have to use different tools. Or maybe it's a timestamp thing?
301 2018-08-23T07:15:43  <provoostenator> I've had the same hash since 0.16.0rc4 it seems, and I've used at least two distinct gitian machines.
302 2018-08-23T07:22:42  *** phwalkr has quit IRC
303 2018-08-23T07:35:18  *** Victorsueca has quit IRC
304 2018-08-23T07:36:07  *** Krellan has quit IRC
305 2018-08-23T07:36:29  *** Victorsueca has joined #bitcoin-core-dev
306 2018-08-23T07:36:53  *** Krellan has joined #bitcoin-core-dev
307 2018-08-23T07:36:57  *** vexbuy has joined #bitcoin-core-dev
308 2018-08-23T07:37:07  *** csknk has joined #bitcoin-core-dev
309 2018-08-23T07:41:53  *** vexbuy has quit IRC
310 2018-08-23T07:42:33  <luke-jr> jonasschnelli: the tarball isn't deterministically made..
311 2018-08-23T07:42:49  <luke-jr> IMO it's a good thing that we have obviously-different SDKs ;)
312 2018-08-23T07:42:56  <jonasschnelli> Yes. Indeed
313 2018-08-23T07:50:31  *** vexbuy has joined #bitcoin-core-dev
314 2018-08-23T08:20:31  *** elichai2 has joined #bitcoin-core-dev
315 2018-08-23T08:32:20  *** gnappuraz has joined #bitcoin-core-dev
316 2018-08-23T08:44:08  <gnappuraz> Hi there, I was wondering if there is any documentation relative to the different archives declared in the main Makefile. Cause I don't really understand the logic of how stuff is organized. i.e. what is the difference between libbitcoin_util and libbitcoin_common?
317 2018-08-23T08:49:07  <jonasschnelli> gnappuraz: these are semi-independent modules. Those modules get linked for the different binaries we have (bitcoin-tx, bitcoin-cli, bitcoind, bitcoin-qt).
318 2018-08-23T08:49:41  <jonasschnelli> But they do not have an ABI or are meant to be used with other applications
319 2018-08-23T08:59:27  *** gnappuraz has quit IRC
320 2018-08-23T08:59:54  *** ossifrage has quit IRC
321 2018-08-23T09:06:25  *** ossifrage has joined #bitcoin-core-dev
322 2018-08-23T09:08:33  *** gnappuraz has joined #bitcoin-core-dev
323 2018-08-23T09:08:46  <ossifrage> g++ is really annoying sometimes, a 'make -j4' just managed to OOM this crappy box
324 2018-08-23T09:11:42  <gnappuraz> thx for the answer. But let's assume that I have to add a new file that if meant to be use by both bitcoind and bitcoin-qt, the rationale behind putting it in COMMON vs UTIL should be: if used by bitcoin-cli then UTIL, otherwise COMMON (since bitcoin-cli links UTIL but not COMMON)?
325 2018-08-23T09:23:01  *** GAit has joined #bitcoin-core-dev
326 2018-08-23T09:24:07  *** AaronvanW has joined #bitcoin-core-dev
327 2018-08-23T09:24:34  *** AaronvanW has quit IRC
328 2018-08-23T09:26:33  *** AaronvanW has joined #bitcoin-core-dev
329 2018-08-23T10:04:48  *** e4xit has joined #bitcoin-core-dev
330 2018-08-23T10:06:26  <Empact> Is it possible to extend the ability to restart appveyor builds to core devs?
331 2018-08-23T10:08:37  *** SopaXorzTaker has joined #bitcoin-core-dev
332 2018-08-23T10:14:53  <ken2812221_> MarcoFalke: You can checkout https://ci.appveyor.com/gitHubTeams and https://ci.appveyor.com/project/Drahtbot/bitcoin/settings/permissions to help you setting up appveyor permissions.
333 2018-08-23T10:16:43  *** ken2812221_ is now known as ken2812221
334 2018-08-23T10:31:32  <Empact> ken2812221: Thanks - set up the former but don't have access to the latter.
335 2018-08-23T10:37:51  *** plankers has quit IRC
336 2018-08-23T10:39:36  *** vexbuy_ has joined #bitcoin-core-dev
337 2018-08-23T10:43:06  *** vexbuy has quit IRC
338 2018-08-23T10:44:52  <ken2812221> Empact: I was talking to MarcoFalke, he's the owner of appveyor ci account.
339 2018-08-23T10:47:18  *** Empact has quit IRC
340 2018-08-23T11:10:21  *** gnappuraz has quit IRC
341 2018-08-23T11:16:35  *** vexbuy has joined #bitcoin-core-dev
342 2018-08-23T11:20:18  *** vexbuy_ has quit IRC
343 2018-08-23T11:24:15  *** vexbuy has quit IRC
344 2018-08-23T11:33:07  *** promag has joined #bitcoin-core-dev
345 2018-08-23T11:37:03  *** promag has quit IRC
346 2018-08-23T11:39:31  *** _flow_ has joined #bitcoin-core-dev
347 2018-08-23T11:41:39  <_flow_> What can I do to get https://github.com/bitcoin/bitcoin/pull/13621 merged? It improves the, currently terriable, configuration mechanics of bitcoind a bit (while still not perfect) and re-enables a recently disabled test.
348 2018-08-23T11:43:03  <wumpus> we're really careful merging changes to the order of configuration file parsing, slight changes to precedence will mess up things for current users and there are no good tests for this
349 2018-08-23T11:43:06  *** lnostdal has joined #bitcoin-core-dev
350 2018-08-23T11:43:18  <jonasschnelli> _flow_: it needs more developers weighting in on the importance of that PR
351 2018-08-23T11:43:21  <wumpus> _flow_: so I suppose my answer is "add good tests"!
352 2018-08-23T11:44:11  <wumpus> those would also help to show what is improved, as they would fail on the old code and pass on the new one
353 2018-08-23T11:45:09  <wumpus> also: does this affect the GUI
354 2018-08-23T11:45:17  <wumpus> I see you did update the existing test, good
355 2018-08-23T11:45:56  <wumpus> but I think it needs to extended, to include the failure cases that you describe
356 2018-08-23T11:49:33  *** Gnappuraz has joined #bitcoin-core-dev
357 2018-08-23T11:49:36  *** vexbuy has joined #bitcoin-core-dev
358 2018-08-23T11:54:01  *** cyber55 has quit IRC
359 2018-08-23T11:55:36  *** belcher_ has joined #bitcoin-core-dev
360 2018-08-23T12:01:09  <_flow_> wumpus: jonasschnelli: thanks. I'll consider adding a test for the issue the user in https://github.com/bitcoin/bitcoin/pull/12255#issuecomment-403666092 is experiencing
361 2018-08-23T12:02:22  *** Guyver2 has joined #bitcoin-core-dev
362 2018-08-23T12:22:30  *** rafalcpp_ has quit IRC
363 2018-08-23T12:34:20  *** vexbuy_ has joined #bitcoin-core-dev
364 2018-08-23T12:38:35  *** vexbuy has quit IRC
365 2018-08-23T12:57:39  *** pergaminho has joined #bitcoin-core-dev
366 2018-08-23T13:09:53  *** belcher_ has quit IRC
367 2018-08-23T13:23:36  *** jcorgan has quit IRC
368 2018-08-23T13:23:58  *** rafalcpp has joined #bitcoin-core-dev
369 2018-08-23T13:26:29  *** Victorsueca has quit IRC
370 2018-08-23T13:27:44  *** Victorsueca has joined #bitcoin-core-dev
371 2018-08-23T13:30:59  *** vexbuy_ has quit IRC
372 2018-08-23T13:46:48  *** vexbuy has joined #bitcoin-core-dev
373 2018-08-23T13:54:16  *** Emcy_ has quit IRC
374 2018-08-23T13:56:39  <jonasschnelli> how do I compile to detect pass-the-end iterator violations? I once had this setup but can no longer reproduce it.
375 2018-08-23T13:56:46  <jonasschnelli> (ideally gcc/debian)
376 2018-08-23T13:58:18  *** Krellan has quit IRC
377 2018-08-23T13:58:27  <jonasschnelli> Oh.. is it maybe DEBUG=1 for the depends build?
378 2018-08-23T13:59:01  *** Krellan has joined #bitcoin-core-dev
379 2018-08-23T14:08:32  *** Rootsudo has joined #bitcoin-core-dev
380 2018-08-23T14:11:51  *** csknk has quit IRC
381 2018-08-23T14:12:48  *** wxss has joined #bitcoin-core-dev
382 2018-08-23T14:38:53  *** petertod1 is now known as petertodd
383 2018-08-23T14:40:26  *** d_t has joined #bitcoin-core-dev
384 2018-08-23T14:41:06  <achow101> gmaxwell: luke-jr: there's an additional "absurdly high fee" that is enforced as a relay policy. it's triggered by a tx with a fee higher than 0.01 BTC (or something like that)
385 2018-08-23T14:42:51  <achow101> gmaxwell: also, insanely high fee is something that I can only find in 0.8 and earlier. I think that guy is making his own altcoin from 0.8.6 as people tend to do
386 2018-08-23T14:47:38  <treyzania> why do people fork from that far back?
387 2018-08-23T15:01:35  *** Giszmo has quit IRC
388 2018-08-23T15:04:00  *** michaelsdunn1 has joined #bitcoin-core-dev
389 2018-08-23T15:17:52  *** SopaXorzTaker has quit IRC
390 2018-08-23T15:24:36  *** pergaminho has quit IRC
391 2018-08-23T15:31:52  *** pergaminho has joined #bitcoin-core-dev
392 2018-08-23T15:39:40  *** wxss has quit IRC
393 2018-08-23T15:41:11  <cfields> fanquake: I was waiting to make sure that my results matched jonasschnelli's before signing
394 2018-08-23T15:41:41  <cfields> jonasschnelli: yes, it's DEBUG=1 for depends
395 2018-08-23T15:42:09  <cfields> jonasschnelli: for sigs, note that this is the first detached sig in a new release series. So we'll switch to a 0.17 branch to match.
396 2018-08-23T15:42:46  <jonasschnelli> cfields: ack. I'll PR after you pushed your commit (if you havent)
397 2018-08-23T15:43:42  <cfields> jonasschnelli: pushed
398 2018-08-23T15:47:32  <cfields> jonasschnelli: for future reference, I use "--orphan" when creating new git branches in the detached sigs repo, to avoid sharing any history between them.
399 2018-08-23T15:48:02  *** Emcy_ has joined #bitcoin-core-dev
400 2018-08-23T15:48:08  *** profmac has quit IRC
401 2018-08-23T15:48:24  <jonasschnelli> cfields: noted
402 2018-08-23T15:48:44  *** profmac has joined #bitcoin-core-dev
403 2018-08-23T15:56:31  *** Giszmo has joined #bitcoin-core-dev
404 2018-08-23T16:01:49  <gmaxwell> achow101: no there isn't. re: relay policy.
405 2018-08-23T16:02:22  <gmaxwell> achow101: you're probably confused because the check is in ATMP, but it's always called with it disabled when handlign a transaction from the network.
406 2018-08-23T16:09:33  *** Gnappuraz has quit IRC
407 2018-08-23T16:15:48  *** Rootsudo has quit IRC
408 2018-08-23T16:16:29  *** Rootsudo has joined #bitcoin-core-dev
409 2018-08-23T16:17:02  *** Rootsudo has joined #bitcoin-core-dev
410 2018-08-23T16:17:21  *** Rootsudo has quit IRC
411 2018-08-23T16:18:44  *** Rootsudo has joined #bitcoin-core-dev
412 2018-08-23T16:19:39  *** Rootsudo has joined #bitcoin-core-dev
413 2018-08-23T16:40:25  *** sipa has joined #bitcoin-core-dev
414 2018-08-23T17:10:30  *** leishman has joined #bitcoin-core-dev
415 2018-08-23T17:36:45  *** Emcy has joined #bitcoin-core-dev
416 2018-08-23T17:37:12  *** Emcy_ has quit IRC
417 2018-08-23T17:39:03  *** profmac has quit IRC
418 2018-08-23T17:48:23  *** rex4539 has joined #bitcoin-core-dev
419 2018-08-23T17:50:24  *** profmac has joined #bitcoin-core-dev
420 2018-08-23T17:50:59  *** vexbuy has quit IRC
421 2018-08-23T18:00:05  *** d_t has quit IRC
422 2018-08-23T18:00:23  *** Empact has joined #bitcoin-core-dev
423 2018-08-23T18:00:57  *** Krellan has quit IRC
424 2018-08-23T18:01:59  *** pergaminho has quit IRC
425 2018-08-23T18:02:08  *** Emcy has quit IRC
426 2018-08-23T18:06:42  *** Rootsudo has joined #bitcoin-core-dev
427 2018-08-23T18:07:29  <wumpus> what's up with travis, btw? it looks like it's failing on all PRs
428 2018-08-23T18:09:21  <sipa> feature_block fails?
429 2018-08-23T18:09:30  <wumpus> there's something wrong with this new appveyor build
430 2018-08-23T18:09:47  <sipa> oh, it's appveyor that fails, not travis
431 2018-08-23T18:09:47  <wumpus> "Error making request with Error Code ServiceUnavailable and Http Status Code ServiceUnavailable. No further error information was returned by the service."
432 2018-08-23T18:10:31  *** SopaXorzTaker has joined #bitcoin-core-dev
433 2018-08-23T18:10:46  <wumpus> I've also seen feature_block fail though, but not consistently
434 2018-08-23T18:11:38  <wumpus> on the very-last test (the reorg)
435 2018-08-23T18:26:01  *** leishman has quit IRC
436 2018-08-23T18:31:42  *** nullptr| has quit IRC
437 2018-08-23T18:32:45  *** nullptr| has joined #bitcoin-core-dev
438 2018-08-23T18:34:16  *** Emcy has joined #bitcoin-core-dev
439 2018-08-23T18:39:34  *** math_ has joined #bitcoin-core-dev
440 2018-08-23T18:43:35  <MarcoFalke> ken2812221: Empact: I set the permissions to view and run builds for our github team
441 2018-08-23T18:43:40  <MarcoFalke> Could someone verify that?
442 2018-08-23T18:44:41  *** promag has joined #bitcoin-core-dev
443 2018-08-23T18:46:18  <Empact> MarcoFalke: I'm still not seeing an option to restart the appveyor
444 2018-08-23T18:47:10  <MarcoFalke> You need to be in the group that can assign labels to issues on our GitHub, I can't see who is in there
445 2018-08-23T18:50:38  <Empact> Gotcha, yeah I'm not in the group
446 2018-08-23T18:51:23  <wumpus> let's see if I can restart appveryot
447 2018-08-23T18:53:16  <wumpus> I don't see it https://ci.appveyor.com/project/DrahtBot/bitcoin/build/master.127, but maybe I'm looking over it
448 2018-08-23T18:53:36  <wumpus> (logged in through github)
449 2018-08-23T18:56:41  <promag> wumpus: can I get #13100 in HP now that 13529 is merged?
450 2018-08-23T18:56:45  <gribble> https://github.com/bitcoin/bitcoin/issues/13100 | gui: Add dynamic wallets support by promag · Pull Request #13100 · bitcoin/bitcoin · GitHub
451 2018-08-23T18:57:08  <wumpus> promag: the meeting is in four minutes, but sure...
452 2018-08-23T18:57:45  <MarcoFalke> hmm maybe I forgot to press "save"?
453 2018-08-23T18:57:48  <MarcoFalke> Mind to try again?
454 2018-08-23T18:58:27  <promag> wumpus: thanks
455 2018-08-23T19:00:04  <sipa> meeting?
456 2018-08-23T19:00:12  <wumpus> #startmeeting
457 2018-08-23T19:00:12  <lightningbot> Meeting started Thu Aug 23 19:00:12 2018 UTC.  The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
458 2018-08-23T19:00:12  <lightningbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
459 2018-08-23T19:00:16  <sipa> hi
460 2018-08-23T19:00:35  <gmaxwell> hi
461 2018-08-23T19:00:35  <achow101> hi
462 2018-08-23T19:00:59  <jonasschnelli> hi
463 2018-08-23T19:01:04  <promag> hi
464 2018-08-23T19:01:04  <wumpus> #bitcoin-core-dev Meeting: wumpus sipa gmaxwell jonasschnelli morcos luke-jr btcdrak sdaftuar jtimon cfields petertodd kanzure bluematt instagibbs phantomcircuit codeshark mi
465 2018-08-23T19:01:08  <wumpus> chagogo marcofalke paveljanik NicolasDorier jl2012 achow101 meshcollider jnewbery maaku fanquake promag provoostenator
466 2018-08-23T19:01:09  *** meshcollider_ has joined #bitcoin-core-dev
467 2018-08-23T19:01:27  <wumpus> who has a topic?
468 2018-08-23T19:01:42  <meshcollider> Hi
469 2018-08-23T19:02:04  <jonasschnelli> topic proposal: tor plugable transport
470 2018-08-23T19:02:08  <wumpus> MarcoFalke: I still don't see it, though I'm not sure what kind of button I'm looking for
471 2018-08-23T19:02:41  <wumpus> let's start with high prio as usual
472 2018-08-23T19:02:46  <wumpus> #topic high priority for review
473 2018-08-23T19:03:23  <jonasschnelli> I'd like to add #14032 if that is appropriate
474 2018-08-23T19:03:25  <gribble> https://github.com/bitcoin/bitcoin/issues/14032 | Add p2p layer encryption with ECDH/ChaCha20Poly1305 by jonasschnelli · Pull Request #14032 · bitcoin/bitcoin · GitHub
475 2018-08-23T19:03:50  <wumpus> https://github.com/bitcoin/bitcoin/projects/8  two were merged, one was added, this leaves five open
476 2018-08-23T19:04:26  <wumpus> jonasschnelli: sure
477 2018-08-23T19:04:44  <cfields> jonasschnelli: thanks, I'm excited to read that.
478 2018-08-23T19:05:01  <jonasschnelli> I'm also excited to hear your feedback. :)
479 2018-08-23T19:05:04  <wumpus> yes, very good to see a PR about that
480 2018-08-23T19:05:07  <promag> jonasschnelli: do you think you can extract some commits to new PR's?
481 2018-08-23T19:05:16  <MarcoFalke> I'd like to add #13961 (it is only refactoring, but drops the memory usage at compile time by 100Mb, also compiles faster)
482 2018-08-23T19:05:18  <gribble> https://github.com/bitcoin/bitcoin/issues/13961 | util: Replace boost::signals2 with std::function by MarcoFalke · Pull Request #13961 · bitcoin/bitcoin · GitHubAsset 1Asset 1
483 2018-08-23T19:05:40  <jonasschnelli> promag: I guess it's hard to make it smaller...
484 2018-08-23T19:05:46  <wumpus> MarcoFalke: ok
485 2018-08-23T19:05:46  <cfields> MarcoFalke: also nice :)
486 2018-08-23T19:05:49  <achow101> topic suggestion: hardware wallet support
487 2018-08-23T19:06:44  <wumpus> #topic tor pluggable transport (jonasschnelli)
488 2018-08-23T19:06:59  <jonasschnelli> Tor has this concept of pluggable transports
489 2018-08-23T19:07:01  <promag> jonasschnelli: could create a PR only with the first 3 commits, but let's talk later
490 2018-08-23T19:07:04  <jonasschnelli> https://www.torproject.org/docs/pluggable-transports.html.en
491 2018-08-23T19:07:08  <jonasschnelli> https://gitweb.torproject.org/torspec.git/tree/pt-spec.txt
492 2018-08-23T19:07:08  *** spinza has quit IRC
493 2018-08-23T19:07:21  <Empact> jonasshnelli: First three commits look independent as prep work
494 2018-08-23T19:07:25  <jonasschnelli> It a great concept to bypass DPI, censored envs...
495 2018-08-23T19:07:56  * sipa suggests using gramtropy generated text over IRC
496 2018-08-23T19:07:58  <jonasschnelli> Tor can start subprocesses that bootstrap a SOCK proxy to communicate with the counterparty
497 2018-08-23T19:08:06  <gmaxwell> jonasschnelli: Why do you think we need something special for that?  We have proxy support already, which is all that is generally required for that sort of thing.
498 2018-08-23T19:08:17  <jonasschnelli> I think supporting PT should be not to complicated in Core
499 2018-08-23T19:08:30  <gmaxwell> I think it's so not complicated that we already have it. :)
500 2018-08-23T19:08:36  <wumpus> I don't understand
501 2018-08-23T19:08:40  <jonasschnelli> I'm unsure about the reverse proxy, the env vars and the auto-exec of that subprocess...
502 2018-08-23T19:08:46  <wumpus> pluggable transports plug into *tor*
503 2018-08-23T19:08:48  <sipa> jonasschnelli: what specific form of transport do you suggest?
504 2018-08-23T19:08:53  <wumpus> bitcoin core can use tor
505 2018-08-23T19:08:57  <wumpus> what else is needed?
506 2018-08-23T19:09:12  <jonasschnelli> sipa: its not about specific transports,.. IMO its an API for other transport layers...
507 2018-08-23T19:09:29  <sipa> jonasschnelli: it is to make tor communicate with other tor nodes using different transport layers
508 2018-08-23T19:09:35  <sipa> what transport layer do you suggest?
509 2018-08-23T19:09:37  <gmaxwell> The API is socks.
510 2018-08-23T19:09:40  <wumpus> looks like a layer violation to worry about this? or do you want to make tor tunnel over bitcoin instead of the other way around?
511 2018-08-23T19:09:43  <luke-jr> I also don't get it
512 2018-08-23T19:09:46  <jonasschnelli> Maybe I got it wrong,.. but it looked to me after a pure transport layer (not tor)
513 2018-08-23T19:10:07  <sipa> yes and no
514 2018-08-23T19:10:07  <wumpus> yes, socks (+optional control port) is good enough as API
515 2018-08-23T19:10:20  <wumpus> it's the only way it's recommended to use tor
516 2018-08-23T19:10:20  <sipa> jonasschnelli: i'm very confused by what you suggest
517 2018-08-23T19:10:45  <sipa> so far you've said "we can use tor pluggable transport", which is true for any tor user
518 2018-08-23T19:10:51  <phantomcircuit> jonasschnelli, the pluggable transport presents a socks interface which tor uses instead of directly connecting
519 2018-08-23T19:10:58  <sipa> how is it bitcoin specific, or what do you specifically propose?
520 2018-08-23T19:11:00  <wumpus> integrating tor's code into bitcoin core is not a good idea
521 2018-08-23T19:11:07  <kanzure> merge tor source tree
522 2018-08-23T19:11:09  <phantomcircuit> it's not something that should be configured by things using tor
523 2018-08-23T19:11:25  <wumpus> indeed, it's part of the tor configuration
524 2018-08-23T19:11:27  <jonasschnelli> The idea is to make Bitcoin work in DPI env in case it would get blocked,.. like China or Iran
525 2018-08-23T19:11:31  <jonasschnelli> Without relying on tor
526 2018-08-23T19:11:36  <gmaxwell> wumpus: I think jonasschnelli suggests that we have support for these obfscuated transports without putting tor in the middle.  I think we already do (in fact, I used one of the ones made for tor to bridge bitcoin nodes last year when we were worried about china blocking bitcoin... before later changing to an icecast stream so that it wouldn't be identifyable by traffic analysis)
527 2018-08-23T19:11:42  <jonasschnelli> Core could directly use obfs4
528 2018-08-23T19:11:47  <gmaxwell> jonasschnelli: we already can.
529 2018-08-23T19:11:56  <jonasschnelli> or meek
530 2018-08-23T19:12:03  <sipa> jonasschnelli: ah!
531 2018-08-23T19:12:14  <wumpus> gmaxwell: ohh, so using *part* of tor, just the obfuscation part...
532 2018-08-23T19:12:18  <sipa> you mean using obfuscation for our own connections, even non tor ones
533 2018-08-23T19:12:21  <jonasschnelli> Since its a separated project and a seperate binary
534 2018-08-23T19:12:32  <gmaxwell> wumpus: well the obfuscation parts of tor aren't parts of tor, technicaly. But yes.
535 2018-08-23T19:12:35  <wumpus> this sounds like scope creep to me
536 2018-08-23T19:12:51  <gmaxwell> jonasschnelli: have you trie using obfs4. I did previously, it just worked.
537 2018-08-23T19:12:55  <gmaxwell> tried*
538 2018-08-23T19:13:02  <sipa> i would say if we're worried about that, you should also just use tor already
539 2018-08-23T19:13:04  <wumpus> gmaxwell: it's tor project code
540 2018-08-23T19:13:13  <wumpus> maybe not part of the tor repository, I don't know
541 2018-08-23T19:13:15  <phantomcircuit> i dont see any reason to do this, anybody who needs this can already configure it with the socks proxy support
542 2018-08-23T19:13:24  <wumpus> yes
543 2018-08-23T19:13:39  <jonasschnelli> okay... fair points. /topic
544 2018-08-23T19:13:56  <gmaxwell> AFAIK, it just works.  It's kinda limited though because none of them resist traffic analysis and it's SUPER easy to identify bitcoin traffic with traffic analysis. :)
545 2018-08-23T19:14:05  *** plankers has joined #bitcoin-core-dev
546 2018-08-23T19:14:27  <phantomcircuit> gmaxwell, iirc they're all super easy to identify anyways
547 2018-08-23T19:14:33  <gmaxwell> If there was some addon we needed to make more of them work, I suppose that would be cool, but I dunno what that would be. At least the original obfs proxy that I used before was just a socks proxy.
548 2018-08-23T19:15:10  <gmaxwell> phantomcircuit: well I think the idea in tor is that there are private ones.
549 2018-08-23T19:15:24  <jonasschnelli> I think core <-> core could use the propose p2p encryption (with NewHope), but for censoring, is was unsure if we should relay on Tor or on the underlaying independent obfuscation projects.
550 2018-08-23T19:15:25  <wumpus> oh the obfs layer itself is also a socks proxy?
551 2018-08-23T19:15:42  <gmaxwell> wumpus: yes
552 2018-08-23T19:15:47  <wumpus> tor is socks proxies all the way down and nothing else isn't it :)
553 2018-08-23T19:16:02  <sipa> pigeons at the bottom of the stack
554 2018-08-23T19:16:09  <wumpus> anyhow, yes, then we can already use it
555 2018-08-23T19:16:22  <wumpus> sipa: pigeons for transaction broadcasting
556 2018-08-23T19:16:24  <phantomcircuit> gmaxwell, the only ones that work are the domain fronting ones, so presumably only the private ones actually work
557 2018-08-23T19:16:46  *** plankers has joined #bitcoin-core-dev
558 2018-08-23T19:18:03  <wumpus> well the non-private ones become known very soon
559 2018-08-23T19:18:07  <gmaxwell> (hm the thing I used before wasn't obfs4 it was an earlier version, they all look like they implement proxies regardless)
560 2018-08-23T19:18:15  <wumpus> no matter how good the obfuscation is, if it's public that you run one...
561 2018-08-23T19:19:04  <gmaxwell> In any case, I think jonasschnelli's spirt is right. I just don't actually think we need to do anything more than have socks support.
562 2018-08-23T19:19:21  <wumpus> yes
563 2018-08-23T19:19:24  <jonasschnelli> Even better then. :)
564 2018-08-23T19:19:35  <gmaxwell> (it's certantly better to do protocol obfscuation via external proxies than to try to bake anything in...)
565 2018-08-23T19:19:39  <sipa> how does the proxy know which peers need to use the obfuscation?
566 2018-08-23T19:19:42  <phantomcircuit> gmaxwell, meek is the only one that works in china, it's domain fronting with gce
567 2018-08-23T19:19:52  <sipa> you can't just blanket obfuscate all connections
568 2018-08-23T19:20:04  <sipa> as our normal peer discovery will find peers that don't support it
569 2018-08-23T19:20:24  <achow101> sipa: I think this would only work for outgoing connections
570 2018-08-23T19:20:25  <wumpus> sipa: you wouldn't want to use normal peer discovery in such cases
571 2018-08-23T19:20:44  <sipa> achow101: yes, of course
572 2018-08-23T19:20:45  <sipa> wumpus: right
573 2018-08-23T19:21:24  <wumpus> something needs to de-obfuscate incoming connections too, but I suppose that works by forwarding a listening port?
574 2018-08-23T19:21:39  <gmaxwell> I think perhaps the useful 'feature' that might come for this is being able to provide per-peer proxy configs to addnode/connect.
575 2018-08-23T19:21:55  <wumpus> or does it really need SOCKS5 binding
576 2018-08-23T19:22:04  <jonasschnelli> I think achow101 has a topic
577 2018-08-23T19:22:25  <wumpus> (which we don't support, we only use the proxy for outgoing, SOCKS binding is really really obscure)
578 2018-08-23T19:22:37  <gmaxwell> I don't think there is need for socks5 binding.
579 2018-08-23T19:22:48  <wumpus> okay
580 2018-08-23T19:22:55  <wumpus> per-peer proxies could be useful, yes
581 2018-08-23T19:23:10  <cfields> gmaxwell: interesting. I believe that'd be trivial to add, other than the syntax handling
582 2018-08-23T19:23:52  <sipa> "netmask X uses proxy Y"
583 2018-08-23T19:23:56  <gmaxwell> wumpus: https://www.comparitech.com/blog/vpn-privacy/hide-openvpn-traffic-obfsproxy-on-windows-and-linux-ec2/  you can see how you can hide arbritary crap with obfsproxy.
584 2018-08-23T19:24:07  <wumpus> gmaxwell: thanks
585 2018-08-23T19:24:25  <gmaxwell> sipa: don't want to go too far or proxychains becomes the right tool (it's a multiproxy routing daemon)
586 2018-08-23T19:24:32  <sipa> ha
587 2018-08-23T19:24:47  <sipa> that sounds like an improvements over sidechains or treechains
588 2018-08-23T19:24:48  <phantomcircuit> gmaxwell, proxychains is never the right tool >.>
589 2018-08-23T19:24:50  <gmaxwell> (it can even nest proxies)
590 2018-08-23T19:25:05  <gmaxwell> phantomcircuit: but how else will I hax you through 7 proxies?
591 2018-08-23T19:25:34  <wumpus> fwiw ssh has built-in support for proxy chaining in the client
592 2018-08-23T19:26:19  <wumpus> #topic hardware wallet support (achow101)
593 2018-08-23T19:26:29  <achow101> So with PSBT we can kind of do hardware wallet things
594 2018-08-23T19:26:34  <gmaxwell> in any case, being able to net-scope or node scope proxies is something I would have used before, e.g. "addnode my node in the office via my ssh -D socks proxy".
595 2018-08-23T19:26:53  <achow101> I've been working on a tool that takes a psbt and does the hardware wallet things: https://github.com/achow101/HWI
596 2018-08-23T19:27:06  <jonasschnelli> great work achow101!
597 2018-08-23T19:27:13  *** SopaXorzTaker has quit IRC
598 2018-08-23T19:27:20  <gmaxwell> achow101: cool.
599 2018-08-23T19:27:28  <wumpus> woohoo!
600 2018-08-23T19:27:31  <achow101> but to actually get hww support, we still need a way to get derivation paths for imported keys and such. and for those keys to be part of the keypool
601 2018-08-23T19:27:59  <jonasschnelli> why not a xpub watch-only wallet support?
602 2018-08-23T19:28:09  *** clarkmoody has joined #bitcoin-core-dev
603 2018-08-23T19:28:20  <achow101> I know sipa has the output descriptors stuff that would work for this, but I think that still quite a ways off
604 2018-08-23T19:28:41  <sipa> i can prioritize it more, but it's unclear how to keep things compatible
605 2018-08-23T19:29:00  <sipa> do we want the old keystore based system and the new descriptor based system side by side?
606 2018-08-23T19:29:01  <jonasschnelli> importing keys seems meh-ish for hardware-wallet. IMO the ideal use case for BIP32 pub key derivation
607 2018-08-23T19:29:11  <sipa> that wouldn't be too hard, i think
608 2018-08-23T19:29:25  <gmaxwell> I don't see why we shouldn't deploy the very narrow thing we need to make all the common hardware wallets work, so long as it won't prevent us from using descriptor based logic in tuhe future.
609 2018-08-23T19:29:26  <sipa> but it would be much nicer if we also get to convert old keystores to desceiptors
610 2018-08-23T19:29:32  <jonasschnelli> Also, xpub watch-only do not need keypools
611 2018-08-23T19:29:42  <gmaxwell> jonasschnelli: yes it does.
612 2018-08-23T19:29:59  <jonasschnelli> I don't see why but happy to learn...
613 2018-08-23T19:30:03  <sipa> it's the equivalent of their gap limit
614 2018-08-23T19:30:07  <gmaxwell> It doesn't have private key material, but it needs _something_ to look ahead with pubkeys and watch for payments.
615 2018-08-23T19:30:16  <achow101> I did #14021 for importing keypaths into the wallet with importmulti, but it isn't ideal
616 2018-08-23T19:30:19  <gribble> https://github.com/bitcoin/bitcoin/issues/14021 | Import key origin data through importmulti by achow101 · Pull Request #14021 · bitcoin/bitcoin · GitHubAsset 1Asset 1
617 2018-08-23T19:30:49  <achow101> it ... works, but kinda sucks as you have to import more keys as you use them. And they aren't in the keypool
618 2018-08-23T19:30:49  <jonasschnelli> gmaxwell: yes. Right. Correction: no keypool written to disk, only xpub, then derive on load
619 2018-08-23T19:31:03  <sipa> i fear for more combinations and special cases to deal with
620 2018-08-23T19:31:31  <gmaxwell> jonasschnelli: well it doesn't really matter if we save it to disk or not. It's harmless to do so.
621 2018-08-23T19:31:57  <jonasschnelli> Yes. Thats right.
622 2018-08-23T19:32:06  <gmaxwell> achow101: obviously as a stopgap importing keys at least allows development and/or testing.
623 2018-08-23T19:32:59  <gmaxwell> achow101: but I assume what you really want is a wallet whos master key reflects the third-party keychain.  I fear the mess of mixed wallets.
624 2018-08-23T19:33:04  <jonasschnelli> achow101: do you have a plan to make hww interaction more hassle free, like including USBHID in core </joke>?
625 2018-08-23T19:33:29  <achow101> jonasschnelli: the idea I had was to use the HWI scripts and call them from Core.
626 2018-08-23T19:33:58  <achow101> they implement all of the usb stuff
627 2018-08-23T19:34:03  <gmaxwell> I think that should be out of of scope. We should implement a simple fork and stdio interface (or similar) to talk to hardware specific drivers.
628 2018-08-23T19:34:32  <jonasschnelli> could one of the "hardware-wallets" just be a hot keystore? Similar to SSH Agent?
629 2018-08-23T19:34:39  <gmaxwell> right.
630 2018-08-23T19:34:40  <sipa> have metadata in the wallet that says "for these outputs, attempt to sign through device driver X"
631 2018-08-23T19:34:52  <sipa> which is just an executable that gets invoked and sent a psbt
632 2018-08-23T19:34:53  <gmaxwell> Even one using SGX+TPM, for example.
633 2018-08-23T19:35:25  <gmaxwell> the drivers could then also provide their own UI, if required.
634 2018-08-23T19:35:38  <gmaxwell> E.g. to get a passphrase.
635 2018-08-23T19:35:51  <gmaxwell> or remind you to plug in the device.
636 2018-08-23T19:35:52  <achow101> yup. I plan on adding a UI to HWI once I get everything actually working
637 2018-08-23T19:36:00  <jonasschnelli> Yes. Becaue of the UI, I initially though every vendor must provide its own script/binary...
638 2018-08-23T19:36:11  <sipa> jonasschnelli: yes
639 2018-08-23T19:36:19  <jonasschnelli> Because again, plugins then endup outside of the vendors control
640 2018-08-23T19:36:25  <jonasschnelli> Leading to outdated software
641 2018-08-23T19:36:37  <gmaxwell> I'm sure it would eventually be possible to have "generic" drivers that work for a family of devices, but I think the industry is too immature for that.
642 2018-08-23T19:36:48  <jonasschnelli> Indeed
643 2018-08-23T19:37:07  <gmaxwell> sipa: how would we handle, e.g. "this script is multisig, with device X and device Y" ?
644 2018-08-23T19:37:18  <gmaxwell> (er driver x and driver y)
645 2018-08-23T19:37:20  <sipa> gmaxwell: list two drivers
646 2018-08-23T19:37:27  <gmaxwell> sounds good.
647 2018-08-23T19:37:34  <jonasschnelli> I was hoping for a protocol/API where Core starts a process and passes all relevant data and the vendors plugin there with rich UI "drivers"
648 2018-08-23T19:37:41  <sipa> gmaxwell: the drivers would be metadata to descriptor records
649 2018-08-23T19:37:45  <achow101> gmaxwell: wouldn't that be "this script has two keys and key 1 uses driver X and key 2 uses driver Y"
650 2018-08-23T19:37:55  <sipa> so they would apply to all outputs derived using the same chain(s)
651 2018-08-23T19:38:11  <sipa> achow101: doesn't even need to say which key is which device
652 2018-08-23T19:38:20  <sipa> just "try signing using X, try signing using ay"
653 2018-08-23T19:38:24  <sipa> *Y
654 2018-08-23T19:39:42  <achow101> anyways, the main point of bringing up this topic was that I wanted to ask whether we should implement xpub watch only wallets or just wait for sipa to finish output descriptor wallets
655 2018-08-23T19:39:46  <gmaxwell> This kind of interface could easily be used for things other than HWI. For example, green address has a 2fa service, where the service will sign its part of a multisig with you, if you pass a 2fa.  This could be supported by a simple driver that sends the tx to a service and gets the sig back.
656 2018-08-23T19:40:25  <gmaxwell> jonasschnelli: I don't think we should be providing the HWI specific UI, because if we provide the UI we would constrain innovation there.
657 2018-08-23T19:41:00  <jonasschnelli> Not Core. But the vendor could have a driver with UI
658 2018-08-23T19:41:12  <jonasschnelli> (and network interface and whatnot)
659 2018-08-23T19:41:13  <gmaxwell> right. I agree.
660 2018-08-23T19:42:01  <gmaxwell> The interface between core and the driver is basically just a PSBT plus arguments provided in the driver config.
661 2018-08-23T19:42:14  <gmaxwell> achow101: well I think thats between you and sipa mostly
662 2018-08-23T19:42:28  <gmaxwell> I hope we don't end up with even more combinitorial explosion in the wallet, however.
663 2018-08-23T19:42:37  <jonasschnelli> Yes. But IMO that only one part. How to get pubkey&metdata into core (and eventually multisig stuff) is still unlcear to me
664 2018-08-23T19:42:52  <gmaxwell> like having to handle a wallet with mixed old plain keys, hd keys, achowpubkey, and descriptors all at once.
665 2018-08-23T19:42:55  <jonasschnelli> (in a non RPCish way)
666 2018-08-23T19:42:59  <sipa> jonasschnelli: descriptors :)
667 2018-08-23T19:43:18  <jonasschnelli> sipa: I mean plug-n-play. :)
668 2018-08-23T19:43:20  <sipa> jonasschnelli: i'll try to create a writeup on how to integrate them in the wallet
669 2018-08-23T19:43:26  <sipa> jonasschnelli: "import a descriptor"
670 2018-08-23T19:43:41  <gmaxwell> the descriptor thing could eventually be  "create new wallet with descriptor"  "paste in descriptor" "confirm"..
671 2018-08-23T19:43:43  <sipa> ah, you mean GUI
672 2018-08-23T19:43:47  <sipa> i have no clue.
673 2018-08-23T19:43:53  <jonasschnelli> You plugin a HWW, the Core detects it and tells you "do you wanna use XYZ".
674 2018-08-23T19:43:58  <jonasschnelli> *then
675 2018-08-23T19:44:16  <gmaxwell> doubt we're going to have any detection anytime soon...
676 2018-08-23T19:44:32  <gmaxwell> as that would require shipping a pile of vendor specific detection and interface code.
677 2018-08-23T19:44:48  <jonasschnelli> Yes. But if we form an API or similar, we should not only focus on the PSBT part,... also on the how do I create a watch-only wallet (eventually also multisig)
678 2018-08-23T19:45:04  <jonasschnelli> But I guess the descriptors are the answer here
679 2018-08-23T19:45:32  <gmaxwell> Create a new wallet from a descriptor, or import a descriptor into a wallet.
680 2018-08-23T19:46:05  <gmaxwell> sipa: maybe you can work with achow 101 on a descriptor-lite implementation that would be forward compatible with the full thing later, and is enough to support the common hardware wallets?
681 2018-08-23T19:46:34  <sipa> gmaxwell: descriptors are also in the codebase
682 2018-08-23T19:46:56  <sipa> *already
683 2018-08-23T19:47:12  <gmaxwell> well I meant basically "enough stuff to be able to create an xpub with path autofilling watching wallet"
684 2018-08-23T19:47:13  <sipa> the hard part is integrating them in the wallet of course
685 2018-08-23T19:47:32  <sipa> but that isn't harder or easier depending on what the descriptors support
686 2018-08-23T19:48:07  <achow101> gmaxwell: the problem with that is the wallet format needs to change
687 2018-08-23T19:48:49  <sipa> i'll try to think it through
688 2018-08-23T19:51:28  <wumpus> any other topics?
689 2018-08-23T19:51:49  <gmaxwell> jonasschnelli: wanted to talk about encryption!
690 2018-08-23T19:52:04  <wumpus> #topic P2P encryption (jonasschnelli)
691 2018-08-23T19:52:10  <jonasschnelli> Not really... :) but happy to do
692 2018-08-23T19:52:28  <gmaxwell> heh, what just going to open that big PR and go on vacation? :P
693 2018-08-23T19:52:46  <jonasschnelli> The question for me now is, is the PR to large (2000lines) and how and when to add NewHope (quantum resistant handshake)
694 2018-08-23T19:53:35  <jonasschnelli> If it is to large, how to split it into PR that make partially sense on its own (could add the crypto stuff beforehand, but meh)
695 2018-08-23T19:53:42  <gmaxwell> so the PR can probably be broken into three parts:  (1) prepatory refactors, (2) inclusion of new primitives, (3) the thing itself.
696 2018-08-23T19:53:58  <jonasschnelli> Yes. Agree.
697 2018-08-23T19:54:05  <jonasschnelli> But would we merge 2 without 3?
698 2018-08-23T19:54:21  <jonasschnelli> Maybe 2 with concrete chances that 3 gets merged
699 2018-08-23T19:54:28  <gmaxwell> Yes. Exactly.
700 2018-08-23T19:54:45  <jonasschnelli> 1 will be very small
701 2018-08-23T19:54:53  <gmaxwell> Well if we changed our mind we could always take it out. We've put other primitives in in advance before.
702 2018-08-23T19:55:10  <gmaxwell> e.g. bip32 key derrivation went unused for years.
703 2018-08-23T19:55:14  <sipa> BIP32 derivation was in the codebase for 3 years before being used :)
704 2018-08-23T19:55:15  <sipa> jinx.
705 2018-08-23T19:55:21  <jonasschnelli> Indeed. But its 2018 now. :)
706 2018-08-23T19:55:24  <jonasschnelli> Okay. I'll do the split then.
707 2018-08-23T19:55:49  <wumpus> granted, that was before people neurotically deleted dead code :-)
708 2018-08-23T19:55:53  <gmaxwell> It's fine,  I think that chacha stuff is mostly a distraction from the review of that PR.  It needs to be reviewed but mostly for build integration, etc.
709 2018-08-23T19:56:05  <gmaxwell> wumpus: or even not-dead-yet code!
710 2018-08-23T19:56:10  <jonasschnelli> heh
711 2018-08-23T19:56:11  <sipa> wumpus: it wasn't dead! it had tests!
712 2018-08-23T19:56:14  <wumpus> gmaxwell: even preemptively
713 2018-08-23T19:56:42  <cfields> jonasschnelli: taking a quick look at your PR, was the handshake spec modified to be sent first thing? Or am I misreading the code?
714 2018-08-23T19:56:49  <jonasschnelli> Yes. I can also try to extend the existing ChaCha20 RNG to be used for the crypto part
715 2018-08-23T19:56:53  <wumpus> jonasschnelli: anyhow if it simplifies review, please do split it up
716 2018-08-23T19:56:58  <jonasschnelli> cfields: yes
717 2018-08-23T19:57:11  <jonasschnelli> cfields: https://gist.github.com/jonasschnelli/c530ea8421b8d0e80c51486325587c52#Handshake
718 2018-08-23T19:57:20  <jonasschnelli> Great. Will split
719 2018-08-23T19:57:27  <promag> nice
720 2018-08-23T19:57:57  <gmaxwell> jonasschnelli: when you split make sure all the parts are visible. E.g. I'll want to look at (3) while reviewing (1) just in case some refactor seems like it makes no sense on its own. :)
721 2018-08-23T19:58:03  <cfields> jonasschnelli: woohoo, thanks
722 2018-08-23T19:58:42  <jonasschnelli> gmaxwell: Yes. Good point. Will do.
723 2018-08-23T19:59:00  <jonasschnelli> I guess keeping the "overall" PR somewhere helps to see the goal.
724 2018-08-23T19:59:14  <gmaxwell> sipa has done that in the past, e.g. with the segwit rebase.
725 2018-08-23T19:59:30  <jonasschnelli> The unanswered question is when and how to add the NewHope handshake part
726 2018-08-23T19:59:38  <wumpus> yes, indeed, though we probably want to change the one in high-priority then
727 2018-08-23T19:59:53  <jonasschnelli> Yes. Lets remove it for now
728 2018-08-23T19:59:56  <jonasschnelli> (ill do that)
729 2018-08-23T20:00:07  <wumpus> ok
730 2018-08-23T20:00:12  <wumpus> #endmeeting
731 2018-08-23T20:00:12  <lightningbot> Meeting ended Thu Aug 23 20:00:12 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
732 2018-08-23T20:00:12  <lightningbot> Minutes:        http://www.erisian.com.au/meetbot/bitcoin-core-dev/2018/bitcoin-core-dev.2018-08-23-19.00.html
733 2018-08-23T20:00:12  <lightningbot> Minutes (text): http://www.erisian.com.au/meetbot/bitcoin-core-dev/2018/bitcoin-core-dev.2018-08-23-19.00.txt
734 2018-08-23T20:00:12  <lightningbot> Log:            http://www.erisian.com.au/meetbot/bitcoin-core-dev/2018/bitcoin-core-dev.2018-08-23-19.00.log.html
735 2018-08-23T20:00:21  <wumpus> jonasschnelli: I did already
736 2018-08-23T20:00:24  <jonasschnelli> thanks
737 2018-08-23T20:00:40  <gmaxwell> jonasschnelli: well my thought is that we make another PR on top of these-- likely with two commits, one that just adds the code to the build, the other which crams it into the handshake, so we see the marginal cost.
738 2018-08-23T20:01:11  <jonasschnelli> Yes. Exactly.
739 2018-08-23T20:02:05  <jonasschnelli> gmaxwell: Do you know if it is sane to replace using SHAKE as the output "KDF/XOF" with HKDF?
740 2018-08-23T20:02:13  <jonasschnelli> Or would that be a custom version of NewHope?
741 2018-08-23T20:03:01  <gmaxwell> jonasschnelli: we could just take its output as is for now. Lets change the absolute minimum for the first pass.
742 2018-08-23T20:03:14  <jonasschnelli> okay
743 2018-08-23T20:03:21  <cfields> jonasschnelli: I'm lagging quite far behind, so "go read the full spec" is a fine answer if you'd like. But I don't see any extensibility in the handshake spec? What's the reason for hard-coding a bare 32byte key without some sort of versioning/typing?
744 2018-08-23T20:03:40  <sipa> cfields: versioning outside of the key is identifiable
745 2018-08-23T20:03:46  <cfields> got it.
746 2018-08-23T20:03:52  <cfields> thanks
747 2018-08-23T20:03:53  <jonasschnelli> yes..
748 2018-08-23T20:04:06  <jonasschnelli> cfields: also, the additional NewHope handshake is not in the specs right now.
749 2018-08-23T20:04:18  <gmaxwell> cfields: we are avoiding having any fixed bytes so that it's harder for dumb DPI to block the traffic. A new protocol can distinguish itself by being incompatible. :)
750 2018-08-23T20:04:36  <jonasschnelli> But IMO censorship resistance is not property of that proposal...
751 2018-08-23T20:04:37  <cfields> jonasschnelli: right, I was just trying to figure out how the transition would work. I understand the issue now.
752 2018-08-23T20:05:14  <jonasschnelli> Not having censorship resistance as a property of the new protocol, doesn't mean, we should use static, identifiable data if avoidable
753 2018-08-23T20:05:23  <gmaxwell> in general upgrading this stuff, even with versioning is messy.
754 2018-08-23T20:05:57  <jonasschnelli> and that, yes.
755 2018-08-23T20:06:37  <jonasschnelli> Does anyone knows why appveyor has build issues when travis and gitian is fine with it? https://ci.appveyor.com/project/DrahtBot/bitcoin/build/master.148
756 2018-08-23T20:06:41  <gmaxwell> like you send key type B and then the other side doesn't support B only A.... so what, now you start over?  all doable just fine.
757 2018-08-23T20:07:24  <jonasschnelli> I just figured out we don't keep node stats outside of the connection lifetime...
758 2018-08-23T20:07:30  <jonasschnelli> (except addrman's service bits)
759 2018-08-23T20:07:32  <gmaxwell> But the extra complexity makes me feel like we're not likely to do something like newhope until the next major protocol rework, if it doesn't go in the first ersion.
760 2018-08-23T20:07:54  <jonasschnelli> Yes. It should go into the first, deployed version.
761 2018-08-23T20:07:57  <jonasschnelli> But maybe not in the same PR
762 2018-08-23T20:08:42  <jonasschnelli> Leading to the possibility of incompatible master<->master peers, which I think is fine
763 2018-08-23T20:09:09  <jonasschnelli> (or merge them almost simultaneous)
764 2018-08-23T20:15:06  <gmaxwell> yea, incompatible master/master doesn't worry me too much. If we're worried we could add a hidden option to disable the crypto and default it to off until we're ready.
765 2018-08-23T20:17:31  <wumpus> putting it behind an option initially sounds like a good idea in any case
766 2018-08-23T20:17:52  <jonasschnelli> Yes. Thats done already (-netencryption)
767 2018-08-23T20:17:56  <jonasschnelli> Disabled by default
768 2018-08-23T20:18:03  <wumpus> right
769 2018-08-23T20:20:18  *** profmac has quit IRC
770 2018-08-23T20:21:17  <MarcoFalke> jonasschnelli: Appveyor is a different build system
771 2018-08-23T20:21:29  <MarcoFalke> You'd have to add new files to the cmake file
772 2018-08-23T20:21:40  <MarcoFalke> (Assuming that that is the issue)
773 2018-08-23T20:22:06  <jonasschnelli> So all new files need to be added to the Makefile and to a special CI file?
774 2018-08-23T20:22:29  <wumpus> we need documentation for this appveyor stuff
775 2018-08-23T20:22:44  <jonasschnelli> Sorry,... I missed the whole PR about that
776 2018-08-23T20:22:46  <wumpus> it's a different build system and no one really has an idea how it works
777 2018-08-23T20:22:57  <gmaxwell> what does it do?
778 2018-08-23T20:23:02  <wumpus> well maybe MarcoFalke does :)
779 2018-08-23T20:23:07  *** profmac has joined #bitcoin-core-dev
780 2018-08-23T20:23:09  <MarcoFalke> no, lol
781 2018-08-23T20:23:19  <achow101> gmaxwell: it's for building windows things IIRC
782 2018-08-23T20:23:39  <achow101> building in a windows environment
783 2018-08-23T20:23:43  <jonasschnelli> Do we really need a Visual Studio CI?
784 2018-08-23T20:23:45  <wumpus> otherwise I'm going to ignore it
785 2018-08-23T20:24:00  <wumpus> and merge if only travis passes
786 2018-08-23T20:24:38  <MarcoFalke> I think it is fine to ignore for now because it is experimental
787 2018-08-23T20:25:09  <achow101> MarcoFalke: is there supposed to a button to restart appveyor builds? sf so, i don't see one
788 2018-08-23T20:25:10  *** vexbuy has joined #bitcoin-core-dev
789 2018-08-23T20:25:22  <wumpus> jonasschnelli: I think it's hardly reasonable to expect people submitting PRs to update two build systems
790 2018-08-23T20:25:44  <MarcoFalke> achow101: hmm, yeah there should be a button
791 2018-08-23T20:26:51  <wumpus> especially if we all have no clue how it works
792 2018-08-23T20:27:06  <achow101> presumably ken2812221 knows since he was the one to suggest it?
793 2018-08-23T20:31:14  *** hebasto has joined #bitcoin-core-dev
794 2018-08-23T20:32:48  *** profmac has quit IRC
795 2018-08-23T20:35:24  <jonasschnelli> Maybe it is possible to run the appveyor CI on PRs but not report back (with green-checkmark or red-cross)
796 2018-08-23T20:35:51  *** profmac has joined #bitcoin-core-dev
797 2018-08-23T20:38:21  <jonasschnelli> I'm not going to touch an .vcxproj XML...
798 2018-08-23T20:39:18  *** Victorsueca has quit IRC
799 2018-08-23T20:40:29  *** Victorsueca has joined #bitcoin-core-dev
800 2018-08-23T20:41:49  <wumpus> ya it's a proprietary format too
801 2018-08-23T20:42:32  *** Empact has quit IRC
802 2018-08-23T20:44:45  <luke-jr> smh
803 2018-08-23T20:46:24  *** clarkmoody has quit IRC
804 2018-08-23T20:49:36  *** elichai2 has quit IRC
805 2018-08-23T20:55:43  <ryanofsky> would be nice if appveyor errors on github were only shown as warnings. it is actually pretty straightforward to edit .vcxproj files, though
806 2018-08-23T21:01:41  <wumpus> I'm sure it's learnable, but I think we already ask enough things from contributors
807 2018-08-23T21:03:20  *** phwalkr has joined #bitcoin-core-dev
808 2018-08-23T21:04:20  <wumpus> not that it would hurt to have documentation on how to add files, even with the current build system
809 2018-08-23T21:05:28  *** thib has joined #bitcoin-core-dev
810 2018-08-23T21:11:15  <MarcoFalke> Hmm, anyone knows how to have one of the linux jobs in travis compile with clang?
811 2018-08-23T21:11:36  <MarcoFalke> I installed clang and then set CC=clang and CXX=clang++, but these are simply ignored
812 2018-08-23T21:12:21  <MarcoFalke> All those build systems are hokuspokus to me
813 2018-08-23T21:13:32  <wumpus> $1/configure CC="${CLANGPATH}/bin/clang " CXX="${CLANGPATH}/bin/clang++" OBJCXX="${CLANGPATH}/bin/clang++"
814 2018-08-23T21:13:35  <wumpus> is what I have
815 2018-08-23T21:15:47  <cfields> MarcoFalke: I believe CC and CXX can't be overridden during configure
816 2018-08-23T21:16:13  <wumpus> I've been doing it for years
817 2018-08-23T21:16:14  <luke-jr> cfields: eh, they should be
818 2018-08-23T21:16:27  <cfields> er sorry...
819 2018-08-23T21:16:34  <cfields> MarcoFalke: I believe CC and CXX can't be overridden during configure for depends builds
820 2018-08-23T21:16:44  <cfields> patch coming up
821 2018-08-23T21:17:22  <wumpus> never tried it with depends builds
822 2018-08-23T21:17:34  <MarcoFalke> Hmm, maybe I remember wrongly that it worked with depends builds
823 2018-08-23T21:17:35  <wumpus> so that could well be true
824 2018-08-23T21:17:56  <MarcoFalke> patch welcome, obviously :)
825 2018-08-23T21:20:32  *** meshcollider_ has quit IRC
826 2018-08-23T21:30:58  *** hebasto has quit IRC
827 2018-08-23T21:34:21  *** profmac has quit IRC
828 2018-08-23T21:35:39  *** profmac has joined #bitcoin-core-dev
829 2018-08-23T21:41:31  <cfields> MarcoFalke: https://github.com/theuni/bitcoin/commit/0d00fd5901102d9ca2b99d6f17a3bd96c946e3b7
830 2018-08-23T21:42:21  <cfields> after doing that, need to 'make' in depends again. It should be instant, though.
831 2018-08-23T22:04:05  *** owowo has quit IRC
832 2018-08-23T22:06:46  *** lnostdal has quit IRC
833 2018-08-23T22:19:15  *** Varunram has quit IRC
834 2018-08-23T22:19:27  *** Varunram has joined #bitcoin-core-dev
835 2018-08-23T22:20:06  *** Guyver2 has quit IRC
836 2018-08-23T22:36:07  *** lukedashjr has joined #bitcoin-core-dev
837 2018-08-23T22:38:43  *** luke-jr has quit IRC
838 2018-08-23T22:41:23  *** lukedashjr is now known as luke-jr
839 2018-08-23T23:00:40  *** michaelsdunn1 has quit IRC
840 2018-08-23T23:37:06  *** Zenton has quit IRC
841 2018-08-23T23:59:29  *** profmac has quit IRC