12018-08-23T00:05:53 *** promag has joined #bitcoin-core-dev
22018-08-23T00:19:11 *** Victorsueca has quit IRC
32018-08-23T00:22:35 *** Victorsueca has joined #bitcoin-core-dev
42018-08-23T00:39:01 *** d9b4bef9 has quit IRC
52018-08-23T00:40:14 *** d9b4bef9 has joined #bitcoin-core-dev
62018-08-23T00:59:11 *** AaronvanW has quit IRC
72018-08-23T00:59:24 *** Zenton has joined #bitcoin-core-dev
82018-08-23T00:59:49 *** AaronvanW has joined #bitcoin-core-dev
92018-08-23T01:03:57 *** AaronvanW has quit IRC
102018-08-23T01:17:10 *** Krellan has quit IRC
112018-08-23T01:17:45 *** Krellan has joined #bitcoin-core-dev
122018-08-23T01:21:07 *** peevsie has joined #bitcoin-core-dev
132018-08-23T01:26:05 *** promag has quit IRC
142018-08-23T01:26:59 *** unholymachine has joined #bitcoin-core-dev
152018-08-23T02:03:25 *** Jmabsd has joined #bitcoin-core-dev
162018-08-23T02:04:16 <Jmabsd> sipa, re "uint256's ToString() method", what's the exact code location? it's not in uint256.h/c
172018-08-23T02:04:49 <Jmabsd> ah dear, wait, it's there - "class uint256 : public base_blob<256>", and then base_blob has a toString, obviously.
182018-08-23T02:05:50 <Jmabsd> ToString just calls GetHex, and it's defined as "return HexStr(std::reverse_iterator<.." - there we have it. thx!
192018-08-23T02:12:54 <sipa> exactly.
202018-08-23T02:16:12 *** Jmabsd has quit IRC
212018-08-23T02:22:46 *** Lightsword_ has joined #bitcoin-core-dev
222018-08-23T02:28:58 *** achow101_ has joined #bitcoin-core-dev
232018-08-23T02:29:31 *** rex4539 has quit IRC
242018-08-23T02:29:31 *** [b__b] has quit IRC
252018-08-23T02:29:31 *** dcousens has quit IRC
262018-08-23T02:29:32 *** e4xit has quit IRC
272018-08-23T02:29:33 *** jonasschnelli has quit IRC
282018-08-23T02:29:33 *** sipa has quit IRC
292018-08-23T02:29:33 *** pierre_rochard has quit IRC
302018-08-23T02:29:33 *** Lightsword has quit IRC
312018-08-23T02:29:33 *** achow101 has quit IRC
322018-08-23T02:29:33 *** AdrianG has quit IRC
332018-08-23T02:29:39 *** Lightsword_ is now known as Lightsword
342018-08-23T02:29:58 *** achow101_ is now known as achow101
352018-08-23T02:30:33 *** dcousens has joined #bitcoin-core-dev
362018-08-23T02:36:32 *** Jmabsd has joined #bitcoin-core-dev
372018-08-23T02:37:05 *** keymone has joined #bitcoin-core-dev
382018-08-23T02:41:00 <Jmabsd> oh, the HD wallet stuff uses reverse byte order too: https://github.com/bitcoin/bitcoin/blob/master/src/wallet/rpcwallet.cpp#L3059
392018-08-23T02:43:12 <Jmabsd> and the merkle root is printed in reverse order.
402018-08-23T02:46:34 <gmaxwell> 21:29:12 < gmaxwell> Jmabsd: the point is that bitcoin's UI uses the same stuff to print all these 256 bit hashes.
412018-08-23T02:46:49 <Jmabsd> gmaxwell, yeap and also 160bit.
422018-08-23T02:47:09 <Jmabsd> (this one https://github.com/bitcoin/bitcoin/blob/master/src/wallet/rpcwallet.cpp#L3059 is 160bit)
432018-08-23T02:47:28 <Jmabsd> gmaxwell, is there any 256bit hash that is *not* printed in this order? also,
442018-08-23T02:47:56 <gmaxwell> I don't know what code would do that printing...
452018-08-23T02:47:57 <Jmabsd> i see the stronger, material, original reason for this ordering was to get the zero bits in block hashes have the zeroes in the beginning
462018-08-23T02:48:31 <gmaxwell> 21:30:21 < gmaxwell> block hash is interpreted as a 256-bit little endian number for target comparison as luke says, so then bitcoin needed to print it that way too, otherwise the 0s would have been on the right which would have been confusing... and the same code was then used to print other hashes.
472018-08-23T02:48:43 <Jmabsd> however may there be some more general, common-sense reason too to print hashes in reverse order, or it's all just about symmetry with the block hash representation?
482018-08-23T02:48:45 *** profmac has quit IRC
492018-08-23T02:49:05 <Jmabsd> ahaa, to get a proper ordering, interesting
502018-08-23T02:50:08 <Jmabsd> that makes sense yes. what code in Bitcoin Core compares two uint256:es, which would be used for sorting?
512018-08-23T02:52:30 *** plankers has quit IRC
522018-08-23T02:56:20 <Jmabsd> HEX STRING -----(funny reverse order deserialization)----> uint256 structure which is actually a 32B byte vector ------(sort function, compares the uint256:es by normal < > logic) ----> sorted list ----- (serialize with funny reverse order again) ----> alphanumeric order sorted list
532018-08-23T02:56:27 <Jmabsd> that only makes sense if the < > operator is little endian :)
542018-08-23T02:57:12 <Jmabsd> ahh i see base_blob's < operator and Compare method
552018-08-23T02:59:03 <Jmabsd> the < logics are in int256.h's base_blob, and implemented as "friend inline bool operator<(const base_blob& a, const base_blob& b) { return a.Compare(b) < 0; }", "inline int Compare(const base_blob& other) const { return memcmp(data, other.data, sizeof(data)); }".
562018-08-23T02:59:22 <Jmabsd> memcmp here would react at the first byte that's different. this comparison logics are suitable for a *big endian* representation of a number right -
572018-08-23T02:59:25 <dcousens> Jmabsd at this point, we're simply stuck with it
582018-08-23T02:59:46 <dcousens> You can reason about it in a thousand ways, network byte ordering, etc, but, its merely what was done, and now its stuck
592018-08-23T03:00:09 <Jmabsd> i compare "0x12 0x54 0xfd 0x55" with "0x12 0x54 0xf0 0x55", in this case memcmp will give a negative value due to the third slot's difference
602018-08-23T03:00:58 <Jmabsd> dcousens: yeap i see. i'm trying to make just a bit of sense here, so that, if you make a Dcousens Protocol to do some Bitcoin-related stuff, and you have some structure in your protocol that you hash and then give to people, in what order should it be.
612018-08-23T03:01:16 <Jmabsd> for instance, for symmetry with Bitcoin, if you make a Dcousens Bitcoin Transaction format, then obviously its hash should be printed in reverse order.
622018-08-23T03:01:34 <Jmabsd> (which is some Bitcoin transaction for some purpose)
632018-08-23T03:01:55 <Jmabsd> e.g. Lightning channels certainly represent the channel open/close transactions as txid:s with reverse order, and maybe some other hashes too.
642018-08-23T03:04:07 <Jmabsd> dcousens: so the uint256 < operator is a big endian thing.
652018-08-23T03:04:24 <Jmabsd> wait where's the PoW bits counting done again - i guess it should count the *last* bits (at the last byte positions of the uint256 structure)?
662018-08-23T03:04:35 <Jmabsd> otherwise the PoW counting and the uint256 < operator are not symmetrical
672018-08-23T03:04:38 <luke-jr> it doesn't count bits at all
682018-08-23T03:04:39 <Jmabsd> or, wait
692018-08-23T03:04:40 <Jmabsd> they are
702018-08-23T03:04:51 <Jmabsd> er.
712018-08-23T03:05:08 <Jmabsd> luke-jr: where is the zero counting done again
722018-08-23T03:05:29 <luke-jr> it's not
732018-08-23T03:05:47 <luke-jr> "zero counting" is a simplification for end users
742018-08-23T03:05:56 <luke-jr> the protocol itself simply does a <
752018-08-23T03:07:16 <dcousens> personally, I use the terms `txhash` for the normal byte ordering, and `txid` for reverse byte ordering
762018-08-23T03:07:55 <Jmabsd> luke-jr: https://github.com/bitcoin/bitcoin/blob/master/src/uint256.h#L49 is this the < operator used, or is the logics somewhere else?
772018-08-23T03:08:01 <dcousens> if they need access to the raw data (aka, hashing), they typically deal with the hash, if they are doing lookups, I give the reverse byte order hex string
782018-08-23T03:08:11 <Jmabsd> dcousens: and for blocks you call it what?
792018-08-23T03:08:24 <dcousens> blockhash blockid
802018-08-23T03:08:30 <dcousens> respectively
812018-08-23T03:08:30 *** Victorsueca has quit IRC
822018-08-23T03:08:55 <Jmabsd> dcousens: and if you ever print a 256bit or 160bit hash in a pubkeyscript, then you print it in *byte order* right
832018-08-23T03:09:04 <Jmabsd> dcousens: merkle root "id" versus "hash" too? :-))
842018-08-23T03:09:05 <dcousens> define print
852018-08-23T03:09:18 <Jmabsd> dcousens: "show to a user in a web page or console"
862018-08-23T03:09:26 *** jb55 has quit IRC
872018-08-23T03:09:30 <dcousens> yeah, in normal byte order for that
882018-08-23T03:09:42 <dcousens> again, I only make the distinction for block/tx id's
892018-08-23T03:09:44 *** Victorsueca has joined #bitcoin-core-dev
902018-08-23T03:09:58 <Jmabsd> dcousens: aha. Bitcoin Core would print the merkle root in reverse order. anyhow right.
912018-08-23T03:10:05 <Jmabsd> dcousens: i think your approach makes sense.
922018-08-23T03:11:54 *** peevsie has quit IRC
932018-08-23T03:11:57 <dcousens> and merkleroots*
942018-08-23T03:15:29 *** cyber55 has joined #bitcoin-core-dev
952018-08-23T03:15:31 <Jmabsd> dcousens: you reverse-order your merkle roots too ???
962018-08-23T03:15:59 <Jmabsd> wait, hashes are such are binary/byte concepts so the whole idea of relating to them as comparable numbers, is a concept that Bitcoin adds on top of them, right? the inventor of SHA256 never related to hashing as SHA256(byte string) => 256bit integer which may be little or big endian, right?
972018-08-23T03:18:51 <dcousens> for display
982018-08-23T03:18:53 <dcousens> IIRC
992018-08-23T03:19:12 <dcousens> Bitcoin core reverse orders any 256-bit hex string, basically
1002018-08-23T03:19:14 <Jmabsd> dcousens: aha. so "blockid", "txid" and "merkle root" you do reverse order. any more? HD wallet seeds?
1012018-08-23T03:19:32 <dcousens> I'd say that is all
1022018-08-23T03:19:36 <Jmabsd> (y)
1032018-08-23T03:19:38 *** lone3lf has joined #bitcoin-core-dev
1042018-08-23T03:19:50 <dcousens> "blockid", "txid", and merkle roots... if I remember correctly.
1052018-08-23T03:20:23 <dcousens> Typically, I refer to it as "merkleRootId"
1062018-08-23T03:21:06 <achow101> Jmabsd: Bitcoin Core prints everything that is a byte array in reverse byte order
1072018-08-23T03:21:15 <achow101> Jmabsd: except for byte arrays in scripts. those are printed as-is
1082018-08-23T03:21:35 <achow101> literally everything else (block hash, merkle root, txid, tx hash, whatever) is reverse byte order
1092018-08-23T03:21:48 <Jmabsd> achow101: if you print a transaction's body, that's in normal byte order no ??
1102018-08-23T03:22:18 <Jmabsd> sorry for being spammy about this byte order question, in a way it's trivial however as soon as a user is trying to use this, if the ordering is wrong then things break.
1112018-08-23T03:22:19 <achow101> Jmabsd: a transaction is an object, not a byte array
1122018-08-23T03:22:36 <dcousens> achow101 except private keys
1132018-08-23T03:22:48 <Jmabsd> achow101: right so Bitcoin Core defines both 256bit and 160bit hashes as subclasses of "base_blob", is this what you mean by byte array?
1142018-08-23T03:22:55 <achow101> Jmabsd: yes
1152018-08-23T03:23:08 <Jmabsd> achow101: is anything else than 256bit and 160bit hashes represented as "base_blob"??
1162018-08-23T03:23:13 <achow101> no
1172018-08-23T03:23:35 <Jmabsd> achow101: right. (what you say i get confirmed by "grep -r base_blob *" too - it's only mentioned in uint256.*.)
1182018-08-23T03:24:16 <achow101> dcousens: private keys are printed as WIF :)
1192018-08-23T03:24:16 <Jmabsd> just for reference, where is the comparison of PoW, that is against the difficulty target, and in a reorg?
1202018-08-23T03:24:18 *** lone3lf has quit IRC
1212018-08-23T03:24:25 <Jmabsd> that should be a "<" operator on the uint256_t type no?
1222018-08-23T03:24:36 <achow101> Jmabsd: src/pow.*
1232018-08-23T03:24:48 <dcousens> achow101: my point was, in terms of a community standard
1242018-08-23T03:25:24 <achow101> Jmabsd: https://github.com/bitcoin/bitcoin/blob/master/src/pow.cpp#L87
1252018-08-23T03:25:34 <dcousens> the standard is reverse byte order for blockid, txid, merklerootid ... I don't think there is anything else you would want to adhere to in reverse byte order.
1262018-08-23T03:25:50 *** lone3lf has joined #bitcoin-core-dev
1272018-08-23T03:26:12 <dcousens> And therefore, even when displaying, I would _only_ do reverse byte order for those things, e.g, if you are printing a private key, you wouldn't print it in reveres byte order
1282018-08-23T03:26:34 <achow101> dcousens: it's actually anything that can be interpreted as an integer. e.g. version numbers are little endian which is a form of "reverse byte order"
1292018-08-23T03:26:53 <dcousens> I don't think I've seen the locktime printed as reverse byte
1302018-08-23T03:27:04 <dcousens> But maybe!
1312018-08-23T03:27:17 <achow101> dcousens: is it not reversed from whatever it is stored as?
1322018-08-23T03:27:59 <dcousens> it is stored in LE
1332018-08-23T03:28:37 <dcousens> so not reversed afaik
1342018-08-23T03:28:38 <achow101> right, so it is printed as a big endian decimal which is reverse byte order of little endian
1352018-08-23T03:29:17 <dcousens> ha
1362018-08-23T03:29:22 <dcousens> yeah I suppose
1372018-08-23T03:29:38 <dcousens> but if I saw it in hex
1382018-08-23T03:29:43 <dcousens> I'd probably assume LE, not BE
1392018-08-23T03:29:56 <dcousens> unless I saw `0x...` haha, then I'd assume BE
1402018-08-23T03:29:58 <dcousens> what a mess
1412018-08-23T03:31:43 *** justanotheruser has quit IRC
1422018-08-23T03:36:46 *** rls has quit IRC
1432018-08-23T03:46:23 *** phwalkr has joined #bitcoin-core-dev
1442018-08-23T04:06:56 <Jmabsd> wait, for mainnet, configuration includes: consensus.powLimit = uint256S("00000000ffffffffffffffffffffffffffffffffffffffffffffffffffffffff");
1452018-08-23T04:07:05 <Jmabsd> this means that a single proof of work may not.. what?
1462018-08-23T04:09:15 *** fanquake has joined #bitcoin-core-dev
1472018-08-23T04:09:52 <fanquake> cfields are you doing detached sigs now?
1482018-08-23T04:13:40 <Jmabsd> dcousens,achow101: facepalm lol.
1492018-08-23T04:14:38 <Jmabsd> dcousens,achow101: i like dcousens' way of relating to it that only blockid, transactionid and merklerootid should be reverse byte order, that limits the possible damage from ambiguous order conventions too ha.
1502018-08-23T04:17:04 *** lone3lf has quit IRC
1512018-08-23T04:17:52 <Jmabsd> achow101: the "nBits" value, which is copied from the block header and is applied in PoW here https://github.com/bitcoin/bitcoin/blob/master/src/pow.cpp#L80 : "arith_uint256 bnTarget;bnTarget.SetCompact(nBits, &fNegative, &fOverflow);" , it's in the Bitcoin-internal 32bit floating point format right??
1522018-08-23T04:18:10 *** fanquake has quit IRC
1532018-08-23T04:18:14 <achow101> Jmabsd: yes
1542018-08-23T04:18:37 <Jmabsd> achow101: actually what code generates that bits value??
1552018-08-23T04:19:08 <Jmabsd> achow101: at the genesis block, the bits value is *very low* and then it's getting *higher and higher*, slowly approaching 2^256 right?
1562018-08-23T04:19:39 *** vexbuy_ has quit IRC
1572018-08-23T04:19:41 <Jmabsd> achow101: each PoW generated out there, has the block hash proof of work 256bit-integer-compared with this target value right?
1582018-08-23T04:19:56 *** lone3lf has joined #bitcoin-core-dev
1592018-08-23T04:20:25 <gmaxwell> it goes up and down, whatever is required to keep the block rate on target.
1602018-08-23T04:20:44 <Jmabsd> so, Bitcoin Core does "if (UintToArith256(hash) > bnTarget)" - so you're checking that the block hash is "more work" than the target value, and meanwhile LibBitcoin does "to_uint256(hash()) <= target;", https://github.com/libbitcoin/libbitcoin/blob/master/src/chain/header.cpp#L464
1612018-08-23T04:21:02 <Jmabsd> so LibBitcoin is checking that... err.. the target is.. BIGGER than the PoW?
1622018-08-23T04:21:11 <Jmabsd> gmaxwell ah yes.
1632018-08-23T04:21:52 <Jmabsd> LibBitcoin comments "Ensure actual work is at least claimed amount (smaller is more work)." - i wonder how they got this inverse definition of work, he. thoughts?
1642018-08-23T04:22:38 <achow101> Jmabsd: Bitcoin Core checks that if the block hash is bigger than the target, it should fail, hence the return false that follows
1652018-08-23T04:22:54 <Jmabsd> Ah. yes.
1662018-08-23T04:23:01 <achow101> libbitcoin checks that the hash is less than the target, and if it is, it continues
1672018-08-23T04:23:03 <achow101> they mean the same thing
1682018-08-23T04:23:12 <Jmabsd> achow101: noted. and... what's the point here?
1692018-08-23T04:23:27 <gmaxwell> I've been asking myself that.
1702018-08-23T04:23:30 <Jmabsd> you want the work to be SMALLER than the target? so the "bits" value is going to zero??
1712018-08-23T04:23:50 <Jmabsd> so zero means 256 leading zero bits to qualify? :-}
1722018-08-23T04:24:12 <achow101> Jmabsd: yes. the hash should be less than the target
1732018-08-23T04:24:39 <Jmabsd> achow101: this makes sense if "leading zero bits" means "the most significant bits in the block hash viewed as a 256bit unsigned integer" -
1742018-08-23T04:24:46 <Jmabsd> then obviously zero bits means a SMALLER integer
1752018-08-23T04:25:02 <Jmabsd> ok interesting, this is a very curious piece of endianness juggling, ha.
1762018-08-23T04:25:16 <achow101> Jmabsd: the whole target checking works like this:
1772018-08-23T04:25:30 <achow101> interpret the hash (as output by the hash function) as a 256 bit little endian integer
1782018-08-23T04:26:07 <achow101> take nBits, decompress it and interpret it as a 256 bit little endian integer
1792018-08-23T04:26:31 <achow101> if the hash is less than or equal to the decompressed nBits, then it is valid.
1802018-08-23T04:27:07 <achow101> The whole reverse byte order thing is to get "256 bit little endian integer" into "256 big endian integer" because humans are used to looking at things in big endian
1812018-08-23T04:27:21 <Jmabsd> right. (and, nBits is a custom value format so "little endian integer" here doesn't make sense as such. point is just, "make a 256bit integer type so you can compare it, and don't mess up!" he.)
1822018-08-23T04:27:33 <achow101> yep
1832018-08-23T04:28:13 <Jmabsd> achow101: and then for sorting blockid/txid/merklerootid:s, you want the internal 256bit integer sorting to correspond to sorting the string serialization in big endian representation form lol
1842018-08-23T04:29:05 <gmaxwell> there isn't any normative sorting in the protocol.
1852018-08-23T04:30:02 *** d9b4bef9 has quit IRC
1862018-08-23T04:30:36 <Jmabsd> mhm.
1872018-08-23T04:30:55 <Jmabsd> so we can only understand that transaction id:s and merkle root id:s (and in Bitcoin Core's case also hd wallet seeds) got this reverse byte order,
1882018-08-23T04:31:00 *** vexbuy has joined #bitcoin-core-dev
1892018-08-23T04:31:04 <Jmabsd> was just a coincidence of how Satoshi happened to write the code
1902018-08-23T04:31:17 *** d9b4bef9 has joined #bitcoin-core-dev
1912018-08-23T04:31:25 <gmaxwell> they didn't "get" any reverse order.
1922018-08-23T04:31:28 <Jmabsd> at least we can't see more sense in it today.
1932018-08-23T04:31:45 <Jmabsd> gmaxwell, well in the human readable string representation, they do, right
1942018-08-23T04:31:46 <gmaxwell> everywhere inside the system there isn't any reversing. It's just printed that way.
1952018-08-23T04:31:56 <gmaxwell> yes, as was explained to you a day ago.
1962018-08-23T04:32:27 <gmaxwell> Because blockhashes make sense to print in that order, every other 'big number' just used the same printing code.
1972018-08-23T04:32:31 <Jmabsd> yeap exactly - the convention of printing transaction id:s and merkle root id:s in reverse order, we can only understand it to be a coincidental design choice that originated from printing block id:s in reverse order, which does have sense in that you get the zeroes in the beginning that way, which is educative.
1982018-08-23T04:33:14 <Jmabsd> exactly. ok i think i got it now. regarding dcousens' terminology of "block id" vs "block hash", "tx id" vs "tx hash", and "merkle root id" vs "merkle root hash", is this a widely accepted convention, or is it just his thing??
1992018-08-23T04:33:27 <Jmabsd> where "id" would mean reverse order and "hash" would be normal order
2002018-08-23T04:34:52 <gmaxwell> I'm not familar with it.
2012018-08-23T04:35:14 <gmaxwell> Really, if byte orders are tripping you up you're going to be really confused by every other detail of the system. :)
2022018-08-23T04:35:47 <Jmabsd> lolol. no i just look at this with particular attention as any user who would get his data trashed would get really confused, ha
2032018-08-23T04:36:00 <Jmabsd> i mean, say someone makes a program asking a user to pass a "block hash" as an argument
2042018-08-23T04:36:10 <Jmabsd> and the guy gives it in reverse order, and then the program says "no such block exists, bye"
2052018-08-23T04:36:38 <Jmabsd> that kind of consequental malbehavior
2062018-08-23T04:36:50 <Jmabsd> what about "id" versus "hash", is this an established convention re block/tx/merkleroot?
2072018-08-23T04:36:50 <achow101> Jmabsd: fun fact, bitcoin core does exactly that
2082018-08-23T04:36:57 <Jmabsd> achow101: :o where?
2092018-08-23T04:37:11 <achow101> if you give it the actual block hash, it will not find the block
2102018-08-23T04:37:26 <gmaxwell> achow just means that the input is in the same order as the output.
2112018-08-23T04:37:43 <gmaxwell> 21:36:50 < Jmabsd> what about "id" versus "hash", > 21:34:51 < gmaxwell> I'm not familar with it.
2122018-08-23T04:38:03 <Jmabsd> achow101: derp derp. exactly.
2132018-08-23T04:38:10 <Jmabsd> > achow just means that the input is in the same order as the output.
2142018-08-23T04:38:12 <Jmabsd> what do you mean?
2152018-08-23T04:38:45 <gmaxwell> what was was trying to say is that the order bytes are in are the most trivial of 101 things you have to get right for software to work. I don't really spend any time thinking about it. Sometimes I'll do something and it won't work because I needed to reorder something, so I do, and I move on.
2162018-08-23T04:38:47 <achow101> Jmabsd: you give it the block hash in the way that it is printed, aka reverse byte order. if you give it how it actually is, it will not find the block
2172018-08-23T04:39:53 <Jmabsd> achow101: exactly. so all over the whole Bitcoin ecosystem, those three values (block HASH, transaction HASH and merkle root HASH aka ID whatever), when you're dealing with hex representation, it must ALWAYS be in reverse order bc otherwise you'll screw up
2182018-08-23T04:39:55 <gmaxwell> achow101: there is no spoon^w"how it is". I mean, it "is" electrical charges, but you enter it via moving buttons. Everything we deal with is an arbritary representation.
2192018-08-23T04:40:14 <Jmabsd> so if you have a tool / app / web site whatever that tells you which block, then you give it in reverse order
2202018-08-23T04:40:25 <Jmabsd> if you have a CLI tool / app / whatever that asks which block or tx, then you input it in reverse order, etc.
2212018-08-23T04:40:34 <achow101> Jmabsd: yes
2222018-08-23T04:40:53 <Jmabsd> however aside from block hashes, transaction hashes and merkle roots, you're free to use the normal order.
2232018-08-23T04:41:08 <achow101> Jmabsd: no
2242018-08-23T04:41:14 <Jmabsd> achow101: no? :o
2252018-08-23T04:41:27 <achow101> Jmabsd: those still use reverse byte order
2262018-08-23T04:41:33 <Jmabsd> yea that's what i said:
2272018-08-23T04:41:48 <achow101> oh, I misread, nvm
2282018-08-23T04:42:05 <Jmabsd> achow101: if anywhere in this world you show a user a block hash, tx hash or merkle root hash, or ask him for it, then you do it in reverse order - but, for other values, such as for instance if you make a HD wallet and print him a hex dump of a seed, or,
2292018-08-23T04:42:17 <gmaxwell> hash160s are probably also in that order in the interface, if any show up in the interface anywhere.
2302018-08-23T04:42:20 <Jmabsd> some other hash, even if it relates to the bitcoin protocol, then normal order is fine
2312018-08-23T04:42:33 <Jmabsd> for instance, the 32B and 20B hashes in pubkeyscripts! those would be normal order
2322018-08-23T04:42:45 <achow101> gmaxwell: it depends on where the hash160 is, apparently
2332018-08-23T04:42:53 <gmaxwell> pubkeyscripts is not an arith/hash_n, it's just bytes.
2342018-08-23T04:42:54 <Jmabsd> gmaxwell: exactly, Bitcoin Core's HDwallet seeds are 160bit and presented in reverse order, yes
2352018-08-23T04:43:12 *** justanotheruser has joined #bitcoin-core-dev
2362018-08-23T04:43:15 <gmaxwell> in any case, trying to come up with a greater theme is probably a waste of time. You do whatever works.
2372018-08-23T04:43:16 <achow101> if it is being output in, e.g. getaddressinfo it will be in reverse byte order. but in decodescript, it is not
2382018-08-23T04:43:30 <gmaxwell> There is a historical reason that it works the way it does, sure, but all that matters is that you do what works.
2392018-08-23T04:43:44 <Jmabsd> achow101,gmaxwell: maybe to help the world a bit, it's better actually to talk about "block id", "transaction id" and "merkle root id", and by "id" you mean "reverse order", even though it's not universally established nomenclature - tihs was a nice suggestion by dcousens.
2402018-08-23T04:44:21 <gmaxwell> if you want to predict what something will be (why?) then I believe "is it stored as a big uint type in the system" is the necessary and sufficient predictor.
2412018-08-23T04:44:47 <gmaxwell> I think if you're spening enough time thinking about this that you need names for it, you're probably doomed to never do anything useful.
2422018-08-23T04:44:57 <Jmabsd> achow101: actually when do you ever show a user the merkle root ID...
2432018-08-23T04:45:03 <Jmabsd> the merkle root 256bit value is soo internal
2442018-08-23T04:45:09 <Jmabsd> you never compare it with anything else too
2452018-08-23T04:45:12 <achow101> Jmabsd: in getblock
2462018-08-23T04:45:22 <gmaxwell> I can tell you that it's possible to work for years on the system and never think about any of this for more than a moment.
2472018-08-23T04:45:26 <Jmabsd> achow101: you make a lookup of a block based on its merkle root or what :o
2482018-08-23T04:45:42 <achow101> Jmabsd: nope, just as output for a decoded block
2492018-08-23T04:45:54 <Jmabsd> right. and it's reverse order. derp derp. yeap.
2502018-08-23T04:45:58 <gmaxwell> and I've seen from years in this channel that people that come in asking lots of questions about byte order almost all just get frustrated and give up.
2512018-08-23T04:46:30 <Jmabsd> okok i got it. anyhow all clear now. thx.
2522018-08-23T04:47:07 <gmaxwell> the order it prints big uints is arbritary, but no more or less arbritary than the decision to print them in hex instead of base2 or base64 or something else.
2532018-08-23T04:47:47 <gmaxwell> the order made sense for one thing, and then everything else just did the same because they share printing code and the order was irrelevant for everything else.
2542018-08-23T04:48:22 <Jmabsd> achow101: if you ever printed a merkle *node* to a user, would you do it in reverse order, i guess *not*.
2552018-08-23T04:48:37 <gmaxwell> of course.
2562018-08-23T04:49:06 <gmaxwell> if its represented as a uint in the software, it gets printed in that order.
2572018-08-23T04:49:28 <gmaxwell> so it would be 'revese' if you mean printing an interior node hash.
2582018-08-23T04:49:35 <gmaxwell> er. reverse.
2592018-08-23T04:50:34 <achow101> Jmabsd: it would be in reverse because I would use a uint256 and that prints in reverse :p
2602018-08-23T04:50:46 <achow101> (in a modification of Bitcoin Core)
2612018-08-23T04:50:56 <gmaxwell> there are already a zillion random fields that show up in the UI in reverse order that litterally no one ever thought about. They just printed a uint and thats how the uints print method works.
2622018-08-23T04:51:07 <gmaxwell> So, for example, there is something that prints a hash of the utxo set.
2632018-08-23T04:51:16 <gmaxwell> that was added last year.
2642018-08-23T04:51:45 <gmaxwell> Without looking I'm sure it's in "reverse" order. No one ever even thought about it. Because the order is irrelevant, so they just get the default behavior.
2652018-08-23T04:52:09 <Jmabsd> gmaxwell, a hash of all utxo:s in the blockchain at a given block height?
2662018-08-23T04:52:30 <gmaxwell> at the current height.
2672018-08-23T04:52:45 <gmaxwell> it's useful for checking for corruption in the utxo set.
2682018-08-23T04:52:52 <Jmabsd> derp derp, ok Bitcoin Core gonna kick in more reverse order hex conventions in the ecosystem over time, he he. however users won't care really about those.
2692018-08-23T04:52:59 <Jmabsd> aha yep.
2702018-08-23T04:53:42 <Jmabsd> one could call it "utxo set id" to subtly emphasise that it's reverse order.
2712018-08-23T04:53:54 <gmaxwell> or, one could not care because it's a total waste of time.
2722018-08-23T04:54:13 <gmaxwell> no one ever had to think about it at all.
2732018-08-23T04:54:42 <gmaxwell> if you get too busy thinking about minutia you'll probably find your mind becomes too crowded to think about more complicated things. :)
2742018-08-23T05:02:53 *** profmac has joined #bitcoin-core-dev
2752018-08-23T05:13:20 <gmaxwell> achow101: https://bitcoin.stackexchange.com/questions/78292/insanely-high-fee lol I wonder if we should make it _harder_ to bypass that...
2762018-08-23T05:13:39 <gmaxwell> I wonder how many people hit that and then frantically work around it without realizing that its saving them from doom.
2772018-08-23T05:14:54 <gmaxwell> like maybe to bypass it the node should present you with a randomly generated nonogram puzzle. :P
2782018-08-23T05:15:55 <luke-jr> insanely high fee isn't enforced as a relay policy, is it? O.o
2792018-08-23T05:16:23 <gmaxwell> no.
2802018-08-23T05:16:39 <gmaxwell> it's just enforced at the user interfaces.
2812018-08-23T05:16:43 <gmaxwell> and its easy to bypass.
2822018-08-23T05:17:06 <gmaxwell> but I've several times seen people asking about how they could bypass it and it turns out it was saving them.
2832018-08-23T05:17:10 <gmaxwell> (as is the case in that link)
2842018-08-23T05:30:00 *** Krellan has quit IRC
2852018-08-23T05:30:52 *** Krellan has joined #bitcoin-core-dev
2862018-08-23T06:13:54 *** plankers has joined #bitcoin-core-dev
2872018-08-23T06:16:03 *** vexbuy has quit IRC
2882018-08-23T06:35:05 *** schmidty has quit IRC
2892018-08-23T06:40:27 *** vexbuy has joined #bitcoin-core-dev
2902018-08-23T06:51:49 *** jonasschnelli_ has joined #bitcoin-core-dev
2912018-08-23T06:52:02 *** jonasschnelli_ is now known as jonasschnelli
2922018-08-23T06:53:33 *** Jmabsd has quit IRC
2932018-08-23T06:56:30 *** vexbuy has quit IRC
2942018-08-23T06:57:53 <jonasschnelli> cfields: I wonder why our OSX SDKs have a different hashs: https://github.com/bitcoin-core/gitian.sigs/commit/2ed8b5fba57623c0160eebc0c77ee4f9f2fe041a#diff-4bb2ae50ac744a39b2feaf294eb21d8eR9. https://github.com/bitcoin-core/gitian.sigs/pull/781/files#diff-0016af707809da5b52923d223d1943fcR9
2952018-08-23T06:58:00 <jonasschnelli> It's not a problem,... just wondering
2962018-08-23T07:05:36 *** [b__b] has joined #bitcoin-core-dev
2972018-08-23T07:06:18 <provoostenator> They're all different? https://github.com/bitcoin-core/gitian.sigs/pull/780/files#diff-32221b5f40db6596d8213cb93b29c705R9
2982018-08-23T07:07:53 <provoostenator> The SDK file isn't distributed, right?
2992018-08-23T07:08:58 <jonasschnelli> looks like... I though we have all shared the same tarball? But maybe not
3002018-08-23T07:13:55 <provoostenator> I used OSX to extract the relevant stuff from the thing I downloaded from Apple. On Linux you have to use different tools. Or maybe it's a timestamp thing?
3012018-08-23T07:15:43 <provoostenator> I've had the same hash since 0.16.0rc4 it seems, and I've used at least two distinct gitian machines.
3022018-08-23T07:22:42 *** phwalkr has quit IRC
3032018-08-23T07:35:18 *** Victorsueca has quit IRC
3042018-08-23T07:36:07 *** Krellan has quit IRC
3052018-08-23T07:36:29 *** Victorsueca has joined #bitcoin-core-dev
3062018-08-23T07:36:53 *** Krellan has joined #bitcoin-core-dev
3072018-08-23T07:36:57 *** vexbuy has joined #bitcoin-core-dev
3082018-08-23T07:37:07 *** csknk has joined #bitcoin-core-dev
3092018-08-23T07:41:53 *** vexbuy has quit IRC
3102018-08-23T07:42:33 <luke-jr> jonasschnelli: the tarball isn't deterministically made..
3112018-08-23T07:42:49 <luke-jr> IMO it's a good thing that we have obviously-different SDKs ;)
3122018-08-23T07:42:56 <jonasschnelli> Yes. Indeed
3132018-08-23T07:50:31 *** vexbuy has joined #bitcoin-core-dev
3142018-08-23T08:20:31 *** elichai2 has joined #bitcoin-core-dev
3152018-08-23T08:32:20 *** gnappuraz has joined #bitcoin-core-dev
3162018-08-23T08:44:08 <gnappuraz> Hi there, I was wondering if there is any documentation relative to the different archives declared in the main Makefile. Cause I don't really understand the logic of how stuff is organized. i.e. what is the difference between libbitcoin_util and libbitcoin_common?
3172018-08-23T08:49:07 <jonasschnelli> gnappuraz: these are semi-independent modules. Those modules get linked for the different binaries we have (bitcoin-tx, bitcoin-cli, bitcoind, bitcoin-qt).
3182018-08-23T08:49:41 <jonasschnelli> But they do not have an ABI or are meant to be used with other applications
3192018-08-23T08:59:27 *** gnappuraz has quit IRC
3202018-08-23T08:59:54 *** ossifrage has quit IRC
3212018-08-23T09:06:25 *** ossifrage has joined #bitcoin-core-dev
3222018-08-23T09:08:33 *** gnappuraz has joined #bitcoin-core-dev
3232018-08-23T09:08:46 <ossifrage> g++ is really annoying sometimes, a 'make -j4' just managed to OOM this crappy box
3242018-08-23T09:11:42 <gnappuraz> thx for the answer. But let's assume that I have to add a new file that if meant to be use by both bitcoind and bitcoin-qt, the rationale behind putting it in COMMON vs UTIL should be: if used by bitcoin-cli then UTIL, otherwise COMMON (since bitcoin-cli links UTIL but not COMMON)?
3252018-08-23T09:23:01 *** GAit has joined #bitcoin-core-dev
3262018-08-23T09:24:07 *** AaronvanW has joined #bitcoin-core-dev
3272018-08-23T09:24:34 *** AaronvanW has quit IRC
3282018-08-23T09:26:33 *** AaronvanW has joined #bitcoin-core-dev
3292018-08-23T10:04:48 *** e4xit has joined #bitcoin-core-dev
3302018-08-23T10:06:26 <Empact> Is it possible to extend the ability to restart appveyor builds to core devs?
3312018-08-23T10:08:37 *** SopaXorzTaker has joined #bitcoin-core-dev
3322018-08-23T10:14:53 <ken2812221_> MarcoFalke: You can checkout https://ci.appveyor.com/gitHubTeams and https://ci.appveyor.com/project/Drahtbot/bitcoin/settings/permissions to help you setting up appveyor permissions.
3332018-08-23T10:16:43 *** ken2812221_ is now known as ken2812221
3342018-08-23T10:31:32 <Empact> ken2812221: Thanks - set up the former but don't have access to the latter.
3352018-08-23T10:37:51 *** plankers has quit IRC
3362018-08-23T10:39:36 *** vexbuy_ has joined #bitcoin-core-dev
3372018-08-23T10:43:06 *** vexbuy has quit IRC
3382018-08-23T10:44:52 <ken2812221> Empact: I was talking to MarcoFalke, he's the owner of appveyor ci account.
3392018-08-23T10:47:18 *** Empact has quit IRC
3402018-08-23T11:10:21 *** gnappuraz has quit IRC
3412018-08-23T11:16:35 *** vexbuy has joined #bitcoin-core-dev
3422018-08-23T11:20:18 *** vexbuy_ has quit IRC
3432018-08-23T11:24:15 *** vexbuy has quit IRC
3442018-08-23T11:33:07 *** promag has joined #bitcoin-core-dev
3452018-08-23T11:37:03 *** promag has quit IRC
3462018-08-23T11:39:31 *** _flow_ has joined #bitcoin-core-dev
3472018-08-23T11:41:39 <_flow_> What can I do to get https://github.com/bitcoin/bitcoin/pull/13621 merged? It improves the, currently terriable, configuration mechanics of bitcoind a bit (while still not perfect) and re-enables a recently disabled test.
3482018-08-23T11:43:03 <wumpus> we're really careful merging changes to the order of configuration file parsing, slight changes to precedence will mess up things for current users and there are no good tests for this
3492018-08-23T11:43:06 *** lnostdal has joined #bitcoin-core-dev
3502018-08-23T11:43:18 <jonasschnelli> _flow_: it needs more developers weighting in on the importance of that PR
3512018-08-23T11:43:21 <wumpus> _flow_: so I suppose my answer is "add good tests"!
3522018-08-23T11:44:11 <wumpus> those would also help to show what is improved, as they would fail on the old code and pass on the new one
3532018-08-23T11:45:09 <wumpus> also: does this affect the GUI
3542018-08-23T11:45:17 <wumpus> I see you did update the existing test, good
3552018-08-23T11:45:56 <wumpus> but I think it needs to extended, to include the failure cases that you describe
3562018-08-23T11:49:33 *** Gnappuraz has joined #bitcoin-core-dev
3572018-08-23T11:49:36 *** vexbuy has joined #bitcoin-core-dev
3582018-08-23T11:54:01 *** cyber55 has quit IRC
3592018-08-23T11:55:36 *** belcher_ has joined #bitcoin-core-dev
3602018-08-23T12:01:09 <_flow_> wumpus: jonasschnelli: thanks. I'll consider adding a test for the issue the user in https://github.com/bitcoin/bitcoin/pull/12255#issuecomment-403666092 is experiencing
3612018-08-23T12:02:22 *** Guyver2 has joined #bitcoin-core-dev
3622018-08-23T12:22:30 *** rafalcpp_ has quit IRC
3632018-08-23T12:34:20 *** vexbuy_ has joined #bitcoin-core-dev
3642018-08-23T12:38:35 *** vexbuy has quit IRC
3652018-08-23T12:57:39 *** pergaminho has joined #bitcoin-core-dev
3662018-08-23T13:09:53 *** belcher_ has quit IRC
3672018-08-23T13:23:36 *** jcorgan has quit IRC
3682018-08-23T13:23:58 *** rafalcpp has joined #bitcoin-core-dev
3692018-08-23T13:26:29 *** Victorsueca has quit IRC
3702018-08-23T13:27:44 *** Victorsueca has joined #bitcoin-core-dev
3712018-08-23T13:30:59 *** vexbuy_ has quit IRC
3722018-08-23T13:46:48 *** vexbuy has joined #bitcoin-core-dev
3732018-08-23T13:54:16 *** Emcy_ has quit IRC
3742018-08-23T13:56:39 <jonasschnelli> how do I compile to detect pass-the-end iterator violations? I once had this setup but can no longer reproduce it.
3752018-08-23T13:56:46 <jonasschnelli> (ideally gcc/debian)
3762018-08-23T13:58:18 *** Krellan has quit IRC
3772018-08-23T13:58:27 <jonasschnelli> Oh.. is it maybe DEBUG=1 for the depends build?
3782018-08-23T13:59:01 *** Krellan has joined #bitcoin-core-dev
3792018-08-23T14:08:32 *** Rootsudo has joined #bitcoin-core-dev
3802018-08-23T14:11:51 *** csknk has quit IRC
3812018-08-23T14:12:48 *** wxss has joined #bitcoin-core-dev
3822018-08-23T14:38:53 *** petertod1 is now known as petertodd
3832018-08-23T14:40:26 *** d_t has joined #bitcoin-core-dev
3842018-08-23T14:41:06 <achow101> gmaxwell: luke-jr: there's an additional "absurdly high fee" that is enforced as a relay policy. it's triggered by a tx with a fee higher than 0.01 BTC (or something like that)
3852018-08-23T14:42:51 <achow101> gmaxwell: also, insanely high fee is something that I can only find in 0.8 and earlier. I think that guy is making his own altcoin from 0.8.6 as people tend to do
3862018-08-23T14:47:38 <treyzania> why do people fork from that far back?
3872018-08-23T15:01:35 *** Giszmo has quit IRC
3882018-08-23T15:04:00 *** michaelsdunn1 has joined #bitcoin-core-dev
3892018-08-23T15:17:52 *** SopaXorzTaker has quit IRC
3902018-08-23T15:24:36 *** pergaminho has quit IRC
3912018-08-23T15:31:52 *** pergaminho has joined #bitcoin-core-dev
3922018-08-23T15:39:40 *** wxss has quit IRC
3932018-08-23T15:41:11 <cfields> fanquake: I was waiting to make sure that my results matched jonasschnelli's before signing
3942018-08-23T15:41:41 <cfields> jonasschnelli: yes, it's DEBUG=1 for depends
3952018-08-23T15:42:09 <cfields> jonasschnelli: for sigs, note that this is the first detached sig in a new release series. So we'll switch to a 0.17 branch to match.
3962018-08-23T15:42:46 <jonasschnelli> cfields: ack. I'll PR after you pushed your commit (if you havent)
3972018-08-23T15:43:42 <cfields> jonasschnelli: pushed
3982018-08-23T15:47:32 <cfields> jonasschnelli: for future reference, I use "--orphan" when creating new git branches in the detached sigs repo, to avoid sharing any history between them.
3992018-08-23T15:48:02 *** Emcy_ has joined #bitcoin-core-dev
4002018-08-23T15:48:08 *** profmac has quit IRC
4012018-08-23T15:48:24 <jonasschnelli> cfields: noted
4022018-08-23T15:48:44 *** profmac has joined #bitcoin-core-dev
4032018-08-23T15:56:31 *** Giszmo has joined #bitcoin-core-dev
4042018-08-23T16:01:49 <gmaxwell> achow101: no there isn't. re: relay policy.
4052018-08-23T16:02:22 <gmaxwell> achow101: you're probably confused because the check is in ATMP, but it's always called with it disabled when handlign a transaction from the network.
4062018-08-23T16:09:33 *** Gnappuraz has quit IRC
4072018-08-23T16:15:48 *** Rootsudo has quit IRC
4082018-08-23T16:16:29 *** Rootsudo has joined #bitcoin-core-dev
4092018-08-23T16:17:02 *** Rootsudo has joined #bitcoin-core-dev
4102018-08-23T16:17:21 *** Rootsudo has quit IRC
4112018-08-23T16:18:44 *** Rootsudo has joined #bitcoin-core-dev
4122018-08-23T16:19:39 *** Rootsudo has joined #bitcoin-core-dev
4132018-08-23T16:40:25 *** sipa has joined #bitcoin-core-dev
4142018-08-23T17:10:30 *** leishman has joined #bitcoin-core-dev
4152018-08-23T17:36:45 *** Emcy has joined #bitcoin-core-dev
4162018-08-23T17:37:12 *** Emcy_ has quit IRC
4172018-08-23T17:39:03 *** profmac has quit IRC
4182018-08-23T17:48:23 *** rex4539 has joined #bitcoin-core-dev
4192018-08-23T17:50:24 *** profmac has joined #bitcoin-core-dev
4202018-08-23T17:50:59 *** vexbuy has quit IRC
4212018-08-23T18:00:05 *** d_t has quit IRC
4222018-08-23T18:00:23 *** Empact has joined #bitcoin-core-dev
4232018-08-23T18:00:57 *** Krellan has quit IRC
4242018-08-23T18:01:59 *** pergaminho has quit IRC
4252018-08-23T18:02:08 *** Emcy has quit IRC
4262018-08-23T18:06:42 *** Rootsudo has joined #bitcoin-core-dev
4272018-08-23T18:07:29 <wumpus> what's up with travis, btw? it looks like it's failing on all PRs
4282018-08-23T18:09:21 <sipa> feature_block fails?
4292018-08-23T18:09:30 <wumpus> there's something wrong with this new appveyor build
4302018-08-23T18:09:47 <sipa> oh, it's appveyor that fails, not travis
4312018-08-23T18:09:47 <wumpus> "Error making request with Error Code ServiceUnavailable and Http Status Code ServiceUnavailable. No further error information was returned by the service."
4322018-08-23T18:10:31 *** SopaXorzTaker has joined #bitcoin-core-dev
4332018-08-23T18:10:46 <wumpus> I've also seen feature_block fail though, but not consistently
4342018-08-23T18:11:38 <wumpus> on the very-last test (the reorg)
4352018-08-23T18:26:01 *** leishman has quit IRC
4362018-08-23T18:31:42 *** nullptr| has quit IRC
4372018-08-23T18:32:45 *** nullptr| has joined #bitcoin-core-dev
4382018-08-23T18:34:16 *** Emcy has joined #bitcoin-core-dev
4392018-08-23T18:39:34 *** math_ has joined #bitcoin-core-dev
4402018-08-23T18:43:35 <MarcoFalke> ken2812221: Empact: I set the permissions to view and run builds for our github team
4412018-08-23T18:43:40 <MarcoFalke> Could someone verify that?
4422018-08-23T18:44:41 *** promag has joined #bitcoin-core-dev
4432018-08-23T18:46:18 <Empact> MarcoFalke: I'm still not seeing an option to restart the appveyor
4442018-08-23T18:47:10 <MarcoFalke> You need to be in the group that can assign labels to issues on our GitHub, I can't see who is in there
4452018-08-23T18:50:38 <Empact> Gotcha, yeah I'm not in the group
4462018-08-23T18:51:23 <wumpus> let's see if I can restart appveryot
4472018-08-23T18:53:16 <wumpus> I don't see it https://ci.appveyor.com/project/DrahtBot/bitcoin/build/master.127, but maybe I'm looking over it
4482018-08-23T18:53:36 <wumpus> (logged in through github)
4492018-08-23T18:56:41 <promag> wumpus: can I get #13100 in HP now that 13529 is merged?
4502018-08-23T18:56:45 <gribble> https://github.com/bitcoin/bitcoin/issues/13100 | gui: Add dynamic wallets support by promag · Pull Request #13100 · bitcoin/bitcoin · GitHub
4512018-08-23T18:57:08 <wumpus> promag: the meeting is in four minutes, but sure...
4522018-08-23T18:57:45 <MarcoFalke> hmm maybe I forgot to press "save"?
4532018-08-23T18:57:48 <MarcoFalke> Mind to try again?
4542018-08-23T18:58:27 <promag> wumpus: thanks
4552018-08-23T19:00:04 <sipa> meeting?
4562018-08-23T19:00:12 <wumpus> #startmeeting
4572018-08-23T19:00:12 <lightningbot> Meeting started Thu Aug 23 19:00:12 2018 UTC. The chair is wumpus. Information about MeetBot at http://wiki.debian.org/MeetBot.
4582018-08-23T19:00:12 <lightningbot> Useful Commands: #action #agreed #help #info #idea #link #topic.
4592018-08-23T19:00:16 <sipa> hi
4602018-08-23T19:00:35 <gmaxwell> hi
4612018-08-23T19:00:35 <achow101> hi
4622018-08-23T19:00:59 <jonasschnelli> hi
4632018-08-23T19:01:04 <promag> hi
4642018-08-23T19:01:04 <wumpus> #bitcoin-core-dev Meeting: wumpus sipa gmaxwell jonasschnelli morcos luke-jr btcdrak sdaftuar jtimon cfields petertodd kanzure bluematt instagibbs phantomcircuit codeshark mi
4652018-08-23T19:01:08 <wumpus> chagogo marcofalke paveljanik NicolasDorier jl2012 achow101 meshcollider jnewbery maaku fanquake promag provoostenator
4662018-08-23T19:01:09 *** meshcollider_ has joined #bitcoin-core-dev
4672018-08-23T19:01:27 <wumpus> who has a topic?
4682018-08-23T19:01:42 <meshcollider> Hi
4692018-08-23T19:02:04 <jonasschnelli> topic proposal: tor plugable transport
4702018-08-23T19:02:08 <wumpus> MarcoFalke: I still don't see it, though I'm not sure what kind of button I'm looking for
4712018-08-23T19:02:41 <wumpus> let's start with high prio as usual
4722018-08-23T19:02:46 <wumpus> #topic high priority for review
4732018-08-23T19:03:23 <jonasschnelli> I'd like to add #14032 if that is appropriate
4742018-08-23T19:03:25 <gribble> https://github.com/bitcoin/bitcoin/issues/14032 | Add p2p layer encryption with ECDH/ChaCha20Poly1305 by jonasschnelli · Pull Request #14032 · bitcoin/bitcoin · GitHub
4752018-08-23T19:03:50 <wumpus> https://github.com/bitcoin/bitcoin/projects/8 two were merged, one was added, this leaves five open
4762018-08-23T19:04:26 <wumpus> jonasschnelli: sure
4772018-08-23T19:04:44 <cfields> jonasschnelli: thanks, I'm excited to read that.
4782018-08-23T19:05:01 <jonasschnelli> I'm also excited to hear your feedback. :)
4792018-08-23T19:05:04 <wumpus> yes, very good to see a PR about that
4802018-08-23T19:05:07 <promag> jonasschnelli: do you think you can extract some commits to new PR's?
4812018-08-23T19:05:16 <MarcoFalke> I'd like to add #13961 (it is only refactoring, but drops the memory usage at compile time by 100Mb, also compiles faster)
4822018-08-23T19:05:18 <gribble> https://github.com/bitcoin/bitcoin/issues/13961 | util: Replace boost::signals2 with std::function by MarcoFalke · Pull Request #13961 · bitcoin/bitcoin · GitHubAsset 1Asset 1
4832018-08-23T19:05:40 <jonasschnelli> promag: I guess it's hard to make it smaller...
4842018-08-23T19:05:46 <wumpus> MarcoFalke: ok
4852018-08-23T19:05:46 <cfields> MarcoFalke: also nice :)
4862018-08-23T19:05:49 <achow101> topic suggestion: hardware wallet support
4872018-08-23T19:06:44 <wumpus> #topic tor pluggable transport (jonasschnelli)
4882018-08-23T19:06:59 <jonasschnelli> Tor has this concept of pluggable transports
4892018-08-23T19:07:01 <promag> jonasschnelli: could create a PR only with the first 3 commits, but let's talk later
4902018-08-23T19:07:04 <jonasschnelli> https://www.torproject.org/docs/pluggable-transports.html.en
4912018-08-23T19:07:08 <jonasschnelli> https://gitweb.torproject.org/torspec.git/tree/pt-spec.txt
4922018-08-23T19:07:08 *** spinza has quit IRC
4932018-08-23T19:07:21 <Empact> jonasshnelli: First three commits look independent as prep work
4942018-08-23T19:07:25 <jonasschnelli> It a great concept to bypass DPI, censored envs...
4952018-08-23T19:07:56 * sipa suggests using gramtropy generated text over IRC
4962018-08-23T19:07:58 <jonasschnelli> Tor can start subprocesses that bootstrap a SOCK proxy to communicate with the counterparty
4972018-08-23T19:08:06 <gmaxwell> jonasschnelli: Why do you think we need something special for that? We have proxy support already, which is all that is generally required for that sort of thing.
4982018-08-23T19:08:17 <jonasschnelli> I think supporting PT should be not to complicated in Core
4992018-08-23T19:08:30 <gmaxwell> I think it's so not complicated that we already have it. :)
5002018-08-23T19:08:36 <wumpus> I don't understand
5012018-08-23T19:08:40 <jonasschnelli> I'm unsure about the reverse proxy, the env vars and the auto-exec of that subprocess...
5022018-08-23T19:08:46 <wumpus> pluggable transports plug into *tor*
5032018-08-23T19:08:48 <sipa> jonasschnelli: what specific form of transport do you suggest?
5042018-08-23T19:08:53 <wumpus> bitcoin core can use tor
5052018-08-23T19:08:57 <wumpus> what else is needed?
5062018-08-23T19:09:12 <jonasschnelli> sipa: its not about specific transports,.. IMO its an API for other transport layers...
5072018-08-23T19:09:29 <sipa> jonasschnelli: it is to make tor communicate with other tor nodes using different transport layers
5082018-08-23T19:09:35 <sipa> what transport layer do you suggest?
5092018-08-23T19:09:37 <gmaxwell> The API is socks.
5102018-08-23T19:09:40 <wumpus> looks like a layer violation to worry about this? or do you want to make tor tunnel over bitcoin instead of the other way around?
5112018-08-23T19:09:43 <luke-jr> I also don't get it
5122018-08-23T19:09:46 <jonasschnelli> Maybe I got it wrong,.. but it looked to me after a pure transport layer (not tor)
5132018-08-23T19:10:07 <sipa> yes and no
5142018-08-23T19:10:07 <wumpus> yes, socks (+optional control port) is good enough as API
5152018-08-23T19:10:20 <wumpus> it's the only way it's recommended to use tor
5162018-08-23T19:10:20 <sipa> jonasschnelli: i'm very confused by what you suggest
5172018-08-23T19:10:45 <sipa> so far you've said "we can use tor pluggable transport", which is true for any tor user
5182018-08-23T19:10:51 <phantomcircuit> jonasschnelli, the pluggable transport presents a socks interface which tor uses instead of directly connecting
5192018-08-23T19:10:58 <sipa> how is it bitcoin specific, or what do you specifically propose?
5202018-08-23T19:11:00 <wumpus> integrating tor's code into bitcoin core is not a good idea
5212018-08-23T19:11:07 <kanzure> merge tor source tree
5222018-08-23T19:11:09 <phantomcircuit> it's not something that should be configured by things using tor
5232018-08-23T19:11:25 <wumpus> indeed, it's part of the tor configuration
5242018-08-23T19:11:27 <jonasschnelli> The idea is to make Bitcoin work in DPI env in case it would get blocked,.. like China or Iran
5252018-08-23T19:11:31 <jonasschnelli> Without relying on tor
5262018-08-23T19:11:36 <gmaxwell> wumpus: I think jonasschnelli suggests that we have support for these obfscuated transports without putting tor in the middle. I think we already do (in fact, I used one of the ones made for tor to bridge bitcoin nodes last year when we were worried about china blocking bitcoin... before later changing to an icecast stream so that it wouldn't be identifyable by traffic analysis)
5272018-08-23T19:11:42 <jonasschnelli> Core could directly use obfs4
5282018-08-23T19:11:47 <gmaxwell> jonasschnelli: we already can.
5292018-08-23T19:11:56 <jonasschnelli> or meek
5302018-08-23T19:12:03 <sipa> jonasschnelli: ah!
5312018-08-23T19:12:14 <wumpus> gmaxwell: ohh, so using *part* of tor, just the obfuscation part...
5322018-08-23T19:12:18 <sipa> you mean using obfuscation for our own connections, even non tor ones
5332018-08-23T19:12:21 <jonasschnelli> Since its a separated project and a seperate binary
5342018-08-23T19:12:32 <gmaxwell> wumpus: well the obfuscation parts of tor aren't parts of tor, technicaly. But yes.
5352018-08-23T19:12:35 <wumpus> this sounds like scope creep to me
5362018-08-23T19:12:51 <gmaxwell> jonasschnelli: have you trie using obfs4. I did previously, it just worked.
5372018-08-23T19:12:55 <gmaxwell> tried*
5382018-08-23T19:13:02 <sipa> i would say if we're worried about that, you should also just use tor already
5392018-08-23T19:13:04 <wumpus> gmaxwell: it's tor project code
5402018-08-23T19:13:13 <wumpus> maybe not part of the tor repository, I don't know
5412018-08-23T19:13:15 <phantomcircuit> i dont see any reason to do this, anybody who needs this can already configure it with the socks proxy support
5422018-08-23T19:13:24 <wumpus> yes
5432018-08-23T19:13:39 <jonasschnelli> okay... fair points. /topic
5442018-08-23T19:13:56 <gmaxwell> AFAIK, it just works. It's kinda limited though because none of them resist traffic analysis and it's SUPER easy to identify bitcoin traffic with traffic analysis. :)
5452018-08-23T19:14:05 *** plankers has joined #bitcoin-core-dev
5462018-08-23T19:14:27 <phantomcircuit> gmaxwell, iirc they're all super easy to identify anyways
5472018-08-23T19:14:33 <gmaxwell> If there was some addon we needed to make more of them work, I suppose that would be cool, but I dunno what that would be. At least the original obfs proxy that I used before was just a socks proxy.
5482018-08-23T19:15:10 <gmaxwell> phantomcircuit: well I think the idea in tor is that there are private ones.
5492018-08-23T19:15:24 <jonasschnelli> I think core <-> core could use the propose p2p encryption (with NewHope), but for censoring, is was unsure if we should relay on Tor or on the underlaying independent obfuscation projects.
5502018-08-23T19:15:25 <wumpus> oh the obfs layer itself is also a socks proxy?
5512018-08-23T19:15:42 <gmaxwell> wumpus: yes
5522018-08-23T19:15:47 <wumpus> tor is socks proxies all the way down and nothing else isn't it :)
5532018-08-23T19:16:02 <sipa> pigeons at the bottom of the stack
5542018-08-23T19:16:09 <wumpus> anyhow, yes, then we can already use it
5552018-08-23T19:16:22 <wumpus> sipa: pigeons for transaction broadcasting
5562018-08-23T19:16:24 <phantomcircuit> gmaxwell, the only ones that work are the domain fronting ones, so presumably only the private ones actually work
5572018-08-23T19:16:46 *** plankers has joined #bitcoin-core-dev
5582018-08-23T19:18:03 <wumpus> well the non-private ones become known very soon
5592018-08-23T19:18:07 <gmaxwell> (hm the thing I used before wasn't obfs4 it was an earlier version, they all look like they implement proxies regardless)
5602018-08-23T19:18:15 <wumpus> no matter how good the obfuscation is, if it's public that you run one...
5612018-08-23T19:19:04 <gmaxwell> In any case, I think jonasschnelli's spirt is right. I just don't actually think we need to do anything more than have socks support.
5622018-08-23T19:19:21 <wumpus> yes
5632018-08-23T19:19:24 <jonasschnelli> Even better then. :)
5642018-08-23T19:19:35 <gmaxwell> (it's certantly better to do protocol obfscuation via external proxies than to try to bake anything in...)
5652018-08-23T19:19:39 <sipa> how does the proxy know which peers need to use the obfuscation?
5662018-08-23T19:19:42 <phantomcircuit> gmaxwell, meek is the only one that works in china, it's domain fronting with gce
5672018-08-23T19:19:52 <sipa> you can't just blanket obfuscate all connections
5682018-08-23T19:20:04 <sipa> as our normal peer discovery will find peers that don't support it
5692018-08-23T19:20:24 <achow101> sipa: I think this would only work for outgoing connections
5702018-08-23T19:20:25 <wumpus> sipa: you wouldn't want to use normal peer discovery in such cases
5712018-08-23T19:20:44 <sipa> achow101: yes, of course
5722018-08-23T19:20:45 <sipa> wumpus: right
5732018-08-23T19:21:24 <wumpus> something needs to de-obfuscate incoming connections too, but I suppose that works by forwarding a listening port?
5742018-08-23T19:21:39 <gmaxwell> I think perhaps the useful 'feature' that might come for this is being able to provide per-peer proxy configs to addnode/connect.
5752018-08-23T19:21:55 <wumpus> or does it really need SOCKS5 binding
5762018-08-23T19:22:04 <jonasschnelli> I think achow101 has a topic
5772018-08-23T19:22:25 <wumpus> (which we don't support, we only use the proxy for outgoing, SOCKS binding is really really obscure)
5782018-08-23T19:22:37 <gmaxwell> I don't think there is need for socks5 binding.
5792018-08-23T19:22:48 <wumpus> okay
5802018-08-23T19:22:55 <wumpus> per-peer proxies could be useful, yes
5812018-08-23T19:23:10 <cfields> gmaxwell: interesting. I believe that'd be trivial to add, other than the syntax handling
5822018-08-23T19:23:52 <sipa> "netmask X uses proxy Y"
5832018-08-23T19:23:56 <gmaxwell> wumpus: https://www.comparitech.com/blog/vpn-privacy/hide-openvpn-traffic-obfsproxy-on-windows-and-linux-ec2/ you can see how you can hide arbritary crap with obfsproxy.
5842018-08-23T19:24:07 <wumpus> gmaxwell: thanks
5852018-08-23T19:24:25 <gmaxwell> sipa: don't want to go too far or proxychains becomes the right tool (it's a multiproxy routing daemon)
5862018-08-23T19:24:32 <sipa> ha
5872018-08-23T19:24:47 <sipa> that sounds like an improvements over sidechains or treechains
5882018-08-23T19:24:48 <phantomcircuit> gmaxwell, proxychains is never the right tool >.>
5892018-08-23T19:24:50 <gmaxwell> (it can even nest proxies)
5902018-08-23T19:25:05 <gmaxwell> phantomcircuit: but how else will I hax you through 7 proxies?
5912018-08-23T19:25:34 <wumpus> fwiw ssh has built-in support for proxy chaining in the client
5922018-08-23T19:26:19 <wumpus> #topic hardware wallet support (achow101)
5932018-08-23T19:26:29 <achow101> So with PSBT we can kind of do hardware wallet things
5942018-08-23T19:26:34 <gmaxwell> in any case, being able to net-scope or node scope proxies is something I would have used before, e.g. "addnode my node in the office via my ssh -D socks proxy".
5952018-08-23T19:26:53 <achow101> I've been working on a tool that takes a psbt and does the hardware wallet things: https://github.com/achow101/HWI
5962018-08-23T19:27:06 <jonasschnelli> great work achow101!
5972018-08-23T19:27:13 *** SopaXorzTaker has quit IRC
5982018-08-23T19:27:20 <gmaxwell> achow101: cool.
5992018-08-23T19:27:28 <wumpus> woohoo!
6002018-08-23T19:27:31 <achow101> but to actually get hww support, we still need a way to get derivation paths for imported keys and such. and for those keys to be part of the keypool
6012018-08-23T19:27:59 <jonasschnelli> why not a xpub watch-only wallet support?
6022018-08-23T19:28:09 *** clarkmoody has joined #bitcoin-core-dev
6032018-08-23T19:28:20 <achow101> I know sipa has the output descriptors stuff that would work for this, but I think that still quite a ways off
6042018-08-23T19:28:41 <sipa> i can prioritize it more, but it's unclear how to keep things compatible
6052018-08-23T19:29:00 <sipa> do we want the old keystore based system and the new descriptor based system side by side?
6062018-08-23T19:29:01 <jonasschnelli> importing keys seems meh-ish for hardware-wallet. IMO the ideal use case for BIP32 pub key derivation
6072018-08-23T19:29:11 <sipa> that wouldn't be too hard, i think
6082018-08-23T19:29:25 <gmaxwell> I don't see why we shouldn't deploy the very narrow thing we need to make all the common hardware wallets work, so long as it won't prevent us from using descriptor based logic in tuhe future.
6092018-08-23T19:29:26 <sipa> but it would be much nicer if we also get to convert old keystores to desceiptors
6102018-08-23T19:29:32 <jonasschnelli> Also, xpub watch-only do not need keypools
6112018-08-23T19:29:42 <gmaxwell> jonasschnelli: yes it does.
6122018-08-23T19:29:59 <jonasschnelli> I don't see why but happy to learn...
6132018-08-23T19:30:03 <sipa> it's the equivalent of their gap limit
6142018-08-23T19:30:07 <gmaxwell> It doesn't have private key material, but it needs _something_ to look ahead with pubkeys and watch for payments.
6152018-08-23T19:30:16 <achow101> I did #14021 for importing keypaths into the wallet with importmulti, but it isn't ideal
6162018-08-23T19:30:19 <gribble> https://github.com/bitcoin/bitcoin/issues/14021 | Import key origin data through importmulti by achow101 · Pull Request #14021 · bitcoin/bitcoin · GitHubAsset 1Asset 1
6172018-08-23T19:30:49 <achow101> it ... works, but kinda sucks as you have to import more keys as you use them. And they aren't in the keypool
6182018-08-23T19:30:49 <jonasschnelli> gmaxwell: yes. Right. Correction: no keypool written to disk, only xpub, then derive on load
6192018-08-23T19:31:03 <sipa> i fear for more combinations and special cases to deal with
6202018-08-23T19:31:31 <gmaxwell> jonasschnelli: well it doesn't really matter if we save it to disk or not. It's harmless to do so.
6212018-08-23T19:31:57 <jonasschnelli> Yes. Thats right.
6222018-08-23T19:32:06 <gmaxwell> achow101: obviously as a stopgap importing keys at least allows development and/or testing.
6232018-08-23T19:32:59 <gmaxwell> achow101: but I assume what you really want is a wallet whos master key reflects the third-party keychain. I fear the mess of mixed wallets.
6242018-08-23T19:33:04 <jonasschnelli> achow101: do you have a plan to make hww interaction more hassle free, like including USBHID in core </joke>?
6252018-08-23T19:33:29 <achow101> jonasschnelli: the idea I had was to use the HWI scripts and call them from Core.
6262018-08-23T19:33:58 <achow101> they implement all of the usb stuff
6272018-08-23T19:34:03 <gmaxwell> I think that should be out of of scope. We should implement a simple fork and stdio interface (or similar) to talk to hardware specific drivers.
6282018-08-23T19:34:32 <jonasschnelli> could one of the "hardware-wallets" just be a hot keystore? Similar to SSH Agent?
6292018-08-23T19:34:39 <gmaxwell> right.
6302018-08-23T19:34:40 <sipa> have metadata in the wallet that says "for these outputs, attempt to sign through device driver X"
6312018-08-23T19:34:52 <sipa> which is just an executable that gets invoked and sent a psbt
6322018-08-23T19:34:53 <gmaxwell> Even one using SGX+TPM, for example.
6332018-08-23T19:35:25 <gmaxwell> the drivers could then also provide their own UI, if required.
6342018-08-23T19:35:38 <gmaxwell> E.g. to get a passphrase.
6352018-08-23T19:35:51 <gmaxwell> or remind you to plug in the device.
6362018-08-23T19:35:52 <achow101> yup. I plan on adding a UI to HWI once I get everything actually working
6372018-08-23T19:36:00 <jonasschnelli> Yes. Becaue of the UI, I initially though every vendor must provide its own script/binary...
6382018-08-23T19:36:11 <sipa> jonasschnelli: yes
6392018-08-23T19:36:19 <jonasschnelli> Because again, plugins then endup outside of the vendors control
6402018-08-23T19:36:25 <jonasschnelli> Leading to outdated software
6412018-08-23T19:36:37 <gmaxwell> I'm sure it would eventually be possible to have "generic" drivers that work for a family of devices, but I think the industry is too immature for that.
6422018-08-23T19:36:48 <jonasschnelli> Indeed
6432018-08-23T19:37:07 <gmaxwell> sipa: how would we handle, e.g. "this script is multisig, with device X and device Y" ?
6442018-08-23T19:37:18 <gmaxwell> (er driver x and driver y)
6452018-08-23T19:37:20 <sipa> gmaxwell: list two drivers
6462018-08-23T19:37:27 <gmaxwell> sounds good.
6472018-08-23T19:37:34 <jonasschnelli> I was hoping for a protocol/API where Core starts a process and passes all relevant data and the vendors plugin there with rich UI "drivers"
6482018-08-23T19:37:41 <sipa> gmaxwell: the drivers would be metadata to descriptor records
6492018-08-23T19:37:45 <achow101> gmaxwell: wouldn't that be "this script has two keys and key 1 uses driver X and key 2 uses driver Y"
6502018-08-23T19:37:55 <sipa> so they would apply to all outputs derived using the same chain(s)
6512018-08-23T19:38:11 <sipa> achow101: doesn't even need to say which key is which device
6522018-08-23T19:38:20 <sipa> just "try signing using X, try signing using ay"
6532018-08-23T19:38:24 <sipa> *Y
6542018-08-23T19:39:42 <achow101> anyways, the main point of bringing up this topic was that I wanted to ask whether we should implement xpub watch only wallets or just wait for sipa to finish output descriptor wallets
6552018-08-23T19:39:46 <gmaxwell> This kind of interface could easily be used for things other than HWI. For example, green address has a 2fa service, where the service will sign its part of a multisig with you, if you pass a 2fa. This could be supported by a simple driver that sends the tx to a service and gets the sig back.
6562018-08-23T19:40:25 <gmaxwell> jonasschnelli: I don't think we should be providing the HWI specific UI, because if we provide the UI we would constrain innovation there.
6572018-08-23T19:41:00 <jonasschnelli> Not Core. But the vendor could have a driver with UI
6582018-08-23T19:41:12 <jonasschnelli> (and network interface and whatnot)
6592018-08-23T19:41:13 <gmaxwell> right. I agree.
6602018-08-23T19:42:01 <gmaxwell> The interface between core and the driver is basically just a PSBT plus arguments provided in the driver config.
6612018-08-23T19:42:14 <gmaxwell> achow101: well I think thats between you and sipa mostly
6622018-08-23T19:42:28 <gmaxwell> I hope we don't end up with even more combinitorial explosion in the wallet, however.
6632018-08-23T19:42:37 <jonasschnelli> Yes. But IMO that only one part. How to get pubkey&metdata into core (and eventually multisig stuff) is still unlcear to me
6642018-08-23T19:42:52 <gmaxwell> like having to handle a wallet with mixed old plain keys, hd keys, achowpubkey, and descriptors all at once.
6652018-08-23T19:42:55 <jonasschnelli> (in a non RPCish way)
6662018-08-23T19:42:59 <sipa> jonasschnelli: descriptors :)
6672018-08-23T19:43:18 <jonasschnelli> sipa: I mean plug-n-play. :)
6682018-08-23T19:43:20 <sipa> jonasschnelli: i'll try to create a writeup on how to integrate them in the wallet
6692018-08-23T19:43:26 <sipa> jonasschnelli: "import a descriptor"
6702018-08-23T19:43:41 <gmaxwell> the descriptor thing could eventually be "create new wallet with descriptor" "paste in descriptor" "confirm"..
6712018-08-23T19:43:43 <sipa> ah, you mean GUI
6722018-08-23T19:43:47 <sipa> i have no clue.
6732018-08-23T19:43:53 <jonasschnelli> You plugin a HWW, the Core detects it and tells you "do you wanna use XYZ".
6742018-08-23T19:43:58 <jonasschnelli> *then
6752018-08-23T19:44:16 <gmaxwell> doubt we're going to have any detection anytime soon...
6762018-08-23T19:44:32 <gmaxwell> as that would require shipping a pile of vendor specific detection and interface code.
6772018-08-23T19:44:48 <jonasschnelli> Yes. But if we form an API or similar, we should not only focus on the PSBT part,... also on the how do I create a watch-only wallet (eventually also multisig)
6782018-08-23T19:45:04 <jonasschnelli> But I guess the descriptors are the answer here
6792018-08-23T19:45:32 <gmaxwell> Create a new wallet from a descriptor, or import a descriptor into a wallet.
6802018-08-23T19:46:05 <gmaxwell> sipa: maybe you can work with achow 101 on a descriptor-lite implementation that would be forward compatible with the full thing later, and is enough to support the common hardware wallets?
6812018-08-23T19:46:34 <sipa> gmaxwell: descriptors are also in the codebase
6822018-08-23T19:46:56 <sipa> *already
6832018-08-23T19:47:12 <gmaxwell> well I meant basically "enough stuff to be able to create an xpub with path autofilling watching wallet"
6842018-08-23T19:47:13 <sipa> the hard part is integrating them in the wallet of course
6852018-08-23T19:47:32 <sipa> but that isn't harder or easier depending on what the descriptors support
6862018-08-23T19:48:07 <achow101> gmaxwell: the problem with that is the wallet format needs to change
6872018-08-23T19:48:49 <sipa> i'll try to think it through
6882018-08-23T19:51:28 <wumpus> any other topics?
6892018-08-23T19:51:49 <gmaxwell> jonasschnelli: wanted to talk about encryption!
6902018-08-23T19:52:04 <wumpus> #topic P2P encryption (jonasschnelli)
6912018-08-23T19:52:10 <jonasschnelli> Not really... :) but happy to do
6922018-08-23T19:52:28 <gmaxwell> heh, what just going to open that big PR and go on vacation? :P
6932018-08-23T19:52:46 <jonasschnelli> The question for me now is, is the PR to large (2000lines) and how and when to add NewHope (quantum resistant handshake)
6942018-08-23T19:53:35 <jonasschnelli> If it is to large, how to split it into PR that make partially sense on its own (could add the crypto stuff beforehand, but meh)
6952018-08-23T19:53:42 <gmaxwell> so the PR can probably be broken into three parts: (1) prepatory refactors, (2) inclusion of new primitives, (3) the thing itself.
6962018-08-23T19:53:58 <jonasschnelli> Yes. Agree.
6972018-08-23T19:54:05 <jonasschnelli> But would we merge 2 without 3?
6982018-08-23T19:54:21 <jonasschnelli> Maybe 2 with concrete chances that 3 gets merged
6992018-08-23T19:54:28 <gmaxwell> Yes. Exactly.
7002018-08-23T19:54:45 <jonasschnelli> 1 will be very small
7012018-08-23T19:54:53 <gmaxwell> Well if we changed our mind we could always take it out. We've put other primitives in in advance before.
7022018-08-23T19:55:10 <gmaxwell> e.g. bip32 key derrivation went unused for years.
7032018-08-23T19:55:14 <sipa> BIP32 derivation was in the codebase for 3 years before being used :)
7042018-08-23T19:55:15 <sipa> jinx.
7052018-08-23T19:55:21 <jonasschnelli> Indeed. But its 2018 now. :)
7062018-08-23T19:55:24 <jonasschnelli> Okay. I'll do the split then.
7072018-08-23T19:55:49 <wumpus> granted, that was before people neurotically deleted dead code :-)
7082018-08-23T19:55:53 <gmaxwell> It's fine, I think that chacha stuff is mostly a distraction from the review of that PR. It needs to be reviewed but mostly for build integration, etc.
7092018-08-23T19:56:05 <gmaxwell> wumpus: or even not-dead-yet code!
7102018-08-23T19:56:10 <jonasschnelli> heh
7112018-08-23T19:56:11 <sipa> wumpus: it wasn't dead! it had tests!
7122018-08-23T19:56:14 <wumpus> gmaxwell: even preemptively
7132018-08-23T19:56:42 <cfields> jonasschnelli: taking a quick look at your PR, was the handshake spec modified to be sent first thing? Or am I misreading the code?
7142018-08-23T19:56:49 <jonasschnelli> Yes. I can also try to extend the existing ChaCha20 RNG to be used for the crypto part
7152018-08-23T19:56:53 <wumpus> jonasschnelli: anyhow if it simplifies review, please do split it up
7162018-08-23T19:56:58 <jonasschnelli> cfields: yes
7172018-08-23T19:57:11 <jonasschnelli> cfields: https://gist.github.com/jonasschnelli/c530ea8421b8d0e80c51486325587c52#Handshake
7182018-08-23T19:57:20 <jonasschnelli> Great. Will split
7192018-08-23T19:57:27 <promag> nice
7202018-08-23T19:57:57 <gmaxwell> jonasschnelli: when you split make sure all the parts are visible. E.g. I'll want to look at (3) while reviewing (1) just in case some refactor seems like it makes no sense on its own. :)
7212018-08-23T19:58:03 <cfields> jonasschnelli: woohoo, thanks
7222018-08-23T19:58:42 <jonasschnelli> gmaxwell: Yes. Good point. Will do.
7232018-08-23T19:59:00 <jonasschnelli> I guess keeping the "overall" PR somewhere helps to see the goal.
7242018-08-23T19:59:14 <gmaxwell> sipa has done that in the past, e.g. with the segwit rebase.
7252018-08-23T19:59:30 <jonasschnelli> The unanswered question is when and how to add the NewHope handshake part
7262018-08-23T19:59:38 <wumpus> yes, indeed, though we probably want to change the one in high-priority then
7272018-08-23T19:59:53 <jonasschnelli> Yes. Lets remove it for now
7282018-08-23T19:59:56 <jonasschnelli> (ill do that)
7292018-08-23T20:00:07 <wumpus> ok
7302018-08-23T20:00:12 <wumpus> #endmeeting
7312018-08-23T20:00:12 <lightningbot> Meeting ended Thu Aug 23 20:00:12 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)
7322018-08-23T20:00:12 <lightningbot> Minutes: http://www.erisian.com.au/meetbot/bitcoin-core-dev/2018/bitcoin-core-dev.2018-08-23-19.00.html
7332018-08-23T20:00:12 <lightningbot> Minutes (text): http://www.erisian.com.au/meetbot/bitcoin-core-dev/2018/bitcoin-core-dev.2018-08-23-19.00.txt
7342018-08-23T20:00:12 <lightningbot> Log: http://www.erisian.com.au/meetbot/bitcoin-core-dev/2018/bitcoin-core-dev.2018-08-23-19.00.log.html
7352018-08-23T20:00:21 <wumpus> jonasschnelli: I did already
7362018-08-23T20:00:24 <jonasschnelli> thanks
7372018-08-23T20:00:40 <gmaxwell> jonasschnelli: well my thought is that we make another PR on top of these-- likely with two commits, one that just adds the code to the build, the other which crams it into the handshake, so we see the marginal cost.
7382018-08-23T20:01:11 <jonasschnelli> Yes. Exactly.
7392018-08-23T20:02:05 <jonasschnelli> gmaxwell: Do you know if it is sane to replace using SHAKE as the output "KDF/XOF" with HKDF?
7402018-08-23T20:02:13 <jonasschnelli> Or would that be a custom version of NewHope?
7412018-08-23T20:03:01 <gmaxwell> jonasschnelli: we could just take its output as is for now. Lets change the absolute minimum for the first pass.
7422018-08-23T20:03:14 <jonasschnelli> okay
7432018-08-23T20:03:21 <cfields> jonasschnelli: I'm lagging quite far behind, so "go read the full spec" is a fine answer if you'd like. But I don't see any extensibility in the handshake spec? What's the reason for hard-coding a bare 32byte key without some sort of versioning/typing?
7442018-08-23T20:03:40 <sipa> cfields: versioning outside of the key is identifiable
7452018-08-23T20:03:46 <cfields> got it.
7462018-08-23T20:03:52 <cfields> thanks
7472018-08-23T20:03:53 <jonasschnelli> yes..
7482018-08-23T20:04:06 <jonasschnelli> cfields: also, the additional NewHope handshake is not in the specs right now.
7492018-08-23T20:04:18 <gmaxwell> cfields: we are avoiding having any fixed bytes so that it's harder for dumb DPI to block the traffic. A new protocol can distinguish itself by being incompatible. :)
7502018-08-23T20:04:36 <jonasschnelli> But IMO censorship resistance is not property of that proposal...
7512018-08-23T20:04:37 <cfields> jonasschnelli: right, I was just trying to figure out how the transition would work. I understand the issue now.
7522018-08-23T20:05:14 <jonasschnelli> Not having censorship resistance as a property of the new protocol, doesn't mean, we should use static, identifiable data if avoidable
7532018-08-23T20:05:23 <gmaxwell> in general upgrading this stuff, even with versioning is messy.
7542018-08-23T20:05:57 <jonasschnelli> and that, yes.
7552018-08-23T20:06:37 <jonasschnelli> Does anyone knows why appveyor has build issues when travis and gitian is fine with it? https://ci.appveyor.com/project/DrahtBot/bitcoin/build/master.148
7562018-08-23T20:06:41 <gmaxwell> like you send key type B and then the other side doesn't support B only A.... so what, now you start over? all doable just fine.
7572018-08-23T20:07:24 <jonasschnelli> I just figured out we don't keep node stats outside of the connection lifetime...
7582018-08-23T20:07:30 <jonasschnelli> (except addrman's service bits)
7592018-08-23T20:07:32 <gmaxwell> But the extra complexity makes me feel like we're not likely to do something like newhope until the next major protocol rework, if it doesn't go in the first ersion.
7602018-08-23T20:07:54 <jonasschnelli> Yes. It should go into the first, deployed version.
7612018-08-23T20:07:57 <jonasschnelli> But maybe not in the same PR
7622018-08-23T20:08:42 <jonasschnelli> Leading to the possibility of incompatible master<->master peers, which I think is fine
7632018-08-23T20:09:09 <jonasschnelli> (or merge them almost simultaneous)
7642018-08-23T20:15:06 <gmaxwell> yea, incompatible master/master doesn't worry me too much. If we're worried we could add a hidden option to disable the crypto and default it to off until we're ready.
7652018-08-23T20:17:31 <wumpus> putting it behind an option initially sounds like a good idea in any case
7662018-08-23T20:17:52 <jonasschnelli> Yes. Thats done already (-netencryption)
7672018-08-23T20:17:56 <jonasschnelli> Disabled by default
7682018-08-23T20:18:03 <wumpus> right
7692018-08-23T20:20:18 *** profmac has quit IRC
7702018-08-23T20:21:17 <MarcoFalke> jonasschnelli: Appveyor is a different build system
7712018-08-23T20:21:29 <MarcoFalke> You'd have to add new files to the cmake file
7722018-08-23T20:21:40 <MarcoFalke> (Assuming that that is the issue)
7732018-08-23T20:22:06 <jonasschnelli> So all new files need to be added to the Makefile and to a special CI file?
7742018-08-23T20:22:29 <wumpus> we need documentation for this appveyor stuff
7752018-08-23T20:22:44 <jonasschnelli> Sorry,... I missed the whole PR about that
7762018-08-23T20:22:46 <wumpus> it's a different build system and no one really has an idea how it works
7772018-08-23T20:22:57 <gmaxwell> what does it do?
7782018-08-23T20:23:02 <wumpus> well maybe MarcoFalke does :)
7792018-08-23T20:23:07 *** profmac has joined #bitcoin-core-dev
7802018-08-23T20:23:09 <MarcoFalke> no, lol
7812018-08-23T20:23:19 <achow101> gmaxwell: it's for building windows things IIRC
7822018-08-23T20:23:39 <achow101> building in a windows environment
7832018-08-23T20:23:43 <jonasschnelli> Do we really need a Visual Studio CI?
7842018-08-23T20:23:45 <wumpus> otherwise I'm going to ignore it
7852018-08-23T20:24:00 <wumpus> and merge if only travis passes
7862018-08-23T20:24:38 <MarcoFalke> I think it is fine to ignore for now because it is experimental
7872018-08-23T20:25:09 <achow101> MarcoFalke: is there supposed to a button to restart appveyor builds? sf so, i don't see one
7882018-08-23T20:25:10 *** vexbuy has joined #bitcoin-core-dev
7892018-08-23T20:25:22 <wumpus> jonasschnelli: I think it's hardly reasonable to expect people submitting PRs to update two build systems
7902018-08-23T20:25:44 <MarcoFalke> achow101: hmm, yeah there should be a button
7912018-08-23T20:26:51 <wumpus> especially if we all have no clue how it works
7922018-08-23T20:27:06 <achow101> presumably ken2812221 knows since he was the one to suggest it?
7932018-08-23T20:31:14 *** hebasto has joined #bitcoin-core-dev
7942018-08-23T20:32:48 *** profmac has quit IRC
7952018-08-23T20:35:24 <jonasschnelli> Maybe it is possible to run the appveyor CI on PRs but not report back (with green-checkmark or red-cross)
7962018-08-23T20:35:51 *** profmac has joined #bitcoin-core-dev
7972018-08-23T20:38:21 <jonasschnelli> I'm not going to touch an .vcxproj XML...
7982018-08-23T20:39:18 *** Victorsueca has quit IRC
7992018-08-23T20:40:29 *** Victorsueca has joined #bitcoin-core-dev
8002018-08-23T20:41:49 <wumpus> ya it's a proprietary format too
8012018-08-23T20:42:32 *** Empact has quit IRC
8022018-08-23T20:44:45 <luke-jr> smh
8032018-08-23T20:46:24 *** clarkmoody has quit IRC
8042018-08-23T20:49:36 *** elichai2 has quit IRC
8052018-08-23T20:55:43 <ryanofsky> would be nice if appveyor errors on github were only shown as warnings. it is actually pretty straightforward to edit .vcxproj files, though
8062018-08-23T21:01:41 <wumpus> I'm sure it's learnable, but I think we already ask enough things from contributors
8072018-08-23T21:03:20 *** phwalkr has joined #bitcoin-core-dev
8082018-08-23T21:04:20 <wumpus> not that it would hurt to have documentation on how to add files, even with the current build system
8092018-08-23T21:05:28 *** thib has joined #bitcoin-core-dev
8102018-08-23T21:11:15 <MarcoFalke> Hmm, anyone knows how to have one of the linux jobs in travis compile with clang?
8112018-08-23T21:11:36 <MarcoFalke> I installed clang and then set CC=clang and CXX=clang++, but these are simply ignored
8122018-08-23T21:12:21 <MarcoFalke> All those build systems are hokuspokus to me
8132018-08-23T21:13:32 <wumpus> $1/configure CC="${CLANGPATH}/bin/clang " CXX="${CLANGPATH}/bin/clang++" OBJCXX="${CLANGPATH}/bin/clang++"
8142018-08-23T21:13:35 <wumpus> is what I have
8152018-08-23T21:15:47 <cfields> MarcoFalke: I believe CC and CXX can't be overridden during configure
8162018-08-23T21:16:13 <wumpus> I've been doing it for years
8172018-08-23T21:16:14 <luke-jr> cfields: eh, they should be
8182018-08-23T21:16:27 <cfields> er sorry...
8192018-08-23T21:16:34 <cfields> MarcoFalke: I believe CC and CXX can't be overridden during configure for depends builds
8202018-08-23T21:16:44 <cfields> patch coming up
8212018-08-23T21:17:22 <wumpus> never tried it with depends builds
8222018-08-23T21:17:34 <MarcoFalke> Hmm, maybe I remember wrongly that it worked with depends builds
8232018-08-23T21:17:35 <wumpus> so that could well be true
8242018-08-23T21:17:56 <MarcoFalke> patch welcome, obviously :)
8252018-08-23T21:20:32 *** meshcollider_ has quit IRC
8262018-08-23T21:30:58 *** hebasto has quit IRC
8272018-08-23T21:34:21 *** profmac has quit IRC
8282018-08-23T21:35:39 *** profmac has joined #bitcoin-core-dev
8292018-08-23T21:41:31 <cfields> MarcoFalke: https://github.com/theuni/bitcoin/commit/0d00fd5901102d9ca2b99d6f17a3bd96c946e3b7
8302018-08-23T21:42:21 <cfields> after doing that, need to 'make' in depends again. It should be instant, though.
8312018-08-23T22:04:05 *** owowo has quit IRC
8322018-08-23T22:06:46 *** lnostdal has quit IRC
8332018-08-23T22:19:15 *** Varunram has quit IRC
8342018-08-23T22:19:27 *** Varunram has joined #bitcoin-core-dev
8352018-08-23T22:20:06 *** Guyver2 has quit IRC
8362018-08-23T22:36:07 *** lukedashjr has joined #bitcoin-core-dev
8372018-08-23T22:38:43 *** luke-jr has quit IRC
8382018-08-23T22:41:23 *** lukedashjr is now known as luke-jr
8392018-08-23T23:00:40 *** michaelsdunn1 has quit IRC
8402018-08-23T23:37:06 *** Zenton has quit IRC
8412018-08-23T23:59:29 *** profmac has quit IRC