1 2020-12-12T00:00:06  *** r2wj <r2wj!~r2wj@> has joined #bitcoin-core-dev
  2 2020-12-12T00:02:35  <achow101> I'll try an updated native_cctools first
  3 2020-12-12T00:02:58  <sipa> cool
  4 2020-12-12T00:03:15  <sipa> it's also possible that jonasschnelli's codesign tool is too old!
  5 2020-12-12T00:03:28  <sipa> (as native_cctools was updated for 0.21)
  6 2020-12-12T00:03:41  <achow101> Well it's worked for the previous rc
  7 2020-12-12T00:04:29  <sipa> it may be that some alignment thing changed that doesn't matter in every situation
  8 2020-12-12T00:06:27  *** jb55 <jb55!~jb55@gateway/tor-sasl/jb55> has quit IRC (Remote host closed the connection)
  9 2020-12-12T00:07:45  <sipa> maybe we should just make some arbitrary binary-changing commits for rc4, and hope that that fixes it
 10 2020-12-12T00:07:48  <sipa> *hides*
 11 2020-12-12T00:09:37  *** Talkless <Talkless!~Talkless@mail.dargis.net> has quit IRC (Quit: Konversation terminated!)
 12 2020-12-12T00:09:57  *** mol_ <mol_!~mol@unaffiliated/molly> has joined #bitcoin-core-dev
 13 2020-12-12T00:12:49  <sipa> jonasschnelli: what version is your cctools? if you have the ability to change it, use 949.0.1
 14 2020-12-12T00:13:07  <sipa> achow101: what version is your codesigning tool on macos?
 15 2020-12-12T00:13:22  <achow101> no idea
 16 2020-12-12T00:13:47  <achow101> there's no version command for codesign or codesign_allocate
 17 2020-12-12T00:14:13  *** mol <mol!~mol@unaffiliated/molly> has quit IRC (Ping timeout: 265 seconds)
 18 2020-12-12T00:14:59  <sipa> this line in codesign_allocate changed in the latest cctools:
 19 2020-12-12T00:15:10  <sipa> object->seg_linkedit->vmsize = rnd32(object->seg_linkedit->filesize, get_segalign_from_flag(&arch_signs[i].arch_flag));
 20 2020-12-12T00:15:39  <achow101> where do you see this?
 21 2020-12-12T00:17:10  <sipa> https://github.com/tpoechtrager/cctools-port/commit/466063c7f7486762a5a8f0d98c48dc5ffbe42f74#diff-c0274ccf413d9c63c15dc20f765faa1dd750ae7a74b65dfda9afbe1e91ea137cR711
 22 2020-12-12T00:17:22  <sipa> this is in the version we're using in the gitian build
 23 2020-12-12T00:17:48  <sipa> but given that it recently changed (may 2020), perhaps somehow jonasschnelli is using an apple equivalent that is older
 24 2020-12-12T00:17:57  <sipa> *march 2020
 25 2020-12-12T00:19:22  <sipa> actually, no
 26 2020-12-12T00:19:30  <sipa> rnd -> rnd32 doesn't change anything
 27 2020-12-12T00:21:08  <achow101> eh.. they don't make tags for new versions.. great
 28 2020-12-12T00:22:41  *** dermoth_ <dermoth_!~dermoth@unaffiliated/dermoth> has joined #bitcoin-core-dev
 29 2020-12-12T00:23:01  *** dermoth <dermoth!~dermoth@unaffiliated/dermoth> has quit IRC (Disconnected by services)
 30 2020-12-12T00:23:03  *** dermoth_ is now known as dermoth
 31 2020-12-12T00:24:18  <sipa> got it!
 32 2020-12-12T00:24:34  <sipa> cctools v921 uses 4k as default alignment for x86_64
 33 2020-12-12T00:24:44  <sipa> oh, no, i misread :(
 34 2020-12-12T00:28:01  <sipa> what is the full output from that x86_64-apple-darwin-otool -l Bitcoin-Qt command?
 35 2020-12-12T00:28:08  *** proofofkeags <proofofkeags!~proofofke@174-16-212-53.hlrn.qwest.net> has joined #bitcoin-core-dev
 36 2020-12-12T00:29:52  <achow101> of which version of Bitcoin-Qt
 37 2020-12-12T00:30:11  <sipa> no matter
 38 2020-12-12T00:31:01  <achow101> https://0bin.net/paste/ArdAXxJ3#67qz6LxpqPGqIXWODToWwyFxorHcouHH1CzJGxdBB2I
 39 2020-12-12T00:31:04  <gribble> https://github.com/bitcoin/bitcoin/issues/67 | does not compile with boost_1_45_0 on OS X 10.6 · Issue #67 · bitcoin/bitcoin · GitHub
 40 2020-12-12T00:31:15  <achow101> bad bot
 41 2020-12-12T00:35:47  <sipa> the __LINKEDIT "filesize" entry is the same for the codesign-output-binary and the gitian-reconstructed one, right?
 42 2020-12-12T00:35:51  <sipa> just vmsize is different
 43 2020-12-12T00:35:57  <achow101> yes
 44 2020-12-12T00:36:06  <sipa> yeah, if it wasnt't, more than 1 byte would differ
 45 2020-12-12T00:38:17  <sipa> literally every version of cctools i can find uses 0x1000 as segment alignment for X86_64
 46 2020-12-12T00:38:51  <sipa> so the correct rounding of filesize 500144 to vmsize is 0x7b000
 47 2020-12-12T00:39:15  <sipa> why is jonas' tool rounding it to 0x7c000 (like he's using a segment alignment of 0x2000 or 0x4000)
 48 2020-12-12T00:39:47  <sipa> now, libstuff uses 0x2000 for unknown CPU architectures
 49 2020-12-12T00:40:43  <sipa> so maybe somehow the wrong architecture is passed down?
 50 2020-12-12T00:41:24  <sipa> see https://opensource.apple.com/source/cctools/cctools-949.0.1/libstuff/arch.c.auto.html look for "{ CPU_TYPE_X86_64,    LITTLE_ENDIAN_BYTE_SEX, 0x7fff5fc00000LL,  0x1000 },"
 51 2020-12-12T00:41:59  <achow101> updating cctools (specifically codesign_allocate) did not make a difference
 52 2020-12-12T00:42:33  *** jeremyrubin <jeremyrubin!~jr@c-73-15-215-148.hsd1.ca.comcast.net> has quit IRC (Ping timeout: 260 seconds)
 53 2020-12-12T00:42:54  <achow101> jonasschnelli: are you using the same signing machine as before?
 54 2020-12-12T00:43:00  <sipa> and in https://opensource.apple.com/source/cctools/cctools-949.0.1/misc/codesign_allocate.c.auto.html look for "object->seg_linkedit->vmsize ="
 55 2020-12-12T00:43:22  <achow101> IIRC he has one of the M1 macs for testing stuff, maybe he used that for signing?
 56 2020-12-12T00:44:36  *** Eagle[TM] <Eagle[TM]!~EagleTM@unaffiliated/eagletm> has quit IRC (Ping timeout: 240 seconds)
 57 2020-12-12T00:50:47  <sipa> i suspect that if we add 1024 bytes to the sig size in codesign_allocate it may work
 58 2020-12-12T00:50:56  <sipa> if you do:
 59 2020-12-12T00:51:05  <sipa> codesign_allocate -i <unsigned Bitcoin-Qt filename> -a x86_64 $((225312+1024)) -o <some temp file>
 60 2020-12-12T00:51:08  <sipa> on linux
 61 2020-12-12T00:51:28  <sipa> it may just work, with the existing signature
 62 2020-12-12T00:51:41  <achow101> I'm wondering why this size makes a difference to how the signature validates
 63 2020-12-12T00:51:50  <achow101> istm it shouldn't be covered by the signature itself
 64 2020-12-12T00:52:06  <sipa> i believe the code signature covers the entire binary, with the signature itself wiped out (this is a guess...)
 65 2020-12-12T00:52:11  <sipa> so its size matters
 66 2020-12-12T00:52:25  <sipa> that would explain why a tool like codesign_allocate exists in the first place
 67 2020-12-12T00:52:34  <sipa> as its behavior is nominal for the signing process
 68 2020-12-12T00:52:40  <achow101> but the size of the signature isn't known until the signature is made
 69 2020-12-12T00:53:13  <sipa> i assume that the codesign tool does now the signature size beforehand
 70 2020-12-12T00:53:16  <sipa> *know
 71 2020-12-12T00:53:26  <sipa> or it pads it to make sure it does
 72 2020-12-12T00:53:39  <achow101> i suppose it just needs to fall within the page size
 73 2020-12-12T00:53:46  <achow101> and that's probably a big enough range
 74 2020-12-12T00:53:51  <sipa> right
 75 2020-12-12T00:54:24  <achow101> so apple makes codesign_allocate open source, but not codesign itself?
 76 2020-12-12T00:54:47  <sipa> apparently
 77 2020-12-12T00:54:56  <sipa> but codesign invokes codesign_allocate
 78 2020-12-12T00:55:09  <achow101> supposedly
 79 2020-12-12T00:55:18  <sipa> and you can override which codesign_allocate to use... as long as it's signed by apple
 80 2020-12-12T00:56:44  <sipa> if you run "codesign_allocate -i <gitian Bitcoin-Qt filename> -a x86_64 $((225312+1024)) -o <newname>" in linux, does the resulting binary work in macos?
 81 2020-12-12T00:57:06  <sipa> oh, nvm
 82 2020-12-12T00:57:11  <sipa> nope, won't work
 83 2020-12-12T00:57:21  <sipa> that'll actually make space in the binary
 84 2020-12-12T00:58:12  *** chjj_ <chjj_!~chjj@unaffiliated/chjj> has joined #bitcoin-core-dev
 85 2020-12-12T01:00:01  <achow101> what if I set arch to ARM64 so it uses 0x4000 as the page size?
 86 2020-12-12T01:00:59  *** vasild <vasild!~vd@gateway/tor-sasl/vasild> has joined #bitcoin-core-dev
 87 2020-12-12T01:03:33  <achow101> and codesign_allocate doesn't like it
 88 2020-12-12T01:08:19  *** chjj <chjj!~chjj@unaffiliated/chjj> has quit IRC (Quit: WeeChat 2.2)
 89 2020-12-12T01:08:26  <sipa> yeah i don't expect that'd work
 90 2020-12-12T01:08:56  <sipa> achow101: but if you run codesign_allocate with 225312, on macos, you also get the 0x7c000 vmsize?
 91 2020-12-12T01:09:54  *** chjj_ <chjj_!~chjj@unaffiliated/chjj> has quit IRC (Quit: WeeChat 2.9)
 92 2020-12-12T01:10:16  *** chjj <chjj!~chjj@unaffiliated/chjj> has joined #bitcoin-core-dev
 93 2020-12-12T01:11:14  <achow101> Ah, yes
 94 2020-12-12T01:12:41  <sipa> so perhaps... the codesign tool isn't invoking it with size 225312
 95 2020-12-12T01:12:53  <sipa> mayne it invokes it twice
 96 2020-12-12T01:13:20  <sipa> once with a prospective size, to get the to-be-signed binary, and then again with the real size
 97 2020-12-12T01:13:39  <achow101> Now, I think the installation of codesign_allocate is different from the open source one
 98 2020-12-12T01:13:54  <sipa> that'd explain the filesize of LINKEDIT being correct, but vmsize isn't
 99 2020-12-12T01:14:01  <achow101> Because it matches jonas's, not gitian
100 2020-12-12T01:17:13  *** mj_node <mj_node!~mj_node@> has quit IRC (Quit: Leaving)
101 2020-12-12T01:19:15  <sipa> right
102 2020-12-12T01:23:43  <sipa> if you on linux run codesign_allocate with a larger size first, and then again withnthe correct size, does the __linkedit stuff end up being correct?
103 2020-12-12T01:33:56  <achow101> what's a big enough size?
104 2020-12-12T01:35:02  <sipa> $((225312+1024))
105 2020-12-12T01:39:03  *** vasild <vasild!~vd@gateway/tor-sasl/vasild> has quit IRC (Ping timeout: 240 seconds)
106 2020-12-12T01:39:21  <achow101> nope. also still 7b
107 2020-12-12T01:40:36  <sipa> after both commands, or even after the first one?
108 2020-12-12T01:40:41  <achow101> both
109 2020-12-12T01:40:44  <sipa> oh, i'm wrong
110 2020-12-12T01:40:46  *** vasild <vasild!~vd@gateway/tor-sasl/vasild> has joined #bitcoin-core-dev
111 2020-12-12T01:40:54  <achow101> well both as in both both commands and first one
112 2020-12-12T01:40:55  <sipa> 0x1000 is not 1024, but 4096
113 2020-12-12T01:41:04  <sipa> so $((225312+4024))
114 2020-12-12T01:41:10  <sipa> so $((225312+4096))
115 2020-12-12T01:41:12  <achow101> oh right
116 2020-12-12T01:43:36  <achow101> that works
117 2020-12-12T01:43:57  <achow101> although I will say that this is not a suitable workaround
118 2020-12-12T01:44:55  <sipa> we could produce in the gitian output the information on what size to resize to
119 2020-12-12T01:45:11  <sipa> it can be deterministically inferred from the macos-signed binary
120 2020-12-12T01:46:25  <achow101> i suppose
121 2020-12-12T01:46:58  <achow101> it feels like there should be a better way
122 2020-12-12T01:47:17  <sipa> yes
123 2020-12-12T01:47:37  <sipa> there is clearly a discrepancy between the macos codesign_allocate tool and the opensource one
124 2020-12-12T01:47:54  <sipa> and i'm confused why that isn't a problem for other projects
125 2020-12-12T01:48:47  <achow101> maybe no one has tried to make a new release recently?
126 2020-12-12T01:49:14  <achow101> we didn't see this a few weeks ago for rc2
127 2020-12-12T01:52:38  <sipa> yeah, baffling
128 2020-12-12T01:54:17  <sipa> lol @ actively {-beating our heads against Apple's walled garden wall-} debugging this issue
129 2020-12-12T01:56:41  <sipa> there is a macports version of cctools, presumably using the open source code
130 2020-12-12T01:56:53  <sipa> if you run that on macos, what behavior does it have?
131 2020-12-12T01:57:22  <achow101> I don't have macports installed
132 2020-12-12T02:01:14  <achow101> I found an old version of codesign source on apple's opensource site: https://opensource.apple.com/source/security_systemkeychain/security_systemkeychain-55191.1/src/codesign.cpp.auto.html
133 2020-12-12T02:01:26  <achow101> it disappears eventually, but that might just mean it got moved to somewhere else
134 2020-12-12T02:02:12  *** AaronvanW <AaronvanW!~AaronvanW@unaffiliated/aaronvanw> has quit IRC ()
135 2020-12-12T02:02:21  <sipa> that one doesn't invoke codesign_allocate
136 2020-12-12T02:02:52  <sipa> as far i can see, but it may be in another source file
137 2020-12-12T02:12:42  *** EagleTM <EagleTM!~EagleTM@unaffiliated/eagletm> has joined #bitcoin-core-dev
138 2020-12-12T02:18:08  *** ConsciousFlesh <ConsciousFlesh!~cFlesh@unaffiliated/consciousflesh> has joined #bitcoin-core-dev
139 2020-12-12T02:21:34  *** twistedline_ <twistedline_!~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8> has joined #bitcoin-core-dev
140 2020-12-12T02:22:06  *** twistedline <twistedline!~twisted@unaffiliated/twistedline> has quit IRC (Read error: Connection reset by peer)
141 2020-12-12T02:24:27  *** twistedline_ <twistedline_!~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8> has quit IRC (Remote host closed the connection)
142 2020-12-12T02:24:53  *** twistedline_ <twistedline_!~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8> has joined #bitcoin-core-dev
143 2020-12-12T02:27:22  <achow101> sipa: macports cctools uses 0x7b000
144 2020-12-12T02:31:27  *** twistedline_ <twistedline_!~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8> has quit IRC (Remote host closed the connection)
145 2020-12-12T02:31:54  *** twistedline_ <twistedline_!~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8> has joined #bitcoin-core-dev
146 2020-12-12T02:32:21  <ConsciousFlesh> Are there any devs here who were involved with, or around when the `RPCHelpMan()` function replaced throwing a runtime error to display JSON-RPC help text?
147 2020-12-12T02:33:00  *** twistedline_ <twistedline_!~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8> has quit IRC (Remote host closed the connection)
148 2020-12-12T02:33:19  *** twistedline_ <twistedline_!~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8> has joined #bitcoin-core-dev
149 2020-12-12T02:33:43  <sipa> achow101: great, so the apple codesign_allocate just behaves differently from their published source code
150 2020-12-12T02:48:42  *** ConsciousFlesh <ConsciousFlesh!~cFlesh@unaffiliated/consciousflesh> has left #bitcoin-core-dev ("If anyone was involved in the RPCHelpMan update I mentioned could you pm me? Had some questions about it's implementation.")
151 2020-12-12T02:49:27  *** twistedline_ <twistedline_!~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8> has quit IRC (Remote host closed the connection)
152 2020-12-12T02:49:54  *** twistedline_ <twistedline_!~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8> has joined #bitcoin-core-dev
153 2020-12-12T02:54:57  *** twistedline_ <twistedline_!~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8> has quit IRC (Remote host closed the connection)
154 2020-12-12T02:55:21  *** twistedline_ <twistedline_!~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8> has joined #bitcoin-core-dev
155 2020-12-12T02:58:58  *** EagleTM <EagleTM!~EagleTM@unaffiliated/eagletm> has quit IRC (Ping timeout: 246 seconds)
156 2020-12-12T02:59:11  <achow101> i guess we shouldn't be that surprised that they do different things
157 2020-12-12T03:24:32  *** twistedline_ <twistedline_!~twisted@2601:14d:8500:a77d:2985:36c3:6c1f:47b8> has quit IRC (Remote host closed the connection)
158 2020-12-12T03:26:43  *** miketwenty1 <miketwenty1!~miketwent@ec2-18-205-136-236.compute-1.amazonaws.com> has joined #bitcoin-core-dev
159 2020-12-12T03:32:25  *** twistedline <twistedline!~twisted@unaffiliated/twistedline> has joined #bitcoin-core-dev
160 2020-12-12T03:52:33  <achow101> sipa: adding the run codesign_allocate twice trick to detached-sig-apply.sh does make rc3 pass signature validation
161 2020-12-12T03:52:53  <sipa> great
162 2020-12-12T03:53:27  <sipa> so we just need something to take a size from the code-signatures.tgz
163 2020-12-12T03:53:41  <sipa> and modify jonas' script to compute what the right pre-allocate size is
164 2020-12-12T03:53:54  <achow101> I wonder if maybe it would be easier to codesign_allocate the unsigned dmg with a larger than necessary but fixed sized chunk
165 2020-12-12T03:54:29  <achow101> the gitian signer's codesign allocate shrinks it, but I don't think effects the vmsize
166 2020-12-12T03:54:47  <sipa> looking at the codesign_allocate tool, it just only ever increments the vmsize
167 2020-12-12T03:55:08  <sipa> which is going to be the rounded-up version of the largest value the filesize ever had
168 2020-12-12T03:55:37  <sipa> and whatever we do, we need to replicate the apple tool's behavior
169 2020-12-12T03:55:38  <achow101> right, so we could always overshoot by a lot when we make the unsigned dmg
170 2020-12-12T03:55:47  <sipa> yup
171 2020-12-12T03:55:52  <sipa> that's a possibility
172 2020-12-12T03:56:24  <luke-jr> [20:46:44] <jonasschnelli> is there a way to speeup gitians "Upgrading system, may take a while (log in var/install.log)"? <-- I just run the base VM and update it
173 2020-12-12T03:56:25  <achow101> I'm going to go through history and see what the largest sigs we've had before is
174 2020-12-12T03:57:48  <achow101> any idea if an overly large vmsize has any negative effects?
175 2020-12-12T03:59:48  <sipa> i suspect if we crank up the codesign --pagesize value the sig will be much smaller
176 2020-12-12T04:00:57  <sipa> it signs every page separately
177 2020-12-12T04:01:11  <achow101> oh, is that why the sig is so big and varies a lot
178 2020-12-12T04:12:30  *** davec <davec!~davec@072-183-054-196.res.spectrum.com> has joined #bitcoin-core-dev
179 2020-12-12T04:43:27  *** justanotheruser <justanotheruser!~justanoth@unaffiliated/justanotheruser> has joined #bitcoin-core-dev
180 2020-12-12T04:56:11  *** mj_node <mj_node!~mj_node@> has joined #bitcoin-core-dev
181 2020-12-12T05:00:00  *** Emcy <Emcy!~Emcy@unaffiliated/emcy> has quit IRC (Read error: Connection reset by peer)
182 2020-12-12T05:01:48  *** Emcy <Emcy!~Emcy@unaffiliated/emcy> has joined #bitcoin-core-dev
183 2020-12-12T05:05:41  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has joined #bitcoin-core-dev
184 2020-12-12T05:05:41  <bitcoin-git> [bitcoin] pox opened pull request #20635: fix misleading comment about call to non-existing function (master...master) https://github.com/bitcoin/bitcoin/pull/20635
185 2020-12-12T05:05:42  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has left #bitcoin-core-dev
186 2020-12-12T05:08:36  *** dhruvm <dhruvm!~noreply@c-73-158-59-66.hsd1.ca.comcast.net> has quit IRC (Ping timeout: 240 seconds)
187 2020-12-12T05:17:26  *** r2wj <r2wj!~r2wj@> has quit IRC (Remote host closed the connection)
188 2020-12-12T05:18:15  *** fearbeag <fearbeag!~sseanicid@bras-base-clwdon2201w-grc-18-216-209-44-58.dsl.bell.ca> has joined #bitcoin-core-dev
189 2020-12-12T05:23:52  *** flag <flag!~flag@net-93-66-71-105.cust.vodafonedsl.it> has quit IRC (Quit: leaving)
190 2020-12-12T05:25:43  *** sledges <sledges!~sledges@> has joined #bitcoin-core-dev
191 2020-12-12T05:31:28  *** flag <flag!~flag@net-93-66-71-105.cust.vodafonedsl.it> has joined #bitcoin-core-dev
192 2020-12-12T05:33:03  *** fearbeag <fearbeag!~sseanicid@bras-base-clwdon2201w-grc-18-216-209-44-58.dsl.bell.ca> has quit IRC (Ping timeout: 260 seconds)
193 2020-12-12T06:22:26  *** miketwenty1 <miketwenty1!~miketwent@ec2-18-205-136-236.compute-1.amazonaws.com> has quit IRC (Remote host closed the connection)
194 2020-12-12T06:44:30  *** mj_node <mj_node!~mj_node@> has quit IRC (Remote host closed the connection)
195 2020-12-12T06:44:50  *** mj_node <mj_node!~mj_node@> has joined #bitcoin-core-dev
196 2020-12-12T07:00:19  <jonasschnelli> sipa, achow101: I'm reading the backlog... for your info,... I haven't changed anything on my signing computer since signing rc2 (which was successful).
197 2020-12-12T07:01:12  <achow101> jonasschnelli: good to know. the backlog is quite a journey..
198 2020-12-12T07:01:21  <jonasschnelli> AFAIK detached-sig-create.sh under the hood, it uses the macOS codesign tool (which I haven't changed)
199 2020-12-12T07:01:32  <jonasschnelli> achow101: yeah.. halfway through.
200 2020-12-12T07:01:35  <jonasschnelli> Very interesting.
201 2020-12-12T07:03:11  <jonasschnelli> 2020-12-11T23:57:47  <sipa> an alternative is making jonasschnelli downgrade his codesign_allocate tool to one that's compatible with the native_ccttols we use, and then tell his codesign tool to use that
202 2020-12-12T07:03:24  <jonasschnelli> ^ as said,... I haven't upgraded anything since rc2
203 2020-12-12T07:04:03  <jonasschnelli> 2020-12-12T00:12:49  <sipa> jonasschnelli: what version is your cctools? if you have the ability to change it, use 949.0.1
204 2020-12-12T07:05:02  <jonasschnelli> I guess sipa wants to know what clang version I have on my signing mac. Why would this be relevant? IMO codesign is not part of the llvm suite
205 2020-12-12T07:05:14  <jonasschnelli> And cctools on the gitian host should be irrelevant
206 2020-12-12T07:05:15  <achow101> cctools is separate from clang
207 2020-12-12T07:05:27  <achow101> it's part of xcode's developer tools
208 2020-12-12T07:06:04  <achow101> jonasschnelli: the tl;dr is that codesign_allocate as distributed by apple is different from codesign_allocate compiled from source
209 2020-12-12T07:06:28  <achow101> codesign uses codesign_allocate under the hood somewhere and that's causing us problems
210 2020-12-12T07:06:40  <jonasschnelli> achow101: thats very interesting...
211 2020-12-12T07:07:46  <sipa> jonasschnelli: can you try whag happens if you pass --pagesize 0 to the codesign tool?
212 2020-12-12T07:08:28  <sipa> that's unrelated, but it may result in smaller ans more predictable signature sizes
213 2020-12-12T07:08:58  <jonasschnelli> sipa: okay... I executed "./detached-sig-create.sh -s "Bitcoin" --pagesize 0"
214 2020-12-12T07:09:14  <sipa> no no, to the codesign tool
215 2020-12-12T07:09:24  <sipa> so inside detached-sig-create.sh
216 2020-12-12T07:09:28  <achow101> sipa: I've figured out what that codesign source I found doesn't have a call to codesign_allocate - the actual signing is way deeper in the stack and isn't open source. the code I found is just the wrapper to expose things as a command line tool
217 2020-12-12T07:09:31  <jonasschnelli> sipa: okay
218 2020-12-12T07:09:39  <sipa> or does it pass everything through?
219 2020-12-12T07:09:45  <achow101> passing extra args to detached-sig-create should pass through to codesign
220 2020-12-12T07:09:52  <sipa> ok
221 2020-12-12T07:10:39  <sipa> achow101: yeah, i tried browsing but i couldn't even find any place where the pagesize argument was used
222 2020-12-12T07:11:23  <jonasschnelli> https://bitcoin.jonasschnelli.ch/signature-osx_pagesize_0.tar.gz
223 2020-12-12T07:11:46  <jonasschnelli> I changed L26 to ${CODESIGN} --pagesize 0 -f --file-list ${TEMPLIST} "$@" "${BUNDLE}"
224 2020-12-12T07:12:25  <jonasschnelli> but why was rc2 fine and rc3 isn't?
225 2020-12-12T07:12:31  <achow101> beats me
226 2020-12-12T07:12:39  <achow101> I think we've found an edge case
227 2020-12-12T07:12:49  <jonasschnelli> as I suspected
228 2020-12-12T07:13:04  <achow101> it's weird that this is not present in the open soure build though
229 2020-12-12T07:13:11  <sipa> jonasschnelli: it appears that the macos tool uses a different segment alignment
230 2020-12-12T07:13:32  <sipa> as your signed result has an alignment of 8192 or 16384 bytes
231 2020-12-12T07:13:45  <sipa> while the gitian result is 4096 byte aligned
232 2020-12-12T07:14:00  <jonasschnelli> okay
233 2020-12-12T07:14:11  <sipa> but the apple (open source) definitely uses 4096
234 2020-12-12T07:14:29  <sipa> it may be that rc2 just accidentally was already 8192 aligned
235 2020-12-12T07:14:37  <sipa> we could check
236 2020-12-12T07:14:53  <sipa> jonasschnelli: how big is the .sign file now?
237 2020-12-12T07:15:13  <achow101> sipa: I checked earlier, it's 7b0000
238 2020-12-12T07:15:54  <sipa> ok, i have no explanation then :)
239 2020-12-12T07:16:04  <jonasschnelli> sipa: jonasschnelli  staff  18496 12 Dez 08:10 Bitcoin-Qt.sign
240 2020-12-12T07:17:52  <sipa> much better!
241 2020-12-12T07:18:12  <sipa> the previous ones were 200 kB or so
242 2020-12-12T07:18:35  <jonasschnelli> I can try to gbuild the macOS signer with that
243 2020-12-12T07:18:54  <sipa> it's worth a try
244 2020-12-12T07:19:13  <sipa> if it was a weird edge case due to alignments, maybe this is enough to fix it
245 2020-12-12T07:26:09  *** kristapsk_ <kristapsk_!~KK@gateway/tor-sasl/kristapsk> has quit IRC (Remote host closed the connection)
246 2020-12-12T07:26:59  *** kristapsk_ <kristapsk_!~KK@gateway/tor-sasl/kristapsk> has joined #bitcoin-core-dev
247 2020-12-12T07:38:59  <achow101> --pagesize 0 might actually make it possible for us to hack together a tool for codesigning
248 2020-12-12T07:39:07  <achow101> that doesn't require macOS
249 2020-12-12T07:39:20  <jonasschnelli> pagesize 0 seems also not to work
250 2020-12-12T07:39:20  <jonasschnelli> https://bitcoin.jonasschnelli.ch/bitcoin-osx-signed_pagesize0.dmg
251 2020-12-12T07:41:38  <achow101> I was going to suggest that we try using the macports codesign_allocate, but the manpage for codesign says "The system will not accept such substitutes unless they are specially signed (by Apple)." on that topic, so I don't think it will work
252 2020-12-12T07:42:02  <sipa> indeed
253 2020-12-12T07:42:25  <sipa> jonasschnelli: same problem?
254 2020-12-12T07:42:39  <sipa> achow101: is it again the vmsize being off?
255 2020-12-12T07:42:53  <achow101> checking
256 2020-12-12T07:43:11  <sipa> jonasschnelli: is that file the signed binary that comes out of the codesign tool?
257 2020-12-12T07:44:05  <jonasschnelli> sipa: no... that file above is the gitian signer result with using the signature.tar.gz that was built with pagesize 0
258 2020-12-12T07:44:20  <achow101> i need to compare against the result of the codesign tool
259 2020-12-12T07:44:27  <sipa> jonasschnelli: we'll also need the codesign output
260 2020-12-12T07:44:41  <jonasschnelli> okay.. one sec
261 2020-12-12T07:46:33  <jonasschnelli> https://bitcoin.jonasschnelli.ch/bitcoin-0.21.0rc3-osx-unsigned-pagesize-0.zip
262 2020-12-12T07:46:38  *** alko89 <alko89!~alko89@unaffiliated/alko89> has quit IRC (Quit: ZNC 1.7.5 - https://znc.in)
263 2020-12-12T07:46:48  *** alko89 <alko89!~alko89@unaffiliated/alko89> has joined #bitcoin-core-dev
264 2020-12-12T07:47:47  <achow101> yes same problem
265 2020-12-12T07:48:03  <achow101> codesign_allocate really likes 7c000 for some reason
266 2020-12-12T07:48:05  <sipa> achow101: what are the vmsize values now?
267 2020-12-12T07:48:18  <sipa> wait what
268 2020-12-12T07:48:38  <sipa> what is the filesize field and the vmsize field in both?
269 2020-12-12T07:48:42  <achow101> gitian is 0x4800 and codesign is 7c000
270 2020-12-12T07:49:08  <sipa> dafuq
271 2020-12-12T07:49:11  <achow101> fileoff and filesize are the same
272 2020-12-12T07:49:21  <achow101> both are otherwise identical except for vmsize
273 2020-12-12T07:49:28  <sipa> and what is that filesize?
274 2020-12-12T07:49:30  * jonasschnelli throws away his mac
275 2020-12-12T07:49:42  <achow101> filesize 293328
276 2020-12-12T07:49:44  <sipa> jonasschnelli: a just decision
277 2020-12-12T07:50:07  <achow101> signature is 18496
278 2020-12-12T07:50:40  <sipa> 0x48000 is the correct vmsize
279 2020-12-12T07:51:05  *** alko89 <alko89!~alko89@unaffiliated/alko89> has quit IRC (Client Quit)
280 2020-12-12T07:51:22  <sipa> well, "correct", from my understanding of what the opensource allocate tool does
281 2020-12-12T07:51:28  <achow101> yes
282 2020-12-12T07:51:34  <sipa> and that's no surprise, because that's also what it did
283 2020-12-12T07:51:52  *** alko89 <alko89!~alko89@unaffiliated/alko89> has joined #bitcoin-core-dev
284 2020-12-12T07:51:55  <sipa> but so this means it is not a rounding issue at all
285 2020-12-12T07:52:11  <sipa> unless it's rounding up to a multiple of 0x7c000 !
286 2020-12-12T07:53:29  <sipa> if you on macos run codesign_allocate infile -a x86_64 18496 -o outfile on the unsigned binary, does it also produce 0x7c000 as vmsize?
287 2020-12-12T07:54:11  <jonasschnelli> let me try
288 2020-12-12T07:55:29  <sipa> jonasschnelli: did you accidentally codesign an already-signed binary?
289 2020-12-12T07:55:52  <jonasschnelli> sipa:.... hmm.. that could be for the latest just done build
290 2020-12-12T07:55:56  <sipa> vmsize is not expected to go down in that case
291 2020-12-12T07:56:13  <sipa> so you have to start with the unsigned binary
292 2020-12-12T07:56:13  *** sr_gi <sr_gi!~sr_gi@> has quit IRC (Read error: Connection reset by peer)
293 2020-12-12T07:56:36  <achow101> vmsize 0x0000000000043188
294 2020-12-12T07:56:36  *** sr_gi <sr_gi!~sr_gi@> has joined #bitcoin-core-dev
295 2020-12-12T07:56:43  <jonasschnelli> I try it again with pagesize 0 and make sure it's not signed already
296 2020-12-12T07:56:49  <achow101> that's not a multple of 0x1000, 0x2000, or 0x4000
297 2020-12-12T07:56:58  <sipa> achow101: wut
298 2020-12-12T07:57:41  <achow101> ah wrong file
299 2020-12-12T07:57:42  <achow101> vmsize 0x0000000000048000
300 2020-12-12T07:57:52  <achow101> that's correct and what we expect
301 2020-12-12T07:58:37  <sipa> achow101: with the macos tool?
302 2020-12-12T07:58:42  <achow101> yes
303 2020-12-12T07:59:34  <sipa> and just to be sure... if you pass 225312 as size?
304 2020-12-12T08:00:08  <sipa> (i know we did that before, but perhaps that before we knew that history of sig sizes the binary has been through mattered)
305 2020-12-12T08:00:11  <achow101> as before vmsize 0x000000000007c000
306 2020-12-12T08:00:17  <sipa> ok
307 2020-12-12T08:00:54  <sipa> well, if it's an alignment issue, 0x48000 is 0x4000 aligned, so unlikely to cause problem
308 2020-12-12T08:01:08  <jonasschnelli> https://bitcoin.jonasschnelli.ch/bitcoin-0.21.0rc3-osx-unsigned-again-pagesize0.zip   <--- signed with pagesize 0, check that is was not already codesigned
309 2020-12-12T08:01:08  <sipa> so i expect this will be a solution
310 2020-12-12T08:01:12  <jonasschnelli> (gitian output follows)
311 2020-12-12T08:02:50  <achow101> sipa: it's aligning against 0x2000
312 2020-12-12T08:03:37  <sipa> achow101: you found that by trying more sizes?
313 2020-12-12T08:03:45  <achow101> if I use 4096 as the size, I get 46000. same with 8192
314 2020-12-12T08:04:04  <jonasschnelli> and if anyone wants to test 0.20.2rc1, I just created the detached signature (no gitian build yet):  https://bitcoin.jonasschnelli.ch/signature-osx_0.20.2rc1.tar.gz
315 2020-12-12T08:04:05  <sipa> ok
316 2020-12-12T08:04:18  <sipa> achow101: we could patch out cctools to do the same
317 2020-12-12T08:04:21  <sipa> *our
318 2020-12-12T08:04:32  <sipa> it's a tiny change
319 2020-12-12T08:04:47  <achow101> I'm not optimistic that that's a lasting solution
320 2020-12-12T08:05:55  <sipa> yeah, agree
321 2020-12-12T08:06:27  <sipa> but it's simpler than trying to hack around the issue with multiple preallocate calls and passing the size through
322 2020-12-12T08:07:19  <achow101> https://github.com/achow101/bitcoin/commit/26f8ebcca26eb74b6546b5d551f3ec468e2878be is my current proposal
323 2020-12-12T08:07:27  <achow101> if we combine with pagesize 0, then it'll probably always work
324 2020-12-12T08:08:10  <achow101> the largest signature since 0.12.1 is 407424
325 2020-12-12T08:10:55  <sipa> achow101: if the gitian preallocate takes it to a large *odd* multiple of 0x1000, will the macos sign tool not increase it further to a multiple of 0x2000 ?
326 2020-12-12T08:11:23  <sipa> if it's anything like the opensource version, it won't
327 2020-12-12T08:11:39  <sipa> but who knows what it does
328 2020-12-12T08:12:20  <jonasschnelli> oh!
329 2020-12-12T08:12:30  <jonasschnelli> pagesize 0 result in a valid signatue (through gitian)
330 2020-12-12T08:12:43  <jonasschnelli> I didn't before because I signed an already signed binary
331 2020-12-12T08:13:07  <jonasschnelli> https://bitcoin.jonasschnelli.ch/bitcoin-osx-signed_pagesize0_clean.dmg
332 2020-12-12T08:13:33  <jonasschnelli> (used detach signature from https://bitcoin.jonasschnelli.ch/bitcoin-0.21.0rc3-osx-unsigned-again-pagesize0.zip )
333 2020-12-12T08:13:52  <achow101> I think that is currently the expected result
334 2020-12-12T08:14:12  <sipa> jonasschnelli: yeah, matches our understanding
335 2020-12-12T08:14:47  <achow101> sipa: it does not appear to shrink the vmsize at all
336 2020-12-12T08:15:07  <sipa> achow101: yes, but it may increase it further
337 2020-12-12T08:15:09  *** tripleslash <tripleslash!~triplesla@unaffiliated/imsaguy> has quit IRC (Ping timeout: 260 seconds)
338 2020-12-12T08:15:26  <sipa> (i don't expect it to, but we should check i think)
339 2020-12-12T08:15:50  <achow101> i'll try against the macports cctools
340 2020-12-12T08:16:08  *** tripleslash <tripleslash!~triplesla@unaffiliated/imsaguy> has joined #bitcoin-core-dev
341 2020-12-12T08:19:30  <achow101> it preserves it
342 2020-12-12T08:20:15  <sipa> great
343 2020-12-12T08:20:52  <sipa> 500000 is probably overkill if we're going to use pagesize 0
344 2020-12-12T08:21:21  <sipa> but that does like a pretty simple fix
345 2020-12-12T08:21:38  <achow101> probably, but I also have no idea what goes into the signature size
346 2020-12-12T08:22:00  <achow101> i'll open a pr when i wake up
347 2020-12-12T08:22:13  <jonasschnelli> thanks achow101!
348 2020-12-12T08:22:15  <jonasschnelli> and sipa
349 2020-12-12T08:22:51  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has joined #bitcoin-core-dev
350 2020-12-12T08:22:53  <bitcoin-git> [bitcoin] MarcoFalke pushed 6 commits to master: https://github.com/bitcoin/bitcoin/compare/6a4806367177...ffc4d0499020
351 2020-12-12T08:22:53  <bitcoin-git> bitcoin/master 5aaeb6c Russell Yanofsky: MOVEONLY: Move IsBDBFile, IsSQLiteFile, and ListWalletDir
352 2020-12-12T08:22:54  <bitcoin-git> bitcoin/master 6ee9cbd Russell Yanofsky: refactor: Replace ListWalletDir() function with ListDatabases()
353 2020-12-12T08:22:55  <bitcoin-git> bitcoin/master 6a7a636 Russell Yanofsky: refactor: Drop call to GetWalletEnv in wallet salvage code
354 2020-12-12T08:22:57  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has left #bitcoin-core-dev
355 2020-12-12T08:23:02  <jonasschnelli> I knew that when I'm going to wake up that you guys will have a solution
356 2020-12-12T08:23:09  <achow101> lol
357 2020-12-12T08:23:11  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has joined #bitcoin-core-dev
358 2020-12-12T08:23:11  <bitcoin-git> [bitcoin] MarcoFalke merged pull request #20275: wallet: List all wallets in non-SQLite and non-BDB builds (master...pr/exist) https://github.com/bitcoin/bitcoin/pull/20275
359 2020-12-12T08:23:12  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has left #bitcoin-core-dev
360 2020-12-12T08:24:13  <sipa> achow101: number of pages, and "resources" are counted separately
361 2020-12-12T08:24:39  <sipa> with pagesize=0 the executable code is always 1 page
362 2020-12-12T08:27:02  <sipa> so i expect it to be mostly constant with pagesize=0
363 2020-12-12T08:34:00  <jonasschnelli> I'm also getting an invalid signature for 0.20.2rc1
364 2020-12-12T08:34:12  <jonasschnelli> shall I codesign with --pagesize 0?
365 2020-12-12T08:35:12  <sipa> jonasschnelli: that may work (50% chance)
366 2020-12-12T08:36:17  <sipa> jonasschnelli: basically any pagesize you try (has to be zero or a power of 2) has an independent 50% chance of being succesful with the current tooling
367 2020-12-12T08:36:28  <sipa> with achow's fix it should always work
368 2020-12-12T08:36:41  <sipa> better just wait; it can be backported too
369 2020-12-12T08:38:07  <jonasschnelli> ok
370 2020-12-12T09:40:34  *** mj_node <mj_node!~mj_node@> has quit IRC (Quit: Leaving)
371 2020-12-12T09:49:25  *** Victor_sueca <Victor_sueca!~Victorsue@unaffiliated/victorsueca> has quit IRC (Ping timeout: 264 seconds)
372 2020-12-12T09:52:51  *** eugene-ff <eugene-ff!~eugene_ff@2604:2000:1383:472b:759a:a9e9:8a19:69d2> has quit IRC (Remote host closed the connection)
373 2020-12-12T10:04:53  *** kexkey <kexkey!~kexkey@static-198-54-132-158.cust.tzulo.com> has quit IRC (Ping timeout: 260 seconds)
374 2020-12-12T10:19:26  *** Victor_sueca <Victor_sueca!~Victorsue@unaffiliated/victorsueca> has joined #bitcoin-core-dev
375 2020-12-12T10:37:28  *** instagibbs <instagibbs!~greg@061093103011.ctinets.com> has quit IRC (Ping timeout: 246 seconds)
376 2020-12-12T10:42:38  *** Pavlenex <Pavlenex!~Thunderbi@> has joined #bitcoin-core-dev
377 2020-12-12T11:07:42  *** jonatack <jonatack!~jon@> has quit IRC (Ping timeout: 256 seconds)
378 2020-12-12T11:08:12  *** Victor_sueca <Victor_sueca!~Victorsue@unaffiliated/victorsueca> has quit IRC (Ping timeout: 256 seconds)
379 2020-12-12T11:18:28  *** Kenny51Hand <Kenny51Hand!~Kenny51Ha@static.> has joined #bitcoin-core-dev
380 2020-12-12T11:23:34  *** Kenny51Hand <Kenny51Hand!~Kenny51Ha@static.> has quit IRC (Ping timeout: 272 seconds)
381 2020-12-12T11:24:04  *** jonatack <jonatack!~jon@> has joined #bitcoin-core-dev
382 2020-12-12T11:29:37  *** jonatack <jonatack!~jon@> has quit IRC (Ping timeout: 264 seconds)
383 2020-12-12T11:30:03  *** jonatack <jonatack!~jon@> has joined #bitcoin-core-dev
384 2020-12-12T11:35:49  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has joined #bitcoin-core-dev
385 2020-12-12T11:35:51  <bitcoin-git> [bitcoin] MarcoFalke pushed 3 commits to master: https://github.com/bitcoin/bitcoin/compare/ffc4d0499020...b18978066d87
386 2020-12-12T11:35:51  <bitcoin-git> bitcoin/master fad68af MarcoFalke: p2p: Ignore non-version msgs before version msg
387 2020-12-12T11:35:52  <bitcoin-git> bitcoin/master faaad1b MarcoFalke: p2p: Ignore version msgs after initial version msg
388 2020-12-12T11:35:53  <bitcoin-git> bitcoin/master b189780 MarcoFalke: Merge #20079: p2p: Treat handshake misbehavior like unknown message
389 2020-12-12T11:35:55  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has left #bitcoin-core-dev
390 2020-12-12T11:36:09  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has joined #bitcoin-core-dev
391 2020-12-12T11:36:09  <bitcoin-git> [bitcoin] MarcoFalke merged pull request #20079: p2p: Treat handshake misbehavior like unknown message (master...2010-p2pNoVersion) https://github.com/bitcoin/bitcoin/pull/20079
392 2020-12-12T11:36:10  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has left #bitcoin-core-dev
393 2020-12-12T11:40:11  *** sledges <sledges!~sledges@> has quit IRC (Remote host closed the connection)
394 2020-12-12T11:58:27  *** Victor_sueca <Victor_sueca!~Victorsue@unaffiliated/victorsueca> has joined #bitcoin-core-dev
395 2020-12-12T12:11:27  *** Eagle[TM] <Eagle[TM]!~EagleTM@unaffiliated/eagletm> has joined #bitcoin-core-dev
396 2020-12-12T12:18:50  *** Talkless <Talkless!~Talkless@mail.dargis.net> has joined #bitcoin-core-dev
397 2020-12-12T12:21:30  *** Talkless <Talkless!~Talkless@mail.dargis.net> has quit IRC (Client Quit)
398 2020-12-12T12:21:46  *** peterrizzo_ <peterrizzo_!~peterrizz@ool-44c18924.dyn.optonline.net> has joined #bitcoin-core-dev
399 2020-12-12T12:35:03  *** Talkless <Talkless!~Talkless@mail.dargis.net> has joined #bitcoin-core-dev
400 2020-12-12T12:35:10  *** slidercrank1 <slidercrank1!~slidercra@s91904426.blix.com> has joined #bitcoin-core-dev
401 2020-12-12T12:40:47  *** rex4539 <rex4539!~rex4539@balticom-197-78.balticom.lv> has joined #bitcoin-core-dev
402 2020-12-12T13:08:34  *** Victor_sueca <Victor_sueca!~Victorsue@unaffiliated/victorsueca> has quit IRC (Ping timeout: 256 seconds)
403 2020-12-12T13:23:58  *** instagibbs <instagibbs!~greg@061093103011.ctinets.com> has joined #bitcoin-core-dev
404 2020-12-12T13:24:44  *** instagibbs <instagibbs!~greg@061093103011.ctinets.com> has quit IRC (Client Quit)
405 2020-12-12T13:27:04  *** mol <mol!~mol@unaffiliated/molly> has joined #bitcoin-core-dev
406 2020-12-12T13:30:14  *** mol_ <mol_!~mol@unaffiliated/molly> has quit IRC (Ping timeout: 260 seconds)
407 2020-12-12T13:35:50  *** vasild_ <vasild_!~vd@gateway/tor-sasl/vasild> has joined #bitcoin-core-dev
408 2020-12-12T13:35:51  *** vasild <vasild!~vd@gateway/tor-sasl/vasild> has quit IRC (Disconnected by services)
409 2020-12-12T13:35:53  *** vasild_ is now known as vasild
410 2020-12-12T13:49:00  *** eugene-ff <eugene-ff!~eugene_ff@2604:2000:1383:472b:a9f4:e07:a3ac:95af> has joined #bitcoin-core-dev
411 2020-12-12T13:53:27  *** slidercrank1 <slidercrank1!~slidercra@s91904426.blix.com> has quit IRC (Remote host closed the connection)
412 2020-12-12T13:55:59  *** Victorsueca <Victorsueca!~Victorsue@unaffiliated/victorsueca> has joined #bitcoin-core-dev
413 2020-12-12T13:56:42  *** miketwenty1 <miketwenty1!~miketwent@ec2-18-205-136-236.compute-1.amazonaws.com> has joined #bitcoin-core-dev
414 2020-12-12T13:57:21  *** justanotheruser <justanotheruser!~justanoth@unaffiliated/justanotheruser> has quit IRC (Ping timeout: 272 seconds)
415 2020-12-12T13:58:01  *** rh0nj <rh0nj!~rh0nj@> has quit IRC (Remote host closed the connection)
416 2020-12-12T13:59:08  *** rh0nj <rh0nj!~rh0nj@> has joined #bitcoin-core-dev
417 2020-12-12T14:06:27  *** twistedline <twistedline!~twisted@unaffiliated/twistedline> has quit IRC (Remote host closed the connection)
418 2020-12-12T14:06:53  *** twistedline <twistedline!~twisted@unaffiliated/twistedline> has joined #bitcoin-core-dev
419 2020-12-12T14:08:44  *** da39a3ee5e6b4b0d <da39a3ee5e6b4b0d!~da39a3ee5@2403:6200:8876:bbcd:2474:9519:9fb5:8678> has joined #bitcoin-core-dev
420 2020-12-12T14:12:46  *** veox <veox!~veox@> has joined #bitcoin-core-dev
421 2020-12-12T14:18:47  *** kristapsk_ <kristapsk_!~KK@gateway/tor-sasl/kristapsk> has quit IRC (Remote host closed the connection)
422 2020-12-12T14:19:09  *** kristapsk_ <kristapsk_!~KK@gateway/tor-sasl/kristapsk> has joined #bitcoin-core-dev
423 2020-12-12T14:20:26  *** twistedline <twistedline!~twisted@unaffiliated/twistedline> has quit IRC (Ping timeout: 264 seconds)
424 2020-12-12T14:27:37  *** twistedline <twistedline!~twisted@unaffiliated/twistedline> has joined #bitcoin-core-dev
425 2020-12-12T14:36:44  *** peterrizzo_ <peterrizzo_!~peterrizz@ool-44c18924.dyn.optonline.net> has quit IRC (Quit: peterrizzo_)
426 2020-12-12T14:38:37  *** twistedline <twistedline!~twisted@unaffiliated/twistedline> has quit IRC (Ping timeout: 246 seconds)
427 2020-12-12T14:44:14  *** da39a3ee5e6b4b0d <da39a3ee5e6b4b0d!~da39a3ee5@2403:6200:8876:bbcd:2474:9519:9fb5:8678> has quit IRC (Quit: My MacBook has gone to sleep. ZZZzzz…)
428 2020-12-12T14:45:35  *** twistedline <twistedline!~twisted@unaffiliated/twistedline> has joined #bitcoin-core-dev
429 2020-12-12T14:48:12  *** da39a3ee5e6b4b0d <da39a3ee5e6b4b0d!~da39a3ee5@2403:6200:8876:bbcd:2474:9519:9fb5:8678> has joined #bitcoin-core-dev
430 2020-12-12T14:55:36  *** mol <mol!~mol@unaffiliated/molly> has quit IRC (Ping timeout: 240 seconds)
431 2020-12-12T15:04:03  *** Pavlenex <Pavlenex!~Thunderbi@> has quit IRC (Quit: Pavlenex)
432 2020-12-12T15:09:02  *** Victorsueca <Victorsueca!~Victorsue@unaffiliated/victorsueca> has quit IRC (Ping timeout: 256 seconds)
433 2020-12-12T15:22:54  *** fearbeag <fearbeag!~sseanicid@bras-base-clwdon2201w-grc-18-216-209-44-58.dsl.bell.ca> has joined #bitcoin-core-dev
434 2020-12-12T15:27:02  *** miketwenty1 <miketwenty1!~miketwent@ec2-18-205-136-236.compute-1.amazonaws.com> has quit IRC (Remote host closed the connection)
435 2020-12-12T15:29:30  *** darosior <darosior!~darosior@> has quit IRC (Remote host closed the connection)
436 2020-12-12T15:30:03  *** darosior <darosior!~darosior@> has joined #bitcoin-core-dev
437 2020-12-12T15:32:04  *** justanotheruser <justanotheruser!~justanoth@unaffiliated/justanotheruser> has joined #bitcoin-core-dev
438 2020-12-12T15:36:31  *** Guyver2 <Guyver2!Guyver@guyver2.xs4all.nl> has joined #bitcoin-core-dev
439 2020-12-12T15:42:13  *** miketwenty1 <miketwenty1!~miketwent@ec2-18-205-136-236.compute-1.amazonaws.com> has joined #bitcoin-core-dev
440 2020-12-12T15:46:36  *** fearbeag <fearbeag!~sseanicid@bras-base-clwdon2201w-grc-18-216-209-44-58.dsl.bell.ca> has quit IRC (Ping timeout: 240 seconds)
441 2020-12-12T15:48:43  *** miketwenty1 <miketwenty1!~miketwent@ec2-18-205-136-236.compute-1.amazonaws.com> has quit IRC (Remote host closed the connection)
442 2020-12-12T15:48:50  *** da39a3ee5e6b4b0d <da39a3ee5e6b4b0d!~da39a3ee5@2403:6200:8876:bbcd:2474:9519:9fb5:8678> has quit IRC (Quit: My MacBook has gone to sleep. ZZZzzz…)
443 2020-12-12T15:49:17  *** miketwenty1 <miketwenty1!~miketwent@> has joined #bitcoin-core-dev
444 2020-12-12T15:52:38  *** Pavlenex <Pavlenex!~Thunderbi@> has joined #bitcoin-core-dev
445 2020-12-12T15:53:35  *** miketwenty1 <miketwenty1!~miketwent@> has quit IRC (Ping timeout: 258 seconds)
446 2020-12-12T15:56:02  *** Victorsueca <Victorsueca!~Victorsue@unaffiliated/victorsueca> has joined #bitcoin-core-dev
447 2020-12-12T15:57:17  *** belcher_ <belcher_!~belcher@unaffiliated/belcher> has joined #bitcoin-core-dev
448 2020-12-12T15:59:27  *** tryphe <tryphe!~tryphe@unaffiliated/tryphe> has quit IRC (Read error: Connection reset by peer)
449 2020-12-12T15:59:35  *** tryphe_ <tryphe_!~tryphe@unaffiliated/tryphe> has joined #bitcoin-core-dev
450 2020-12-12T16:00:02  *** belcher <belcher!~belcher@unaffiliated/belcher> has quit IRC (Ping timeout: 256 seconds)
451 2020-12-12T16:04:25  *** jonatack <jonatack!~jon@> has quit IRC (Ping timeout: 240 seconds)
452 2020-12-12T16:06:09  <prusnak> do you know if https://github.com/shesek is on IRC?
453 2020-12-12T16:06:14  *** belcher_ is now known as belcher
454 2020-12-12T16:06:23  *** jonatack <jonatack!~jon@> has joined #bitcoin-core-dev
455 2020-12-12T16:06:36  <belcher> prusnak yes hes called shesek
456 2020-12-12T16:06:54  <prusnak> shesek: no such nick
457 2020-12-12T16:07:27  <prusnak> thanks, i was asking because he's offline most probably
458 2020-12-12T16:25:55  <achow101> sipa: I found the codesign source. It's in a library called "Security": https://opensource.apple.com/source/Security/Security-59306.140.5/OSX/libsecurity_codesigning/lib/
459 2020-12-12T16:29:41  <achow101> Would it be worthwhile to write a tool that lets us code sign macOS from linux? Given that we can read the original code signing source, I don't think it would be too difficult to write one.
460 2020-12-12T16:30:27  *** wumpus <wumpus!~ircclient@pdpc/supporter/professional/wumpus> has quit IRC (Read error: Connection reset by peer)
461 2020-12-12T16:32:04  *** rex4539_ <rex4539_!~rex4539@> has joined #bitcoin-core-dev
462 2020-12-12T16:32:06  *** wumpus <wumpus!~ircclient@pdpc/supporter/professional/wumpus> has joined #bitcoin-core-dev
463 2020-12-12T16:33:27  *** rex4539 <rex4539!~rex4539@balticom-197-78.balticom.lv> has quit IRC (Read error: Connection reset by peer)
464 2020-12-12T16:36:49  *** rex4539_ <rex4539_!~rex4539@> has quit IRC (Ping timeout: 260 seconds)
465 2020-12-12T16:40:55  *** fearbeag <fearbeag!~sseanicid@bras-base-clwdon2201w-grc-18-216-209-44-58.dsl.bell.ca> has joined #bitcoin-core-dev
466 2020-12-12T16:42:59  *** jonatack <jonatack!~jon@> has quit IRC (Quit: jonatack)
467 2020-12-12T16:49:44  *** dhruv <dhruv!~dhruv@> has joined #bitcoin-core-dev
468 2020-12-12T16:50:35  *** dhruv <dhruv!~dhruv@> has quit IRC (Client Quit)
469 2020-12-12T16:51:09  *** dhruvm <dhruvm!~dhruv@> has joined #bitcoin-core-dev
470 2020-12-12T16:58:08  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has joined #bitcoin-core-dev
471 2020-12-12T16:58:08  <bitcoin-git> [bitcoin] achow101 opened pull request #20638: Mac codesign fixed alloc (master...mac-codesign-fixed-alloc) https://github.com/bitcoin/bitcoin/pull/20638
472 2020-12-12T16:58:09  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has left #bitcoin-core-dev
473 2020-12-12T17:01:11  *** mol <mol!~mol@unaffiliated/molly> has joined #bitcoin-core-dev
474 2020-12-12T17:05:54  *** jonatack <jonatack!~jon@> has joined #bitcoin-core-dev
475 2020-12-12T17:08:40  *** Victorsueca <Victorsueca!~Victorsue@unaffiliated/victorsueca> has quit IRC (Ping timeout: 256 seconds)
476 2020-12-12T17:10:16  *** jonatack <jonatack!~jon@> has quit IRC (Ping timeout: 240 seconds)
477 2020-12-12T17:10:42  *** jonatack <jonatack!~jon@> has joined #bitcoin-core-dev
478 2020-12-12T17:17:43  *** dermoth_ <dermoth_!~dermoth@unaffiliated/dermoth> has joined #bitcoin-core-dev
479 2020-12-12T17:18:02  *** dermoth <dermoth!~dermoth@unaffiliated/dermoth> has quit IRC (Disconnected by services)
480 2020-12-12T17:18:04  *** dermoth_ is now known as dermoth
481 2020-12-12T17:25:49  *** davterra <davterra!~davterra@> has joined #bitcoin-core-dev
482 2020-12-12T17:30:17  *** Pavlenex <Pavlenex!~Thunderbi@> has quit IRC (Quit: Pavlenex)
483 2020-12-12T17:38:35  <sipa> achow101: maybe, depending on how hard it is to compile that for linux
484 2020-12-12T17:40:24  *** owowo <owowo!~ovovo@unaffiliated/ovovo> has quit IRC (Ping timeout: 256 seconds)
485 2020-12-12T17:40:58  *** fearbeag <fearbeag!~sseanicid@bras-base-clwdon2201w-grc-18-216-209-44-58.dsl.bell.ca> has quit IRC (Ping timeout: 256 seconds)
486 2020-12-12T17:45:22  *** owowo <owowo!~ovovo@unaffiliated/ovovo> has joined #bitcoin-core-dev
487 2020-12-12T17:58:58  <wumpus> achow101: nice!
488 2020-12-12T18:01:43  <achow101> sipa: I don't think we can compile it ourselves. It would have to be a completely separate implementation.
489 2020-12-12T18:02:31  <sipa> achow101: that sounds nontrivial
490 2020-12-12T18:02:58  <wumpus> assuming, of course, it's the same as the binary they ship
491 2020-12-12T18:03:39  <sipa> the benefit is that if someone actually makes that work, we may be able to replace it with multiparty RSA or s
492 2020-12-12T18:03:40  <achow101> it appears to be nontrivial. the code is a bit hard to understand
493 2020-12-12T18:03:42  <sipa> o
494 2020-12-12T18:04:05  <achow101> I just wanted to make it use rfc6979
495 2020-12-12T18:04:07  <wumpus> if it's RSA in the first place
496 2020-12-12T18:04:21  <achow101> the cert is rsa
497 2020-12-12T18:05:43  <achow101> it's amazing how hard it is to actually find the function that does the cryptographic signature
498 2020-12-12T18:07:01  <sipa> i assume the signature contais a certnl chain too
499 2020-12-12T18:07:08  <wumpus> you'd say if you're going to do something like per-page signing you'd use an algorithm with short signatures :)
500 2020-12-12T18:07:16  <sipa> which may complicate thihgs
501 2020-12-12T18:07:39  <sipa> and the code for deciding exactly is being signed may be nontrivial too
502 2020-12-12T18:07:58  <wumpus> that explains why no one else made a portable signer i guess
503 2020-12-12T18:08:29  <achow101> I'm surprised no one has put in the effort to make one given that it's actually open source
504 2020-12-12T18:08:31  <sipa> i wonder how many people even care about doing codesigning macos binaries on a non-macos platform
505 2020-12-12T18:09:15  <wumpus> true, not that many projects are doing deterministic builds
506 2020-12-12T18:10:25  <wumpus> though even for non-deterministic builds it can often be useful to do all builds on a single build host platform
507 2020-12-12T18:11:10  *** mol_ <mol_!~mol@unaffiliated/molly> has joined #bitcoin-core-dev
508 2020-12-12T18:12:32  <achow101> even with per-page signing, shouldn't the signature be way larger than 200k?
509 2020-12-12T18:12:35  <achow101> given our binary size
510 2020-12-12T18:13:55  <sipa> what is the actual binary size without resources?
511 2020-12-12T18:14:24  <sipa> i think jonasschnelli's tgz has a file that lists all the resources
512 2020-12-12T18:14:28  *** mol <mol!~mol@unaffiliated/molly> has quit IRC (Ping timeout: 265 seconds)
513 2020-12-12T18:15:12  <achow101> 26891600 Dec 12 11:40 Bitcoin-Qt
514 2020-12-12T18:16:51  <sipa> do you know how many bits the RSA key is?
515 2020-12-12T18:17:48  <achow101> hmm. the cert is not in contrib/macdeploy.. I thought it was
516 2020-12-12T18:17:49  <sipa> achow101: perhaps you can find in that apple source code what the default page size is
517 2020-12-12T18:19:34  <sipa> perhaps the default page size is 32k or 64k
518 2020-12-12T18:19:35  <achow101> turns out codesign has a lot of --verbose options
519 2020-12-12T18:19:42  <achow101> the page size is 4096 in the rc3 sig
520 2020-12-12T18:19:56  <sipa> hmm
521 2020-12-12T18:20:02  <sipa> that makes little sense
522 2020-12-12T18:21:01  <achow101> here is the full verbose output of codesign -d: https://0bin.net/paste/2gi9BEF9#YOEGkrrhbpToojZbGKJhHvcYp5H1zV3MfO+skMBRotg
523 2020-12-12T18:21:07  <sipa> well that"s 35 bytes per sig
524 2020-12-12T18:21:13  <achow101> it looks like it's actually just a ton of hashes and the signature is over all of them?
525 2020-12-12T18:21:20  <achow101> it says Signature size=8973
526 2020-12-12T18:21:21  <sipa> ah!
527 2020-12-12T18:21:35  <sipa> it's a merkle tree over the binary?
528 2020-12-12T18:21:54  <sipa> that makes sens
529 2020-12-12T18:22:24  <achow101> couldn't say. but 6463 * 32 is in the ballpark of the signature size
530 2020-12-12T18:23:00  <achow101> and there's 6432 hashes. that also fits with the page size and binary size, 6432 * 4096 is near the binary size
531 2020-12-12T18:23:50  <achow101> *6463 hashes
532 2020-12-12T18:24:26  <sipa> yup
533 2020-12-12T18:24:34  <sipa> i suspect that's it exactly
534 2020-12-12T18:24:54  <achow101> making hashes of each page seems excessive though
535 2020-12-12T18:25:06  <sipa> it allows for fast per-page validation
536 2020-12-12T18:25:27  <sipa> without needing to hash all the leaves every time
537 2020-12-12T18:28:29  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has joined #bitcoin-core-dev
538 2020-12-12T18:28:29  <bitcoin-git> [bitcoin] jbampton opened pull request #20639: doc: fix case of GitHub (master...fix-case-of-github) https://github.com/bitcoin/bitcoin/pull/20639
539 2020-12-12T18:28:30  *** bitcoin-git <bitcoin-git!~bitcoin-g@x0f.org> has left #bitcoin-core-dev
540 2020-12-12T18:32:48  <achow101> I don't think it's a merkle tree. I'm pretty sure it's just a flat list of hashes
541 2020-12-12T18:33:01  <achow101> that's what gets hashes again and signed
542 2020-12-12T18:35:50  <wumpus> that makes sense
543 2020-12-12T18:36:36  <wumpus> a hash would be much more efficient to verify per page, after all, and the signature check can be done once at load time for the hashes section
544 2020-12-12T18:37:39  <wumpus> per page (or bunch-of-pages) makes sense as it decreases the granularity of loading, a hash over the entire binary means the entire binary has to be read
545 2020-12-12T18:38:17  <achow101> so our trick to do a single page may not be ideal
546 2020-12-12T18:38:32  <wumpus> well if it works as a workaround
547 2020-12-12T18:38:50  <wumpus> but not in the long term no i guess
548 2020-12-12T18:39:28  <achow101> well we now know mostly how big the signature will be, so we could compute an overestimate instead
549 2020-12-12T18:40:08  <sipa> achow101: why not? do we care about not loading all pages all at once?
550 2020-12-12T18:40:13  <wumpus> if someone wants to reverse engineer this i'd prefer having a signature *verifier* first, so the gitian process can detect if it went wrong
551 2020-12-12T18:40:51  <wumpus> sipa: i guess so? there's always a lot of data in a binary that's not 'hot', might never be accessed at all, especially as we static link qt
552 2020-12-12T18:41:17  <achow101> i dunno
553 2020-12-12T18:41:29  <achow101> it seems like there's a default for a reason
554 2020-12-12T18:41:51  <achow101> and it's clearly architecture optimized
555 2020-12-12T18:41:53  <wumpus> that said, given the memory requiremetns i'm not sure we care about paging much
556 2020-12-12T18:42:24  <achow101> verification of the signature itself should only happen once
557 2020-12-12T18:42:41  <sipa> oh you think it's re-verifying the pages every time they get loaded again from disk
558 2020-12-12T18:42:43  <wumpus> yes, but the hash might be done more often, say if a page is paged out and in again?
559 2020-12-12T18:42:44  <achow101> I would guess this is some kind of runtime integrity check
560 2020-12-12T18:42:44  <sipa> that does make sense
561 2020-12-12T18:42:59  <sipa> but just the hash, not the full sig
562 2020-12-12T18:43:03  <achow101> yeah
563 2020-12-12T18:43:23  <wumpus> yes, the sig verification for the hashes section is likely done only once
564 2020-12-12T18:43:30  <sipa> but that means that the list of hashes must be loaded at all times
565 2020-12-12T18:43:43  <wumpus> yes
566 2020-12-12T18:44:45  <wumpus> so it'd be a compromise--too small page size and the list of hashes is huge, too big page size and it has very large paging granularity
567 2020-12-12T18:46:23  <wumpus> and i'm not sure--if it wouldn't check every time a page is re-loaded from disk it might open up some kinds of exploits, espeically for network file systems if manipulation of the data can happen without the OS knowing
568 2020-12-12T18:47:27  <wumpus> but that's not certain
569 2020-12-12T18:48:05  <sipa> if it was a Merkle tree instead they wouldn't need to keep the hash lish loaded, but it'd be twice the size
570 2020-12-12T18:52:09  <achow101> I think we can produce a good enough overestimate to allow for the default page size
571 2020-12-12T18:53:09  <achow101> (((unsigned_size / 4096) + 1) * 32) + 50000
572 2020-12-12T18:53:30  <sipa> yup
573 2020-12-12T18:53:38  <wumpus> good!
574 2020-12-12T18:53:39  <achow101> the sig is 9k, and there's an additional 10k of overhead. with + 50k, we get 30k of extra
575 2020-12-12T18:54:21  <achow101> i'll update the pr
576 2020-12-12T18:55:45  <phantomcircuit> it seems to be using https://en.wikipedia.org/wiki/Cryptographic_Message_Syntax
577 2020-12-12T18:56:17  <achow101> oh!
578 2020-12-12T18:56:26  <achow101> so openssl can be used?
579 2020-12-12T18:56:55  *** adiabat <adiabat!~adiabat@> has quit IRC (Ping timeout: 272 seconds)
580 2020-12-12T18:57:13  *** adiabat <adiabat!~adiabat@> has joined #bitcoin-core-dev
581 2020-12-12T18:59:08  <phantomcircuit> achow101, well to quote a comment 'Sign a Mach-O binary, using liberal dollops of that special Mach-O magic sauce.'
582 2020-12-12T18:59:09  <phantomcircuit> so
583 2020-12-12T18:59:23  <achow101> when I read that, I was not encouraged
584 2020-12-12T18:59:57  <sipa> how to construct the Xcode extract for the 0.20 and 0.19 builds?
585 2020-12-12T19:00:21  <achow101> sipa: see contrib/macdeploy/README in the 0.20 branch
586 2020-12-12T19:00:26  <achow101> it explains how for both
587 2020-12-12T19:00:59  <wumpus> if the download is still available on the apple site, hopefully
588 2020-12-12T19:01:14  <sipa> https://github.com/bitcoin-core/docs/blob/master/gitian-building/gitian-building-mac-os-sdk.md seems to refer to very old stuff
589 2020-12-12T19:01:24  <sipa> i guess it should be updated or deleted?
590 2020-12-12T19:01:33  <sipa> or changed into a link to the macdeploy stuff?
591 2020-12-12T19:01:39  <achow101> I think it should link to macdeploy
592 2020-12-12T19:02:08  <wumpus> yea the information on a branch should probably only describe for that branch
593 2020-12-12T19:02:47  <achow101> for some reason the 0.19 branch doesn't describe the extract process, but 0.20 does as "the old way"
594 2020-12-12T19:03:29  <phantomcircuit> achow101, yeha i cant figure out where it's actually signing either
595 2020-12-12T19:03:52  <phantomcircuit> im sure if i could download that entire directory from git or as a tar ball i could
596 2020-12-12T19:04:02  <phantomcircuit> but no
597 2020-12-12T19:04:13  *** Zenton <Zenton!~user@unaffiliated/vicenteh> has quit IRC (Read error: Connection reset by peer)
598 2020-12-12T19:04:13  <phantomcircuit> classic apple
599 2020-12-12T19:04:56  <achow101> phantomcircuit: mirror of it here https://github.com/apple-opensource/Security/
600 2020-12-12T19:05:11  <achow101> slightly older version, but close enough
601 2020-12-12T19:06:06  <achow101> I got to SecCodeSigner::Signer::signCodeDirectory in signer.cpp before I got super lost
602 2020-12-12T19:06:45  <phantomcircuit> achow101, yeah that's definitely the right function
603 2020-12-12T19:07:22  <phantomcircuit> i got to CMSEncoderAddSigners but can't tell what it's actually doing
604 2020-12-12T19:07:29  <phantomcircuit> at least i think that's a decent place to look
605 2020-12-12T19:08:07  <phantomcircuit> also, if(hashDict != NULL) {assert(hashDict != NULL); they must not trust their compiler very much
606 2020-12-12T19:08:17  *** Zenton <Zenton!~user@unaffiliated/vicenteh> has joined #bitcoin-core-dev
607 2020-12-12T19:08:27  <achow101> CMSEncoderCopyEncodedContent claims to do the actual signing, but I find no evidence that supports that. it just kinda goes into a while function stack that I can't understand
608 2020-12-12T19:08:37  <achow101> *wild
609 2020-12-12T19:09:45  <achow101> CMSEncoderAddSigners just adds some kind of signer object to a list
610 2020-12-12T19:10:08  *** Zenton <Zenton!~user@unaffiliated/vicenteh> has quit IRC (Read error: Connection reset by peer)
611 2020-12-12T19:11:03  <phantomcircuit> achow101, CMSEncoderUpdateContent
612 2020-12-12T19:13:30  <phantomcircuit> wait this looks like it's generating hashes and then encrypting them???
613 2020-12-12T19:13:46  <achow101> I think they use encrypt and sign interchangeably
614 2020-12-12T19:14:13  *** joelklabo <joelklabo!~textual@108-196-216-127.lightspeed.sntcca.sbcglobal.net> has quit IRC (Quit: Textual IRC Client: www.textualapp.com)
615 2020-12-12T19:14:43  <phantomcircuit> achow101, https://github.com/apple-opensource/Security/blob/3aaa9af09496be26b5c15ff011e3d65ca75821ac/libsecurity_smime/lib/cmsencode.c#L389
616 2020-12-12T19:14:51  <phantomcircuit> iono kinda looks like it's actually encrypting things
617 2020-12-12T19:15:25  <phantomcircuit> of course there's about 10 levels of indirection though so who can even tell
618 2020-12-12T19:15:41  <achow101> I think it's just some more generic code that can also handle encryption?
619 2020-12-12T19:15:44  <achow101> honstly it's hard to know
620 2020-12-12T19:16:31  <sipa> for RSA encrypting and signing are kind-of interchangeable (but the devil is in the details)
621 2020-12-12T19:18:32  *** joelklabo <joelklabo!~textual@108-196-216-127.lightspeed.sntcca.sbcglobal.net> has joined #bitcoin-core-dev
622 2020-12-12T19:20:20  <phantomcircuit> sipa, just gotta figure out where the ciphcx object is set.. which is uhhh
623 2020-12-12T19:21:31  *** Zenton <Zenton!~user@unaffiliated/vicenteh> has joined #bitcoin-core-dev
624 2020-12-12T19:29:18  <sipa> ok, downloading xocde 10.2.1
625 2020-12-12T19:29:21  <sipa> *xcode
626 2020-12-12T19:57:50  *** eugene-ff <eugene-ff!~eugene_ff@2604:2000:1383:472b:a9f4:e07:a3ac:95af> has quit IRC (Remote host closed the connection)
627 2020-12-12T20:04:50  <wumpus> can you see what hash algorithm they use? if so it might be most straightforward to check if the hashes are encrypted or otherwise obfuscating by hashing pages and looking if they appear in the signature section of the file
628 2020-12-12T20:06:04  <wumpus> pages are bound to have an address aligned to the start of the file, otherwise it'd be wildly inefficient
629 2020-12-12T20:08:31  *** eugene-ff <eugene-ff!~eugene_ff@2604:2000:1383:472b:a9f4:e07:a3ac:95af> has joined #bitcoin-core-dev
630 2020-12-12T20:13:18  *** eugene-ff <eugene-ff!~eugene_ff@2604:2000:1383:472b:a9f4:e07:a3ac:95af> has quit IRC (Ping timeout: 260 seconds)
631 2020-12-12T20:15:43  <achow101> they use sha256, so it shouldn't be hard to check
632 2020-12-12T20:21:24  *** fearbeag <fearbeag!~sseanicid@bras-base-clwdon2201w-grc-18-216-209-44-58.dsl.bell.ca> has joined #bitcoin-core-dev
633 2020-12-12T20:27:02  *** sr_gi <sr_gi!~sr_gi@> has quit IRC (Read error: Connection reset by peer)
634 2020-12-12T20:27:26  *** sr_gi <sr_gi!~sr_gi@> has joined #bitcoin-core-dev
635 2020-12-12T20:37:36  *** fearbeag <fearbeag!~sseanicid@bras-base-clwdon2201w-grc-18-216-209-44-58.dsl.bell.ca> has quit IRC (Ping timeout: 240 seconds)
636 2020-12-12T20:42:33  <achow101> The hashes are definitely embedded
637 2020-12-12T20:43:05  <achow101> From the 2nd page to the end of the file, the hashes in the signed binary match the unsigned one
638 2020-12-12T20:43:26  <achow101> the 2nd page contains the alloc thing, so it's different, but if the alloc is done, it matches
639 2020-12-12T20:43:55  <achow101> only the first page doesn't match. and then there are 3 "special" hashes that are of... something else
640 2020-12-12T20:45:50  <wumpus> achow101: oh that's great!
641 2020-12-12T20:46:07  <achow101> actually the first page matches, it just doesn't match the unsigned one
642 2020-12-12T20:47:19  <wumpus> maybe it updates something in the header first
643 2020-12-12T20:50:03  <sipa> well, we know it does :)
644 2020-12-12T20:50:12  <sipa> if it didn't, this whole problem wouldn't exist
645 2020-12-12T20:50:27  <sipa> the vmsize/filesize entries on the __LINKEDIT section are changed
646 2020-12-12T20:50:56  <wumpus> yeah then it makes sense for the hash to match the new state, not the original one
647 2020-12-12T20:51:01  <wumpus> right, true :)
648 2020-12-12T20:58:47  <achow101> yep, first 2 pages differ from unsigned because of the addition of the code signing section
649 2020-12-12T20:59:21  <achow101> so the hashes in the sig are 4096 byte pages from beginning of the signed binary to the beginning of the code signature
650 2020-12-12T20:59:29  <achow101> this is something we can verify ourselves
651 2020-12-12T21:07:58  *** jonatack <jonatack!~jon@> has quit IRC (Ping timeout: 256 seconds)
652 2020-12-12T21:08:58  <sipa> ah, neat
653 2020-12-12T21:09:57  <achow101> i figured out 2 of the 3 special hashes
654 2020-12-12T21:10:51  *** eugene-ff <eugene-ff!~eugene_ff@2604:2000:1383:472b:25fd:4b36:9c2b:340> has joined #bitcoin-core-dev
655 2020-12-12T21:10:51  *** eugene-ff <eugene-ff!~eugene_ff@2604:2000:1383:472b:25fd:4b36:9c2b:340> has quit IRC (Client Quit)
656 2020-12-12T21:17:41  <wumpus> that's fast :)
657 2020-12-12T21:18:06  <sipa> last one may be some metadata with the signature/hashing parameters?
658 2020-12-12T21:18:56  <achow101> looking at the source, I think it's "entitlements"
659 2020-12-12T21:18:59  <achow101> but I don't know what that means
660 2020-12-12T21:19:26  <sipa> the codesign tool has a way to provide something called entitlements to it
661 2020-12-12T21:21:14  <achow101> one of the hashes is of the file Bitcoin-Qt.app/Contents/_CodeSignature/CodeResources, and the other is Bitcoin-Qt.app/Contents/Info.plist
662 2020-12-12T21:22:02  <achow101> but this last hash is not any of the files in the bundle
663 2020-12-12T21:23:24  <wumpus> if we don't pass "entitlements", maybe it's a hash of empty data?
664 2020-12-12T21:23:49  <achow101> I don't think so
665 2020-12-12T21:24:00  <achow101> The hash is Bitcoin-Qt.app/Contents
666 2020-12-12T21:24:08  <achow101> 1761bc7a0f37e9809acd22c53f42ff21622ea02093c14058e7daa3a466c95d97
667 2020-12-12T21:29:01  <sipa> is this hash the same for other releases?
668 2020-12-12T21:29:13  <sipa> if it's just some constant we don't actually care what it's a hash of
669 2020-12-12T21:34:20  *** astrolince <astrolince!astrolince@gateway/shell/matrix.org/x-pefoseecitekmcnv> has joined #bitcoin-core-dev
670 2020-12-12T21:34:29  <wumpus> right a proto-verifyer could just compare it against the fixed hash and raise alarm if it doesn't match
671 2020-12-12T21:34:44  <achow101> it's the same in 0.20.1, 0.21.0rc2, and 0.21.0rc3
672 2020-12-12T21:35:05  <achow101> afaik we don't have any entitlements so it should just be the hash of an empty something
673 2020-12-12T21:35:54  <sipa> google is of no help
674 2020-12-12T21:39:33  *** fearbeag <fearbeag!~sseanicid@bras-base-clwdon2201w-grc-18-216-209-44-58.dsl.bell.ca> has joined #bitcoin-core-dev
675 2020-12-12T21:40:43  <wumpus> "
676 2020-12-12T21:40:45  <wumpus> The specific entitlements will be embedded in the signature of an application. If you are having trouble, it can help to look at what the signature actually says about the entitlements: $ codesign -d --entitlements - Example.app will show an XML property list similar to the one above.
677 2020-12-12T21:41:04  <achow101> it gives nothing so there are no entitlements
678 2020-12-12T21:41:07  <wumpus> so it's a hash of something that is in the signature itself, that sounds strange but...
679 2020-12-12T21:41:22  <sipa> maybe it's just <xml></xml> or something
680 2020-12-12T21:41:31  <wumpus> I'd expect so
681 2020-12-12T21:42:04  <achow101> the source code has it as a datatype of CFDataRef which I think just means it's a void *
682 2020-12-12T21:42:10  <sipa> wumpus: sounds similar to bitcoin's use of a sighash byte
683 2020-12-12T21:42:18  <sipa> part of the signature itself, and also signed
684 2020-12-12T21:43:11  <wumpus> sipa: ah yes
685 2020-12-12T21:53:40  <achow101> is there such a thing as a generic DER decoder?
686 2020-12-12T21:53:54  <sipa> yes
687 2020-12-12T21:55:22  <sipa> https://github.com/pornin/DDer
688 2020-12-12T21:55:34  *** owowo <owowo!~ovovo@unaffiliated/ovovo> has quit IRC (Ping timeout: 256 seconds)
689 2020-12-12T21:55:44  <sipa> ha, thomas pornin
690 2020-12-12T21:55:46  <sipa> i've met him
691 2020-12-12T21:57:41  <achow101> hmm, openssl should be able to decode this if it was properly CMS DER, but it doesn't like it
692 2020-12-12T21:58:07  <wumpus> it apparently DiskRep::component(cdEntitlementSlot) gets the data that is hashed to verify the entitlement slot, there are different implementations of DispRep though, but in diskrep this is the file "CodeEntitlements"
693 2020-12-12T21:58:20  <wumpus> filediskrep*
694 2020-12-12T21:59:30  *** abetusk <abetusk!~abe@> has joined #bitcoin-core-dev
695 2020-12-12T22:00:17  *** owowo <owowo!~ovovo@unaffiliated/ovovo> has joined #bitcoin-core-dev
696 2020-12-12T22:00:38  <wumpus> in CodeDirectory::slotIsPresent a non-zero digest in the slot means that is is present
697 2020-12-12T22:02:38  <wumpus> i guess i should be looking at BundleDiskRep::component(cdEntitlementSlot) as it can fetch components from the executable itself
698 2020-12-12T22:02:44  <achow101> I found a bunch of ASN.1 templates, this means it should be possible to figure the serialization, right?
699 2020-12-12T22:03:04  <sipa> achow101: can you show me the DER data?
700 2020-12-12T22:03:39  <achow101> sipa: in the bitcoin-detached-sigs repo osx/dist/Bitcoin-Qt.app/Contents/MacOS/Bitcoin-Qt.sign
701 2020-12-12T22:03:45  <achow101> that's the code signature. in theory it's DER
702 2020-12-12T22:03:59  <achow101> (checkout 0.21 branch)
703 2020-12-12T22:03:59  <sipa> you don't actually need the schema to decode DER, it's self-descriptive
704 2020-12-12T22:04:09  <sipa> the scheme just helps with naming stuff
705 2020-12-12T22:04:30  <achow101> I'm not convinced it's actually DER
706 2020-12-12T22:04:59  <wumpus> *i have no clue what i'm doing dog image*
707 2020-12-12T22:05:08  <wumpus> this looks interesting: https://chromium.googlesource.com/experimental/chromium/src/+/refs/wip/bajones/webvr/chrome/browser/safe_browsing/signature_evaluator_mac.mm
708 2020-12-12T22:06:16  <sipa> it's not
709 2020-12-12T22:06:25  <wumpus> ok
710 2020-12-12T22:07:01  <sipa> $ file Bitcoin-Qt.sign
711 2020-12-12T22:07:01  <sipa> Bitcoin-Qt.sign: Mac OS X Detached Code Signature (non-executable) - 216249 bytes
712 2020-12-12T22:07:23  <achow101> wait file recognizes this???
713 2020-12-12T22:09:16  <sipa> hmm where does file keeps its magic definitions these days?
714 2020-12-12T22:09:28  <sipa> i only find a precompiled magic.mgc file
715 2020-12-12T22:09:46  <achow101> I think that's it
716 2020-12-12T22:09:51  <achow101> gotta find the source for file
717 2020-12-12T22:09:54  <wumpus> from MachORep::compoenent to EmbeddedSignatureBlob::component now
718 2020-12-12T22:10:31  <sipa> https://github.com/file/file/blob/master/magic/Magdir/apple
719 2020-12-12T22:11:05  <wumpus> "An EmbeddedSignatureBlob is a SuperBlob indexed by component slot number." of coursee
720 2020-12-12T22:11:53  <wumpus> class EmbeddedSignatureBlob : public SuperBlobCore<EmbeddedSignatureBlob, 0xfade0cc0, uint32_t>
721 2020-12-12T22:13:13  <sipa> _0x0ff: belong0xfade0cc1Mac OS X Detached Code Signature
722 2020-12-12T22:13:48  <sipa> 0 belong 0xfade0cc1 Mac OS X Detached Code Signature
723 2020-12-12T22:14:31  <wumpus> typedef SuperBlob<0xfade0cc1> DetachedSignatureBlob;  // indexed by main architecture
724 2020-12-12T22:14:52  <wumpus> superblobs, all of them
725 2020-12-12T22:15:09  <sipa> turtles all the way down
726 2020-12-12T22:15:15  <wumpus> hehe
727 2020-12-12T22:15:24  <achow101> blobloblob
728 2020-12-12T22:17:58  <wumpus> / as a genuine Blob (e.g. for insertion into a SuperBlob).
729 2020-12-12T22:18:22  <wumpus> a superblob is a map of integers to blobs
730 2020-12-12T22:18:46  <sipa> getting deeper
731 2020-12-12T22:19:27  *** yunier2002 <yunier2002!~yunier200@c-73-85-126-91.hsd1.fl.comcast.net> has joined #bitcoin-core-dev
732 2020-12-12T22:19:44  <wumpus> "// echoes from parent BlobCore (the C++ type system is too restrictive here)" uh
733 2020-12-12T22:21:50  <wumpus> tbh seems like it's actually all fairly straightforward C++ code
734 2020-12-12T22:22:19  <wumpus> the comments are way scarier than the code itself
735 2020-12-12T22:24:41  <sipa> wumpus: you're biased
736 2020-12-12T22:24:44  <sipa> :)
737 2020-12-12T22:25:45  <wumpus> i... suppose hehe
738 2020-12-12T22:26:30  <achow101> i've somehow wandered into some code in this libsecurity that's Copyright Mozilla
739 2020-12-12T22:44:25  *** Guyver2 <Guyver2!Guyver@guyver2.xs4all.nl> has quit IRC (Quit: Going offline, see ya! (www.adiirc.com))
740 2020-12-12T22:54:51  <sipa> so do we know how to extract the blobs from the EmbeddedSignatureBlob ?
741 2020-12-12T23:00:51  <achow101> nope?
742 2020-12-12T23:02:31  *** Talkless <Talkless!~Talkless@mail.dargis.net> has quit IRC (Quit: Konversation terminated!)
743 2020-12-12T23:04:18  <achow101> That unknown hash earlier is apparently "internal requirements", not entitlements
744 2020-12-12T23:11:38  <wumpus> kind of, it's u32 magic (BE), u32 (BE) size, then u32 (LE) number of blobs, then per blob u32 (LE) blob type, u32 LE offset
745 2020-12-12T23:11:47  <wumpus> then the data
746 2020-12-12T23:12:47  <wumpus> blob type is the slot id, so 5 for the cdEntitlementSlot, but apprantly we're no longer looking for that
747 2020-12-12T23:13:41  <sipa> ok so 3 blobs
748 2020-12-12T23:14:26  <sipa> first blob is type 0, length 0x24
749 2020-12-12T23:14:40  <wumpus> which is good as it doesn't have that one
750 2020-12-12T23:14:47  <sipa> ah no, not length; offset
751 2020-12-12T23:15:01  <sipa> the individual blobs don't have a length?
752 2020-12-12T23:15:16  <wumpus> sipa: that seems correct?
753 2020-12-12T23:15:28  <wumpus> yes, offset
754 2020-12-12T23:15:33  <wumpus> the length is in the blob itself
755 2020-12-12T23:15:37  <sipa> the first blob has magic fa de 0c 02
756 2020-12-12T23:15:46  <wumpus> blobs are all <magic> <length>
757 2020-12-12T23:15:57  <sipa> got it
758 2020-12-12T23:16:12  <achow101> list of magics in OSX/libsecurity_codesigning/lib/CSCommonPriv.h
759 2020-12-12T23:16:51  <sipa> Mac OS X Code Directory is the first one
760 2020-12-12T23:16:55  <wumpus> so yes we have 0 at offset 0x24, 2 at offset 0x2f7a4 and 0x100 at offset 0x2f864
761 2020-12-12T23:17:16  <achow101> sipa: yes. this contains our list of hashes
762 2020-12-12T23:17:19  <wumpus> (all integers are BE, it seemed from the code that they'd be LE but apparently they're swapped somewhere)
763 2020-12-12T23:18:04  <achow101> it looks like the structure is the class members
764 2020-12-12T23:18:08  <achow101> in order as listed
765 2020-12-12T23:18:09  <sipa> second one has magic fa de 0c 01, Mac OS X Code Requirement Set
766 2020-12-12T23:18:45  <wumpus> there's your requirement set to hash i guess
767 2020-12-12T23:18:54  <sipa> last blob has magic fa de 0b 01
768 2020-12-12T23:19:57  <wumpus> class BlobWrapper : public Blob<BlobWrapper, 0xfade0b01>
769 2020-12-12T23:20:01  <sipa> the first blob has magic fa de 0c 02
770 2020-12-12T23:20:05  <sipa> https://gist.github.com/nmoinvaz/cd8651ffb3659161423534b29824510a
771 2020-12-12T23:20:22  <achow101> oh nice
772 2020-12-12T23:21:33  <achow101> hmm, it can't be the structure as listed because I run into some strings before I'm supposed to
773 2020-12-12T23:21:38  <achow101> but it's pretty close
774 2020-12-12T23:21:59  <wumpus> euhh the third's slot id is 0x10000 which is cdSignatureSlot = 0x10000,|     |       |       // CMS signature
775 2020-12-12T23:22:19  *** Mercury_Vapor <Mercury_Vapor!~Mercury_V@174-082-166-092.res.spectrum.com> has quit IRC (Remote host closed the connection)
776 2020-12-12T23:22:24  <achow101> that might be DER encoded
777 2020-12-12T23:22:26  <wumpus> which is wrapped in a BlobWrapper which is simply a blob header with arbitrary binary dat
778 2020-12-12T23:22:45  *** Mercury_Vapor <Mercury_Vapor!~Mercury_V@174-082-166-092.res.spectrum.com> has joined #bitcoin-core-dev
779 2020-12-12T23:22:51  *** miketwenty1 <miketwenty1!~miketwent@ec2-34-202-224-110.compute-1.amazonaws.com> has joined #bitcoin-core-dev
780 2020-12-12T23:22:58  <sipa> this has so many layers, i'd be surprised if it isn't somehow exploitable...
781 2020-12-12T23:24:40  <achow101> In class CodeDirectory, everything in the class decodes in order up to spare3
782 2020-12-12T23:25:00  <achow101> then we get to the offset for identifier string and the identifer string begins there, as expected
783 2020-12-12T23:25:31  <wumpus> otoh the blob format is super simple, there could be a mistake of course, but yes e.g. DER parsers are notoriously exploitable
784 2020-12-12T23:26:53  *** Mercury_Vapor <Mercury_Vapor!~Mercury_V@174-082-166-092.res.spectrum.com> has quit IRC (Remote host closed the connection)
785 2020-12-12T23:27:19  *** Mercury_Vapor <Mercury_Vapor!~Mercury_V@174-082-166-092.res.spectrum.com> has joined #bitcoin-core-dev
786 2020-12-12T23:30:09  <wumpus> the CMS signature is produced in SecCodeSigner::Signer::signCodeDirectory in signer.cpp
787 2020-12-12T23:30:27  *** veox <veox!~veox@> has quit IRC (Remote host closed the connection)
788 2020-12-12T23:30:39  <achow101> it seems like the message that is signed is the CodeDirectory blob
789 2020-12-12T23:31:48  <wumpus> that's what the comment says too "Generate the CMS signature for a (finished) CodeDirectory.", in addition it takes a hashDict and hashList
790 2020-12-12T23:32:13  *** Barno7 <Barno7!~bob@> has quit IRC (Remote host closed the connection)
791 2020-12-12T23:38:05  <wumpus> where hashDict is apparently "v2" and hashList for "v1"
792 2020-12-12T23:40:43  <wumpus> the encoding is indeed the Cryptographic Message Syntax what phantomcircuit linked to with apple specific tags
793 2020-12-12T23:49:31  <phantomcircuit> <sipa> this has so many layers, i'd be surprised if it isn't somehow exploitable...
794 2020-12-12T23:49:33  <phantomcircuit> indeed
795 2020-12-12T23:53:08  <achow101> wumpus: that Requirement Set blob is indeed the unknown hash.
796 2020-12-12T23:55:19  *** miketwenty1 <miketwenty1!~miketwent@ec2-34-202-224-110.compute-1.amazonaws.com> has quit IRC (Remote host closed the connection)
797 2020-12-12T23:56:43  *** jeremyrubin <jeremyrubin!~jr@c-73-15-215-148.hsd1.ca.comcast.net> has joined #bitcoin-core-dev
798 2020-12-12T23:56:56  *** fearbeag <fearbeag!~sseanicid@bras-base-clwdon2201w-grc-18-216-209-44-58.dsl.bell.ca> has quit IRC (Ping timeout: 240 seconds)
799 2020-12-12T23:58:00  *** miketwenty1 <miketwenty1!~miketwent@ec2-18-205-136-236.compute-1.amazonaws.com> has joined #bitcoin-core-dev