19:02:28 #startmeeting 19:02:28 Meeting started Thu Oct 29 19:02:28 2020 UTC. The chair is jonasschnelli. Information about MeetBot at http://wiki.debian.org/MeetBot. 19:02:28 Useful Commands: #action #agreed #help #info #idea #link #topic. 19:02:32 hi 19:02:45 hi 19:02:47 hi 19:02:51 lightningbot? 19:02:51 hi 19:02:54 hi 19:03:02 jonasschnelli: Error: Can't start another meeting, one is in progress. 19:03:17 bad bot... 19:03:28 maybe it's dead 19:03:40 he refuses to work for me... 19:03:56 MarcoFalke: Error: Can't start another meeting, one is in progress. 19:04:00 however,... lets start anyways 19:04:01 #bitcoin-core-dev Meeting: wumpus sipa gmaxwell jonasschnelli morcos luke-jr sdaftuar jtimon cfields petertodd kanzure bluematt instagibbs phantomcircuit codeshark michagogo marcofalke paveljanik NicolasDorier jl2012 achow101 meshcollider jnewbery maaku fanquake promag provoostenator aj Chris_Stewart_5 dongcarl gwillen jamesob ken281221 ryanofsky gleb moneyball kvaciral ariard digi_james 19:04:02 amiti fjahr jeremyrubin lightlike emilengler jonatack hebasto jb55 elichai2 19:04:33 no proposed topics today I guess. 19:04:37 hi 19:04:41 two topics 19:04:41 any last-minute ones? 19:04:47 luke-jr: proposed one 19:04:55 [11:02] #proposedmeetingtopic Disable S390x travis build 19:05:05 [19:46] #proposedmeetingtopic allowing sqlite wallet regression into 0.21 19:05:28 jonasschnelli: I had one: Disable S390x travis build 19:05:31 #action someone should update the channel topic: Meeting topics https://gist.github.com/moneyball/071d608fdae217c2a6d7c35955881d8a 19:06:01 but it's not updated? 19:06:01 Okay... lets start with 0.21 milestone first 19:06:02 #topic 0.21 milestone 19:06:08 https://github.com/bitcoin/bitcoin/pulls?q=is%3Aopen+is%3Apr+milestone%3A0.21.0 19:06:19 jonasschnelli: http://gnusha.org/bitcoin-core-dev/proposedmeetingtopics.txt 19:06:20 luke-jr: yeah. someone should change that link 19:06:25 https://github.com/bitcoin/bitcoin/milestone/45 19:06:37 my topic fits into this one :p 19:07:00 #20205 had the milestone removed, but should be a blocker 19:07:03 https://github.com/bitcoin/bitcoin/issues/20205 | wallet: Properly support a wallet id by achow101 · Pull Request #20205 · bitcoin/bitcoin · GitHub 19:07:45 luke-jr: can you elaborate why it is a blocker? What is broken without it? 19:08:00 jonasschnelli: without it, sqlite wallets are missing a unique id 19:08:13 luke-jr: what user function does it break? 19:08:13 jonasschnelli: BDB did this for us, but sqlite does not 19:08:25 why would they need one? 19:08:26 jonasschnelli: not having a unique id is a regression from the old wallets 19:08:34 is ryanofsky here? 19:08:37 MarcoFalke: to distinguish between different wallets and renames/moves/copies 19:08:39 the question is what unique ids do for us and whether not having one is a regression 19:08:46 because we don't use them for anything currently 19:08:55 achow101: Knots does 19:09:02 have we expose the unique ID over our APIs or in the GUI? 19:09:07 jonasschnelli: no 19:09:07 it's an existing and long standing wallet feature 19:09:30 also, we do up until now use them to disallow opening the same wallet twice 19:09:31 it's a "feature" that's a byproduct of bdb and was only used by the bdb handling previously 19:09:38 is it a feature, or a necessary restriction? 19:09:45 I see luke-jr point... seems like a Knots issue too me. Core has never published or promised unique wallet id... however,.. seems to be easy to fix 19:09:48 sipa: the unique id is certainly a feature 19:10:03 I have no strong opinion either way, just that the way the luke-jr originally proposed was not the right way to add an id 19:10:11 A uuid per wallet seems neat and useful. I agree with luke-jr 19:10:13 jonasschnelli: I can't fix it in Knots alone; I would have to simply remove all functionality that uses it 19:10:28 i tend to agree that it's a potentially useful feature 19:10:51 but i don't know if it's a feature right now we need to make promises about 19:10:57 Lets keep an eye on #20205... 19:11:00 https://github.com/bitcoin/bitcoin/issues/20205 | wallet: Properly support a wallet id by achow101 · Pull Request #20205 · bitcoin/bitcoin · GitHub 19:11:00 adding it doesn't require any promioses. 19:11:03 sqlite wallets will work fine if you load 2 at the same time 19:11:10 absolute worst case, we'd just not use it 19:11:19 unsure if it is a 0.21 bugfix or a 0.22 feature. 19:11:26 jonasschnelli: it's a regression 19:11:30 luke-jr: if we never end up using it, it's hard to call it a bugfix now 19:11:40 luke-jr: why do you say you can't fix it in knots? 19:11:42 I guess adding it in 0.22 when we have 0.21 users creating sqlite wallets is not ideal 19:11:42 sipa: the wallet is losing a feature 19:11:44 I think if we have an ID we should use someting that is already pressent. Manybe the first public key. 19:12:06 luke-jr: so what? it's a new type wallet; it has no features except the ones we say it has 19:12:07 emzy: "first" is undefined, especially for descriptor wallets 19:12:13 I guess having sqlite wallets without ID and some with may lead to bugs. 19:12:15 jnewbery: 1) Core wallets will be missing it, and the same problems persist as adding ti later 19:12:18 anyway, i'll comment on thePR 19:12:33 jnewbery: 2) if I do anything to the wallet format in Knots, Core historically refuses to consider compatibility with it 19:12:35 i'm in favor of having it, but i don't think it's fair to call it a bugfi 19:12:46 luke-jr: obviously 19:13:13 I added the 0.21 milestone to 20205 19:13:13 lets continue on GitHub about whether its a bugfix or not 19:13:15 no offense, but what you do in knots isn't relevant in this discussion 19:13:38 sipa: see the question I am answering 19:13:48 ok 19:14:04 anything else that is relevant for the 0.21 milestone? 19:14:20 for 0.20, we backported a fix that never got merged to master 19:14:20 jonasschnelli: Only the things that are tagged 19:14:31 luke-jr: oh? 19:14:38 please help review 0.21: https://github.com/bitcoin/bitcoin/milestone/45 thanks 19:14:40 part of #18818 19:14:41 luke-jr: Yeah, at least we should cherry-pick what was backported 19:14:43 https://github.com/bitcoin/bitcoin/issues/18818 | Fix release tarball generated by gitian by luke-jr · Pull Request #18818 · bitcoin/bitcoin · GitHub 19:14:57 yes 19:15:07 agree 19:15:10 it would be ideal to just get it all merged in tho :p 19:15:20 Though, it isn't a clean cherry-pick IIRC 19:15:44 GitHub doesn't say there's conflicts O.o 19:16:42 #topic Disable S390x travis build (jnewbery) 19:17:35 The S390x builds on travis are very flakey and often fail, presumably because of contention issues at travis 19:17:36 (should we also talk about the travis-ci migration?) 19:17:38 is it just s390x? 19:17:50 all of travis is pretty much broken for the secp256k1 repo 19:17:59 :/ 19:18:02 s390x was the first sign, but soon after, everything stopped 19:18:37 yeah.. I personally tend to ignore the CI icons ons pulls... its getting non-useful 19:18:39 they were added to test under a big-endian platform. I'm not sure how many bugs have been caught by adding it, but I don't think it's worth keeping it given the costs of having to frequently rerun failed builds 19:18:43 Is someone actually using bitcoin core on x390x? 19:18:44 let me check if that's still the case 19:18:53 emzy: it's the only Big Endian platform Travis had 19:19:00 emzy: no, it's just to guarantee big-endian compatibility 19:19:04 s390x seems to be the least reliable 19:19:16 jnewbery: we could have a bot that restarts s390x if the rest pass? 19:19:28 but should we wait and see if .com has this issue? 19:19:50 (which, even if nobody actually uses bitcoin core on a big endian platform, is a good test for the correctness of the code, as it tends to expose implemention-dependency that isn't otherwise exposed) 19:19:54 s390x found at least one bug, so I think it is useful 19:19:58 Travis is only giving us 1 month to move, so.. 19:20:01 CIs are loosing their value if there are a large percentage of false "positives" (false fails) 19:20:11 jonasschnelli: indeed, rapidly 19:20:41 luke-jr: is power9 big endian? 19:20:47 (typically) 19:20:52 sipa: it's bi-endian, but Travis only supports LE afaik 19:21:00 MarcoFalke: was that a new bug that was introduced, or one that had been around for a long time? 19:21:06 luke-jr: on your power9 system, is the OS LE or BE? 19:21:13 sipa: I run LE myself 19:21:18 jnewbery: both. So it was two bugs actually 19:21:21 I still recommend to continue build run our own CI system. bitcoinbuilds.org is a great start, ... runs more or less smooths since months,... can be easly extended and is cheeper than travis 19:21:54 and quicker 19:21:57 jonasschnelli: We need integration with GitHub, otherwise no one will notice a failing build or even check them 19:22:04 MarcoFalke: I have it 19:22:09 Just not on the master branch.... 19:22:18 Will add it to the GUI repo soon 19:22:22 It's not complicated 19:22:22 who runs bitcoinbuilds.org? 19:22:26 luke-jr: me 19:22:52 Also, we need to be able to modify the config in-tree, not out-of-tree 19:23:04 Yes. I'll add that soon. 19:23:27 jonasschnelli: what architecture(s) does it support? 19:23:37 Not saying it can replace travis,... but it may show the road to a successful and fast CI which we can easly maintain ourselfs 19:23:53 can we make it use the GCC compile farm? :P 19:24:02 sipa: its using libvirt under the hood 19:24:13 Can it run qemu-s390x? 19:24:15 jonasschnelli: that doesn't answer.. :P 19:24:36 I think it is plain amd64 architecture 19:25:12 Currently it can only run ubuntu18,... but we could add other servers with other architectures or qemu others (slow) 19:25:21 GCC's farm has also ppc64, aarch64, sparc64, and mipsel 19:25:55 jonasschnelli: can it work with just a shell? 19:26:10 luke-jr: how do you mean that? no web frontend? 19:26:19 jonasschnelli: I mean SSH into GCC's farm and run builds there 19:26:33 would be possible... 19:26:37 contribution welcome 19:26:40 another project is already using it for their CI 19:26:50 which project? 19:26:59 "HansLambermont uses the compile farm for continuous build integration of the Stellarium project." 19:27:02 as said... running and expanding our own CI seems to me the best way forward to finally get back to a CI state that is useful 19:27:49 "Stellarium is a free open source planetarium for your computer. It shows a realistic sky in 3D, just like what you see with the naked eye, binoculars or a telescope." 19:28:01 Our demand of funcionability goes beyond most CI product offerings,... 19:28:02 I find it hard to believe that there isn't a reliable CI service that we can use and that we need to invent our own 19:28:03 jonasschnelli: I'm open to help with it. 19:28:13 emzy: great to hear. 19:28:14 jnewbery: indeed... 19:28:51 jnewbery: me too.. but our demand is non-normal 19:28:55 does jenkins still exist? 19:29:02 self hosted? 19:29:04 jnewbery: All CI services are "lol we only support docker" 19:29:17 MarcoFalke: is that a problem? 19:29:29 sipa: Doesn't jenkins count as running our own? 19:29:35 MarcoFalke: yes 19:29:50 there is a different between running our own and inventing our own :) 19:30:06 sipa: They don't run sanitizers or wine or ... 19:30:10 sipa: it's already invented. It runs since 1 year 19:30:22 (but missed additional features) 19:30:27 *misses 19:30:34 MarcoFalke: i'm confused what that has to do with docker 19:31:03 sure you can find a docker image that contains sanitizer-enabled compilers and/or wine? 19:31:04 sipa: You'll have to start the docker daemon with additional permissions 19:31:05 so as far as Travis goes, are we just not migrating? 19:31:24 i think we should keep our options open 19:31:25 sipa: On travis we can start our own docker daemon 19:31:56 luke-jr: I'd be surprised if travis improved by changing the domain name 19:32:00 MarcoFalke: ok, i probably don't know enough about docker then... i'm not sure what that means or implies 19:32:10 MarcoFalke: well, .com has historically been their commercial product? 19:32:25 things could improve with travis.com... i'm not sure 19:32:44 a question is if that requires giving them write permission to the repo, are we open to doing so? 19:32:55 We used to send them money and support was still ignoring requests 19:32:59 sipa: apparently it's write permissions to the user, not the repo specifically :/ 19:33:19 luke-jr: we can create a dummy bitcoin-core-ci user or something, no? 19:33:36 MarcoFalke: they responded to me (maybe not timely or helpfully, I forget the details) 19:33:54 sipa: maybe, but I think that violates GitHub's ToS 19:34:04 luke-jr: well response is "lol, we don't support that" or "we'll discuss this internally" 19:34:12 MarcoFalke: ☺ 19:34:39 luke-jr: Then DrahtBot would violate GitHub's ToS 19:34:40 Migrations to .com makes sense to me as we have no alternative ready that has the same depths of testing... and it might get better (or worse?) 19:35:24 back to the topic.... should we disable S390x? 19:35:25 MarcoFalke: aha, they have an exception for bots, nm 19:35:38 jonasschnelli: ACK 19:35:45 ack 19:35:51 jonasschnelli: Well, we could move it to cirrus ci for now 19:35:52 if it doesn't work, it should be disabled 19:35:57 ack on disabling 19:36:11 MarcoFalke: Cirrus has BE? :P 19:36:17 in qemu 19:36:21 >_< 19:36:45 They are working on getting native arm though 19:36:46 * luke-jr should try getting x86_64 gitian working on his ppc64 host via qemu… 19:36:59 (apparently I will have to patch qemu) 19:37:17 even just doing BE builds in qemu for master (and not for every PR) would be a major win over not having it at all 19:37:32 same with some other platforms 19:37:34 I'll create a pull to move it to cirrus 19:37:44 that'd be great 19:38:18 Indeed. Building **everything** per each PR push is crazy 19:38:26 (but great if it works) 19:38:52 why is it crazy? It seems optimal if it works 19:38:53 don't agree on crazy :P 19:39:15 s/crazy/crazy-system-demanding 19:39:34 if we have the resources, sure 19:40:00 and it seems like we don't have the resources because large percentages of the builds fail for no reason 19:40:07 cpu time is cheaper than developer time 19:40:10 fail isn't a resource issue 19:40:21 resource shortage would just mean long waits 19:40:26 indeed 19:40:38 jnewbery: agree... but if you can't be sure whether you have to debug the build or if it was just a glitch in the CI,... its a waste of time 19:40:39 we're not low on resources, it's just broken, and we need to find something that works 19:40:48 whatever we find is better than something that's broken 19:40:50 ideally CI would prioritise builds, but otherwise include everything 19:40:58 I don't think resources are the problem. The problem is the lack of ci infrastructure to trigger builds on working hardware, and someone who maintains the ci infrastructure. 19:41:01 eg, put master above PRs 19:41:12 maybe have a button where someone can say "I'm actually waiting on this" 19:41:21 we need a CI-infrastructur-maintainer *duck* 19:41:31 jonasschnelli: I thought you just volunteered? :P 19:41:34 I volunteer jonasschnelli 19:41:39 no way... 19:41:45 it needs to be some linux/server crack 19:41:59 [19:22:22] who runs bitcoinbuilds.org? [19:22:26] luke-jr: me 19:42:02 we'll say that counts 19:42:09 MarcoFalke: I agree. The problem is that we don't have a build manager 19:42:12 I start stuff,... but not finish them 19:42:27 XD 19:42:31 jonasschnelli: i fear this will be a problem if we'd migrate to bitcoinbuilds.org for part or all of our CI 19:42:39 things will break, and will require maintenance 19:42:48 oh crap don't tell me i missed the meeting almost because of the DST change here *facepalm* 19:42:55 sipa: we should certenly not migrate before a group of people have commited time to it 19:42:56 wumpus: lol 19:42:58 wumpus: still 17 minutes left! 19:42:58 I think all these problems are solvable by someone who has the time and desire to figure out the problems. Part of that might be researching different CI options 19:43:01 wumpus: lol 19:43:02 wumpus: hah 19:43:17 wumpus: DST is evil 19:43:23 sipa: yess 19:43:27 wumpus: nobody will ever know: the bot is broken to 19:43:29 too* 19:43:30 Ideally we have a team of people willing to maintain the CI 19:43:39 jnewbery: I did look at all available ci options a year ago 19:43:40 If it is all on my shoulders,.. it will break for sure 19:43:47 jonasschnelli: like we don't have our hands full maintaining Core 19:43:52 As I said most of them are docker-only, so they can't run our scripts 19:43:55 If we have actual needs beyond what CI providers offer, then we should pay someone to maintain a CI cluster, no? 19:44:10 luke-jr: there are enought people with server admin skills but no coding skills willing to contribute. 19:44:19 dongcarl: that implies we have excess funding 19:44:28 Only cirrus ci and travis offer full vms 19:44:29 dongcarl: sure. Paing for it should be possible 19:44:30 funding can be found, i'm sure 19:44:37 I expect funding will not be a problem 19:44:50 oh, so it's only my funding that's a problem :/ 19:44:55 sigh 19:45:06 luke-jr: Are build servers *that* expensive? 19:45:18 no... but the maintenance men.hours 19:45:18 on that note, funding might be good for security bug bounties too 19:45:38 yanmaani: no, but maintaining a CI platform probably is 19:45:49 oh right, misread 19:45:56 I think a CI team is better than a single persion. 19:46:02 *or woman hours :) 19:46:03 emzy: for sure 19:46:12 jnewbery: oops. yes. 19:46:30 Have people had issues with Cirrus Ci lately? 19:46:45 MarcoFalke: I started ignoring it because when it does have problems I can't restart it 19:46:47 MarcoFalke: i can't remember any... i also have no idea what actually runs on it 19:46:49 They used to abort sometimes, but setting kvm:true fixed it I think 19:47:06 sipa: Only the thread sanitizer and one fuzzer 19:47:35 is there any reason why everything couldn't run on cirrus? 19:47:43 We could also move all jobs to Cirrus Ci, which would probably mean we need to pay them 19:47:54 free tier is only 8 cpus or so 19:48:09 probably cheaper than maintaining our own 19:48:14 if it works decently 19:48:14 probably 19:48:33 Do we have any needs beyond what Cirrus provides? 19:48:35 Semaphore offered us also free service https://semaphoreci.com 19:48:56 dongcarl: s390x and arm would be nice, but we can use qemu 19:49:36 jonasschnelli: but that's just Docker? 19:49:50 jonasschnelli: If it is just docker, it won't work 19:50:00 luke-jr: they have x86 and macOS... all docker I guess 19:50:37 If someone knows another non-docker ci, let me know, but I don't think there is 19:50:37 MarcoFalke: hmm, does KVM work inside stock Docker? 19:50:43 [13bitcoin] 15gwillen opened pull request #20264: test: Make secp tests optional in `make check` (06master...06feature-optional-secp-check) 02https://github.com/bitcoin/bitcoin/pull/20264 19:51:00 luke-jr: we don't use kvm 19:51:24 I doubt it does 19:51:39 qemu-user works in docker 19:51:42 MarcoFalke: we could 19:52:07 luke-jr: Patches welcome :) 19:52:17 I don't have background in kvm, so I can't help here 19:52:22 MarcoFalke: without a plan, no point 19:52:38 MarcoFalke: KVM is just virtualising a machine - so you can do anything as if it was real hardware 19:52:46 luke-jr: git grep docker ./ci 19:52:52 then replace docker with kvm 19:53:07 sure, but unless Docker supports KVM, it won't help 19:53:20 jup, I don't know an answer to that 19:53:31 Other topics? 19:53:36 (short ones) 19:54:17 #endmeeting