Using GitHub Actions for Releasing

Project Infrastructure GitHub
1

Following the discussion with @tstellar and @tobiashieta in Discord, here are some random ideas:

  • We can also use GHA for nightly build, then push the artifact to somewhere like nightly.llvm.org , artifacts older than X days are deleted. This enables a few things:

    • Services like godbolt can pull the latest trunk. IIRC godbolt now builds offline which sometimes results in a trunk build that’s days/weeks/months older
    • Projects like ld64.lld is still in its early stage and people tend to use the ToT version, hence projects like keith/ld64.lld exists, we can use nightly instead of relying on someone else’s build
    • This also makes our “does it reproduce on trunk” issue when working with new bug reports easier

  • macOS / Windows builds can be signed with a LLVM Foundation Apple Developer certificate so no more workaround / turning off gatekeeper is required.

1 Like
2

I think having some kind of nightly builds would be great. If someone wants to write an GitHub actions job for this, we can deploy it and see how it goes. The job should use a standardized script for doing the builds. It could re-use test-release.sh possibly or it could be a new script. And it should be runable on the free-tier of runners.

Do you have more information about how this signing process would work?

Personally, I don’t think this is necessary. We have the tools to implement prototypes for some of these things (especially the nightly builds), we just need someone who has the time and motivation to do it.

3
  • Obtain a Developer account from Apple, and generate DeveloperID certificate, install
  • security find-identity -v -p codesigning, you see something like 1) ABCDEFG "Developer ID Application: XXXXX"
  • Run this one-liner
find . -type f  -not -type l |parallel -- file --no-dereference {}|grep 'Mach-O'|cut -f1 -d ':'|parallel codesign --force --timestamp --options runtime -s "ABCDEFG" {}
4

I have both of them, but I’m only familiar with Darwin platforms

5

That’s probably a problem, we probably need to setup ccache for it?.
Alternatively, we could use self-hosted runner, similar to what swift did.

6

It’s doable, we have tests for the release branch that are able to build llvm, clang, and lld together. We are using sccache for those builds, but they work even without that.

7

Yeah, we may want to do this eventually. I just think it’s easier to start with the GitHub hosted runners and then expand to something better/faster once it’s working.

8
  • Run this one-liner

This isn’t quite enough these days - you also need to notarize the binaries by packing them up in a zip and submitting it to Apple with xcrun notarytool submit <file.zip> --keychain-profile AC_PASSWORD --wait. For regular application bundles, you’d also ideally staple the notarization result onto the binaries (so that macOS gatekeeper can verify them offline), but that doesn’t work for standalone, non-bundle executables.

9

I support having a standard set of binaries for each release built by a CI instead of relying on the testers. I still think the release testers are very valuable since they run more in-depth testing most of the time. But as we have seen the time commitment makes it hard to get binaries out for each release.

But we need to discuss the goal for these binaries. Are they there to provide all features of LLVM to all users? That’s a pretty complex goal in that case.

As I see it there are very different users of LLVM:

  • People who want a fast C/C++/Fortran toolchain
  • People who want to link to LLVM / Clang libraries
  • People who use MLIR and associated libraries
  • … etc there are many uses here

We have some problems with the current binaries:

  • The binaries are done without many optimizations (no LTO, no PGO, etc) so they are not well suited for a fast toolchain.
  • Windows binaries only supply the toolchain. No extra libraries or tools for development around LLVM
    – This is actually complex to solve since Windows binaries are more stringent with their ABI compatibility. To provide development binaries for windows you might need to have multiple sets of libraries shipped.
  • The size of the current packages is pretty big since they contain all the static development libraries.
  • The current build is long and expensive since we build almost all features

I wonder if our first attempt at this should be to provide official binaries for the toolchain. This means clang, lld, llvm-binutils, compiler-rt, and libc++. These binaries can be faster to build, they can offer basic needs for many users and we can enable LTO optimizations (and even later PGO) for faster binaries without having to tackle the bitcode-in-static-libraries issue.

This can then be expanded with a development libraries package of LLVM to address those users.

I think splitting the binaries we want to provide into specific packages for specific users is better than trying to make a one-size-fits-all.

At least that’s my two cents.

1 Like
10

I know, what we did for Swift Toolchain is we notarize the installer , and developer sign /staple the executable themselves. However I don’t think anyone uses LLVM toolchain with Xcode due to changes introduced by Apple downstream changes, so here we are

11

I think NDK wraps the toolchain in a DMG for notarize / staple?

12

I wonder if we really need to do the 3 phase builds for releases or if this can be skipped.

13

I think we want to do a two-stage build. We want to make sure the final product is built with the latest Clang to get the latest optimizations etc.

14

Sure, but we can use GHA for nightly with single-stage first?

15

Yeah, I’d probably start with single stage for nightly builds and then maybe look at trying to download the previous nights build to use as the compiler. But for release builds, I agree with @tobiashieta that a two-stage build make sense.

16

SGTM.

I think releases should do something like this: llvm-project/DistributionExample.cmake at main · llvm/llvm-project (github.com)

1 Like
17

We have only 2000minutes build time per month, not to mention macOS runners has a 10x multiplier, there’s no way this is enough without some sort of CCache in place

18

That’s the quota for private repositories. We have unlimited use of the standard runners for llvm-project.

2 Likes
19

I’ll try spend sometime working with macOS, (or is there attempts like this already in place?)

20

Not that I am aware of.

But if want to do this - I would really recommend writing the script in Python so it can work on Windows later. If we write it in .sh we would need to have a .bat script later and duplicate all the effort.

Python is already used for a lot of scripting in LLVM and it would fit I think.

1 Like