I am re-submitting this proposal now that we have more reliable automation for building our release binaries.
GitHub has a feature called Immutable Releases. Projects can designate a release as immutable which means that new assets (e.g. source tarballs and binaries) cannot be added and existing assets can not be modified.
I believe that this feature can mitigate one of our current security weaknesses, which is the fact that any one of our 1600+ contributors with write access can create or update release assets.
I would like to propose that we enable this feature for all past and future releases and adopt the following new process for creating releases and managing assets on GitHub:
The day before the release, the manager will tag the release and an automated process will create a ‘draft’ release on GitHub.
After the ‘draft’ release is created automation will upload the release sources and binaries to the ‘draft’ release page. This process takes around 4 hours (latest run).
On the day of the release, the release manager will check to make sure all the binaries and sources have been uploaded to the ‘draft’ release page, re-running the builds if necessary.
Once all the sources and binaries are uploaded, the release manager will review them and then sign the release assets.
Once the signatures are uploaded, the release manager will publish the release, and create the release announcement on Discourse.
If there is a mistake or something is missing, a release manager may create a new tag and release to fix the issue, but this is up to the discretion of the release manager.
The main concern in that thread was the possibility of TOCTOU-style supply chain attacks.
Do we report checksums of executables from our build pipelines? Barring fully reproducible builds, that would likely be the easiest way to ensure that nothing’s been tampered with between artifact upload and signing.
@whuhn All the artifacts we build are upload along with attestations. The release manager uses the attestations to confirm the binaries are valid before signing them.
I would like to propose that we enable this feature for all past and future releases
Just a note that these aren’t exactly the same option (some may already know this, but I just wanted to highlight it for when planning logistics): ticking the GitHub checkbox enables its action for all future releases, while that checkbox is ticked. Past releases would remain mutable, but the first mutation of any sort of it will also trigger making that release immutable. In my experimentation, there won’t be a warning message or banner that editing the past release makes it immutable (unlike a new release, which will have a warning about that effect).
That said, it sounds reasonable to me. There doesn’t seem much reason to edit releases after release. A corrupt release can still be yanked. And a mistake means the patch number just gets incremented once more than originally expected. New release artifacts could hypothetically even be added later in a similar way (e.g. a new tag at the same commit for publishing more artifacts).
We added this to the @infra-area-team meeting agenda today, and we think this is a reasonable, uncontroversial proposal. It’s been posted for 10 days and generated 5 comments. The main impact here is on the release team, so as long as you are working closely with them to address their concerns, we should consider this approved.
We had a side discussion about how some release packages for alternative operating systems are produced, where a community member might take responsibility for building packages for a platform of interest, such as z/OS or Windows, and there might be some latency in producing that package. It was suggested that you’ve created some new process for linking to a separate repo for these alternative platform releases, but I wasn’t able to dig up a reference confirming this claim. Regardless, I assume you and the release team will come up with a reasonable solution.
Regarding community build packages, the tarball for windows is currently community built iirc. But there would be no need for that build to exist anymore if we moved to a more modern installer and could fit a complete build there(as in all projects, and not a subset, notably missing are mlir and flang as well as multiple targets), as currently we are space(or file count) limited by the installer which has caused issues in the past. Not sure what came out of trying Wix @tstellar?
The only community build we have on the release page now is 32-bit arm (@omjavaid ), so this would have to be moved to our “third-party release binaries” thread like the one we had for llvm-22.
Starting with llvm-23, we will have the WiX installer for Windows, so that should resolve any issues with installer size.