[RFC] Official LLVM Binary Releases for Arm and AArch64 Platforms

Project Infrastructure Release Testers
1

Problem Statement

This RFC addresses the need for official LLVM 19.x binaries for ARM and AArch64 Linux, as well as Windows ARM64 (WOA) platforms. Currently, users on these architectures lack access to official pre-built binaries since the release process has moved to GitHub actions workflow.

Historic Context

Over past decade, We, Linaro has produced official prebuilt LLVM binaries for Arm architecture platforms (both Linux and Windows) till LLVM 19, when official release was migrated to GitHub actions workflow.

About Linaro LLVM Binaries

Linaro LLVM binaries for ARM and AArch64 Linux platforms are produced using the official release scripts provided by the LLVM project, available at llvm/utils/release.

Linaro produces LLVM binary release for following platform triples

  • armv7-linux-gnueabihf
  • aarch64-linux-gnu
  • aarch64-pc-windows-msvc

Build Platform Information

Linux

The builds are executed on Arm Ampere® Altra® Server 2U Mt. Jade NVMe hardware, servers running Ubuntu (LTS - 1) and utilize Docker containers to create isolated and reproducible build environments.

Windows

The Windows ARM64 LLVM binaries were being produced using a Microsoft Surface Pro X up to the 18.x release. Starting with the 19.x release, the binaries are produce on Microsoft Surface Pro 9 featuring the 11th Generation Snapdragon X Elite processor, equipped with 32 GB RAM and a 1 TB SSD. Both machines runs Windows 11 and utilizes Visual Studio 2022 alongside the latest LLVM tools available prior to building the LLVM release.

Reasons Why LLVM 19.x Arm platform binaries missing.

  • The current LLVM GitHub Actions release workflows are not supported to build for ARM architectures.
  • GitHub hosted Linux ARM64 runners are relatively new and have not been tested for LLVM build process. Without thorough testing release CI on GHA may be flaky or inconsistent.
  • The capabilities, stability, and resource constraints of Windows ARM64 runners in GitHub Actions are not well-understood.
  • Arm v7 (32bit - Arm) Architecture support is not supported by GitHub actions.
  • There is delay and lack of clarity on dedicated funding to procure or maintain the necessary runners (either self-hosted or higher-tier GitHub-hosted runners) to perform these builds efficiently.

Proposed Solution

For LLVM 19.x release we (Linaro) propose to upload binaries built in our infrastructure to LLVM release page. We can work with LLVM community and release managers to resolve any concerns. I understand we already have such precedent where Windows x64 19.x release binaries were third-party binaries uploaded to LLVM release page.

Linaro LLVM 19.x binaries can be found at: Linaro Snapshots

For future we voltuneer to evaluate GitHub actions workflows and make change to the existing workflows for Arm Linux and Windows release.

We understand that current LLVM GHA funding situation may not be ideal for the workloads to run efficiently on GitHub runners and may require higher tier runners or strong self-hosted runners. We welcome funding from the community and we are also working offline with our partners to resolve this issue.

CC: @tobiashieta @tstellar @DavidSpickett

2

I’m looking forward to when we can get github actions to produce release binaries at parity with what we’ve been shipping so far, but it seems we’re not there yet.

Your proposal sounds good to me.

1 Like
3

I think I am fine with this in general. But we still have to be careful so we don’t end up adding back all the binaries we had before 19 without adding any additional control back.

@omjavaid - who would be doing the binaries? how many people have access to the infrastructure you are talking about? What’s the risk of us accepting these binaries if someone is able to manipulate the systems or tarballs?

I think we need to have some kind of singing done at least, so we can verify the origins.

4

Linaro’s build infrastructure is only accessible by certain Linaro employees (Toolchain Team). However we can make arrangement to make sure only one person who is responsible for builds have access to build directories.

Windows on Arm builds will be done by a machine which will be on my desk and will be accessible by me only.

I can take responsibility for signing both Windows and Arm/AArch64 Linux builds.

5

With these restrictions and procedures implemented - I would be happy to move forward with this suggestion.

WDYT @tstellar ?

6

This is fine with me.

7

Thanks everyone for support and feedback.

@tobiashieta What are the next step. May I upload the release using the upload script. Also are there any particular signing guidelines we are following for signing release binaries?

8

I will message you directly to figure out the signing.

9

I have uploaded following binaries to official release page:

➜ sha256sum LLVM-19.1.0-woa64.exe
4e1ff3c84ff0ae11c7fded3a35bb616a6672800d134d646675ba261d6457401d

➜ sha256sum LLVM-19.1.1-woa64.exe
f256585b8e39cf95ce3191eb4002e269a755217f8492b240653c94c0ae077c8a

➜ sha256sum LLVM-19.1.2-woa64.exe
76df5995e015e317072a3887b52f9a2163338f4eae23f09e5d3fcb025317f334

Thanks folks for helping out.

1 Like
10

Is there a plan to offer linux-aarch64 binaries on the official release page as well?

11

Yes, I’m working on getting some resources for linux-aarch64 builds.

1 Like
12

Is it expected that linux-aarch64 builds will be available soon?

13

I have a prototype here: Depot runners arm by tstellar · Pull Request #120786 · llvm/llvm-project · GitHub

14

For the Windows ARM64 builds, there is another option that we should consider. We could cross-compile the toolchain from the Windows builders in GitHub Actions. We at The Browser Company have been able to do this with good results so far. We are building (and distributing) a Windows ARM64 Swift toolchain which is built atop of LLVM (and clang). The distribution contains a full LLVM toolchain capable of building C/C++ and Swift code. The GitHub Actions workflow definitions are a fork from GitHub - compnerd/swift-build: Alternate Swift Builds, which would be a good starting point for this should we be interested in exploring the use of cross-compilation for the LLVM Windows distributions.

15

Yes, it’s quite doable to cross compile llvm based toolchains - not only for windows, but also for Linux and macOS too.

For Linux, I cross compile aarch64 Linux toolchains on x86_64 Linux here: llvm-mingw/.github/workflows/build.yml at 20241217 · mstorsjo/llvm-mingw · GitHub All of my Windows toolchains are cross compiled from Linux too.

That said, the LLVM release building script may require a bit of rework to work for cross builds, as it is a multi-stage process. (It would require doing the first stage as a native build, the second one as a cross build, and possibly skip doing a third verification pass.) Doing PGO also becomes harder.

That said, GitHub have announced that there will be free runners with both Linux and Windows on aarch64 - initially this was meant to be made available by the end of this year, but the schedule seems to have slipped a bit.

16

We do have a small budget and could use some of the larger ARM64 Windows runners provided by GitHub, I just haven’t had time to look into this yet.

@mstorsjo We’ve replaced the release script with a CMake cache file, so that should make cross-builds this a little easier.

1 Like
17

I wonder if Linaro would be able to contribute resources from CodeLinaro - that’s a good environment for building with availability of both x86_64 and aarch64 targets.

18

We (Linaro) currently do not have AArch64 Windows runners suitable for contributing as a GH action runner. However we will keep building and uploading Arm architecture release binaries in our buildbot and release infrastructure as long we dont have same binaries uploaded by Github actions CI.