I am pleased to announce that AWS has given us a very generous donation which will allow us to add AArch64/Linux pre-commit CI builders. I would like to get feedback from the community about the most effective way to use these builders. Should we use the same configuration on AArch64/Linux that we use for X86_64/Linux or should we build something different? Some alternate configurations might be:
Building with assertions disabled.
A bootstrapping build.
Enabling LLVM_LINK_LLVM_DYLIB
Something else?
Interested to hear what everyone thinks about this.
I don’t think using the same configuration as we do for x86_64 would be cost effective. On Linux we do things like test libc++ in two different configurations on every clang/llvm change to test for clang breakages. I don’t think we gain meaningful test coverage from also running that on AArch64 with respect to the cost (currently something like 20% of our runtime for llvm changes).
Overall, I think I would like to see the configuration be as “default” as possible, while being run on AArch64. Building with assertions disabled might catch some errors in that configuration, but I think we lose more test coverage of assertions on AArch64 than we gain in the assertions disabled config. I’m not sure what you mean by boostrapping build here. The existing CI does a bootstrapping build for the runtimes. Doing a full bootstrap build where we compile clang and then use the fresh clang to build/test the project would have a huge cost. We lose a lot of caching because the compiler is different everytime. That also has a large latency hit. The dylib configuration has a decent startup time penalty on check targets and I don’t think it provides enough additional coverage (like say a shared lib build would) to justify using it in CI. I don’t think any expensive configurations are going to be viable for cost reasons. We might have to limit what projects we test for cost reasons as well.
I would also like to stress the maintenance cost of premerge. Maintaining the current infrastructure is a large amount of work. There is a bit less of this sense we can use depot with AWS, but I think it’s important to ensure that there are at least some people in the community who have access to the AWS project(s) and can hack on things. Going through a stabilization period like we did with the new premerge system where checks aren’t marked as failing initially would also be good. In my experience, getting something going is not too difficult, but ensuring it is reliable takes a lot more effort. A lot of the reliability issues were seeing on premerge these days would be platform agnostic though.
It’s hard to answer without knowing what our current pre-commit configurations are. Where could I find this information?
In general, if our coverage is already quite robust then it will be hard to “invent” new meaningful configs and I would either:
Duplicate, or
Replace some of the X86_64 runners with AArch64.
Otherwise, we should just identify configs that are not tested yet.
As for a specific request, I would include MLIR’s e2e/integration tests (hidden behind MLIR_INCLUDE_INTEGRATION_TESTS) - that’s one major testing gap in our pre-commit CI - at least from our perspective.
Btw, can you share which AWS instances LLVM is getting access to? I am mostly interested whether that hardware includes SVE.
Have we considered enabling features that are unique to AArch64 and could help with finding bugs? I’m specifically thinking about Hardware-assisted AddressSanitizer (HWASan) which has significantly lower overhead (although still non-trivial) compared to ASan.
Context: I’m one of the team at Linaro who provide post-commit bots for Arm (32-bit) and AArch64.
One way to think about it is what problems were caught only by non-x86 post-commit builders.
A common problem is tests that require a backend to be enabled but don’t have REQUIRES:. However, these are pretty trivial to fix and if we limited the build to only AArch64 then you limit the coverage for suggestions like Petr’s. So I would not try to address this failure mode with pre-commit (unless we had excess capacity, seems unlikely).
There are sometimes ABI differences but they are fairly rare. For instance when we steal bits from pointers by assuming alignment of certain types. These would be checked by doing a normal build on AArch64 hardware, don’t need to do anything unusual.
(we get far more ABI/bitness issues for 32-bit Arm)
For runtimes being on the hardware really helps. Libcxx Linaro is supplying workers for already, libc already has their own thing I think, compiler-rt would be good to run, and flang-rt is included anyway if Flang is built.
LLDB would benefit a lot from AArch64 pre-commit, SVE testing is a nice bonus if it’s there. Linaro’s post commit testing is already on Gravition. We have had many “can I trigger an AArch64 run manually?” requests. Again, nothing special about the AArch64 LLDB build.
There are probably more surprises when it comes to AArch64 as an architecture, but it’s my default these days so they’ll be less obvious to me.
You can find all of Linaro’s existing bots in llvm-zorg, and I can get you any other details you might need.
(I’m not involved in managing LLVM’s CI, but have worked on similar things in the past.)
I think this comes down to the available resources. As a starting point, I’d pick a stock standard configuration which closely duplicates one of the existing x86 configs. The downside to this as pointed out above is that it doesn’t maximize coverage per dollar, but the upside is it means that a failure in the new bot (but not on the existing bot) is much easier to attribute and debug. In the early days of any new architecture of a CI system, that’s super important for building user trust and keeping the maintainer sane.
Even the stock standard build is going to cover some new territory. In particular, AArch64 SVE will expose all the scalable vector logic by default. We do have this covered in LIT tests, but native testing always finds something.
Once you have that in place, if you have additional resources you can consider adding extra configurations. Personally, I’d like to see a no-asserts build, but that’s only because that’s one of my personal frequent screw ups. I’ll note this is no way architectural specific, so this would just be treating all our pre-commit CI resources as one big pool, and splitting the desired configurations across them.
In general, I would strongly prioritize fewer configs run faster over more configs run slower. Pre-commit CI needs to be fast to be useful.
I generally think that the marginal value of testing LLVM/Clang/… on AArch64 in addition to X86 in the same configuration is quite low. Given how our tests for these projects work, they are nearly entirely host independent. For these projects, to get actual value out of the new runners, we need to be testing a different configuration on them.
Two that I think would be valuable are:
Building without assertions enabled. Because devs will almost always build with assertions, this class of issues (-debug without REQUIRES, assert-only variables causing warnings, etc) is easy to miss and would be valuable to diagnose in pre-merge testing.
Using a BUILD_SHARED_LIBS=ON build. This detects incorrect library dependencies a lot more reliably. This is a less common issue, but still a common recurring problem.
Of course, for runtimes (or runtime-adjacent projects like lldb), simply running on a different architecture does provide a lot of value, even if the used configuration is otherwise exactly the same.
I don’t think this is the case? To actually exercise SVE logic we’d have to do something like build llvm-test-suite with the right configuration. Pre-merge testing currently does not cover llvm-test-suite at all. So we’d just be running the same lit tests we do on X86. Am I missing something?
In the past, for ARMv7, we have caught bugs in Clang where the ABI was different, but only when self-hosting (the system clang didn’t show the problem). ARMv8 is much close to Itanium ABI to be a problem, so I tend to agree that it would be more beneficial to have different builds.
Assuming the x86 bots build all targets, pretty much, yeah.
I would like to see the test-suite as a pre-commit builder, and this could be one thing we do. But I recognize this may be a whole different can of worms, so not going out of my way to propose it.
We don’t need llvm-test-suite for this - we have e2e/integration/runnable tests in MLIR (hidden behind MLIR_INCLUDE_INTEGRATION_TESTS). Some of these tests are specific to SVE or SME and are only enabled when either:
the underlying hardware supports SVE/SME, or
an emulator has been specified (via. ARM_EMULATOR_MLIR_RUNNER_EXECUTABLE).
Most of these tests run a single compute kernel (e.g. a GEMM or a convolution), so something that is:
much more complex than most/all LIT tests in LLVM,
not as complex as some of the examples used with llvm-test-suite (e.g. SNAP).
If we are only considering LLVM LIT tests, then yes. If we look beyond that (e.g. MLIR e2e tests), then there are options that we could consider. Not sure about other sub-projects.
One of the things the test-suite covers is optimizations gone wrong, including vectorization, and we have no SVE coverage for that, even if we turn the integration tests on. We caught a lot of these bugs back in the ARMv7 days with NEON.
From the main repo, I think the runtimes and standard libraries would be a good additional coverage, in addition to the integration tests you mention.
I don’t have a lot to add over what’s already been said. My, now ancient, memory of the Linaro AArch64 build bots, were that the runtimes, especially sanitizers, were the most frequent source of AArch64 specific problems.
Last time I looked I’m not sure if we have an AArch64 linux, self-hosted, test-suite, using compiler-rt, libc++ libc++abi, libunwind. That would be useful to complement the GNU runtime libraries builds.
I agree with Petr Hosek that runtime tests with some of the AArch64 specific hardware features enabled like MTE, PACRET, BTI will be helpful. Especially with libunwind, libc++abi used.
Would definitely be good to not duplicate the existing AArch64 builders Linaro provide.
For testing that enabling MTE, PACRET and BTI don’t affect the correctness of the program; anything that is a full build and run executable will work. The libc++/libc++abi/libunwind tests and the llvm testsuite are what I had in mind. I think of it like running the test-suite with the sanitizers enabled.
What this won’t do is test if MTE, PACRET and BTI are providing the fault detection or mitigations that they are intended to, only that they’ve been applied in a way that doesn’t affect the tests.
There are unit tests for MTE, PACRET and BTI but these can be run on any host machine.
MTE (memory tagging) specifically, is not present on any AWS hardware. The others are though.
Not very useful to you, but just to be clear, Linaro does not run anything like this.
LLDB will run extension specific tests if the test runner detects those features. So we do run pointer authentication debugging tests, but didn’t have to do extra work to enable it (and that is run on Graviton).
For llvm-test-suite the most we’re doing is adding SVE codegen options. SVE 1 / SVE 2, fixed length, variable length etc.
I get why we’re trying to not duplicate, but this is more about pre-commit than just a new builder.
I think there’s a reasonable expectation to run basic checks on pre-commit, and depending on which graviton instance we got from AWS, it’s quick enough to run full builds, perhaps self-hosting builds and perhaps with test-suite.
If all we have is a 4 core machine, then just building LLVM will be a struggle, so I’d keep it to simple builds + check-all (with special conditions to make it worth it).
I’ll note this is no way architectural specific, so this would just be treating all our pre-commit CI resources as one big pool, and splitting the desired configurations across them.