RFC: Using GitHub Actions for CI testing on the release/* branches

Hi,

I would like to start using GitHub Actions[1] for CI testing on the release/*
branches. As far as I know we don't have any buildbots listening to the
release branches, and I think GitHub Actions are a good way for us to quickly
bring-up some CI jobs there.

My proposal is to start by adding two post-commit CI jobs to the release/9.x branch.
One for building and testing (ninja checka-all) llvm/clang/lld on Linux,
Windows, and Mac, and another for detecting ABI changes since the 9.0.0 release.

I have already implemented these two CI jobs in my llvm-project fork on GitHub[2][3],
but in order to get these running in the main repository, I would need to:

1. Create a new repository in the LLVM organization called 'actions' for storing some custom
builds steps for our CI jobs (see [4]).
2. Commit yaml CI definitions to the .github/workflows directory in the release/9.x
branch.

In the future, I would also like to add buil and tests jobs for other sub-projects
once I am able to get those working.

In addition to being used for post-commit testing, having these CI definitions in the
main tree will make it easier for me (or anyone) to do pre-commit testing for the
release branch in a personal fork. It will also allow me to experiment with some new
workflows to help make managing the releases much easier.

I think this will be a good way to test Actions in a low traffic environment to
see if they are something we would want to use for CI on the master branch.

Given that we are close to the end of the 9.0.1 cycle, unless there are any
strong objections, I would like to get this enabled by Mon Nov 18, to maximize its
usefulness. Let me know what you think.

Thanks,
Tom

[1] https://github.com/features/actions
[2] https://github.com/tstellar/llvm-project/commit/952d80e8509ecc95797b2ddbf1af40abad2dcf4e/checks?check_suite_id=305765621
[3] https://github.com/tstellar/llvm-project/commit/6d74f1b81632ef081dffa1e0c0434f47d4954423/checks?check_suite_id=303074176
[4] https://github.com/tstellar/actions

Not having given it deep thought/analysis, nor understanding much of the GIT infrastructure here, but: Sounds good to me, for whatever that’s worth :slight_smile:

Although I’ve not had any Github Action experience, and despite being one who has raised objections to other parts of adopting github features, this sounds like a sensible idea. It seems odd to me that we don’t already have some form of CI for our releases, so anything that improves on that is great.

James

Hi Tom,

This sounds very interesting! +1 from me.
What does this imply for the release testers?
Would it be possible / desirable for us to add self-hosted runners for
other architectures? I'm assuming GitHub only provides x86, right? I
totally missed any details about that in their docs, but it seems to
be implied here [1]. I'm not saying we should go all-possible-arches
from the start, we can definitely try it out only for x86 this time
around if you think that would be the best approach.

Thanks,
Diana

[1] Self-hosted runners for GitHub Actions is now in beta | The GitHub Blog

Sounds great to me!

Hey Tom,

That sounds really useful. Would it be possible to include LLDB as
well? We have a subset of tests (unit & lit) that can be run without
Python/SWIG by passing LLDB_DISABLE_PYTHON=ON to CMake.

Thanks,
Jonas

Hey Tom,

That sounds really useful. Would it be possible to include LLDB as
well? We have a subset of tests (unit & lit) that can be run without
Python/SWIG by passing LLDB_DISABLE_PYTHON=ON to CMake.

I can try to add an lldb builder. Would you be able to provide me
with the cmake arguments plus a check target where all tests are expected
to pass?

Thanks,
Tom

+1, I'll be really curious to know how this works out.

Philip

Hi,

I would like to start using GitHub Actions[1] for CI testing on the release/*
branches. As far as I know we don't have any buildbots listening to the
release branches, and I think GitHub Actions are a good way for us to quickly
bring-up some CI jobs there.

My proposal is to start by adding two post-commit CI jobs to the release/9.x branch.
One for building and testing (ninja checka-all) llvm/clang/lld on Linux,
Windows, and Mac, and another for detecting ABI changes since the 9.0.0 release.

I have already implemented these two CI jobs in my llvm-project fork on GitHub[2][3],
but in order to get these running in the main repository, I would need to:

1. Create a new repository in the LLVM organization called 'actions' for storing some custom
builds steps for our CI jobs (see [4]).
2. Commit yaml CI definitions to the .github/workflows directory in the release/9.x
branch.

In the future, I would also like to add buil and tests jobs for other sub-projects
once I am able to get those working.

In addition to being used for post-commit testing, having these CI definitions in the
main tree will make it easier for me (or anyone) to do pre-commit testing for the
release branch in a personal fork. It will also allow me to experiment with some new
workflows to help make managing the releases much easier.

I think this will be a good way to test Actions in a low traffic environment to
see if they are something we would want to use for CI on the master branch.

Given that we are close to the end of the 9.0.1 cycle, unless there are any
strong objections, I would like to get this enabled by Mon Nov 18, to maximize its
usefulness. Let me know what you think.

The CI tests are running now on the stable branch. Here is the
first run [1]. Passing commits[2] now have a green check mark in
the web UI that you can click on to view the results.

These tests have already caught 1 bug that did not appear in my local
builds, so they are proving to be very useful.

Now that this is in place, I'm going to start looking into some more automation
around backporting fixes to the stable branch. I think we could do something like
filing backport requests as a GitHub issues and then have that issue automatically
trigger a pull request and do some pre-commit testing. I'm still trying to learn
what is technically possible with GitHub and if anyone has any ideas or suggestions,
let me know. I'm hoping to have something in place for the 10.0.1 release cycle.

Thanks,
Tom

[1] https://github.com/llvm/llvm-project/commit/6c86aa62d502ac2df58e207d4792544cc96d79bd/checks?check_suite_id=316648311
[2] https://github.com/llvm/llvm-project/commit/6c86aa62d502ac2df58e207d4792544cc96d79bd

Hi Tom,

This sounds very interesting! +1 from me.
What does this imply for the release testers?
Would it be possible / desirable for us to add self-hosted runners for
other architectures? I'm assuming GitHub only provides x86, right? I
totally missed any details about that in their docs, but it seems to
be implied here [1]. I'm not saying we should go all-possible-arches
from the start, we can definitely try it out only for x86 this time
around if you think that would be the best approach.

Nothing changes for release testers at this point. The main goal here is
to get some post-commit testing so we can catch issues in the branch early.

Longer term I think have self-hosted runners would be great, because it would
allow us to fully automate the testing and uploading we do when a release gets
tagged.

You are correct that GitHub only provides x86 machines, however, they don't
have enough disk space (14GB) to be able to run the test-release.sh script, so
adding x86 self-hosted builders with more disk space would still be very useful.

-Tom

Hi Tom,

First of: I’m a big fan of adding more automatic tests to find bugs. Great work!

On the other hand, I think we should consider creating some sort of test strategy for the LLVM project:

  • What tests do we expect users to run before uploading patches?

  • What tests do we expect users to run before merging?

  • What tests do we run after merging?

  • What failures must be fixed, what failures can be ignored?

  • What do we check for on the build bots?

  • What do we check for on the release branches?

  • What do we check for on the pre-merge tests?

  • Which CI tools do we want to use (github, Jenkins, bulidbot, …)?

  • What about running clang-tidy and clang-format?

  • What CMake configurations should we check? (Release/Debug, assertions, …)

  • Do we want to run tests with Sanitizers?

  • Which of these systems do we expect users to monitor?
    I suppose it would be good to have a document that summarizes all of this so that we

  1. do not test the same thing twice and
  2. do not miss any important checks.
    Does something like this exist?
    Is anyone working on this?

Best,
Christian

In addition to being used for post-commit testing, having these CI definitions in the
main tree will make it easier for me (or anyone) to do pre-commit testing for the
release branch in a personal fork.

I find this part especially useful. I’m using these actions on a personal branch to validate patches on Linux, OSX and Windows, which gives me more confidence on its portability.