TL;DR: I’d like to set up nightly testing based on Docker images. This would be a non-disruptive, non-PR-gating addition.
Docker seems to be the way modern projects assemble their environment for testing and production. (I personally have used it for development also.) As such, I think we should use it for running our nightly integration tests. I have very successfully done this several times in the past. Given that this is relatively standard these days, I’m going to assume that everyone either doesn’t care or is in agreement for the rest of this post.
I’ve put together a prototype workflow which runs the integration tests out of a Docker container with all the necessary software pre-installed. The prototype workflow builds the image (with the Docker cache, so rarely re-builds it) then runs the build and test inside the container. This ensures that the pre-installed software matches the scripts in the CIRCT repo.
Setting up this workflow different repos (different organizations really) is a bit involved since it involves an organization admin configuring permissions for the container registry then a workflow secret. It’s not too much work but after doing it this way, I’m thinking it’s not worth the effort.
Instead, I think we should set up a CIRCT Docker repo in the circt github repo and use it to build the image(s) which are used by the CIRCT integration builds. This method would make forking while maintaining operation of the nightly tests easier. It would, however, mean when adding/updating a test requirement (e.g. updating Verilator) changing both the Docker repo to generate a new image (with the upgraded Verilator) and the CIRCT nightly workflow to point to the new image. That’s a fair trade-off (IMO) given that these requirements rarely change.
I do not think we should include MLIR in said image since we are deeply coupled to it for build and it changes relatively frequently. Also, we’ve already got a caching solution for the llvm submodule which works well. The Docker image should only contain the all the optional requirements for building and integration testing of MLIR and CIRCT.
Does everyone agree that this is the way to go? If so, I can set it up if I’m made an owner for the CIRCT organization, then PR into the CIRCT repo with an additional workflow to do the nightly (and on-demand) integration testing.
+1 from me. I wanted to do this a while back but didn’t get to it.
Big question: What would you install in this container? Verilator? Other than that, I don’t think we have many external dependencies that aren’t in the default ubuntu containers.
NPCOMP has a bigger problem in this area, because it actively requires pybind, pytorch, etc.
I’m interested in seeing how you do this and adapting to npcomp. My docker skills are more as an end user and not for real automation.
On the npcomp side, we have more deps but they are all stock installs from upstream. The main issue I run in to is making sure that everything builds with the cxx11 abi disabled as that is how the binary deps are built (but newer os images default it to on).
Thanks! I’ve created a “machine account” for CIRCT since I don’t want to install a secret for my personal account. Can you invite “circt-ma” to join the CIRCT org as a collaborator?
Does anyone know a secure way to share passwords with others? I set up the MA and an email address for the MA which I should probably share with the CIRCT owners.
Done. As for password sharing: it would be nice to be able to credential any user for this. When you say “install a secret”, it’s not clear what you’re doing? Perhaps you can elaborate on (or better yet, document) what you’re doing?
A “machine user” is just a GitHub user which is used for automated tasks. GH doesn’t know that it’s special so there’s no way to credential users for it.
I don’t recall turning these on for this particular org, so I assumed they were on by default. You’d think that members would be able to view the org settings though not modify them!
Hmmm… It pushed successfully now, but I don’t see it under the org packages. @stephenneuendorffer can you go to Packages · CIRCT · GitHub and if you see a Docker image make it public?
