Bazel and "check-mlir" target: does Bazel support running lit tests?

We are building MLIR-based project that must be built in Bazel environment. I was able to build our project by using rules from utils/bazel subtree with LLVM as external repository. But now I would like to create lit testing environment similar to as it is done in LLVM and MLIR. Looking inside utils/bazel I was not able to discover how to do this. I also was not able to google it.

  1. Does LLVM Bazel build system support running lit tests in some way? If yes, how?
  2. Assuming “yes”, where can I see how to add this support to my project also?

Thank you in advance.

We have some internal rules to enable lit-based tests, so it’s definitely possible under bazel. Not sure why the upstream only supports unit tests. Perhaps @gcmn or @chandlerc know.

We have this in TensorFlow: tensorflow/glob_lit_test.bzl at master · tensorflow/tensorflow · GitHub

Iree also has some support for this: iree/build_tools/bazel at main · google/iree · GitHub (and see usage in various test/ directories). @gcmn wrote most of it.

Bazel is a high cost to develop and support for these things, and we eventually had to stop spending time supporting the upstream bazel build in terms of features. It works for what it does and we use it but it is not complete and far (far) from providing the options we get in the cmake build. (We treat our cmake build as source of truth and support bazel in an as needed basis)

1 Like

One thing I recall: for leaf projects with relatively few tests (is. Tf and iree), both of the sources listed here can work ok. However, llvm’s test suite is huge (and laid out in a way that isn’t very conducive to the opinions bazel takes on how things should be done). I know that internal to Google (where these build files originated) there is dedicated tooling for scaling to that volume. I’m not sure how much that is a factor in reality, but knowing that definitely discouraged people from going down the rabbit hole in the upstream llvm/oss side.

1 Like

Sorry I was out on vacation when this thread happened. Internally at google we use a custom program to run lit tests (more details in this llvm-dev thread). I’ve got Bazel working with lit and we’re using it in IREE (replacing a hacky shell script we were using before), but llvm-lit is a whole other beast on top of lit that’s deeply entangled with the CMake build system. Additionally, lit insists on writing files with test timings and such, which is totally incompatible to Bazel’s sandboxing. In IREE, we’re able to get around that by setting the test_exec_root, but that isn’t really an option for upstream lit tests. I tried out just getting the tests of lit itself to work under Bazel and ran into that issue. Fundamentally, Bazel is already a test runner and lit is a test runner and it’s pretty hard to run one inside the other. It ended up being too much of a rabbit hole to go down.

If someone wanted to look into adding these, the lit test rules we’ve got in IREE are designed to be general, and I’d be happy to help. I think it might involve starting a conversation about making the temp file outputs from lit optional. And you will run into the scaling issues Stella mentions, though I’m not sure whether they’re actually a show stopper or just a waste of resources. I’m pretty sure that if it were my job, I could get lit to work alright under Bazel, maybe by shifting all the config parsing to build time and turning the test into just a shell test, but alas I have other priorities :-/

2 Likes

Thank you all for the help. Based on your input we decided to run lit tests for our project outside of Bazel by using lit runner directly.