LLVM E2E tests

We (SAP) recently ran into a problem regarding the seldomly used indirectbr/blockaddr IR instructions, reported here: [x86] Undefined temporary symbol .Ltmp0 created by blockaddress · Issue #161164 · llvm/llvm-project · GitHub
We found a workaround for this and patched our LLVM fork.

Regardless if we may able to achieve a fix on LLVM-side, we spent another thought on how we as LLVM user can ensure future stability of our proprietary compiler. We are currently considering contributing E2E test cases (LLVM IR → assembly) to LLVM, that represent the typical output of our compiler, also in complex scenarios. This would aim to ensure, that no new development inside of LLVM triggers any functional regression on our side.

So my questions would be:

  1. Would such E2E tests generally be appreciated?
  2. Where would we put such tests? Logically they don’t really fit in the test directory of any transform, since they test the whole pipeline end to end.

Best,
Marc

I’m not a contributor to LLVM subproject, but in general there are two ways.

The first one, as you already mentioned, is contributing your tests upstream. In general you should have a good feedback on anything that improves our coverage.

The second is to track upstream and report failures before they slip into releases. Definitely more work for you than the first option, but for tests that can’t be upstreamed you don’t have much of a choice.

Could you clarify the end-to-end here? Is it only the llc or a combination of opt and llc?

Generally speaking we’re welcome any kind of codegen (usually llc) or middle-end (usually opt) tests, as long as they improve the coverage. If the test is running a combination of opt and llc, is it possible to split them into separate tests for each tool?

Sounds like the tests you’re picturing are “large”-ish chunks of LLVM IR, then tested through a full middle end pass pipeline and through a backend of choice to assembly?

This is probably not the sort of test that should go into the LLVM tests directory - large means possibly slow, and means the failures aren’t as precise and actionable as we like to keep tests there.

So maybe it could go into the test-suite, though those run less often, etc.

And then there’s probably some questions to ask/answer about what these scenarios represent and whether they’re aligned with what the LLVM community wants to sign up to support. (if the community doesn’t want to sign up to support them - might be on you to run the tests & triage the failures before presenting them to the community - as someone else mentioned, you could do this frequently to reduce the latency/increase the chance you catch something before it’s baked in too far)

There is some precedent in llvm-test-suite/Bitcode (mostly for halide codegen) and the test-suite can differentiate between “benchmarks only” and “benchmarks + unit/correctness tests” mode. The tests have to be able to run though and produce some kind of “finished correctly” output.

Though admittedly debugging when things in the Bitcode directory fail is pretty annoying… so hoping the function/modules don’t become too big…

Also wanted to mention that many groups just operate in a different manner where the group itself makes sure it has a branch/development version that keeps up with day-to-day upstream changes and then runs internal tests themselfes and just providing minimized reproducers on the upstream PR as they find them. That tends to be the model that works best as people are most receptive when the breakages are “fresh” and when there’s a person responsible/reacting to questions about the minimized problem report… (and you can run as much company internal code as you want in this mode)

1 Like

I would like to note that we have the cross-project-tests directory, some of which go from C all the way to objdump. I presume this would continue to be OK for (small, self-contained) future tests.

I don’t know which bots are running these tests, though.

With end-to-end I mean input → assembly (or event better an executable binary)

Not necessarily. E.g. the issue, mentioned above, can be reproduced in < 200 lines of IR. But yes, it would also be desirable to have the option for longer running tests as well.

Thanks for that hint. I had already browsed around in that repo, but it seemed rather benchmark related suite to me. For that reason I wanted to inquire your opinion if such tests could go there.

I now uploaded a draft of the first testcase here:

Please let me know if I miss something here. I also attached the IR file, that the bitcode was generated from to ease debugging.

It’s been a while since I introduced cross-project-tests, but the intent was for these tests to still be targeted at small specific features, like most lit tests, but where the behaviour to test requires elements of multiple LLVM sub-projects (e.g. clang and LLVM binary utilities). I get the impression the tests being discussed here don’t fall under that category.

2 Likes