# LLVM E2E tests

**URL:** <https://discourse.llvm.org/t/llvm-e2e-tests/88801>\
**Category:** LLVM Project\
**Tags:** llvm, testing\
**Created:** [November 6, 2025, 4:39pm UTC](https://discourse.llvm.org/t/llvm-e2e-tests/88801 "2025-11-06T16:39:02Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![marcauberer](https://sea1.discourse-cdn.com/flex021/user_avatar/discourse.llvm.org/marcauberer/32/26007_2.png) [@marcauberer](https://discourse.llvm.org/u/marcauberer)\
**Post date:** [November 6, 2025, 4:39pm UTC](https://discourse.llvm.org/t/llvm-e2e-tests/88801/1 "2025-11-06T16:39:02Z")

</div>

We (SAP) recently ran into a problem regarding the seldomly used indirectbr/blockaddr IR instructions, reported here: [[x86] Undefined temporary symbol .Ltmp0 created by blockaddress · Issue #161164 · llvm/llvm-project · GitHub](https://github.com/llvm/llvm-project/issues/161164)  
We found a workaround for this and patched our LLVM fork.

Regardless if we may able to achieve a fix on LLVM-side, we spent another thought on how we as LLVM user can ensure future stability of our proprietary compiler. We are currently considering contributing E2E test cases (LLVM IR → assembly) to LLVM, that represent the typical output of our compiler, also in complex scenarios. This would aim to ensure, that no new development inside of LLVM triggers any functional regression on our side.

So my questions would be:

1. Would such E2E tests generally be appreciated?
2. Where would we put such tests? Logically they don’t really fit in the test directory of any transform, since they test the whole pipeline end to end.

Best,  
Marc

---

<div class="post-metadata">

**Author:** ![Endill](https://sea1.discourse-cdn.com/flex021/user_avatar/discourse.llvm.org/endill/32/13614_2.png) [@Endill](https://discourse.llvm.org/u/Endill)\
**Post date:** [November 6, 2025, 5:32pm UTC](https://discourse.llvm.org/t/llvm-e2e-tests/88801/2 "2025-11-06T17:32:10Z")

</div>

I’m not a contributor to LLVM subproject, but in general there are two ways.

The first one, as you already mentioned, is contributing your tests upstream. In general you should have a good feedback on anything that improves our coverage.

The second is to track upstream and report failures before they slip into releases. Definitely more work for you than the first option, but for tests that can’t be upstreamed you don’t have much of a choice.

---

<div class="post-metadata">

**Author:** ![mshockwave](https://sea1.discourse-cdn.com/flex021/user_avatar/discourse.llvm.org/mshockwave/32/13674_2.png) [@mshockwave](https://discourse.llvm.org/u/mshockwave)\
**Post date:** [November 6, 2025, 5:43pm UTC](https://discourse.llvm.org/t/llvm-e2e-tests/88801/3 "2025-11-06T17:43:54Z")

</div>

> [@marcauberer](#):
>
> E2E test cases (LLVM IR → assembly)

Could you clarify the end-to-end here? Is it only the `llc` or a combination of `opt` and `llc`?

Generally speaking we’re welcome any kind of codegen (usually `llc`) or middle-end (usually `opt`) tests, as long as they improve the coverage. If the test is running a combination of `opt` and `llc`, is it possible to split them into separate tests for each tool?

---

<div class="post-metadata">

**Author:** ![dblaikie](https://sea1.discourse-cdn.com/flex021/user_avatar/discourse.llvm.org/dblaikie/32/977_2.png) [@dblaikie](https://discourse.llvm.org/u/dblaikie)\
**Post date:** [November 6, 2025, 6:41pm UTC](https://discourse.llvm.org/t/llvm-e2e-tests/88801/4 "2025-11-06T18:41:17Z")

</div>

Sounds like the tests you’re picturing are “large”-ish chunks of LLVM IR, then tested through a full middle end pass pipeline and through a backend of choice to assembly?

This is probably not the sort of test that should go into the LLVM tests directory - large means possibly slow, and means the failures aren’t as precise and actionable as we like to keep tests there.

So maybe it could go into the `test-suite`, though those run less often, etc.

And then there’s probably some questions to ask/answer about what these scenarios represent and whether they’re aligned with what the LLVM community wants to sign up to support. (if the community doesn’t want to sign up to support them - might be on you to run the tests & triage the failures before presenting them to the community - as someone else mentioned, you could do this frequently to reduce the latency/increase the chance you catch something before it’s baked in too far)

---

<div class="post-metadata">

**Author:** ![MatzeB](https://sea1.discourse-cdn.com/flex021/user_avatar/discourse.llvm.org/matzeb/32/13514_2.png) [@MatzeB](https://discourse.llvm.org/u/MatzeB)\
**Post date:** [November 6, 2025, 7:32pm UTC](https://discourse.llvm.org/t/llvm-e2e-tests/88801/5 "2025-11-06T19:32:21Z")

</div>

There is some precedent in `llvm-test-suite/Bitcode` (mostly for halide codegen) and the test-suite can differentiate between “benchmarks only” and “benchmarks + unit/correctness tests” mode. The tests have to be able to run though and produce some kind of “finished correctly” output.

Though admittedly debugging when things in the `Bitcode` directory fail is pretty annoying… so hoping the function/modules don’t become too big…

---

<div class="post-metadata">

**Author:** ![MatzeB](https://sea1.discourse-cdn.com/flex021/user_avatar/discourse.llvm.org/matzeb/32/13514_2.png) [@MatzeB](https://discourse.llvm.org/u/MatzeB)\
**Post date:** [November 6, 2025, 7:34pm UTC](https://discourse.llvm.org/t/llvm-e2e-tests/88801/6 "2025-11-06T19:34:39Z")

</div>

Also wanted to mention that many groups just operate in a different manner where the group itself makes sure it has a branch/development version that keeps up with day-to-day upstream changes and then runs internal tests themselfes and just providing minimized reproducers on the upstream PR as they find them. That tends to be the model that works best as people are most receptive when the breakages are “fresh” and when there’s a person responsible/reacting to questions about the minimized problem report… (and you can run as much company internal code as you want in this mode)

---

<div class="post-metadata">

**Author:** ![lenary](https://sea1.discourse-cdn.com/flex021/user_avatar/discourse.llvm.org/lenary/32/17025_2.png) [@lenary](https://discourse.llvm.org/u/lenary)\
**Post date:** [November 7, 2025, 3:47am UTC](https://discourse.llvm.org/t/llvm-e2e-tests/88801/7 "2025-11-07T03:47:57Z")

</div>

I would like to note that we have the `cross-project-tests` directory, some of which go from C all the way to objdump. I presume this would continue to be OK for (small, self-contained) future tests.

I don’t know which bots are running these tests, though.

---

<div class="post-metadata">

**Author:** ![marcauberer](https://sea1.discourse-cdn.com/flex021/user_avatar/discourse.llvm.org/marcauberer/32/26007_2.png) [@marcauberer](https://discourse.llvm.org/u/marcauberer)\
**Post date:** [November 7, 2025, 3:30pm UTC](https://discourse.llvm.org/t/llvm-e2e-tests/88801/8 "2025-11-07T15:30:53Z")

</div>

> [@mshockwave](#):
>
> Could you clarify the end-to-end here? Is it only the `llc` or a combination of `opt` and `llc`?

With end-to-end I mean input → assembly (or event better an executable binary)

> [@dblaikie](#):
>
> Sounds like the tests you’re picturing are “large”-ish chunks of LLVM IR, then tested through a full middle end pass pipeline and through a backend of choice to assembly?

Not necessarily. E.g. the issue, mentioned above, can be reproduced in \< 200 lines of IR. But yes, it would also be desirable to have the option for longer running tests as well.

> [@MatzeB](#):
>
> There is some precedent in `llvm-test-suite/Bitcode` (mostly for halide codegen) and the test-suite can differentiate between “benchmarks only” and “benchmarks + unit/correctness tests” mode. The tests have to be able to run though and produce some kind of “finished correctly” output.

Thanks for that hint. I had already browsed around in that repo, but it seemed rather benchmark related suite to me. For that reason I wanted to inquire your opinion if such tests could go there.

I now uploaded a draft of the first testcase here:

> <https://github.com/llvm/llvm-test-suite/pull/293>
>
> As discussed in this \[forum question\](https://discourse.llvm.org/t/llvm-e2e-test…s/88801), we hereby contribute the first SAP e2e test, that tests our common usage of \`indirectbr\`/\`blockaddr\` to achieve systematic outlining of functions.

Please let me know if I miss something here. I also attached the IR file, that the bitcode was generated from to ease debugging.

---

<div class="post-metadata">

**Author:** ![jh7370](https://sea1.discourse-cdn.com/flex021/user_avatar/discourse.llvm.org/jh7370/32/2079_2.png) [@jh7370](https://discourse.llvm.org/u/jh7370)\
**Post date:** [November 11, 2025, 11:38am UTC](https://discourse.llvm.org/t/llvm-e2e-tests/88801/9 "2025-11-11T11:38:40Z")

</div>

It’s been a while since I introduced `cross-project-tests`, but the intent was for these tests to still be targeted at small specific features, like most lit tests, but where the behaviour to test requires elements of multiple LLVM sub-projects (e.g. clang and LLVM binary utilities). I get the impression the tests being discussed here don’t fall under that category.
