Hello!
I am writing an instrumentation pass using LLVM. I have unit tests using llvm-lit in the style of the tests in llvm/test to make sure the IR changes show up in the correct place. I would also like to have some end-to-end tests that start from a .c file, compile the program, run it, and check the output.
I imagine I could do this as a test in the same style, but I'm wondering if that's the llvm-correct way of doing it. Can anyone advise?
Thank you!
Eli
Typically the “end-to-end” test cases reside in projects/test-suite. All the test cases there build executable binaries from source, run those and compare the output (as well as the return code) to what’s expected. You should probably put your test cases in there. Please note also that there are subcategories there - there are single source file and multiple source file programs. There are those that are meant as benchmarks, unit tests, full applications, etc. I’m sure you can find an appropriate place to add your test cases to.
Ah, thank you, that makes sense.
Is there any chance you can point me toward some documentation for how to add tests to the test-suite? I’ve found a fair bit on how to run the tests, but nothing on how to add new ones. Is it just as simple as adding testcase.c and testscase.reference_output in the appropriate directory?
Thanks!
I am far from an expert on this since I’ve only ever added one or two of these IIRC. But as far as I remember, for most purposes, that’s all you need. Add a source file, add an expected file and add the source file to the CMakeLists.txt (along with any additional CFLAGS/CXXFLAGS/LDFLAGS)…
Of course, if you need anything custom, there are likely examples in the test suite that you can use for guidance.
Also, this may be of some help:
https://llvm.org/docs/TestSuiteMakefileGuide.html
Thank you very much. I’ll give it a shot!
Typically this would be tested alongside the runtime component in compiler-rt.
For example, PGO instrumentation has its “end-to-end” tests in https://github.com/llvm-mirror/compiler-rt/tree/master/test/profile
– Sean Silva
Ah, that makes a lot of sense. Thank you! Do you have any tips (or points to documentation) for how to run compiler-rt tests? Just invoking llvm-lit doesn’t seem to work
Ah, that makes a lot of sense. Thank you! Do you have any tips (or points
to documentation) for how to run compiler-rt tests? Just invoking llvm-lit
doesn't seem to work
They should run as part of check-all when you have compiler-rt checked out
in llvm/projects/ (or the equivalent with the monorepo setup, which I think
is just setting the appropriate LLVM_EXTERNAL_*_SOURCE_DIR cmake variable).
I forget, but I'd be surprised if there isn't a "check-compiler-rt" or
similar. I've also used llvm-lit in the past (look at what
check-compiler-rt does; I think it just calls llvm-lit).
-- Sean Silva