Please, can anybody show me the required steps for running the LLVM regression tests, or point me to a detailed doc?
I read the available documentation in https://llvm.org/docs/TestingGuide.html but that’s mostly a general description on what’s available rather than an “guide”.
This is what I tried:
- Installed both LLVM and CLANG using CMAKE on a Mac OS X computer. (Sources compiled fine, and everything seems to work as I’m already at the final steps of a custom LLVM target implementation)
- Python 2.7.10 is already installed.
- From the command line (terminal app) I tried to execute python lit.site.cfg.py , however I get the following error:
File “/Users/joan/LLVM+CLANG/build/test/lit.site.cfg.py”, line 6, in
config.host_triple = “x86_64-apple-darwin16.7.0”
NameError: name ‘config’ is not defined
There seem to be something obvious that I miss, so any help is appreciated.
Please be aware that I’m an experienced C, C++, Obj-C programmer (now retired) but I do not speak a word of python.
Thanks,
John
I read the available documentation in LLVM Testing Infrastructure Guide — LLVM 18.0.0git documentation but that's mostly a general description on what’s available rather than an “guide”.
It mentions the main commands: "make check-all" will run the
regression tests for all built components. There are also more
specific targets like "check" (LLVM only), "check-clang", ...
If you're using ninja you can get it to list them with something like
"ninja -t targets | grep check".
- From the command line (terminal app) I tried to execute python lit.site.cfg.py , however I get the following error:
File "/Users/joan/LLVM+CLANG/build/test/lit.site.cfg.py", line 6, in <module>
config.host_triple = "x86_64-apple-darwin16.7.0"
NameError: name 'config' is not defined
A tool bin/llvm-lit is generated for running tests either individually
or when pointed at a test directory. It takes care of the needed
imports, PYTHONPATHs and so on. It's invoked by the "check" targets in
make. That's the only real way to interact with lit stuff directly.
It's mostly useful when debugging a single test that's failed
"bin/llvm-lit -v /path/to/test.ll" quickly tells you if your
attempted fix has worked. I occasionally use it to test all CodeGen
for a particular target without running other things that I don't care
about right now ("bin/llvm-lit -v
/path/to/llvm/test/CodeGen/AArch64").
Cheers.
Tim.
Hi Tim,
When I started with this, I just downloaded the sources for the LLVM site, I did not even created a git for it. Then ran the CMAKE app (not from the command line) with the default settings other than also adding the ‘experimental’ targets, and boom a fully functional XCode project was created, so I gave no further thought on that. I am not even aware of what was involved under the hood, but this has worked fine since I have been able to create a totally new custom target from that install with no issues. That’s why now I’m surprised that the tests doesn’t seem to even be there. I think that CMAKE is one of the major ways to install LLVM, and I may try to install it and ‘clean’ compile from the ground up again, but I would need to understand what’s required to actually get the tests available.
Thanks,
Hi Joan,
When I started with this, I just downloaded the sources for the LLVM site, I did not even created a git for it. Then ran the CMAKE app (not from the command line) with the default settings other than also adding the ‘experimental’ targets, and boom a fully functional XCode project was created, so I gave no further thought on that.
Oh, Xcode. Now that's a large wrinkle in matters. I take it that means
you used "cmake -G Xcode"? I think most people who work on LLVM use
either the "Unix Makefiles" or he "Ninja" backends for CMake, which do
generate the check destinations needed. The xcode backend would
completely replace any make/ninja files CMake would generate so that
would explain what you're seeing.
Some more people (I think) have a separate xcode build directory
purely for indexing/IDE but do their main builds in a make/ninja
environment.
Anyway, I've just generated an Xcode project myself and (by running
"xcodebuild -list") there does appear to be an "xcodebuild -target
check" (etc). I've not actually run them because at the moment I only
have access to a laptop that would take quite a while to finish, but I
think that's probably your best bet.
Cheers.
Tim.
Hi Tim,
Thanks again for that.
I meant to say that did NOT use any terminal command to create the project, I just ran the CMake app. I mean I didn’t even use the ‘terminal' app for that.
So, I tried now what you suggested, and in my case, after running xcodebuild -list, a long list of targets appear, but none with the text “check” in it. I’m utterly confused on what to try next.
John
Hi Tim,
Anyway, I seem to have managed to make it work. The ‘make' commands do not work properly after you create a Xcode project with CMake, but the ‘check’ Targets are still there to be run from the Xcode IDE (only as a whole though).
However, the 'llvm-lit' tool still works just fine, so I can just use that to run sets of tests from the command line.
Thanks for your help!
John