My question is with regards to builds which enable multiple targets, say AArch64 and X86, as part of a build.
When lit is executed as part of check-all, the suite selects a default target via querying the clang/llvm binaries and/or the artifacts of cmake configuration. This target is the one used exclusively for any tests originating from that configuration.
Does this not result in missing test coverage?
Tests for generic passes which query TargetInstrInfo, for example, will only be covered for a single target.
To achieve this coverage, it seems that one would have to configure LLVM once for each target, but thatās effectively duplicating the full process of configuration and compilation just to change the value of what amounts to one macro.
Is there another way to do this? Or a better way to choose which target is used by check-all? Two paths Iāve followed are:
Trying to find some way to change the default target after building, but it seems to be hard-coded and compiled into the executables
Following litās execution to the point it selects which clang/llvm binary to use and the options to give. Thereās some promise here, but complications with defining substitutions have given me some pause (and also made me write up this post!)
The generic target tests are supposed to just test generic things, whilst the target-specific tests that run so long as the target is enabled are supposed to give coverage for the backend in question. In theory, at least.
Incorrect. It is simply the default target, if no explicit triple is specified in the test. Many tests do specify an explicit triple, which is obeyed. In general, tests for a particular target architecture (X86, Sparc, whatever) are included if the LLVM build includes that target. Tests in target-specific directories generally ought to specify a triple, because that target might not be the default.
It is also the case that many tests work correctly only in a particular operating system environment (Windows, Linux, MacOS, etc) and again those tests specify constraints (REQUIRES or UNSUPPORTED directives, or similar constraints expressed in a lit config file per-directory) so they are run only in the appropriate environment. In principle we try to have buildbots that cover all these different environments. In practice we do cover a great many of them.
I understand that some tests specify and test a particular target, and some do not.
I understand that tests that do not specify a target may test generic things.
However, I would venture a guess to say that a vast majority of passes in LLVM use the target in some capacity through things like TargetInstrInfo and TargetTransformInfo.
You could argue that the loss in test coverage for these functions in an amalgamated build is a degenerate case and that separate compilation of each back end is the price you pay, but Iām approaching this as a developer that wishes to build LLVM once and use the result to generate multiple toolchains out of convenience and efficiency.
I had hoped there was an easier way to switch the default target.
I donāt spend a lot of my time poking around in passes, but my impression is the āmiddle endā optimizer doesnāt use that stuff much, and when it does itās not usually critical to the operation of the pass. The story is likely a bit different in the backend, which does still have many common passes but is more likely to be taking advantage of target-specific things. However, most tests are very targeted to what they are trying to test, and the infrequency of target variant results in the generic tests tells me that thereās not a lot of target-specific corners in those tests, and hence not in the passes they are testing.
As for switching the default target, there is a trick you can use. If the name of the clang executable has a prefix that is a triple, that is the default triple it will use. For example, x86_64-linux-clang has a default triple of x86_64-linux. Rename it to armv7a-linux-clang and poof, the default triple is armv7a-linux. (Assuming you have built with both the X86 and ARM targets.)
Soft links, hard links, copies, all should work. I donāt know if other tools have the same feature, but certainly clang works this way.
The clang trick works, and we actually use that in our deliverables to choose.
However, the generated check-all target (i.e. bin/llvm-lit) doesnāt have that luxury. It hard-codes the cmake-configured default targets out of $build/bin. It looks like, however, that LLVM_TARGET_TRIPLE_ENV might be what Iām looking for, as far as testing goes.
Perhaps Iām square-peg-round-holing this and should just call lit on my own, but Iām worried about losing coverage tied to check-all that isnāt obvious, like ācheck-clang-pythonā.