This document presents our reflection for LLVM libc testing, where we are now and where we want to be in the near future. With inputs from @michaelrj-google @lntue and @nickdesaulniers.
Current state
In this section we describe the current status for configuring, building and testing LLVM libc.
Configuration
Configuration is two fold, one is provided via json config files, the other part is derived from the selected target options.
Json configuration
We currently support libc client customization via machine readable json file following the principles in [RFC] Systematic way to introduce and use libc config options.
A first config.json file in the config folder serves as a list of all user flags alongside with its human readable documentation and default value.
Configs are further refined by platform and architecture, each file taking precedence over its predecessor (e.g. customization for the bare metal platform).
Note:
- The json options are untyped. They can be any json value.
- There is currently no 1-to-1 relationship between json defined options and C++ preprocessor. Each author is responsible for implementing the logic behind the option.
Target derived configuration
When building LLVM libc, some features are enabled depending on the “target options” (see below). This is done via LLVMLibCCheckCpuFeatures where we iterate through a list of features and compile a dummy program to check whether the feature is present. For instance, we can tell whether the targeted platform supports AVX2.
By “target options” we mean the conjunction of target triple (i.e., LIBC_TARGET_TRIPLE) and target tuning options (i.e., LIBC_COMPILE_OPTIONS_DEFAULT). The latter allows passing options such as -march / -mtune or more specific options like -mno-avx2.
Building
The final LLVM libc library machine code is a combination of json and target options. This configuration is not exposed directly, but rather derived from the target options and enabled when available.
For instance, passing -march=haswell will enable FMA usage where relevant.
Testing
Inside CMake, we currently have two ways of testing the configurations:
- A manual one where we add as many targets as needed (e.g. memory function testing), this allows testing json and target options by specifying them explicitly.
- An automatic one that tests the cartesian product of predefined options. Historically, this has been setup for math functions and is driven by functions in LLVMLibCFlagRules. We basically create many different versions of each internal target, allowing us to individually select the state of each option. Dependencies between functions are also taken into account.
Issues with the current state
- The logic to transfer json defined options to C++ preprocessor is ad hoc and spread out through the build system. If it goes missing it will not trigger any error.
- For building the libc, we currently don’t have a way to produce the full list of options that have been taken into account (a bit like the
.configfile when building the Linux Kernel). - We have two mechanisms for testing, none of which covers the entire configuration space in a principled way.
Proposed improvements
1. Link json config to C++ preprocessor
By adding a preprocessor key on some of the tags we can instruct CMake to automatically forward the configuration to the compiler.
"LIBC_CONF_PRINTF_DISABLE_FLOAT": {
"value": true,
"doc": "Disable printing floating point values in printf and friends.",
"preprocessor": "LIBC_COPT_PRINTF_DISABLE_FLOAT"
},
This would result in CMake generating a -DLIBC_COPT_PRINTF_DISABLE_FLOAT=1.
2. Infer feature usage from code
We could have a tool (clang based or simpler) to grep through target sources and look for #if lines containing LIBC_TARGET_CPU_HAS_(XXX) and attach this information to the target. This way we could list all features participating in different code paths.
3. Testing
Gather build configuration options
Using the provided target triple (i.e., LIBC_TARGET_TRIPLE) we can determine the availability of the cpu features either through direct compilation or cross compilation.
We can write the union of the json and target options in a file describing how the libc was built (a la Linux Kernel .config).
Testing for some specific target options
For some particular target options (see above), the set of cpu features is completely defined.
Additionally, we assume that all json options are orthogonal and so we don’t need to create the cartesian product of all combinations, for N options we can have N variations with only one of them being active at a time.
Thorough testing
We want to be able to test all interesting combinations of json and target options. Target options are not independent and most of the time some target options imply other target options (e.g., for x86 FMA is available when AVX is).
To test all target options it is probably sufficient to enumerate a fixed list of target triple for each supported architecture that will exercise all interesting combinations of target options.
For x86, the x86-64 psABI describes Microarchitecture levels with corresponding cpu features, testing all of them should cover all cpu feature combinations : x86-64-v1, x86-64-v2, x86-64-v3 and x86-64-v4.
We hope to define similar hierarchy of options for other architectures.
A thorough testing of LLVM libc will consist in running tests for {architecture} × {target triple} × {json option}.
Test runners
When the target architecture does not match the host architecture there are actually several possibilities:
- Compile and run tests only when
targetmatcheshost. - Run tests only when
targetmatcheshost. But cross compile if possible - there is value in checking that it compiles without warning. - Cross compile and only run test binaries on -separate- dedicated hardware.
- Cross compile and cross test.
We believe that 4 holds the most value.
We may have missed important use cases so please use this thread to discuss and comment the proposal.
Thank you