Testing libc++ documentation page is incorrect or incomplete

I’ve been trying to follow the Testing libc++ documentation in order to exercise some tests against a modified glibc installation. Unfortunately the documentation seems to be incorrect or incomplete, particularly in the Usage section.

In order to preempt other “did you turn it on” questions, I’ll state that make check-cxx works perfectly fine for me.

First, the documentation suggests running make -C <build> cxx-test-depends from the root of the monorepo. <build> is presumably intended to mean the top of the build directory for the full LLVM build (e.g., the current directory when cmake was run unless the -B option was specified). This is implied by the follow on commands that indicate that llvm-lit is expected to be found in the <buid>/bin directory. However, the cxx-test-depends is not recognized at that level.

$ make -C ../llvm-project-build cxx-test-depends
make: Entering directory '/.../llvm-project-build'
make: *** No rule to make target 'cxx-test-depends'.  Stop.
make: Leaving directory '/.../llvm-project-build'

The simple <build>/bin/llvm-lit commands that follow that in the documentation don’t seem to work for me. I was eventually able to figure out how to get an individual test to run, but instead of doing as the documentation indicated:

$ ../llvm-project-build/bin/llvm-lit -sv libcxx/test/std/depr/depr.c.headers/stdlib_h.pass.cpp
llvm-lit: /home/tom/projects/llvm-project/libcxx/test/lit.cfg.py:7: fatal: You seem to be running Lit directly -- you should be running Lit through <build>/bin/llvm-lit, which will ensure that the right Lit configuration file is used.

I had to do:

$ ../llvm-project-build/bin/llvm-lit -sv ../llvm-project-build/runtimes/runtimes-bins/libcxx/test/std/depr/depr.c.headers/stdlib_h.pass.cpp
...
Testing Time: 0.17s
  Passed: 1

Finally, once I figured that out, I wanted to try running tests with an alternate system include root. The documentation suggests doing something like:

<build>/bin/llvm-lit ... --param=compile_flags='-isystem /path/to/include'

But that seemed to have no effect. I was eventually able to do the testing I wanted to do by modifying the runtimes/runtimes-bins/libcxx/test/lit.site.cfg file in the <build> directory, but I suspect a better method exists.

I’ll be happy to submit a patch for the documentation, but will need guidance on what the best practices actually are for doing one off testing like this these days.

Tom.

Hi Tom,

Thanks for your report!

There are two ways to build libc++
https://libcxx.llvm.org/BuildingLibcxx.html
I use the default build and things work for me. Looking at the output I assume you have used the Bootstrapping build. Is that correct?

Cheers,
Mark

Hi, Mark, thanks for the response!

Yes, I am doing a bootstrap build. Specifically:

$ mkdir llvm-project-build
$ cd llvm-project-build
$ CC=gcc CXX=g++ cmake \
    -G "Unix Makefiles" \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_INSTALL_PREFIX="$INSTALL_DIR" \
    -DLLVM_ENABLE_PROJECTS=clang \
    -DLLVM_ENABLE_RUNTIMES="compiler-rt;libcxx;libcxxabi" \
    -DLLVM_BUILD_TESTS=ON \
    "../llvm-project/llvm"
$ make -j 4 && make install

Thanks a lot for the heads up. There are/were a couple of problems here.

First, the documentation of what Lit parameters are supported was out-of-date, I just updated it and I will cherry-pick that back to LLVM 15. I hope the updated documentation is more useful and less misleading, please don’t hesitate to provide feedback and I’ll make it better. Long story short, what you want to do instead of using --param compile_flags=XXXXX is define your own base Lit config. That way, you will make sure that you don’t have duplicate system header search paths, which would render the tests brittle when using --param compile_flags=XXX. You can base your own base Lit config on e.g. libcxx/test/configs/llvm-libc++-shared.cfg.in. I would expect that you need something like this:

lit_config.load_config(config, '@CMAKE_CURRENT_BINARY_DIR@/cmake-bridge.cfg')

config.substitutions.append(('%{flags}', ''))
config.substitutions.append(('%{compile_flags}',
    '-nostdinc -I %{include} -I %{target-include} -isystem /your/include/path -I %{libcxx}/test/support'
))
config.substitutions.append(('%{link_flags}',
    '-nostdlib -L %{lib} -Wl,-rpath,%{lib} -lc++ -l your-system-library'
))
config.substitutions.append(('%{exec}',
    '%{executor} --execdir %T -- '
))

import os, site
site.addsitedir(os.path.join('@LIBCXX_SOURCE_DIR@', 'utils'))
import libcxx.test.params, libcxx.test.newconfig
libcxx.test.newconfig.configure(
    libcxx.test.params.DEFAULT_PARAMETERS,
    libcxx.test.features.DEFAULT_FEATURES,
    config,
    lit_config
)

This acts kind of like a CMake toolchain file – it tells the test suite how to build a hello world program in your desired setup, and everything else in the test suite is defined on top of that. Once you have this “toolchain file” created, you need to point the test suite to it by passing -DLIBCXX_TEST_CONFIG=<path-to-lit-config> at CMake configuration time.

The other issue you encountered (but have found the workaround for) is that the Runtimes build does not set up llvm-lit correctly, which is why you have to use lit -sv llvm-project-build/runtimes/runtimes-bins/libcxx/test/etc instead of just lit -sv libcxx/test/etc. This is a known issue, and honestly I’m not sure how to best fix that, I have to look into it.

Thank you, @ldionne!

The changes to the testing page look great. I validated the runtimes-test-depends target for a bootstrap build. The link to libcxx/utils/libcxx/test/params.py is especially useful too!

The information about creating a custom site config seems useful. In my case, I just needed to do a one-off test, so it ended up being easier to temporarily update libcxx/test/configs/llvm-libc++-shared.cfg.in in the monorepo and run make check-cxx as usual. That might be worth mentioning as a simple test technique; it nicely avoids having to re-run cmake and perform a separate build (especially nice when doing bootstrap builds).

The other issue you encountered (but have found the workaround for) is that the Runtimes build does not set up llvm-lit correctly, which is why you have to use lit -sv llvm-project-build/runtimes/runtimes-bins/libcxx/test/etc instead of just lit -sv libcxx/test/etc. This is a known issue, and honestly I’m not sure how to best fix that, I have to look into it.

Glad to hear that is a known issue. Is there a GitHub issue tracking the problem?