How do I run llvm's asan tests?

Hi llvm-dev,

I'm trying to figure out how to contribute to LLVM, in particular a followup to
kcc's commit 6bde702a in sanitzer_suppressions.cc. However I can't find a way
to get the tests to run before I even change anything.

The relevant unit test is
compiler-rt/lib/sanitizer_common/tests/sanitizer_suppressions_test.cc, so I
expect I want `ninja check-asan` (right?). Here's what I tried, on my Debian
machine, following (and adjusting) the outdated llvm documentation about
setting up a GCC toolchain:

    # Install the GCC toolchain.
    # See http://llvm.org/docs/GettingStarted.html#getting-a-modern-host-c-toolchain
    # Adjusted for GCC 7.4.0, which doesn't ship in a bzip2 archive.
    gcc_version=7.4.0
    wget https://ftp.gnu.org/gnu/gcc/gcc-$\{gcc\_version\}/gcc\-$\{gcc\_version\}\.tar\.gz
    wget https://ftp.gnu.org/gnu/gcc/gcc-$\{gcc\_version\}/gcc\-$\{gcc\_version\}\.tar\.gz\.sig
    wget https://ftp.gnu.org/gnu/gnu-keyring.gpg
    signature_invalid=`gpg --verify --no-default-keyring --keyring
./gnu-keyring.gpg gcc-${gcc_version}.tar.gz.sig`
    if [ $signature_invalid ]; then echo "Invalid signature" ; exit 1 ; fi
    tar -xvzf gcc-${gcc_version}.tar.gz
    cd gcc-${gcc_version}
    ./contrib/download_prerequisites
    cd ..
    mkdir gcc-${gcc_version}-build
    cd gcc-${gcc_version}-build
    mkdir -p $HOME/toolchains
    $PWD/../gcc-${gcc_version}/configure --prefix=$HOME/toolchains
--enable-languages=c,c++
    make -j$(nproc)
    make install

    # Clone llvm and try to test it.
    mkdir -p ~/clients
    cd ~/clients
    git clone https://github.com/llvm/llvm-project.git
    # Get a coffee
    cd llvm-project
    mkdir build
    cd build
    CC=$HOME/toolchains/bin/gcc CXX=$HOME/toolchains/bin/g++ \
      cmake -G Ninja
-DCMAKE_CXX_LINK_FLAGS="-Wl,-rpath,$HOME/toolchains/lib64
-L$HOME/toolchains/lib64"
-DLLVM_ENABLE_PROJECTS='clang;compiler-rt;libcxx;libcxxabi' ../llvm
    ninja check-asan
    # Get another coffee

After about 30 minutes, the ninja command fails with an error about how the
string header isn't found:

    FAILED: projects/compiler-rt/lib/asan/tests/ASAN_NOINST_TEST_OBJECTS.asan_noinst_test.cc.x86_64-inline.o
    cd /usr/local/google/home/jacobsa/clients/llvm-project/build/projects/compiler-rt/lib/asan/tests
&& /usr/local/google/home/jacobsa/clients/llvm-project/build/./bin/clang
-fPIC -fvisibility-inlines-hidden -Werror=date-time -std=c++11 -Wall
-Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual
-Wno-missing-field-initializers -pedantic -Wno-long-long
-Wimplicit-fallthrough -Wno-maybe-uninitialized -Wno-noexcept-type
-Wdelete-non-virtual-dtor -Wno-comment -fdiagnostics-color -Wall
-std=c++11 -Wno-unused-parameter -Wno-unknown-warning-option
-DGTEST_NO_LLVM_RAW_OSTREAM=1 -DGTEST_HAS_RTTI=0
-I/usr/local/google/home/jacobsa/clients/llvm-project/llvm/utils/unittest/googletest/include
-I/usr/local/google/home/jacobsa/clients/llvm-project/llvm/utils/unittest/googletest
-I/usr/local/google/home/jacobsa/clients/llvm-project/compiler-rt/include
-I/usr/local/google/home/jacobsa/clients/llvm-project/compiler-rt/lib
-I/usr/local/google/home/jacobsa/clients/llvm-project/compiler-rt/lib/asan
-I/usr/local/google/home/jacobsa/clients/llvm-project/compiler-rt/lib/sanitizer_common/tests
-fno-rtti -O2 -Wno-format -Werror=sign-compare -Wno-non-virtual-dtor
-Wno-variadic-macros -gline-tables-only -DASAN_HAS_BLACKLIST=1
-DASAN_HAS_EXCEPTIONS=1 -DASAN_UAR=0 -m64 -c -o
ASAN_NOINST_TEST_OBJECTS.asan_noinst_test.cc.x86_64-inline.o
/usr/local/google/home/jacobsa/clients/llvm-project/compiler-rt/lib/asan/tests/asan_noinst_test.cc
    In file included from
/usr/local/google/home/jacobsa/clients/llvm-project/compiler-rt/lib/asan/tests/asan_noinst_test.cc:17:
    In file included from
/usr/local/google/home/jacobsa/clients/llvm-project/compiler-rt/lib/asan/tests/asan_test_utils.h:18:
    /usr/local/google/home/jacobsa/clients/llvm-project/compiler-rt/lib/asan/tests/asan_test_config.h:19:10:
fatal error: 'string' file not found
    #include <string>
             ^~~~~~~~
    1 error generated.

This doesn't happen for `ninja check-all` (but despite its name that target
doesn't seem to run the test), so I wonder if it's a hermeticism issue in the
asan tests? It's also very possible I've just done it wrong. Help?

Thanks,
Aaron

Your steps seem to be correct....

I've just done this, and it worked fine:

git clone https://github.com/llvm/llvm-project.git
cd llvm-project
mkdir build
cd build
cmake -G Ninja -DLLVM_ENABLE_PROJECTS='clang;compiler-rt;' ../llvm
ninja check-asan

My gcc is "gcc (Debian 7.3.0-5) 7.3.0"

No good ideas what's broken on your side...

--kcc

clang can’t find your STL headers, I guess. You probably have some GCC libraries installed in /usr/lib for some recent version of GCC (8.N), but no headers for that same version. Clang’s GCC installation detection logic will find the libraries, and use the corresponding version to calculate standard library include paths.

The easiest fix is probably to run sudo apt-get install libstdc++-8-dev, assuming you have some libraries for libstdc++8 but no headers.

Thanks Kostya and Reid for confirming the usual process. After a tremendous
amount of faffing around, I did manage to make this work on a Google Cloud
Engine instance that I created from scratch, as follows:

    gcloud compute instances create debian --image-project
debian-cloud --image-family debian-9 --machine-type n1-standard-16
--project <redacted> --zone australia-southeast1-a --boot-disk-size
1TB
    gcloud compute ssh debian

    sudo apt-get install gcc g++ git cmake ninja-build
    gcc --version # gcc (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
    g++ --version # ++ (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
    cmake --version # cmake version 3.7.2
    ninja --version # 1.7.2

    git clone https://github.com/llvm/llvm-project.git
    cd llvm-project
    mkdir build
    cd build
    cmake -G Ninja -DLLVM_ENABLE_PROJECTS='clang;compiler-rt;' ../llvm
    ninja check-asan

    # But that doesn't actually run the test I want, so:
    ninja check-sanitizer

So I guess this was a problem with my local setup (a relatively stock
Google-internal linux image) in the end. Sorry for the noise.

Aaron

If that’s the case, then it’s extra likely that you just need to run sudo apt-get install libstdc++-8-dev locally to fix the issue. I had the same problem (clang not finding STL headers). I’ll send you a link to the internal bug off list.