buildbot failure in LLVM on sanitizer-x86_64-linux-fast

Didn’t reproduce on my OS X asan build.

This is probably an old bug in the unittests/MC test code or possibly in the code that it is testing. I recommend disabling unittests/MC on asan configurations until it can be investigated, but I don’t know how to do that. Other options are to disable unittests/MC unconditionally, or roll back r280455 until unittests/MC can be fixed.

Is there someone who can reproduce this who can take it over?

Nine tests on five builders have test failures here. Again, these are old bugs that were previously undetected.

llvm/unittests/MC
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-bootstrap/builds/12834
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fast/builds/16015

tsan/tests/rtl/TsanRtlTest
tsan/tests/unit/TsanUnitTest
http://lab.llvm.org:8011/builders/sanitizer_x86_64-freebsd/builds/13492

unittests/DebugInfo/DWARF
unittests/DebugInfo/PDB
unittests/ExecutionEngine/Orc
unittests/ExecutionEngine/MCJIT
unittests/Transforms/IPO
unittests/Transforms/Utils
http://lab.llvm.org:8011/builders/llvm-mips-linux/builds/17616
http://lab.llvm.org:8011/builders/clang-ppc64le-linux-multistage/builds/2331

If we need to get these builders clean quickly we can temporarily downgrade the new error to a warning. Patch:

downgrade-googletest-error.patch (673 Bytes)

The Buildbot has detected a new failure on builder sanitizer-x86_64-linux-fast while building llvm.
Full details are available at:
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fast/builds/16009

lit.py: /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/utils/lit/lit/formats/googletest.py:39: error: unable to discover google-tests in ‘/mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm_build_asan/unittests/MC/MCTests’: Command ‘[’/mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm_build_asan/unittests/MC/MCTests’, ‘–gtest_list_tests’]’ returned non-zero exit status 1

This is a new lit error message introduced by r280455. I expect that the MCTests executable is hitting an asan error when run to report what tests it contains. Before r280455 the test silently passed as a result. Now the test fails.

Ivan Krasin wrote:

URL: http://llvm.org/viewvc/llvm-project?rev=280550&view=rev
lit: print process output, if getting the list of google-tests failed.

Good idea, Ivan.

The mips and ppc64le builders haven’t rebuilt with this yet.
http://lab.llvm.org:8011/builders/llvm-mips-linux
http://lab.llvm.org:8011/builders/clang-ppc64le-linux-multistage

The freebsd builder rebuilt but provided no new output. Exit code -11 is presumably SIGSEGV.
http://lab.llvm.org:8011/builders/sanitizer_x86_64-freebsd/builds/13509

lit.py: /usr/home/buildslave/as-bldslv5/sanitizer_x86_64-freebsd/llvm.src/utils/lit/lit/formats/googletest.py:41: error: unable to discover google-tests in ‘/usr/home/buildslave/as-bldslv5/sanitizer_x86_64-freebsd/llvm.obj/projects/compiler-rt/lib/tsan/tests/rtl/TsanRtlTest’: Command ‘[’/usr/home/buildslave/as-bldslv5/sanitizer_x86_64-freebsd/llvm.obj/projects/compiler-rt/lib/tsan/tests/rtl/TsanRtlTest’, ‘–gtest_list_tests’]’ returned non-zero exit status -11. Process output:

lit.py: /usr/home/buildslave/as-bldslv5/sanitizer_x86_64-freebsd/llvm.src/utils/lit/lit/formats/googletest.py:41: error: unable to discover google-tests in ‘/usr/home/buildslave/as-bldslv5/sanitizer_x86_64-freebsd/llvm.obj/projects/compiler-rt/lib/tsan/tests/unit/TsanUnitTest’: Command ‘[’/usr/home/buildslave/as-bldslv5/sanitizer_x86_64-freebsd/llvm.obj/projects/compiler-rt/lib/tsan/tests/unit/TsanUnitTest’, ‘–gtest_list_tests’]’ returned non-zero exit status -11. Process output:

One of the x86_64 builders has a result:
http://lab.llvm.org:8011/builders/sanitizer-x86_64-linux-fast

lit.py: /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/utils/lit/lit/formats/googletest.py:41: error: unable to discover google-tests in ‘/mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm_build_asan/unittests/MC/MCTests’: Command ‘[’/mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm_build_asan/unittests/MC/MCTests’, ‘–gtest_list_tests’]’ returned non-zero exit status 1. Process output:

==27367==ERROR: AddressSanitizer: initialization-order-fiasco on address 0x000002de0c90 at pc 0x000000bad579 bp 0x7ffd584afed0 sp 0x7ffd584afec8
READ of size 8 at 0x000002de0c90 thread T0
#0 0xbad578 in llvm::TargetRegistry::RegisterTarget(llvm::Target&, char const*, char const*, bool (*)(llvm::Triple::ArchType), bool) /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/Support/TargetRegistry.cpp:99:9
#1 0x5f94ba in LLVMInitializeAArch64TargetInfo /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/Target/AArch64/TargetInfo/AArch64TargetInfo.cpp:23:3
#2 0x420d2c in InitializeAllTargetInfos /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm_build_asan/include/llvm/Config/Targets.def:26:1
#3 0x420d2c in Context /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/unittests/MC/DwarfLineTables.cpp:30
#4 0x420d2c in __cxx_global_var_init /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/unittests/MC/DwarfLineTables.cpp:49
#5 0x420d2c in _GLOBAL__sub_I_DwarfLineTables.cpp /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/unittests/MC/DwarfLineTables.cpp
#6 0xf8e50c in __libc_csu_init (/mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm_build_asan/unittests/MC/MCTests+0xf8e50c)
#7 0x7f7c5a46fed4 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21ed4)
#8 0x4aa6a9 in _start (/mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm_build_asan/unittests/MC/MCTests+0x4aa6a9)

0x000002de0c90 is located 16 bytes inside of global variable ‘llvm::TheARM64Target’ defined in ‘/mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/lib/Target/AArch64/TargetInfo/AArch64TargetInfo.cpp:17:8’ (0x2de0c80) of size 208
registered at:
#0 0x4bc87d in __asan_register_globals.part.15 /mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm/projects/compiler-rt/lib/asan/asan_globals.cc:338
#1 0x5f955b in asan.module_ctor (/mnt/b/sanitizer-buildbot3/sanitizer-x86_64-linux-fast/build/llvm_build_asan/unittests/MC/MCTests+0x5f955b)

TargetRegistry.cpp:99:

// Check if this target has already been initialized, we allow this as a
// convenience to some clients.
if (T.Name)
return;

Hey Greg,

I've applied your downgrade patch in r280674 to unbreak the build.
I've been investigating another issue with asan unittests on OS X, and
it looks like https://reviews.llvm.org/D24184 should fix a few others,
but not all. I filed http://llvm.org/PR30285.

-Ahmed