I looked around in the logs, looking for the config/build commands that would reproduce the failure. The problem is that the bot seems to be using a script which I don’t have access to (buildbot_bootstrap.sh). So, I looked for cmake/ninja calls. AFAICT, the bot does a bootstrap and then runs the testsuite with the final build configured with -DLLVM_USE_SANITIZER=Memory.
I tried that locally, but the build dies very early in tblgen:
I’m not sure how to proceed from here. The bot is clearly building things in a different way, but I don’t know how to duplicate it. Is there a way for me to use the same script that the bot is using? What is the general advice on reproducing buildbot failures?
They appear to use a prebuilt libstdc++ shared object.
I think the bot usually gives a readable error report, but it doesn’t work for this test because the test is passing stderr to FileCheck. Lots of tests do that, and we should find a way to make that work. We might want to pass MSAN_OPTIONS=log_path=/tmp/something.log and then cat that file from lit if it’s non-empty.
Thanks, Reid. I've gotten the script and I'm now running it locally. It's
running into trouble in llvm_build2_asan, so I'll have to kick it a bit
first.
I think the bot usually gives a readable error report, but it doesn't work
for this test because the test is passing stderr to FileCheck. Lots of
tests do that, and we should find a way to make that work. We might want
to pass MSAN_OPTIONS=log_path=/tmp/something.log and then cat that file
from lit if it's non-empty.
Yeah. From the output, I can see part of the msan failure, but I can't see
the whole thing. Worse, I can't even replicate it.
#1 0x2888c1d in llvm::raw_ostream::write(char const*, unsigned long) /home/dtoolsbot/build/sanitizer-x86_64-linux-bootstrap/build/llvm/lib/Support/raw_ostream.cpp:332
But since this is a lit test, we don’t see the whole output and so this requires a local rebuild and rerun
(and then you need llvm-symbolizer in PATH).
I wonder if we can configure the lit test runner to print the (tail of) test output on failure.
Thanks. I've now found and fixed the bug. Another thing that may help in
the future is to have asan display a warning/note when it can't find
llvm-symbolizer. This would help with figuring out why it's giving an
unreadable trace.
You'd have to teach FileCheck, actually, since it's the one that consumes
stderr in this case. Rather than doing that, why not use
*SAN_OPTIONS=log_file=blah.txt, and teach lit to dump that on failure?
I wished in the recent past that FileCheck would print its input upon failure. It currently only prints a single line with “possible intended match: blah” which may or may not be the line you’re interested in. I can work on this if people think that this would be a good feature.