Buildbots missing stack traces

Yesterday I’ve received failure reports due to crashes from a bunch of different build bots, all of which are missing symbolized stack traces, here’s a list:

I believe this used to work out-of-the-box for most buildbots. I wonder if something regressed in the default buildbot template.

This is rendering certain fixes impossible, especially when the crash is not reproducible on a platform where developer has access to.

Looking at your links, all of the buildbots are doing Release builds, so no debug info. I cannot speak for the other buildbots, but for the one I manage, we do it for two reasons. The first is that this is the configuration we test internally, and second, it allows for a faster turnaround.

The CMake commands are in the build details, so you should be able to reproduce the issue by running those. If not, at least for the buildbots that I manage, you are welcome to contact me if you need some help reproducing a failure on the buildbot that I manage.

Looking at your links, all of the buildbots are doing Release builds, so no debug info.

Even without full debug info, most ELF platforms keep the symbol table in the final binary, and it can be used to produce a somewhat useful symbolized stack trace. If you follow the advice in the logs and put llvm-symbolizer on PATH somewhere, these stack traces should become more useful:

[ RUN      ] LexerTest.BrokenStringLiteral
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):

That would be really helpful.

If this was a recent regression, I suspect it is because PATH changed, which happens often.

The LLVM stack dumper also looks for llvm-symbolizer in the parent directory of argv0, as shown here:

This means stack traces work out of the box for all binaries in the LLVM bin directory, so it works for everything… except unit test binaries.

I think the solution is probably to add LLVM build/bin/ to PATH when running unit tests in lit, and that should fix things.