How to debug a test that fails only on some build bots?

I tried to land a small fix before going on vacation at the end of the year, but I had to revert it because it broke on a few of the build bots, specifically clang-cmake-thumbv7-a15, clang-cmake-armv7-a15, and llvm-hexagon-elf.

It seemed to work as expected on the others.

Unfortunately, the build logs don’t give many clues. The fix included a new test, and that test failed with an exit code of 1 but with no messages. For example:

Here are a couple of strategies that helped me in the past:

  • Try to copy the cmake invocation from the bot’s logs and reproduce it locally.

→ Particularly the bots you list I think all don’t build an X86 backend, so maybe that’s it.

  • Ask for help from the maintainer of the bot
  • [last resort] Commit a change that duplicates the failing command without piping it through FileCheck, so you can see the output in the bot’s log.

– adrian

I think you meant to put the test into test/DebugInfo/X86/ ? Your test specifies an x86 target, but those buildbots don’t build the x86 target. llc is probably producing some error message like “error: unable to get target for ‘i686’, see --version and --triple.”, but you’re redirecting stderr, so it isn’t visible. You should try to avoid this when possible. Your test doesn’t have any positive CHECK lines, so FileCheck passes. You should generally try to include some positive CHECK line in regression tests. And you never need to check for the string “Assertion failed”; the test will fail if any command on the RUN line fails or crashes. -Eli

Thanks! It looks like this is sorted out now.

Adding to the discussion:

A classic gotcha is the fact that llc/opt target LLVM_DEFAULT_TARGET_TRIPLE by default (which is different for some bots).

Even using “llc -march” / “opt -march” is not always enough as it only changes the architecture but not the operating system/environment (actually a lot of tests get this wrong, but it usually doesn’t make them fail).

The correct way do things is to always provide
when invoking llc or opt with any transformations that use TargetTransformInfo.

  • Matthias