Need help reproducing a bug

Hi,

Recently I committed a change (r330175) that passed all my testing, but failed on several bots. Namely, these are the failed ones:
http://lab.llvm.org:8011/builders/clang-with-thin-lto-ubuntu/builds/9803
http://lab.llvm.org:8011/builders/clang-with-lto-ubuntu/builds/8173
http://lab.llvm.org:8011/builders/lld-x86_64-freebsd/builds/18082

I reverted the change (r330180), but now I’m stuck with how to proceed with it, as I can’t reproduce any of these.

So far I’ve tried building clang with asan and using this sanitized clang to build clang and lld one more time and run make check - none of these failed on my machine. What else could I try to catch the issue?

In case you are interested in details and/or want to try to reproduce it, you’ll need to revert r330180 (and thus reapply r330175). The change is about using a new faster SSAUpdater in Jump Threading, more details are available in the phabricator: https://reviews.llvm.org/D44282.

Thanks,
Michael

Hi,

Recently I committed a change (r330175) that passed all my testing, but
failed on several bots. Namely, these are the failed ones:
http://lab.llvm.org:8011/builders/clang-with-thin-lto-ubuntu/builds/9803
http://lab.llvm.org:8011/builders/clang-with-lto-ubuntu/builds/8173
http://lab.llvm.org:8011/builders/lld-x86_64-freebsd/builds/18082

Note what *specifically* failed:
* compare-compilers compare stage3 and stage4 compilers failed ( 0 secs )
* compare-tablegen-inc-files compare stage3 and stage4 Tablegen inc
files failed ( 1 secs )

I.e. it wasn't tests that failed.

Hi,

Recently I committed a change (r330175) that passed all my testing, but
failed on several bots. Namely, these are the failed ones:
http://lab.llvm.org:8011/builders/clang-with-thin-lto-ubuntu/builds/9803
http://lab.llvm.org:8011/builders/clang-with-lto-ubuntu/builds/8173
http://lab.llvm.org:8011/builders/lld-x86_64-freebsd/builds/18082

Note what *specifically* failed:
* compare-compilers compare stage3 and stage4 compilers failed ( 0 secs )
* compare-tablegen-inc-files compare stage3 and stage4 Tablegen inc
files failed ( 1 secs )

I.e. it wasn't tests that failed.

Failing that tests means the compiler doesn't produce deterministic output because the stage3 and stage 4 compiler has to be the same.

Not sure about LLD test.

Steven

Thanks everyone! What are the best tools/techniques to expose such non-deterministic behavior? My hope is to reproduce it on a smaller test (e.g. use some sanitizer and thus make the compiler fail when building the test) - Currently these failures only tell me “there is some bug in your code” without any hints where to look for it.

Michael

Thanks everyone! What are the best tools/techniques to expose such
non-deterministic behavior? My hope is to reproduce it on a smaller test
(e.g. use some sanitizer and thus make the compiler *fail* when building the
test) - Currently these failures only tell me “there is some bug in your
code” without any hints where to look for it.

Hmm, have you tried -DLLVM_ENABLE_ASSERTIONS=ON
-DLLVM_ENABLE_EXPENSIVE_CHECKS=ON -DLLVM_REVERSE_ITERATION=ON
(especially the last one)?

Hi Michael,

Last year I had a problem with reproducibility that I detected in the generated assembly for the out-of-tree target I was working on that seemed like non-deterministic code generation. There was nothing incorrect in the alternative code being emitted, but it was making me nervous that it was different at all.

After some investigation it turned out to be a consequence of iterator ordering. The scheduler was working out the prioritisation of basic blocks and there was no issue if one BB had a different priority to another. However, if two BBs had the same priority, the implicit ordering was based on the hash in the map that was being iterated, and this could vary between platform (Windows vs Linux) and between recompiles on the same platform.

Just a thought, but perhaps you might have a look at how ordering is handled in your BB or MI iterators.

MartinO

Yep, code generation choices based on the iteration order of a hash (or unstable sorted) container is the most common source of non-determinism. Using the reverse iteration mode that Roman mentioned is about the only tool we have & doesn’t cover all cases.

Another possibility is order of evaluation of function arguments (eg: f(g(), h()) - if the evaluation order of ‘g’ and ‘h’ effects the code generation, then this differ between GCC and Clang).