Non-determinism in libFuzzer tests on AArch64

Hi,

I've seen this issue pop up on the mailing list a couple of times
before with no resolution [1][2][3].

It has been getting really bad for our AArch64 buildbots lately. You
can see a few examples at the end of my email [4], but we have the
same failures on a couple of other bots as well.

Does anyone have any idea how to reproduce / investigate these issues?
If not, is it reasonable to mark the tests as unsupported on AArch64?

Thanks,
Diana

[1] https://lists.llvm.org/pipermail/llvm-dev/2018-June/124334.html
[2] http://lists.llvm.org/pipermail/llvm-dev/2019-January/128861.html
[3] https://lists.llvm.org/pipermail/llvm-dev/2018-November/127701.html
[4] clang-cmake-aarch64-lld failures:
http://lab.llvm.org:8011/builders/clang-cmake-aarch64-lld/builds/6264
http://lab.llvm.org:8011/builders/clang-cmake-aarch64-lld/builds/6262
http://lab.llvm.org:8011/builders/clang-cmake-aarch64-lld/builds/6260
http://lab.llvm.org:8011/builders/clang-cmake-aarch64-lld/builds/6255
http://lab.llvm.org:8011/builders/clang-cmake-aarch64-lld/builds/6253
http://lab.llvm.org:8011/builders/clang-cmake-aarch64-lld/builds/6250
http://lab.llvm.org:8011/builders/clang-cmake-aarch64-lld/builds/6247
http://lab.llvm.org:8011/builders/clang-cmake-aarch64-lld/builds/6245
http://lab.llvm.org:8011/builders/clang-cmake-aarch64-lld/builds/6237
http://lab.llvm.org:8011/builders/clang-cmake-aarch64-lld/builds/6235
http://lab.llvm.org:8011/builders/clang-cmake-aarch64-lld/builds/6229

I have worked on fixing/uncovering non-determinism in llvm and clang [1] [2] [3] [4]. The main causes there were iteration of unordered containers, hashing on pointers and using std::sort on equal elements. Although I am not sure if the libfuzzer issues are the same/similar.

[1] http://lists.llvm.org/pipermail/llvm-dev/2016-November/107098.html

[2] http://lists.llvm.org/pipermail/llvm-dev/2017-July/115025.html

[3] http://lists.llvm.org/pipermail/llvm-dev/2017-August/116975.html

[4] http://lists.llvm.org/pipermail/llvm-dev/2017-October/118639.html

--Mandeep

I frankly don’t even know who supports libFuzzer on aarch64.
Kuba?

+Mitch Phillips who may get involved with libFuzzer/aarch64 in near future.

[CC the right George’s email]

I frankly don’t even know who supports libFuzzer on aarch64.
Kuba?

+Mitch Phillips who may get involved with libFuzzer/aarch64 in near future.

Hi,

I’ve seen this issue pop up on the mailing list a couple of times
before with no resolution [1][2][3].

It has been getting really bad for our AArch64 buildbots lately. You
can see a few examples at the end of my email [4], but we have the
same failures on a couple of other bots as well.

Does anyone have any idea how to reproduce / investigate these issues?
If not, is it reasonable to mark the tests as unsupported on AArch64?

I think that’s fine, although AFAIK, these tests are very stable on x86, right?

Kuba

I frankly don’t even know who supports libFuzzer on aarch64.
Kuba?

+Mitch Phillips who may get involved with libFuzzer/aarch64 in near future.

Hi,

I’ve seen this issue pop up on the mailing list a couple of times
before with no resolution [1][2][3].

It has been getting really bad for our AArch64 buildbots lately. You
can see a few examples at the end of my email [4], but we have the
same failures on a couple of other bots as well.

Does anyone have any idea how to reproduce / investigate these issues?
If not, is it reasonable to mark the tests as unsupported on AArch64?

I think that’s fine, although AFAIK, these tests are very stable on x86, right?

Correct.
Last time I’ve checked, some of the libFuzzer tests
were very slow, and thus flaky, due to lsan/aarch64 being very slow.
https://github.com/google/sanitizers/issues/703

–kcc