LLVM make check-lit results *not reproducible* in certain cases WAS: [llvm-commits]: Initial cut of ARM MC ELF emitter (PATCH)

Project Infrastructure LLVM Dev List Archives
1

Hi everyone,

I am sending this email after some time consuming exploration into how
LLVM executes its tests. I first noticed this issue when Rafael
noticed that my patch I sent in to llvm-commits broke some tests.

I was initially very skeptical, as I had run the make check test from
the build directory before sending in the patch, but as it turns out,
several issues came together in a perfect storm to give a royal
punkin' to yours truly.

After several time-consuming make clean/make check cycles among my
various llvm build directories, I began to doubt my sanity when across
certain recompile make clean/make check operation pairs on the SAME
SOURCE directory, different tests began to fail from the list by
Rafael, or sometimes, NO tests failed (and in one case, 100% of tests
failed!)

The answer turned out to be both simpler and more complex than I had
imagined. Whenever you have complex integration tests that call out to
various utilities, it is wise to make sure that at the very least, the
locations of said utilities are anchored to specific directories.

At least at first it seemed to be the case, according to the
autogenerated test/lit.site.config file in the individual llvm build
directories, recreated here for your reading pleasure:

2

So, if I understand you correctly the summary is that doing

* Build llvm and install it in /foo/bar
* Build llvm-gcc and install it in /foo/bar
* export PATH=/foo/bar/bin:$PATH
* Build llvm again in a unrelated build directory

and then running "make check-lit" in the second directory causes
/foo/bar/bin/llc and similar tools to be used instead of the recently
build tools? If that is the case, that really looks like a bug. Could
you please report it?

Thanks for reading!

-jason

Thanks,

3

So, if I understand you correctly the summary is that doing

* Build llvm and install it in /foo/bar
* Build llvm-gcc and install it in /foo/bar
* export PATH=/foo/bar/bin:$PATH
* Build llvm again in a unrelated build directory

and then running "make check-lit" in the second directory causes
/foo/bar/bin/llc and similar tools to be used instead of the recently
build tools? If that is the case, that really looks like a bug. Could
you please report it?

Just did. PR8199

Thanks!

-jason