llvm-stress for fuzzing llvm

Project Infrastructure LLVM Dev List Archives
1

Hi,

Compiling lots of 'junk' helps in catching bugs. I added a new tool (located under llvm/tools/llvm-stress) for generating random LL files. The tool can be used to test different llvm components using various compilation flags. Until now, I only found bugs in the codegen, and not in general llvm optimizations. This probably means that the generated tests are currently too simple for the higher-level optimizations.

The command line below generates a random ll file, and llc compiles this file. It often crashes.

./llvm-stress -seed $RANDOM -o tmp.ll -size 1000 ; ./llc tmp.ll -mcpu=corei7-avx -mattr=+avx -o /dev/null

The "-seed" flag sets the initial seed to be used by the random function. I implemented a simple portable 'random' function so that the result should be identical on all platforms. The initial seed also appears in the name of the generated function. The "-size" parameter sets the size of the generated random file.

Nadav

2

Hi Nadav,

Great tool!

Attached is a relatively small patch to add random vector types.

Thanks,
Joey

vec.diff (1.44 KB)

3

Nadav,

Thanks, this is neat! Here is a patch which optionally enables
generation of the other floating-point types. Please review.

-Hal

llvm_stress_fp_types.patch (2.91 KB)

4

Wow, nifty tool! I’ve already found a couple crashes!

It is also really easy to pinpoint what is causing the error. Whenever you trigger a bug, run llvm-stress with the same seed but a really small size that doesn’t trigger the bug (e.g. like 10). Then do binary search on the size. Eventually you find exactly the cutoff of size that triggers the bug (e.g. 539 runs fine, but 540 crashes), and then you can diff the crashing and non-crashing .ll files and there should only be a tiny difference.

–Sean Silva

5

I’m finding it useful to replace the main loop with:

for (unsigned i = 0, n = SizeCL/Modifiers.size(); i < n; ++i) {
Modifiers[i%Modifiers.size()]->Act();
}

That way, changing the size by 1 adds exactly one instruction, which makes delta debugging MUCH easier. Maybe it would be worth changing?

–Sean Silva

6

Sean,

Thanks for looking at this. Knowing that the last instruction triggered the bug is often not enough. I use bugpoint to reduce the failing test. The reason is that some of the bugs may be caused by the interaction between several instruction. Having said that, I think that the change that you proposed is a good one. Can you send a patch ?

Thanks,
Nadav

7

LGTM, please commit.

8

Here is that patch.

Btw, I’ve just been using bugpoint, and it’s really nifty!

–Sean Silva

2012/2/27 Rotem, Nadav <nadav.rotem@intel.com>

forloop.diff (764 Bytes)

9

Attached is a simple bash script I wrote to run llvm-stress several times, might be useful!

Thanks
Joey

run_stress.sh (306 Bytes)