bugpoint --tool-args and --safe-tool-args

I have a failing test and bugpoint would be the perfect tool to help
narrow it down.

llc is the failing tool. It fails with one set of options and passes
with another. I was hoping to use bugpoint like this:

bugpoint -safe-llc -run-llc <testcase> -tool-args <failing args> -safe-tool-args <passing args>

Unfortunately, this doesn't seem to be possible. According to the
documentation both -tool-args and -safe-tool-args consume all remaining
arguments so bugpoint would never see -safe-tool-args above.

I tried it and indeed it looks like -safe-tool-args never makes it to
llc.

For now I can build another llc which makes <failing args> default and
then just use -safe-tool-args above but it's a bit painful.

Has anyone tried to use bugpoint like this before? Any tips?

                          -David

I've occasionally used shell script wrappers when I couldn't get bugpoint command-line parsing to work the way I wanted it to.

Looking at CommandLine.rst, PositionalEatsArgs is supposed to allow specifying something like the command you wrote, but I'm not sure it ever worked correctly; nothing other than bugpoint uses it, as far as I can tell.

-Eli

This is what works for me, when I want to reduce an llc failure triggering a failure with a “foo bar baz” in the assert message (the grep for the error message avoids reducing to an unrelated failure):

$ cat run_llc.sh

#!/bin/sh
if ! $HOME/dev/llvm/build/bin/llc “$@” 2>&1 | grep “foo bar baz”; then
exit 0
else
exit $?
fi

$ PATH=$HOME/path/to/my/llvm/build/bin:$PATH bugpoint -compile-custom -compile-command ./run_llc.sh test.ll

  • Matthias

This is what works for me, when I want to reduce an llc failure triggering a failure with a “foo bar baz” in the assert message (the grep for the error message avoids reducing to an unrelated failure):

$ cat run_llc.sh

#!/bin/sh
if ! $HOME/dev/llvm/build/bin/llc “$@” 2>&1 | grep “foo bar baz”; then
exit 0
else
exit $?
fi

$ PATH=$HOME/path/to/my/llvm/build/bin:$PATH bugpoint -compile-custom -compile-command ./run_llc.sh test.ll

Also:

  • Make sure the script is executable chmod +x run_llc.sh.
  • Try first that your run script behaves as expected. If you see bugpoint running for a long time without making real progress it’s likely that the returncodes of your script aren’t as you expect.

Oh yeah, that makes sense. Thanks!

                          -David

Matthias Braun via llvm-dev <llvm-dev@lists.llvm.org> writes:

Ugh. Scripts won't work either, at least not by default. The problem
is that I need non-default options for both the safe and unsafe
compiler.

I hacked in an -unsafe-path option to let me point to a script for the
"bad" compiler. We'll see if that works. If it does, would it be a
useful addition?

                           -David

Matthias Braun <mbraun@apple.com> writes:

Have you read http://blog.llvm.org/2015/11/reduce-your-testcases-with-bugpoint-and.html?m=1 ?

That’s how I always use bugpoint and have done things like this. I always seem to have to brush up my bash skills every time I use it.

For me the bugpoint arguments never seemed to quite work so I always use the custom compile script and since I always forget how to use bugpoint I wrote a script for reducing test cases. It usies either bugpoint or creduce depending on whether it is a frontend or backend crash (although sometimes using creduce on .ll files removes more stuff than bugpoint does).
If that sounds useful the script can be found here: <https://github.com/CTSRD-CHERI/clang/blob/master/utils/creduce_crash_testcase.py>.

You can just point it at the test case and it will parse the RUN: line and try to reduce that. It works for run lines using llc,opt, or %clang/%clang_cc1 (and various CHERI specific lit substitutions).
However, it only works for crashes and infinite loops (default timeout is more than 20 seconds → assume infinite loop). I’ve not yet added support for good tool / bad tool but it should be easy to add by just slightly changing the generated shell script that is used for reducing. If more people are interested in this I could add the feature, clean up the script and create a review.

Alex

Thanks for the pointer!

Unfortunately, I am not sure this will help. Right now I've got a bad
code generator. I would like to use the code generator mode to pinpoint
where in the test it is going wrong. The blog post talks about testcase
reduction, which isn't quite what I need. I need the splitting behavior
of the code generator mode, which I'm not sure -compile-custom supports.
I'll give it a go and see what happens.

That post talks about the confusing nature of bugpoint and I have to
agree. I'm thinking about writing up an RFC to improve the user
experience but I've got to find time to do it. :slight_smile:

                               -David

Sean Silva <chisophugis@gmail.com> writes: