Running the programs in the LLVM test suite with a pass and without it.

Hi guys,

I have a pass called “MyPass”. And I can run this pass in the LLVM test suite programs and produce a report with the statistics of the execution of this pass. Now, I would like to run the programs in LLVM test suite and generate a report comparing the results obtained without using the pass with the results obtained using the pass. Is there any way to do this?

Thank you,


Hi Douglas,

Yes, you can use the "llc-beta" test suite option to do what you're looking for. That runs the tests with and without a set of command line options and provides a comparison.

1. Make your pass disabled by default and enabled via a command line option (e.g., --enable-mypass),
2. Tell the test-suite Makefiles to run your pass as the beta option*,. Look in <llvm/projects/test-suite/Makefile.programs*,
3. Run the testsuite in night test mode (the "simple" mode does not support beta options at this time) via something like "make TEST=nightly report.html",
4. Open the report to see your results.

* Look in <llvm/projects/test-suite/Makefile.programs. The options are enabled by target, search for "BETAOPTION" to find them. You'll see examples for setting the LLC beta option via LLCBETAOPTION. If your pass is a machine function pass, that's where you'll want to add it. If it's run by 'opt' instead, you'll want to use OPTBETAOPTION.


If by "stats" you mean execution time, then you may use the Makefile support for "llc beta" as Jim suggested. I use the following without editing the Makefiles:

TEST=nightly ENABLE_LLCBETA=1 LLCBETAOPTION="-my-pass" report.csv

To compare across separate runs, I just load the report.csv into a spreadsheet and define some formulas.

If you actually want to compare output from the -stats option, then you can compare the * files (one for each benchmark) from two separate runs. I attached the script that I use to do this.

-Andy (2.36 KB)