Automatic Regression Finding Tool

Dear LLVM Developers,

in the last few months Theodoros (theodort@inf.ethz.ch) and I have created a tool which automatically finds regressions related to dead code elimination.
We have collected more than 200 regressions with respect to trunk and would like to report them.

However, as we do not wish to spam the GitHub issues, we would like to ask how to best hand over these findings.

An example of how a current issue/report looks like can be found at https://github.com/llvm/llvm-project/issues/52965 .
Any feedback on the report template is appreciated.

Yours sincerely,
Yann Girsberger

Hi Yann, this is a great initiative.

On the template I had a few ideas.

Could you put a dividing line between each pair of command/output? Or
perhaps just one after the "can not eliminate foo but..." line. Then
it's easier to see that each command and output is paired starting
from that point.

You have the output of -v, what is the use case for that? The most
obvious thing is the target. I primarily work on Arm so it is good to
see what the default target should be but I wonder if there's a better
way to do that. Not saying you have to remove the -v output, I just
wonder if an explicit --target option to clang would give the reader
the same info and help those of us not running on x86. (I'm not
suggesting you go back and redo 200 reports though, maybe for new
ones)

Having both assembly outputs is great. We (Linaro) just added this to
one of our regression loops and a lot of cases can be triaged with
just that.

However the phrasing is a bit imprecise on first read. "Started with"
I think, did the bisect start with this rev or did the problem start
there? After a few seconds you realise of course the latter is the
only thing that makes sense but it could be clearer. Maybe "Regression
started with".
(you can probably spend ages naming that bit. Some people prefer last
good/first bad, sometimes I want to put it the other way around as
before/after)

Would it be possible to attach the test case and the before/after as
an archive to the issue? (I found out recently github doesn't allow .c
files but an archive containing them is fine) Then it's quick to
download and diff locally if it can't be eyeballed.

Last thing, that I'm not sure is possible automatically. Tagging the
user who wrote the patch. Easy enough to do after the fact but perhaps
github has some way to map commit to github username.

Hi David,

thank you very much for your feedback!

I integrated it and other feedback into new reports.
(https://github.com/llvm/llvm-project/issues/53322)

Is there any interest from Linaro (or the ARM community in general) in similar regressions for ARM targets?

If you have any further feedback on the reports, please let us know.

Best,
Yann

I integrated it and other feedback into new reports.

Looks great.

Is there any interest from Linaro (or the ARM community in general) in similar regressions for ARM targets?

We (Linaro) have been doing benchmarking related to code size and
speed using SPEC but nothing like this where you've got different
generated (I assume) examples each time. I've added Maxim on CC who
set up our benchmarking infrastructure and better knows what we test.

Certainly sounds like a good addition and it would be interesting to
find out whether it's detailed enough to catch target specific issues.
Is your approach detailed anywhere? (I think I saw mention of a paper)