Hi,
Has anyone gathered statistics on reference workloads (*) for (transformation) passes
that are enabled / run by default (e.g., -O3) but most of the
time don’t do any effective transformation?
Even better if we also have such statistics for passes that are not enabled by default
(e.g., loop fusion, distribution, interchange, NewGVN).
And yet even better if people have some idea / data for the reason for ineffectiveness.
Bad heuristics / decision-making? Are some of these transformations useless most of the time?
Or maybe they are useful but their implementation in LLVM is not powerful enough. Or maybe
they incur a significant compile-time overhead.
If not, it would also be helpful if anyone who has tried gathering similar statistics has any
advice on how to approach it (my rough idea is initially
use --print-changed / --print-after-all on these workloads for -O3 and then try to
slide in passes that are not enabled by default; although that’s harder to do it right).
@Hideto: In the last LLVM meeting you gave a related talk [1]. Do you maybe have
the full statistics and / or ways to reproduce them?
Best,
Stefanos
(*) SPEC, Polybench, Cryptographic libraries, Genome alignment, Image Processing, Graph Processing, Web browsers, databases like sqlite etc.
[1] https://www.youtube.com/watch?v=QvF68tOt_w8
I think optimization remarks is a good framework for measuring whether a Pass does any work.
But unfortunately it is not widely adopted by Passes outside loop transformations. And even for the Passes that use it, my impression was that not every predicates are annotated with optimization remarks.
-Min
Hi Min,
Honestly, I don’t think optimization remarks are good to check whether a pass did any work. A far better option, to the best of my knowledge, is, as I mentioned, the --print-after-all family [1]
Optimization remarks may help on the last question I posed and that is to check why a transformation was not applied. Although, if one reaches this point, they will need to use way more
tools than remarks.
And even for the Passes that use it, my impression was that not every predicates are annotated with optimization remarks.
Definitely not and a lot of remarks are not that descriptive either.
Best,
Stefanos
[1] https://godbolt.org/z/58Ms7qW4s
Στις Δευ, 24 Μαΐ 2021 στις 11:07 μ.μ., ο/η Min-Yih Hsu <minyihh@uci.edu> έγραψε:
For the new pass manager, opt --print-changed(=quiet) is very nice for filtering out passes that didn’t do anything. You could add something to StandardInstrumentations to do something besides just printing the IR out when it changes.
Hideto (cc'ed) did look into this last year and he has some data.
You can find a quick summary in his lighting talk last year [1] and
also read a little bit about it in our paper draft! (attached),
especially Section 2.2 and Figure 3 are interesting here.
Not to say that we should not build some functionality upstream
to do this regularly 
~ Johannes
[1] Lightning Talks Session 2 - YouTube
Towards_Compile_Time_Reducing_Compiler_Optimization_Selection_via_Machine_Learning.pdf (1.86 MB)
Hi Arthur,
opt --print-changed(=quiet)
I don’t know how that argument “just works” now, I remember fighting it like… two months ago 
For instance, opt 12.0.0: https://godbolt.org/z/bd49oMqn5
You could add something to StandardInstrumentations to do something besides just printing the IR out when it changes.
Cool! I didn’t know about that, thanks!
Best,
Stefanos
Στις Δευ, 24 Μαΐ 2021 στις 11:55 μ.μ., ο/η Arthur Eubanks <aeubanks@google.com> έγραψε:
Hi Johannes,
Yes, I CC’d Hideto on the opening message, I should’ve CC’d all the contributors. If the full data is available, that would be a great start!
And thanks for the paper! I didn’t know about it.
Not to say that we should not build some functionality upstream
to do this regularly
I have a (bad) feeling that I’ll need to do a lot of drudgework to get this data, so, I might as well help to set something up if other people are interested.
The closest thing I know is a compile-time-measurement website (?) Nikita Popov (CC’d) had set up.
Best,
Stefanos
Στις Τρί, 25 Μαΐ 2021 στις 12:01 π.μ., ο/η Johannes Doerfert <johannesdoerfert@gmail.com> έγραψε:
Hi all,
Yes, I CC’d Hideto on the opening message, I should’ve CC’d all the contributors. If the full data is available, that would be a great start!
And thanks for the paper! I didn’t know about it.
What I did in the last year is to dump results to temporary files at PasManager.h[0], and then look over all results.
I don’t think it is a good way to accumulate the results so it is necessary to consider another way if we want the functionality in the trunk.
[0] https://github.com/llvm/llvm-project/blob/d881319cc5606baa7668405a296d0960a83a1e4c/llvm/include/llvm/IR/PassManager.h#L509
Hi Hideto,
Thanks, it makes sense. It’s similar to --print-after-all / --print-changed.
Best,
Stefanos
Στις Τρί, 25 Μαΐ 2021 στις 4:45 π.μ., ο/η Hideto Ueno <uenoku.tokotoko@gmail.com> έγραψε: