LLVM compile-time regression tracking?

Project Infrastructure LLVM Dev List Archives
#1

Hi,

Does the LLVM project perform any kind of tracking for commit-by-commit compile-time changes? It looks like LNT only tracks run-time performance (and to be honest I wasn’t able to make heads or tails of the results even for that – the interface was pretty unintuitive to me.)

While it is “normal” that each new LLVM release regresses compile-time by 2-3%, LLVM 10 seems to be going for a new record here, with optimized builds regressing by around 5-15%, and that’s already after mitigating the largest regression: https://perf.rust-lang.org/compare.html?start=58b834344fc7b9185e7a50db1ff24e5eb07dae5e&end=f9c17b2d9bd4ca61bb0eb8b226dd61aaf254156c

It does not seem like these regressions are specific to Rust either, see for example https://bugs.llvm.org/show_bug.cgi?id=44408.

The Rust project tracks compile-time results for each commit (see link above), and thanks to that can quickly spot and revert changes that negatively affect performance. I think it would be really great if something similar would exist for LLVM…

Regards,
Nikita

#2

Hi,

#3

Hi,

Hi,

Hi,

Does the LLVM project perform any kind of tracking for commit-by-commit compile-time changes? It looks like LNT only tracks run-time performance (and to be honest I wasn’t able to make heads or tails of the results even for that – the interface was pretty unintuitive to me.)

At some point Apple was feeding compile-time data to LNT, but it seems like the jobs aren’t active anymore: [http://green.lab.llvm.org/green/view/Compile%20Time/](http://green.lab.llvm.org/green/view/Compile Time/)

Yes, it looks like the matrix view is broken. One of the relevant jobs is http://104.154.54.203/db_default/v4/nts/machine/1353 which has a fairly recent submission http://104.154.54.203/db_default/v4/nts/131539

But it looks like we need to adjust the ‘Run order’. It is currently using git commit hashes, which messes up the ordering of submissions. And submissions by the bot seem to be very irregular, due to various failures.

While it is “normal” that each new LLVM release regresses compile-time by 2-3%, LLVM 10 seems to be going for a new record here, with optimized builds regressing by around 5-15%, and that’s already after mitigating the largest regression: https://perf.rust-lang.org/compare.html?start=58b834344fc7b9185e7a50db1ff24e5eb07dae5e&end=f9c17b2d9bd4ca61bb0eb8b226dd61aaf254156c

It does not seem like these regressions are specific to Rust either, see for example https://bugs.llvm.org/show_bug.cgi?id=44408.

The Rust project tracks compile-time results for each commit (see link above), and thanks to that can quickly spot and revert changes that negatively affect performance. I think it would be really great if something similar would exist for LLVM…

The Rust tracking looks awesome! It’s great that we get feedback from there, thanks for keeping an eye on that.

In theory, the bots on green dragon should submit CTMark numbers regularly for CTMark on a few configurations (although not for every commit), but as mentioned earlier there are a few issues that crept in. I’ll try to reach out to the right people to have a look.

Cheers,
Florian

#4

FTR I ended up writing a service for this myself. Pretty graphs available at: http://llvm-compile-time-tracker.com/graphs.php.

We’re on a downwards trend right now, let’s keep it that way :slight_smile:

Regards,

Nikita