Igalia has been tracking the performance of Clang and GCC on RISC-V hardware over the past year on our own LNT instance, but recently thanks to the work of @ldionne and the infrastructure team the official http://lnt.llvm.org instance is now back up and running and the results are available to view there.
For those not familiar, LNT can be used to track regressions as well as compare the difference in performance across various configurations. Given that the official instance has been down for some time, I’ve included some info on how to use it below.
Quick start guide
To get started, visit http://lnt.llvm.org and select the “nts” test-suite. This is the llvm-test-suite test-suite and also contains SPEC CPU 2017. You’ll be greeted with a list of “machines” and “runs”:
A machine is a certain configuration, e.g. Clang running on a Banana Pi F3 with the flags -O3 -march=rva22u64_v. A run is a specific commit that is being tested on a machine. Currently each machine produces a new run every day.
If you click on “view results” for a run, you’ll see the changes between that run and the run immediately before it for that machine:
You want to look for the “run-over-run changes” section. The “run-over-baseline” is used for comparing against e.g. a specific release of Clang but that hasn’t been set up yet.
LNT tracks both changes to the execution time as well as binary size. The changes highlighted are those that are deemed statistically significant by LNT to remove some of the noise.
To see what commit of Clang/GCC a run used, expand the “run info” panel and look for the “cc_src_revision” field:
Daily reports
If you want to see what regressed/improved across all machines, you can use the daily report page to see all the changes detected in one day:
The sparkline to the right shows the performance over a few days, and the different colours indicate when the binary hash changed. You can filter for specific machines in the top which is useful if you e.g. want to skip the GCC changes.
Comparing across machines (configurations)
You can also compare performance results across different machines. For example, to compare the latest build of Clang against the latest build of GCC click on the page for the bfipf3-rva22u64_v-ReleaseLTO__clang machine, and then select the bpif3-rva22u64_v-ReleaseLTO__gcc machine under the “compare” section:
This will bring you to the interface here: http://lnt.llvm.org/db_default/v4/nts/19?compare_to=18
The interface is the same as comparing against runs on the same machine. A regression means that Clang is slower than GCC, and an improvement means vice-versa.
Viewing profiles
LNT also records perf profiles which can be viewed directly from the web interface. For example, comparing Clang to GCC we can see that this MultiSource/VersaBench/bmm/bmm is 3.5x slower on Clang versus GCC:
To get a better idea of why GCC outperforms Clang here you can click on the “profile” button. This will lead you to an interface like this:
Clang is on the left and GCC is on the right. The top contains a comparison of various perf counters. Below you can select various functions to compare against. This is sometimes useful because one compiler may inline another function, like in this case GCC has inlined matmult into main.
In the bottom left there are options to control the view which let you view either the assembly as outputted from objdump or a CFG. On the bottom right you can switch between viewing various counters like instructions or cycles, and if you want to view them as absolute, relative or cumulative. These counters are displayed inline on the right hand side of the assembly.
In this particular example, we can use the profiling information to see that GCC is able to avoid a strided vector load by performing loop interchange.
Next steps
Hopefully we can get the resources to track other popular targets like X86 and AArch64, either from the LLVM foundation or other interested parties.
The LNT instance also needs to be upgraded to host over https.
And finally the LNT interface is currently being rewritten which will make this UI easier to use, and hopefully we will see an RFC for it soon.