https://llvm-compile-time-tracker.com/ is currently fairly C-heavy. I’ve been recently doing some clang optimizations for C++23, and I think those aren’t tracked at all. I’ve been using a somewhat representative subset of the Chromium build to inform these.
We weren’t quite sure where to ask; “Clang Frontend” on discourse was our best guess
Pasting from there:
Is this interesting at all?
As-is, this doubles the (wall clock) size of llvm-compile-tracker. This can be tweaked by including fewer TUs, but then it’s less representative of the build
As-is, this hardcodes the release flags always. If desired, I could add a 2nd .cmake file with our debug flags (and slightly different generated headers) and load those in debug builds
…but since it only compiles and not really links, LTO will likely never work
Is just adding all files to this repro the way to go, or should they go into a submodule’d-in-repro? (It’s 108 MB, but git compresses them to 17 MB in the index and in transit. So it’s neither very small nor very large. If we ever want to refresh them, maybe it’s nicer if the history of that isn’t in this repro? Not sure how y’all are thinking about this.)
If there’s interest, I’m happy to spend some time on this to make it nice. But also no hard feelings if you’d rather not add this.
I think it’d be cool if llvm-compile-time-tracker also tracked some C++ code. I think Chromium is a decent candidate (reasonably modern C++ version, fairly hermetic build in that it also uses hermetic sysroot and libc++), but I’m obviously biased.
+1 to a more modern C++ code base, but -0.5 to Chromium sources. I’d prefer if we could make a more targeted selection that is less costly and find some other, smaller program(s) that exercise(s) similar characteristics (perhaps SPEC CPU 2026 could serve as inspiration?).
Right now we have Bullet (C++98), tramp3d-v4 (single file, C++14), kimwitu++ (C++14), 7zip (default = C++17). stage1 also has LLVM/Clang (C++17). One or two more recent code bases would be a valuable addition. These should be particularly targeted at Clang, I think the existing benchmarks cover LLVM reasonably well already.
I think test should be compilable in all configurations (debuginfo/nodebuginfo, asserts/NDEBUG, LTO/ThinLTO/no LTO, O0/O1/…) and linkable (otherwise it is not really possible to measure size-text, etc.). IMHO all tests should also be executable: wrong code can be generated arbitrarily fast.
Currently, one c-t-t test takes ~9.5 minutes (assuming stage1 ccache hits), of which 5.5 minutes are the stage2 clang build. Doubling the lit time substantially reduces the amount of tests done. I personally also value short roundtrip times (which may also include a queue of other commits before the interesting one). (Also, most c-t-t experiments are related to LLVM and veeeery few people work on Clang perf, so doubling latency and reducing the amount of tests per day from ~150 to ~110 might be a bad trade-off.)
This makes me think that Chromium sources are not an ideal fit here. Moreover, 60 heavy TUs also seems somewhat large and I’m not convinced that each of these exercises substantially different important not-yet-covered code paths that make it worth having them all.
Thanks for getting the discussion started. I think CTMark certainly could do with some modernized workloads.
Just to note, I think the selection could be adjusted to fewer TUs, to adjust the increase of compile-time.
One potential issue with the current selection is that it pulls in a very large number of header files.
Ideally yes, if someone is willing to spend the time to come find such workloads that would be great. If not, using the chromium extracted sources would already be a good improvement, if it can be made to work properly with all build modes.
Looks like C++11/14, so unlikely to add much coverage over LLVM/Clang.
I did look around a bit, but C++20 seems to be used very sparely – which makes me think that C++20 workloads (let alone intensive concepts use) are not yet representative for a significant user fraction. I agree that CTMark could receive some newer benchmark (e.g. SPEC-inspired benchmarks), but on c-t-t we do have a somewhat representative modern C++ code base with LLVM/Clang.
From the referenced motivational PRs, two do show significant improvements on c-t-t, one is for an off-by-default warning (wouldn’t be tested anyway?), and one is specifically for concepts (C++20).
To check the “much smaller” part, I tried two header-only C++20 libraries with C++23/C++26 features behind feature-test macros:
mp-units (MIT): concepts, class-type NTTPs, consteval. 24 examples, all compile at -std=c++20/23 and run with deterministic output.
beman.execution (Apache-2.0 WITH LLVM-exception), a reference implementation of C++26 std::execution (P2300): concepts-heavy sender/receiver code, coroutines. Self-contained assert-based tests, no Catch2/gtest.
Unlike tramp3d-v4 (~11% frontend at -O2), these spend roughly 40–90% of their -O2 time in Clang’s frontend, which is the part thakis wants to track.
I measured instructions:u at -O2 with clang HEAD, and with each of the recent fixes reverted (or applied, for the open #227398). Noise is ≤0.04%.
change
beman.execution
mp-units
existing CTMark C++
#226620 constraint cache by expression
+6.4%
+2.3%
0.0%
#227398 (open) satisfied concept-ids
−1.1%
−0.8%
0.0%
#226610getDeclAttrs cache
+1.1%
+0.5%
+0.3%
#228956DiagStateMap cache
+0.6%
+0.2%
+0.3%
(existing CTMark C++ = tramp3d-v4 + kimwitu++ + Bullet)
So the concepts work is invisible to CTMark today, and these two libraries catch it clearly. #226605 is for an off-by-default warning and wouldn’t show up in any default-flags benchmark. I haven’t measured #161671 yet.
The beman tests are highly redundant (presumably dominated by the shared headers): even the smallest ones show ~11% for #226620. So a small selection is enough, e.g. mp-units currency, kalman_filter-example_6, foot_pound_second and beman’s exec-when-all. That’s about 95G instructions at -O2, less than 2× tramp3d-v4, and since it links and runs it should work in every c-t-t configuration (I have only tried -O2 so far).
Caveats: mp-units needs a host standard library with <format> (libstdc++ 13+ or a recent libc++) and orders unit symbols by __PRETTY_FUNCTION__, so one example’s reference output differs between Clang and GCC. Both look manageable.
I think that Chromium may be a bit too much.
As part of my fellowship, I found a number of regressions that didn’t appear on CTT, but did appear on other projects. In particular, I would recommend some of the TUs from mp-units and range-v3, and some of the Boost.MPL tests.
change
mp-units
range-v3
Boost.MPL
existing CTMark C++
#161671 constraint normalization (-std=c++20; flat at c++17)
As I said, happy to not do anything. @MaskRay, @AnonMiraj, are you planning on adding the ones you identified?
I did look around a bit, but C++20 seems to be used very sparely – which makes me think that C++20 workloads (let alone intensive concepts use) are not yet representative for a significant user fraction.
I’m a bit confused by this comment. C++23 is (mostly) a superset of the earlier versions, so it’d track perf of both pre-C++20 features and also of C++23 features, right? What point is this paragraph trying to make? That C++20+ build time doesn’t matter?
C++17 code is already extensively tested through the stage2 LLVM build, so new benchmarks should be heavy on C++20/23-only features. Given that projects that heavily use such features seem rare right now, I deduced that these features might not yet be relevant for most Clang users. Given that we have a somewhat limited time budget, features that see very little use shouldn’t dominate that budget.
Thanks for writing this up and helping with the performance in concepts.
As someone who slowed down concepts in clang 22 a lot, I think I would have give a +1 on adding benchmarks to template-heavy C++ code!
@cor3ntin and I are outside of chromium developing and to be honest we couldn’t gather much meaningful real-world performance data during development (I appreciate that the Chromium folks noticed the issue and helped improve it rather than simply reverting the changes), so a benchmark running on the production use cases would definitely be helpful to us developers who work extensively on C++ templates. (cc @mizvekov)
As for other alternatives mentioned above, e.g. range-v3 – I should mention that we actually tested that locally before we roll out #161671, but only for checking correctness. Still they’re regression tests and I think they’re less representative of real-world production workloads than the ones from Chromium.
Side note about the statistics from @AnonMiraj: we brought back much of the performance loss since #161671 in clang 23 - though there is still room - I believe we’re currently not 89% slow on range-v3.
I give a +1 to adding Chromium: it’s a significant LLVM use case that does not seem to be well-represented in CTMark currently; from our (Sony’s) perspective, the tradeoff of increasing CTMark build time against having it detect more regressions is straightforwardly positive. Doubling the wall time is unfortunate, and it would be preferable if a smaller set of TUs could be selected that would still be representative, but in the absence of (or in addition to!) other projects that have a similar performance profile I think this would be an improvement - if the increased time does turn out to be an issue, or we find suitable substitutes, we could always remove it later. Regarding the other proposed CTMark projects, it’d be good to see if any of them have performance that correlates at least moderately with Chromium, since that would significantly weaken the case for including Chromium.
