[RFC] LLVM LTS

LLVM Project
1

Hi,

At US LLVM 2024, we had a round table discussion about the possibility of an LLVM Long-Term Support release (LLVM LTS), and what it may look like. This RFC continues that discussion.

The roundtable wasn’t recorded, but Tom’s and David’s presentation is excellent for background on releases: https://www.youtube.com/watch?v=HwrQyOQBIpQ, https://llvm.org/devmtg/2024-10/slides/techtalk/Edelsohn-GCC-and-LLVMDevelopment-and-Maintenance.pdf .

First of all, why LTS?

Well, because having LLVM LTS is from “convenient” to “essential” for some toolchain applications. Two good examples are

  • building a Linux or BSD distro with LLVM/Clang as the main compiler, and
  • building an SDK or a custom compiler on top of LLVM.

Linux or BSD distribution lifecycle spans many years, with 4 years being the usual frequency at which an average user is comfortable upgrading systems. Within those 4 years, distros need a supported compiler for rebuilds. When picking GCC for the distro compiler, this means getting 2 years of support from the GCC community, and then another 2 years of supporting the compiler internally. For LLVM/Clang, that would mean half a year of community support, and then 3.5 years of internal support – significantly more than GCC.

A similar logic goes for SDKs that are distributed to application developers. Sure, some of them upgrade their development environment every month, but many use installed SDK until it breaks – e.g., doesn’t support a new core necessary for the project. Therefore, companies building SDKs on top of LLVM have to invest into internal support of old LLVM versions.

LLVM LTS is about pooling resources to create a good base for the long-term maintenance of LLVM.

Secondly, who will do the LTS work?

At the round table discussion, it quickly became clear that the LLVM community highly values speed of development. This means that the LTS process should not add to the normal development or contribution workflows. This means that no patch author should be required to do more work because their patch is going to be backported to the LTS version. Consequently, the LTS work will need to be staffed by companies / stakeholders interested in LLVM LTS.

Some of the activities I foresee for the LTS include:

  • Backporting patches
  • Testing individual patches
  • Setting up additional buildbot builders
  • Monitoring LTS buildbots and investigate regressions
  • Managing releases

Lastly, what would LLVM LTS look like?

Um, I don’t know. My first stab at an MVP is this:

  • Each LLVM LTS release will be based on a normal LLVM release
  • Patches will be backported if they …
    • … fix a bug for supported architectures
    • … implement a feature critical for “supported projects”
      • e.g., hardware errata mitigations
      • e.g., security issues mitigations
    • Note [1]
  • All other patches will NOT be backported
  • Supported target architectures
    • AArch64 Linux (self-hosting)
    • x86_64 Linux (self-hosting)
    • Note [2]
  • Supported projects
    • Linux kernel (latest LTS release)
    • Note [3]
  • Schedule
    • Every 2 years a particular LLVM release is designated as LTS, say LLVM 20
    • LLVM upstream community supports LLVM 20 for the first 6 months as per the normal release process.
    • Once 6 months are up, the LTS team continues to support LLVM 20 branch for another 24 months.
    • LTS overlap with previous LTS is 6 months

So… what do you think?

Notes:
[1] Exact policies for backports are to be decided and refined – please provide ideas and suggestions!

[2] Other architectures are, obviously, candidates for LTS as well, but each supported architecture should have at least one stakeholder willing to provide engineering resources to backport, test, and maintain LLVM LTS releases for that architecture. E.g., Linaro would support AArch64 architecture, and I am optimistic that we can find a couple of stakeholders for x86_64 as well.

[3] “Supported project” is a source base so important that we will backport a new compiler feature [to an otherwise very conservative branch] to fix an issue in it. Or, in other words, these are major projects that the LLVM LTS should always be able to compile for any LTS-supported architecture.

8 Likes
LLVM Qualification WG sync-ups meeting minutes
LLVM Weekly - #577, January 20th 2025
2

Thanks for your RFC! LLVM LTS is very helpful for the release engineers.

1 Like
3

Thank you for this RFC!

Do you envision the LTS release being just for LLVM, or for LLVM/Clang/libc++/compiler-rt/et al? The scope of the work will impact just how much effort is needed for companies and stakeholders.

Also, do you know if any companies or stakeholders have already tentatively said they’d be willing to commit to the efforts in two-year increments? One concern I have is that corporate attention spans are much shorter than two years, but once our community promises an LTS release, we’re on the hook for that support even if corporate priorities shift. So ideally, I think we’d probably like to have several different sources of labor lined up.

5 Likes
4

Totally agree with Aaron on all points. Suggestions in this area have to be backed by commitments to do the work, otherwise we’ll go nowhere.

1 Like
5

Situation around standalone builds, which are lightly used by LLVM community, but heavily used by OS distributions, is an example when we tried to fulfill external needs without someone’s commitment to do the work.

1 Like
6

Agreed with Aaron here. Additionally: Who are the users requesting this? Have they sent mail representing their project to discourse?

1 Like
7

Talking as one of the two release managers here (but I am not talking for Tom, he might feel different).

IF we were to have a LTS version I wouldn’t want to add that to my workload as a RM. I manage one release a year and that’s plenty. The difficult part of the release process is to get people to care about “old code” in any case, I feel like LTS would just make that problem worse.

But that’s all theoretical, I have seen very little requests for LTS (that doesn’t mean there isn’t any, but I haven’t seen it in discourse).

I maintain a toolchain based on LLVM for our company that has a longer lifespan than our current releases, and while I backport patches from main to our branch at times the fixes are often highly specific to something we ran into in our code and not at all generally applicable. So I am still sceptical this is something we need.

Happy to be proven wrong, but I think it can be maintained separately from the community in that case - and if it catches on we could always make it official then.

8

I think it should be determined by the amount of testing each sub-project receives for the LLVM LTS branch.

I believe that testing and CI should be central parts of the LLVM LTS effort, and that support for a particular feature/target should be measured by the quality of testing coverage. In other words, if AArch64 Linux has an LTS CI bot for libc++, but x86_64 doesn’t, then libc++ is supported on AArch64 but not on x86_64.

Speaking with AArch64 Linux buildbots in mind, we will have no problem adding LTS versions of all mainline buildbots that Linaro maintains. The stream of changes flowing into LTS will be much smaller compared to main, so the additional testing bandwidth should be manageable. That means for AArch64 Linux we can include clang, flang, compiler-rt, lld, lldb, libc++ and everything else that current buildbots test.

9

One of the goals of this RFC is to give potential sponsors an opportunity to show their interest. I hope that in a few weeks we will have a public list of interested companies.

Regarding @AaronBallman comment about community promising LTS support without explicit backing by company sponsors – I agree that it’s a non-starter. The general LLVM community may adopt LTS releases down the road, but in the beginning there needs to be a dedicated LTS team supported by companies benefiting from the LTS branches and releases.

10

IF we were to have a LTS version I wouldn’t want to add that to my workload as a RM.

Yes, I agree with that. I’m coming into this discussion with the mindset that the general LLVM community may help with advice and backport reviews, but the LTS team should not depend or count on that.

I maintain a toolchain based on LLVM for our company that has a longer lifespan than our current releases

I would appreciate you sharing more details on your day-to-day downstream maintenance activities, and what the LTS branch could provide to reduce your overhead. I mean if there were a branch with tested backports from which you could cherry-pick bug fixes – wouldn’t it simplify your downstream job?

11

It might, it all depends on the criteria of fixes that a LTS branch would pick. We usually find bugs in an obscure area of the COFF format or missed optimization that makes a huge difference for us specifically, I don’t think many of those fixes would be accepted in a upstream LTS.

We also focuses very heavily on Windows x64, and upstream backing for that platform is already pretty thin. So I doubt anyone would back that for LTS.

That said - there are probably other downstream users that might benefit and if there is actual committed resources to it, I won’t oppose it - but I think it would make sense to start with a “unofficial” LLVM LTS while the waters are tested to reduce the burden on the LLVM community.

1 Like
12

I think it’s worth stressing out that CI capacity seems a much easier problem to solve than review capacity for backport patches. Clang is constantly short on experienced reviewers, and I doubt other subprojects fare much better in this regard.

13

My expectation is that a LTS release would be most useful for Linux Distribution LTS releases or other platforms that have an LTS rather live at head model of development. I’m not personally familiar with that type of work though.

At Arm we maintain a Functional Safety (FuSa for short) Qualified toolchain derived from LLVM. These toolchains have an extended maintenance period going out 5+ years.

This is not strictly the same thing as an LTS release, as the changes are limited to security and a certain class of bugs, mostly Arm and AArch64 code-generation bugs that affect the qualified scope.

While I don’t think any FuSa toolchain will want to use an upstream LTS release directly as it will have too many unrelated changes, an LTS release could provide a good source of backported issues that a FuSa toolchain could evaluate and either cherry-pick or adapt. Similarly to the Linux distribution case a LTS release could help downstream FuSa toolchain teams share more work. Like Linux Distributions, there aren’t too many FuSa toolchains though.

Our development model when we do find a bug is to fix it in upstream and then backport to an older branch. For at least the first few years we find that most fixes are usually straightforward to backport, often just needing a cherry-pick, although there are exceptions when there is a major change like opaque pointers. Most review work is light as the majority of the review work can be reused from the main branch. There is a possibility that backend code is less prone to major refactoring changes than other areas of the code.

I’ve discussed this internally at Arm and we’re theoretically intested in actively helping, but we’d need a stronger statement on where this would likely be used before we can commit.

2 Likes
14

Well, perhaps my perspective is a little different, since for Gentoo the primarily problem are the LLVM libraries rather than the tools. Let’s start with some background.

Right now, we support LLVM versions back to LLVM 15. This is because we still have to deal with important packages that do not support the API of any of the newer versions. And we already lost some functionality by removing LLVM 14.

This roughly means that we (which primarily means me) have to:

  1. Test the state of main branch not to have ton of regressions to deal with when 20.x is branched.
  2. Test 19.x releases and request backports, so we don’t end up having day-zero patchsets.
  3. Maintain patchsets for the 4 remaining releases.

I can imagine other distributions are facing similar problems. Add to that projects like Rust that vendor a specific LLVM version (currently meaning 16.x to 19.x for us), and which we need to maintain in multiple versions to support building different projects (insert a rant on stability of Rust).

Admittedly, LLVM and Clang themselves aren’t technically high-maintenance packages. Still, their size alone means that any backport — and especially backport that needs to be carried all the way back, and that is likely not to apply cleanly to old versions — is a lot of effort. Add to that more effort if you want tests to pass 2 years later.

So, if an “LTS” release would mean that we can, say, rely on projects sticking to a single LLVM version until the next release comes out, and having that release receive patches upstream would certainly be an improvement for us.

In the end, even if I had to handle the backports myself, it would be an improvement, because these patches would benefit others and hopefully be reviewed by someone more competent than myself, rather than having every downstream handle the same thing separately.

Of course, this all relies on upstreams choosing to either focus on the LTS release, or follow new LLVM releases timely. It wouldn’t work if they just slowly moved from release to release as they do now.

1 Like
15

I have some skepticism here: if they find it problematic to migrate from LLVM N to LLVM N+1 in a timely manner, migration from LLVM N to LLVM N+4 is not going to be any less problematic. I can easily see how you can end up supporting multiple LTS releases, ultimately ending up with even wider gap between trunk and the oldest version you need to support.

1 Like
16

Well, there are two ways to look at this. On one hand, going one version forward is probably easier than four versions forward at once. On the other, the total effort of going one version forward four times is probably larger than going four versions forward once.

17

Speaking as the FreeBSD ports/package maintainer (@DimitryAndric maintains the base system LLVM) this would add a small amount of additional work as the ports would need more frequent updates. So long as releases with tarballs are cut at a reasonable frequency, this would be easy and might even encourage me to keep up with infrastructure refactoring I sometimes let slide.

Having LTS branches would have little immediate effect in the number of LLVM ports I maintain (currently LLVM 11-19, a development snapshot, and three CHERI variants).

I have a vague hypothesis that LTS releases would encourage downstream API consumers who don’t stay up to date all the time to pick an LTS rather than some arbitrary release which might eventually make it easier to thin out the list of supported releases. I would expect that any such effect would take years to develop.

For the base system, it’s possible we might settle on LTS versions for stable branches, but it’s not clear to me that would be a good tradeoff for us. We find the cost of compiler/libc++ version skew between branches to be quite high so while we don’t merge all releases, we try not to let them lag too much. FWIW, it’s a bit unclear if FreeBSD 15.0 (~December 2025) will ship with LLVM 20 or 21. It probably depends on how much disruption there is in default warnings/errors.

18

I’m under the impression that the release branches are the answer to users wanting stability over improvements. The LTS Linux distro should take some release version and leave it alone. Updating to a different release or patching the first one means less stability and more work.

I would speculate that a LTS model would be popular with the groups that currently resist moving to a new branch for fear of breakage, and that contributions from those groups would be sparse.

1 Like
19

Answering from the other angle here to @mgorny whose reply was mostly about LLVM-as-a-library use, I’ll focus on Clang as a system compiler - which I also promised @AaronBallman my thoughts on some time ago.

Right now, the Clang release process isn’t amenable to how distributions work. It’s not really clear often when the RC period is starting, the RC period doesn’t seem long enough to begin and do thorough testing and reporting of issues w/ a large package base (keeping in mind that we’ll have other stuff to do, we need to then investigate bugs, look for patterns and figure out if they’re worth raising w/ LLVM upstream, determine for runtime problems if there’s UB in the relevant package, …) before the release is made.

But even once the release is made, there’s a pretty brief period for point releases. @mgorny and I have tried to backport fixes before at the .3/.4 point and been rejected “because we don’t want to make too many more” (though often they get made anyway). But supposing we get up to .6/.7, that’s still only several weeks of coverage from the initial release, then the point-releases stop.

You end up with an “unsupported” release of LLVM for most of its lifetime before the next LLVM major release. This isn’t great for motivation for maintaining LLVM as a system toolchain, nor is it great in terms of motivation for reporting bugs or investigating them at all, given they may well have been fixed on main and with no way to save others from the work by backporting to a branch.

i.e. there’s two problems:

  1. The run-up to branching feels abrupt often without enough signalling, and is too short even then, to really do enough testing to flag up potential problems;
  2. Releases don’t live for very long, leaving most of the year unsupported

I think having branches open for a period after release (far longer than currently) would be a significant improvement rather than one release per N being designated LTS.

4 Likes
20

This RFC goes some way towards ameliorating that (extends the support period, though stops short of multiple supported versions much less LTSes).

Not sure what the current status for 19.1.x is, @tobiashieta mentioned he’d already consider extending the support period for this release.

2 Likes