LLVM 14 Release Retrospective

LLVM Project
1

Hi,

The LLVM 14 release cycle is over now, and I would like to do a retrospective to get feedback about how the release process went. If you want to provide feedback, please answer one of the following questions in the comments:

  • What went well?
  • What did not go well?
  • What can be improved?

As a reminder, we implemented two major changes in this cycle, the first was the new automated backport process with pre-commit CI and the second was the new stable release cycle with faster more frequent stable releases. Feedback on these changes or any other parts of the release process will be greatly appreciated.

2

Thanks for managing all of this process!

My response is from a very packaging-centric perspective, adjust bias compensators accordingly.

Went well:

  • Switching to the new release cycle
  • Responsiveness on discussing (though not necessarily fixing) issues by the community and RM
  • Automated backports (also dealt with conflicts :+1:)

Not so well:

  • I was trying to build all the RCs for LLVM 14 in conda-forge, and there was pretty substantial breakage between the last RC & GA around the whole tarball situation - this get not get fixed on the 14.x branch despite being milestoned since 14.0.1.
  • Changes around CLANG_SONAME were not in the release notes and not well documented or consistent across platforms.
  • Since 14.0.5 (and still on 14.0.6), we’re experiencing hangs on windows in compiler-rt, lld, openmp. Haven’t had time to debug or raise an issue.

Improvements:

  • Do not allow infrastructure changes between the last RC and GA - if the changes are necessary, add another RC.
  • Already the 4 RCs of LLVM 14 were on the lower side compared to the 4, 5, 6 RCs of LLVM 13, 12, 11 (respectively). I’m concerned that the 2 RCs proposed for LLVM 15 will not be enough to ensure good release quality (unless there are other procedural changes I’m not aware of), especially if people aren’t aware of this reduction and don’t prioritize testing from rc1 (rather than waiting for an rc3 that might not happen).
  • Currently, there seems to be no real “release blocker” category that actually blocks a release - perhaps I’m wrong in this perception, but everything unfixed at a specific time seems to just get pushed to the next (patch) release. I understand the advantages for the higher-frequency “ship-what-we-have” mindset, but it would IMO be worthwhile to consider something like that (to be used sparingly).
3

I think one issue here is that there is a substantial number of known mis-compiles and crashes. (see Help Wanted: Improve correctness and reliability of LLVM by fixing known issues - #6 by fhahn)

Hopefully we will be able to slowly reduce those numbers, which should put us in a position to be slightly stricter with release-blocking issues.