As a project, Clang has gotten negative public feedback about our lack of communicating changes to users. There are comments on places like Hacker News or Reddit where users have (rightfully) been confused as to what changes happen in a given release, leading to misinformation like Clang not adding support for C++20 features: https://news.ycombinator.com/item?id=28761464.
As a code reviewer, I’ve been asking for release notes in Clang more aggressively in the past release or two in an effort to combat this, and I’ve asked other reviewers to do the same. However, it was observed that our developer documentation really doesn’t talk about release notes at all. So I would like to update our developer policy to make it clear that we expect release notes to be written for significant user-facing changes. I am purposefully trying not to be too prescriptive and leaving room for interpretation as to what’s “significant” or “user-facing” because each project may have a slightly different answer for that. Instead, I opted to use a non-exhaustive list of example situations where a release note would likely be appropriate.
Thank you for starting this effort! I think it is better to add release notes with any new fix or feature than in time before final release - usually with end up with poor release notes and many cool things are missed. It even may look like the development of XYZ is almost dead but on the other hand, there are so many new features - just not documented properly.
Not only Clang needs this effort, all LLVM subprojects should document new (major) changes/features as they are merged to main.
this includes a list of fixed bugs since the previous release. (Possibly generated, and/or reversed, being able to see for a bug in which release it’s fixed)
mostly relevant to see if your bugs are fixed in a release, if improvements happened to ABI/Feature compat with MSVC
progress on implementing features (maybe feature X ain’t completed in the release, however, seeing that it makes progress helps in managing my frustration, just be explicit it ain’t useable yet)
If you don’t implement nr 1, I would like to stress that some changes as response for bugs are as valuable as new features. So please also think about the release notes when looking at bugs.
I’ll leave it up to you to see what you’ll take a long and what is for later.
Thanks for all the effort! This really makes me happy!
Yeah big thumbs up from me as well. I have the dubious task of summarizing each release internally and it always feels like I am missing a lot of changes from the release notes.
Another smaller issue that’s a bit related is that the release notes are so spread out that it’s hard to find an exhaustive list (some subprojects doesn’t seem to publish any notes at all). Would be nice to have it in a more central place.
I can’t believe I missed that in the non-exhaustive list of examples. Oops! I’ve added bug fixing as a specific example, because that’s a very important one to make sure we communicate to people. I also updated the bullet on diagnostics to include re-grouping a diagnostic under a new name. I think the progress and new formatting suggestions are definitely in scope for what I’d hope to see release notes for, but I left them off to keep the list brief instead of making it feel more exhaustive than it really is.
FWIW, we do have release notes somewhat centralized: https://releases.llvm.org/ (the Release Notes column) and LLVM Download Page (each one has a release notes link). Or were you looking for something different than that?
Thanks for writing this up, it’s nice to have the requirements be explicit!
The risk of missing things is real, but there are some downsides to writing notes as we go:
Requires casual contributors to learn our unusual doc markup language, set up the build system, installing extra python software, dealing with doc buildbot failures. This escalates the already-high barrier to getting something landed.
It’s easier to group and summarize related changes at the end, and to maintain consistency and perspective.
I expect to continue needing to summarize changes at release time and to go through git history to do so, and having to convince people to draft release notes in code review seems like it may just be an extra burden.
I expect that need will be lessened greatly. FWIW, I’ve been requiring release notes in Clang for everything in Clang 15 (and a fair amount of stuff in late Clang 14 as well) and I’ve had zero people have issues writing a release note that couldn’t be quickly addressed by the code reviewer. Most often, the question is “where in the release notes should this go?” rather than anything to do with the markdown (which authors already need to know when writing documentation). I disagree there’s any more (significant) burden here than asking people to understand lit to be able to write tests.
The only downside I’ve seen with requiring release notes is that it is a minor source of annoyance when rebasing a patch because of how often release notes step on each other. This is a small price to pay for communicating appropriately with our users.
If we could use markdown, I think the burden would be minimal. ReST is unusual enough that contributors often don’t know it at all, and I miss things in review all the time (like mistaken use of markdown backticks). So at least one of us has to run the actual build…
We’re not disagreeing then, as lit is also a significant barrier . But that’s harder to solve.
We have built bots that tell you when you get the .rst wrong. And again, .rst is required for writing all of our documentation so I disagree that this is a burden that should excuse someone from writing a release note when requested. That said, I think it would be lovely if precommit CI would also build documentation so people could learn about the issues early (but then again, precommit CI is broken so often that I question its utility in general most days).
Your concern is valid in the abstract, but until we have people actually saying “I can’t write a release note because I don’t know how”, I don’t think this concern is sufficient for us to not document an expectation that people write release notes when appropriate or requested by a reviewer.
FWIW, I watch the sphinx bots every day and just fix up people’s mistakes rather than bother them with it – fixing the bot is trivial and only needs to happen a few times a month.
For what it’s worth, my understanding is a vast majority of folks don’t actually install/run sphinx (I know I don’t!) and just copy off of the previous examples, and end up doing a pretty good job. Internally we do the same thing (we’ve extended the llvm documentation format), and even newbies have little/no problems. Our sphinx validation bot fails almost n ever as a result.
I think the concerns over RST files is overblown: our sphinx bot is almost completely green.
Despite Aaron and I encouraging all our the folks to the CFE (and I believe Aaron in clang-tools) to add release notes, we haven’t really seen many failures; way fewer than with everything ELSE that goes wrong during post commit:
If we are THAT concerned about this (which I don’t agree that we have any reason to be), we should probably add the sphinx to the pre-commit CI.
Sony’s internal release note process has always included basically diffing the Options.td file and all the Diagnostic*Kinds.td files, so we can decide whether an option or diagnostic difference (new, changed, removed) should be mentioned. I doubt we would stop doing that, because we also make platform-specific decisions about whether to bother, but having fewer things to look at would be nice!
Thanks for doing this!
For last few libc++ releases I’ve gathered the release notes shortly before the release was branched. We’re now looking at the release notes actively during reviews and I noticed that the release notes were quite up-to-date for the LLVM 14 release.
I haven’t seen contributors having an issue with writing .rst files. In libc++ we validate the documentation in the pre-commit pipeline.
I am wondering if we could add a special label in github issues. And once the corresponding issue is closed, we could generate one release note automatically.
As a user of the LLVM C++ API. E.g. The IR Builder C++ API, it would be helpful to include in the release notes, any changes to that API. I.e. as presented to the user in the /usr/include/llvm/* header files.
Or, if not in the release notes, add LLVM documentation with a page like:
“Upgrading from LLVM version 14 to version 15”, that would explain what needs to change in the user’s code to work with the API changes between 14 and 15.
For example, if a method changes from needing 3 parameters to needing 4 parameters, please leave the old 3 parameter one with a “soon to be depreciated, please use method Y instead” message when compiling, rather than the current approach of just deleting the old 3 parameter one. One is then left guessing how to fix user code to match the new LLVM version.
. But that’s harder to solve.