LLVM 3.4 stable releases

Project Infrastructure LLVM Dev List Archives
1

Hi,

I would like to try again to do stable releases for LLVM 3.4.
Even though we were unsuccessful with stable releases for LLVM
3.3, I learned some things going through the process, which I think
will increase the chance for success with LLVM 3.4.

So, here is my TODO list for a successful 3.4.1 release:

1. Get volunteers to help

This is probably the most important thing on this list. Stable releases
need to be a community effort, there is too much work for just a few people to
do.

2. Choose a release manger.

I will volunteer for this, but if someone else wants to do it, I have no
problem relinquishing the role.

3. Make sure this page: http://llvm.org/docs/ReleaseProcess.html is updated and
  correct:

It would be great if someone who just participated in the release process could
look over this page and make sure it is correct and easy to follow. For example,
to me it's not clear how to compare the results of the test-release.sh script from
different release candidates.

4. Come up with a list of all the platforms which must be tested in
   order to make a release.

5. Determine whether or not stable releases should maintain a stable ABI.

It seems like the preference in the community is to keep a stable ABI
with stable releases. I'm curious what uses cases require a stable ABI?

6. Add patch level to LLVM version (e.g. 3.4.0)

7. Figure out the .so name to use for stable releases (i.e. Should we
use libLLVM-3.4.so or libLLVM-3.4.1.so)

8. Add support for stable releases to the test-release.sh script.
   This may go along with adding the patch level to LLVM version.

9. Choose a release date for LLVM 3.4.1

My recommendation is to release LLVM 3.4.1 3 months after 3.4 and have it
be the only stable release. Once we get better as a community at doing
stable releases, then maybe we can consider doing more.

I think this list is a good starting point. Anyone else have any ideas
or other feedback?

Thanks,
Tom

2

1. Get volunteers to help

I'm available to do point release builds and tests, as I do for releases.

3. Make sure this page: How To Validate a New Release — LLVM 18.0.0git documentation is updated

and

  correct:

I may have to add a few things... but nothing serious.

4. Come up with a list of all the platforms which must be tested in

   order to make a release.

So, this depends on who is available at the time the point release comes
out. The work Bill did to organize all this was not small, and it takes
time to do it all, and people take holidays, move jobs, etc.

Until we have an official commitment from companies that care about LLVM,
in which someone should always be available to do the release from that
company, we can't commit ourselves to anything. If you remember the
official release process and the number of binaries released, there was
nothing "official" about that process either.

5. Determine whether or not stable releases should maintain a stable ABI.

As I said earlier, I think this is paramount.

What this means is that building blocks will remain compatible with the
release, as to minimise re-compilation of out-of-tree projects.

I can think of things like IRBuilder, DIBuilder, APInt, StringRef, but also
the Debug Information metadata format, IR semantics, pass order,
command-line options, etc.

In essence, release 3.4.1 should have nothing more than bug fixes to
existing infrastructure, either crashes, conformance or performance, but
never change the semantics.

7. Figure out the .so name to use for stable releases (i.e. Should we

use libLLVM-3.4.so or libLLVM-3.4.1.so)

AFAICR, the LSB uses symlinks like:

libLLVM-3.4.so -> libLLVM-3.4.so.3
libLLVM-3.4.so.3 -> libLLVM-3.4.so.3.4.1
libLLVM-3.4.so.3.4 (deprecated)
libLLVM-3.4.so.3.4.1

I'm not sure LLVM does any of that, but we'd have to do something similar
if we are to add point releases.

9. Choose a release date for LLVM 3.4.1

My recommendation is to release LLVM 3.4.1 3 months after 3.4 and have it
be the only stable release. Once we get better as a community at doing
stable releases, then maybe we can consider doing more.

I'd say we only do a release if a serious enough bug shows up after we
release LLVM. I wouldn't go for performance reviews right now because
that's impossible to manage at our level of commitment.

We cannot test all programs in all platforms. We cannot benchmark all
benchmarks on all platforms. We may not even be able to compile it for all
platforms...

For those reasons, it's impossible to ascertain the quality of any revision
after the official release unless you do the release process every time,
which won't happen because it takes three months to do so.

* My Impressions

If I got it right, you're referring to a different kind of point release,
just checking out a release and testing it, as if it was just another
release in between. I personally see no value in doing so. The only value
is to backport bug-fixes into the original branch-3.4 and make a release
for production systems that have tested 3.4 and need a bug fixed.

My reasoning is this:

1. The LLVM community follows trunk. All developers, all out-of-tree
developers and all libraries and sub-projects do so. There is no point in
marking release A or B when no one even cares about releases.

2. Users that need stability (Google for Renderscript, Apple of iOS, etc)
get a random revision, test it thoroughly, fix a few bugs directly, and
ship. No one, AFAIK, uses or depend on any specific release of LLVM. We do
releases more to track conformance and performance than anything else.

3. Users that do need a specific release (research projects, proprietary
R&D) will stick to a release and not change for years. Three months won't
make any difference for them, and they will only move to a newer version if
there is a significant feature added or bug fixed.

For those reasons, releasing anything bug bug-fixes in between releases is
not just unnecessary, but a waste of time. Besides, buildbots already do
most of this job (compile checks, test-suites, etc), so if you do want to
release a revision, I'd advise you to do so unofficially and based on
buildbot status.

If you're instead, referring to bug-fix releases, than backporting specific
commits to the old branch won't generally break any API or semantics, and
it should be safe for us to test and ship. Also, it should only take one
cycle of tests, since there won't be many changes and most of it was
already tested on release anyway.

Makes sense?

cheers,
--renato

3

> 1. Get volunteers to help
>

I'm available to do point release builds and tests, as I do for releases.

3. Make sure this page: How To Validate a New Release — LLVM 18.0.0git documentation is updated
> and
>
  correct:
>

I may have to add a few things... but nothing serious.

4. Come up with a list of all the platforms which must be tested in
> order to make a release.
>

So, this depends on who is available at the time the point release comes
out. The work Bill did to organize all this was not small, and it takes
time to do it all, and people take holidays, move jobs, etc.

Until we have an official commitment from companies that care about LLVM,
in which someone should always be available to do the release from that
company, we can't commit ourselves to anything. If you remember the
official release process and the number of binaries released, there was
nothing "official" about that process either.

5. Determine whether or not stable releases should maintain a stable ABI.
>

As I said earlier, I think this is paramount.

What this means is that building blocks will remain compatible with the
release, as to minimise re-compilation of out-of-tree projects.

I can think of things like IRBuilder, DIBuilder, APInt, StringRef, but also
the Debug Information metadata format, IR semantics, pass order,
command-line options, etc.

In essence, release 3.4.1 should have nothing more than bug fixes to
existing infrastructure, either crashes, conformance or performance, but
never change the semantics.

7. Figure out the .so name to use for stable releases (i.e. Should we
> use libLLVM-3.4.so or libLLVM-3.4.1.so)
>

AFAICR, the LSB uses symlinks like:

libLLVM-3.4.so -> libLLVM-3.4.so.3
libLLVM-3.4.so.3 -> libLLVM-3.4.so.3.4.1
libLLVM-3.4.so.3.4 (deprecated)
libLLVM-3.4.so.3.4.1

I'm not sure LLVM does any of that, but we'd have to do something similar
if we are to add point releases.

9. Choose a release date for LLVM 3.4.1
>

> My recommendation is to release LLVM 3.4.1 3 months after 3.4 and have it
> be the only stable release. Once we get better as a community at doing
> stable releases, then maybe we can consider doing more.
>

I'd say we only do a release if a serious enough bug shows up after we
release LLVM. I wouldn't go for performance reviews right now because
that's impossible to manage at our level of commitment.

We cannot test all programs in all platforms. We cannot benchmark all
benchmarks on all platforms. We may not even be able to compile it for all
platforms...

For those reasons, it's impossible to ascertain the quality of any revision
after the official release unless you do the release process every time,
which won't happen because it takes three months to do so.

* My Impressions

If I got it right, you're referring to a different kind of point release,
just checking out a release and testing it, as if it was just another
release in between. I personally see no value in doing so. The only value
is to backport bug-fixes into the original branch-3.4 and make a release
for production systems that have tested 3.4 and need a bug fixed.

Sorry this wasn't clear. My assumption was that we would be backing-porting
fixes from master to 3.4 and making a release from the 3.4 branch.

4

Right, then we should be safe regarding API stability, and it should be
pretty simple to release in one cycle, rather than the current three
candidates.

cheers,
--renato

5

Hi,

6

I'm happy to run tests.

Ben

7

I would also be in for building & testing.

Cheers,

8

Hi,

1. Get volunteers to help

This is probably the most important thing on this list. Stable releases
need to be a community effort, there is too much work for just a few people to
do.

Count me in.

2. Choose a release manger.

I will volunteer for this, but if someone else wants to do it, I have no
problem relinquishing the role.

I would have been interested a few months ago but since I am going to
start a new job next Tuesday.
I don't think that would be reasonable.

4. Come up with a list of all the platforms which must be tested in
   order to make a release.

5. Determine whether or not stable releases should maintain a stable ABI.

It seems like the preference in the community is to keep a stable ABI
with stable releases. I'm curious what uses cases require a stable ABI?
From the distro pov, all libraries depending on LLVM or clang libraries.

That is starting to be important.
If you change the ABI, we, distribution, would have to rebuild all
packages depending on
LLVM & Clang. If we keep the same ABI, this work won't be necessary.
Changing an ABI is really a nightmare from the distribution point of view...

6. Add patch level to LLVM version (e.g. 3.4.0)

7. Figure out the .so name to use for stable releases (i.e. Should we
use libLLVM-3.4.so or libLLVM-3.4.1.so)

libLLVM-3.4.so should be OK if the ABI does not change.

9. Choose a release date for LLVM 3.4.1

My recommendation is to release LLVM 3.4.1 3 months after 3.4 and have it
be the only stable release. Once we get better as a community at doing
stable releases, then maybe we can consider doing more.

I am not sure that it is about date but more about the number of
critical bugs in the current stable release...

I would add a 10) point.
Clearly document what kind of patches are acceptable from a stable release.

For me, it should be only important bug fixes, no new feature.

Sylvestre

9

From: "Sylvestre Ledru" <sylvestre@debian.org>
To: "Tom Stellard" <tom@stellard.net>, llvmdev@cs.uiuc.edu
Cc: "erik verbruggen" <erik.verbruggen@me.com>
Sent: Thursday, January 16, 2014 12:20:38 PM
Subject: Re: [LLVMdev] LLVM 3.4 stable releases

> Hi,

> 1. Get volunteers to help
>
> This is probably the most important thing on this list. Stable
> releases
> need to be a community effort, there is too much work for just a
> few people to
> do.
Count me in.

Me too! Tom, thanks for doing this!

>
> 2. Choose a release manger.
>
> I will volunteer for this, but if someone else wants to do it, I
> have no
> problem relinquishing the role.

I'm certainly willing to help with this (I'm a bit hesitant to say that I'll do it, because I'm not sure what's involved with actually creating the binaries). I'll certainly help with culling the bug fixes to be back ported into the stable branch.

I would have been interested a few months ago but since I am going to
start a new job next Tuesday.
I don't think that would be reasonable.
> 4. Come up with a list of all the platforms which must be tested in
> order to make a release.
>
> 5. Determine whether or not stable releases should maintain a
> stable ABI.
>
> It seems like the preference in the community is to keep a stable
> ABI
> with stable releases. I'm curious what uses cases require a stable
> ABI?
From the distro pov, all libraries depending on LLVM or clang
libraries.
That is starting to be important.
If you change the ABI, we, distribution, would have to rebuild all
packages depending on
LLVM & Clang. If we keep the same ABI, this work won't be necessary.
Changing an ABI is really a nightmare from the distribution point of
view...
> 6. Add patch level to LLVM version (e.g. 3.4.0)
>
> 7. Figure out the .so name to use for stable releases (i.e. Should
> we
> use libLLVM-3.4.so or libLLVM-3.4.1.so)

libLLVM-3.4.so should be OK if the ABI does not change.

+1

But, we might also want to symlink a libLLVM-3.4.1.so to it; that way if you ship a binary that depends on some bug fix in 3.4.1 (and explicitly link against 3.4.1), it can't accidentally be run in an environment with only 3.4 (obviously, there are other ways to do this, but that seems the simplest).

>
> 9. Choose a release date for LLVM 3.4.1
>
> My recommendation is to release LLVM 3.4.1 3 months after 3.4 and
> have it
> be the only stable release. Once we get better as a community at
> doing
> stable releases, then maybe we can consider doing more.
I am not sure that it is about date but more about the number of
critical bugs in the current stable release...

I would add a 10) point.
Clearly document what kind of patches are acceptable from a stable
release.

For me, it should be only important bug fixes, no new feature.

I think we should take bug fixes, generally, (crashes and especially miscompiles), but only those that are did not involve any significant amount of refactoring (or are tied to new features).

-Hal

10

Yup, so am I.

– Erik

11

> From: "Sylvestre Ledru" <sylvestre@debian.org>
> To: "Tom Stellard" <tom@stellard.net>, llvmdev@cs.uiuc.edu
> Cc: "erik verbruggen" <erik.verbruggen@me.com>
> Sent: Thursday, January 16, 2014 12:20:38 PM
> Subject: Re: [LLVMdev] LLVM 3.4 stable releases
>
> > Hi,
>
> > 1. Get volunteers to help
> >
> > This is probably the most important thing on this list. Stable
> > releases
> > need to be a community effort, there is too much work for just a
> > few people to
> > do.
> Count me in.

Me too! Tom, thanks for doing this!

> >
> > 2. Choose a release manger.
> >
> > I will volunteer for this, but if someone else wants to do it, I
> > have no
> > problem relinquishing the role.

I'm certainly willing to help with this (I'm a bit hesitant to say that I'll do it, because I'm not sure what's involved with actually creating the binaries). I'll certainly help with culling the bug fixes to be back ported into the stable branch.

I think usually the testers are responsible for creating the binaries.
The release manager is in charge of committing the patches to the stable
branch and also making sure the release process guidelines are being
followed.

I think tracking down all the bug-fixes could be difficult, so it would be
good to have several people helping with this.

-Tom

12

Hi,

Thanks for the feedback. Here is a summary of the responses.
These items are still up for discussion, but if there are no
objections in the next few days, I will add these to the
release documentation:

+ Stable release must be ABI compatible with the current major
  release.

+ Only bug fixes will be accepted in stable releases. All changes
  to the stable branch must first be committed to trunk (when applicable)
  and approved by the code appropriate code owner.

+ Shorter release cycle with 1 release candidate.

+ Shared library name will remain libLLVM-Major.Minor.so, but
  a libLLVM-Major.Minor.Patch.so symlink will be added.

+ Supported platforms will be determined by test coverage.

+ The release manager will determine when to make a stable release based
  on input from the community. This will be a judgement call, but
  the basic guidelines are to do a release if there are a large number
  of bug-fixes sitting in the stable branch or some critical bugs are
  found after a release that affect a large number of users.

TODO list:

+ Add patch level to LLVM version.
+ Add support for stable releases to test-release.sh script.
+ Update testing and release documentations.
+ Identify candidate patches for the stable branch.

-Tom

13

Thanks for the feedback. Here is a summary of the responses.
These items are still up for discussion, but if there are no
objections in the next few days, I will add these to the
release documentation:

Hi Tom,

Thanks for the summary, I agree with mostly everything, but I have some
specific comments inline...

+ Shared library name will remain libLLVM-Major.Minor.so, but

  a libLLVM-Major.Minor.Patch.so symlink will be added.

You mean a library M.m.p will be created and a symlink M.m will point to
it, right?

+ Supported platforms will be determined by test coverage.

I'm in two minds about it. While it makes no sense to re-build for all
archs if the bug-fix is specific to only one, it'll confuse people if you
need to use the same toolchain across different architectures, and we'll
end up replying on the list: "oh, 3.5.1 is only for ARM and x86, while
3.5.2 is only for Mips...", etc. I think we should release all binaries,
and maybe test less extensively on the architectures that have no
bug-fixes, but still build the binaries, the libraries, and symlinks.

The problem arises when no one will build and test for arch A and we simply
cannot release something without even knowing it runs. We shouldn't stop
the release because of that, and that would be an acceptable case where
there simply is no patch-release for that arch. I would expect it not to
happen for the architectures listed as "supported" in the documentation,
and in time, it should be "frowned upon" for "supported" architectures NOT
to do a patch release.

cheers,
--renato

14

> Thanks for the feedback. Here is a summary of the responses.
> These items are still up for discussion, but if there are no
> objections in the next few days, I will add these to the
> release documentation:
>

Hi Tom,

Thanks for the summary, I agree with mostly everything, but I have some
specific comments inline...

+ Shared library name will remain libLLVM-Major.Minor.so, but
>
  a libLLVM-Major.Minor.Patch.so symlink will be added.
>

You mean a library M.m.p will be created and a symlink M.m will point to
it, right?

Yes.

libLLVM-M.m.so -> libLLVM-M.m.p.so

+ Supported platforms will be determined by test coverage.
>

I'm in two minds about it. While it makes no sense to re-build for all
archs if the bug-fix is specific to only one, it'll confuse people if you
need to use the same toolchain across different architectures, and we'll
end up replying on the list: "oh, 3.5.1 is only for ARM and x86, while
3.5.2 is only for Mips...", etc. I think we should release all binaries,
and maybe test less extensively on the architectures that have no
bug-fixes, but still build the binaries, the libraries, and symlinks.

How many people actually depend on the binaries posted on llvm.org? I
agree that if we are providing binaries we should try to build binaries
for the same architectures each release. However, it seems to me like a
majority of LLVM's users are using a binary produced somewhere else, so
I wonder if we even need to worry about which architectures we build for.

I'm much more concerned about platform testing and validation than I am
the binaries.

The problem arises when no one will build and test for arch A and we simply
cannot release something without even knowing it runs. We shouldn't stop
the release because of that, and that would be an acceptable case where
there simply is no patch-release for that arch. I would expect it not to
happen for the architectures listed as "supported" in the documentation,
and in time, it should be "frowned upon" for "supported" architectures NOT
to do a patch release.

What you've described is the main problem I'm trying to avoid. Where we say we
support ARM, for example, and then there is a critical fix for X86 somewhere
in the core libraries, and we can't do a release because there are no ARM testers
to build binaries.

As far as I understand, there only two "supported" or "first-tier" platforms:
X86 and ARM. I just want it to be clear what to do in case there are no testers
for one of these platforms. Do we cancel the release, or do we take the position
that if no one is willing to provide testing resources for a platform, then it
is not really worthy of "supported" or "first-tier" status.

-Tom

15

I'm much more concerned about platform testing and validation than I am
the binaries.

That's a good point. Distributions should trust our source branches because
we tested on the release platforms, not because our binary releases are out.

As far as I understand, there only two "supported" or "first-tier"

platforms:
X86 and ARM.

Yes. That makes three architectures: x86, x86_64 and ARM32.

ARM64, Mips, PPC and others still aren't "first-tier" so we don't need to
worry about them right now.

Do we cancel the release, or do we take the position

that if no one is willing to provide testing resources for a platform,
then it
is not really worthy of "supported" or "first-tier" status.

We can't cancel the status of first-tier because patch releases are not yet
official. Heck, even skipping an official release wouldn't necessarily
remove the status if we have buildbots, and extensive offline tests
elsewhere. So we can't play that card.

Since both x86 and ARM communities are large enough, I don't think we'll
ever be without hands to at least build the release and run a test-suite.
That should give us enough peace of mind to progress with the release. But
as more platforms start joining the release process, we'll have to make
sure they can cope with the process when demand comes.

Furthermore, as it stands, we have at least two people willing to build and
test for each platform. As a last resort, I could do all three releases,
since I have the hardware available, so we can still test the patch release
on all three major archs. Other people in this list can also do the same,
so I think that we're covered.

cheers,
--renato

16

What’s the state here?

There are three commits I’ve made to Clang that I’d like to see in the 3.4 release branch if for no other reason than to help out folks bootstrapping on old Linux distributions with too-old installed versions of GCC/libstdc++. These are r199632, r199633, and r199769. Let me know if you can merge them or I should or how we can get a nice stable tree that folks can check out and build as the first step of getting a bootstrap.

17

Hi Chandler,

We haven't set a fixed date or anything, but if I got it right, those three
patches are independent from any API changes, so they should be good to go
on 3.4.1. Have you tried to apply them locally on a 3.4 branch? I think the
first steps are to re-base and send a merge request to the commits list. We
can follow from there...

Tom, would be good to track all the patches that go in, so at least we have
a way to update the change log.

cheers,
--renato

18

There are three commits I've made to Clang that I'd like to see in the
3.4 release branch if for no other reason than to help out folks
bootstrapping on old Linux distributions with too-old installed versions of
GCC/libstdc++. These are r199632, r199633, and r199769. Let me know if you
can merge them or I should or how we can get a nice stable tree that folks
can check out and build as the first step of getting a bootstrap.

Hi Chandler,

We haven't set a fixed date or anything, but if I got it right, those
three patches are independent from any API changes, so they should be good
to go on 3.4.1. Have you tried to apply them locally on a 3.4 branch? I
think the first steps are to re-base and send a merge request to the
commits list. We can follow from there...

I don't think there is any need to rebase or send merge requests etc...
They'll apply cleanly, or I can manually merge them if need be. The
question is not how to merge them but *if* we should merge them, and if
someone will create tags / release numbers to track this.

19

That's what I meant by "merge request", not if they *would* apply, but if
they *should* be applied. :wink:

It wasn't clear from my first email, I apologise, but I'm trying to come up
with a process where we can easily detect, either now and later if you
search the list, which patches are being proposed for back-porting. Today,
we use "[PATCH]" for trunk, maybe we should have something like "[3.4.1]"
or "[PATCH 3.4.1]" for the back-ports.

The current release process is a bit confused in that we don't have a clear
idea of which patches are being applied to each release candidate, because
they're all spread out in the list without a clear identification, normally
just copying the release manager, or waiting for him/her to pick it up from
the discussion. I'd like to reduce this overhead by having clear semantics,
so that the release manager could just filter for a specific pattern and be
sure to have caught all back-port patches.

That'd also make it easier to write the change log, just by filtering
emails on a mailing list.

Makes sense?

cheers,
--renato

20

> I don't think there is any need to rebase or send merge requests etc...
> They'll apply cleanly, or I can manually merge them if need be. The
> question is not how to merge them but *if* we should merge them, and if
> someone will create tags / release numbers to track this.
>

That's what I meant by "merge request", not if they *would* apply, but if
they *should* be applied. :wink:

It wasn't clear from my first email, I apologise, but I'm trying to come up
with a process where we can easily detect, either now and later if you
search the list, which patches are being proposed for back-porting. Today,
we use "[PATCH]" for trunk, maybe we should have something like "[3.4.1]"
or "[PATCH 3.4.1]" for the back-ports.

What have for the Mesa project is a -stable list that we cc on all patches
that are candidates for the stable branch. This is really easy to do with git
since you can add the CC: mesa-stable annotation to the commit and it is
automatically sent to the stable list when you use git send-email. Also,
since the CC: annotation is in the commit message when it is committed to
master you can have a script auto-cherry pick all the stable patches to
the stable branch.

For LLVM, we could set up a stable branch and cc that on all stable patch
candidates. This would make it easier for multiple people to track which
patches are being proposed for stable. Though, this is probably the
equivalent of what Renato suggested, since people could just as easily
setup mail filters for the [PATCH 3.4.1] tag.

In addition, I think it would be nice to do what Mesa does and add a standard
annotation in the commit message for stable candidates. Something like:
NOTE: This is a candidate for the 3.4 branch so we could partially

This would help automate the backporting process and make the release manager's
life much easier.

-Tom