4.0.1 Release Schedule + Need feedback for improving stable releases

Project Infrastructure Release Testers
1

Hi,

I would like to start a discussion about improvements to the
stable release process, but first, here is a proposed schedule
for the 4.0.1 release.

May 1, 2017 -rc1
May 22, 2017 Deadline for submitting merge request
May 29, 2017 Deadline for merging changes.
June 3, 2017 -rc2
June 10, 2017 Final Release

This is slightly different from previous stable releases in that
we are doing an early -rc1 release. This matches what we do for
major releases, and I think it will help us get more feedback from
testers sooner.

There is now a script in utils/release, called merge-request.sh, which
you can use to submit merge requests to bugzilla. It's recommend
that you use this script in order to give us consistent bugzilla entries.
However, you can still file bugzilla requests manually if for some
reason you are unable to use the script.

If you file bugs manually, make sure you mark it as a blocker for
the release metabug: 32061 and add a link the commit on phabricator
in the url field: e.g http://reviews.llvm.org/rL12345

Description for the bug should be in the form of:

Merge r123456 into the 4.0 branch : <80 character commit message summary>

* Stable Release Improvements *

I would like to start an informal discussion about stable release
improvements. My goals for this discussion are to find ways to
get more community members/organizations using the official stable
releases, and also to help simplify the release process for users,
testers, and release managers.

* Ideas:

1. Longer support for stable release. We currently have 2 major and 2
stable releases per year. Each major release is effectively supported
for only 3 months, because after the x.y.1 release , we move on to the
next major version. Our current release schedule is approximately:

March: x.y.0
June x.y.1
Sept: (x+1).y.0
Dec: (x+1).y.1

One idea is to move to 6 releases per year and have an additional
x.y.2 release after (x+1).y.0, but before (x+1).y.0. This would
mean that stable releases are supported for ~ 8 months. So something
like:

Jan: (x-1).y.1
March: x.y.0
May: (x-1).y.2
July: x.y.1
Sept: (x+1).y.0
Nov: x.y.2

What do people think about this or other alternative release schedules?
Would this kind of extended support make stable releases more
attractive to users/organizations that use LLVM.

My main concern with adding an extra release is placing an extra
burden on release testers. For release testers is doing two
more releases per year practical?

2. Automated release testing. This is connected to the previous
idea. Can we automate the release testing? It would be great if
we had some release tester bots that would test the release
candidates and upload the binaries. It seems like this should
be doable within the current buildbot infrastructure. Is the
only thing that keeps us from doing this, lack of dedicated test
hardware?

3. Automated merge requests. I would like to come up with a way
that we can tag commit messages so that a post-commit hook would
be able to automatically create merge requests in bugzilla.
There are a few ways I can think of to tag the messages, but I'm
just going to propose that adding the following line:

STABLE: 4.0

To commit messages would make it a candidate for an automated
merge request. Any thoughts?

4. Other Ideas. Does anyone else have other ideas for improving
stable releases?

Thanks,
Tom

2

Hi,

I would like to start a discussion about improvements to the
stable release process, but first, here is a proposed schedule
for the 4.0.1 release.

May 1, 2017 -rc1
May 22, 2017 Deadline for submitting merge request
May 29, 2017 Deadline for merging changes.
June 3, 2017 -rc2
June 10, 2017 Final Release

This is slightly different from previous stable releases in that
we are doing an early -rc1 release. This matches what we do for
major releases, and I think it will help us get more feedback from
testers sooner.

Sounds good to me.

There is now a script in utils/release, called merge-request.sh, which
you can use to submit merge requests to bugzilla. It's recommend
that you use this script in order to give us consistent bugzilla entries.
However, you can still file bugzilla requests manually if for some
reason you are unable to use the script.

If you file bugs manually, make sure you mark it as a blocker for
the release metabug: 32061 and add a link the commit on phabricator
in the url field: e.g http://reviews.llvm.org/rL12345

Description for the bug should be in the form of:

Merge r123456 into the 4.0 branch : <80 character commit message summary>

* Stable Release Improvements *

I would like to start an informal discussion about stable release
improvements. My goals for this discussion are to find ways to
get more community members/organizations using the official stable
releases, and also to help simplify the release process for users,
testers, and release managers.

* Ideas:

1. Longer support for stable release. We currently have 2 major and 2
stable releases per year. Each major release is effectively supported
for only 3 months, because after the x.y.1 release , we move on to the
next major version. Our current release schedule is approximately:

March: x.y.0
June x.y.1
Sept: (x+1).y.0
Dec: (x+1).y.1

One idea is to move to 6 releases per year and have an additional
x.y.2 release after (x+1).y.0, but before (x+1).y.0. This would
mean that stable releases are supported for ~ 8 months. So something
like:

Jan: (x-1).y.1
March: x.y.0
May: (x-1).y.2
July: x.y.1
Sept: (x+1).y.0
Nov: x.y.2

What do people think about this or other alternative release schedules?
Would this kind of extended support make stable releases more
attractive to users/organizations that use LLVM.

My main concern with adding an extra release is placing an extra
burden on release testers. For release testers is doing two
more releases per year practical?

I'd be fine with it. I don't find the release testing itself to be
particularly burdensome. The hard part of releases is chasing down and
fixing bugs, but for .2 releases I expect the process to be pretty
smooth, basically building and testing a build with a few extra
patches on top of .1.

2. Automated release testing. This is connected to the previous
idea. Can we automate the release testing? It would be great if
we had some release tester bots that would test the release
candidates and upload the binaries. It seems like this should
be doable within the current buildbot infrastructure. Is the
only thing that keeps us from doing this, lack of dedicated test
hardware?

I think it's more a problem of finding someone to configure and
maintain the bots.

3. Automated merge requests. I would like to come up with a way
that we can tag commit messages so that a post-commit hook would
be able to automatically create merge requests in bugzilla.
There are a few ways I can think of to tag the messages, but I'm
just going to propose that adding the following line:

STABLE: 4.0

To commit messages would make it a candidate for an automated
merge request. Any thoughts?

I'm not sure this is worth it. Aren't many of the "we should merge
this to stable" decisions made after the commit lands?

A commit hook I'd really like is something that updates the bugzilla
when a commit message mentions a PR. That's a different story though
:slight_smile:

Cheers,
Hans

3

Tom - In terms of release testing - What about using OpenSUSE build
service? It could be possible to setup a toolchain and try to build a
ton of packages. Most of this could be easily automated and you may
significantly extend the amount of code coverage you get. I don't
think build service is able to run make test, but you'd possibly flush
out build and link issues. (As well as seeing how compatible packages
are with clang)

4

OBS can run tests on a lot of platforms, see the official llvm package
for example: https://build.opensuse.org/package/show/devel:tools:compiler/llvm4
and clang-trunk package which is kind-of-daily snapshots:
https://build.opensuse.org/project/show/home%3Anamtrac%3Aclang

If we want to use OBS, I am ready to help with integration and such.

Regards,
ismail

5

tbh I think swapping out clang for gcc and building against the most
bleeding edge wouldn't be a bad thing. I'm not sure if clang can build
the latest linux kernel or what patchset would be required as well as
grub2 and glibc, but besides those I'm not aware of any particularly
nasty blockers.

opensuse+clang (without gcc at all) is in my mind an ideal goal. I
don't know what weird things gentoo has corrupted clang with, but
someone with gentoo and clang may also be able to help flush out a
whole world level of testing.

Just some drive-by thoughts since additional testing is coming up as a topic..

6

Hi,

I would like to start a discussion about improvements to the
stable release process, but first, here is a proposed schedule
for the 4.0.1 release.

May 1, 2017 -rc1
May 22, 2017 Deadline for submitting merge request
May 29, 2017 Deadline for merging changes.
June 3, 2017 -rc2
June 10, 2017 Final Release

This is slightly different from previous stable releases in that
we are doing an early -rc1 release. This matches what we do for
major releases, and I think it will help us get more feedback from
testers sooner.

Sounds good to me.

There is now a script in utils/release, called merge-request.sh, which
you can use to submit merge requests to bugzilla. It's recommend
that you use this script in order to give us consistent bugzilla entries.
However, you can still file bugzilla requests manually if for some
reason you are unable to use the script.

If you file bugs manually, make sure you mark it as a blocker for
the release metabug: 32061 and add a link the commit on phabricator
in the url field: e.g http://reviews.llvm.org/rL12345

Description for the bug should be in the form of:

Merge r123456 into the 4.0 branch : <80 character commit message summary>

* Stable Release Improvements *

I would like to start an informal discussion about stable release
improvements. My goals for this discussion are to find ways to
get more community members/organizations using the official stable
releases, and also to help simplify the release process for users,
testers, and release managers.

* Ideas:

1. Longer support for stable release. We currently have 2 major and 2
stable releases per year. Each major release is effectively supported
for only 3 months, because after the x.y.1 release , we move on to the
next major version. Our current release schedule is approximately:

March: x.y.0
June x.y.1
Sept: (x+1).y.0
Dec: (x+1).y.1

One idea is to move to 6 releases per year and have an additional
x.y.2 release after (x+1).y.0, but before (x+1).y.0. This would
mean that stable releases are supported for ~ 8 months. So something
like:

Jan: (x-1).y.1
March: x.y.0
May: (x-1).y.2
July: x.y.1
Sept: (x+1).y.0
Nov: x.y.2

What do people think about this or other alternative release schedules?
Would this kind of extended support make stable releases more
attractive to users/organizations that use LLVM.

My main concern with adding an extra release is placing an extra
burden on release testers. For release testers is doing two
more releases per year practical?

I'd be fine with it. I don't find the release testing itself to be
particularly burdensome. The hard part of releases is chasing down and
fixing bugs, but for .2 releases I expect the process to be pretty
smooth, basically building and testing a build with a few extra
patches on top of .1.

2. Automated release testing. This is connected to the previous
idea. Can we automate the release testing? It would be great if
we had some release tester bots that would test the release
candidates and upload the binaries. It seems like this should
be doable within the current buildbot infrastructure. Is the
only thing that keeps us from doing this, lack of dedicated test
hardware?

I think it's more a problem of finding someone to configure and
maintain the bots.

3. Automated merge requests. I would like to come up with a way
that we can tag commit messages so that a post-commit hook would
be able to automatically create merge requests in bugzilla.
There are a few ways I can think of to tag the messages, but I'm
just going to propose that adding the following line:

STABLE: 4.0

To commit messages would make it a candidate for an automated
merge request. Any thoughts?

I'm not sure this is worth it. Aren't many of the "we should merge
this to stable" decisions made after the commit lands?

Yes, good point.

A commit hook I'd really like is something that updates the bugzilla
when a commit message mentions a PR. That's a different story though
:slight_smile:

I like this idea better than what I had in mind, this way instead
of parsing commit messages, we would just need to look at closed bugs
to figure out what to merge.

7

Hi,

I would like to start a discussion about improvements to the
stable release process, but first, here is a proposed schedule
for the 4.0.1 release.

May 1, 2017 -rc1
May 22, 2017 Deadline for submitting merge request
May 29, 2017 Deadline for merging changes.
June 3, 2017 -rc2
June 10, 2017 Final Release

This is slightly different from previous stable releases in that
we are doing an early -rc1 release. This matches what we do for
major releases, and I think it will help us get more feedback from
testers sooner.

There is now a script in utils/release, called merge-request.sh, which
you can use to submit merge requests to bugzilla. It's recommend
that you use this script in order to give us consistent bugzilla entries.
However, you can still file bugzilla requests manually if for some
reason you are unable to use the script.

I like the fact that there is a script!
It’d be really nice to have an entry on the website describing how to use this. I always struggle to find the right email that tells me how to handle cherry-picks.

If you file bugs manually, make sure you mark it as a blocker for
the release metabug: 32061 and add a link the commit on phabricator
in the url field: e.g http://reviews.llvm.org/rL12345

Description for the bug should be in the form of:

Merge r123456 into the 4.0 branch : <80 character commit message summary>

* Stable Release Improvements *

I would like to start an informal discussion about stable release
improvements. My goals for this discussion are to find ways to
get more community members/organizations using the official stable
releases, and also to help simplify the release process for users,
testers, and release managers.

* Ideas:

1. Longer support for stable release. We currently have 2 major and 2
stable releases per year. Each major release is effectively supported
for only 3 months, because after the x.y.1 release , we move on to the
next major version. Our current release schedule is approximately:

March: x.y.0
June x.y.1
Sept: (x+1).y.0
Dec: (x+1).y.1

One idea is to move to 6 releases per year and have an additional
x.y.2 release after (x+1).y.0, but before (x+1).y.0. This would
mean that stable releases are supported for ~ 8 months. So something
like:

Jan: (x-1).y.1
March: x.y.0
May: (x-1).y.2
July: x.y.1
Sept: (x+1).y.0
Nov: x.y.2

What do people think about this or other alternative release schedules?
Would this kind of extended support make stable releases more
attractive to users/organizations that use LLVM.

My main concern with adding an extra release is placing an extra
burden on release testers. For release testers is doing two
more releases per year practical?

2. Automated release testing. This is connected to the previous
idea. Can we automate the release testing? It would be great if
we had some release tester bots that would test the release
candidates and upload the binaries. It seems like this should
be doable within the current buildbot infrastructure. Is the
only thing that keeps us from doing this, lack of dedicated test
hardware?

+1 for the automation! This is key to me.

I’m not sure we need *dedicated* hardware for this: the release testing does not consume a lot of cycles continuously, does it? I expect we need some hardware for a limited period of time when preparing a release.

What’s missing in my opinion is rather the software infrastructure: an easy way to run a script and validate the compiler more extensively than the LLVM test-suite.
Maybe we should lead an effort towards an “extended test-suite”: having the ability to clone a bunch of selected open-source project and build & run their unitests. Even if it is something that takes a full day to run, it seems possible to do this on each RCs.

3. Automated merge requests. I would like to come up with a way
that we can tag commit messages so that a post-commit hook would
be able to automatically create merge requests in bugzilla.
There are a few ways I can think of to tag the messages, but I'm
just going to propose that adding the following line:

STABLE: 4.0

To commit messages would make it a candidate for an automated
merge request. Any thoughts?

I’m not a big fan of hijacking commit messages for this.
What does this by? Just saving running the script?
I rather have this as part of the scripts that will be available after we migrate to git.
For instance you can already commit from git with `git llvm push` (see Getting Started with the LLVM System — LLVM 16.0.0git documentation ), we can imagine extending this in the future to be able to send merge request easily from the command line.

8

Hi Tom,

First, kudos for the new merge scripts.

(...) I don't find the release testing itself to be
particularly burdensome. The hard part of releases is chasing down and
fixing bugs, but for .2 releases I expect the process to be pretty
smooth, basically building and testing a build with a few extra
patches on top of .1.

I agree with Hans. This is why dot releases cannot have any new
feature in it, no matter how small.

I think it's more a problem of finding someone to configure and
maintain the bots.

Yup.

Finding the available hardware that is upstream, outside firewalls and
ready for deployment as buildbots, either all year round or on/off
during releases, is going to be hard.

If/when we move to Jenkins, this could be easier, but right now, it's
pretty hard.

I'm not sure this is worth it. Aren't many of the "we should merge
this to stable" decisions made after the commit lands?

Again, agreed. The decision process should be a conscious one, and
with review from the release manager, code owner, etc.

Simple scripting won't go all the way to providing that auditing path
and could make release branches unstable.

Release testing isn't complicated enough to automate that part, at
least not yet.

If/when we have more tests beyond the test-suite, we'll have to
investigate each individual test on their own.

For now, involving distros and large projects directly is the easiest
way to test the release.

cheers,
--renato