1 Week Before 2.0 Branch Creation

LLVM Developers,

It is now 1 week before I will create the 2.0 release branch.

I'm asking that all platform maintainers and available llvm developers
review the nightly tester results. Please XFAIL any dejagnu tests that are
currently failing, fix any warnings, and review the results of the full
llvm test suite. If a test failure is determined to be something that
needs to be fixed before the release, please fix it or file a bugzilla
bug, and set the target to 2.0.

Any enhancements scheduled for the 2.0 release should go in ASAP. This
work should be completed before May 7th. Please try to avoid checking
in large changes right before the deadline.

I will be creating the 2.0 release branch on May 7th at 9:00PM PST.

Reminder of Release Schedule:

* April 30th - 1 week before code freeze warning email.
* May 7th - Code freeze and branch creation.
* May 14th - Pre-releases available for additional testing. All
documentation changes committed to branch and mainline.
* May 21st - Release.

If you have any questions or concerns, please let me know.

-Tanya

Tanya M. Lattner wrote:

I'm asking that all platform maintainers and available llvm developers
review the nightly tester results. Please XFAIL any dejagnu tests that are
currently failing, fix any warnings, and review the results of the full
llvm test suite. If a test failure is determined to be something that
needs to be fixed before the release, please fix it or file a bugzilla
bug, and set the target to 2.0.

Any enhancements scheduled for the 2.0 release should go in ASAP. This
work should be completed before May 7th. Please try to avoid checking
in large changes right before the deadline.

I very much want to get my regalloc refactoring changes in but I don't
have a consistent platform to test them on. Someone broke llvm-gcc
bootstrapping late last week or this week.

Can we please freeze features for a while and get some stability in
the tree before we branch 2.0? I really need to test these patches
and it seems like the tree has been broken every other day for a week.

                                    -Dave

I very much want to get my regalloc refactoring changes in but I don't
have a consistent platform to test them on. Someone broke llvm-gcc
bootstrapping late last week or this week.

How large of a change have you made? With 3 days before the branch creation, I strongly advise people not to be checking in major changes.

Can we please freeze features for a while and get some stability in
the tree before we branch 2.0? I really need to test these patches
and it seems like the tree has been broken every other day for a week.

This week has been a bit crazy with people checking in their changes for 2.0. When checking in, people are suppose to be running the llvm-test suite and regression tests. Unfortunately, this doesn't always seem to happen and its hard to control. I know its not a great solution, but you can do preliminary tests with the last stable version of cvs that you (or check the test results to find that date).

Creating the branch is a way of freezing the code. If the tree isn't stable on Monday, I will have to make a decision on how to proceed (ie. backing out changes).

We may need to change our proceedures for releases in the future. This is how we have done it in the past with no problem, but LLVM is growing much more rapidly now.

-Tanya

Tanya M. Lattner wrote:

How large of a change have you made? With 3 days before the branch creation, I strongly advise people not to be checking in major changes.

Depends how you look at it. Structurally, it separates two files into
four and moves some functionality from one class to a new class, so in a
sense that's a big change. Code-logic-wise, it does nothing at all. I
will send the patch to the commits list today. Hopefully someone can
look at it and decide whether to apply it.

> We may need to change our proceedures for releases in the future.
> This is how we have done it in the past with no problem, but LLVM is
> growing much more rapidly now.

In my experience, a code freeze lasts for a fair amount of time (on the
order of months). The way I've seen it done in many projects is that
there's a deadline to get all new feature work in (with more than a
week's notice!). Then the new branch is created. The next two or three
months, only bugfixes are allowed on the release branch. Some projects
close the development branch to force bugs to be fixed first, while
others run two branches in parallel. I would lean toward the latter and
trust people to be responsible enough to fix bugs before release.

The release is done when there are no new regressions and all tests
created for new features pass. Of course, this means that folks
should be creating tests for their features.

Do we want some kind of discussion about what this process should be
followed by a formal proposal drafted by a few people for comment and
possible adoption?

                                 -Dave

Tanya M. Lattner wrote:

How large of a change have you made? With 3 days before the branch
creation, I strongly advise people not to be checking in major changes.

Depends how you look at it. Structurally, it separates two files into
four and moves some functionality from one class to a new class, so in a
sense that's a big change. Code-logic-wise, it does nothing at all. I
will send the patch to the commits list today. Hopefully someone can
look at it and decide whether to apply it.

> We may need to change our proceedures for releases in the future.
> This is how we have done it in the past with no problem, but LLVM is
> growing much more rapidly now.

In my experience, a code freeze lasts for a fair amount of time (on the
order of months). The way I've seen it done in many projects is that
there's a deadline to get all new feature work in (with more than a
week's notice!). Then the new branch is created. The next two or three
months, only bugfixes are allowed on the release branch. Some projects
close the development branch to force bugs to be fixed first, while
others run two branches in parallel. I would lean toward the latter and
trust people to be responsible enough to fix bugs before release.

The release is done when there are no new regressions and all tests
created for new features pass. Of course, this means that folks
should be creating tests for their features.

Do we want some kind of discussion about what this process should be
followed by a formal proposal drafted by a few people for comment and
possible adoption?

It would be good to have a mailing list for test results where 'make check' results could be posted so that there is some reference and people could avoid repeating builds.

Aaron

Aaron Gray wrote:

It would be good to have a mailing list for test results where 'make check' results could be posted so that there is some reference and people could avoid repeating builds.

There's the llvm-testresults list, but I find it less than fully useful
because it's not immediately obvious from scanning message subjects if
there's been a test failure. It's a lot of messages to wade through and
read to get this information.

What about a Tinderbox-like setup where we could consult a web page to
see the current status of the repository? Boost has a nice setup:

http://engineering.meta-comm.com/boost-regression/CVS-RC_1_34_0/developer/index_release.html

It's probably more complex than what we need. Maybe we just need a
page grouping each test under it's suite and marking the result on
each architecture. Something like this:

<begin crappy ASCII art>

llvm-testresults :slight_smile:

-Chris

Great, feeling silly, I'll signon to that then :slight_smile:

Aaron

Shall attach results or put them inline ?

Aaron

Aaron Gray wrote:

Aaron Gray wrote:

It would be good to have a mailing list for test results where 'make
check'
results could be posted so that there is some reference and people could
avoid repeating builds.

llvm-testresults :slight_smile:

Great, feeling silly, I'll signon to that then :slight_smile:

Shall attach results or put them inline ?

Aaron

What you see on llvm-testresults are posted automatically by running the
nightly tester.

Thanks, yes I gathered that, just signed on to the list so its newish territory.

Aaron

Tanya M. Lattner wrote:

How large of a change have you made? With 3 days before the branch
creation, I strongly advise people not to be checking in major changes.

Depends how you look at it. Structurally, it separates two files into
four and moves some functionality from one class to a new class, so in a
sense that's a big change. Code-logic-wise, it does nothing at all. I
will send the patch to the commits list today. Hopefully someone can
look at it and decide whether to apply it.

This sounds like great stuff, and is definitely progress towards a better RA. However, it doesn't provide any user-visible feature, so I think it makes sense to apply this immediately after 2.0 branches, just to reduce risk.

> We may need to change our proceedures for releases in the future.
> This is how we have done it in the past with no problem, but LLVM is
> growing much more rapidly now.

The way I've seen it done in many projects is that
there's a deadline to get all new feature work in (with more than a
week's notice!). Then the new branch is created.

To be fair, this has been the plan for quite some time:
http://lists.cs.uiuc.edu/pipermail/llvmdev/2007-March/008322.html

We should probably find a way to publicize this better on the web page. As an aside, the web page definitely needs a major overhaul :slight_smile:

In my experience, a code freeze lasts for a fair amount of time (on the order of months). The next two or three months, only bugfixes are allowed on the release branch. Some projects close the development branch to force bugs to be fixed first, while others run two branches in parallel. I would lean toward the latter and trust people to be responsible enough to fix bugs before release.

Right, this is a very common approach, used by many projects.

For LLVM, we have traditionally taken a different approach: keep mainline as stable as possible at all times, then make the release branch a short cycle that only accepts *critical* bug fixes (even things like miscompilations or compile crashes aren't taken if they are at all risky).

The idea is that we always keep mainline CVS close to release quality. This is good for everyone doing daily development of course. In practice, minor regressions occasionally slip in during the development cycle, so the tree isn't *always* happy. However, these problems are usually "shallow" problems that are easily fixed, they don't require major new rearchitecture, etc.

One major difference between LLVM and some other projects is that we usually promptly fix incoming bug reports (particularly regressions) as they happen, on mainline. There is no need to wait for a release branch for this.

In general, branches are an extremely expensive way to spend human time. I prefer to keep branches (release and otherwise) as minimal and short-lived as possible. This is one reason that we almost never use branches for mainline development, even for large projects: we find a way to break the project down into incremental pieces, which are independently easy to test and are less scary than the big rewrite :slight_smile:

The release is done when there are no new regressions and all tests
created for new features pass. Of course, this means that folks
should be creating tests for their features.

Absolutely. We definitely do this today. In general, I think that the LLVM project has done an excellent job of making each release monotonically better than the previous one. LLVM 2.0 has been used to compile *large* quanitites of source code. While it contains a large number of new features, I am quite confident it will be our best release yet.

Do we want some kind of discussion about what this process should be
followed by a formal proposal drafted by a few people for comment and
possible adoption?

Sure, I think that makes sense.

-Chris

Either way, just don't rot13 them or something :slight_smile:

-Chris

There's the llvm-testresults list, but I find it less than fully useful
because it's not immediately obvious from scanning message subjects if
there's been a test failure. It's a lot of messages to wade through and
read to get this information.

Right.

What about a Tinderbox-like setup where we could consult a web page to
see the current status of the repository? Boost has a nice setup:

http://engineering.meta-comm.com/boost-regression/CVS-RC_1_34_0/developer/index_release.html

It's probably more complex than what we need. Maybe we just need a
page grouping each test under it's suite and marking the result on
each architecture. Something like this:

I think that this is a great idea. However, instead of picking up yet another setting of testing infrastructure, I think we should make what we have already (the nightly testers) better. In particular, the main page of the tester:
http://llvm.org/nightlytest/

Already captures a lot of this: it tells you the number of unexpected failures, whether or not the build succeeded etc. You can even drill down to a specific machine, e.g.:
http://llvm.org/nightlytest/machine.php?machine=120

I see several problems with this, all of which are solvable:

1. The tester script doesn't update and rebuild the CFE, so often you get
    failures due to an out-of-date CFE.
2. There is no way to group by architecture, target-triple, etc. 3. Minor stupid stuff: on the per-machine page, you can click on the test
    failures number and go to the list of unexpected failures, but you
    can't do that on the main page.

-Chris

How large of a change have you made? With 3 days before the branch
creation, I strongly advise people not to be checking in major changes.

Depends how you look at it. Structurally, it separates two files into
four and moves some functionality from one class to a new class, so in a
sense that's a big change. Code-logic-wise, it does nothing at all. I
will send the patch to the commits list today. Hopefully someone can
look at it and decide whether to apply it.

While its not necessarily a large functionality change, its probably too much to review with this notice. Chris would have final say on this, but in general its bit close to the deadline.

> We may need to change our proceedures for releases in the future.
> This is how we have done it in the past with no problem, but LLVM is
> growing much more rapidly now.

In my experience, a code freeze lasts for a fair amount of time (on the
order of months). The way I've seen it done in many projects is that
there's a deadline to get all new feature work in (with more than a
week's notice!). Then the new branch is created. The next two or three
months, only bugfixes are allowed on the release branch. Some projects
close the development branch to force bugs to be fixed first, while
others run two branches in parallel. I would lean toward the latter and
trust people to be responsible enough to fix bugs before release.

In my defense, there has been more than one email about the release schedule. In the future, I will send them out more often and also find a good place to put this on our webpage (I've been meaning to overhaul it, but havent had time).

The release is done when there are no new regressions and all tests
created for new features pass. Of course, this means that folks
should be creating tests for their features.

This is the case today. I verify on our major platforms that there are no regressions from the previous release. Older releases may not have been as strict with this requirement (pre 1.7 I was not the release manager). Regressions should be fixed and we do delay the release if necessary.

Do we want some kind of discussion about what this process should be
followed by a formal proposal drafted by a few people for comment and
possible adoption?

You should be aware of this:
http://llvm.org/docs/HowToReleaseLLVM.html

Some of the process is not documented, but I always send emails about the next steps. You can look back in the mailing archives to see how we have done this is in the past.

Since this is my domain, I will draft something up. However, nothing will change for this release.

-Tanya

It would be good to have a mailing list for test results where 'make check'
results could be posted so that there is some reference and people could
avoid repeating builds.

There's the llvm-testresults list, but I find it less than fully useful
because it's not immediately obvious from scanning message subjects if
there's been a test failure. It's a lot of messages to wade through and
read to get this information.

What about a Tinderbox-like setup where we could consult a web page to
see the current status of the repository? Boost has a nice setup:

http://engineering.meta-comm.com/boost-regression/CVS-RC_1_34_0/developer/index_release.html

It's probably more complex than what we need. Maybe we just need a
page grouping each test under it's suite and marking the result on
each architecture. Something like this:

I agree the test results are not displayed well. Its not well known, but I have been redesigning our databases and scripts to enable better display of the data. Right now its not very easy to do what you describe.

-Tanya

Hi Chris,

I think that this is a great idea. However, instead of picking up yet
another setting of testing infrastructure, I think we should make what
we have already (the nightly testers) better. In particular, the main
page of the tester:
http://llvm.org/nightlytest/

Already captures a lot of this: it tells you the number of unexpected
failures, whether or not the build succeeded etc. You can even drill
down to a specific machine, e.g.:
http://llvm.org/nightlytest/machine.php?machine=120

I see several problems with this, all of which are solvable:

1. The tester script doesn't update and rebuild the CFE, so often you
get failures due to an out-of-date CFE.
2. There is no way to group by architecture, target-triple, etc.
3. Minor stupid stuff: on the per-machine page, you can click on the
test failures number and go to the list of unexpected failures, but
you can't do that on the main page.

Are you aware of buildbot? It's quite widely used and flexible.

    http://buildbot.sourceforge.net/

    "By automatically rebuilding and testing the tree each time
    something has changed, build problems are pinpointed quickly, before
    other developers are inconvenienced by the failure. The guilty
    developer can be **identified and harassed without human
    intervention**. By running the builds on a variety of platforms,
    developers who do not have the facilities to test their changes
    everywhere before checkin will at least know shortly afterwards
    whether they have broken the build or not. Warning counts, lint
    checks, image size, compile time, and other build parameters can be
    tracked over time, are more visible, and are therefore easier to
    improve."

    ** -- my emphasis.

I'd suggest at least one machine given over to always building whenever
anything changes, and have the other nightly volunteers as now. Using
just nightly volunteers isn't great because breakage is noticed too late
and can affect too many people by then.

An example of the output is http://www.python.org/dev/buildbot/trunk/

Cheers,

Ralph.

That looks nifty. If someone wanted to set it up on their machine, we could give it a try and see how it works.

-Chris

I don’t know if others would be interested as well, but I should be able to set it up at least on one of my machines over the summer, and on a continuing basis if its found useful. Should be at least useful to me with the work I’ll be doing on LLVM over the summer.

(NB: a speedy x86-64 linux box currently)

-Chandler

I’d definitely be interested in seeing this happen. If you get it working, I might be able to contribute a build slave as well.

–Owen