[RFC] Helping release management

Hi,

I am sending this proposal to get feedbacks on how we could make the tagging of bug fixes and regressions more obvious. The idea is to provide easily accessible information to help deciding what to cherry-pick in a release branch.

* Context *

People shipping compilers based on LLVM may not completely align with the official releases of LLVM. Thus, the stabilization of each custom release may happen at different period of time. Because of that, release managers have to come up with their own strategy to decide which commits should be cherry-picked during the stabilization of their release branch.

For the official LLVM releases, people (committers, code owners, etc.) notice LLVM release managers that a given commit is worth pulling into the release. I would like to put in place something more systematic and that plays nicely with scripting and such that would extend this mechanism.

* Proposal *

1. Use [Fix] for commit related to bug fixes.
2. Add a description of the problem in the commit message to help answer the following questions:
- What is fixed?
- Which targets are impacted?
- What is required to trigger the bug? (I.e., how often the end users may encounter it.)
- When was the bug introduced?

#1 At the very least, I would like that each bug fix has a tag on the first line of the commit (i.e., what ends up in the subject line of the related email.) Something like [Fix] would do.
Thanks to that tag, it would be possible to easily filter bug fixes in email and other cherry-picking helper tools, I believe.

#2 Although it may be difficult to come up with that information, I believe it should be provided as the best of the committer knowledge. Indeed, this kind of information is useful to help release managers to ascertain how relevant is a change for their release and thus help them to decide whether to cherry-pick this change or not.

What do people think?

Thanks,
-Quentin

From: llvm-dev [mailto:llvm-dev-bounces@lists.llvm.org] On Behalf Of
Quentin Colombet via llvm-dev
Sent: Monday, May 02, 2016 1:35 PM
To: llvm-dev <llvm-dev@lists.llvm.org>
Subject: [llvm-dev] [RFC] Helping release management

Hi,

I am sending this proposal to get feedbacks on how we could make the
tagging of bug fixes and regressions more obvious. The idea is to provide
easily accessible information to help deciding what to cherry-pick in a release
branch.

* Context *

People shipping compilers based on LLVM may not completely align with the
official releases of LLVM. Thus, the stabilization of each custom release may
happen at different period of time. Because of that, release managers have
to come up with their own strategy to decide which commits should be
cherry-picked during the stabilization of their release branch.

For the official LLVM releases, people (committers, code owners, etc.) notice
LLVM release managers that a given commit is worth pulling into the release.
I would like to put in place something more systematic and that plays nicely
with scripting and such that would extend this mechanism.

This seems like a great idea to me.

* Proposal *

1. Use [Fix] for commit related to bug fixes.

a specific form would be best: [Fix] PR12345, or [FIX] Bug introduced in r56789.

2. Add a description of the problem in the commit message to help answer
the following questions:
- What is fixed?
- Which targets are impacted?
- What is required to trigger the bug? (I.e., how often the end users may
encounter it.)
- When was the bug introduced?

This is super important for things that regress, and then get fixed before a PR is even filed. Even for a PR if the problem was introduced in a known
revision, having that in the commit message helps to understand whether some other revision is really safe to pull into a release.

#1 At the very least, I would like that each bug fix has a tag on the first line of
the commit (i.e., what ends up in the subject line of the related email.)
Something like [Fix] would do.
Thanks to that tag, it would be possible to easily filter bug fixes in email and
other cherry-picking helper tools, I believe.

#2 Although it may be difficult to come up with that information, I believe it
should be provided as the best of the committer knowledge. Indeed, this
kind of information is useful to help release managers to ascertain how
relevant is a change for their release and thus help them to decide whether to
cherry-pick this change or not.

What do people think?

I like it.

The first line of the commit is interpreted as a "title" by "git based" tools.
I rather have a self-contained first line and avoid *any* external reference there.
(The commit message can be as long as you want).

1. Use [Fix] for commit related to bug fixes.

I'm not really such a big fan of this format, adds too much noise.

2. Add a description of the problem in the commit message to help answer the following questions:
- What is fixed?
- Which targets are impacted?
- What is required to trigger the bug? (I.e., how often the end users may encounter it.)
- When was the bug introduced?

...but this is how a commit message should look like anyway. Worst
examples in the past are references so some internal bug report systems
without actual description for the rest of the world.

Joerg

Hi Joerg,

1. Use [Fix] for commit related to bug fixes.

I'm not really such a big fan of this format, adds too much noise.

What alternatives do you have in mind?

2. Add a description of the problem in the commit message to help answer the following questions:
- What is fixed?
- Which targets are impacted?
- What is required to trigger the bug? (I.e., how often the end users may encounter it.)
- When was the bug introduced?

...but this is how a commit message should look like anyway.

Sure, but we lack a standardized way to highlight the commits that are fixes IMHO.
It is a pain to go through all the commit logs without a bit of guidance, even when the commit is well documented.

Also, if this is how it is supposed to look, maybe we could write it down as a recommendation somewhere :).

Worst
examples in the past are references so some internal bug report systems
without actual description for the rest of the world.

Agreed.

Cheers,
-Quentin

From: "Mehdi Amini via llvm-dev" <llvm-dev@lists.llvm.org>
To: "Kevin B Smith" <kevin.b.smith@intel.com>
Cc: llvm-dev@lists.llvm.org
Sent: Monday, May 2, 2016 4:26:30 PM
Subject: Re: [llvm-dev] [RFC] Helping release management

>
>
>
>> From: llvm-dev [mailto:llvm-dev-bounces@lists.llvm.org] On Behalf
>> Of
>> Quentin Colombet via llvm-dev
>> Sent: Monday, May 02, 2016 1:35 PM
>> To: llvm-dev <llvm-dev@lists.llvm.org>
>> Subject: [llvm-dev] [RFC] Helping release management
>>
>> Hi,
>>
>> I am sending this proposal to get feedbacks on how we could make
>> the
>> tagging of bug fixes and regressions more obvious. The idea is to
>> provide
>> easily accessible information to help deciding what to cherry-pick
>> in a release
>> branch.
>>
>> * Context *
>>
>> People shipping compilers based on LLVM may not completely align
>> with the
>> official releases of LLVM. Thus, the stabilization of each custom
>> release may
>> happen at different period of time. Because of that, release
>> managers have
>> to come up with their own strategy to decide which commits should
>> be
>> cherry-picked during the stabilization of their release branch.
>>
>> For the official LLVM releases, people (committers, code owners,
>> etc.) notice
>> LLVM release managers that a given commit is worth pulling into
>> the release.
>> I would like to put in place something more systematic and that
>> plays nicely
>> with scripting and such that would extend this mechanism.
>>
>
> This seems like a great idea to me.
>
>> * Proposal *
>>
>> 1. Use [Fix] for commit related to bug fixes.
>
> a specific form would be best: [Fix] PR12345, or [FIX] Bug
> introduced in r56789.

The first line of the commit is interpreted as a "title" by "git
based" tools.
I rather have a self-contained first line and avoid *any* external
reference there.
(The commit message can be as long as you want).

+1

The title of each commit should briefly describe what the commit does. If we'd like to prefix the title with [Fix] or similar to indicate commits primarily intended as bug fixes, that sounds useful to me, but not essential. All of the other metadata should appear later in the commit message - it is the presence of this metadata we can search for in the commit logs that will be important.

-Hal

If the goal is to help with release management, handle it similar to
phabricator etc, i.e. "MT3.8: yes" or so as data.

Joerg

Hi Joerg,

1. Use [Fix] for commit related to bug fixes.

I'm not really such a big fan of this format, adds too much noise.

What alternatives do you have in mind?

If the goal is to help with release management, handle it similar to
phabricator etc, i.e. "MT3.8: yes" or so as data.

This is a post commit thing, right?

I do not think people actively set properties (in the SVN sense) on their commits or are you referring to something else?

Just include it in the commit message. That's easy to process with tools
already.

Joerg

Hi Joerg,

  1. Use [Fix] for commit related to bug fixes.

I’m not really such a big fan of this format, adds too much noise.

What alternatives do you have in mind?

If the goal is to help with release management, handle it similar to
phabricator etc, i.e. “MT3.8: yes” or so as data.

This is a post commit thing, right?

I do not think people actively set properties (in the SVN sense) on their commits or are you referring to something else?

Just include it in the commit message. That’s easy to process with tools
already.

Oh ok, then you are suggesting a tagging mechanism like I was :).

I am sending this proposal to get feedbacks on how we could make the tagging of bug fixes and regressions more obvious. The idea is to provide easily accessible information to help deciding what to cherry-pick in a release branch.

* Context *

People shipping compilers based on LLVM may not completely align with the official releases of LLVM. Thus, the stabilization of each custom release may happen at different period of time. Because of that, release managers have to come up with their own strategy to decide which commits should be cherry-picked during the stabilization of their release branch.

(Unrelated to your proposal, I'm curious how common it is to base
releases of LLVM-based tools off the upstream release branches vs.
other revisions.)

For the official LLVM releases, people (committers, code owners, etc.) notice LLVM release managers that a given commit is worth pulling into the release. I would like to put in place something more systematic and that plays nicely with scripting and such that would extend this mechanism.

* Proposal *

1. Use [Fix] for commit related to bug fixes.

I think we're mostly pretty good at referencing PR's in commit
messages already. That's also easy to grep for, so maybe that's good
enough?

2. Add a description of the problem in the commit message to help answer the following questions:
- What is fixed?
- Which targets are impacted?
- What is required to trigger the bug? (I.e., how often the end users may encounter it.)
- When was the bug introduced?

This sounds like the kind of information that should be in a great
commit message anyways.

But I'm also thinking that maybe we could be better at using our bug
tracker? Whether a bug is a feature request, something that was always
broken, or a regression (and from what version), sounds like a perfect
fit for a bug tracker. Someone doing a release could then query the
Bugzilla to see e.g. what regression bugs were fixed in a certain time
span.

#1 At the very least, I would like that each bug fix has a tag on the first line of the commit (i.e., what ends up in the subject line of the related email.) Something like [Fix] would do.
Thanks to that tag, it would be possible to easily filter bug fixes in email and other cherry-picking helper tools, I believe.

If we really do want to make a guideline about this, I propose we
standardize on suffixing the first line of the commit with (PRnnn).

Cheers,
Hans

Hi Hans,

Since you are actively doing this kind of things, your feedbacks is particularly valuable.
Thanks!

I am sending this proposal to get feedbacks on how we could make the tagging of bug fixes and regressions more obvious. The idea is to provide easily accessible information to help deciding what to cherry-pick in a release branch.

* Context *

People shipping compilers based on LLVM may not completely align with the official releases of LLVM. Thus, the stabilization of each custom release may happen at different period of time. Because of that, release managers have to come up with their own strategy to decide which commits should be cherry-picked during the stabilization of their release branch.

(Unrelated to your proposal, I'm curious how common it is to base
releases of LLVM-based tools off the upstream release branches vs.
other revisions.)

For the official LLVM releases, people (committers, code owners, etc.) notice LLVM release managers that a given commit is worth pulling into the release. I would like to put in place something more systematic and that plays nicely with scripting and such that would extend this mechanism.

* Proposal *

1. Use [Fix] for commit related to bug fixes.

I think we're mostly pretty good at referencing PR's in commit
messages already. That's also easy to grep for, so maybe that's good
enough?

When a PR is available, that is certainly good enough.
I do not want to make filling a PR mandatory for each bug we fix though. Having a PR is great, but I can see why we may not want to create one each time we fix something.

If we do want to go into that direction though, I believe we would need to provide more support to make that easier. (E.g. filing via command line, getting a number back and feeding this number to a commit.)

2. Add a description of the problem in the commit message to help answer the following questions:
- What is fixed?
- Which targets are impacted?
- What is required to trigger the bug? (I.e., how often the end users may encounter it.)
- When was the bug introduced?

This sounds like the kind of information that should be in a great
commit message anyways.

But I'm also thinking that maybe we could be better at using our bug
tracker? Whether a bug is a feature request, something that was always
broken, or a regression (and from what version), sounds like a perfect
fit for a bug tracker. Someone doing a release could then query the
Bugzilla to see e.g. what regression bugs were fixed in a certain time
span.

Sounds great to me.

Would that kind of workflow ease your job for tracking what should be pulled into the release branch after you’ve branched?
What would be the best workflow for such task for you?

#1 At the very least, I would like that each bug fix has a tag on the first line of the commit (i.e., what ends up in the subject line of the related email.) Something like [Fix] would do.
Thanks to that tag, it would be possible to easily filter bug fixes in email and other cherry-picking helper tools, I believe.

If we really do want to make a guideline about this, I propose we
standardize on suffixing the first line of the commit with (PRnnn).

That would work for me.

Anywhere in the commit message would work as well (like Mehdi and Hal said), though I tend to prefer the first line (but Mehdi does not like it :)).

Cheers,
-Quentin

+1 to everything Hans said. We should standardize how we reference bugs from commits, and then get better about tagging the issues in ways that are useful to release managers.

We do both. We strongly prefer to base our releases on llvm.org releases, but y’all frequently don’t comply with our timetables. :slight_smile:

Jim Rowan
jmr@codeaurora.org
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by the Linux Foundation

It'd be great if whatever we choose to implement this has a way to add a reference to a PR / non-PR'ed bug after the commit is done. That way forgetting to add a PR tag to a commit message is not a big deal, and does not require a revert and recommit. A field in Diffusion that I can update manually (but gets autofilled if the commit message has "[FIX]" or "PRXXXX") will be perfect.

-- Sanjoy

1. Use [Fix] for commit related to bug fixes.

I think we're mostly pretty good at referencing PR's in commit
messages already. That's also easy to grep for, so maybe that's good
enough?

When a PR is available, that is certainly good enough.
I do not want to make filling a PR mandatory for each bug we fix though. Having a PR is great, but I can see why we may not want to create one each time we fix something.

At least for me, if I run into a problem that takes some work to
reduce, debug, etc., I find filing a PR worth it, if nothing else to
keep track of the work for my own sake.

For bugs which one just stumbles across in the code and which can be
fixed with a quick patch, I agree that filing a bug doesn't make
sense.

I suspect the kind of fixes that are important for a release branch
are mostly of the former kind, though.

2. Add a description of the problem in the commit message to help answer the following questions:
- What is fixed?
- Which targets are impacted?
- What is required to trigger the bug? (I.e., how often the end users may encounter it.)
- When was the bug introduced?

This sounds like the kind of information that should be in a great
commit message anyways.

But I'm also thinking that maybe we could be better at using our bug
tracker? Whether a bug is a feature request, something that was always
broken, or a regression (and from what version), sounds like a perfect
fit for a bug tracker. Someone doing a release could then query the
Bugzilla to see e.g. what regression bugs were fixed in a certain time
span.

Sounds great to me.

Would that kind of workflow ease your job for tracking what should be pulled into the release branch after you’ve branched?
What would be the best workflow for such task for you?

My ideal workflow for discovering commits suitable for merging is when
folks cc me on the commit message, because then I don't have to do any
work :slight_smile:

But besides that, I do rely on the bug tracker quite a lot. It's hard
to keep up with the commits list, but keeping up with filed and fixed
bugs is doable, and a good source of information. It's usually easy to
tell from a PR whether it's serious or not, and most people are good
about indicating what version they used, if it started failing
recently, what target they're using, etc.

Cheers,
Hans

I’m with Hans that we should use bugzilla/phab better. It’s far easier to update bugs than format commit messages to conform to free text parseable content.

I don’t think we should restrict the commit message more than it is, nor that we should reprimand / block / revert patches because of downstream release processes. I also don’t think we should change everyone’s process to fit to everyone’s else’s process. That doesn’t scale.

I personally want to see all downstream releases participating on the upstream release process, and all their release managers actively engaged in the public process. Anything else will mean additional cost to us all, and go against the open source nature of the project.

My tuppence.
Renato

Note that you already get clickable links in phabricator and when viewing commit messages (depending on the terminal or graphical git client) simply by refering to bugreports with an URL instead of just the number:

http://llvm.org/PR1234

  • Matthias

+1 to everything Hans said. We should standardize how we reference bugs from
commits, and then get better about tagging the issues in ways that are
useful to release managers.

We have a standard:

http://llvm.org/docs/DeveloperPolicy.html#commit-messages

"If the patch fixes a bug in bugzilla, please include the PR# in the message."

Also, we don't want to control how people write commit messages. There
was a long discussion when we wrote that document, and the consensus
was:

"For minor violations of these recommendations, the community normally
favors reminding the contributor of this policy over reverting. Minor
corrections and omissions can be handled by sending a reply to the
commits mailing list."

Meaning any parsing you may want to do on commit messages will be
broken by the fact that not all of them will be strictly following our
not-strict guidelines.

I like the idea of a custom phabricator field, so we can make the references
clickable from reviews.

Customizing Phab is indeed a good idea, but one that is still not a
sure way to get info from. Nor is Bugzilla.

Until now, the mechanism we're using to back-port patches to stable
releases has been for developers to recommend a patch and for the
code-owner to approve (and merge) the back-port. I agree this doesn't
scale to multiple release schedules, but I don't want back-ports going
in without oversight, nor I think we can consolidate all release
management into a tool that can easily go offline for extended
periods, thus unnecessarily delaying the release process.

I think it's perfectly fine for downstream release managers to propose
patches to the upstream release, and then pull their changes from the
release branch, because that will be validated. I also think we could
do more stable releases a year, since less validation is needed for
each incremental one (most tests are automated in test-release.sh) and
the downstream releases will provide us with a further level of
quality assurance that we don't have today.

If we all work together, it's less work for everyone. If we require
that everyone abide to some complicated and obscure rules on commit
messages and tooling usage, we'll make it harder for everyone else.

cheers,
--renato

Sony has (almost) always built releases on top of upstream formal releases. The timing means we carry a release branch for kind of a long time, which is inconvenient, but there are some compensating advantages.

We have thought about releasing based on something more current, but we have some internal processes to rework before we can make that viable.

–paulr