RFC: New Automated Release Workflow (using Issues and Pull Requests)

tstellar · December 17, 2021, 9:15pm

Hi,

Here is a proposal for a new automated workflow for managing parts of the release
process. I've been experimenting with this over the past few releases and
now that we have migrated to GitHub issues, it would be possible for us to
implement this in the main repo.

The workflow is pretty straight forward, but it does use pull requests. My
idea is to enable pull requests for only this automated workflow and not
for general development (i.e. We would still use Phabricator for code review).
Let me know what you think about this:

# Workflow

* On an existing issue or a newly created issue, a user who wants to backport
one or more commits to the release branch adds a comment:

/cherry-pick <commit_sha> <..>

* This starts a GitHub Action job that attempts to cherry-pick the commit(s)
to the current release branch.

* If the commit(s) can be cherry-picked cleanly, then the GitHub Action:
     * Pushes the result of the cherry-pick to a branch in the
       llvmbot/llvm-project repo called issue<n>, where n is the number of the
       GitHub Issue that launched the Action.

* Adds this comment on the issue: /branch llvmbot/llvm-project/issue<n>

* Creates a pull request from llvmbot/llvm-project/issue<n> to
llvm/llvm-project/release/XX.x

* Adds a comment on the issue: /pull-request #<n>
where n is the number of the pull request.

* If the commit(s) can't be cherry-picked cleanly, then the GitHub Action job adds
the release:cherry-pick-failed label to the issue and adds a comment:
"Failed to cherry-pick <commit_sha> <..>" along with a link to the failing
Action.

* If a user has manually cherry-picked the fixes, resolved the conflicts, and
pushed the result to a branch on github, they can automatically create a pull
request by adding this comment to an issue: /branch <user>/<repo>/<branch>

* Once a pull request has been created, this launches more GitHub Actions
to run pre-commit tests.

* Once the tests complete successfully and the changes have been approved
by the release manager, the pull request can me merged into the release branch.

* After the pull request is merged, a GitHub Action automatically closes the
associated issue.

Some Examples:

Cherry-pick success: https://github.com/tstellar/llvm-project/issues/729
Cherry-pick failure: https://github.com/tstellar/llvm-project/issues/730
Manual Branch comment: https://github.com/tstellar/llvm-project/issues/710

# Motivation

Why do this? The goal is to make the release process more efficient and transparent.
With this new workflow, users can get automatic and immediate feedback when a commit
they want backported doesn't apply cleanly or introduces some test failures. With
the current process, these kinds of issues are communicated by the release manager,
and it can be days or even weeks before a problem is discovered and communicated back
to the users.

Another advantage of this workflow is it introduces pre-commit CI to the release branch,
which is important for the stability of the branch and the releases, but also gives
the project an opportunity to experiment with new CI workflows in a way that
does not disrupt development on the main branch.

# Implementation

If this proposal is accepted, I would plan to implement this for the LLVM 14 release cycle based
on the following proof of concept that I have been testing for the last few releases:

https://github.com/tstellar/llvm-project/blob/release-automation/.github/workflows/release-workflow.yml

github.com

tstellar/llvm-project/blob/release-automation/.github/workflows/release-merge-pr.yml

name: Release Backport - Merge Pull Request

on:
  pull_request:
    types: [closed]

jobs:
  update-linked-issue:
    runs-on: ubuntu-latest
      #if: github.event.pull_request.merged

    steps:
      - name: Install PyGithub
        run: pip install PyGithub

      - name: Update Isssue
        shell: python
        run: |
          import github
          import re

This file has been truncated. show original

Thanks,
Tom

mehdi_amini · December 17, 2021, 9:25pm

Hi,

Hi,

Here is a proposal for a new automated workflow for managing parts of the release
process. I’ve been experimenting with this over the past few releases and
now that we have migrated to GitHub issues, it would be possible for us to
implement this in the main repo.

The workflow is pretty straight forward, but it does use pull requests. My
idea is to enable pull requests for only this automated workflow and not
for general development (i.e. We would still use Phabricator for code review).
Let me know what you think about this:

Workflow

On an existing issue or a newly created issue, a user who wants to backport
one or more commits to the release branch adds a comment:

/cherry-pick <commit_sha> <…>

This starts a GitHub Action job that attempts to cherry-pick the commit(s)
to the current release branch.

If the commit(s) can be cherry-picked cleanly, then the GitHub Action:

Pushes the result of the cherry-pick to a branch in the
llvmbot/llvm-project repo called issue, where n is the number of the
GitHub Issue that launched the Action.

Adds this comment on the issue: /branch llvmbot/llvm-project/issue

Creates a pull request from llvmbot/llvm-project/issue to
llvm/llvm-project/release/XX.x

Adds a comment on the issue: /pull-request #
where n is the number of the pull request.

If the commit(s) can’t be cherry-picked cleanly, then the GitHub Action job adds
the release:cherry-pick-failed label to the issue and adds a comment:
“Failed to cherry-pick <commit_sha> <…>” along with a link to the failing
Action.

If a user has manually cherry-picked the fixes, resolved the conflicts, and
pushed the result to a branch on github, they can automatically create a pull
request by adding this comment to an issue: /branch //

Once a pull request has been created, this launches more GitHub Actions
to run pre-commit tests.

Once the tests complete successfully and the changes have been approved
by the release manager, the pull request can me merged into the release branch.

After the pull request is merged, a GitHub Action automatically closes the
associated issue.

Some Examples:

Cherry-pick success: Merge eeb4266f8137c232f0f218a727dd12b5d4f52adc into 13.0.1 · Issue #729 · tstellar/llvm-project · GitHub
Cherry-pick failure: https://github.com/tstellar/llvm-project/issues/730
Manual Branch comment: https://github.com/tstellar/llvm-project/issues/710

Since your workflow can trigger actions from comments in the issues, why do you need pull-requests at all? Can’t you trigger the pre-merge testing action on the branch from the issue? Then you can “approve” it with a /merge LGTM comment in the issue directly and let the action merge it for example.

tstellar · December 17, 2021, 9:52pm

Hi,

    Hi,

    Here is a proposal for a new automated workflow for managing parts of the release
    process. I've been experimenting with this over the past few releases and
    now that we have migrated to GitHub issues, it would be possible for us to
    implement this in the main repo.

    The workflow is pretty straight forward, but it does use pull requests. My
    idea is to enable pull requests for only this automated workflow and not
    for general development (i.e. We would still use Phabricator for code review).
    Let me know what you think about this:

    # Workflow

    * On an existing issue or a newly created issue, a user who wants to backport
    one or more commits to the release branch adds a comment:

    /cherry-pick <commit_sha> <..>

    * This starts a GitHub Action job that attempts to cherry-pick the commit(s)
    to the current release branch.

    * If the commit(s) can be cherry-picked cleanly, then the GitHub Action:
      * Pushes the result of the cherry-pick to a branch in the
      llvmbot/llvm-project repo called issue<n>, where n is the number of the
      GitHub Issue that launched the Action.

      * Adds this comment on the issue: /branch llvmbot/llvm-project/issue<n>

      * Creates a pull request from llvmbot/llvm-project/issue<n> to
      llvm/llvm-project/release/XX.x

      * Adds a comment on the issue: /pull-request #<n>
      where n is the number of the pull request.

    * If the commit(s) can't be cherry-picked cleanly, then the GitHub Action job adds
    the release:cherry-pick-failed label to the issue and adds a comment:
    "Failed to cherry-pick <commit_sha> <..>" along with a link to the failing
    Action.

    * If a user has manually cherry-picked the fixes, resolved the conflicts, and
    pushed the result to a branch on github, they can automatically create a pull
    request by adding this comment to an issue: /branch <user>/<repo>/<branch>

    * Once a pull request has been created, this launches more GitHub Actions
    to run pre-commit tests.

    * Once the tests complete successfully and the changes have been approved
    by the release manager, the pull request can me merged into the release branch.

    * After the pull request is merged, a GitHub Action automatically closes the
    associated issue.

    Some Examples:

    Cherry-pick success: Merge eeb4266f8137c232f0f218a727dd12b5d4f52adc into 13.0.1 · Issue #729 · tstellar/llvm-project · GitHub
    Cherry-pick <Issues · tstellar/llvm-project · GitHub; failure: Backport 885fb9a257faa14ec33ad972df81801483c85da2 · Issue #730 · tstellar/llvm-project · GitHub
    Manual Branch comment: Assertion 'Symbol' failed with -fpatchable-function-entry=2 + -Oz AArch64 · Issue #710 · tstellar/llvm-project · GitHub

Since your workflow can trigger actions from comments in the issues, why do you need pull-requests at all? Can't you trigger the pre-merge testing action on the branch from the issue? Then you can "approve" it with a //merge LGTM/ comment in the issue directly and let the action merge it for example.

Yes, it would be possible to emulate pull request features with GitHub Actions. You
would also have to implement some kind of reporting mechanism to report results
back to the issue. I personally don't think it would be worth the effort to
do a lot of extra work to get what pull requests give us for free (someone else
would be welcome to implement this if they wanted).

If we did decide we don't want to use Pull Requests in the main repo for this,
I think the alternatives would be to use Pull Requests in the llvmbot
repo, or just drop this part of the proposal (in which case I would go back
to using Pull Request in my personal account for testing).

-Tom

mehdi_amini · December 17, 2021, 10:16pm

Hi,

Hi,

Here is a proposal for a new automated workflow for managing parts of the release
process. I’ve been experimenting with this over the past few releases and
now that we have migrated to GitHub issues, it would be possible for us to
implement this in the main repo.

The workflow is pretty straight forward, but it does use pull requests. My
idea is to enable pull requests for only this automated workflow and not
for general development (i.e. We would still use Phabricator for code review).
Let me know what you think about this:

Workflow

On an existing issue or a newly created issue, a user who wants to backport
one or more commits to the release branch adds a comment:

/cherry-pick <commit_sha> <…>

This starts a GitHub Action job that attempts to cherry-pick the commit(s)
to the current release branch.

If the commit(s) can be cherry-picked cleanly, then the GitHub Action:

Pushes the result of the cherry-pick to a branch in the
llvmbot/llvm-project repo called issue, where n is the number of the
GitHub Issue that launched the Action.

Adds this comment on the issue: /branch llvmbot/llvm-project/issue

Creates a pull request from llvmbot/llvm-project/issue to
llvm/llvm-project/release/XX.x

Adds a comment on the issue: /pull-request #
where n is the number of the pull request.

If the commit(s) can’t be cherry-picked cleanly, then the GitHub Action job adds
the release:cherry-pick-failed label to the issue and adds a comment:
“Failed to cherry-pick <commit_sha> <…>” along with a link to the failing
Action.

If a user has manually cherry-picked the fixes, resolved the conflicts, and
pushed the result to a branch on github, they can automatically create a pull
request by adding this comment to an issue: /branch //

Once a pull request has been created, this launches more GitHub Actions
to run pre-commit tests.

Once the tests complete successfully and the changes have been approved
by the release manager, the pull request can me merged into the release branch.

After the pull request is merged, a GitHub Action automatically closes the
associated issue.

Some Examples:

Cherry-pick success: https://github.com/tstellar/llvm-project/issues/729
Cherry-pick <https://github.com/tstellar/llvm-project/issues/729Cherry-pick> failure: https://github.com/tstellar/llvm-project/issues/730 <https://github.com/tstellar/llvm-project/issues/730>
Manual Branch comment: https://github.com/tstellar/llvm-project/issues/710 <https://github.com/tstellar/llvm-project/issues/710>

Since your workflow can trigger actions from comments in the issues, why do you need pull-requests at all? Can’t you trigger the pre-merge testing action on the branch from the issue? Then you can “approve” it with a //merge LGTM/ comment in the issue directly and let the action merge it for example.

Yes, it would be possible to emulate pull request features with GitHub Actions. You
would also have to implement some kind of reporting mechanism to report results
back to the issue. I personally don’t think it would be worth the effort to
do a lot of extra work to get what pull requests give us for free (someone else
would be welcome to implement this if they wanted).

My main worry right now is to enable pull-request on the repo before we manage to figure out the workflow with GitHub issues, and in particular the story about how to manage notifications at the repo level.

If we did decide we don’t want to use Pull Requests in the main repo for this,
I think the alternatives would be to use Pull Requests in the llvmbot
repo, or just drop this part of the proposal (in which case I would go back
to using Pull Request in my personal account for testing).

Using pull-requests on another mirror repo (even create a “release-testing” repo for this) would at least segregate all this from the monorepo, which would address my current issues with the whole “subscribed to all activity or lose too much” that we have with GitHub.

dblaikie · December 18, 2021, 12:47am

Sounds pretty good to me - wouldn’t mind knowing more about/a good summary of the effects of this on project/repo/etc notifications that Mehdi’s mentioning. (be good to have a write up of the expected impact/options to then discuss - from the thread so far I understand some general/high level concerns, but it’s not clear to me exactly how it plays out)

tstellar · December 18, 2021, 2:38am

Sounds pretty good to me - wouldn't mind knowing more about/a good summary of the effects of this on project/repo/etc notifications that Mehdi's mentioning. (be good to have a write up of the expected impact/options to then discuss - from the thread so far I understand some general/high level concerns, but it's not clear to me exactly how it plays out)

The impact is really going to depend on the person and what notification preferences they
have/want. If you are already watching the repo with the default settings, then you probably
won't notice much of a difference given the current volume of notifications.

If people want to give their notification preferences, I can try to look at how
this change will impact specific configurations.

-Tom

dblaikie · December 18, 2021, 11:04pm

Sounds pretty good to me - wouldn’t mind knowing more about/a good summary of the effects of this on project/repo/etc notifications that Mehdi’s mentioning. (be good to have a write up of the expected impact/options to then discuss - from the thread so far I understand some general/high level concerns, but it’s not clear to me exactly how it plays out)

The impact is really going to depend on the person and what notification preferences they
have/want. If you are already watching the repo with the default settings, then you probably
won’t notice much of a difference given the current volume of notifications.

I think I’m on the default settings - which does currently mean a notification for every issue update, which is a lot. Given that llvm-bugs@email.llvm.org has been re-enabled, sending mail only on issue creation, I & others might opt back in to that behavior by disabling the baseline “notify on everything” to “notify only on issues I’m mentioned in”.

I guess currently the only email that github is generating is one email per issue update. We don’t have any pull requests, so there aren’t any emails for that, yeah?

So this new strategy might add a few more back-and-forth on each cherrypick issue (for those using llvm-bugs & disabling general issue notifications, this will not be relevant to them - there won’t be more issues created, just more comments on existing issues). But there will be some more emails generated related to the pull requests themselves, I guess? So each cherrypick goes from 2 emails to llvm-bugs (the issue creation and closure) to, how many? 4 (2 for llvm-bugs and I guess at least 2 for the pull request - one to make the request and one to close it - maybe a couple more status ones along the way?)

If people want to give their notification preferences, I can try to look at how
this change will impact specific configurations.

@Mehdi AMINI - are there particular scenarios you have in mind that’d be good to work through?

mehdi_amini · December 18, 2021, 11:47pm

It’s more the unknown that I’m cautious about here. In particular the recent experience with the issues’ notifications lend me to think we likely should have more thorough consideration to how things are fitting as a whole before adding more things on GitHub. You mentioned above how emails are issued for these new pull-requests, so I’d like to see a better characterization of this (quantity and example of such emails) as well as some documentation (what are the developer workflows, how to manage notifications configuration on GitHub, and client-side email filtering, etc.).

Using a separate repo if possible alleviates these constraints by sharding things early, which seems to fit better into how GitHub organizes things: a repo seems a really a unit of itself, and everything seems “scoped” at this level (notification, permissions, etc.).

tstellar · December 20, 2021, 5:16pm

     > Sounds pretty good to me - wouldn't mind knowing more about/a good summary of the effects of this on project/repo/etc notifications that Mehdi's mentioning. (be good to have a write up of the expected impact/options to then discuss - from the thread so far I understand some general/high level concerns, but it's not clear to me exactly how it plays out)
     >

    The impact is really going to depend on the person and what notification preferences they
    have/want. If you are already watching the repo with the default settings, then you probably
    won't notice much of a difference given the current volume of notifications.

I think I'm on the default settings - which does currently mean a notification for every issue update, which is a lot. Given that llvm-bugs@email.llvm.org <mailto:llvm-bugs@email.llvm.org> has been re-enabled, sending mail only on issue creation, I & others might opt back in to that behavior by disabling the baseline "notify on everything" to "notify only on issues I'm mentioned in".

I guess currently the only email that github is generating is one email per issue update. We don't have any pull requests, so there aren't any emails for that, yeah?

So this new strategy might add a few more back-and-forth on each cherrypick issue (for those using llvm-bugs & disabling general issue notifications, this will not be relevant to them - there won't be more issues created, just more comments on existing issues). But there will be some more emails generated related to the pull requests themselves, I guess? So each cherrypick goes from 2 emails to llvm-bugs (the issue creation and closure) to, how many? 4 (2 for llvm-bugs and I guess at least 2 for the pull request - one to make the request and one to close it - maybe a couple more status ones along the way?)

I think the number of net new comments on issues will be very minimal or none at all. The
automated comments that are created by this process are replacing comments that I'm already making
manually.

So 2+ for pull requests is probably a good estimate. I still need to figure out how many notifications
get generated for Actions with the default settings.

--Tom

tstellar · December 20, 2021, 11:24pm

 &gt; Sounds pretty good to me \- wouldn&#39;t mind knowing more about/a good summary of the effects of this on project/repo/etc notifications that Mehdi&#39;s mentioning\. $be good to have a write up of the expected impact/options to then discuss \- from the thread so far I understand some general/high level concerns, but it&#39;s not clear to me exactly how it plays out$
 &gt;

The impact is really going to depend on the person and what notification preferences they
have/want\.  If you are already watching the repo with the default settings, then you probably
won&#39;t notice much of a difference given the current volume of notifications\.
I think I'm on the default settings - which does currently mean a notification for every issue update, which is a lot. Given that llvm-bugs@email.llvm.org <mailto:llvm-bugs@email.llvm.org> has been re-enabled, sending mail only on issue creation, I & others might opt back in to that behavior by disabling the baseline "notify on everything" to "notify only on issues I'm mentioned in".

I guess currently the only email that github is generating is one email per issue update. We don't have any pull requests, so there aren't any emails for that, yeah?

So this new strategy might add a few more back-and-forth on each cherrypick issue (for those using llvm-bugs & disabling general issue notifications, this will not be relevant to them - there won't be more issues created, just more comments on existing issues). But there will be some more emails generated related to the pull requests themselves, I guess? So each cherrypick goes from 2 emails to llvm-bugs (the issue creation and closure) to, how many? 4 (2 for llvm-bugs and I guess at least 2 for the pull request - one to make the request and one to close it - maybe a couple more status ones along the way?)
I think the number of net new comments on issues will be very minimal or none at all. The
automated comments that are created by this process are replacing comments that I'm already making
manually.

So 2+ for pull requests is probably a good estimate. I still need to figure out how many notifications
get generated for Actions with the default settings.

I did some research on the notifications and here is what I came up with:

From what I can tell, notifications for actions are only sent to the
user that initiated the event that led to the actions, so there would
be no global notifications sent for the actions used by this workflow.

There have been 131 bugs marked as release blockers in the llvm-13 cycle,
this includes the 13.0.0 and 13.0.1 release. In the best case scenario,
this proposal would generate 2 additional notifications per issue
(1 for creating a pull request and 1 for merging it), and 0 net new
issue comments (the automated comments just replace manual comments).

If you assume that no manual comments would be replaced by the automation,
then in the typical use case there would be a maximum of 4 notifications
generated from issues (/cherry-pick comment, cherry-pick failed comment,
/branch comment, /pull-request comment). In addition to the 2 pull
request notifications.

Based on this, my estimate is that this proposal will produce between
(2 * 131) = 262 and (6 * 131) = 786 net new notifications every 6 months.
Or between 1.46 and 4.367 net new notifications per day.

For comparison, on Fri Dec 17, I received 115 email notifications from
the llvm/llvm-project repo.

The pull request emails should be easy for people to filter out of their
inboxes with a rule. Pull request emails would have llvm/llvm-project in
the To: field and have '(PR #123456)' at the end of the Subject: field
(where 123456 is pull request number).

For people who filter out the pull request notifications, they would have between
0 and 2.9 net new notifications per day.

- Tom

preames · December 20, 2021, 11:34pm

 &gt; Sounds pretty good to me \- wouldn&#39;t mind knowing more about/a good summary of the effects of this on project/repo/etc notifications that Mehdi&#39;s mentioning\. $be good to have a write up of the expected impact/options to then discuss \- from the thread so far I understand some general/high level concerns, but it&#39;s not clear to me exactly how it plays out$
 &gt;

The impact is really going to depend on the person and what notification preferences they
have/want\.  If you are already watching the repo with the default settings, then you probably
won&#39;t notice much of a difference given the current volume of notifications\.
I think I'm on the default settings - which does currently mean a notification for every issue update, which is a lot. Given that llvm-bugs@email.llvm.org <mailto:llvm-bugs@email.llvm.org> has been re-enabled, sending mail only on issue creation, I & others might opt back in to that behavior by disabling the baseline "notify on everything" to "notify only on issues I'm mentioned in".

I guess currently the only email that github is generating is one email per issue update. We don't have any pull requests, so there aren't any emails for that, yeah?

So this new strategy might add a few more back-and-forth on each cherrypick issue (for those using llvm-bugs & disabling general issue notifications, this will not be relevant to them - there won't be more issues created, just more comments on existing issues). But there will be some more emails generated related to the pull requests themselves, I guess? So each cherrypick goes from 2 emails to llvm-bugs (the issue creation and closure) to, how many? 4 (2 for llvm-bugs and I guess at least 2 for the pull request - one to make the request and one to close it - maybe a couple more status ones along the way?)
I think the number of net new comments on issues will be very minimal or none at all. The
automated comments that are created by this process are replacing comments that I'm already making
manually.

So 2+ for pull requests is probably a good estimate. I still need to figure out how many notifications
get generated for Actions with the default settings.
I did some research on the notifications and here is what I came up with:

From what I can tell, notifications for actions are only sent to the
user that initiated the event that led to the actions, so there would
be no global notifications sent for the actions used by this workflow.

There have been 131 bugs marked as release blockers in the llvm-13 cycle,
this includes the 13.0.0 and 13.0.1 release. In the best case scenario,
this proposal would generate 2 additional notifications per issue
(1 for creating a pull request and 1 for merging it), and 0 net new
issue comments (the automated comments just replace manual comments).

If you assume that no manual comments would be replaced by the automation,
then in the typical use case there would be a maximum of 4 notifications
generated from issues (/cherry-pick comment, cherry-pick failed comment,
/branch comment, /pull-request comment). In addition to the 2 pull
request notifications.

Based on this, my estimate is that this proposal will produce between
(2 * 131) = 262 and (6 * 131) = 786 net new notifications every 6 months.
Or between 1.46 and 4.367 net new notifications per day.

For comparison, on Fri Dec 17, I received 115 email notifications from
the llvm/llvm-project repo.

The pull request emails should be easy for people to filter out of their
inboxes with a rule. Pull request emails would have llvm/llvm-project in
the To: field and have '(PR #123456)' at the end of the Subject: field
(where 123456 is pull request number).

For people who filter out the pull request notifications, they would have between
0 and 2.9 net new notifications per day.

This seems both fairly minimal, and well justified to me.

Philip

mehdi_amini · December 21, 2021, 2:21am

Sounds pretty good to me - wouldn’t mind knowing more about/a good summary of the effects of this on project/repo/etc notifications that Mehdi’s mentioning. (be good to have a write up of the expected impact/options to then discuss - from the thread so far I understand some general/high level concerns, but it’s not clear to me exactly how it plays out)

The impact is really going to depend on the person and what notification preferences they
have/want. If you are already watching the repo with the default settings, then you probably
won’t notice much of a difference given the current volume of notifications.

I think I’m on the default settings - which does currently mean a notification for every issue update, which is a lot. Given that llvm-bugs@email.llvm.org mailto:[llvm-bugs@email.llvm.org](mailto:llvm-bugs@email.llvm.org) has been re-enabled, sending mail only on issue creation, I & others might opt back in to that behavior by disabling the baseline “notify on everything” to “notify only on issues I’m mentioned in”.

I guess currently the only email that github is generating is one email per issue update. We don’t have any pull requests, so there aren’t any emails for that, yeah?

So this new strategy might add a few more back-and-forth on each cherrypick issue (for those using llvm-bugs & disabling general issue notifications, this will not be relevant to them - there won’t be more issues created, just more comments on existing issues). But there will be some more emails generated related to the pull requests themselves, I guess? So each cherrypick goes from 2 emails to llvm-bugs (the issue creation and closure) to, how many? 4 (2 for llvm-bugs and I guess at least 2 for the pull request - one to make the request and one to close it - maybe a couple more status ones along the way?)

I think the number of net new comments on issues will be very minimal or none at all. The
automated comments that are created by this process are replacing comments that I’m already making
manually.

So 2+ for pull requests is probably a good estimate. I still need to figure out how many notifications
get generated for Actions with the default settings.

I did some research on the notifications and here is what I came up with:

From what I can tell, notifications for actions are only sent to the
user that initiated the event that led to the actions, so there would
be no global notifications sent for the actions used by this workflow.

There have been 131 bugs marked as release blockers in the llvm-13 cycle,
this includes the 13.0.0 and 13.0.1 release. In the best case scenario,
this proposal would generate 2 additional notifications per issue
(1 for creating a pull request and 1 for merging it), and 0 net new
issue comments (the automated comments just replace manual comments).

If you assume that no manual comments would be replaced by the automation,
then in the typical use case there would be a maximum of 4 notifications
generated from issues (/cherry-pick comment, cherry-pick failed comment,
/branch comment, /pull-request comment). In addition to the 2 pull
request notifications.

Based on this, my estimate is that this proposal will produce between
(2 * 131) = 262 and (6 * 131) = 786 net new notifications every 6 months.
Or between 1.46 and 4.367 net new notifications per day.

For comparison, on Fri Dec 17, I received 115 email notifications from
the llvm/llvm-project repo.

The pull request emails should be easy for people to filter out of their
inboxes with a rule. Pull request emails would have llvm/llvm-project in
the To: field and have ‘(PR #123456)’ at the end of the Subject: field
(where 123456 is pull request number).

Actually it isn’t enough: there isn’t a way to filter on regexes in gmail for example. Until GitHub allows the use of some different alias / target / cc-email or similar mechanisms, it’ll be hard to filter GitHub emails accurately / reliably.

There are also the confusing aspects of starting to use pull-requests in the monorepo, but only for some branches, which seem undesirable to me.

You didn’t really elaborate about why not use a repo dedicated for managing your actions and everything else? Seems like a conservative choice that would provide isolation while still having the feature you desired, wouldn’t it?

tstellar · December 21, 2021, 5:18am

     >>
     >> > Sounds pretty good to me - wouldn't mind knowing more about/a good summary of the effects of this on project/repo/etc notifications that Mehdi's mentioning. (be good to have a write up of the expected impact/options to then discuss - from the thread so far I understand some general/high level concerns, but it's not clear to me exactly how it plays out)
     >> >
     >>
     >> The impact is really going to depend on the person and what notification preferences they
     >> have/want. If you are already watching the repo with the default settings, then you probably
     >> won't notice much of a difference given the current volume of notifications.
     >>
     >> I think I'm on the default settings - which does currently mean a notification for every issue update, which is a lot. Given that llvm-bugs@email.llvm.org <mailto:llvm-bugs@email.llvm.org> <mailto:llvm-bugs@email.llvm.org> has been re-enabled, sending mail only on issue creation, I & others might opt back in to that behavior by disabling the baseline "notify on everything" to "notify only on issues I'm mentioned in".
     >>
     >> I guess currently the only email that github is generating is one email per issue update. We don't have any pull requests, so there aren't any emails for that, yeah?
     >>
     >> So this new strategy might add a few more back-and-forth on each cherrypick issue (for those using llvm-bugs & disabling general issue notifications, this will not be relevant to them - there won't be more issues created, just more comments on existing issues). But there will be some more emails generated related to the pull requests themselves, I guess? So each cherrypick goes from 2 emails to llvm-bugs (the issue creation and closure) to, how many? 4 (2 for llvm-bugs and I guess at least 2 for the pull request - one to make the request and one to close it - maybe a couple more status ones along the way?)
     >>
     >
     > I think the number of net new comments on issues will be very minimal or none at all. The
     > automated comments that are created by this process are replacing comments that I'm already making
     > manually.
     >
     > So 2+ for pull requests is probably a good estimate. I still need to figure out how many notifications
     > get generated for Actions with the default settings.
     >

    I did some research on the notifications and here is what I came up with:

      From what I can tell, notifications for actions are only sent to the
    user that initiated the event that led to the actions, so there would
    be no global notifications sent for the actions used by this workflow.

    There have been 131 bugs marked as release blockers in the llvm-13 cycle,
    this includes the 13.0.0 and 13.0.1 release. In the best case scenario,
    this proposal would generate 2 additional notifications per issue
    (1 for creating a pull request and 1 for merging it), and 0 net new
    issue comments (the automated comments just replace manual comments).

    If you assume that no manual comments would be replaced by the automation,
    then in the typical use case there would be a maximum of 4 notifications
    generated from issues (/cherry-pick comment, cherry-pick failed comment,
    /branch comment, /pull-request comment). In addition to the 2 pull
    request notifications.

    Based on this, my estimate is that this proposal will produce between
    (2 * 131) = 262 and (6 * 131) = 786 net new notifications every 6 months.
    Or between 1.46 and 4.367 net new notifications per day.

    For comparison, on Fri Dec 17, I received 115 email notifications from
    the llvm/llvm-project repo.

    The pull request emails should be easy for people to filter out of their
    inboxes with a rule. Pull request emails would have llvm/llvm-project in
    the To: field and have '(PR #123456)' at the end of the Subject: field
    (where 123456 is pull request number).

Actually it isn't enough: there isn't a way to filter on regexes in gmail for example. Until GitHub allows the use of some different alias / target / cc-email or similar mechanisms, it'll be hard to filter GitHub emails accurately / reliably.

Matching on '(PR #' might be enough if people wanted to try it.

There are also the confusing aspects of starting to use pull-requests in the monorepo, but only for some branches, which seem undesirable to me.

Yeah, this is one of the downsides of using pull-requests in the monorepo.

You didn't really elaborate about why not use a repo dedicated for managing your actions and everything else? Seems like a conservative choice that would provide isolation while still having the feature you desired, wouldn't it?

One advantage of using the monorepo for the pull-requests is that it simplifies
the workflow. With pull-requests in a secondary repo, there would have to be an
additional step to sync from the secondary repo into the monorepo after the
pull-request was merged. You would also need to setup a sync from the monorepo
to the secondary repo in case something was pushed directly to the monorepo.

Using the monorepo for pull requests also makes it easier to enable self-hosted
action runners, because you would only need to set them up for one repo and
wouldn't have to figure out how to coordinate their usage between two.
And if we were able to get GitHub to give us access to some of their more powerful
runners, it might be more difficult to enable them for both repos.

- Tom

mehdi_amini · December 23, 2021, 3:51am

Sounds pretty good to me - wouldn’t mind knowing more about/a good summary of the effects of this on project/repo/etc notifications that Mehdi’s mentioning. (be good to have a write up of the expected impact/options to then discuss - from the thread so far I understand some general/high level concerns, but it’s not clear to me exactly how it plays out)

The impact is really going to depend on the person and what notification preferences they
have/want. If you are already watching the repo with the default settings, then you probably
won’t notice much of a difference given the current volume of notifications.

I think I’m on the default settings - which does currently mean a notification for every issue update, which is a lot. Given that llvm-bugs@email.llvm.org mailto:[llvm-bugs@email.llvm.org](mailto:llvm-bugs@email.llvm.org) <mailto:llvm-bugs@email.llvm.org mailto:[llvm-bugs@email.llvm.org](mailto:llvm-bugs@email.llvm.org)> has been re-enabled, sending mail only on issue creation, I & others might opt back in to that behavior by disabling the baseline “notify on everything” to “notify only on issues I’m mentioned in”.

I guess currently the only email that github is generating is one email per issue update. We don’t have any pull requests, so there aren’t any emails for that, yeah?

So this new strategy might add a few more back-and-forth on each cherrypick issue (for those using llvm-bugs & disabling general issue notifications, this will not be relevant to them - there won’t be more issues created, just more comments on existing issues). But there will be some more emails generated related to the pull requests themselves, I guess? So each cherrypick goes from 2 emails to llvm-bugs (the issue creation and closure) to, how many? 4 (2 for llvm-bugs and I guess at least 2 for the pull request - one to make the request and one to close it - maybe a couple more status ones along the way?)

I think the number of net new comments on issues will be very minimal or none at all. The
automated comments that are created by this process are replacing comments that I’m already making
manually.

So 2+ for pull requests is probably a good estimate. I still need to figure out how many notifications
get generated for Actions with the default settings.

I did some research on the notifications and here is what I came up with:

From what I can tell, notifications for actions are only sent to the
user that initiated the event that led to the actions, so there would
be no global notifications sent for the actions used by this workflow.

There have been 131 bugs marked as release blockers in the llvm-13 cycle,
this includes the 13.0.0 and 13.0.1 release. In the best case scenario,
this proposal would generate 2 additional notifications per issue
(1 for creating a pull request and 1 for merging it), and 0 net new
issue comments (the automated comments just replace manual comments).

If you assume that no manual comments would be replaced by the automation,
then in the typical use case there would be a maximum of 4 notifications
generated from issues (/cherry-pick comment, cherry-pick failed comment,
/branch comment, /pull-request comment). In addition to the 2 pull
request notifications.

Based on this, my estimate is that this proposal will produce between
(2 * 131) = 262 and (6 * 131) = 786 net new notifications every 6 months.
Or between 1.46 and 4.367 net new notifications per day.

For comparison, on Fri Dec 17, I received 115 email notifications from
the llvm/llvm-project repo.

The pull request emails should be easy for people to filter out of their
inboxes with a rule. Pull request emails would have llvm/llvm-project in
the To: field and have ‘(PR #123456)’ at the end of the Subject: field
(where 123456 is pull request number).

Actually it isn’t enough: there isn’t a way to filter on regexes in gmail for example. Until GitHub allows the use of some different alias / target / cc-email or similar mechanisms, it’ll be hard to filter GitHub emails accurately / reliably.

Matching on ‘(PR #’ might be enough if people wanted to try it.

Gmail ignores punctuation unfortunately, this is partly why the current situation with issue notification is also quite messy and hard to manage on the client side.

There are also the confusing aspects of starting to use pull-requests in the monorepo, but only for some branches, which seem undesirable to me.

Yeah, this is one of the downsides of using pull-requests in the monorepo.

You didn’t really elaborate about why not use a repo dedicated for managing your actions and everything else? Seems like a conservative choice that would provide isolation while still having the feature you desired, wouldn’t it?

One advantage of using the monorepo for the pull-requests is that it simplifies
the workflow. With pull-requests in a secondary repo, there would have to be an
additional step to sync from the secondary repo into the monorepo after the
pull-request was merged. You would also need to setup a sync from the monorepo
to the secondary repo in case something was pushed directly to the monorepo.

Using the monorepo for pull requests also makes it easier to enable self-hosted
action runners, because you would only need to set them up for one repo and
wouldn’t have to figure out how to coordinate their usage between two.
And if we were able to get GitHub to give us access to some of their more powerful
runners, it might be more difficult to enable them for both repos.

I would hope that we can get access to this for the entire organization and just for one repo.

Ben_Boeckel · December 23, 2021, 1:32pm

The Message-Id header is pretty unique for GitHub events, but I don't
think Gmail indexes those either (AFAICT, filters can only affect what
is indexed).

Pressure on Google to fix such things would be amazing .

--Ben

Renato_Golin2 · December 23, 2021, 5:53pm

Hi Tom,

Would this be any user or users with certain permissions in the repo (like code owners, release managers)?

Ignoring malicious action, any user creating a cherry-pick at any time, may create confusion if two users are trying to pick changes that need multiple (non-sequential) commits each.

An alternative would be to build a branch off the release branch (ex. “release-x.y.z-$username”) and pick the commits on that branch, run the pre-commit tests, and then merge to the release branch if it’s all green.

Because the merge is atomic, and the tests passed on the alternative branch, the probability of the release branch breaking is lower.

Of course, interaction between the users’ branches can still break, and well, further tests that are not present in the pre-commit tests, can also.

But with atomic merges of cherry-picks in a linear sequence will also make it easier to bisect in case anything goes wrong with the release candidate.

If only a subset of users can merge, then they’d do one at a time and this problem wouldn’t be a big issue and we’d avoid a complicated infrastructure setup.

Does that make sense?

cheers,
–renato

tstellar · December 24, 2021, 3:11am

    * On an existing issue or a newly created issue, a user who wants to backport
    one or more commits to the release branch adds a comment:

    /cherry-pick <commit_sha> <..>

Hi Tom,

Would this be *any* user or users with certain permissions in the repo (like code owners, release managers)?

Any user can do this.

Ignoring malicious action, *any* user creating a cherry-pick at any time, may create confusion if two users are trying to pick changes that need multiple (non-sequential) commits each.

An alternative would be to build a branch off the release branch (ex. "release-x.y.z-$username") and pick the commits on that branch, run the pre-commit tests, and then merge to the release branch if it's all green.

This is actually how it works. The cherry-picked commits get
pushed to a branch called issue<issue#> and the pull request is created
off of that branch.

-Tom

Renato_Golin2 · December 24, 2021, 11:49am

Ah, awesome, thanks!

tstellar · January 15, 2022, 6:53am

Hi,

Here is a proposal for a new automated workflow for managing parts of the release
process. I've been experimenting with this over the past few releases and
now that we have migrated to GitHub issues, it would be possible for us to
implement this in the main repo.

The workflow is pretty straight forward, but it does use pull requests. My
idea is to enable pull requests for only this automated workflow and not
for general development (i.e. We would still use Phabricator for code review).
Let me know what you think about this:

Hi,

Thanks for the feedback on this. I've posted a patch to implement this proposal:
⚙ D117386 github: Add actions to automate part of the release workflow. The only change is that the pull requests will
not be in the llvm/llvm-project repo, but will instead be in llvmbot/llvm-project.
This will reduce the number of notifications and avoid confusion about whether or
not we are using pull requests for this project.

-Tom

Topic		Replies	Views
RFC: New Automated Release Workflow Part 2 LLVM Project rfc	8	641	February 21, 2022
Help reviewing github-actions automation Release Testers	0	286	September 8, 2022
[RFC] Helping release management LLVM Dev List Archives	57	142	May 18, 2016
Creating a new repository for release branch pull requests LLVM Project	3	326	May 17, 2022
RFC / FYI: Pull Request Greetings For New Contributors LLVM Project	29	1085	April 5, 2024

RFC: New Automated Release Workflow (using Issues and Pull Requests)

Workflow

Workflow

Related Topics