GitHub Migration Schedule and Plans

Hi,

We're less than 2 weeks away from the developer meeting, so I wanted to
give an update on the GitHub migration and what's (hopefully) going to
happen during the developer meeting.

Everyone who has added their information to the github-usernames.txt
file in SVN before today should have received an invite to become a collaborator
on the llvm-project repository. If you did not receive an invite and think
you should have, please contact me off-list. I will continue to monitor the
file for new updates and periodically send out new batches of invites.

There is still some ongoing work to get the buildbots ready and the mailing lists
ready, but we are optimistic that the work will be done in time.

The team at GitHub has finished implementing the "Require Linear History"
branch protection that we requested. The feature is in beta and currently
enabled in the llvm-project repository. This means that we will have the
option to commit directly via git, in addition to using the git-llvm script.
A patch that updates git-llvm to push to git instead of svn can be found here:
https://reviews.llvm.org/D67772. You should be able to test it out on your
own fork of the llvm-project repository.

The current plan is to begin the final migration steps on the evening (PDT)
of October 21. Here is what will happen:

1. Make SVN read-only.
2. Turn-off the SVN->git update process.
3. Commit the new git-llvm script directly to github.
4. Grant all contributors write access to the repository.
5. Email lists announcing that the migration is complete.

Once the migration is complete, if you run into any issues, please file
a bug, and mark it as a blocker for the github metabug PR39393.

If you have any questions or think I am missing something, please
let me know.

Thanks,
Tom

Hi,

We’re less than 2 weeks away from the developer meeting, so I wanted to
give an update on the GitHub migration and what’s (hopefully) going to
happen during the developer meeting.

Everyone who has added their information to the github-usernames.txt
file in SVN before today should have received an invite to become a collaborator
on the llvm-project repository. If you did not receive an invite and think
you should have, please contact me off-list. I will continue to monitor the
file for new updates and periodically send out new batches of invites.

There is still some ongoing work to get the buildbots ready and the mailing lists
ready, but we are optimistic that the work will be done in time.

The team at GitHub has finished implementing the “Require Linear History”
branch protection that we requested. The feature is in beta and currently
enabled in the llvm-project repository. This means that we will have the
option to commit directly via git, in addition to using the git-llvm script.
A patch that updates git-llvm to push to git instead of svn can be found here:
https://reviews.llvm.org/D67772. You should be able to test it out on your
own fork of the llvm-project repository.

The current plan is to begin the final migration steps on the evening (PDT)
of October 21. Here is what will happen:

  1. Make SVN read-only.
  2. Turn-off the SVN->git update process.
  3. Commit the new git-llvm script directly to github.
  4. Grant all contributors write access to the repository.

Is the repo configured to forbid contributors to create new branches? I’m worried about the “jungle” it can become quickly if we leave open the possibility to create branches “at will” in the repo, I rather leave this to maintainers.

  1. Email lists announcing that the migration is complete.

Once the migration is complete, if you run into any issues, please file
a bug, and mark it as a blocker for the github metabug PR39393.

If you have any questions or think I am missing something, please
let me know.

This is fantastic! Thank you so much for doing this work Tom :slight_smile:

Hi,

We're less than 2 weeks away from the developer meeting, so I wanted to
give an update on the GitHub migration and what's (hopefully) going to
happen during the developer meeting.

Everyone who has added their information to the github-usernames.txt
file in SVN before today should have received an invite to become a collaborator
on the llvm-project repository. If you did not receive an invite and think
you should have, please contact me off-list. I will continue to monitor the
file for new updates and periodically send out new batches of invites.

It seems like not everyone received email notifications of the invites
(it may depend on your settings). You should be able to see your invite
if you go to https://github.com/llvm/llvm-project/invitations.

-Tom

    Hi,

    We're less than 2 weeks away from the developer meeting, so I wanted to
    give an update on the GitHub migration and what's (hopefully) going to
    happen during the developer meeting.

    Everyone who has added their information to the github-usernames.txt
    file in SVN before today should have received an invite to become a collaborator
    on the llvm-project repository. If you did not receive an invite and think
    you should have, please contact me off-list. I will continue to monitor the
    file for new updates and periodically send out new batches of invites.

    There is still some ongoing work to get the buildbots ready and the mailing lists
    ready, but we are optimistic that the work will be done in time.

    The team at GitHub has finished implementing the "Require Linear History"
    branch protection that we requested. The feature is in beta and currently
    enabled in the llvm-project repository. This means that we will have the
    option to commit directly via git, in addition to using the git-llvm script.
    A patch that updates git-llvm to push to git instead of svn can be found here:
    https://reviews.llvm.org/D67772. You should be able to test it out on your
    own fork of the llvm-project repository.

    The current plan is to begin the final migration steps on the evening (PDT)
    of October 21. Here is what will happen:

    1. Make SVN read-only.
    2. Turn-off the SVN->git update process.
    3. Commit the new git-llvm script directly to github.
    4. Grant all contributors write access to the repository.

Is the repo configured to forbid contributors to create new branches? I'm worried about the "jungle" it can become quickly if we leave open the possibility to create branches "at will" in the repo, I rather leave this to maintainers.

I haven't been able to find a way to restrict branch creation for committers,
I'm not sure if this is even possible.

We could try to enforce this rule in the git-llvm script, but this would
mean making use of the script mandatory, which was our original plan, but
that was based on the assumption that the "Require Linear History"
protection would not be ready in time.

Generally, would it be better if we kept use of the script mandatory so that
we can handle this and other potential restrictions in the future?

- Tom

Hi,

We’re less than 2 weeks away from the developer meeting, so I wanted to
give an update on the GitHub migration and what’s (hopefully) going to
happen during the developer meeting.

Everyone who has added their information to the github-usernames.txt
file in SVN before today should have received an invite to become a collaborator
on the llvm-project repository. If you did not receive an invite and think
you should have, please contact me off-list. I will continue to monitor the
file for new updates and periodically send out new batches of invites.

There is still some ongoing work to get the buildbots ready and the mailing lists
ready, but we are optimistic that the work will be done in time.

The team at GitHub has finished implementing the “Require Linear History”
branch protection that we requested. The feature is in beta and currently
enabled in the llvm-project repository. This means that we will have the
option to commit directly via git, in addition to using the git-llvm script.
A patch that updates git-llvm to push to git instead of svn can be found here:
https://reviews.llvm.org/D67772. You should be able to test it out on your
own fork of the llvm-project repository.

The current plan is to begin the final migration steps on the evening (PDT)
of October 21. Here is what will happen:

  1. Make SVN read-only.
  2. Turn-off the SVN->git update process.
  3. Commit the new git-llvm script directly to github.
  4. Grant all contributors write access to the repository.

Is the repo configured to forbid contributors to create new branches? I’m worried about the “jungle” it can become quickly if we leave open the possibility to create branches “at will” in the repo, I rather leave this to maintainers.

I haven’t been able to find a way to restrict branch creation for committers,
I’m not sure if this is even possible.

I think you can just go to the branch settings, add a new branch protection rule, match on everything but master, and check “Restrict who can push to matching branches”.

Personally, I'd prefer to avoid requiring a custom tool for pushing commits, if possible.

As for creating undesired branches; contrary to accidentally pushing merge commits on the master branch, any accidentally created branch should be possible to remove without any permanent traces left behind (especially as it doesn't affect the master branch). So for that purpose alone, I'd vote for not requiring git-llvm for pushing.

Btw, out of curiousity, what mechanism does GitHub offer for verifying that the push actually comes from git-llvm? (I'm sorry if this part was discussed elsewhere, but I don't remember seeing it discussed lately.)

// Martin

Hi,

We’re less than 2 weeks away from the developer meeting, so I wanted to
give an update on the GitHub migration and what’s (hopefully) going to
happen during the developer meeting.

Everyone who has added their information to the github-usernames.txt
file in SVN before today should have received an invite to become a collaborator
on the llvm-project repository. If you did not receive an invite and think
you should have, please contact me off-list. I will continue to monitor the
file for new updates and periodically send out new batches of invites.

There is still some ongoing work to get the buildbots ready and the mailing lists
ready, but we are optimistic that the work will be done in time.

The team at GitHub has finished implementing the “Require Linear History”
branch protection that we requested. The feature is in beta and currently
enabled in the llvm-project repository. This means that we will have the
option to commit directly via git, in addition to using the git-llvm script.
A patch that updates git-llvm to push to git instead of svn can be found here:
https://reviews.llvm.org/D67772. You should be able to test it out on your
own fork of the llvm-project repository.

The current plan is to begin the final migration steps on the evening (PDT)
of October 21. Here is what will happen:

  1. Make SVN read-only.
  2. Turn-off the SVN->git update process.
  3. Commit the new git-llvm script directly to github.
  4. Grant all contributors write access to the repository.

Is the repo configured to forbid contributors to create new branches? I’m worried about the “jungle” it can become quickly if we leave open the possibility to create branches “at will” in the repo, I rather leave this to maintainers.

I haven’t been able to find a way to restrict branch creation for committers,
I’m not sure if this is even possible.

We could try to enforce this rule in the git-llvm script, but this would
mean making use of the script mandatory, which was our original plan, but
that was based on the assumption that the “Require Linear History”
protection would not be ready in time.

Generally, would it be better if we kept use of the script mandatory so that
we can handle this and other potential restrictions in the future?

Personally, I’d prefer to avoid requiring a custom tool for pushing
commits, if possible.

As for creating undesired branches; contrary to accidentally pushing merge
commits on the master branch, any accidentally created branch should be
possible to remove without any permanent traces left behind (especially as
it doesn’t affect the master branch). So for that purpose alone, I’d vote
for not requiring git-llvm for pushing.

The original reason for the tool is about linear history (contrary to branches, this can’t be undone).

Btw, out of curiousity, what mechanism does GitHub offer for verifying
that the push actually comes from git-llvm? (I’m sorry if this part was
discussed elsewhere, but I don’t remember seeing it discussed lately.)

I mentioned it on LLVM-dev a couple of times a while back (here for example: http://lists.llvm.org/pipermail/llvm-dev/2019-February/129896.html )

But apparently the concrete implementation was discussed on cfe-dev@ only, so here are the details: http://lists.llvm.org/pipermail/cfe-dev/2019-April/062063.html

    >
    >
    >
    > Hi,
    >
    > We're less than 2 weeks away from the developer meeting, so I wanted to
    > give an update on the GitHub migration and what's (hopefully) going to
    > happen during the developer meeting.
    >
    > Everyone who has added their information to the github-usernames.txt
    > file in SVN before today should have received an invite to become a collaborator
    > on the llvm-project repository. If you did not receive an invite and think
    > you should have, please contact me off-list. I will continue to monitor the
    > file for new updates and periodically send out new batches of invites.
    >
    > There is still some ongoing work to get the buildbots ready and the mailing lists
    > ready, but we are optimistic that the work will be done in time.
    >
    > The team at GitHub has finished implementing the "Require Linear History"
    > branch protection that we requested. The feature is in beta and currently
    > enabled in the llvm-project repository. This means that we will have the
    > option to commit directly via git, in addition to using the git-llvm script.
    > A patch that updates git-llvm to push to git instead of svn can be found here:
    > https://reviews.llvm.org/D67772. You should be able to test it out on your
    > own fork of the llvm-project repository.
    >
    > The current plan is to begin the final migration steps on the evening (PDT)
    > of October 21. Here is what will happen:
    >
    > 1. Make SVN read-only.
    > 2. Turn-off the SVN->git update process.
    > 3. Commit the new git-llvm script directly to github.
    > 4. Grant all contributors write access to the repository.
    >
    >
    > Is the repo configured to forbid contributors to create new branches? I'm worried about the "jungle" it can become quickly if we leave open the possibility to create branches "at will" in the repo, I rather leave this to maintainers.
    >

    I haven't been able to find a way to restrict branch creation for committers,
    I'm not sure if this is even possible.

I think you can just go to the branch settings, add a new branch protection rule, match on everything but master, and check "Restrict who can push to matching branches".

I tried this, and the branch protections only come into effect after a branch
has been creating, so this doesn't prevent new branches. It's actually worse
than doing nothing, because once the branch is created the branch protection
prevents you from deleting it.

-Tom

      >>
      >
      > I haven't been able to find a way to restrict branch creation
      for committers,
      > I'm not sure if this is even possible.
      >
      > We could try to enforce this rule in the git-llvm script, but
      this would
      > mean making use of the script mandatory, which was our
      original plan, but
      > that was based on the assumption that the "Require Linear
      History"
      > protection would not be ready in time.
      >
      > Generally, would it be better if we kept use of the script
      mandatory so that
      > we can handle this and other potential restrictions in the
      future?

      Personally, I'd prefer to avoid requiring a custom tool for
      pushing
      commits, if possible.

      As for creating undesired branches; contrary to accidentally
      pushing merge
      commits on the master branch, any accidentally created branch
      should be
      possible to remove without any permanent traces left behind
      (especially as
      it doesn't affect the master branch). So for that purpose alone,
      I'd vote
      for not requiring git-llvm for pushing.

The original reason for the tool is about linear history (contrary to
branches, this can't be undone).

Yes, I see (or saw) the reason for the tool if a linear history couldn't be guaranteed otherwise, but I'm more hesitant if the only current purpose is preventing branch creation.

      Btw, out of curiousity, what mechanism does GitHub offer for
      verifying
      that the push actually comes from git-llvm? (I'm sorry if this
      part was
      discussed elsewhere, but I don't remember seeing it discussed
      lately.)

I mentioned it on LLVM-dev a couple of times a while back (here for example:
http://lists.llvm.org/pipermail/llvm-dev/2019-February/129896.html )

But apparently the concrete implementation was discussed on cfe-dev@ only,
so here are the details:
http://lists.llvm.org/pipermail/cfe-dev/2019-April/062063.html

Ok, thanks for the references! That's a rather interesting feature they have.

// Martin

Hi,

We’re less than 2 weeks away from the developer meeting, so I wanted to
give an update on the GitHub migration and what’s (hopefully) going to
happen during the developer meeting.

Everyone who has added their information to the github-usernames.txt
file in SVN before today should have received an invite to become a collaborator
on the llvm-project repository. If you did not receive an invite and think
you should have, please contact me off-list. I will continue to monitor the
file for new updates and periodically send out new batches of invites.

There is still some ongoing work to get the buildbots ready and the mailing lists
ready, but we are optimistic that the work will be done in time.

The team at GitHub has finished implementing the “Require Linear History”
branch protection that we requested. The feature is in beta and currently
enabled in the llvm-project repository. This means that we will have the
option to commit directly via git, in addition to using the git-llvm script.
A patch that updates git-llvm to push to git instead of svn can be found here:
https://reviews.llvm.org/D67772. You should be able to test it out on your
own fork of the llvm-project repository.

The current plan is to begin the final migration steps on the evening (PDT)
of October 21. Here is what will happen:

  1. Make SVN read-only.
  2. Turn-off the SVN->git update process.
  3. Commit the new git-llvm script directly to github.
  4. Grant all contributors write access to the repository.

Is the repo configured to forbid contributors to create new branches? I’m worried about the “jungle” it can become quickly if we leave open the possibility to create branches “at will” in the repo, I rather leave this to maintainers.

I haven’t been able to find a way to restrict branch creation for committers,
I’m not sure if this is even possible.

I think you can just go to the branch settings, add a new branch protection rule, match on everything but master, and check “Restrict who can push to matching branches”.

I tried this, and the branch protections only come into effect after a branch
has been creating, so this doesn’t prevent new branches. It’s actually worse
than doing nothing, because once the branch is created the branch protection
prevents you from deleting it.

-Tom

FWIW, we’re interested in periodically (weekly) tagging well-tested/stable revisions, but via a branch instead of just a tag so we can include which cherrypicks (e.g. reverts or fixes) are needed. We do this with the current svn repo so we’d just be porting existing functionality to github.

(Also, I’m not sure this announcement thread with all the *-dev lists is the best place to discuss branching policy, but I wanted to get this bit in since y’all brought it up :slight_smile: )

Hi,

We’re less than 2 weeks away from the developer meeting, so I wanted to
give an update on the GitHub migration and what’s (hopefully) going to
happen during the developer meeting.

Everyone who has added their information to the github-usernames.txt
file in SVN before today should have received an invite to become a collaborator
on the llvm-project repository. If you did not receive an invite and think
you should have, please contact me off-list. I will continue to monitor the
file for new updates and periodically send out new batches of invites.

There is still some ongoing work to get the buildbots ready and the mailing lists
ready, but we are optimistic that the work will be done in time.

The team at GitHub has finished implementing the “Require Linear History”
branch protection that we requested. The feature is in beta and currently
enabled in the llvm-project repository. This means that we will have the
option to commit directly via git, in addition to using the git-llvm script.
A patch that updates git-llvm to push to git instead of svn can be found here:
https://reviews.llvm.org/D67772. You should be able to test it out on your
own fork of the llvm-project repository.

The current plan is to begin the final migration steps on the evening (PDT)
of October 21. Here is what will happen:

  1. Make SVN read-only.
  2. Turn-off the SVN->git update process.
  3. Commit the new git-llvm script directly to github.
  4. Grant all contributors write access to the repository.

Is the repo configured to forbid contributors to create new branches? I’m worried about the “jungle” it can become quickly if we leave open the possibility to create branches “at will” in the repo, I rather leave this to maintainers.

I haven’t been able to find a way to restrict branch creation for committers,
I’m not sure if this is even possible.

I think you can just go to the branch settings, add a new branch protection rule, match on everything but master, and check “Restrict who can push to matching branches”.

I tried this, and the branch protections only come into effect after a branch
has been creating, so this doesn’t prevent new branches. It’s actually worse
than doing nothing, because once the branch is created the branch protection
prevents you from deleting it.

Ah, interesting :slight_smile:
Since for now git-llvm is necessary to push to the repo, I guess creating a branch won’t even really be possible? So when adding the support for branch creation to git-llvm we could print a disclaimer about it or something?

-Tom

FWIW, we’re interested in periodically (weekly) tagging well-tested/stable revisions, but via a branch instead of just a tag so we can include which cherrypicks (e.g. reverts or fixes) are needed. We do this with the current svn repo so we’d just be porting existing functionality to github.

I would likely include this in the general category of “project maintainer” created branch?
The main difference with SVN, is that on GitHub a fork is “cheap” and everyone can have their own. Also compared to SVN you don’t really know ahead of time when you type git clone or git pull that you may pull a lot of noise.
I’m not against branches for the sake of it, I just have some bad memories from a repo where every developer was pushing their work in progress branches all the time and they never got deleted: to the point where some git CLI auto-completion was unusable (there were >10k branches and probably as many tags).

(Also, I’m not sure this announcement thread with all the *-dev lists is the best place to discuss branching policy, but I wanted to get this bit in since y’all brought it up :slight_smile: )

(we discussed branches in past GitHub migration thread, we just never had a dedicated email thread to user-branches, as it didn’t seem like there was a strong need for it)

        >
        >
        >
        > >
        > >
        > >
        > > Hi,
        > >
        > > We're less than 2 weeks away from the developer meeting, so I wanted to
        > > give an update on the GitHub migration and what's (hopefully) going to
        > > happen during the developer meeting.
        > >
        > > Everyone who has added their information to the github-usernames.txt
        > > file in SVN before today should have received an invite to become a collaborator
        > > on the llvm-project repository. If you did not receive an invite and think
        > > you should have, please contact me off-list. I will continue to monitor the
        > > file for new updates and periodically send out new batches of invites.
        > >
        > > There is still some ongoing work to get the buildbots ready and the mailing lists
        > > ready, but we are optimistic that the work will be done in time.
        > >
        > > The team at GitHub has finished implementing the "Require Linear History"
        > > branch protection that we requested. The feature is in beta and currently
        > > enabled in the llvm-project repository. This means that we will have the
        > > option to commit directly via git, in addition to using the git-llvm script.
        > > A patch that updates git-llvm to push to git instead of svn can be found here:
        > > https://reviews.llvm.org/D67772. You should be able to test it out on your
        > > own fork of the llvm-project repository.
        > >
        > > The current plan is to begin the final migration steps on the evening (PDT)
        > > of October 21. Here is what will happen:
        > >
        > > 1. Make SVN read-only.
        > > 2. Turn-off the SVN->git update process.
        > > 3. Commit the new git-llvm script directly to github.
        > > 4. Grant all contributors write access to the repository.
        > >
        > >
        > > Is the repo configured to forbid contributors to create new branches? I'm worried about the "jungle" it can become quickly if we leave open the possibility to create branches "at will" in the repo, I rather leave this to maintainers.
        > >
        >
        > I haven't been able to find a way to restrict branch creation for committers,
        > I'm not sure if this is even possible.
        >
        >
        > I think you can just go to the branch settings, add a new branch protection rule, match on everything but master, and check "Restrict who can push to matching branches".
        >

        I tried this, and the branch protections only come into effect after a branch
        has been creating, so this doesn't prevent new branches. It's actually worse
        than doing nothing, because once the branch is created the branch protection
        prevents you from deleting it.

Ah, interesting :slight_smile:
Since for now `git-llvm` is necessary to push to the repo, I guess creating a branch won't even really be possible? So when adding the support for branch creation to `git-llvm` we could print a disclaimer about it or something?

With my latest changes in https://reviews.llvm.org/D67772 it is actually possible
to create a new branch with git-llvm, so we could print a disclaimer or block this,
however we still can't stop people from creating a new branch using the
normal git command.

-Tom

Oh, of course, because the same branch protection we use for master can’t apply to branches before they are created in the repo.

Thanks,

If the master branch is protected, we can always as people to be nice
and not create new branches and clean up if it happens by accident or
misinformation.

After a while, mistakes should become infrequent enough that it's ok.

FWIW, I'm with Mehdi on this. I'd much rather people used their own
forks for development branches, and leave only release branches on the
master repo.

Also, for everyone involved in this: thank you! The "migration" list
on your first email is so short and simple that I almost didn't
believe. :slight_smile:

cheers,
--renato

While I agree that people should create branches in their own forks, I’d also like to point out that one is not obligated to download everything from a given upstream repository. For example, if one only cares about master and release branches from LLVM but not tags or “random” branches, then one can change .git/config to look like this:

[remote "truth"]
  url = https://github.com/llvm/llvm-project.git
  tagOpt = --no-tags
  fetch = +refs/heads/master:refs/remotes/truth/master
  fetch = +refs/heads/release/*:refs/remotes/truth/release/*

IMO – we should generally discourage random developer branches, and instead encourage people to fork the repo and do their work there. But I’d concur that it’s not so big a concern as to require it to be prohibited via tooling, since they’re easily removable. I can imagine some cases where a temporary dev branch in the main repo might be appropriate to use, and it’d be nice not to prohibit such a valid use.

With regards to these sorts of “vendor release branches”, though – I’m strongly against putting those in the main repo. That usage would certainly be best handled with a separate Google “fork” of the llvm-project repository – and similarly for anyone else who wants to do it. Putting such branches into the main llvm repo means that everyone pulling it pulls down all the google release branches by default, which is really unnecessary clutter (check out https://github.com/llvm/llvm-project-legacy-branches if you’d like to see what such clutter starts to look like…note that github doesn’t even display all of them in its branch/tag selector popup because there’s too many.)