Updates on SVN to GitHub migration

TLDR: Official monorepo repository will be published on
Tuesday, Oct 23, 2018. After this date, you should modify
your workflows to use the monorepo ASAP. Current workflows
will be supported for at most 1 more year.

Hi,

We had 2 round-tables this week at the Developer Meeting to
discuss the SVN to GitHub migration, and I wanted to update
the rest of the community on what we discussed.

The most important outcome from that meeting is that we
now have a timeline for completing the transition which looks
like this:

Tues Oct 23, 2018:

The latest monorepo prototype[1] will be moved over to the LLVM
organization github project[2] and will begin mirroring the current
SVN repository. Commits will still be made to the SVN repository
just as they are today.

All community members should begin migrating their workflows that
rely on SVN or the current git mirrors to use the new monorepo.

For CI jobs or internal mirrors pulling from SVN or
http://llvm.org/git/*.git you should modify them to pull from
the new monorepo and also to deal with the new repository
layout.

For Developers, you should begin using the new monorepo
for your development and using the provided scripts[3]
to commit your code. These scripts will allow to commit
to SVN from the monorepo without using git-svn

[1] https://github.com/llvm-git-prototype/llvm
[2] https://github.com/llvm/
[3] https://llvm.org/docs/GettingStarted.html#for-developers-to-work-with-a-git-monorepo

TLDR: Official monorepo repository will be published on
Tuesday, Oct 23, 2018. After this date, you should modify
your workflows to use the monorepo ASAP. Current workflows
will be supported for at most 1 more year.

Hi,

We had 2 round-tables this week at the Developer Meeting to
discuss the SVN to GitHub migration, and I wanted to update
the rest of the community on what we discussed.

The most important outcome from that meeting is that we
now have a timeline for completing the transition which looks
like this:

Step 1:

Tues Oct 23, 2018:

The latest monorepo prototype[1] will be moved over to the LLVM
organization github project[2] and will begin mirroring the current
SVN repository. Commits will still be made to the SVN repository
just as they are today.

All community members should begin migrating their workflows that
rely on SVN or the current git mirrors to use the new monorepo.

For CI jobs or internal mirrors pulling from SVN or
http://llvm.org/git/*.git you should modify them to pull from
the new monorepo and also to deal with the new repository
layout.

For Developers, you should begin using the new monorepo
for your development and using the provided scripts[3]
to commit your code. These scripts will allow to commit
to SVN from the monorepo without using git-svn

Sorry hit send before I was done. Here is the rest of the mail:

Step 2:

Around the time of next year's developer meeting (1 year at the most),
we will turn off commit access to the SVN server and enable commit
access to the monorepo. At this point the monorepo will become the
'one source of truth' for the project. Community members *must* have
updated their workflows by this date and are encouraged to begin
updating workflows ASAP.

A lot of people asked at the developer meeting about the future
of bugzilla and phabricator and whether or not we will use
github issues and pull requests. These are important questions,
but are unrelated to the migration of the code.

We also came up with a TODO list for things we want to accomplish
as a community in the next year and beyond related to github. I
am working on putting these into bugzilla so we can track progress
better and I will send a follow-up email about this.

-Tom

(+openmp-dev, they should know about this!)

Recapping the "Concerns" (https://llvm.org/docs/Proposals/GitHubMove.html#id12) there is a proposal of "single-subproject Git mirrors" for people who are only contributing to standalone subprojects. I think this will be easy in the transition period, we can just continue to move the current official git mirrors. Will this "service" be continued after GitHub becomes the 'one source of truth'? I'd strongly vote for yes, but I'm not sure how that's going to work on a technical level.

Thanks,
Jonas

(+openmp-dev, they should know about this!)

Recapping the “Concerns”
(https://llvm.org/docs/Proposals/GitHubMove.html#id12) there is a
proposal of “single-subproject Git mirrors” for people who are only
contributing to standalone subprojects.

I think this will be easy in the
transition period, we can just continue to move the current official git
mirrors.

I am not sure I understand this sentence?

Will this “service” be continued after GitHub becomes the ‘one
source of truth’? I’d strongly vote for yes, but I’m not sure how that’s
going to work on a technical level.

I suspect that a possible problem will be that we can’t guarantee in the future that a particular subdirectory in the monorepo won’t move, be split, reorganized, or that its build script will be tested in CI independently of the rest of the repo.
Otherwise providing a read-only repo that contains only a subset of the monorepo isn’t very difficult: someone has to provide a machine to continuously update the read-only “view”.

I had a short side-conversation at one of the roundtables about existing
users of the subproject repositories. It would be helpful to have
instructions about the best way to move local branches in those
repositories to the monorepo and some scripts to help with the
transition. I know someone posted an example project a while ago with
some scripts but my sense is that those scripts were particular to that
project and maybe not generally applicable.

Once the monorepo goes live (tomorrow?), what happens to the existing
subproject mirrors? Do they get wiped away and replaced with history
from the monorepo? Or are entirely new mirrors created? Or do they
just continue to mirror SVN until SVN becomes read-only?

The first option would essentially be a rewrite of history for the
subproject repositories. We'll need to know if/when that is going to
happen.

                          -David

Jonas Hahnfeld via Openmp-dev <openmp-dev@lists.llvm.org> writes:

(+openmp-dev, they should know about this!)

Recapping the "Concerns"
(https://llvm.org/docs/Proposals/GitHubMove.html#id12) there is a
proposal of "single-subproject Git mirrors" for people who are only
contributing to standalone subprojects.

I think this will be easy in the
transition period, we can just continue to move the current official
git
mirrors.

I am not sure I understand this sentence?

Sorry, "We can just continue to use the current official git mirrors" (because SVN will stay the "official" repo during the transition period).
I've mixed this with the thought that these subproject mirrors could be "moved" to the official LLVM namespace on GitHub, instead of living in https://github.com/llvm-mirror/.

Will this "service" be continued after GitHub becomes the 'one
source of truth'? I'd strongly vote for yes, but I'm not sure how
that's
going to work on a technical level.

I suspect that a possible problem will be that we can't guarantee in
the future that a particular subdirectory in the monorepo won't move,
be split, reorganized, or that its build script will be tested in CI
independently of the rest of the repo.
Otherwise providing a read-only repo that contains only a subset of
the monorepo isn't very difficult: someone has to provide a machine to
continuously update the read-only "view".

Sure, the monorepo will be the "master" repository. I think it's enough to mirror a subdirectory from that, if easily possible.

Cheers,
Jonas

I had a short side-conversation at one of the roundtables about existing
users of the subproject repositories. It would be helpful to have
instructions about the best way to move local branches in those
repositories to the monorepo and some scripts to help with the
transition. I know someone posted an example project a while ago with
some scripts but my sense is that those scripts were particular to that
project and maybe not generally applicable.

Once the monorepo goes live (tomorrow?), what happens to the existing
subproject mirrors? Do they get wiped away and replaced with history
from the monorepo? Or are entirely new mirrors created? Or do they
just continue to mirror SVN until SVN becomes read-only?

After Tuesday, the existing sub-projects mirrors will continue to mirror SVN
as they do today. It's undecided what will happen once SVN becomes read-only.

-Tom

When I view monorepo, the google stable branch have the problem that always delete all the contents, and then copy the stable branch.
May us skip those delte branch actions.
Refere to


SHA-1: 00353aa9741e8419a0ab140559fd6fdfaaac04af

* Updating branches/google/stable to r309660

llvm-svn=309869
llvm-svn=309867
llvm-svn=309865
llvm-svn=309863
llvm-svn=309861
llvm-svn=309859


SHA-1: 6546fed885dc829d0e6b3cffab9b1b5c540c2dd8

* Cleaning up stable branch

llvm-svn=309857


* Updating branches/google/stable to r308006

llvm-svn=308373
llvm-svn=308371
llvm-svn=308369
llvm-svn=308367
llvm-svn=308365
llvm-svn=308363


SHA-1: 412992b89c7a09aaa13c503213d4238b8c657f4c

* Cleaning up stable branch

llvm-svn=308361

James and Tom,

One question. Would the monorepo be fixed and would it not be re-created after the day?

I am deeply cosmetic-checking the repo. For now, I haven’t found critical issues, though.

Takumi

It would be helpful to have instructions about the best way to move local branches in those repositories to the monorepo and some scripts to help with the transition.

I put together https://reviews.llvm.org/D53414.

At this point, Tuesday is definitely not going to happen. I’ll recreate the prototype tomorrow to fix the issue you discovered, and aim for Thursday to make it final unless more changes are needed.

But yes, the goal is to, very soon, declare the conversion is “final”, publish it at the official url, and no longer recreate it even if we find a minor deficiency.

This will allow people to start converting their internal processes to use it and depend on hashes safely. So, if you or anyone else wants to suggest changes, now’s the best time.

Thanks! In our case we have a branch off master for each component and
we've developed there, merging from master from time to time (but never
our local branch to master). I guess that puts us in the "multirepo
with merges" category. We don't really care about maintaining the
history of merges from master. Even rebasing our changes on the curent
monorepo master would probably be fine (understanding that there will be
merge conflicts we'll need to resolve).

The biggest challenge will be corrdinating changes to subprojects that
depend on each other. We don't want to just rebase all changes from
subproject X and then all changes from subproject Y, because we'll end
up with unbuildable history for most of the resulting commits. In the
foggy days of the past, I did our svn-to-git conversion and we had
similar needs, merging multiple svn repositories into one git
repository. I had to carefully craft scripts to get the commit ordering
right. I could probably do the same much more easily with git.

I've not used subtree merges in quite the way described so we'll have to
experiment a bit and see what works.

                               -David

Justin Lebar <jlebar@google.com> writes:

Looks like the prototype repository [1] is missing quite a lot of tags, like all of the RELEASE_NNN tags [2].

[1] https://github.com/llvm-git-prototype/llvm
[2] https://llvm.org/svn/llvm-project/cfe/tags/

Apparently the GitHub UI is not great about showing tags. If you clone
and do git tag you'll see them. Someone at one of the dev. meeting
roundtables found a way to see them on GitHub but I don't remember the
details.

                      -David

Jacob Carlborg via llvm-dev <llvm-dev@lists.llvm.org> writes:

They are not shown in the project graph, but if you open the “branch” drop down it has a tab named ‘Tags’.

I see that now, thanks. Sorry for the noise.

It shows some tags there, but not all of them. But clicking "releases" then "Tags" will show this page [1], which seems to include all of them.

[1] https://github.com/llvm-git-prototype/llvm/tags

What’s the status here?

Can someone keep https://llvm.org/docs/Proposals/GitHubMove.html updated with the current status of things?

And once things are usable, probably update https://llvm.org/docs/GettingStarted.html#for-developers-to-work-with-a-git-monorepo as well.

Some status update wrt GitHub SVN bridge.

It does not work for any non-trivial (= LLVM) repo. I filled the issue
there, however, there is no ETA when it will be fixed. Even worse,
there are no promises that the issue will be addressed at all. Though
they are aware that this is the issue for us.

It’d be nice to know what about our repository is breaking it. Do they have any idea what that is?

For example – I think that we probably will want to archive+discard many of the random branches and tags currently in the repository. If the large number of branches and tags is breaking it, then maybe it just starts working after we do so.