GIT mirroring

Hello Everyone,

It seems given the decent amount of discussions it's time to make
small announcement.

So, official git mirrors are available for some subset of LLVM
projects. They were used by some LLVM developers for couple of months
already and seem to be stable enough.

Right now there is a mirroring for LLVM itself and clang. More can be
added per requests. The mirrors are made via running of git-svn on
llvm.org inside post-commit hook, so they should be in theory as fresh
as possible. Also, one can recreate git-svn metadata and enable such
nice features like "git svn dcommit". The mirrors track only trunk
mostly due to not so git'ish way of doing branches / tags in LLVM
repository.

The URLs for clone are:

http://llvm.org/git/llvm.git
http://llvm.org/git/clang.git

It's not yet decided whether we'll run e.g. gitweb for mirrors, most
probably - there won't be anything like this, viewvc should be enough
for almost all purposes.

PS: Patches to various sorts of documentation scattered here and there
are welcome :slight_smile:

Thanks!

Cheers,
Rafael

Anton Korobeynikov <anton@korobeynikov.info> writes:

So, official git mirrors are available for some subset of LLVM
projects. They were used by some LLVM developers for couple of months
already and seem to be stable enough.

Woo hoo!

as possible. Also, one can recreate git-svn metadata and enable such
nice features like "git svn dcommit". The mirrors track only trunk

Yee haw! Is there some documentation on how to set this up for LLVM?

Thanks Anton and everyone who worked on this!

                               -Dave

Thank you for setting this up, Anton!

This is how I use the Git mirror with git-svn:

For the initial clone and setup:

$ git clone http://llvm.org/git/llvm.git
$ cd llvm
$ git config --add remote.origin.fetch '+refs/remotes/git-svn:refs/remotes/git-svn'
$ git fetch
$ git svn init https://llvm.org/svn/llvm-project/llvm/trunk
$ git svn rebase -l

This will quickly build the git-svn metadata by using the magical remotes/git-svn branch fetched from the origin.

To update I run:

$ git fetch
$ git svn rebase -l

And to commit:

$ git svn dcommit
$ git fetch
$ git svn rebase -l

I have sometimes seen git-svn refusing to dcommit, claiming that I have uncommitted files in my tree. I think this happens when I forget to resynchronize the metadata after committing. Anyway, the solution is to wipe away all of .git/svn and rebuild it with "git svn rebase -l"

/jakob

Thanks so much for setting this up, Anton!

I notice that .gitignore in the root has tools/clang. This makes it necessary to use ‘–force’ if using ‘git submodule’. I do not know if there are any drawbacks to this - if someone knows, it would be interesting to know what those drawbacks are.

One alternative would be for individuals who clone into tools/clone without using submodules to instead add tools/clang explicitly to .git/info/exclude.

Thanks again,

Mark

To rebuild, it would be enough to remove
.git/svn/refs/remotes/git-svn/.rev_map.*

My usual way to resync;

$ git fetch llvm.org (is remote name)

$ git update-ref refs/remotes/git-svn llvm.org/master

$ git svn fetch
Partial-rebuilding
.git/svn/refs/remotes/git-svn/.rev_map.91177308-0d34-0410-b5e6-96231b3b80d8
...
Currently at 124651 = 071d3af0de273b1079d79f7f979264f28d567373
r124653 = cde5a400dbc9655eddf0f383585d3cf67c11c539
...
r124704 = e349d2c61256f11180b7112ab592b4ae700f3dcf
Done rebuilding
.git/svn/refs/remotes/git-svn/.rev_map.91177308-0d34-0410-b5e6-96231b3b80d8

...Takumi

I have a small script (attached) in my $PATH so I can just do:

$ git svnmu llvm

git-svnmu (90 Bytes)

And I just changed

.git/config such that both fetch strings point to origin/master.

Good evening, Tobi.

I've been using Jakob's commands below, but it has stopped working for me. This happened roughly around the time when Anton added the other branches, but I'm not sure if that was the cause or not. The symptom is that the "git svn rebase -l" command does nothing except say that the master branch is already up to date, and "git svn dcommit" complains that files are out of date. In both cases, "git svn rebase" (without "-l") solves the problem, but without using the GIT mirror, so it's slow.

I've tried re-creating my git repos from scratch but that did not fix the problem. Any ideas?

The trick with 'git config --add remote.origin.fetch' doesn't work anymore. I tried changing it to refer to the remote's master branch instead, but that has caused strange problems.

I now run 'git update-ref' every time I fetch from the mirror:

  git fetch -p origin
  git update-ref refs/remotes/git-svn origin/master
  git svn rebase -l

Same for committing:

  git svn dcommit
  git fetch -p origin
  git update-ref refs/remotes/git-svn origin/master
  git svn rebase -l

For the initial clone and setup:

  git clone http://llvm.org/git/llvm.git
  cd llvm
  git svn init https://llvm.org/svn/llvm-project/llvm/trunk
  git update-ref refs/remotes/git-svn origin/master
  git svn rebase -l

/jakob

I've tried re-creating my git repos from scratch but that did not fix the problem. Any ideas?

The trick with 'git config --add remote.origin.fetch' doesn't work anymore. I tried changing it to refer to the remote's master branch instead, but that has caused strange problems.

I'm not a git-svn expert, but I suspect the real problem is that
git-svn automagically updates master from git-svn remote. Right now
we're exporting just master and thus stuff appears to form a cycle.
This is just a random thought though :slight_smile:

I've been using Jakob's commands below, but it has stopped working for me. This happened roughly around the time when Anton added the other branches, but I'm not sure if that was the cause or not. The symptom is that the "git svn rebase -l" command does nothing except say that the master branch is already up to date, and "git svn dcommit" complains that files are out of date. In both cases, "git svn rebase" (without "-l") solves the problem, but without using the GIT mirror, so it's slow.

I've tried re-creating my git repos from scratch but that did not fix the problem. Any ideas?

Yes, here the changes necessary:

$ git clone http://llvm.org/git/llvm.git
$ cd llvm
$ git config --add remote.origin.fetch '+refs/remotes/git-svn:refs/remotes/git-svn'
$ git fetch

Skip the last two lines line.

$ git svn init https://llvm.org/svn/llvm-project/llvm/trunk

Add here:
git config svn-remote.svn.fetch ':refs/remotes/origin/master'

$ git svn rebase -l

And here I personally just use a 'git svn fetch'

This is the complete sequence

$ git clone http://llvm.org/git/llvm.git
$ cd llvm
$ git svn init https://llvm.org/svn/llvm-project/llvm/trunk
$ git config svn-remote.svn.fetch ':refs/remotes/origin/master'
$ git svn fetch

LLVM trunk is now in origin/master and can be accessed e.g. by

$git log origin/master

It can be updated by using

$git remote update
$git fetch
$git pull

Cheers
Tobi

You should be able to get rid of this by calling:

git config svn-remote.svn.fetch ':refs/remotes/origin/master'

Now the default svn branch points to refs/remotes/origin/master instead of refs/remotes/git-svn and everything should work automatically.
Let me know if there are any problems with this approach.

Tobi

That would work too.

/jakob