Reminder: LLVM 2.9 Branching in One Week

This is a reminder that we will be branching for LLVM 2.9 in one week!

07:00:00 p.m. Sunday March 6, 2011 PST / 0****3:00:00 a.m. Monday March 7, 2011 GMT

What this means for you:

Please keep a watch on all of your patches going into mainline. And pay close attention to the buildbots and fix any issues quickly.

Also, please try to finish up any last minute feature work. While it won’t be the last time to submit a patch for a work-in-progress feature, the more work and testing that can be done before the branching means that the release verification process will go that much more smoothly.

LLVM 2.9 is going to be an exciting release. It marks the end of an era in some respects, but the beginning of many new ones to come! :slight_smile:

-bw

Hi Bill,
Will the 2.9 branch be reflected in the git mirrors?

Thanks,
Chad

Chad Colgur <colgur@gmail.com> writes:

Will the 2.9 branch be reflected in the git mirrors?

I really, really hope this is the case. The official git mirror
documentation says it is only trunk because the current branching scheme
in not suitable for git. I don't understand why that would be but I'm
not a git-svn expert.

My very short reading of the git-svn manpage makes me think a git-svn
branch from the git mirror would make this happen. So maybe the svn
branch should be created that way?

                                -Dave

It doesn't work because git-svn assumes that all of your branches begin at the same position in the SVN tree, i.e. llvm/branches/*. The problem is that we have some branches in llvm/branches/*, some in llvm/branches/Apple/*, some in llvm/branches/ggrief/*, etc. The end result is that git-svn gets horribly confused.

--Owen

Owen Anderson <resistor@mac.com> writes:

It doesn't work because git-svn assumes that all of your branches
begin at the same position in the SVN tree, i.e. llvm/branches/*. The
problem is that we have some branches in llvm/branches/*, some in
llvm/branches/Apple/*, some in llvm/branches/ggrief/*, etc. The end
result is that git-svn gets horribly confused.

But all of the public-facing branches (I think RELEASE_* are the only
branches most people care about) are in llvm/tags/* (or will be in
llvm/tags/RELEASE_29/* for 2.9), right?

So I guess it will be a problem if 3.0 is in llvm/tags/RELEASE_30/*.
This is perhaps one reason not to use the branching scheme Bill
proposed. That's not a very satisfactory result.

                               -Dave

Owen Anderson <resistor@mac.com> writes:

It doesn't work because git-svn assumes that all of your branches
begin at the same position in the SVN tree, i.e. llvm/branches/*. The
problem is that we have some branches in llvm/branches/*, some in
llvm/branches/Apple/*, some in llvm/branches/ggrief/*, etc. The end
result is that git-svn gets horribly confused.

What options were used with git-svn init? It certainly can support
multiple branch/tag directories:

init
  --tags=<tags_subdir>
  --branches=<branches_subdir>

  You can specify more than one --tags and/or --branches options, in
  case your Subversion repository places tags or branches under multiple
  paths.

                              -Dave

I don't remember exactly what we tried, but as I recall it was the fact that some of the branch directories were nested inside another branch directory that got it all confused.

--Owen

I really, really hope this is the case. The official git mirror
documentation says it is only trunk because the current branching scheme
in not suitable for git. I don't understand why that would be but I'm
not a git-svn expert.

It doesn't work because git-svn assumes that all of your branches begin at the same position in the SVN tree, i.e. llvm/branches/*. The problem is that we have some branches in llvm/branches/*, some in llvm/branches/Apple/*, some in llvm/branches/ggrief/*, etc. The end result is that git-svn gets horribly confused.

Same for tags. And current 2-level system will make stuff even worse
as it seems :slight_smile:

What options were used with git-svn init? It certainly can support
multiple branch/tag directories:

init
--tags=<tags_subdir>
--branches=<branches_subdir>

You can specify more than one --tags and/or --branches options, in
case your Subversion repository places tags or branches under multiple
paths.

The svn repository is free to checkout. So, if you'll find a
reasonable combination of cmdline options (which will provide release
branches / tags only), then please let us know and we'll add the
branches/tags to the repository as well.

I am by far not a 'git' expert. (Every time I use git, I mess something up.) I encourage people to create a 2.9 branch in git if it's possible. But unfortunately I won't be able to do it automatically.

-bw

Anton Korobeynikov <anton@korobeynikov.info> writes:

What options were used with git-svn init? It certainly can support
multiple branch/tag directories:

init
--tags=<tags_subdir>
--branches=<branches_subdir>

You can specify more than one --tags and/or --branches options, in
case your Subversion repository places tags or branches under multiple
paths.

The svn repository is free to checkout.

At some point in the past, an anti-git-svn system had been set up on
llvm.org. Has this been disabled since? I don't manage to do much with
git-svn:

$ git svn clone http://llvm.org/svn/llvm-project/llvm -T trunk llvm-clone
Initialized empty Git repository in /tmp/llvm-clone/.git/
Using higher level of URL: http://llvm.org/svn/llvm-project/llvm =>
http://llvm.org/svn/llvm-project
RA layer request failed: Server sent unexpected return value (403
Forbidden) in response to REPORT request for
'/svn/llvm-project/!svn/vcc/default' at
/home/moy/local/usr/libexec/git-core/git-svn line 5131

(but I'm not really a git-svn expert either)

So, if you'll find a reasonable combination of cmdline options (which
will provide release branches / tags only), then please let us know
and we'll add the branches/tags to the repository as well.

At least one person did that (partly manually):

https://github.com/earl/llvm-mirror

Matthieu,

At some point in the past, an anti-git-svn system had been set up on
llvm.org. Has this been disabled since? I don't manage to do much with
git-svn:

Maybe sure. Anton said it is disabled to access upper directories with svn.
Thus, we (accessing llvm.org remotely) cannot do git-svn with branches
for homebrew.

In contrast, I suppose Anton might create branches with git-svn.
Or I guess Andreas (aka earl) might know howto.
(I guess, Andreas might manipulate lowlevel git to create a branch initially)

Andreas and Anton, could you please launch git release branches for llvm.org?

...Takumi

ps. afaik llvm.org/git/llvm.git is diverged to
https://github.com/earl/llvm-mirror.

Hi list

- snip -

At least one person did that (partly manually):

https://github.com/earl/llvm-mirror

FYI, I did same for clang here:
https://github.com/spurious/clang-mirror
(this repository is diverged to official one
because it includes some old history which official
git-mirror had omitted)

- snip -

ps. afaik llvm.org/git/llvm.git is diverged to
https://github.com/earl/llvm-mirror.

It is not a real problem. We can git-rebase beyond two diverged repository.
1. add preferred git-mirrors as remote and fetch, checkout preferred one.
2. (Branch, work on it, commit)
3. git rebase --onto official-llvm-git-mirror/master my-own-mirror/master..HEAD
4. git svn dcommit
will work. (This assumes the branch "my-own-mirror/master" and
"official-llvm-git-mirror/master"
point _same_ SVN revision.)

(Please refer git-rebase(1) "recovering from upstream rebase"/"hard
case" for more.)

But, of course, I also encourage llvm people to create a 2.9 branch in
official git mirror...

Thank you Yuki! I am using yours and earl's clones (via hg-git, I prefer Mercurial)

P.S. If someone wants these Mercurial URLs, please drop a line.

NAKAMURA Takumi <geek4civic@gmail.com> writes:

Matthieu,

At some point in the past, an anti-git-svn system had been set up on
llvm.org. Has this been disabled since? I don't manage to do much with
git-svn:

Maybe sure. Anton said it is disabled to access upper directories with svn.
Thus, we (accessing llvm.org remotely) cannot do git-svn with branches
for homebrew.

This is exactly right.

Andreas and Anton, could you please launch git release branches for llvm.org?

I think the trouble with branches is the lockdown of the root repository
directory. I tried something like this but it breaks due to the
restrictions:

git svn init --stdlayout https://<user>@llvm.org/svn/llvm-project/llvm \
  --ignore-paths="^.*(Apple|PowerPC.*|SVA|eh-experimental|ggreif|non-call-eh|parallel|release_.*|vector_llvm|wendling|May2007|checker|cremebrulee|start|RELEASE_1.*|RELEASE_2[0-7])"

Obviously, replace <user> with whatever it needs to be to allow dcommit
to work.

Ideally we'd have clang and llvm-gcc git mirrors as well via the
--prefix argument to git-svn init, but let's not get ahead of ourselves.
:slight_smile:

It appears that there's not much those of us who would like git branches
for svn release tags can do without help from the server side.

                            -Dave

Small correction: The branching will happen at 7PM today.

Please watch the build bots like a hawk. :slight_smile:

-bw

Hi David

I think the trouble with branches is the lockdown of the root repository
directory.

Surely not (at the server)

git svn init --stdlayout https://<user>@llvm.org/svn/llvm-project/llvm \
--ignore-paths="^.*(Apple|PowerPC.*|SVA|eh-experimental|ggreif|non-call-eh|parallel|release_.*|vector_llvm|wendling|May2007|checker|cremebrulee|start|RELEASE_1.*|RELEASE_2[0-7])"

Several problems here:
1. Bunch of additional branches / tags are created due to multiple
branch points. I don't recall for llvm, but for clang we'll end with
two tags per each release. Something like:
$ git branch -r
  trunk
  tags/RELEASE_26
  tags/RELEASE_26@84939
  tags/RELEASE_27
  tags/RELEASE_27@102415
  tags/RELEASE_28
  tags/RELEASE_28@115869

The problem will be much worse with new release branch scheme,
basically we'll need to add each branch by hand, etc...
2. We really don't want to push arbitrary branches to git repository.
It's really easy to add branch by an accident, so it will be much
better not to ignore stuff, but except - add by some pattern.
Unfortunately, git-svn does not allow this yet.

So, right now I'm experimenting with various ways of doing stuff, but
the results looks not pretty good.
If anything would give me a working all-in-one cmdline / .git/config
entry - I'd really appreciate this :slight_smile:

Anton Korobeynikov <anton@korobeynikov.info> writes:

Hi David

I think the trouble with branches is the lockdown of the root repository
directory.

Surely not (at the server)

Yes, but ordinary users can't try to experiment and find a set of
options that work as long as the server is locked down. So we have to
go through server admins, which seems inefficient. But, I'm ok with
that if we're making progress.

git svn init --stdlayout https://<user>@llvm.org/svn/llvm-project/llvm \
--ignore-paths="^.*(Apple|PowerPC.*|SVA|eh-experimental|ggreif|non-call-eh|parallel|release_.*|vector_llvm|wendling|May2007|checker|cremebrulee|start|RELEASE_1.*|RELEASE_2[0-7])"

Several problems here:
1. Bunch of additional branches / tags are created due to multiple
branch points. I don't recall for llvm, but for clang we'll end with
two tags per each release. Something like:
$ git branch -r
  trunk
  tags/RELEASE_26
  tags/RELEASE_26@84939
  tags/RELEASE_27
  tags/RELEASE_27@102415
  tags/RELEASE_28
  tags/RELEASE_28@115869

Yep. But is this really a problem? All of these tags and branches must
be useful, or why create them? If they aren't useful, add them to the
--ignore-paths list.

The problem will be much worse with new release branch scheme,
basically we'll need to add each branch by hand, etc...

Why? Ultimately the top level of llvm looks like the standard
subversion layout:

http://llvm.org/svn/llvm-project/llvm/
  # branches/
  # tags/
  # trunk/

2. We really don't want to push arbitrary branches to git repository.
It's really easy to add branch by an accident, so it will be much

You mean add a brach through the git-svn mirror via "git svn branch?"
How can someone do that without having permission on the server? This
seems like an svn permissions issue to me.

Or are you worried about some svn user adding a branch? Again, this
seems like a server permissions issue. The LLVM policy is "no branches"
so why don't we enforce it and only allow the release manager to create
them?

better not to ignore stuff, but except - add by some pattern.
Unfortunately, git-svn does not allow this yet.

That certainly would be a useful feature. It shouldn't be hard to
implement as it's just a perl script. Getting it accepted upstream and
pushed out to clients is the bigger problem.

So, right now I'm experimenting with various ways of doing stuff, but
the results looks not pretty good.

What's not good about them?

So far my experience is that git svn does a really good job following
svn branches and making them available via git. See for example this
posting:

http://www.jukie.net/bart/blog/svn-branches-in-git

More experimentation is necessary, though. I'm still pretty new at
this.

If anything would give me a working all-in-one cmdline / .git/config
entry - I'd really appreciate this :slight_smile:

I just blew mine away to try something else. :slight_smile: But it did work well
before. I'll send the new one to you when it's ready.

I'm playing around with some git-svn stuff here with our repositories.
If I find anything interesting I'll let you know.

I believe the key to making this work is to have a separate "svn commit"
branch in the user's git clone so that people who do dcommit always do
it from there. That way git svn rebase won't screw up their "normal"
git branches. I've tried a bit of this and it seems if one isolates the
git svn stuff to its own branch, things work pretty smoothly.

Graphically:

                       svn
                        > git svn init/fetch
                        V
                    git-svn clone
                        > git clone
                        V
                   user's clone
                    / \
                   / \
                master commit

The user does a "git pull" into their local master, works on it, creates
local branches, etc. just as a normal git user would. Once something is
ready to go upstream, the user does the following:

git commit -a (commit to local git master)
git checkout commit
git merge master
git svn rebase
git svn dcommit

Then the next git clone to master will pick up the change history and
not conflict with the change as it exists in the user's local master.

I did this with the existing LLVM git mirror on my most recent commit to
verify that there were no issues. It was a simple change to a component
that doesn't change much, but I'm going to get some more experience with
this in the coming days and weeks.

I set things up a suggested by Tobias Grosser:

http://permalink.gmane.org/gmane.comp.compilers.clang.devel/12843

It works well for trunk. I think it should work equally well if/when
branches are added to the git mirror. Since no one should be committing
to branches except the release manager, everything will always go
upstream through the local "commit" branch. This keeps the svn metadata
sane.

From my point of view, the main reason to add the branches to the git

mirror is to *greatly* ease the burden for third parties when they
upgrade to a new release AND when they send patches upstream. The git
branch/merge/conflict resolution process is just killer for this. I
imagine most third parties don't work off of trunk. We certainly don't.

                                -Dave

Hi David

I think the trouble with branches is the lockdown of the root repository
directory.

Surely not (at the server)

git svn init --stdlayout https://<user>@llvm.org/svn/llvm-project/llvm \
  --ignore-paths="^.*(Apple|PowerPC.*|SVA|eh-experimental|ggreif|non-call-eh|parallel|release_.*|vector_llvm|wendling|May2007|checker|cremebrulee|start|RELEASE_1.*|RELEASE_2[0-7])"

Several problems here:
1. Bunch of additional branches / tags are created due to multiple
branch points. I don't recall for llvm, but for clang we'll end with
two tags per each release. Something like:
$ git branch -r
   trunk
   tags/RELEASE_26
   tags/RELEASE_26@84939
   tags/RELEASE_27
   tags/RELEASE_27@102415
   tags/RELEASE_28
   tags/RELEASE_28@115869

The problem will be much worse with new release branch scheme,
basically we'll need to add each branch by hand, etc...
2. We really don't want to push arbitrary branches to git repository.
It's really easy to add branch by an accident, so it will be much
better not to ignore stuff, but except - add by some pattern.
Unfortunately, git-svn does not allow this yet.

Why not? As far as I understand --ignore-paths takes a perl regular expression. So we could just provide a regular expression that matches on all paths except the ones we want to keep.

The following expression e.g.

/^.*(?<!trunk|RELEASE_2.).$/m

uses lookbehind to matches on:

   tags/SVA
   tags/eh-experimental
   tags/ggreif
   tags/non-call-eh
   tags/RELEASE_28@115869
   branches/Apple
   branches/PowerPC-A
   branches/PowerPC-B

but not on

   trunk
   tags/RELEASE_28
   tags/RELEASE_29
   tags/RELEASE_27

It probably needs some adjustments to get it right, but I believe this is basically a problem of finding the right regex.

So, right now I'm experimenting with various ways of doing stuff, but
the results looks not pretty good.
If anything would give me a working all-in-one cmdline / .git/config
entry - I'd really appreciate this :slight_smile:

I would love to. However, as David pointed out, this is difficult with the blocked svn access.

Cheers
Tobi

Hi Tobias,

The following expression e.g.

/^.*(?<!trunk|RELEASE_2.).$/m

uses lookbehind to matches on:

Thanks. Clever trick, but...

Variable length lookbehind not implemented in regex
m/^.*(?<!trunk|RELEASE_2.).$/ at /usr/lib/git-core/git-svn line 4078.

:frowning: