Repository Layout

All,

I'm finding the current layout a bit frustrating, not to mention that it
breaks all the documentation. We currently have modules laid out like
this:

   llvm/trunk/...
   llvm/branches/...
   llvm/tags/...

   stacker/trunk/...
   stacker/branches/...
   stacker/tags/...

and similarly for all the other modules (test-suite, java, poolalloc,
etc.). Unfortunately, this "standard" layout interjects a directory
(trunk/branches/tags) below each module name. I would rather see it laid
out like this:

   trunk/llvm/...
   trunk/stacker/...
   trunk/java/...

   branches/llvm/...
   branches/stacker/...
   branches/java/...

   tags/llvm/...
   tags/stacker/...
   tags/java/..

Fortunately, with SVN, making such a change isn't hard and its tracked
in the commits. Anyone object or have other ideas in this realm?

Reid.

Reid,

This is a common problem with Subversion. I wouldn’t recommend using the layout you suggested here, though. Although it’s “just” a convention, please keep the standard layout at the root of each project. (You’re proposing interceding a directory layer between trunk, branches, and tags, etc. I don’t think you’ll find that it solves many management problems.)

If the projects are too granular, merge them together. It might, in fact, be appropriate to use a single project root for the totality of llvm. Getting this correct is by far the most important decision to be made, IMO. It’s very easy to check out a subdirectory of a project, but much harder to check out a union of several projects.

However, if a union of several projects is necessary, Subversion does have an svn:external property[1] which can be used to create precisely that. It’s much more complicated to tag or branch such a beast, however, so it is strongly recommended that such a thing not be used as a unit of release.

Anton, as Reid mentioned, there’s absolutely no need to worry about wasting repository space with copies. However, since copies are so incredibly cheap on the server, there is a need to worry about wasting developer hard disk space. At the client, copies are much more expensive with Subversion than with CVS, so your intuition is still very useful.

— Gordon

[1] http://svnbook.red-bean.com/en/1.0/ch07s03.html

Hi Gordon,

> I'm finding the current layout a bit frustrating, not to mention
> that it breaks all the documentation. [...] I would rather see it
> laid out like this:
>
>
> >
> trunk/llvm/...
> trunk/stacker/...
> trunk/java/...
>
>
> branches/llvm/...
> branches/stacker/...
> branches/java/...
>
>
> tags/llvm/...
> tags/stacker/...
> tags/java/..

Reid,

This is a common problem with Subversion. I wouldn't recommend using
the layout you suggested here, though. Although it's "just" a
convention, please keep the standard layout at the root of each
project. (You're proposing interceding a directory layer between
trunk, branches, and tags, etc. I don't think you'll find that it
solves many management problems.)

I wasn't trying to solve any management problems. I see both schemes
(project at top vs. trunk/tags/branches at top) as having basically the
same management issues. What I was trying to achieve was:

a) a single URL for checking out all llvm-project software. Putting
trunk/tags/branches at the top would give us:
http://llvm.org/svn/llvm-project/trunk as the root of all software.
Without this, you would have one root for each sub-projects (llvm,
stacker, java, ...)

b) not having to fix all the documentation that references paths to
insert a "/trunk" into the path.

If the projects are too granular, merge them together.

Perhaps, but that's not related to the issue at hand.

It might, in fact, be appropriate to use a single project root for the
totality of llvm.

We have that: http://llvm.org/svn/llvm-project.

Getting this correct is by far the most important decision to be
made, IMO. It's very easy to check out a subdirectory of a project,
but much harder to check out a union of several projects.

if tags/trunk/branches was at the top level then getting the "union of
several projects" is achieved by just checking out
"trunk" (http://llvm.org/svn/llvm-project/trunk). Again, I'm not sure
how this is relevant to the original issue. Am I missing something?

However, if a union of several projects is necessary, Subversion does
have an svn:external property[1] which can be used to create precisely
that. It's much more complicated to tag or branch such a beast,
however, so it is strongly recommended that such a thing not be used
as a unit of release.

I'm aware of svn:external and I don't think we should be using it for
internal repositories. We *might* want to use it to integrate another
repository, but I know of no such need currently and can't see it
arising in the future.

Anton, as Reid mentioned, there's absolutely no need to worry about
wasting repository space with copies. However, since copies are so
incredibly cheap on the server, there is a need to worry about wasting
developer hard disk space. At the client, copies are much more
expensive with Subversion than with CVS, so your intuition is still
very useful.

Yes, on the client side, copies are definitely more expensive. This just
underscores the need to set up a way to get "trunk" easily and avoid all
the copies in tags and branches. For example, consider doing something
like this:

svn co http://llvm.org/svn/llvm-project

Right now, what that will get is you is the trunk of all projects and
all the tags and branches for all those projects. Clearly this isn't
desirable. For just the llvm sub-project, this would involve checking
out 28 copies of the llvm source code.

If we had trunk/tags/branches at the top then it is much easier to get
the single copy of the latest software with just:

svn co http://llvm.org/svn/llvm-project/trunk

So, in summary, I'm not quire sure I understand your original caution
about using this approach. What are the management headaches that we
would incur? Are they any higher than putting trunk/tags/branches one
level lower (by having the module name first)? A little more detail
would be helpful.

Thanks,

Reid.

I'm finding the current layout a bit frustrating, not to mention
that it breaks all the documentation. [...] I would rather see it
laid out like this:

  trunk/llvm/...
  trunk/stacker/...
  trunk/java/...

  branches/llvm/...
  branches/stacker/...
  branches/java/...

  tags/llvm/...
  tags/stacker/...
  tags/java/..

Reid,

This is a common problem with Subversion. I wouldn't recommend using
the layout you suggested here, though. Although it's "just" a
convention, please keep the standard layout at the root of each
project. (You're proposing interceding a directory layer between
trunk, branches, and tags, etc. I don't think you'll find that it
solves many management problems.)

I wasn't trying to solve any management problems. I see both schemes
(project at top vs. trunk/tags/branches at top) as having basically the
same management issues. What I was trying to achieve was:

a) a single URL for checking out all llvm-project software. Putting
trunk/tags/branches at the top would give us:
http://llvm.org/svn/llvm-project/trunk as the root of all software.
Without this, you would have one root for each sub-projects (llvm,
stacker, java, ...)

But, unless I misunderstand your suggested layout, that benefit would not extend to branches or tags, because you placed the project as the parent directory of the tag. This makes branches and tags much less useful, because they cannot be used symmetrically with the trunk(s) or created with a simple svn copy command, should more than one subfolder need to be tagged.

If the projects are too granular, merge them together.

Perhaps, but that's not related to the issue at hand.

By "project", I was referring solely to the layout in source control, not any source code integration. Sorry for the imprecise language. I shall use "SVNROOT" instead.

It might, in fact, be appropriate to use a single project root for the
totality of llvm.

We have that: http://llvm.org/svn/llvm-project.

As you note, that directory cannot be (practically) checked out because it will contain all branches and tags. I'm referring to a folder which transparently references other projects as a shortcut for checking out all of the trunks manually, precisely the scenario described in the Red Book.

Getting this correct is by far the most important decision to be
made, IMO. It's very easy to check out a subdirectory of a project,
but much harder to check out a union of several projects.

if tags/trunk/branches was at the top level then getting the "union of
several projects" is achieved by just checking out
"trunk" (http://llvm.org/svn/llvm-project/trunk).

Yes, exactly. But in your proposed layout, that only works for the trunk. That's because you flipped the path from SVNROOT/tags/tagname/folder to SVNROOT/tags/folder/tagname.

Again, I'm not sure
how this is relevant to the original issue. Am I missing something?

However, if a union of several projects is necessary, Subversion does
have an svn:external property[1] which can be used to create precisely
that. It's much more complicated to tag or branch such a beast,
however, so it is strongly recommended that such a thing not be used
as a unit of release.

I'm aware of svn:external and I don't think we should be using it for
internal repositories. We *might* want to use it to integrate another
repository, but I know of no such need currently and can't see it
arising in the future.

Anton, as Reid mentioned, there's absolutely no need to worry about
wasting repository space with copies. However, since copies are so
incredibly cheap on the server, there is a need to worry about wasting
developer hard disk space. At the client, copies are much more
expensive with Subversion than with CVS, so your intuition is still
very useful.

Yes, on the client side, copies are definitely more expensive. This just
underscores the need to set up a way to get "trunk" easily and avoid all
the copies in tags and branches. For example, consider doing something
like this:

svn co http://llvm.org/svn/llvm-project

Right now, what that will get is you is the trunk of all projects and
all the tags and branches for all those projects. Clearly this isn't
desirable. For just the llvm sub-project, this would involve checking
out 28 copies of the llvm source code.

If we had trunk/tags/branches at the top then it is much easier to get
the single copy of the latest software with just:

svn co http://llvm.org/svn/llvm-project/trunk

So, in summary, I'm not quire sure I understand your original caution
about using this approach. What are the management headaches that we
would incur. Are they any higher than putting trunk/tags/branches one
level lower (by having the module name first)? A little more detail
would be helpful.

All I'm saying is this:

1. You'll cause yourself headaches unless the contents of $SVNROOT/trunk is symmetrical with $SVNROOT/{tags,branches}/$name.

2. It sounds like you would be happier with fewer SVNROOTs.

But you should also consider whether this is simply a growing pain; cvs and svn are fundamentally different in that Subversion makes the management structure visible in the URL.

- Gordon

I strongly dislike this approach. This is particularly bad because we have many subprojects. Lots of these subprojects are only loosly coupled to LLVM: things like llvm-tv, llvm-java, llvm-poolalloc, etc.

I think that it is natural for subprojects to go through various development phases and to sink or swim based on their own merits and the contributor base around them. Our goal should be to make it as easy as
possible to check out the desired subset of LLVM and get going quickly.

Instead of trying to make a mega project, I suggest something else. What I'd really like to see is this:

1. I check out the "llvm-project" into a directory. This provides some
    very very basic makefiles and configure script stuff. This does not
    try to solve all the worlds problems, it is just some simple glue.
2. Based on my interests, I check out a subset of the subprojects into
    directories under llvm-project.
3. Each subproject knows the projects it depends on - either as a hard
    dependency (requirement) or a soft dependency (optional component).
4. Once the stuff is checked out, you type "configure". Based on the glue
    in #1, this checks to ensure that the various dependencies required are
    checked out. If not, it errors out, telling you what else to check out
    (or offering to do it for you?)
5. make, make test, make install, etc are provided by the glue. These
    simply recurse into the subdirectories in the appropriate order and
    invoke the respective targets.

With this design, I'd suggest refactoring the LLVM project a little bit. I'd like to see:

1. a subproject for libsupport and libsystem.
2. a subproject for the rest of the "llvm" project. sometime later we can
    separate out vmcore perhaps.
3. a subproject for llvm-gcc, which depends on #1 and #2.
4. a subproject for the new c front-end, which has a hard requirement on
    #1 and a optional requirement for #2.
5. HLVM, llvm-tv, etc (the ones other people care about :slight_smile:

One of the big in-between things is llvm-config. I've grown to *love* this tool, but it only works for the main llvm project. I'd like to see it move up and works as part of the glue, so that it is easy to link/include anything that is part of the "llvm-project".

The overall goal is to make the system more modular (NOT more monolithic) and to make it easier to get stuff started: it should be enough to check out the llvm-project glue (very very quick checkout) and run "make get-me-llvm-gcc-and-build-it" :slight_smile:

As an aside, I've been working on a revamp of the website, to make *it* more modular, which is also required for the docs. I think it's amazing(ly bad) that we don't have a documentation subsection for llvm-gcc anywhere!

Thoughts?

-Chris

I think the trunk/subproject layout is inappropriate, since it also
implies branches/name/subprojects, where a branch is likely to apply
only to a single subproject.

how about a top-level llvm-and-kitchen-sink module which simply
contains svn:externals to the trunks of all the subprojects, allowing
a monolithic checkout for those who want it, and separate modules for
better branching and more isolated development?

This would also provide the "easy (but bandwidth consuming) to get
started" module

Reid,

All,

I'm finding the current layout a bit frustrating, not to mention that it
breaks all the documentation. We currently have modules laid out like
this:

  llvm/trunk/...
  llvm/branches/...
  llvm/tags/...

  stacker/trunk/...
  stacker/branches/...
  stacker/tags/...

and similarly for all the other modules (test-suite, java, poolalloc,
etc.). Unfortunately, this "standard" layout interjects a directory
(trunk/branches/tags) below each module name. I would rather see it laid
out like this:

  trunk/llvm/...
  trunk/stacker/...
  trunk/java/...

  branches/llvm/...
  branches/stacker/...
  branches/java/...

  tags/llvm/...
  tags/stacker/...
  tags/java/..

Fortunately, with SVN, making such a change isn't hard and its tracked
in the commits. Anyone object or have other ideas in this realm?

Your suggestion makes life difficult for someone who wants to mirror
one subproject. Once, llvm-gcc is using svn at llvm.org, most likely
I will create a local mirror to follow llvm-gcc development. In that case
it is easier to mirror

  .../llvm-gcc

then
  
  ../trunk/llvm-gcc/
  ../branches/llvm-gcc/
  ../tags/llvm-gcc

Devang Patel wrote:
> Your suggestion makes life difficult for someone who wants to mirror

one subproject. Once, llvm-gcc is using svn at llvm.org, most likely
I will create a local mirror to follow llvm-gcc development. In that
case
it is easier to mirror

  .../llvm-gcc

then
  
  ../trunk/llvm-gcc/
  ../branches/llvm-gcc/
  ../tags/llvm-gcc

+1 strong agreement from another local mirror-er.

-scooter

Hi Scott,

Devang Patel wrote:
> Your suggestion makes life difficult for someone who wants to mirror
> one subproject. Once, llvm-gcc is using svn at llvm.org, most likely
> I will create a local mirror to follow llvm-gcc development. In that
> case
> it is easier to mirror
>
> .../llvm-gcc
>
> then
>
> ../trunk/llvm-gcc/
> ../branches/llvm-gcc/
> ../tags/llvm-gcc

We backed off this plan. Please see http://llvm.org/SVNLayout.html

Reid.