svnsync of llvm tree

I'm trying to setup a svnsync copy of the entire llvm svn tree.

I get the svnsync repository initialized, and then the first
67357 revisions get copied without issue.

And then it attempts to get rvisions 67358 from llvm.org, and
it fails. Repeatably. It transfers about 22MB of data for
this revision, and then dies:

lidl@hydra-146: svnsync sync file:///p/llvm/svn
Transmitting file data .......
[...]
......svnsync: REPORT of ‘http://llvm.org/svn/llvm-project/llvm’: Could not read chunk size: connection was closed by server (http://llvm.org)

Is there a svnsync seed file anywhere that has this revision
available? If not, can one be created?

Alternatively, can someone investigate the issue with the SVN server
and attempt to figure out what is failing on this revision?

It's completely repeatable. Thanks.

-Kurt

Hello Kurt,

lidl@hydra-146: svnsync sync file:///p/llvm/svn
Transmitting file data .......
[...]
......svnsync: REPORT of ‘http://llvm.org/svn/llvm-project/llvm’: Could not read chunk size: connection was closed by server (http://llvm.org)

You cannot checkout the whole llvm repository. Only trunk or separate
tags/branches.

Why can't I mirror the whole tree?

I'm not doing a "checkout" of the tree (i.e. svn co .../head), but rather
using the svnsync command (packaged with svn) to mirror the entire
repository.

I've done it with FreeBSD's SVN tree, which is very handy for having
lots of different branches checked out locally on a variety of different
machines. The entire FreeBSD SVN repository for that is about 5.3GB, and has
224948 revisions in it. (My mirror last updated yesterday.)

Outside of the LLVM SVN server choking on revision 67358 of the
tree (which appears to be size related and nothing else), the mirroring of
the entire repository was working just fine. I got the first half
(by revision count) before this problem cropped up.

-Kurt

The original reason for this limitation was that complete checkouts
put a lot of stress on the server. We upgraded the server hardware
since but I'm not sure whether we can lift the restriction.

However, we have official git mirrors of most of the projects on
llvm.org, would using them instead of svnsync be an option for you?

- Ben

> Why can't I mirror the whole tree?

The original reason for this limitation was that complete checkouts
put a lot of stress on the server.

While I agree bootstrapping a new mirror from scratch is time and
resource consuming, this can be largely avoided by making a seed
of the svn tree available. (Basically, I think that looks like
a compressed tar file of the svn repository.) Or, a svnadmin dump
of the repo could be made available for the adventurous.

I was under the impression that the overhead to synsync a mirror
of a complete svn repository wasn't too great. svnsync keeps track
of the last revision in a repo that it has successfully fetched,
and after that, will only ever request the changes newer than that.

We upgraded the server hardware
since but I'm not sure whether we can lift the restriction.

However, we have official git mirrors of most of the projects on
llvm.org, would using them instead of svnsync be an option for you?

Well, if the authoritative source code control system for all the
llvm projects is svn, I'd just as soon use svn as the tool at my end.
Having multiple source code control systems pulling and converting
to their own internal storage formats doesn't gain me a lot. I suppose
I can look into using the git mirror of the llvm tree.

However, I think if the resources on the apache server were increased
modestly, the svnsync would work just fine too.

-Kurt

The page at http://llvm.org/docs/GettingStarted.html says that the git
mirrors only include svn trunk. Is that still the case?

If so, it's not a particularly good option for anyone who is
interested in tracking more than just the single line of development.

While I agree bootstrapping a new mirror from scratch is time and
resource consuming, this can be largely avoided by making a seed
of the svn tree available. (Basically, I think that looks like
a compressed tar file of the svn repository.) Or, a svnadmin dump
of the repo could be made available for the adventurous.

Having the same problem as Kurt, I'd like to second his request for a
seed file. The dump seems the more sensible approach, although the tar
will work as well under most circumstances.

Some insight on how FreeBSD does it, can be found here:

  <http://wiki.freebsd.org/SubversionPrimer#Setting_up_a_svnsync_mirror>

Such a seed would have to be updated every now an then, but not too often.

rsync would be another alternative, which I have been using to clone
some repos from SF.net for some time now. They offer this option for
backup purposes.

I was under the impression that the overhead to synsync a mirror
of a complete svn repository wasn't too great. svnsync keeps track
of the last revision in a repo that it has successfully fetched,
and after that, will only ever request the changes newer than that.

Indeed, it does.

However, we have official git mirrors of most of the projects on
llvm.org, would using them instead of svnsync be an option for you?

Well, if the authoritative source code control system for all the
llvm projects is svn, I'd just as soon use svn as the tool at my end.

The git mirrors only contain what's in trunk in the SVN repos.

// Oliver

Oliver Schneider <gmane@assarbad.net> writes:

However, we have official git mirrors of most of the projects on
llvm.org, would using them instead of svnsync be an option for you?

Well, if the authoritative source code control system for all the
llvm projects is svn, I'd just as soon use svn as the tool at my end.

The git mirrors only contain what's in trunk in the SVN repos.

I think it contains branches of several previous releases:

$ git branch -r
  llvm-upstream/master
  llvm-upstream/release_1
  llvm-upstream/release_16
  llvm-upstream/release_20
  llvm-upstream/release_21
  llvm-upstream/release_22
  llvm-upstream/release_23
  llvm-upstream/release_24
  llvm-upstream/release_25
  llvm-upstream/release_26
  llvm-upstream/release_27
  llvm-upstream/release_28
  llvm-upstream/release_29
  llvm-upstream/svn-tags/RELEASE_1
  llvm-upstream/svn-tags/RELEASE_20
  llvm-upstream/svn-tags/RELEASE_21
  llvm-upstream/svn-tags/RELEASE_22
  llvm-upstream/svn-tags/RELEASE_23
  llvm-upstream/svn-tags/RELEASE_24
  llvm-upstream/svn-tags/RELEASE_25
  llvm-upstream/svn-tags/RELEASE_26
  llvm-upstream/svn-tags/RELEASE_27
  llvm-upstream/svn-tags/RELEASE_28
  llvm-upstream/svn-tags/RELEASE_29

It's possible I set that up myself. I can't remember.

                           -Dave

no, you didn't. The git repo contains those branches.

-Jeff