OT: Cloning llvm repo over low speed connection != fun

I remember reading all those examples about how a big repo isn't an
issue and when sitting at some ${LOCATION} I have 1010101010Gbps and
it takes like 30 seconds.. Well.. that may be true when I'm at home,
but I'm not.. and this laptop doesn't have llvm sources installed..

So real world feedback of the pita it can be..
/* Side rant - I wish I didn't even need the llvm sources. I just want
to build libcxxrt */

/* Yes I'm about to do the smarter thing and fetch a shallow clone or
try to get the tarball snapshot from github.. I did however want to
show an example of how big repo on non-corporate infrastructure can be
a challenge. I'd hate for some ambitious community dev to run into the
same thing and turn away as a result */

Hi,

No matter what we do, I hardly think it likely we can optimize the LLVM clone speed for a 20KB/s connection. I agree that corporate internet connections being gigantic pipes means we run the risk of forgetting what home infrastructure looks like, but 20KBps is… around dial-up. Where are you, rural Tibet?

Cheers,

James

Hah! Woo is me you say...

Have you ever stayed in a non-hilton hotel while traveling? I don't
mean couch surfing, but just average place.. I'd even add some above
average places to the list and when you get 100 people all on the same
line.. it can get "slow"...

Lets also factor in networking policies that favor normal internet
traffic.. So they may allow youtube to stream like normal, but ssh or
something else gets throttled a lot.. (which is what's happening in my
case)

I wish I was in Tibet.. I'll have to add that to my things to try if I
visit some day.. :wink:

You may mock me and think this can't ever happen, but I'm happy to
benchmark real world connection speeds while traveling.

I remember reading all those examples about how a big repo isn't an
issue and when sitting at some ${LOCATION} I have 1010101010Gbps and
it takes like 30 seconds.. Well.. that may be true when I'm at home,
but I'm not.. and this laptop doesn't have llvm sources installed..

So real world feedback of the pita it can be..
/* Side rant - I wish I didn't even need the llvm sources. I just want
to build libcxxrt */

/* side rant: aren’t you the one complaining loudly about noisy non-technical thread on llvm-dev?? */

I've been to many places while going to Connects, events and sprints,
and I can attest that hotels, no matter how many stars, have horrible
Internet connection.

One hotel we had to tell the admin to stop throttling SSH and also to
allow connections to more ports than just 25, 80 and 443.

But that affected *everybody*. Using Git or SVN, using Windows or
Linux, trying to download Linux, LLVM, GCC, Linaro stuff.

So, while I do literally feel your pain around 5 weeks per year, so
does every one else in the same situation. There are no solutions to
that problem other than "not going there", which for me, really isn't
an option.

Bottom line is, clone your repos before going away and always keep an
updated repo in a backup, so you can use it to considerably reduce the
download time on remote places.

cheers,
--renato

Please try following approach:

1. Clone with --depth 1
2. Run git fetch --unshallow

In this case you will split download into 2 parts, which can succeed independently from each other. If you cannot get unshallow step working, you can split it further by running git fetch --depth=<number> with increasing numbers.

Yes and please look at the subject - I clearly put OT as well as
didn't cross post to every other ML on the planet. So I'm following my
own advice..

Have you ever stayed in a non-hilton hotel while traveling? I don't
mean couch surfing, but just average place.. I'd even add some above
average places to the list and when you get 100 people all on the same
line.. it can get "slow"...

I've been to many places while going to Connects, events and sprints,
and I can attest that hotels, no matter how many stars, have horrible
Internet connection.

One hotel we had to tell the admin to stop throttling SSH and also to
allow connections to more ports than just 25, 80 and 443.

But that affected *everybody*. Using Git or SVN, using Windows or
Linux, trying to download Linux, LLVM, GCC, Linaro stuff.

But with SVN you don't fetch complete history.

You'll see in my 1st email I said I was going to try this.. (shallow
clone did work) It was meant more as a data point from real world than
a cry for help.

Thanks a lot for looking out though :slight_smile: Long time no talk..

You can do that with Git, too. And SVN is generally a lot slower to
clone the same repo. :slight_smile:

cheers,
--renato

OT: For WebKit you can just download initial SVN snapshot from official site :slight_smile:

GitHub allows that, too, no? Or is that a repo-free tar ball? I never tried...

It's repo-free

FWIW:

[dannyb@dannyb-macbookpro3 11:54:38] ~/sources/llvm (git-svn)-[newgvn-predicates]- :slight_smile: $ gdu --exclude=debug-build -k


521724 .

[dannyb@dannyb-macbookpro3 11:55:00] ~/sources/llvm (git-svn)-[newgvn-predicates]- :slight_smile: $ gdu .git -k


305768 .git

an SVN checkout would have transferred the 521 meg:)

With zlib compression, etc, maybe they are about equal.

But SVN would not have transferred significantly less data here.