IMPORTANT: APT repo temporary switched off

TL;DR: APT repo switched off due to excessive load / traffic

Recently we realized that APT repo generates almost 95% of I/O on
llvm.org and more than 40% of network bandwidth alone. During last 2
weeks the main services on llvm.org (svn, git, bugzilla) had serious
problems with overall connectivity.

We decided to temporary switch APT repo off to see if this would help.
Stay tuned for updates.

From: "Anton Korobeynikov via llvm-foundation" <llvm-foundation@lists.llvm.org>
To: "llvm-dev" <llvm-dev@lists.llvm.org>, "clang-dev Developers" <cfe-dev@cs.uiuc.edu>,
llvm-foundation@lists.llvm.org
Cc: "Sylvestre Ledru" <sylvestre@debian.org>
Sent: Tuesday, May 31, 2016 12:27:21 PM
Subject: [llvm-foundation] IMPORTANT: APT repo temporary switched off

TL;DR: APT repo switched off due to excessive load / traffic

Recently we realized that APT repo generates almost 95% of I/O on
llvm.org and more than 40% of network bandwidth alone.

Interesting. Is that mostly from metadata updates, or from people actually installing LLVM packages?

-Hal

Recently we realized that APT repo generates almost 95% of I/O on
llvm.org and more than 40% of network bandwidth alone.

Interesting. Is that mostly from metadata updates, or from people actually installing LLVM packages?

The latter

Thanks Anton, there were a lot of timeouts on our bots for the last
few weeks, I hope this will make things better.

If we still want to be serving those packages, maybe we could talk to
someone willing to host them?

cheers,
--renato

I'm not surprised by the heavy load but a little more notice would
have been nice. This has broken our CI builds for KLEE [1]. We (the
KLEE developers) can fix this by moving our CI to run in an Ubuntu
14.04 environment where the LLVM version we currently require is part
of the official Ubuntu repositories. So I guess this is the impetus we
need to make the switch :slight_smile:

I (and suspect several others) really on the packages in the APT repo
for our CI builds. You can see see that the LLVM APT repo
is in TravisCI's source white list [2] so there are probably others using it.

[1] https://travis-ci.org/klee/klee/jobs/134303870
[2] https://github.com/travis-ci/apt-source-whitelist/blob/92dbd15f6a12e5f24c2f31f314060a349c37dc68/ubuntu.json#L168

Yes, I imagine that this broke quite a few builds on Travis CI; it
certainly broke a few that I rely on.

Regardless, thanks for having run this infrastructure for so long. Have you
considered publishing the packages to a PPA, so that the community can
continue to benefit from them without llvm.org having to support the
hosting costs?

I'm not surprised by the heavy load but a little more notice would
have been nice. This has broken our CI builds for KLEE [1]. We (the
KLEE developers) can fix this by moving our CI to run in an Ubuntu
14.04 environment where the LLVM version we currently require is part
of the official Ubuntu repositories. So I guess this is the impetus we
need to make the switch :slight_smile:

Hi Dan,

I understand the problems that this have created, and I can see that
we still have people on IRC complaining about it.

But the server were really on its knees and everything else in it
(including all our source repositories) were just not working at all.
It was a hard decision, but one that had to be made, and soon.

Regardless, thanks for having run this infrastructure for so long. Have you considered publishing the packages to a PPA, so that the community can continue to benefit from them without llvm.org having to support the hosting costs?

John,

I'm not well versed with PPAs, but if you have an alternative that we
could do and fix the package problem without reusing the same server,
I think we should pursue it.

cheers,
--renato

Regardless, thanks for having run this infrastructure for so long. Have you considered publishing the packages to a PPA, so that the community can continue to benefit from them without llvm.org having to support the hosting costs?

John,

I'm not well versed with PPAs, but if you have an alternative that we
could do and fix the package problem without reusing the same server,
I think we should pursue it.

I've used PPAs as a developer before and I hated the experience but
that's probably because launchpad (the web interface) is garbage.

I've had a better experience with the OpenSUSE build service [1] which
despite its name lets you build for Ubuntu, Debian, OpenSUSE, Redhat,
Fedora, CentOS and Arch Linux. You have to provide the packaging files
for each packaging system though. It also supports more architectures
than just x86_64.

[1] https://build.opensuse.org/

I've used PPAs as a developer before and I hated the experience but
that's probably because launchpad (the web interface) is garbage.

Indeed!

It seems that the people relying on our apt service wouldn't bother
much where we move it to. So, I'm ok with any solution that replaces
the functionality.

[1] https://build.opensuse.org/

This also sounds interesting... I remember being impressed with
OpenSUSE's validation infrastructure in the past.

Would you be able to cook up some example, so that the rest of the
people that rely on it can check if it works for them, too?

cheers,
--renato

[1] https://build.opensuse.org/

This also sounds interesting... I remember being impressed with
OpenSUSE's validation infrastructure in the past.

Would you be able to cook up some example, so that the rest of the
people that rely on it can check if it works for them, too?

I did make a package [1] a while ago. I've not touched it in a while
so the Arch Linux build broke.

It seems there is already a repo for OpenSUSE [2]. We could try to
incorporate support for other distros there are make our own given
that we actually want to support multiple LLVM versions too.

If you want I could try to hack together a proof of concept using the
existing Debian packaging files targeting one particular LLVM version.

If the proof of concept works well and the community don't mind moving
we should probably talk to the OpenSUSE build service people about
setting up a more permanent home for the packages there and getting
involvement from downstream users because I don't want to maintain
packaging scripts for distributions that I don't even use.

[1] https://build.opensuse.org/package/show/home:delcypher:z3/z3
[2] https://build.opensuse.org/package/show/devel:tools:compiler/llvm

It seems there is already a repo for OpenSUSE [2]. We could try to
incorporate support for other distros there are make our own given
that we actually want to support multiple LLVM versions too.

s/other distos there are make/other distros there or make/

It seems there is already a repo for OpenSUSE [2]. We could try to
incorporate support for other distros there are make our own given
that we actually want to support multiple LLVM versions too.

Looks like they're all 3.8.0 based...

If you want I could try to hack together a proof of concept using the
existing Debian packaging files targeting one particular LLVM version.

If you don't mind, I really have no idea how to do that. :slight_smile:

If the proof of concept works well and the community don't mind moving
we should probably talk to the OpenSUSE build service people about
setting up a more permanent home for the packages there and getting
involvement from downstream users because I don't want to maintain
packaging scripts for distributions that I don't even use.

Absolutely. If we end up using their services, we need to make them
aware and be in sync. :slight_smile:

Also, we should incorporate that maintenance into the same package as
we did the last apt repo.

Just make sure Anton is in the loop, and we should be able to get it
going after the prototype is approved by the other users.

cheers,
--renato

Guys, you are jumping on the gun.
There is no need to migrate to PPA or build.o.o.
llvm.org/apt was providing the mirror, not the build infra.
Changing the URL of the mirror is trivial, changing the build infra is a
complex work.

Clearly, I don't see any real issue in the current situation.

Sylvestre

TL;DR: APT repo switched off due to excessive load / traffic

Recently we realized that APT repo generates almost 95% of I/O on
llvm.org and more than 40% of network bandwidth alone. During last 2
weeks the main services on llvm.org (svn, git, bugzilla) had serious
problems with overall connectivity.

Oups! I wasn't aware of this issue. Thanks for doing that.
At least, we have an answer to a question I had recently:
a lot of people are using it!

We decided to temporary switch APT repo off to see if this would help.
Stay tuned for updates.

OK, could you share the updates? Should I look for an alternative for
the hosting of the repository?

Cheers,
Sylvestre

TL;DR: APT repo switched off due to excessive load / traffic

Recently we realized that APT repo generates almost 95% of I/O on
llvm.org and more than 40% of network bandwidth alone. During last 2
weeks the main services on llvm.org (svn, git, bugzilla) had serious
problems with overall connectivity.

I think this correlates with the increase of package update frequency from
once a week to (sometimes more than) once a day. [1]
I thought that the CI systems like Travis do some caching of packages so it
is not re-downloading new packages for every build of every pull request?
If not, adding one commit to a PR in LDC means 6 downloads from llvm.org...

We decided to temporary switch APT repo off to see if this would help.

Stay tuned for updates.

Because LDC's core CI testing depends on the APT repo, I hope a replacement
solution is found quickly.
Thanks very much,
   Johan

[1] https://groups.google.com/forum/#!topic/llvm-dev/MjhxuNRx9sA

Guys, you are jumping on the gun.
There is no need to migrate to PPA or build.o.o.

I think we all agreed that PPA was a bad idea from start, so don't
worry about it.

llvm.org/apt was providing the mirror, not the build infra.
Changing the URL of the mirror is trivial, changing the build infra is a
complex work.

It's more than just an apt mirror...

Right now, only Debian based distros can use that apt repo, so if we
want to do the same for RPM based ones, we'll have to double the
infrastructure. Furthermore, I can see an argument to keep
distro-based storage in a distro-based infrastructure, not in LLVM, as
we probably should be distro-agnostic.

Dan was saying that the OpenSuse build can host files for all distros,
and maybe even build them, so I don't see any problem in *trying*
that. If anything, we'll end up with another distro being served in
the same way, and we could get Debian and Ubuntu builds to pull files
from there. Even if we go back serving the files from LLVM servers,
the Suse build can continue from the new location. I don't see any
harm in that... As you said, changing the mirror is trivial...

I'm happy with any solution that can be maintained for more than a
week without crashing other crucial LLVM infrastructure.

cheers,
--renato

Sorry to jump the gun. I brought up OBS because someone mentioned PPAs
and I think PPAs in their current form are terrible.

Right now finding new hosting for the mirror seems like the most
important thing. Maybe the LLVM foundation should weigh in here?
Presumably there are funds set aside for something like this? Having
packages for different LLVM versions is incredibly useful and I think
help should be made available to you for hosting should you need it.

Dan.

Another option would be to use a commercial hosting solution like Bintray.
It is even free for OSS projects.
It supports hosting Debian packages and even RPM.

https://bintray.com/docs/usermanual/formats/formats_debianrepositories.html

/Florent

Hello,

TL;DR: APT repo switched off due to excessive load / traffic

Recently we realized that APT repo generates almost 95% of I/O on
llvm.org and more than 40% of network bandwidth alone. During last 2
weeks the main services on llvm.org (svn, git, bugzilla) had serious
problems with overall connectivity.

We decided to temporary switch APT repo off to see if this would help.
Stay tuned for updates.

Until we have a more permanent solution, I set up:

http://llvm-apt.ecranbleu.org/apt/

Please note that it is just a temporary solution, I might turn it off if
the load is too important.

(the server is only used for this).

The sync from the CI is not back yet (should be done in the next few days).

Cheers,
Sylvestre

Thank you very much for setting up an APT repository with LLVM trunk in it.
LDC's CI systems are now working with the binaries available from llvm.org,
but for LLVM trunk we depend on the APT repo.

cheers,
  Johan