Minimum Requirement Infrastructure

Hi folks,

A number of threads have sparked discussions about minimum toolchain
requirements and a few of them have shown that we're very poor in
keeping track of our promises.

The main issue is that we're not publicly testing our minimum requirements.

We claim clang 3.1 can compile all LLVM projects, but no one has a
buildbot with that version, even with just LLVM and Clang. Recently,
libc++ has broken with clang 3.6, and that's because I've upgraded our
bot to 3.9 and stopped tracking.

We (Linaro) need to stay on top of what people are using, not LLVM's
minimum requirements, as we have found a number of issues when
migrating to the current LTS platform (Ubuntu Xenial), and that's what
people are using on ARM right now. So we cannot keep testing all
platforms, and I assume this is true for all companies that contribute
to LLVM.

For that reason, I propose that the Foundation should set some funds
to have x86_64 machines (cloud instances are fine) with the minimum
requirements for at least one Linux build with minimum Clang and one
with minimum GCC.

I don't know if it's possible to have OSX on the cloud, or if it's
easy to have a Windows machine in the same way, but that'd be "in
addition" to the Linux machine. MSVC 2015 support can be tested by
those that have the license, I'm not proposing the Foundation to
purchase it.

The preference for Linux is that it's easier to deploy and control on
the cloud (and it's free), and we're testing the compilation process,
not the functionality (which should be tested elsewhere).

Having a single machine for both GCC and Clang builds should be ok,
since we don't need too many builds a day, and compilation errors are
easy to spot, even on a large commit range. Even if we had one build
each, every day, it would be fine. A dual core cloud instance with 4GB
of RAM would be more than enough.

Can the foundation spare resources for that usage?

If not, do we have another way to ensure the support of minimum
compilers to our projects?

Do we track sub-projects like LLD, LLDB, Compiler-RT and libc++
separately? We must track them, too.

cheers,
--renato

From: "Renato Golin via llvm-foundation" <llvm-foundation@lists.llvm.org>
To: "Renato Golin via llvm-foundation" <llvm-foundation@lists.llvm.org>
Sent: Tuesday, October 18, 2016 5:40:54 AM
Subject: [llvm-foundation] Minimum Requirement Infrastructure

Hi folks,

A number of threads have sparked discussions about minimum toolchain
requirements and a few of them have shown that we're very poor in
keeping track of our promises.

The main issue is that we're not publicly testing our minimum
requirements.

We claim clang 3.1 can compile all LLVM projects, but no one has a
buildbot with that version, even with just LLVM and Clang. Recently,
libc++ has broken with clang 3.6, and that's because I've upgraded
our
bot to 3.9 and stopped tracking.

We (Linaro) need to stay on top of what people are using, not LLVM's
minimum requirements, as we have found a number of issues when
migrating to the current LTS platform (Ubuntu Xenial), and that's
what
people are using on ARM right now. So we cannot keep testing all
platforms, and I assume this is true for all companies that
contribute
to LLVM.

For that reason, I propose that the Foundation should set some funds
to have x86_64 machines (cloud instances are fine) with the minimum
requirements for at least one Linux build with minimum Clang and one
with minimum GCC.

I don't know if it's possible to have OSX on the cloud, or if it's
easy to have a Windows machine in the same way, but that'd be "in
addition" to the Linux machine. MSVC 2015 support can be tested by
those that have the license, I'm not proposing the Foundation to
purchase it.

The preference for Linux is that it's easier to deploy and control on
the cloud (and it's free), and we're testing the compilation process,
not the functionality (which should be tested elsewhere).

Having a single machine for both GCC and Clang builds should be ok,
since we don't need too many builds a day, and compilation errors are
easy to spot, even on a large commit range. Even if we had one build
each, every day, it would be fine. A dual core cloud instance with
4GB
of RAM would be more than enough.

Can the foundation spare resources for that usage?

If not, do we have another way to ensure the support of minimum
compilers to our projects?

Do we track sub-projects like LLD, LLDB, Compiler-RT and libc++
separately? We must track them, too.

Regarding this last question, I think we should discuss this on llvm-dev. Perhaps a separate thread would be appropriate. How we specify our minimum requirements across the various LLVM projects is something we need to discuss with a wide audience, and is a technical matter. We definitely don't do a good job now, either with the specification or the testing, in this regard.

-Hal

I agree. Though, that question depends on how the foundation will
help. Thus, why I started here.

The technical side of it is really minimal. If we get enough hardware,
then tracking largely means "we're building all relevant sub-projects
in these environments already". If not, than this will need help from
the community and can be complicated.

The main question for llvm-dev is "What do we consider 'relevant'?"
(which also can change, depending on hardware investments).

cheers,
--renato

Hi all,

For that reason, I propose that the Foundation should set some funds
to have x86_64 machines (cloud instances are fine)

At PlayStation we already have some remote engineers using Ubuntu on EC2 as their main build environment. At the level you’ve suggested (one build per day) the minimum requirements testing is probably something I can fit in to our existing cloud usage (thus minimizing any cost to the foundation), I’m happy to investigate that possibility in parallel with any conversation you have on llvm-dev.

easy to have a Windows machine in the same way,

This was on my list to explore too but I expect this will be harder to configure, I’m happy to look into this too and report back.

Thanks,

Simon

At PlayStation we already have some remote engineers using Ubuntu on EC2 as
their main build environment. At the level you've suggested (one build per
day) the minimum requirements testing is probably something I can fit in to
our existing cloud usage (thus minimizing any cost to the foundation), I'm
happy to investigate that possibility in parallel with any conversation you
have on llvm-dev.

Hi Simon,

Thanks for the offer, I think it would be brilliant if we had this
publicly with emails like buildbots / green dragon.

It should be reasonably simple to setup a Clang 3.x (1? 4? we need to
make sure what works here) and GCC 4.8 with the appropriate glibc at
the time (a vanilla Ubuntu Trusty would do just fine).

This was on my list to explore too but I expect this will be harder to
configure, I'm happy to look into this too and report back.

I assume this is easier to do for the companies that do have MSVC
licenses, which is normally an internal process. Given the nature of
Windows environments, and the number of existing companies validating
MSVC, I think it should be fine to keep it aside.

I know even less about OSX than I do about Windows, so I'm hoping
Apple can chime in on how to best validate on their environments. But
this may be a question for llvm-dev.

cheers,
--renato