Quad-Core ARMv7 Build Slave Seeks Noble Purpose

Hi guys,

I have this ODROID XU (quad-core ARMv7 Cortex A15 1,6 GHz) box that I want to make available to the LLVM project. I can see that LLVM already has an ARMv7 Cortex A9 system doing Clang, but how should this box be used, if at all? We’re talking a long-term commitment within the realm of being a buildbot slave. It builds LLVM in about 40 minutes, excluding the test suite.

Sincerely,
Mikael Lyngvig

Hi Mikael,

40 minutes is a good iteration time for an ARM CPU. Are you using
some kind of caching mechanism to speed up compilation (ccache?) If
not, it could be even faster :slight_smile:

I think this buildbot will be very useful for the community. What OS
is it running?

You can find instructions on adding a buildbot here:

http://llvm.org/docs/HowToAddABuilder.html

I am also CC'ing Galina.

Dmitri

Hi Dmitri,

I am not using any kind of cache (didn’t even know of ccache). I have now installed ccache. Perhaps ccache should be mentioned in the buildbot document so that every buildbot owner knows about it?

It is currently running Arch Linux ARM. if there are good reasons to switch to something else, I’ll be happy to do that, although I am generally very happy about Arch Linux.

What do you want me to build? LLVM? Clang? Both plus test suite?

Sincerely,
Mikael

Hi Dmitri,

I am not using any kind of cache (didn't even know of ccache). I have now
installed ccache. Perhaps ccache should be mentioned in the buildbot
document so that every buildbot owner knows about it?

Possibly.

It is currently running Arch Linux ARM. if there are good reasons to switch
to something else, I'll be happy to do that, although I am generally very
happy about Arch Linux.

What do you want me to build? LLVM? Clang? Both plus test suite?

Personally, I see more value in building LLVM, Clang, possibly lld,
without test suite. This is the way that most build bots operate.
Test suite takes a long time to run, and this increases the iteration
time.

As an example, you could look at the configuration in zorg repository
that corresponds to this buildbot:

http://lab.llvm.org:8011/builders/llvm-clang-lld-x86_64-debian-fast

It uses CMake/ninja, which will give you the best possible iteration time.

In order to use ccache, you need ensure that these environment
variables are set:

CC="ccache clang"
CXX="ccache clang++"
CCACHE_CPP2=yes
export CC CXX CCACHE_CPP2

(Of course, you can use gcc/g++ as compilers.)

Dmitri

Hi Dmitri,

I am not using any kind of cache (didn't even know of ccache). I have now
installed ccache. Perhaps ccache should be mentioned in the buildbot
document so that every buildbot owner knows about it?

It is currently running Arch Linux ARM. if there are good reasons to
switch to something else, I'll be happy to do that, although I am generally
very happy about Arch Linux.

I love Arch, but it is probably a bit too unstable for a long-term
buildbot. CC'ing Renato who might have some suggestions.

-- Sean Silva

> Hi Dmitri,
>
> I am not using any kind of cache (didn't even know of ccache). I have
now
> installed ccache. Perhaps ccache should be mentioned in the buildbot
> document so that every buildbot owner knows about it?

Possibly.

> It is currently running Arch Linux ARM. if there are good reasons to
switch
> to something else, I'll be happy to do that, although I am generally very
> happy about Arch Linux.
>
> What do you want me to build? LLVM? Clang? Both plus test suite?

Personally, I see more value in building LLVM, Clang, possibly lld,

I think LLD would especially benefit from testing on other architectures,
since LLD is the most prone to accidentally having something
architecture-dependent (e.g. assuming pointers are 64 bits, endianness,
alignment, etc) and so testing it on more architectures will turn up bugs.

-- Sean Silva

A quick question while I am adding a tiny section on ccache to the HowToAddABuilder document: Why do you set CCACHE_CPP2 to yes? From reading the manual, I get the impression that this is much slower than the default and that it should only be used with tricky compilers or when debugging. Is Clang such a tricky compiler? If so, I suppose Clang should be fixed, not the ccache configuration.

– Mikael

I don't remember exactly why I used this option, but it seems like I
was getting diagnostics from system headers, that would otherwise be
suppressed, because the header was found in a system include
directory.

Dmitri

Yes, lld reads binary files so gets all the potential bugs that comes with that. Mach-O binary format is challenging because it uses bit fields structs and the compiler switches the direction of bit fields on big vs little endian archs!

That said, isn’t the ARM machine 32-bit little endian? Other than possible alignment issues, it has the same coverage as i386. What lld needs is big endian build machines.

-Nick

> What do you want me to build? LLVM? Clang? Both plus test suite?

Personally, I see more value in building LLVM, Clang, possibly lld,

I think LLD would especially benefit from testing on other architectures,
since LLD is the most prone to accidentally having something
architecture-dependent (e.g. assuming pointers are 64 bits, endianness,
alignment, etc) and so testing it on more architectures will turn up bugs.

Yes, lld reads binary files so gets all the potential bugs that comes with
that. Mach-O binary format is challenging because it uses bit fields
structs and the compiler switches the direction of bit fields on big vs
little endian archs!

Ouch...

That said, isn’t the ARM machine 32-bit little endian? Other than
possible alignment issues, it has the same coverage as i386. What lld
needs is big endian build machines.

Yeah, big endian would really be nice (ppc, sparc? we must have *some*
big-endian builders...), but just the alignment checking is still a
worthwhile step up from the status quo I think (although IIRC UBSan also
catches these?).

-- Sean Silva

Yes, ARM normally runs as a little-endian and it is a 32-bit CPU. It CAN be configured to be a big-endian system, but that requires hardware support as far as I know.

I do have an old, slow Mac Mini G4 PowerPC (big-endian) that I could hook up as a builder too. I was thinking of it the moment you mentioned big endian. I actually bought it for testing C++ code on because big-endian machines are so great at catching those silly errors that we little-endianers occasionally do.

– Mikael

I ran into the very same problem, so it is definitely required. I’ll add it to the proposed change to the LLVM buildbot documentation.

LOL, a full build with an empty ccache cache took 36 minutes. A full build with all the files in ccache’s cache took about eleven and a half minute, most of it spent linking.

– Mikael

Because ccache splits the compilation step up into components
(preprocess, compile, link, etc.), but passes all the options to each
stage, clang complains about unused build flags.

There might be other reasons that are not down to diagnostic noise, and
certainly ccache has had some of these issues fixed (it better knows
which options are suitable for which stage) since the recommendation
(I'm assuming they exist because it was tested with gcc which does not
complain so much).

Ben

Does clang build on G4 (or G5) Mac OS X? I never managed to get it built :frowning:

Csaba

Mikael, Sean,

I think having an Arch buildbot is a great idea. At Linaro, we normally
test on Debian-derived distros, and having something else entirely is a
good stress test for the compiler, the build configuration, the buildbot
scripts and the test infrastructure.

Regarding what to test, I'd begin with Clang+LLVM check-all. There are
already configurations on Zorg for that kind of setup. You can copy from
the A9 bots and change to "A15". I recommend you to disable make clean
(Clean=false), so that the cycle time becomes minutes, not hours. You
should set this up locally, on your own build master, and let it run for a
few days, and if it's stable, you can go to stage 2.

Stage 2 is either to put it in production (ie. moving the configuration to
Zorg and connecting your bot to LLVM's master), or enhance the testing
capabilities of your bot. The former is *very* simple, but the latter
depends on what you want. Stage 3 would be to put it into production.

Normally, the rules of thumb for ARM bots:
* I woulnd't have bots running check-all AND the test-suite/lldb, because
I want them to be orthogonal, ie. I don't want the test-suite bot stopping
short of testing because of a silly breakage in a new test.
* I wouldn't test lldb if you don't care about it. (I don't, yet). lldb is
a separate project and I had trouble setting it up to run on ARM before.
* Always have more than one buildbot on any configuration. Build time can
be huge, and dev boards are notoriously faulty. I had huge problems with
Panda boards in the past, to the point where I removed them all from the
build rota. The odroid U2 seems more stable, but the XU has some
hardware/kernel problems (randomly re-mounting partitions read-only,
disabling CPUs and never re-enabling them again, cache flush between every
big.LITTLE switch, amongst others).
* Create boot scripts to check for those problems, plus set the CPU
scheduler to "performance" on ALL CPUs. This eases most CPU problems.
* Create a stable configuration and save the image as it will run in
production, to make it easier to re-create bots on the spot
* Have extra spare boards to replace a broken bot, as most of the time,
the easiest path is to re-flash, but you need something running while you
do it
* Running the build on SDcards is ok, but they are more prone to failures
than good quality USB sticks, and those are more prone to failures than
external hard-drives (those are also a lot faster). So, at least, I'd
recommend you to buy a SanDisk Ultra USB stick.
* Make sure you have a decent power supply (dozens of dollars worth) that
can provide *at least* 4amps.

All that may seem daunting, but there is one critical issue of hosting a
buildbot: reliability of the test is equals to reliability of the platform.

In the beginning, there were only a handful of ARM boards, and they were
broken most of the time, and ARM was not considered a stable target. We
changed it by introducing lots of new bots, test-suite and fixing all the
bugs, but once my Pandas starting to fail randomly, the popular belief was
that "failures on ARM are due to the board stability, not my commits", and
sure enough, bugs started to creep in. We don't want that.

So, while I welcome new buildbots for ARM, we must do it right, from the
beginning. We still don't have enough critical mass to be able to have a
faulty bot, unfortunatelly.

cheers,
--renato

This box is running Debian, not Max OS X, so I suppose it should work.

Renato, thanks for your elaborate walk-through of the issues with ARM boards. I’m trying to add some of this to the “How to Build on ARM” document and will submit a patch later on.

I already ran into the problem of cores disappearing, but on Arch Linux (which uses a fairly recent kernel), the missing cores come back as soon as the load falls to zero.

Unfortunately, my personal budget does not allow me more than a single Odroid XU board for the time being. So I’ll have to do with only one board. I happen to have an eMMC card and it is fairly fast.

By the way, how do you set the CPU scheduler to “performance” (procfs something?). Just so that it can be added to the docs.

– Mikael

Renato, thanks for your elaborate walk-through of the issues with ARM
boards. I'm trying to add some of this to the "How to Build on ARM"
document and will submit a patch later on.

Nice, thanks! That would be great!

Unfortunately, my personal budget does not allow me more than a single

Odroid XU board for the time being. So I'll have to do with only one
board. I happen to have an eMMC card and it is fairly fast.

That's good. Still, I'd have it running locally on your master for a week
or so, just to make sure it's stable enough.

To create a local master you need to read this doc:

https://buildbot.readthedocs.org/en/v0.8.6/manual/installation.html

And add Zorg to your master. The way I did was to create a master on
buildbot's user dir, (buildbot create-master ~/buildmaster/llvm), checkout
zorg, symlink the master/builders configs from zorg into the buildmaster
dir, symlink the zorg's Python module into /lib/python2.7/whatever (can't
remember), and start the master via (buildbot master start).

Then, I'd create two buildslaves on the board, one in ~/localbot and one in
~/buildbot, with the former pointing to your local master and the latter
pointing to LLVM's official master, so that it's easy for you to swap. (
http://llvm.org/docs/HowToAddABuilder.html)

On the master, I stripped down all builders from the config and only left
my own boards, so that I don't clutter the waterfall/builders page.

By the way, how do you set the CPU scheduler to "performance" (procfs

something?). Just so that it can be added to the docs.

for ((cpu=0; cpu<`grep -c proc /proc/cpuinfo`; cpu++)); do
  sudo sudo cpufreq-set -c $cpu -g performance
done

cheers,
--renato

I finally got it to work - and my ODROID-XU is now a passive member of the LLVM builder society for the next week or so, until it has proven itself (it monitors changes to the SVN repository and builds in my end without bothering anyone at LLVM.org about its findings). I did write up a “short” treatise on how to do it, in case anybody needs to do this sometime in the future. I expect it to be finished with the final steps of how to link into Zorg, once I get that far.

http://llvm.lyngvig.org/Articles/How-to-Setup-an-Arch-Linux-Buildbot-for-LLVM

The trickiest part is that Arch Linux comes with Buildbot v0.8.8 and LLVM uses v0.8.5. So I had to resort to using a virtual environment and install v0.8.5 instead of v0.8.8.

The next two pending projects are:

  1. To set up my Mac Mini G4 (PowerPC)/Debian to build and test LLD. Several felt that it needed big-endian testing, so that’s what that box will do.
  2. To set up my BeagleBone Black to do something useful, whatever that is. Perhaps it can do an LLD build too, as several requested an ARM build/test of LLD.

– Mikael

http://llvm.lyngvig.org/Articles/How-to-Setup-an-Arch-Linux-Buildbot-for-LLVM

Notice: You only need Ninja for the test build; none of the official LLVM builders use Ninja as far as I know.

My buildbots use ninja.

Please notice that you must specify the absolute path to ninja, otherwise CMake can't figure out where it is located.

Strange, I never had to do this. I just put 'ninja' binary into my PATH.

pushd llvm-trunk/tools > /dev/null

This does not look correct to me, because you are not entering the
'clang' directory. Similarly, for compiler-rt below.

Dmitri