Assuring ARM code quality in LLVM

Project Infrastructure LLVM Dev List Archives
1

Hi all,

I was recently investigating the build bot infrastructure and noticed
that the arm-linux target is failing for quite a long time. I believe
that it means ARM code is not executed all that often in LLVM tests,
is that correct?

We were wondering what kind of support we could give to make sure ARM
code is correct and don't regress, specially before releases (I know
it's a bit late for 2.9).

We have thought about some routes:
- Install some build slaves on our side, so you can test them (linux,
bare-metal), if it's possible to cross-compile on build slaves.
- Provide models or even hardware to whomever maintains the build
slaves, to have a centralised infrastructure, if that's beneficial in
any way.
- Add linkage & execution to LIT tests, probably by using QEMU
models, but that would create a dependence on QEMU for everyone.
- Something else?

As far as I could see, there is no current owner for the ARM tests
(including linkage and execution).

I'd like to get the ball rolling to better support testing and
benchmarking the ARM code, with the ultimate goal of making sure it
won't regress at least on every release, but hopefully for every
patch.

Could you please let me know what should I do, and what are the best
ways of doing so?

cheers,
--renato

2

Hi all,

I was recently investigating the build bot infrastructure and noticed
that the arm-linux target is failing for quite a long time. I believe
that it means ARM code is not executed all that often in LLVM tests,
is that correct?

Hi Renato

Not really apropos to your question specifically, but
I have some downstream patches for ARM/MC/ELF/.o
There were some issues that needs fixing. As soon as I get some time,
I plan to upstream these patches in to trunk.
Hopefully this should improve things a little bit...

-jason

3

Hi Renato,

I was recently investigating the build bot infrastructure and noticed
that the arm-linux target is failing for quite a long time. I believe
that it means ARM code is not executed all that often in LLVM tests,
is that correct?

We were wondering what kind of support we could give to make sure ARM
code is correct and don't regress, specially before releases (I know
it's a bit late for 2.9).

We have thought about some routes:
  - Install some build slaves on our side, so you can test them (linux,
bare-metal), if it's possible to cross-compile on build slaves.

there are several idle ARM buildslaves. There are the following slaves:

gcc35: # 1TB 0.8 GHz Freescale i.MX515 (ARM Cortex-A8) / 512 MB RAM / Efika MX Client Dev Board

gcc50: # 250G 0.6 GHz ARM XScale-80219 / 512 MB RAM / Thecus N2100 NAS

gcc55: # 250G 1.2 GHz ARM Feroceon 88FR131 (kirkwood) / 512 MB RAM / Marvell SheevaPlug

gcc56: # 320G 1.2 GHz ARM Feroceon 88FR131 (kirkwood) / 512 MB RAM / Marvell SheevaPlug

gcc57: # 320G 1.2 GHz ARM Feroceon 88FR131 (kirkwood) / 512 MB RAM / Marvell SheevaPlug

I can give you access to these machines if you like (send me your public ssh
key).

Ciao, Duncan.

4

Hi Duncan,

I could find them on the page
http://google1.osuosl.org:8011/buildslaves but I couldn't tell they're
ARM. Where do I get this list?

Also, I believe they're testing llvm-gcc only (right?). The only
target that was testing LLVM was the one failing, and no target is
testing Clang.

I'd like to have at least one each. To cover the minimum requirements,
would be good to have {v5,v6,v7}x{linux,eabi}x{ARM,Thumb} where
appropriate.

I think the most important right now are at least one v5T, one A8 and
one M3, on both linux (v5, A8) and bare-metal, testing on both ARM and
Thumb(1 or 2) each.

Is there a way to schedule different tests on the same boards,
avoiding collision? Say, while LLVM test runs on A8 bare-metal, Clang
tests run on linux and vice versa.

cheers,
--renato

5

Hi i will summarise the last sucessfull builds by the ARM builder:

  • Revision: 128836 last successful ARM build by the builder - Revision: 128851 failed build - Revision: 128873 failed … 757 - 759 - dito - Revision: 128895 failed … 761 - 769 - dito - Revision: 129068 failed build - Revision: 129072 failed … 772 - now - dito LLVM 2.9 was forked off the trunk at Revision 127228 so the above regressions are not in LLVM 2.9. unfortunally a fold-pcmpeqd-0.ll regression got into the 2.9 fork. - Revision: 127193 last known successful ARM build by the builder before the 2.9 fork. - Revision: 127214 failed - Revision: 127220 failed 2.9 fork - happened here - Revision: 127237 failed … 592 - 593 dito - Revision: 127289 failed - Revision: 127308 last known successful ARM build by the builder after the 2.9 fork. Hope this will help fix the regressions Cheers Xerxes
6

Hi Renato,

there are several idle ARM buildslaves. There are the following slaves:

Hi Duncan,

I could find them on the page
http://google1.osuosl.org:8011/buildslaves but I couldn't tell they're
ARM. Where do I get this list?

http://llvm.org/svn/llvm-project/zorg/trunk/buildbot/osuosl/master/config/slaves.py

Also, I believe they're testing llvm-gcc only (right?).

They're not running anything. I'm happy if you run whatever you like there.

  The only

target that was testing LLVM was the one failing, and no target is
testing Clang.

I'd like to have at least one each. To cover the minimum requirements,
would be good to have {v5,v6,v7}x{linux,eabi}x{ARM,Thumb} where
appropriate.

I think the most important right now are at least one v5T, one A8 and
one M3, on both linux (v5, A8) and bare-metal, testing on both ARM and
Thumb(1 or 2) each.

Is there a way to schedule different tests on the same boards,
avoiding collision? Say, while LLVM test runs on A8 bare-metal, Clang
tests run on linux and vice versa.

Since I don't know anything about ARM I suggest you log into the machines
yourself and do whatever configuration you want.

Ciao, Duncan.

7

Hope this will help fix the regressions

Hi Xerxes,

I see you're the owner of that board, thanks for the detailed
description of the tests.

By what you say, I think that the board itself is serving its purpose,
and 2.9 only got that regression because it wasn't fixed it in time.

My intention was to know what can we (ARM) can do to help LLVM repeat
your success on all tests (clang, llvm, llvm-gcc, dragonegg), at least
before each release. How can we make ARM a first-class target, in
which a regression such as the one you found is a blocker to release.

Duncan,

I'm not sure me connecting to those boards will help much, since I
know very little of the build-bot infrastructure. However, I'd be
happy to help setting them up to run the tests we need (with some help
in the beginning).

From what you've listed, we only have v5T and one cortex A8. We could

provide more A8/A9s, M3s (mBed) boards and also models of all
variations possible.

Does it make any difference running them here at ARM or locally on
OSUOSL? We could easily help them get more boards, too...

On a different topic,

From what I understood, this tests run completely on the target

(compilation of LLVM, unit tests, etc). So, for bare-metal tests that
wouldn't work. We'd need to have a staging server that would compile
to all platforms and run the tests (maybe just a subset) in the
hardware via USB (mBeds) or JTAG on most other boards, or even better,
on models directly.

Do you think this is doable with the build bot infrastructure?

cheers,
--renato

8

Hi Renato,

I was recently investigating the build bot infrastructure and noticed
that the arm-linux target is failing for quite a long time. I believe
that it means ARM code is not executed all that often in LLVM tests,
is that correct?

We were wondering what kind of support we could give to make sure ARM
code is correct and don't regress, specially before releases (I know
it's a bit late for 2.9).

We have thought about some routes:
- Install some build slaves on our side, so you can test them (linux,
bare-metal), if it's possible to cross-compile on build slaves.

It is possible.
We host several build slaves which cross compile. The issue with cross
compilation is that tests do support well the cross scenario. We are
working on this but it moves slower then we wish it would be.

From what you've listed, we only have v5T and one cortex A8. We could

provide more A8/A9s, M3s (mBed) boards and also models of all
variations possible.

Does it make any difference running them here at ARM or locally on
OSUOSL? We could easily help them get more boards, too...

Llvm lab seems is the right place to put them, so people could get an
access to boards for debugging.

As a general note, my observations shows that slow builds gets less
attention from the community then fast ones.
This could be an issue with small boards.
I have been tried native builds on a beagleboard, it took about 4
hours to build without the tests which is a bit too slow.
We will need to find a way to make ARM builds faster.

Thanks

Galina

9

Correction; Should be:

The issue with cross compilation is that tests do support well the cross scenario.

Tests DO NOT support well the cross

Thanks

Galina

10

It is possible.

Good news! :wink:

We host several build slaves which cross compile. The issue with cross
compilation is that tests do [not] support well the cross scenario. We are
working on this but it moves slower then we wish it would be.

Thought so. Again, this is something we can help, because this is very
important for us. Running Linux tests is also important, of course,
but mostly for A* targets.

Since we want the support for (at least) A, R and M families to be as
good as possible, bare-metal tests are fundamental.

We can continue off-list on how can we help to make that easier for
cross-compilation tests.

Llvm lab seems is the right place to put them, so people could get an
access to boards for debugging.

Agreed!

As a general note, my observations shows that slow builds gets less
attention from the community then fast ones.
This could be an issue with small boards.

Unfortunately, this will always be an issue with embedded systems. Not
that they're slow themselves, but the debug connection is never as
fast as we'd like.

If we had multiple boards, say 6 mBeds, we could run 6 tests in
parallel if the LIT tests could do cross-tests easily.

We do that with models already and it works perfectly (and fast). That
could be another solution we should try...

I have been tried native builds on a beagleboard, it took about 4
hours to build without the tests which is a bit too slow.

Yes. Beagle board is single-core and it does take a long time to
compile. This is why we need cross-tests, we compile in a quad-core
machine and test on 4 beagleboards at the same time.

We will need to find a way to make ARM builds faster.

Overclock? :smiley:

Now seriously, I think cross-compilation + parallelization (hardware
or models) should do the trick.

cheers,
--renato