LLC ARM Backend maintainer

I am very interested in seeing a qualification plan for ARM given that it is
a widely used target with several combinations of options/modes to be
tested. I & my team use ARM hardware for running tests and we run all LLVM
test suite tests as part of qualification process. I had started a similar
conversation in llvm-commits, but this is probably the right forum. It will
save everyone a lot of time if we can all agree on qualification tests and
options for ARM and would be happy to initiate the discussion and process
for this.

--Raja

Hi,

It goes without saying that I +1 this.

Cheers,

James

Hi James,

It goes without saying that I +1 this.

Well, the answer should be pretty easy, I think:

1. We should define which ARM-related features (in general, e.g.
platforms, cores, modes, etc.) we consider "release important"
2. We should define the conditions how the features in 1. should be tested
3. Someone should perform such testing for each release, provide help
with reproduction of the problems (consider e.g. PR11107, w/o Bill's
help it would be extremely hard to reproduce the problem, since it
manifests only on arm/darwin).

Any volunteers? :slight_smile:

4. We should be able to guarantee that release-blocking bugs on ARM
targets will be fixed (if technically possible) before the actual
release.

There is no point in define ARM as a release-blocking target if there
is no commitment in actually fixing release bugs. What keeps ARM on
the bench is just the lack of general commitment. Somebody has to own
it, for real.

cheers,
--renato

Admittedly we’re very interested in becoming ARM backend maintainers as our product heavily relies on LLVM.

However, we don’t have testing resources to test both our product and LLVM on a host of target boards. We have some chumbys, beagleboards, iPhones, iPod Touches, tables, Android Phones, etc. And most of those are already booked solid with our own regression tests (most of which are based on llvm-test-suite)

Could ARM enable us with testing hardware/resources?

Thanks!

Joe Abbey
Software Architect
Arxan Technologies, Inc.
1305 Cumberland Ave, Ste 215
West Lafayette, IN 47906
W: 765-889-4756 x2
C: 765-464-9893
jabbey@arxan.com
www.arxan.com

I think we need a group of maintainers rather than a single maintainer given the spectrum of ARM targets/ISAs/platforms required to support and the amount of people/system resources required. I & my team plan to actively participate in the bug-fixing process during the release cycle. If we can divide the bugs among the maintainers and establish a requirement that all open ARM bugs must be fixed/addressed (at least analyzed if cannot be fixed) by release time, it will go a long way in ensuring high quality releases for ARM

–Raja

The ARM-based Raspberry Pi is very close to shipping. The entry-level
unit will retail for just $25 with a higher-end model for $35. I
don't recall who makes it, but there is an ARM desktop computer that
retails for $149, with the same company making an ARM Netbook for
$199.

I have a Gumstix Overo Fire COM which I'm using mostly to experiment
with ARM assembly code and ARM-specific optimizations in C, C++ and
Objective-C. When I get it configured reliably I'd be happy to
dedicate it as a test machine most of the time.

https://www.gumstix.com/store/product_info.php?products_id=227

This has an ARM Cortex A8 with NEON SIMD as well as a Texas
Instruments DSP. The DSP isn't programmed yet but could readily be
with TI's DSP/BIOS and Code Composer Studio.

I just spent days and days and days building all of Anstrom Linux for
it, so it has a full GUI desktop environment, development tools and
God Almighty Knows What Else installed on it.

It is on my home LAN which has a dynamic IP, but I understand there is
a way I can set up a tunnel to my web server which has a fixed IP. I
can also ask my hosting service for an additional IP so if necessary I
could tunnel ALL the ports to my Gumstix board.

I also have a first-generation Apple iPhone which has iOS 3.0.6
installed on it.

To use it for a testing bot I'd need to prevent it from ever going to
sleep; I *think* I can do that by setting Auto-Lock to Never in the
General section of the Settings App.

I would need to package the tests as some kind of iPhone App which
(maybe) I could load on the phone automatically by automating either
iTunes or the iPhone Configuration Utility with AppleScript.

I'm not sure what ARM CPU my first-gen iPhone has but I think it uses
the arm6 instruction set.

Finally I have three Oxford Semiconductor FireWire / USB / Parallel
IDE storage bridge chip target boards.

They all have ARM7TDMI cores with 64 kb to 128 kb of Flash and some
microscopic amount of RAM: my OXFW911 chip has 64 kb of 16-bit Flash
but only 1800 BYTES - not Megabytes, not Kilobytes, but just BYTES -
of RAM.

There wouldn't be enough memory in these things to run a shell, but
any of the tests that could be packaged as a subroutine with a single
entry point could be compiled into the firmware.

These all run 16-but Thumb 1 and 32-bit ARM code. My iOS devices - I
also have a first-gen iPad and an iPhone 4 - can run Thumb 2.

If you'd like me to set these all up to do building and testing I'd be
happy to, but I need some handholding.

I've been trying to bootstrap llvm-gcc4.2 on Ubuntu 11.04 with the
aim of using the CLang analyzer on ZooLIb (http://www.zoolib.org/) but
the bootstrap always fails because of bizarre problems with autoconf's
configure scripts.

I've tried to debug the configure scripts but have yet to have any
joy. IMHO autoconf is A Tool of the Devil. ZooLib is even more
cross-platform than autoconf is, but does not need shell scripts at
all to configure it.

I need to catch some ZZZs now, I've been up all night. I'll have
another go at configuring my Overo this evening, then will try
bootstraping LLVM for x86 again. If that works then I'll bootstrap it
for ARM as well.

You can also run the tests in a emulator. The timing won't be
accurate but you should be able to catch gross speed regressions.

There are several emulators available; the ones I know about are
Softgun, QEMU, the GNU Debugger GDB and the ARM Holdings development
system emulator. There is another one that I came across once whose
name I don't recall.

Some emulators emulate a full computer with I/O devices so you can run
an OS on them. These emulators could then host the build and not just
the test.

There are other simulators that just execute a single subroutine tree
and provide standard input and output.

The cool thing about this kind of emulator is that one can count the
number of instructions required to go from the beginning to the end of
the subroutine, or with the addition of a memory map that gives access
times for various kinds of memory (16-bit Flash or Dynamic RAM) you
can get cycle counts for the subrouine.

The ARM Holdings emulator does this; I used it with great success to
profile an Advanced Encryption Standard encryptor a while back.

Ever Faithful,

Don Quixote

There have been a plethora of ARM Instruction Set Architecture
variants over the years. Which ones do we expect LLVM to support
right now, and which as-yet unsupported ones do we wish to support?

Am I correct that ARM7TDMI is the oldest, slowest core still in use?

The Apple iPad 2 and iPhone 4S have the Apple A9 CPU, which I
understand is a dual-core Cortex A9.

The iPhone 4 and first-generation iPad have the Cortex A8.

Xcode gives on the option of targeting either the arm6 or arm7 ISAs,
with the recommended build being "Universal" in that the build is done
twice with both ISA variants built into the same executable. I
understand older iPhones and iPod Touches support the arm6 but not
arm7.

Ever Faithful,

Don Quixote

The ARM Holdings emulator does this; I used it with great success to
profile an Advanced Encryption Standard encryptor a while back.

It is indeed a useful piece of kit. We do a lot of our internal regression tests on it, and also run LLVM's regression tests every night on it (as well as PlumHall, EEMBC and SpecInt). Unfortunately it's not exactly software we can give away or give access to.

Indeed. My client at the time bought me my copy; it was bundled with
the Oxford Semiconductor OXFW911 FireWire/IDE development board.

The whole development system, I was told, cost USD $ 4,000.00.

Happily, I still have it all, but it might take some time to actually
lay my hands on it.

Besides the memory mapping, the emulator also supports emulating I/O
devices and interactions with other kinds of cores. It's meant to
enable programming System-on-Chip designs before one has real hardware
available.

Cheerfully,

Don Quixote

Well how about as a strawman… taking some options from http://en.wikipedia.org/wiki/List_of_ARM_microprocessor_cores and http://en.wikipedia.org/wiki/List_of_applications_of_ARM_cores

LLVM Supports:

ARMv4T → ARM7TDMI
ARMv5TE → ARM926EJ-S
→ XScale
ARMv6 → ARM1136J(F)-S

ARMv6ZK → ARM1176JZ(F)-S
ARMv7A → Cortex-A8
Cortex-A9
ARMv7M → Cortex-M3

Granted there are many many more listed within in ARM.td, but I think we should start with a small subset and build on it as we begin guaranteeing maintenance. Once we compile a set of ISAs and implementations, we could start logically dividing the load and assigning maintainer roles.

Joe Abbey
Software Architect
Arxan Technologies, Inc.
1305 Cumberland Ave, Ste 215
West Lafayette, IN 47906
W: 765-889-4756 x2
C: 765-464-9893
jabbey@arxan.com
www.arxan.com

Does the LLVM code generator generate Thumb code in addition to ARM code?

For those who don't know ARM, Thumb is a subset of ARM in which each
instruction is 16-bits in size.

ARM instructions are 32-bits; besides working poorly on 16-bit Flash,
compilers have a hard time taking advantage of the ARM ISA's unique
advantages, such as every instruction being conditional and most
instructions have an optional shift or rotate of one of the register
operands. The result is poor code density.

Don't confuse the 16-bit instructions with a 16-bit ISA. Thumb can
still address 32 bits of memory and has 32 bit registers. In fact
when executing Thumb code, a single Thumb instruction is placed in an
instruction prefetch buffer, decompressed to a 32-bit ARM instruction
and only then is it executed.

I read somewhere that the instructions present in the Thumb ISA were
chosen to be what most C compilers generate.

There is not just the Thumb that the ARM7TDMI executes, but there is
also a slightly different and hopefully improved Thumb 2 instruction
set that later cores like the Cortex A8 can execute.

I'm pretty sure any CPU that can run Thumb 2 code can also run the
original Thumb code.

One can mix the two ISA through the use of instructions that switch
the modes; this is called "Interworking". The regular way to return
from an instruction is to put the return address into register 15, but
the instruction BX (if I remember correctly) will both return from a
subroutine and switch the ISA.

I took advantage of this in my AES encryptor, which I wrote in tightly
hand-optimized assembly, by writing the less time-critical outer loops
in slower but more-compact Thumb code, with the more time-critical
inner loops being in the ARM ISA.

In some hopeful future day LLVM will be able to do this as well; if
you have a compilation unit that is mostly Thumb but you inline some
functions or unroll some loops, you should consider interworking to
ARM mode there as well.

I would expect that we would want profiler-guided optimization to drive that.

I'm the code owner of LLVM codegen and targets. I'm also the one of main developers on the original ARM target. That means, I would make the decisions on major development on ARM target if there are decisions to be made.

But my role is very different from what people are looking for in this thread. To properly qualify a target like ARM which are supported on many different CPUs and platforms, we need a group of volunteers. This is critical near release time for obvious reasons, but it's also welcome between releases as well. LLVM.org doesn't explicitly name these "maintainers". I personally don't see a benefit because we don't want to force people to do release qualification when they might not be available. I think what we usually do is for the release manager to gather a group of volunteered. These are the folks who are responsible for qualification the targets of their interests on their preferred platform.

Evan

Testing on arm-darwin outside of Apple is _complicated_. I had it
partially working about two years ago.

The general approach is to use rsh/ssh to remotely execute each test
on the target, which outside of Apple requires jailbreaking. In the
old test system, setting up REMOTE_CLIENT=ssh, REMOTE_PORT, etc. would
automatically use ssh to run the tests. In lit, it looks like
TestRunner.py has support for external shells that may be intended for
this purpose; I can't tell. The test-suite and bugpoint code for
working this way is still present.

The problem with this setup is that it assumes that the target will be
running the tests at the same paths as they were built on the host.
That cannot be achieved outside of Apple because all the NFS
components are not present in the released/jailbroken iOS. It should
be straight-forward to build these pieces from the source, but I
stopped investigating this at this point.

If somebody else gets this working outside of Apple, please post more
instructions to the list. I know I felt more comfortable at least
being able to hand run a few specific tests on arm-darwin against my
ARM patches before submitting since I was making large changes to
calling conventions and they are quite different on arm-darwin. Other
areas of ARM work probably aren't as likely to need to test on
arm-darwin outside of Apple.

deep

Atmel makes a whole bunch of different ARM-based System-on-Chip
modules. These come with some Flash, RAM and I/O all on one chip.

I expect it would be reasonably affordable to buy one of Atmel's
developer target boards for each instruction set variant LLVM
supports.

My Overo Fire COM is Gumstix' fanciest CPU board. I also have a Tobi
expansion board for 100 MBPS ethernet, DVI video, USB for keyboard and
mouse and (I expect) USB hard drives, USB serial for the Linux console
and so on. The CPU board also has WiFi and Bluetooth.

It's got enough memory and is fast enough that with some additional
storage it can be used as a desktop computer.

All their Overo boards have Cortex A8s, and so support ARMv7A.

Gumstix also sells a much cheaper board called the Verdex Pro that use
Marvell chips "With Xscale". I think that means Marvell must have
licensed the Xscale IP from Intel.

The Xscale core indicates that the Verdex Pro supports the ARMv5TE instructions.

The Gumstix Verdex Pross don't have as much memory as the Overos so
they may not be useful as desktop computers but they would certainly
be suitable for running LLVM's test suite and some benchmarks.

I see, so perhaps the LLVM ARM Backend is in need of a method of organizing volunteer qualifiers, as releases near?

Has this generally been organized via this mailing list?

Joe

Joe Abbey
Software Architect
Arxan Technologies, Inc.
1305 Cumberland Ave, Ste 215
West Lafayette, IN 47906
W: 765-889-4756 x2
C: 765-464-9893
jabbey@arxan.com<mailto:jabbey@arxan.com>
www.arxan.com<http://www.arxan.com/>

I see, so perhaps the LLVM ARM Backend is in need of a method of organizing volunteer qualifiers, as releases near?

Has this generally been organized via this mailing list?

Yes. Bill Wendling manages the releases, so you'll want to coordinate with him.

-Jim

Testing on arm-darwin outside of Apple is _complicated_. I had it
partially working about two years ago.

The general approach is to use rsh/ssh to remotely execute each test
on the target, which outside of Apple requires jailbreaking. In the
old test system, setting up REMOTE_CLIENT=ssh, REMOTE_PORT, etc. would
automatically use ssh to run the tests. In lit, it looks like
TestRunner.py has support for external shells that may be intended for
this purpose; I can't tell. The test-suite and bugpoint code for
working this way is still present.

The problem with this setup is that it assumes that the target will be
running the tests at the same paths as they were built on the host.
That cannot be achieved outside of Apple because all the NFS
components are not present in the released/jailbroken iOS. It should
be straight-forward to build these pieces from the source, but I
stopped investigating this at this point.

This is the biggest problem. It's obviously a solvable problem but someone will have to devote some resource to it. It's not going to help with 3.0 release, but it will help future releases tremendously.

Any volunteer?

Evan

The benefit of having people run tests on ARM during the release process is to ferret out bugs that could potentially affect our release platforms. I don't want or expect people to spend all of their time doing full release testing on a platform we aren't going to officially support in 3.0. But during the release, we do generate source tarballs and ask people to download them and test them. I would encourage ARM developers to test them.

That said, we are not looking to officially support ARM for the 3.0 release. It's too late in the process to change the officially supported platforms list.

-bw

I think we need a group of maintainers rather than a single maintainer given the spectrum of ARM targets/ISAs/platforms required to support and the amount of people/system resources required. I & my team plan to actively participate in the bug-fixing process during the release cycle. If we can divide the bugs among the maintainers and establish a requirement that all open ARM bugs must be fixed/addressed (at least analyzed if cannot be fixed) by release time, it will go a long way in ensuring high quality releases for ARM

Its very unrealistic to require that all ARM bugs be fixed for a release. There is no way that this would feasible work and get the release out in a timely manner. You need to have a very concrete list of requirements to consider the release to be qualified for ARM. I suggest looking at we we currently do:
http://llvm.org/docs/HowToReleaseLLVM.html#criteria

Bill is going to update this to reflect us dropping llvm-gcc, but thats the general idea. When determining the release criteria, I would advise starting off small. You don't need to come up with the perfect solution up front. Pick a few tests, bootstrap, and then which processors are to be qualified.

Once the criteria is established, then continuous testing needs to occur via our buildbot infrastructure. Given our short release cycle, we can't not test something for 6 months and then suddenly decide to test it during the release cycle. Too many surprise bugs will show up and may take a very long time to fix. Its much better to continuously test and have a handful of issues come release time.

Volunteers are needed to be the qualifier for some arch/platform for releases. If someone is interested in filling these roles, please talk to Bill as he is the current release manager.

And as always, we need volunteers to fix release blocker bugs during the release cycle. This has sometimes been big problem during a release cycle and hopefully more will start to get more volunteers. This is why we need continuous testing of release criteria because we just don't have enough volunteers to fix everything at the last minute.

Lastly, anyone can contribute to the ARM backend, so I don't think there is anything really stopping people at the moment for helping "maintain" the ARM backend. Evan is the code owner who approves all changes of course.

Thanks,
Tanya