QEMU testing for LIT execution tests

Dan/Daniel/Eric,

I'm testing a bare-metal ARM toolchain, and I've hacked up my local copy of lit to get it to run libcxx tests on qemu. I wanted to pick your collective brains to see if there was a better way of doing this.

What I have implemented is here (don't take this as a formal review request, we can do that later once we figure out the best direction to go):
lit part: ⚙ D5014 Add QEMU args to lit
libcxx part: ⚙ D5015 Add QEMU testing to libcxx's lit config

Talking with Dan Albert earlier, he pointed out an existing review (⚙ D4594 [libcxx] Add support for Android targets to lit.cfg.) where he has implemented something vaguely similar for Android. Looking at that, I've realized that maybe it makes sense to make an interface for the different actions of:
   * Compiling the test
      o Maybe the compiler isn't on the machine that runs the test?
      o Maybe the compile jobs could be distributed to different machines?
   * Running the test
      o Perhaps the test needs to be sent out to a remote device?
      o Perhaps the test can be scheduled on more than one remote device?

Then that interface could be implemented in a couple of generic ways:
   * scp & ssh for remote hosts (or QEMU with an OS image)
   * Wrapper around Android's adb
   * Thin wrapper around "run it in this machine's shell" (what we currently have)

What are your thoughts on this? Is this worth pursuing? What other use-cases can we support by structuring LIT this way? I don't yet have a concrete proposal for what this would look like, but I can throw one together if there's interest.

Cheers,

Jon

No specific comments at the moment, but I do want to say that I very much believe this is worth pursuing. Better testing infrastructure for remote targets will be an absolutely fantastic improvement. Thank you and Dan both for pushing forward on the topic.

Jim

+1

I remember Greg (cc'd) did some of that in compiler-rt (or was it
sanitizers?). He's not active in LLVM any more, but maybe you guys
could share some thoughts.

cheers,
--renato

Hi Jon,

I’m doing work in a similar vein as yours at the moment. We have several arm platforms (hardware and qemu) running either Linux or Android and want to run test-suite on them using x86->arm/aarch64 cross compilers dispatched via LNT.

I’ve had some success bypassing lit and LNT by modifying RunSafely.sh within test-suite. It’s likely this won’t completely solve your problem but I think it would be a good idea to find the right path forward. I posted the patch I made in another thread on the list, titled under “test-suite RunSafely.sh remote execution with test copy”. I’m using a variant on the method you outlined talking with Dan Albert to accomplish it. Like your code it’s a bit of a hack, but I’ve gotten it to work provided the remote host has ssh and scp. I decided against the adb path for simplicity and consistency in the code modifications. Were a solution provided that had both ssh/scp and adb I’d be thrilled.

Finally, I’ve done some work on getting Ubuntu 14.04LTS aarch64 running on Qemu but have not yet posted the instructions online. If you (or anyone) is interested please let me know and I’ll make it a priority to document the steps I used.

-Brian

I hope this isn’t considered spamming the list as I think it’s relevant to QEMU and lit testing on ARM-64 platforms. After a few people contacted me for the instructions to setup Ubuntu aarch64 on QEMU I wrote them up here:

http://rzycki.blogspot.com/2014/08/using-qemu-to-run-ubuntu-arm-64-bit.html

Thanks,

-Brian

Hi Brian,

Not at all spam, we welcome helps with setting up hardware to test the
compiler. Though we probably wouldn't find a place where to link this
from on our docs, it's a good reference on the list.

We have an AArch64 buildbot using QEMU user emulation here, FYI:

http://lab.llvm.org:8011/builders/llvm-aarch64-linux

Thanks!
--renato

Hi Jon,

+1 on adding support to run cross-compiled libcxx tests. +Marshall

To add cross-compilation support to the compiler-rt test suite, I
added '%run' variable to the 'RUN:' lines in each lit test. For
example:

https://github.com/llvm-mirror/compiler-rt/blob/master/test/asan/TestCases/atexit_stats.cc

and substituted '%run' with 'config.emulator' in the lit configuration file:

https://github.com/llvm-mirror/compiler-rt/blob/master/test/lit.common.cfg

which is set via CMake option COMPILER_RT_EMULATOR:

https://github.com/llvm-mirror/compiler-rt/blob/master/test/lit.common.configured.in

To configure the build to use QEMU (or adb, scp, etc):

$ cmake -DCOMPILER_RT_EMULATOR=qemu-aarch64

If you need additional parameters, you can either add them to the
COMPILER_RT_EMULATOR string or add a wrapper shell script.

To ensure new tests include '%run', I added the script 'litlint.py'
and invoke it within the CMake build:

https://github.com/llvm-mirror/compiler-rt/blob/master/lib/sanitizer_common/scripts/litlint.py

Let me know if you have any questions. Feel free to include me on
Phabricator reviews.

-Greg

So I started prototyping remote testing for libcxx over the weekend:
   https://github.com/jroelofs/libcxx/tree/remote_test

The SSHExecutor isn't quite finished yet, but this should give an idea of what I've got in mind.

Does this look generic enough to work for other projects under the llvm-umbrella (LNT, test-suite, compiler_rt, libcxxabi, etc)? Are there use cases that you see where this will not work?

Jon

Hi Jon,

Compiler-rt, libcxx, libcxxabi should all use the same
cross-compilation testing strategy. Compiler-rt already has a working
solution. If it is inadequate for libcxx or libcxxabi, can we start
by addressing those deficiencies?

Thanks,
Greg

Hi Jon,

Compiler-rt, libcxx, libcxxabi should all use the same
cross-compilation testing strategy. Compiler-rt already has a working
solution. If it is inadequate for libcxx or libcxxabi, can we start
by addressing those deficiencies?

One issue to start with is that none of the libcxx/libcxxabi tests have RUN lines. All of the tests are either: *.fail.cpp ones that fail to compile, or *.pass.cpp ones that are expected to exit with a return status of 0 without failing an asserts. Granted, this is more of a test semantic issue, but there are a *lot* of tests to update in order to converge on compiler_rt's 'RUN: ' format.

[Sidenote: I don't see any 'RUN:' lines on the builtins unit tests. How does it work for those ones?]

Another issue, which Dan Albert recently pointed out to me, is that there are several libcxx tests which would require copying in of test data (in addition to test executables). I think that this necessitates inventing a new LIT syntax, something like '// DEPENDS-ON: filename.dat' in order to inform the lit.cfg that the dependent file also needs to be copied to the remote target.

A third issue is that I don't think the current '%run' solution works for full Canadian cross testing (i.e. run lit on BUILD, clang/llvm/whatever on HOST, and the tests themselves on TARGET, where {BUILD,HOST,TARGET} are all different machines). This use case is of particular interest to me. I suppose %run could be renamed %run_target, add %run_host, and then have an implied no-op %run_build on commands that don't have %run_target or %run_host. Perhaps someone might want to have BUILD be a different machine than where LIT runs, in which case %run_build would have to be explicit (but I'm not really interested in that use case).

Cheers,
Jon

Jon,

One issue to start with is that none of the libcxx/libcxxabi tests have RUN lines.

Sorry, I didn't mean to imply that 'RUN:' lines should be added to
libcxx tests. It's fine that every one is implicitly:

    // RUN: %clang %s -o %t && %run %t

I think that this necessitates inventing a new LIT syntax, something like '// DEPENDS-ON: filename.dat'

If you didn't want to change lit, this would be a good place to add a
'RUN:' line. Something like:

    // RUN: %clang %s -o %t && %run %t filename.dat

and then your %run implementation would see the dependency on the
local file and ship it over to TARGET.

A third issue is that I don't think the current '%run' solution works for
full Canadian cross testing (i.e. run lit on BUILD, clang/llvm/whatever
on HOST, and the tests themselves on TARGET, where {BUILD,
HOST,TARGET} are all different machines).

In compiler-rt, I'd configure such a build with the following CMake variables:

LLVM_LIT=$BUILD/lit
CMAKE_C_COMPILER=$HOST/clang
COMPILER_RT_EMULATOR=$TARGET/qemu

For libcxx, how about adding LIBCXX_EMULATOR, and then changing
lit.cfg to acknowledge it when invoking executables? You'd then put
your ssh functionality in a standalone script. If a command-line
interface is too inflexible (or painful), perhaps add a
LIBCXX_PY_EMULATOR variable that points to a Python module.

-Greg

Jon,

One issue to start with is that none of the libcxx/libcxxabi tests have RUN lines.

Sorry, I didn't mean to imply that 'RUN:' lines should be added to
libcxx tests. It's fine that every one is implicitly:

     // RUN: %clang %s -o %t && %run %t

... and then when an explicit 'RUN' line is provided, use that instead? That sounds reasonable.

I think that this necessitates inventing a new LIT syntax, something like '// DEPENDS-ON: filename.dat'

If you didn't want to change lit, this would be a good place to add a
'RUN:' line. Something like:

     // RUN: %clang %s -o %t && %run %t filename.dat

and then your %run implementation would see the dependency on the
local file and ship it over to TARGET.

This also sounds reasonable.

A third issue is that I don't think the current '%run' solution works for
full Canadian cross testing (i.e. run lit on BUILD, clang/llvm/whatever
on HOST, and the tests themselves on TARGET, where {BUILD,
HOST,TARGET} are all different machines).

In compiler-rt, I'd configure such a build with the following CMake variables:

I'm not sure we're on the same page... Your example, and the way that lit is currently set up, assumes BUILD and HOST are the same machine.

LLVM_LIT=$BUILD/lit
CMAKE_C_COMPILER=$HOST/clang
COMPILER_RT_EMULATOR=$TARGET/qemu

Let me give a concrete example:

BUILD=i686-pc-linux-gnu
HOST=i386-pc-win32
TARGET=arm-elf (on say, real hardware, or i686-pc-linux/bin/qemu-system-arm, or i386-pc-win32/bin/qemu-system-arm)

Here I'd like to run lit on the linux machine such that test compilation happens on the windows machine and test execution happens on arm hardware or qemu (where qemu is running on either the windows machine, or the linux machine, depending on how the lit.site.cfg is set up).

Jon

Jon,

One issue to start with is that none of the libcxx/libcxxabi tests have RUN
lines.

Sorry, I didn't mean to imply that 'RUN:' lines should be added to
libcxx tests. It's fine that every one is implicitly:

     // RUN: %clang %s -o %t && %run %t

... and then when an explicit 'RUN' line is provided, use that instead? That
sounds reasonable.

I think that this necessitates inventing a new LIT syntax, something like '//
DEPENDS-ON: filename.dat'

If you didn't want to change lit, this would be a good place to add a
'RUN:' line. Something like:

     // RUN: %clang %s -o %t && %run %t filename.dat

and then your %run implementation would see the dependency on the
local file and ship it over to TARGET.

This also sounds reasonable.

A third issue is that I don't think the current '%run' solution works for
full Canadian cross testing (i.e. run lit on BUILD, clang/llvm/whatever
on HOST, and the tests themselves on TARGET, where {BUILD,
HOST,TARGET} are all different machines).

In compiler-rt, I'd configure such a build with the following CMake variables:

I'm not sure we're on the same page... Your example, and the way that lit is
currently set up, assumes BUILD and HOST are the same machine.

or, at least, it assumes that lit is running on HOST.

Isn't that the point? So lit can use "qemu-arm arm-linux-gnu-clang ...".

Or are we going to run the lit inside the QEMU? I think this solution
would be harder, as you'd need more context for python, etc.

cheers,
--renato

Sorry, this thread diverged a bit from the title, which I think may have
confused things....

This started as a discussion of "how do we get libcxx's lit to run target
executables inside qemu" and turned into "how can we make that generic
enough to support actual hardware as the remote target", and further into
"how about remote testing of the compiles too".

Now as to your question, my intent is *not* to run arm-linux-gnu-clang
inside qemu-arm on the x86 linux machine. And I also don't intend to run lit
inside qemu. What I would like to do however, is something like this:

For the moment, lets assume that I want to run my tests on real hardware,
say a raspberry pi (I'll come back to how qemu ties into this in a bit).
Suppose I have these three machines:

role hostname triple
BUILD tux i686-pc-linux-gnu
HOST bob i386-pc-win32
TARGET pi armv6-unknown-linux-gnueabi

and that I've built a canadian cross toolchain starting on 'tux', where
users on machines like 'bob' build software targeting their raspberry pi's.

I want to test both the i386-pc-win32-clang (which runs on 'bob'), as well
as the executables that it produces (which run on 'pi'), but my testing
infrastructure (i.e. python, LIT, etc) has to be on 'tux'.

In this scenario, the run lines would have to be:

// RUN: %host_run %clang %s -o %t && %target_run %t

Where %host_run is some script that copies in %s and runs
armv6-unknown-linux-gnueabi-clang on 'bob', and then copies out %t.
And where %target_run is another script that copies in and runs %t on 'pi'

And now, lets bring qemu back into the picture: now suppose my target
hardware does not exist yet, so I've got to use qemu. I want to be able to
test two versions of it: 1) one where qemu is built for 'tux' and 2) one
where qemu is built for 'bob'. So now those scripts are as follows:

1)
%host_run is the same as it was before.
%target_run is a script that wraps qemu

2)
%host_run is the same as it was before.
%target_run is a script that copies %t over to 'bob', and runs %t inside
qemu on 'bob'.

TL;DR: I want to run LIT on one machine, clang on another, and the binaries
that it produces on yet another machine (or even emulator).

I hope that clarifies things a bit.

Cheers,
Jon

Jon,

// RUN: %host_run %clang %s -o %t && %target_run %t

lit has control over what it replaces %clang with, so no explicit
%host_run is needed. In compiler-rt, for example, you might configure
with:

    $ cmake -DCOMPILER_RT_TEST_COMPILER=my_clever_script.py

That script would receive the path to the input and output files,
which it could use to ssh to $HOST, compile and copy the output file
back. Earlier I suggested using CMAKE_C_COMPILER. That could work
too, but that variable is also used to compile libcxx - probably not
what you want.

-Greg

CMake runs the compiler to test it and tries to run the resulting
program, and producing a binary for another target might not work
straight away.

--renato

role hostname triple
BUILD tux i686-pc-linux-gnu
HOST bob i386-pc-win32
TARGET pi armv6-unknown-linux-gnueabi

and that I've built a canadian cross toolchain starting on 'tux', where
users on machines like 'bob' build software targeting their raspberry pi's.

Ah! Canadian cross... (*strokes moustache*).

TL;DR: I want to run LIT on one machine, clang on another, and the binaries
that it produces on yet another machine (or even emulator).

Running QEMU, ssh etc. for lit tests is doable (I've done it before
numerous times), but when I did it, I always wanted to, so the
commands were hard-coded into the tests.

I'm guessing that's what your %host_run and %target_run will do
dynamically. It may be complicated, but doable.

cheers,
--renato