Running lit (googletest) tests remotely

Hello all,

we are trying to convert some of the lldb tests to lit (for these
specific tests we are using the googletest format). One of our
requirements is that we are able to run these tests remotely, so that
we are able to verify that we can debug e.g. android arm binaries even
though our development environment runs x86 linux).

As far as I am aware, right now there is no support for that in lit:
running check-lldb-unit target in a cross-compile situation will
attempt to run the run the test binaries as if they were native and
fail horribly.

Based on a not-too-detailed examination of the lit codebase, it does
not seem that it would be too difficult to add this capability: During
test discovery phase, we could copy the required files to the remote
host. Then, when we run the test, we could just prefix the run command
similarly to how it is done for running the tests under valgrind. It
would be up to the user to provide a suitable command for copying and
running files on the remote host (using rsync, ssh, telnet or any
other transport he chooses).

What do you think? Would something like that be a welcome addition to
the llvm testing infrastructure? Has anyone tried to do something like
that and hit major road blocks I did not anticipate?

Or, if you have any suggestions on how to run tests in cross-compile
setting differently, I'd love to hear about them.

regards,
pavel

You might want to look at the compiler-rt tests, which have support for remote runs. Search for the ‘%run’ substitution. It’s probably not directly applicable to gtest tests, though.

This seems to be the crux to me: What does "required files" mean?
- All the executables mentioned in the RUN line? What llvm was compiled as a library, will we copy those too?
- Can tests include other files? Do they need special annotations for that?

As another example: The llvm-testsuite can perform remote runs (test-suite/litsupport/remote.py if you want to see the implementation) that code makes the assumption that the remote devices has an NFS mount so the relevant parts of the filesystem look alike on the host and remote device. I'm not sure that is the best solution as NFS introduces its own sort of flakiness and potential skew in I/O heavy benchmarks but it avoids the question of what to copy to the device.

- Matthias

You may have a look at libcxx/utils/libcxx/test/executor.py ; it contains a
"SSHExecutor" that can be used when running the lit tests for libcxx.

Thank you all for the pointers. I am going to look at these to see if
there is anything that we could reuse, and come back. In the mean
time, I'll reply to Mathiass's comments:

Based on a not-too-detailed examination of the lit codebase, it does
not seem that it would be too difficult to add this capability: During
test discovery phase, we could copy the required files to the remote
host. Then, when we run the test, we could just prefix the run command
similarly to how it is done for running the tests under valgrind. It
would be up to the user to provide a suitable command for copying and
running files on the remote host (using rsync, ssh, telnet or any
other transport he chooses).

This seems to be the crux to me: What does "required files" mean?
- All the executables mentioned in the RUN line? What llvm was compiled as a library, will we copy those too?

For executables, I was considering just listing them explicitly (in
lit.local.cfg, I guess), although parsing the RUN line should be
possible as well. Even with RUN parsing, I expect we would some way to
explicitly add files to the copy list (e.g. for lldb tests we also
need to copy the program we are going to debug).

As for libraries, I see a couple of solutions:
- declare these configurations unsupported for remote executions
- copy over ALL shared libraries
- have automatic tracking of runtime dependencies - all of this
information should pass through llvm_add_library macro, so it should
be mostly a matter of exporting this information out of cmake.
These can be combined in the sense that we can start in the
"unsupported" state, and then add some support for it once there is a
need for it (we don't need it right now).

- Can tests include other files? Do they need special annotations for that?

My initial idea was to just copy over all files in the Inputs folder.
Do you know of any other dependencies that I should consider?

As another example: The llvm-testsuite can perform remote runs (test-suite/litsupport/remote.py if you want to see the implementation) that code makes the assumption that the remote devices has an NFS mount so the relevant parts of the filesystem look alike on the host and remote device. I'm not sure that is the best solution as NFS introduces its own sort of flakiness and potential skew in I/O heavy benchmarks but it avoids the question of what to copy to the device.

Requiring an NFS mount is a non-starter for us (no way to get an
android device to create one), although if we would be able to hook in
a custom script which does a copy to simulate the "mount", we might be
able to work with it. Presently I am mostly thinking about correctness
tests, and I am not worried about benchmark skews

regards,
pl

Thank you all for the pointers. I am going to look at these to see if
there is anything that we could reuse, and come back. In the mean
time, I'll reply to Mathiass's comments:

Based on a not-too-detailed examination of the lit codebase, it does
not seem that it would be too difficult to add this capability: During
test discovery phase, we could copy the required files to the remote
host. Then, when we run the test, we could just prefix the run command
similarly to how it is done for running the tests under valgrind. It
would be up to the user to provide a suitable command for copying and
running files on the remote host (using rsync, ssh, telnet or any
other transport he chooses).

This seems to be the crux to me: What does "required files" mean?
- All the executables mentioned in the RUN line? What llvm was compiled as a library, will we copy those too?

For executables, I was considering just listing them explicitly (in
lit.local.cfg, I guess), although parsing the RUN line should be
possible as well. Even with RUN parsing, I expect we would some way to
explicitly add files to the copy list (e.g. for lldb tests we also
need to copy the program we are going to debug).

As for libraries, I see a couple of solutions:
- declare these configurations unsupported for remote executions
- copy over ALL shared libraries
- have automatic tracking of runtime dependencies - all of this
information should pass through llvm_add_library macro, so it should
be mostly a matter of exporting this information out of cmake.
These can be combined in the sense that we can start in the
"unsupported" state, and then add some support for it once there is a
need for it (we don't need it right now).

Sounds good. An actively managed list of files to copy in the lit configuration is a nice simple solution provided we have some regularily running public bot so we can catch missing things. But I assume setting up a bot was your plan anyway.

- Can tests include other files? Do they need special annotations for that?

My initial idea was to just copy over all files in the Inputs folder.
Do you know of any other dependencies that I should consider?

I didn't notice that we had already developed a convention with the "Inputs" folders, so I guess all that is left to do is making sure all tests actually follow that convention.

As another example: The llvm-testsuite can perform remote runs (test-suite/litsupport/remote.py if you want to see the implementation) that code makes the assumption that the remote devices has an NFS mount so the relevant parts of the filesystem look alike on the host and remote device. I'm not sure that is the best solution as NFS introduces its own sort of flakiness and potential skew in I/O heavy benchmarks but it avoids the question of what to copy to the device.

Requiring an NFS mount is a non-starter for us (no way to get an
android device to create one), although if we would be able to hook in
a custom script which does a copy to simulate the "mount", we might be
able to work with it. Presently I am mostly thinking about correctness
tests, and I am not worried about benchmark skews

Sure, I don't think I would end up with an NFS mount strategy if I would start fresh today. Also the test-suite benchmarks (esp. the SPEC) ones tend to have more complicated harder to track inputs.

- Matthias

Thank you all for the pointers. I am going to look at these to see if
there is anything that we could reuse, and come back. In the mean
time, I’ll reply to Mathiass’s comments:

Based on a not-too-detailed examination of the lit codebase, it does
not seem that it would be too difficult to add this capability: During
test discovery phase, we could copy the required files to the remote
host. Then, when we run the test, we could just prefix the run command
similarly to how it is done for running the tests under valgrind. It
would be up to the user to provide a suitable command for copying and
running files on the remote host (using rsync, ssh, telnet or any
other transport he chooses).

This seems to be the crux to me: What does “required files” mean?

  • All the executables mentioned in the RUN line? What llvm was compiled as a library, will we copy those too?
    For executables, I was considering just listing them explicitly (in
    lit.local.cfg, I guess), although parsing the RUN line should be
    possible as well. Even with RUN parsing, I expect we would some way to
    explicitly add files to the copy list (e.g. for lldb tests we also
    need to copy the program we are going to debug).

As for libraries, I see a couple of solutions:

  • declare these configurations unsupported for remote executions
  • copy over ALL shared libraries
  • have automatic tracking of runtime dependencies - all of this
    information should pass through llvm_add_library macro, so it should
    be mostly a matter of exporting this information out of cmake.
    These can be combined in the sense that we can start in the
    “unsupported” state, and then add some support for it once there is a
    need for it (we don’t need it right now).
    Sounds good. An actively managed list of files to copy in the lit configuration is a nice simple solution provided we have some regularily running public bot so we can catch missing things. But I assume setting up a bot was your plan anyway.
  • Can tests include other files? Do they need special annotations for that?
    My initial idea was to just copy over all files in the Inputs folder.
    Do you know of any other dependencies that I should consider?
    I didn’t notice that we had already developed a convention with the “Inputs” folders, so I guess all that is left to do is making sure all tests actually follow that convention.

The Google-internal execution of LLVM’s tests relies on this property - so at least for the common tests and the targets Google cares about, this property is pretty well enforced.

My plan was to use it on already-running lldb android bots. These run
lldb tests only. They're also currently in "experimental" mode (not
sending emails) because they are not reliable enough (main, but not
only reason is the flakyness of the physical connection to the android
devices). With the new testing strategy, the the connection might not
be stressed so much, so we may actually be able to enable it, but it's
hard to say. In any case, we would be keeping an eye out for the bots,
as we do now.

I've taken a closer look at the existing remote runners suggested in
this thread, and it seems none of them really fit our purpose:
- the one in libc++ only handles the .cpp tests (which are very libc++
specific), and not the (more complicated) ShTest, nor GoogleTest
- the compiler-rt %run substitution seems to be used for running the
tests in an emulator -- for example it would be hard to handle copying
of inputs to the remote machine with a substitution based solution
(and there is no support for that now)
- the one in llvm-testsuite is quite interesting -- it has code which
gains deep understanding of the RUN lines, which should be enough to
modify the sequence of commands to add copy commands to the test
script. However, that does not directly apply to googletest tests and
also is probably a big overkill for them (googletest.py is about 150
LOC, whereas the llvm-testsuite python files are nearly 1k). I am
hoping we can add the remote execution to googletest.py without
significantly increasing it's complexity (a prototype, which did not
handle copying input files was about a 10-line patch).

Therefore, I propose to modify googletest.py, which would add the
ability to run remotely for all gtest-based tests. Right now, we don't
have a need to run ShTests remotely in lldb (we have none), but we may
have soon. If that happens, we can look at porting some of the logic
in llvm-testsuite, so that it is available to other users as well.

How does that sound?

pl