Wrapping lit commands for device testing

Hi,

Tests in compiler-rt run under both iOS simulators and Android devices, which requires “wrapping” the run line with the “adb” command for Android, and adding a prefix / setting some environment variables for simulators/
In case of Android this is done by changing the compilation command to produce a wrapper which already calls “adb”, and in case of iOS simulators by explicitly adding the “%run” command in front of run lines.
The solution is far from perfect though: many tests (e.g. libFuzzer) extensively rely on using arbitrary shell commands to create and manipulate temporary text files.
Performing those commands in a different environment is done in a very ad-hoc manner: from what I see, for sanitizer tests those temporary files are then usually references in the environment variables present in the run command, which are then parsed to carry those files into the executable environment.
Doing this for libFuzzer would be very difficult and would require a lot of modifications to test.

I would prefer a more transparent approach: if LIT configuration was to provide a hook to perform arbitrary replacements/wrapping on each of the executed lines, the wrapping could be performed without modifying tests at all.

My implementation preference would be adding a method `def wrap(self, s)` (str -> str) to the format class (lit/formats/*), and then using it (if non-empty) on each line being run.
I think that is preferable to a method on a `config` object, since line transformation is a very heavyweight operation, akin to other methods available on the `format` class (test discovery semantics and test execution semantics).
Would there be any objections against such an implementation plan?

Regards,
George

Hello George,

I would be very much interested in a (generic) solution that would allow
running tests on a remote host/device. My intended use case for this is
lldb+android, and I am mostly interested in googletest-style tests. In
fact, I proposed something vaguely similar to this (but limited to gtest)
last year <http://lists.llvm.org/pipermail/llvm-dev/2017-May/113370.html>.
However, due to some internal reshuffles, this idea never got past the
prototype stage. Nonetheless, I would still like do something like this
eventually (even better if someone else does it for me :slight_smile: ).

I would love to hear more about your proposal, as my head is currently
filled with questions about how this would work. (I admit I don't know
anything about how libFuzzer is tested.) It's not clear to me whether you
intend to implement a general purpose remote test execution framework, or a
solution tailored to the needs of libFuzzer. Do you automatically handle
copying the inputs to the remote target? How do you identify what the
inputs are? Do you copy over any llvm tools executed in the run lines? What
about their dependent libraries? etc.

Maybe if you could give an example of a typical test you intend to support
and how it would work, a lot of these questions would be answered. If this
turns out to be something that could be reused for our use case, I would be
willing to chip in to help make it happen.

regards,
pavel

Hi,

Tests in compiler-rt run under both iOS simulators and Android devices,

which requires “wrapping” the run line with the “adb” command for Android,
and adding a prefix / setting some environment variables for simulators/

In case of Android this is done by changing the compilation command to

produce a wrapper which already calls “adb”, and in case of iOS simulators
by explicitly adding the “%run” command in front of run lines.

The solution is far from perfect though: many tests (e.g. libFuzzer)

extensively rely on using arbitrary shell commands to create and manipulate
temporary text files.

Performing those commands in a different environment is done in a very

ad-hoc manner: from what I see, for sanitizer tests those temporary files
are then usually references in the environment variables present in the run
command, which are then parsed to carry those files into the executable
environment.

Doing this for libFuzzer would be very difficult and would require a lot

of modifications to test.

I would prefer a more transparent approach: if LIT configuration was to

provide a hook to perform arbitrary replacements/wrapping on each of the
executed lines, the wrapping could be performed without modifying tests at
all.

My implementation preference would be adding a method `def wrap(self,

s)` (str -> str) to the format class (lit/formats/*), and then using it (if
non-empty) on each line being run.

I think that is preferable to a method on a `config` object, since line

transformation is a very heavyweight operation, akin to other methods
available on the `format` class (test discovery semantics and test
execution semantics).

Hi Pavel,

I was thinking about a generic proposal for a while,
but then I’ve realized that a much simpler heuristic suffices for all of my needs:

- Prepend all commands which should execute on the device with %run, which runs a wrapper python script
- Inside that script use a simple regexp to find all paths
- sync all paths to the device
- run the executable on the device
- sync all paths form the device back to host

Hello George,

I would be very much interested in a (generic) solution that would allow
running tests on a remote host/device. My intended use case for this is
lldb+android, and I am mostly interested in googletest-style tests.

I’m not sure about those, all my problems were in LIT tests.

In
fact, I proposed something vaguely similar to this (but limited to gtest)
last year <http://lists.llvm.org/pipermail/llvm-dev/2017-May/113370.html>.
However, due to some internal reshuffles, this idea never got past the
prototype stage. Nonetheless, I would still like do something like this
eventually (even better if someone else does it for me :slight_smile: ).

I would love to hear more about your proposal, as my head is currently
filled with questions about how this would work. (I admit I don't know
anything about how libFuzzer is tested.) It's not clear to me whether you
intend to implement a general purpose remote test execution framework, or a
solution tailored to the needs of libFuzzer.

Seemed generic enough for most of compiler-rt.

Do you automatically handle
copying the inputs to the remote target? How do you identify what the
inputs are?

Yes, the simple heuristic above seems to work here.

Do you copy over any llvm tools executed in the run lines?

I copy the command directly after %run wrapper, but not FileCheck/etc.
%run syncs it over to the device.

What
about their dependent libraries? etc.

Dynamically linked? Mostly we don’t have any, and for those tests where we do,
some rpath substitution magic is required.

Maybe if you could give an example of a typical test you intend to support
and how it would work, a lot of these questions would be answered. If this
turns out to be something that could be reused for our use case, I would be
willing to chip in to help make it happen.

Basically, any of libFuzzer tests with %run prepended, e.g. https://reviews.llvm.org/D45235

George

Thank you for the explanation George. I have too came to believe that a
general all-encompassing solution would be incredibly complicated. Our
(lldb) use case is in many ways similar to compiler-rt (we want to test
code which was not compiled for the host), but then it also has it's own
"thing" (we have *two* processes/binaries, one debugging another,
theoretically they don't even have to have the same bitness).

I am glad you were able to make your use case work. I'll probably try
creating something a bit custom for lldb, when I actually get around to it.

regards,
pavel

Hi Pavel,

I was thinking about a generic proposal for a while,
but then I’ve realized that a much simpler heuristic suffices for all of

my needs:

   - Prepend all commands which should execute on the device with %run,

which runs a wrapper python script

   - Inside that script use a simple regexp to find all paths
   - sync all paths to the device
   - run the executable on the device
   - sync all paths form the device back to host

>
> Hello George,
>
> I would be very much interested in a (generic) solution that would allow
> running tests on a remote host/device. My intended use case for this is
> lldb+android, and I am mostly interested in googletest-style tests.

I’m not sure about those, all my problems were in LIT tests.

> In
> fact, I proposed something vaguely similar to this (but limited to

gtest)

> last year <http://lists.llvm.org/pipermail/llvm-dev/2017-May/113370.html
.
> However, due to some internal reshuffles, this idea never got past the
> prototype stage. Nonetheless, I would still like do something like this
> eventually (even better if someone else does it for me :slight_smile: ).
>
> I would love to hear more about your proposal, as my head is currently
> filled with questions about how this would work. (I admit I don't know
> anything about how libFuzzer is tested.) It's not clear to me whether

you

> intend to implement a general purpose remote test execution framework,

or a

> solution tailored to the needs of libFuzzer.

Seemed generic enough for most of compiler-rt.

> Do you automatically handle
> copying the inputs to the remote target? How do you identify what the
> inputs are?

Yes, the simple heuristic above seems to work here.

> Do you copy over any llvm tools executed in the run lines?

I copy the command directly after %run wrapper, but not FileCheck/etc.
%run syncs it over to the device.

> What
> about their dependent libraries? etc.

Dynamically linked? Mostly we don’t have any, and for those tests where

we do,

some rpath substitution magic is required.

> Maybe if you could give an example of a typical test you intend to

support

> and how it would work, a lot of these questions would be answered. If

this

> turns out to be something that could be reused for our use case, I

would be

> willing to chip in to help make it happen.

Basically, any of libFuzzer tests with %run prepended, e.g.

https://reviews.llvm.org/D45235

George

>
> regards,
> pavel
>
>
>> Hi,
>
>> Tests in compiler-rt run under both iOS simulators and Android devices,
> which requires “wrapping” the run line with the “adb” command for

Android,

> and adding a prefix / setting some environment variables for simulators/
>> In case of Android this is done by changing the compilation command to
> produce a wrapper which already calls “adb”, and in case of iOS

simulators

> by explicitly adding the “%run” command in front of run lines.
>> The solution is far from perfect though: many tests (e.g. libFuzzer)
> extensively rely on using arbitrary shell commands to create and

manipulate

> temporary text files.
>> Performing those commands in a different environment is done in a very
> ad-hoc manner: from what I see, for sanitizer tests those temporary

files

> are then usually references in the environment variables present in the

run

> command, which are then parsed to carry those files into the executable
> environment.
>> Doing this for libFuzzer would be very difficult and would require a

lot

> of modifications to test.
>
>> I would prefer a more transparent approach: if LIT configuration was to
> provide a hook to perform arbitrary replacements/wrapping on each of the
> executed lines, the wrapping could be performed without modifying tests

at

> all.
>
>> My implementation preference would be adding a method `def wrap(self,
> s)` (str -> str) to the format class (lit/formats/*), and then using it

(if