How to execute cross-compiled libcxx tests on a remote target?

Hi Louis,

I saw that you were the one who implemented the new testing system for libcxx, so I figured that you might be able to help me out with my problem.

I'm working on a downstream target and we're now in the process of adding support for libcxx. We would like to leverage the existing libcxx test suite to make sure that the implementation is working correctly. The problem is that our target is a bare-metal target and we need to cross-compile the tests and execute them remotely. There isn't really any documentation for how to do it, but I came across an old email thread from 2015 which said to implement an executor and use that. However, it seems like those executors are not used with the new test format any longer (except for a check whether to use `run.py` or `ssh.py`).

Also, in the original Phabricator review where you first introduced the new format, you said:

>As a side effect of this design, configuration files for the test suite can be as simple as:
>
>config.substitutions.append(('%{cxx}', '<path-to-compiler>'))
>config.substitutions.append(('%{compile_flags}', '<flags>'))
>config.substitutions.append(('%{link_flags}', '<flags>'))
>config.substitutions.append(('%{exec}', '<script-to-execute>'))
>
>This should allow storing lit.cfg files for various configurations directly in the repository

However I'm not sure what exactly I need to do here. Do I need to write my own `lit.site.cfg` and point CMake to it? If possible I'd like to reuse the default- and auto-generated config as much as possible.

We have a script which can execute a cross-compiled binary on our remote target. I just want the test suite to call this script. Could you give me some help on what I need to do to make this work?

Cheers,

Dominik

BCC: LLVM Dev

Hi,

Hi Louis,

thanks for the response and the commit to support this use case!

In the meantime I have implemented a feature to load an additional configuration file from the auto generated one, which solves the problem for me so far. Especially since I also need to set some additional flags (depending on the configuration) which get passed to my execution script, which seems like something which is not supported by your last commit?

By the way, if you're interested I'd be happy to create patch from my extension.

While we're at it, is there a way to declare some tests as unsupported? We have a couple which we do not support on our target and would like to exclude them so they don't show up as failures at all.

Cheers,

Dominik

Hi Louis,

thanks for the response and the commit to support this use case!

In the meantime I have implemented a feature to load an additional configuration file from the auto generated one, which solves the problem for me so far. Especially since I also need to set some additional flags (depending on the configuration) which get passed to my execution script, which seems like something which is not supported by your last commit?

Yes, you can pass e.g. -DLIBCXX_EXECUTOR="<your-remote-execution-script> --host <ip-address> --other-args...".

By the way, if you're interested I'd be happy to create patch from my extension.

Yes, I'd like you to send a Phab review with it so I can see what you've got. I also have a local patch that allows bypassing the default config file, and I was waiting to land other patches before I landed that one. Looking at yours might be useful.

While we're at it, is there a way to declare some tests as unsupported? We have a couple which we do not support on our target and would like to exclude them so they don't show up as failures at all.

You'd have to add `// UNSUPPORTED: <some-Lit-feature-defined-on-your-target>` to each unsupported test. Is that not an option? We'll be happy to take such a patch if your system is part of our test matrix, otherwise another option would be to add it to your downstream fork if you have one.

Cheers,
Louis

Hi Louis,

thanks for the response and the commit to support this use case!

In the meantime I have implemented a feature to load an additional configuration file from the auto generated one, which solves the problem for me so far. Especially since I also need to set some additional flags (depending on the configuration) which get passed to my execution script, which seems like something which is not supported by your last commit?

Yes, you can pass e.g. -DLIBCXX_EXECUTOR="<your-remote-execution-script> --host <ip-address> --other-args...".

Ah, yes that makes sense. Will have to try it for our use-case once I get some time to refactor the current setup :slight_smile:

By the way, if you're interested I'd be happy to create patch from my extension.

Yes, I'd like you to send a Phab review with it so I can see what you've got. I also have a local patch that allows bypassing the default config file, and I was waiting to land other patches before I landed that one. Looking at yours might be useful.

I opened [1] and added you as a reviewer. It's incredibly simple and doesn't do anything special whatsoever, but it solves all problem we had for our use-case. Let me know what you think!

While we're at it, is there a way to declare some tests as unsupported? We have a couple which we do not support on our target and would like to exclude them so they don't show up as failures at all.

You'd have to add `// UNSUPPORTED: <some-Lit-feature-defined-on-your-target>` to each unsupported test. Is that not an option? We'll be happy to take such a patch if your system is part of our test matrix, otherwise another option would be to add it to your downstream fork if you have one.

I need to check how many tests there are exactly that we consider unsupported for our downstream target. I have a feeling there might be too many to add this line to each one by hand, but in the meantime I found `lit.local.cfg` to at least exclude a larger set of tests.

Cheers,

Dominik

Hi Louis,

thanks for the response and the commit to support this use case!

In the meantime I have implemented a feature to load an additional configuration file from the auto generated one, which solves the problem for me so far. Especially since I also need to set some additional flags (depending on the configuration) which get passed to my execution script, which seems like something which is not supported by your last commit?

Yes, you can pass e.g. -DLIBCXX_EXECUTOR=" --host --other-args…".

Ah, yes that makes sense. Will have to try it for our use-case once I get some time to refactor the current setup :slight_smile:

By the way, if you’re interested I’d be happy to create patch from my extension.

Yes, I’d like you to send a Phab review with it so I can see what you’ve got. I also have a local patch that allows bypassing the default config file, and I was waiting to land other patches before I landed that one. Looking at yours might be useful.

I opened [1] and added you as a reviewer. It’s incredibly simple and doesn’t do anything special whatsoever, but it solves all problem we had for our use-case. Let me know what you think!

Thanks, let’s chat on https://reviews.llvm.org/D81824.

Louis