RE: https://reviews.llvm.org/D65311
Internally we use dotest to run the lldb test suite with various RTOS configurations for the test binaries. In these runs we don’t care about the lit tests or the unit tests, because they are OS agnostic. We do this by specifying the compiler, lldb, and test flavor (static testcase + os, dynamic library testcase loaded by an OS image, dynamic library testcase loaded by an OS image running the OS’ debug stub).
With the multiprocess testrunner removed, how do I have lit:
Hey Ted,
1. You can run just the dotest-tests by pointing lit at the
`lit/Suite` directory.
2. You can pass arguments to dotest by passing `dotest-args` in --param.
The invocation would look something like this:
/path/to/llvm/bin/llvm-lit /path/to/lldb/lit/Suite --param
'dotest-args=--foo --bar'
Hope that helps,
Jonas
Is this in the on-line docs somewhere?
Jim
Thanks Jonas.
Is full support for --param fairly recent? I tried it with a version of our master, based on top-of-tree from about a month ago, and it didn't work quite right. It's passing the dotest args, but it's also generating some args, so I'm seeing odd effects.
Here is my run line:
bin/python bin/llvm-lit /local/mnt/ted/8.4/llvm/lldb/lit/Suite --param 'dotest-args=-A v66 -C /prj/dsp/qdsp6/release/internal/HEXAGON/branch-8.4/linux64/latest/Tools/bin/hexagon-clang --executable /local/scratch/ted/8.4/build/bin/lldb -t -v -f RecursiveTypesTestCase.test_recursive_type_1_dwarf'
I only want to run RecursiveTyepsTestCase.test_recursive_type_1_dwarf, but it's running the whole test suite. Here's a dotest line from the run:
/local/mnt/ted/8.4/build/bin/python /local/mnt/ted/8.4/llvm/lldb/test/dotest.py -q --arch=v66 -s /local/mnt/ted/8.4/build/lldb-test-traces --build-dir /local/mnt/ted/8.4/build/lldb-test-build.noindex -S nm -u CXXFLAGS -u CFLAGS --executable /local/mnt/ted/8.4/build/./bin/lldb --dsymutil /local/mnt/ted/8.4/build/./bin/dsymutil --filecheck /local/mnt/ted/8.4/build/./bin/FileCheck -C /local/mnt/ted/8.4/build/./bin/clang --env ARCHIVER=/usr/bin/ar --env OBJCOPY=/usr/bin/objcopy -A v66 -C /prj/dsp/qdsp6/release/internal/HEXAGON/branch-8.4/linux64/latest/Tools/bin/hexagon-clang --executable /local/scratch/ted/8.4/build/bin/lldb -t -v -f RecursiveTypesTestCase.test_recursive_type_1_dwarf --env LLVM_LIBS_DIR=/local/mnt/ted/8.4/build/./lib /local/mnt/ted/8.4/llvm/lldb/packages/Python/lldbsuite/test/functionalities/breakpoint/debugbreak -p TestDebugBreak.py
It's got both --arch= and -A, -C is set to my build directory clang as well as the clang I told it to use, --executable is set twice, and it's got -f RecursiveTypesTestCase.test_recursive_type_1_dwarf and -p TestDebugBreak.py .
These tests that do a "process launch" (which is most of them) invoke the hexagon simulator, but it was never launched. There was also only 1 testcase built in /local/mnt/ted/8.4/build/lldb-test-build.noindex - types/TestRecursiveTypes.test_recursive_type_1_dwarf .
This does not have the patch that removes multiprocess support from dotest.
Ted
Hey Ted,
Thanks Jonas.
Is full support for --param fairly recent? I tried it with a version of our master, based on top-of-tree from about a month ago, and it didn't work quite right. It's passing the dotest args, but it's also generating some args, so I'm seeing odd effects.
It's not something I touched recently, but it's always possible they
made some changes in LLVM.
Here is my run line:
bin/python bin/llvm-lit /local/mnt/ted/8.4/llvm/lldb/lit/Suite --param 'dotest-args=-A v66 -C /prj/dsp/qdsp6/release/internal/HEXAGON/branch-8.4/linux64/latest/Tools/bin/hexagon-clang --executable /local/scratch/ted/8.4/build/bin/lldb -t -v -f RecursiveTypesTestCase.test_recursive_type_1_dwarf'
I only want to run RecursiveTyepsTestCase.test_recursive_type_1_dwarf, but it's running the whole test suite.
Do you know about the lldb-dotest binary? You can still use it to
invoke a single test, similar to how lit does it. You should be able
to just pass your arguments to that.
Here's a dotest line from the run:
/local/mnt/ted/8.4/build/bin/python /local/mnt/ted/8.4/llvm/lldb/test/dotest.py -q --arch=v66 -s /local/mnt/ted/8.4/build/lldb-test-traces --build-dir /local/mnt/ted/8.4/build/lldb-test-build.noindex -S nm -u CXXFLAGS -u CFLAGS --executable /local/mnt/ted/8.4/build/./bin/lldb --dsymutil /local/mnt/ted/8.4/build/./bin/dsymutil --filecheck /local/mnt/ted/8.4/build/./bin/FileCheck -C /local/mnt/ted/8.4/build/./bin/clang --env ARCHIVER=/usr/bin/ar --env OBJCOPY=/usr/bin/objcopy -A v66 -C /prj/dsp/qdsp6/release/internal/HEXAGON/branch-8.4/linux64/latest/Tools/bin/hexagon-clang --executable /local/scratch/ted/8.4/build/bin/lldb -t -v -f RecursiveTypesTestCase.test_recursive_type_1_dwarf --env LLVM_LIBS_DIR=/local/mnt/ted/8.4/build/./lib /local/mnt/ted/8.4/llvm/lldb/packages/Python/lldbsuite/test/functionalities/breakpoint/debugbreak -p TestDebugBreak.py
It's got both --arch= and -A, -C is set to my build directory clang as well as the clang I told it to use, --executable is set twice, and it's got -f RecursiveTypesTestCase.test_recursive_type_1_dwarf and -p TestDebugBreak.py .
Both lit and lldb-dotest are configured using the dotest arguments
that we can configure at CMake configuration time. That would explain
where the extra options come from. If those are not the ones you want,
you can still invoke dotest.py directly.
These tests that do a "process launch" (which is most of them) invoke the hexagon simulator, but it was never launched. There was also only 1 testcase built in /local/mnt/ted/8.4/build/lldb-test-build.noindex - types/TestRecursiveTypes.test_recursive_type_1_dwarf .
Other than the extra arguments I can't think of any reason why this
would behave differently. Lit is just a simple wrapper that invokes
dotest.py with the right arguments.
Hi Jonas,
What I need is a way to run the test suite with arbitrary command line arguments. Sometimes I want to run one or more tests with -f, sometimes I want to run one or more test files with -p, and sometimes I want to run the entire suite, either in parallel or 1 at a time (--no-multiprocess). I might be running from a directory where I've built lldb (but not clang, using clang from an arbitrary location), a directory where I've built everything (but in this case we set everything up with cmake), or a directory where I've checkout out just the sources and have copied the binaries from a distribution, so no cmake.
dotest used to handle these cases; how do I handle them now?
In our environment, lldb will launch and connect to the hexagon simulator, much like how it launches and connects to debugserver or lldb-server. But it has to use a specific version of clang, because we've found that if we have mismatches in clang/simulator/RTOS, bad things can happen. User code is in a shared library that gets loaded by a wrapper run under the RTOS. The RTOS, wrapper and user code all need to be built with the same complier and c/c++ libraries.
Ted
So far the only thing that changed by removing multiprocess is that
`--no-multiprocess` is always enabled. Everything else you describe is
still possible, and will continue to be possible.
Even when we remove the driver functionality from dotest.py this will
all continue to work. The only difference is that dotest.py will
operate on a single file after being invoked by some other driver. The
problem you describe with the dual arguments seems relatively easy to
fix by loading a different configuration in lit. It's not something
I've looked at yet, because everything we care about is configurable
from CMake or overridable by passing --params to lit.
Thanks for the pointers, Jonas.
I cloned a new repo, then ran cmake, setting up arbitrary test compiler and lldb locations. Then ran llvm-lit tools/lldb/lit/Suite , and the test suite ran as expected. There were a few failures, but I've got a plan now.
Ted