A feature I've wanted in lit for a while is a having a timeout per
are patches that implement this idea.
I'm e-mailing llvm-dev rather than llvm-commits
because I want to gather more feedback on my initial implementation and
hopefully some answers to some unresolved issues with my implementation.
Currently in lit you can set a global timeout for
all of the tests but not for each individual test.
The attached patches add
* Support for a new ResultCode called TIMEOUT
* A new command line option --max-individual-test-time
* Support for running external and internal ShTests with a per test timeout
* Support for running GTests with a per test timeout
I wanted to get some initial feedback on the implementation.
* If a timeout is requested the Python psutil module is required.
This module does not ship with Python but is availble via pip
(or on Linux your distribution's package manager). How do people feel
about this? I don't like adding extra dependencies but this module
makes it really easy to kill a process and all its children recursively
in a platform neutral way. Note that if a per test timeout is not requested
then the psutil module is not imported and lit acts just like it did
before my patches.
* If the platform running lit doesn't have psutil installed and a
timeout is requested
an exception will be thrown. Should we provide a more friendly error message?
If so where should this go in lit's code?
* I've not tested these patches on OSX or Windows. Is anyone on those
to give them a try?
* The behaviour of --max-individual-test-time is a little at odds with
the behaviour of --max-time. --max-time will mark unexecuted tests as
UNRESOLVED whereas --max-individual-test-time will mark a test that
ran out of time as TIMEOUT. Is this okay?
* @Chris Matthews. Does the code that emits xunit xml files need to
change in anyway?
* @Daniel Dunbar. If these changes (in some form) end up being
committed would you be happy
to push a new release of lit to PyPy?