Hi everyone, I’ve been experimenting with a prototype for the daemonized test running idea which can be found here.
So far, I’ve just been applying the idea to opt and FileCheck and comparing performance in the llvm/Transforms folder. Lit collects the stdout and stderr from the daemon, as well as the return code, and displays them as normal. If the daemon exits unexpectedly during a test, Lit takes the exit code as the exit code for the test and restarts the daemon, so crashing is taken care of. To try the daemonized testing, invoke Lit with --use-daemon-tools.
On Windows, running all the llvm/Transforms tests with regular Lit takes 137 seconds on my PC. With my prototype for running opt and FileCheck as daemons, it takes just 32 seconds, although 0.97% of tests in the Transforms directory fail incorrectly.
The biggest problem for the approach is static state in LLVM. Static variables are not reset between daemon tasks, so any static variable that may affect the output can break tests - for example, NumDevirtCalls in WholeProgramDevirt.cpp wasn’t being reset between runs, causing the Transforms/WholeProgramDevirt/import.ll test to fail unexpectedly - I think the causes for the remaining failures are similar. The daemonization approach could only be applied universally if all state which is shared between modules and may change the output is removed and forbidden. Another idea I had is, for tests that fail when run using the daemon, to try re-running them without it - this would cover cases where the daemonization causes incorrect failures, but doesn’t solve the issue pointed out by jh7370 that shared state could cause tests to pass incorrectly.
The daemonization is transparent from the perspective of the Lit test, although it may be sensible to make a way for a test to opt out of running tools in daemon mode in case the test relies on state that can’t be properly reset, or even to make it opt-in on a per-invocation basis (e.g. RUN: %opt_daemon rather than RUN: opt).