Daniel Dunbar wrote:
I am going to change the LLVM test-suite RunSafely.sh to only track
'user' time, instead of reporting 'user' + 'sys' time as it currently
does. This will probably cause a spike in nightly test numbers,
although hopefully it will be limited to the smaller tests.
The eventual goal is to report all numbers (independently), so that we
can control for noise better. However, until that happens it is better
to track the most stable & interesting number.
First, you should be aware that the test-suite infrastructure is used by the Automatic Pool Allocation and SAFECode projects for their testing infrastructure. It is also used by our internal research projects and may be used by other research projects at other universities. We use this infrastructure for our research, so changes you make can affect us.
Second, why are you only interested in user time? The reason why we had RunSafely.sh measure user + system time is that it gives a more accurate depiction of how well an optimization works. If a program spends most of its time in the OS, increasing speed in user-space doesn't gain us much. If a transform decreases user time but increases system time, then measuring only user time may show a speedup when measuring user+system will show a loss.
If noise is your concern, I think it would make sense to run the tests several times and report averages and standard deviations. System time should only be reporting OS CPU time, so delays due to interrupts, I/O, etc. should not affect the results. User+System reported for several runs should provide the most accurate evaluation of how well an optimization is improving performance.
Third, IIRC, I think you can change the nightly testers to report just user time by changing how they grab the time measurements in TEST.nightly.Makefile. I think the .time files created by the test suite record user, system, and user+system, and the testing Makefiles just grab the number that they want.
Fourth, if you are intent on changing RunSafely.sh to measure just user time, why not add a feature that toggles whether to measure user+system or user time and *then* change the default behavior to measure user time? That way, I can easily toggle it back. What you've suggested above (more or less) is to remove a feature, add a feature, and then add back the feature you removed. It makes more sense to me to add the one feature you want and then to change the default behavior to use the new feature instead of the old one.
-- John T.