[RFC] Dexter Feature Removals

tl;dr: In this RFC, we (Sony) propose removing the builder and the test discovery mechanism from Dexter. If anyone is currently using these features or has a case to make for their continued support, let us know in a response to this RFC.

Dexter is one of the tools used for LLVM’s debuginfo-tests (within the cross-project-tests project). For some time, it was not running on any buildbots due to errors in Dexter and its associated tests[0] that slipped in due to configuration errors preventing the tests from being run/failures from being detected. Dexter is currently being used internally at Sony, and may be in use by individuals downstream, but has only recently started being tested by buildbots (a pair of SIE buildbots for linux[1] and windows[2]), and the test suite still contains a significant number of XFAILs. These XFAILs are currently being fixed along with a few other errors in the cross-project-tests project.

Beyond just fixing the existing errors, we are also looking at reducing the number of potential failure points in Dexter. To this end we intend to remove two largely redundant features in Dexter: the builder, which is used to build a program for testing, and test discovery, which recursively searches a folder for Dexter tests (as opposed to running tests individually). Both of these features were implemented when Dexter was a standalone tool, and were necessary for any kind of test suite to be created. Since Dexter has merged into the llvm project proper however, these features don’t provide any value to the project.

The builder scripts are used in the Dexter tests in cross-project-tests, but could trivially be replaced by directly invoking clang as part of the lit test without losing any functionality. At Sony we are not using the builder scripts for our internal tests, and at least one of the build scripts (clang_vs2015) doesn’t work at all. Our intention is to remove the build feature entirely for the sake of maintenance; if anyone downstream is using the builder (either the existing scripts or their own custom script) it would be useful to know about.

The test discovery feature has been made almost entirely obsolete by llvm-lit. Using test discovery within cross-project-tests would be actively counter-productive compared to using lit, as it would fold the test results for several different Dexter tests into a single lit result. Unlike the builder, which is a distinct optional step, the discovery code is tangled up with other code in Dexter, which puts an additional burden on maintaining and developing it. It seems unlikely that anybody is depending on this feature given the existence of lit and the fact that test discovery is fairly simple to perform with a script, but as with the builder it would be useful to know about any such cases.

Removing these features is not urgently required, so any objections or comments are welcome, but otherwise these features appear to be unnecessary code bloat that don’t provide value while potentially hampering further development and adding a maintenance burden.

[0] Discussed on the mailing list: [llvm-dev] dexter failures in cross-project-tests
[1] https://lab.llvm.org/buildbot/#/builders/217
[2] Buildbot