I've finished adding the -rundejagnu option to the nightly tester script,
which was the last step to fully support Dejagnu. I think now is the
appropriate time to discuss keeping QMTest or switching to Dejagnu. A lot
of work went into using QMTest, so I think we should make this decision
carefully and before the 1.4 release.
Here are the pros and cons in my eyes, please feel free to add your own
(Note: RUN lines are basically sequences of commands that are put into a
bash script and executed. Here is an example: RUN: llvm-as < %s | llvm-dis
Pros of Dejagnu:
1) Adding a test is very quick and easy. In most cases it requires
just adding a RUN line to the file. No gui needed.
2) XFAILing a test is also quick and easy. You just add a XFAIL
line to a single test with a * for all platforms, or specify part
of a target triplet to match. No gui needed.
3) Because you add the RUN and XFAIL lines to each individual
test, it's all located in one file. This makes it very easy
understand how the test is run and easy to change.
4) You can run just a single sub directory of tests by specifying
that directory as the TESTSUITE. (ex: make check-dejagnu
5) All intermediate output is put into an Output directory and
easily cleaned up w/ proper Makefile support.
6) It is faster then QMTest (54% faster on the sparc, 20% faster
7) The output logs are easy to parse.
8) You can easily search for XFAIL tests.
Cons of Dejagnu:
1) There must be a dg.exp file in each subdirectory to run tests
in that directory. (easily copied though)
Pros of QMTest:
1) Can add names to subdirectories to create testsuites (kinda
like a shortcut).
2) Has a gui (some prefer this).
Cons of QMTest:
1) You have to use the gui to add directories.
2) You have to use the gui to XFAIL a test.
3) It uses something called expectation files that you must load
to view which tests XFAIL. There is no way (that I have found) to
get a complete list of XFAILs..
4) It is also hard to XFAIL across platforms, because it requires
hacking an expectation file for each
target, which must be done with the gui.
5) Intermediate output placement can not be controlled.
6) The output logs are not as clean.
7) Right now we are dependent on a specific version of QMTest.
8) It has several bugs that can cause it to forkbomb when tools
are not built correctly.
I propose that we switch over to using dejagnu by default, renaming
check-dejagnu to check, and deprecate QMTest. We can either remove qmtest
for this release or keep it until 1.5.
I'd appreciate your opinion or any feedback you may have.