QMTest vs. Dejagnu

I've finished adding the -rundejagnu option to the nightly tester script,
which was the last step to fully support Dejagnu. I think now is the
appropriate time to discuss keeping QMTest or switching to Dejagnu. A lot
of work went into using QMTest, so I think we should make this decision
carefully and before the 1.4 release.

Here are the pros and cons in my eyes, please feel free to add your own
opinion.

(Note: RUN lines are basically sequences of commands that are put into a
bash script and executed. Here is an example: RUN: llvm-as < %s | llvm-dis

llvm-as)

Pros of Dejagnu:
  1) Adding a test is very quick and easy. In most cases it requires
  just adding a RUN line to the file. No gui needed.
  2) XFAILing a test is also quick and easy. You just add a XFAIL
  line to a single test with a * for all platforms, or specify part
  of a target triplet to match. No gui needed.
  3) Because you add the RUN and XFAIL lines to each individual
  test, it's all located in one file. This makes it very easy
  understand how the test is run and easy to change.
  4) You can run just a single sub directory of tests by specifying
  that directory as the TESTSUITE. (ex: make check-dejagnu
  TESTSUITE=Regression/CFrontend)
  5) All intermediate output is put into an Output directory and
  easily cleaned up w/ proper Makefile support.
  6) It is faster then QMTest (54% faster on the sparc, 20% faster
  on ppc)
  7) The output logs are easy to parse.
  8) You can easily search for XFAIL tests.

Cons of Dejagnu:
  1) There must be a dg.exp file in each subdirectory to run tests
  in that directory. (easily copied though)

Pros of QMTest:
  1) Can add names to subdirectories to create testsuites (kinda
  like a shortcut).
  2) Has a gui (some prefer this).

Cons of QMTest:
  1) You have to use the gui to add directories.
  2) You have to use the gui to XFAIL a test.
  3) It uses something called expectation files that you must load
  to view which tests XFAIL. There is no way (that I have found) to
  get a complete list of XFAILs..
  4) It is also hard to XFAIL across platforms, because it requires
  hacking an expectation file for each
target, which must be done with the gui.
  5) Intermediate output placement can not be controlled.
  6) The output logs are not as clean.
  7) Right now we are dependent on a specific version of QMTest.
  8) It has several bugs that can cause it to forkbomb when tools
  are not built correctly.

I propose that we switch over to using dejagnu by default, renaming
check-dejagnu to check, and deprecate QMTest. We can either remove qmtest
for this release or keep it until 1.5.

I'd appreciate your opinion or any feedback you may have.

Thanks!
-Tanya

Seems pretty clear to me that dejagnu is a better choice. There's one
more QMTest issue: we only work with 2.03 as our database doesn't read
properly with newer versions. Not having a database seems like a good
way to go and having all the necessary things being easily editable in
text files will help keep the tests up to date and make XFAILing much
easier.

I vote for dumping QMTest in 1.4 or at least deprecating it in 1.4 and
removing it in 1.5. In any event the "check" target in the test
directory should now default to dejagnu.

Reid.

dejagnu sounds like the better choice to me too. I recommend keeping qmtest for this release and then removing it for 1.5.

--Vikram

Just some comments from a QMTest user... Note however, that even with them,
dejagnu looks better.

Cons of QMTest:
1) You have to use the gui to add directories.

I think you're wrong. Manually creating a directory would work, as QMTest does
not place anything special in directories.

2) You have to use the gui to XFAIL a test.

Right.

3) It uses something called expectation files that you must load
to view which tests XFAIL. There is no way (that I have found) to
get a complete list of XFAILs..

Right. This appears to have deep roots in QMTest design, so I don't know how
to make using one file with "RUN" and "XFAIL" lines work -- the allowed
format of expectations file is hardcoded.

4) It is also hard to XFAIL across platforms, because it requires
hacking an expectation file for each
target, which must be done with the gui.

Right and very deplorable. Basically, QMTest is nice tool but it's not clear
how to use it for for cross-platform work like LLVM (or C++ Boost).

5) Intermediate output placement can not be controlled.

I don't understand this one. The intermediate output is placed where tests
want it to be placed; QMTest does not create any such output itself.

6) The output logs are not as clean.

I think the current version supports dejagnu-style logs.

7) Right now we are dependent on a specific version of QMTest.

Yea, that's a big problem.

I propose that we switch over to using dejagnu by default, renaming
check-dejagnu to check, and deprecate QMTest. We can either remove qmtest
for this release or keep it until 1.5.

I'd appreciate your opinion or any feedback you may have.

Looks like dejagnu is better for this task (though I always thought it's some
rusty tcl-based code). You might want to check on QMTest mailing list first,
in case they have some silver bullets at hand. After all QMTest was supposed
to test gcc, which is also cross-platform.

- Volodya

Tanya Lattner wrote:

I've finished adding the -rundejagnu option to the nightly tester script,
which was the last step to fully support Dejagnu. I think now is the
appropriate time to discuss keeping QMTest or switching to Dejagnu. A lot
of work went into using QMTest, so I think we should make this decision
carefully and before the 1.4 release.

Here are the pros and cons in my eyes, please feel free to add your own
opinion.

(Note: RUN lines are basically sequences of commands that are put into a
bash script and executed. Here is an example: RUN: llvm-as < %s | llvm-dis
> llvm-as)

Pros of Dejagnu:
  1) Adding a test is very quick and easy. In most cases it requires
  just adding a RUN line to the file. No gui needed.
  2) XFAILing a test is also quick and easy. You just add a XFAIL
  line to a single test with a * for all platforms, or specify part
  of a target triplet to match. No gui needed.
  3) Because you add the RUN and XFAIL lines to each individual
  test, it's all located in one file. This makes it very easy
  understand how the test is run and easy to change.
  4) You can run just a single sub directory of tests by specifying
  that directory as the TESTSUITE. (ex: make check-dejagnu
  TESTSUITE=Regression/CFrontend)
  5) All intermediate output is put into an Output directory and
  easily cleaned up w/ proper Makefile support.
  6) It is faster then QMTest (54% faster on the sparc, 20% faster
  on ppc)
  7) The output logs are easy to parse.
  8) You can easily search for XFAIL tests.

Cons of Dejagnu:
  1) There must be a dg.exp file in each subdirectory to run tests
  in that directory. (easily copied though)

Can these be located in the source tree and found when running the tests in the object tree? If so, that would save us from having to copy these over into the object tree via the configure script, which would be very nice.

What other packages does DejaGnu require, and are most people likely to have those packages installed?

Pros of QMTest:
  1) Can add names to subdirectories to create testsuites (kinda
  like a shortcut).
  2) Has a gui (some prefer this).

I agree that having a command line tool or being able to edit information inside the test file would be ideal. The GUI is painful.

Cons of QMTest:
  1) You have to use the gui to add directories.
  2) You have to use the gui to XFAIL a test.
  3) It uses something called expectation files that you must load
  to view which tests XFAIL. There is no way (that I have found) to
  get a complete list of XFAILs..
  4) It is also hard to XFAIL across platforms, because it requires
  hacking an expectation file for each
target, which must be done with the gui.

QMTest could use the same expectations file for all platforms; I eventually chose not to do that because I need some tests to pass on some platforms and fail on others.

The problem, I think, is being able to XFAIL across platforms and still be easily able to XFAIL/XPASS tests for specific platforms (i.e. test1 fails on all platforms, but test2 fails on Sparc only).

  5) Intermediate output placement can not be controlled.

Actually, this is not a QMTest issue as much as it is the test code that I wrote. The testing code I wrote (qmtest.py) puts all temporary files into a temporary directory in the object tree. I think I did this because QMTest runs all tests from the $OBJ_ROOT/test directory instead of recursively cd'ing into all of the test directories like the old Makefile system did.

  6) The output logs are not as clean.

FWIW, QMTest can generate an HTML output file.

  7) Right now we are dependent on a specific version of QMTest.

This is due to the fact that QMTest changed their coding interfaces in newer versions,so our QMTest python code no longer works.

  8) It has several bugs that can cause it to forkbomb when tools
  are not built correctly.

I don't know if this is a coding error on my part or a bug in whatever Python library I'm using to execute the program. In any event, the current implementation should be fixable; it just hasn't gotten high priority on my list yet.

I propose that we switch over to using dejagnu by default, renaming
check-dejagnu to check, and deprecate QMTest. We can either remove qmtest
for this release or keep it until 1.5.

Despite my feedback on QMTest, I am getting the impression that Dejagnu is the easier tool to use (although I don't know for sure as I've not used it). Using QMTest is only good if we upgrade to a newer version, and even then, it may not get us the features we want (like not using the GUI, easy XFAIL'ing, etc). Since Dejagnu is already in the tree, I don't see much point in upgrading to a newer version of QMTest.

The only real downside to Dejagnu might be external software dependencies.

Regarding the upgrade path, it would be nice to have only one test system in place for LLVM 1.4. However, if there isn't enough time, I think we should just stick with QMTest and then remove it for LLVM 1.5.

I'd appreciate your opinion or any feedback you may have.

Thanks!
-Tanya

_______________________________________________
LLVM Developers mailing list
LLVMdev@cs.uiuc.edu http://llvm.cs.uiuc.edu
http://mail.cs.uiuc.edu/mailman/listinfo/llvmdev

-- John T.

> Cons of QMTest:
> 1) You have to use the gui to add directories.

I think you're wrong. Manually creating a directory would work, as QMTest does
not place anything special in directories.

I may have worded this poorly, but I think you have to use the gui to add
new directories or tests, or specific tests. Otherwise, it does not know
what to do with those files. I get lots of warnings/errors from QMTest
when it comes across a tests thats not in the database (says something
like "could not load test".

> 5) Intermediate output placement can not be controlled.

I don't understand this one. The intermediate output is placed where tests
want it to be placed; QMTest does not create any such output itself.

This could be how we have written our python files.. but right now if you
run a tool and it generates some output.. where that output is placed is
not controlled. So it makes it difficult to clean up this intermediate
output.

> I propose that we switch over to using dejagnu by default, renaming
> check-dejagnu to check, and deprecate QMTest. We can either remove qmtest
> for this release or keep it until 1.5.
>
> I'd appreciate your opinion or any feedback you may have.

Looks like dejagnu is better for this task (though I always thought it's some
rusty tcl-based code). You might want to check on QMTest mailing list first,
in case they have some silver bullets at hand. After all QMTest was supposed
to test gcc, which is also cross-platform.

Thanks for your feedback!!!

-Tanya

Tanya Lattner wrote:

Cons of QMTest:
1) You have to use the gui to add directories.

I think you're wrong. Manually creating a directory would work, as QMTest does
not place anything special in directories.

I may have worded this poorly, but I think you have to use the gui to add
new directories or tests, or specific tests. Otherwise, it does not know
what to do with those files. I get lots of warnings/errors from QMTest
when it comes across a tests thats not in the database (says something
like "could not load test".

You shouldn't need to add tests using the GUI; the source in llvmdb.py should pick it up automatically (llvmdb.py takes all of the test input files and presents it to QMTest as a test database).

However, if you add a directory (say, llvm/test/NewComponent), then llvmdb.py needs to be modified so that it knows that the directory is part of the test suite and knows which tests to run on the files inside that new directory.

Currently, all of this stuff is hardcoded into llvm/test/QMTest/llvmdb.py, but it could be made configurable (assuming we wanted to keep QMTest).

5) Intermediate output placement can not be controlled.

I don't understand this one. The intermediate output is placed where tests
want it to be placed; QMTest does not create any such output itself.

This could be how we have written our python files.. but right now if you
run a tool and it generates some output.. where that output is placed is
not controlled. So it makes it difficult to clean up this intermediate
output.

Since, with QMTest, all tests are run from $OBJROOT/test, we don't want that control to reside with the tests (they'll place their temporary files in the wrong place). Hence the addition of the %t replacement string in the Pyton version of TestRunner (%t provides the pathname of the temporary directory to use for files).

I propose that we switch over to using dejagnu by default, renaming
check-dejagnu to check, and deprecate QMTest. We can either remove qmtest
for this release or keep it until 1.5.

I'd appreciate your opinion or any feedback you may have.

Looks like dejagnu is better for this task (though I always thought it's some
rusty tcl-based code). You might want to check on QMTest mailing list first,
in case they have some silver bullets at hand. After all QMTest was supposed
to test gcc, which is also cross-platform.

Thanks for your feedback!!!

-Tanya

_______________________________________________
LLVM Developers mailing list
LLVMdev@cs.uiuc.edu http://llvm.cs.uiuc.edu
http://mail.cs.uiuc.edu/mailman/listinfo/llvmdev

QMTest Quality of Implementation can be improved, but if Dejagnu already exists and works well enough, there is little reason to keep QMTest, I think.

-- John T.

> Cons of Dejagnu:
> 1) There must be a dg.exp file in each subdirectory to run tests
> in that directory. (easily copied though)

Can these be located in the source tree and found when running the tests
in the object tree? If so, that would save us from having to copy these
over into the object tree via the configure script, which would be very
nice.

They do not have to be copied over. I run objdir != srcdir and dejagnu
works fine. The trickey part is that you can not specify a particular
testsuite if you have not run the complete suite of tests (only applies to
when objdir != srcdir). This is because dejagnu wants to place the sum
file in that directory which won't exist until you run the complete suite.
There is a bugzilla bug on this, but its really not that big of a deal to
run the Regression/Features tests once (in my opinion).

What other packages does DejaGnu require, and are most people likely to
have those packages installed?

tcl (which most people will have installed) and expect are the only
requirements to install Dejagnu.

> Cons of QMTest:
> 1) You have to use the gui to add directories.
> 2) You have to use the gui to XFAIL a test.
> 3) It uses something called expectation files that you must load
> to view which tests XFAIL. There is no way (that I have found) to
> get a complete list of XFAILs..
> 4) It is also hard to XFAIL across platforms, because it requires
> hacking an expectation file for each
> target, which must be done with the gui.

QMTest could use the same expectations file for all platforms; I
eventually chose not to do that because I need some tests to pass on
some platforms and fail on others.

The problem, I think, is being able to XFAIL across platforms and still
be easily able to XFAIL/XPASS tests for specific platforms (i.e. test1
fails on all platforms, but test2 fails on Sparc only).

> 5) Intermediate output placement can not be controlled.

Actually, this is not a QMTest issue as much as it is the test code that
I wrote. The testing code I wrote (qmtest.py) puts all temporary files
into a temporary directory in the object tree. I think I did this
because QMTest runs all tests from the $OBJ_ROOT/test directory instead
of recursively cd'ing into all of the test directories like the old
Makefile system did.

There are some tests that generate some intermediate output that are not
getting placed in the temp directory. I'm not sure why that is.. but it
must be something that just wasn't noticed til now.

> 6) The output logs are not as clean.

FWIW, QMTest can generate an HTML output file.

That is a good point.

> I propose that we switch over to using dejagnu by default, renaming
> check-dejagnu to check, and deprecate QMTest. We can either remove qmtest
> for this release or keep it until 1.5.

Despite my feedback on QMTest, I am getting the impression that Dejagnu
is the easier tool to use (although I don't know for sure as I've not
used it). Using QMTest is only good if we upgrade to a newer version,
and even then, it may not get us the features we want (like not using
the GUI, easy XFAIL'ing, etc). Since Dejagnu is already in the tree, I
don't see much point in upgrading to a newer version of QMTest.

The only real downside to Dejagnu might be external software dependencies.

I actually think that more people will have tcl installed then python..
but thats just my personal opinion.

Regarding the upgrade path, it would be nice to have only one test
system in place for LLVM 1.4. However, if there isn't enough time, I
think we should just stick with QMTest and then remove it for LLVM 1.5.

The only downside I see to keeping it in for 1.4 is that you (since you
are the one that updates QMTest) will have to update the expectation files
for the tests (there have been several new ones added). But otherwise, its
easy to keep it in the tree.

-Tanya