RFC: Dropping DejaGNU

Hi all,

As you may or may not have noticed, I have gradually been replacing
the testing infrastructure in LLVM. Most of this work is moving LLVM
and Clang to share a single testing tool, 'lit'. See:
  http://llvm.org/cmds/lit.html
for documentation on the tool itself.

If you haven't already tried it, please consider switch to 'make
check-lit' as an alternative to 'make check'. If it doesn't work for
you, or you find it doesn't do something DejaGNU did and you like,
please let me know. My eventual plan is to move to lit entirely and
drop DejaGNU support, so consider yourself warned.

A couple key points about 'make check-lit':
1. It runs the unittests as well, there is no separate 'make
unittests' step. If you like, there is also a 'make check-all' which
will also run the Clang tests at the same time if it happens to be
checked out in the standard location (tools/clang).
2. The public buildbots have switched to it. It shouldn't happen, but
if you do find a discrepancy between DejaGNU and lit, let me know.

If you aren't already familiar with it, here are some of the advantages of lit:
1. It is portable. It already is being used on the Windows buildbot
for Clang, and almost all of our tests "just work".
2. It uses multiple threads to test. While it doesn't yet scale quite
as well as I would like, switching the Linux x86_64 buildbot to it
dropped its cycle time by over 30% (just because running the tests got
about 3x faster).
3. I've tried hard to make it easy to use, and will keep working in
this direction. Eventually I want to replace the TestRunner scripts as
well so that all that is needed to run a single test is 'lit
path/to/the/test'. I also would like to improve the output for
failures to be more informative (particularly when run via buildbot).

See http://llvm.org/PR5217 for my TODO list before dropping DejaGNU,
feel free to add comments, requests, or blocking bugs to that one.

- Daniel

Daniel Dunbar <daniel@zuster.org> writes:

[snip]

1. It is portable. It already is being used on the Windows buildbot
for Clang, and almost all of our tests "just work".

Nice.

Thanks Daniel!

Hi all,

As you may or may not have noticed, I have gradually been replacing
the testing infrastructure in LLVM. Most of this work is moving LLVM
and Clang to share a single testing tool, 'lit'. See:
http://llvm.org/cmds/lit.html
for documentation on the tool itself.

If you haven't already tried it, please consider switch to 'make
check-lit' as an alternative to 'make check'. If it doesn't work for
you, or you find it doesn't do something DejaGNU did and you like,
please let me know. My eventual plan is to move to lit entirely and
drop DejaGNU support, so consider yourself warned.

This is great work Daniel,

One question: Why not switch 'make check' to default to lit and offer 'make check-dejagnu' as a workaround. That forces adoption :slight_smile:

-Chris

Hi all,

As you may or may not have noticed, I have gradually been replacing
the testing infrastructure in LLVM. Most of this work is moving LLVM
and Clang to share a single testing tool, 'lit'. See:
http://llvm.org/cmds/lit.html
for documentation on the tool itself.

If you haven't already tried it, please consider switch to 'make
check-lit' as an alternative to 'make check'. If it doesn't work for
you, or you find it doesn't do something DejaGNU did and you like,
please let me know. My eventual plan is to move to lit entirely and
drop DejaGNU support, so consider yourself warned.

This is great work Daniel,

One question: Why not switch 'make check' to default to lit and offer 'make
check-dejagnu' as a workaround. That forces adoption :slight_smile:

I intend to as soon as it won't break anything. The sole blocker for
doing that is making sure it doesn't break the nightly testing
infrastructure.

- Daniel

I've swapped all of my testing to lit since all I do is native and it's been great. I'm definitely in favor of Chris's suggestion that we move to that as the default.

One quick feature question: one of the nifty things about dejagnu (really the only nifty thing) is that it supported remote execution of binaries - I don't use that right now, but other people may. Any idea if you plan on having lit support that? Honestly I doubt it would be hard in python. Probably a "if someone has a need" feature? :slight_smile:

-eric

If you haven't already tried it, please consider switch to 'make
check-lit' as an alternative to 'make check'. If it doesn't work for
you, or you find it doesn't do something DejaGNU did and you like,
please let me know. My eventual plan is to move to lit entirely and
drop DejaGNU support, so consider yourself warned.

I've swapped all of my testing to lit since all I do is native and it's been
great. I'm definitely in favor of Chris's suggestion that we move to that
as the default.

Yay!

One quick feature question: one of the nifty things about dejagnu (really
the only nifty thing) is that it supported remote execution of binaries - I
don't use that right now, but other people may. Any idea if you plan on
having lit support that? Honestly I doubt it would be hard in python.
Probably a "if someone has a need" feature? :slight_smile:

Should be possible, although I don't know anything about the dejagnu
feature. However, 'lit' has a reasonably complete shell and Tcl parser
so it knows what commands are being called, so at least in theory it
should be able to rewrite them.

- Daniel

Should be possible, although I don't know anything about the dejagnu
feature. However, 'lit' has a reasonably complete shell and Tcl parser
so it knows what commands are being called, so at least in theory it
should be able to rewrite them.

Interesting. Even less of that would be just duplicating the functionality. At any rate everything looks quite nice.

Thanks!

-eric

FWIW, this is a feature we're currently using and caused me to make
numerous changes to the makefiles a while back. If this is entirely
going away, then bringing up a buildbot with a Beagleboard attached is
going to be a much bigger task than I hoped.

deep

Should be possible, although I don't know anything about the dejagnu
feature. However, 'lit' has a reasonably complete shell and Tcl parser
so it knows what commands are being called, so at least in theory it
should be able to rewrite them.

Interesting. Even less of that would be just duplicating the
functionality. At any rate everything looks quite nice.

Why not abstracting the "copy to target," "run on target," "get results from target," etc. commands? For the default (hosted) case, they're pretty much trivial. For remote targets, they can be defined in terms of RSH, SSH, or whatever. As I recall, dejagnu does something along those lines.

Pretty much what I was thinking, yes.

Anyhow, not a major issue on my end at the moment :slight_smile:

-eric

Daniel,

This is really nice :slight_smile:

Is there a way to run only clang test ?

-- Jean-Daniel

I normally just cd tools/clang and then make test.

Thanks,

It tried 'make check-lit' instead of 'make test', that why I don't saw the expected result.

-- Jean-Daniel