Testing documentation and terminology

Project Infrastructure LLVM Dev List Archives
1

Hi all,

as you might have seen, I just did some restructuring on the testing
documentation. Since the changes are only documentation and mostly text
movement, I took the liberty of committing without posting a patch first. Was
that appropriate?.

The incentive for this commit was that I've spent the last week trying to get
a grip on the test suite with moderate success. I think this restructuring
will help new users to get started with the test suite faster. Or, more
directly, I wanted to add some documentation on comparing test results, but
couldn't find a decent spot in the testing documentation structure.

There is still one issue I'd like to discuss here. The test suite
documentation is a bit confusing in terminology. There are three import
concepts: All of the testing stuff in llvm, the llvm/test DejaGNU part, and
the test-suite part.

Currently, the first is called the "LLVM test suite" (which is the subject of
the testing documentation). The second is called the DejaGNU tests, code
fragment tests or llvm/test. The last one is the most confusing, it's called
the whole program tests, test-suite (named after the svn directory it is in,
easily confused with the entire test suite), llvm-test (named after the
directory in which you should place it, easily confused with llvm/test).

I propose a few changes here. First, to use the "whole program tests", you
should checkout the "test-suite" svn module into a "projects/llvm-test"
directory. This would be way more logical if you could just checkout into a
"projects/test-suite" directory.

Furthermore, I would no longer refer to all of llvm's testing stuff as the
"LLVM test suite", but use the term "LLVM test architecture" or "framework" or
something similar (suggestions?). Since the testing stuff is composed of two
distinct parts, I'm not sure the term "suite" is appropriate here. If people
think that "test suite" is still very much appropriate here, I think that
renaming the "test-suite" svn module would be in order.

Lastly, I would refer to the "whole program tests" as the "Test suite", since
it is a suite of testing programs that can be run as a whole. This will also
make sure that the name properly matches the svn module that the "whole
program tests" are housing in.

After these changes, I think the testing framework should be a lot easier to
describe and document and more accessible to new users.

Any comments/questions/objections?

Gr.

Matthijs

2

Hi all,

as you might have seen, I just did some restructuring on the testing
documentation. Since the changes are only documentation and mostly text
movement, I took the liberty of committing without posting a patch first. Was
that appropriate?.

Yep, if you consider it to be obvious, it can be committed without approval.

The incentive for this commit was that I've spent the last week trying to get
a grip on the test suite with moderate success. I think this restructuring
will help new users to get started with the test suite faster. Or, more
directly, I wanted to add some documentation on comparing test results, but
couldn't find a decent spot in the testing documentation structure.

Awesome, we welcome improvements, obviously there are many to make here :slight_smile:

There is still one issue I'd like to discuss here. The test suite
documentation is a bit confusing in terminology. There are three import
concepts: All of the testing stuff in llvm, the llvm/test DejaGNU part, and
the test-suite part.

Ok.

Currently, the first is called the "LLVM test suite" (which is the subject of
the testing documentation). The second is called the DejaGNU tests, code
fragment tests or llvm/test. The last one is the most confusing, it's called
the whole program tests, test-suite (named after the svn directory it is in,
easily confused with the entire test suite), llvm-test (named after the
directory in which you should place it, easily confused with llvm/test).

llvm-test was the name of the CVS module. When we moved to SVN, it got renamed to test-suite, which I consider to be very confusing. We should probably recommend that people check it out into test-suite, and refer to it as that. Alternatively, change the svn module name to something more useful.

Furthermore, I would no longer refer to all of llvm's testing stuff as the
"LLVM test suite", but use the term "LLVM test architecture" or "framework" or
something similar (suggestions?).

LLVM testing infrastructure?

Since the testing stuff is composed of two
distinct parts, I'm not sure the term "suite" is appropriate here. If people
think that "test suite" is still very much appropriate here, I think that
renaming the "test-suite" svn module would be in order.

Right. It is somewhat complicated by the fact that "llvm-test" is really just a collection of programs with some makefile drivers. How about calling it "program-tests" or "llvm-program-tests" or something like that?

Lastly, I would refer to the "whole program tests" as the "Test suite", since
it is a suite of testing programs that can be run as a whole. This will also
make sure that the name properly matches the svn module that the "whole
program tests" are housing in.

Yeah, we definitely need a better name :slight_smile:

After these changes, I think the testing framework should be a lot easier to
describe and document and more accessible to new users.

Sounds great to me,

-Chris

3

Currently, the first is called the “LLVM test suite” (which is the
subject of
the testing documentation). The second is called the DejaGNU tests,
code
fragment tests or llvm/test. The last one is the most confusing,
it’s called
the whole program tests, test-suite (named after the svn directory
it is in,
easily confused with the entire test suite), llvm-test (named after
the
directory in which you should place it, easily confused with llvm/
test).

llvm-test was the name of the CVS module. When we moved to SVN, it
got renamed to test-suite, which I consider to be very confusing. We
should probably recommend that people check it out into test-suite,
and refer to it as that. Alternatively, change the svn module name to
something more useful.

I think at one time we did try to allow people to check it out as test-suite, but there is something in the Makefiles that require it to be llvm-test (I don’t know exactly what though). So this is something that needs to be fixed. It would be great to get it fixed.

-Tanya

4

I think at one time we did try to allow people to check it out as
test-suite, but there is something in the Makefiles that require it to be
llvm-test (I don't know exactly what though). So this is something that
needs to be fixed. It would be great to get it fixed.

From what I've seen, the llvm-test directory is special in that it doesn't get
make'd when you type make in the llvm root directory (which makes sense, but
might also suggest that the projects/ directory is perhaps not the right place
for llvm-test). Anyway, it should be obvious to just modify the Makefile to
also allow test-suite.

Gr.

Matthijs

5

Hi All,

I've finally implemented my proposed changes to the testing documentation. I
think I fixed up the naming everywhere, but I did only a single pass over the
document, so I might have missed something. Review welcome :slight_smile:

It is now recommended to put the test suite in "projects/test-suite" instead
of "projects/llvm-test". I've also updated the configure script and
projects/Makefile to reflect this. I haven't run autoconf, since I don't have
the right version, so someone should probably do that to check my manual
changes to configure. Also, it would perhaps be useful to add a deprecation
warning when using llvm-test, but my autoconf magic is insufficient (plus I
couldn't really test that due to no autoconf-2.6).

Gr.

Matthijs

6

Great!

To really finish this work, the nightly tester script needs to be updated. Its currently full of llvm-test references. Would you be willing to modify this as well?

Thanks,
Tanya

7

Hi Tanya,

To really finish this work, the nightly tester script needs to be updated.
Its currently full of llvm-test references. Would you be willing to modify
this as well?

I could have a go, though I haven't touched the nightly tester script at all
yet. I won't get around to this until somewhere next week, so if anyone else
wants to take this, go ahead :slight_smile:

Gr.

Matthijs