Deprecating libc++'s testit script

Hi All,

I am hoping to remove libcxx/test/testit in the very near future. I
would like all users to use LLVM LIT to run the libc++ tests. testit
is being removed to allow for more complex test formats including
micro-benchmarks and tests for the soon to come filesystem library.

If you are a testit user please reach out to me. I will make sure that
all required functionality from testit is in LIT before making the
switch.

Using CMake to build libc++ is recommended but not required when using
LIT to test. CMake generates a `lit.site.cfg` file that provides
configuration information on how to run the tests. If you are not
using CMake to build libc++ then you should manually create a
`lit.site.cfg` file from `libcxx/test/lit.site.cfg.in`. Please see the
libcxx documentation below about how the `lit.site.cfg` file is used.

The documentation for using CMake and LIT can be found on the landing
page for libcxx at:
http://libcxx.llvm.org

I've written a guide for using LIT to test libcxx on the command line
along with documentation for each option. The documentation can be
found here:
http://libcxx.llvm.org/lit_usage.html

Documentation for LIT command line usage (for the tool itself) can be
found here:
http://llvm.org/docs/CommandGuide/lit.html

Please, please, please reach out to me if you have any questions,
concerns or advice. I would like to help make this transition as
smooth as possible.

/Eric

I dislike this, testit makes it very easy to run the tests on a fresh
netbsd vm without having to install python first...

Joerg

I dislike this, testit makes it very easy to run the tests on a fresh
netbsd vm without having to install python first...

I understand that the simplicity is beneficial to many users. The
reason I want to remove the script outright is because at some point
testit will no longer provide decent test coverage over all of libc++.
testit users may fail to see failing tests in newer parts of libc++
and that would be detrimental to libc++ quality as a whole.

Until a significant number of incompatible tests are checked into
libc++ I see no reason why testit can't remain in the repository but
once using testit becomes a "gotcha" I would like it to be removed.

/Eric

The rest of the LLVM project has required python for running tests for
years, and it hasn't been a problem. I don't think its reasonable to hold
up simplifying the infrastructure because of this (quite narrow) use case.

Joerg,

Would you be okay with this if we got the remote testing stuff working such that tests are driven from a separate machine? That would alleviate the need for python on the machine under test.

Jon

The rest of the LLVM project does not consist of *runtime* tests.

Joerg

Sure, if all it needs e.g. is a ssh server, that's fine with me.

Joerg

The test suite and compiler-rt also use Python to the best of my knowledge.

compiler-rt's builtin part also has a test script. Can't comment on the
test suite, I don't normally run that.

Joerg

>
> > > The rest of the LLVM project has required python for running tests
for
> > > years, and it hasn't been a problem. I don't think its reasonable to
hold
> > > up simplifying the infrastructure because of this (quite narrow) use
> > case.
> >
> > The rest of the LLVM project does not consist of *runtime* tests.
>
>
> The test suite and compiler-rt also use Python to the best of my
knowledge.

compiler-rt's builtin part also has a test script. Can't comment on the
test suite, I don't normally run that.

True, but that script was broken long ago and seems to be unused.