Proposal for improving the llvm nightly tester

hi all,
After having some discussions in the IRC, I am trying here to come up with a proposal for GSoC 2008 for improving the llvm nightly tester[1].Following are the ideas and suggestions that came up in the discussion, if you have any comment or any other suggestion please add them to the list. I have some doubts in some places.

  1. Improvements to the perl script which manage actual testing (./util/NewNightlyTest.pl)
    (a). compiling llvm-gcc
    (b). allowing someone to use an existing tree versus checking out a new one

  2. The improvements to the email that get sent to the llvm-testresults
    (a). finding a point in time where some test failed
    (b). comparing two different sets of test results that aren’t just 2 days apart
    here if you have any suggestions which help someone analysis test results would be grateful.

  3. Running llvm testsuite for all targets, target variants and operating systems(using an simulator for hardware which is not available) ex:gcc project’s compile farm [2]

  4. Running Fortran, Ada, C/C++/ObjC tests as well with test results emailed to the llvm-testresults.

Most of the server side and the database design has already done by “tonic”(IRC nickname at #llvm), so suggestions regarding others would be important, so that we have a fully featured nightly tester .And finally all the php scripts need to be updated according to the required new features.
Thanks in advance. Please add your comments.!
Regards,
-Rajika

[1] - http://llvm.org/nightlytest/
[2] - http://gcc.gnu.org/wiki/CompileFarm

My main complaints about the nightly tester are:
Loading the web pages is waaay tooo sllooooow
The information provided is not terribly useful for tracking down bugs. What bug fixers need is environment+command line options adequate to reproduce the problem. Access to the run logs would be a good start.

Hi Dale,

My main complaints about the nightly tester are:
Loading the web pages is waaay tooo sllooooow

One of the complaints there was that the nightly tester is too slow. I am looking forward to hear any idea from some one to improve that.

The information provided is not terribly useful for tracking down bugs. What bug fixers need is environment+command line options adequate to reproduce the

problem. Access to the run logs would be a good start.

This information can be added to the email that get delivered to the llvm-testresults. Thanks for the inputs!

-Rajika

1. Improvements to the perl script which manage actual testing
(./util/NewNightlyTest.pl)
    (a). compiling llvm-gcc
    (b). allowing someone to use an existing tree versus checking out a
new one
2. The improvements to the email that get sent to the llvm-testresults
     (a). finding a point in time where some test failed
     (b). comparing two different sets of test results that aren't just 2
days apart
here if you have any suggestions which help someone analysis test results
would be grateful.

I think you need to expand on this. Basically, these would be improvements to how the data is displayed and analyzed. The second part is making the emails that get sent out better. They are separate things.

You should come up with ideas on your own about how someone might want to interpret the data and allow that flexibility via the web pages.

3. Running llvm testsuite for all targets, target variants and operating
systems(using an simulator for hardware which is not available) ex:gcc
project's compile farm [2]

How will you test this?

4. Running Fortran, Ada, C/C++/ObjC tests as well with test results emailed
to the llvm-testresults.

Are these different or new tests? Why wouldn't they just be a part of the llvm-test module? Then you don't have to do any modifications to the scripts.

-Tanya

> 4. Running Fortran, Ada, C/C++/ObjC tests as well with test results emailed
> to the llvm-testresults.

Are these different or new tests? Why wouldn't they just be a part of the
llvm-test module? Then you don't have to do any modifications to the
scripts.

Probably means the gcc testsuite.

Ciao,

Duncan.

Eventually, we want to to move all lang. FE tests from llvm/test to llvm-gcc testsuite once nightly tester script start supporting llvm-gcc-4.2 builds.

4. Running Fortran, Ada, C/C++/ObjC tests as well with test results emailed
to the llvm-testresults.

Are these different or new tests? Why wouldn't they just be a part of the
llvm-test module? Then you don't have to do any modifications to the
scripts.

Probably means the gcc testsuite.

Well then this should be clarified in the proposal. :slight_smile:

-Tanya

Thanks all for your ideas, Please review the proposal at Google web app. I have added my proposal there.
Thanks!
-Rajika