NewNightlyTester.pl: split into phases?

Hi all,

when looking at NewNightlyTester.pl, I see it has several distinct
phases:

1. Get a current tree
2. ./configure
3. Build
4. Run a selection of tests
7. Send test results to web site
8. Clean up

Wouldn't it make sense to allow each phase to be activated individually?
Or possibly a subset of phases, something along the lines of

  -from-phase=test -to-phase=report

With that, I could do all kinds of nifty things, such as checking out
once and running the test site for i386 and amd64; or testing compiler
options without having to re-download the source tree; or avoiding
sending irrelevant test results because the test suite isn't properly
installed yet.

Does that make sense? Problems?
I'd be willing to implement this if something like that is welcomed.

Regards,
Jo

Hi all,

when looking at NewNightlyTester.pl, I see it has several distinct
phases:

1. Get a current tree
2. ./configure
3. Build
4. Run a selection of tests
7. Send test results to web site
8. Clean up

Wouldn't it make sense to allow each phase to be activated individually?
Or possibly a subset of phases, something along the lines of

-from-phase=test -to-phase=report

With that, I could do all kinds of nifty things, such as checking out
once and running the test site for i386 and amd64; or testing compiler
options without having to re-download the source tree; or avoiding
sending irrelevant test results because the test suite isn't properly
installed yet.

Does that make sense? Problems?
I'd be willing to implement this if something like that is welcomed.

Its been on my todo for awhile, but this is what I would like to see.
- ability to run the nightly tester from a tree thats already checked out and updated. I think right now it forces you to always check out a new tree and if you don't it reports a build error.
- ability to check out llvm-gcc or update llvm-gcc and build it before running tests. In addition to using a prebuilt binary.

If you fix the two things above then you should be able to just rerun the script multiple times for different targets like you want. It will also solve problems many others have experienced.

A -noreport option is probably a good idea too.

Just be sure not to change what it sends to the website. That needs to stay the same.

-Tanya

- ability to check out llvm-gcc or update llvm-gcc and build it before
running tests. In addition to using a prebuilt binary.

Does it need a prebuilt binary?
I have been suspecting so since it has been failing with BUILD ERROR for
me. I just haven't found the time to verify that yet.

I have yet to try building llvm-gcc from sources though. The
instructions seem to be clear enough, but it's still possible that I'll
hit a hidden snag.
Well, I'll try that anyway. I like my software well-done after all :slight_smile:

If you fix the two things above then you should be able to just rerun
the script multiple times for different targets like you want. It will
also solve problems many others have experienced.

Ensuring that the tree is pristine could be a bit of a challenge. It's
always possible that the user inadvertently changed or deleted a file,
giving a false alarm for all future checks.
Options to avoid that seem to be:
1. Do a 'make dist-clean' before running the tests. Of course, if
dist-clean has bugs, this may create trouble. On the other hand, it
would exercise dist-clean on a regular basis, which might be a Good
Thing.
2. Recreate $BUILDDIR as a copy of a pristine check-out directory each
time the tester is run. This would be much faster than a download, but
still a lot of unnecessary load.
3. For each test, first run svn revert, then ask svn about any
unversioned files and delete them, finally svn update.

(2) would be easiest. (3) would be most failsafe. (1) might be more
useful for the project because it could uncover bugs in the dist-clean
target though. (OTOH one could argue that this should be a separate
regression test.)

A -noreport option is probably a good idea too.

OK, that should be easy to add.

Just be sure not to change what it sends to the website. That needs to
stay the same.

Sure. I think wrapping an if statement around the HTTP send should be
easy; the code of NewNightlyTester.pl isn't very complicated.

On a tangent:
What's the best way to submit patches: Bugzilla? Something else?

Regards,
Jo

- ability to check out llvm-gcc or update llvm-gcc and build it before
running tests. In addition to using a prebuilt binary.

Does it need a prebuilt binary?
I have been suspecting so since it has been failing with BUILD ERROR for
me. I just haven't found the time to verify that yet.

I have yet to try building llvm-gcc from sources though. The
instructions seem to be clear enough, but it's still possible that I'll
hit a hidden snag.
Well, I'll try that anyway. I like my software well-done after all :slight_smile:

You need to have an llvm-gcc, so you either build it yourself or use one of the prebuilt binaries. The nightly tester just needs to know where its located, but it doesn't know anything about llvm-gcc otherwise. Since most people are testing TOT and llvm/llvm-gcc are tied together it makes sense to update llvm-gcc before running the nightly tester. However, it takes quite a bit of time to checkout and build llvm-gcc, so having the option to just update existing sources would be nice.

If you fix the two things above then you should be able to just rerun
the script multiple times for different targets like you want. It will
also solve problems many others have experienced.

Ensuring that the tree is pristine could be a bit of a challenge. It's
always possible that the user inadvertently changed or deleted a file,
giving a false alarm for all future checks.
Options to avoid that seem to be:
1. Do a 'make dist-clean' before running the tests. Of course, if
dist-clean has bugs, this may create trouble. On the other hand, it
would exercise dist-clean on a regular basis, which might be a Good
Thing.
2. Recreate $BUILDDIR as a copy of a pristine check-out directory each
time the tester is run. This would be much faster than a download, but
still a lot of unnecessary load.
3. For each test, first run svn revert, then ask svn about any
unversioned files and delete them, finally svn update.

(2) would be easiest. (3) would be most failsafe. (1) might be more
useful for the project because it could uncover bugs in the dist-clean
target though. (OTOH one could argue that this should be a separate
regression test.)

I'd rather not use the nightly tester to test make dist-clean. I'm not sure how many people actually use make dist anything right now so it probably has bugs too. This should probably be investigated separately.

I think the only way to ensure the tree is pristine is to do a clean checkout like it does now. So I think that should be the preferred way. However, I think its worthwhile to be able to update your tree and build using the nightly tester. Checking out and building llvm can be a time consuming process for people with slow network connections and slow machines. This is worthwhile in my opinion, but its not necessarily the solution to your problem if you want to make sure your tree is perfect.

However, I think #2 is the better solution for your particular problem and I don't think you need to do any changes to the nightly tester, just tell it to use a new build directory.

A -noreport option is probably a good idea too.

OK, that should be easy to add.

Just be sure not to change what it sends to the website. That needs to
stay the same.

Sure. I think wrapping an if statement around the HTTP send should be
easy; the code of NewNightlyTester.pl isn't very complicated.

Cool.

On a tangent:
What's the best way to submit patches: Bugzilla? Something else?

Send to llvm-commits or llvm-dev (as long as its not too big).

-Tanya

P.S. I'm not sure if someone has gotten a chance to look at your dejagnu bugs you filed, but if not.. I'll try to take a look this weekend. Unfortunately, I've been preoccupied this week with family matters.

>> A -noreport option is probably a good idea too.
>
> OK, that should be easy to add.
>
>> Just be sure not to change what it sends to the website. That needs
>> to
>> stay the same.
>
> Sure. I think wrapping an if statement around the HTTP send should be
> easy; the code of NewNightlyTester.pl isn't very complicated.
>

Cool.

Actually it's trivial. There was an undocumented $TESTING variable that
already did exactly what -noreport was supposed to do.

Patch is attached. Option is now named -nosubmit, in line with
-submit-server and -submit-script.

P.S. I'm not sure if someone has gotten a chance to look at your
dejagnu bugs you filed,

No, the bugs have remained undisturbed. (Except that maybe somebody
changed the priority.)

but if not.. I'll try to take a look this
weekend. Unfortunately, I've been preoccupied this week with family
matters.

I have to admit that I was beginning to wonder how fast bug reports and
fixes would get recognized, but I kept reminding myself that all sorts
of things can interfere (seems I guessed right with that).

Oh, and take your time. Some things are more important than others :slight_smile:

Regards,
Jo

NewNightlyTest.pl.nosubmit.patch (1.78 KB)

A -noreport option is probably a good idea too.

OK, that should be easy to add.

Just be sure not to change what it sends to the website. That needs
to
stay the same.

Sure. I think wrapping an if statement around the HTTP send should be
easy; the code of NewNightlyTester.pl isn't very complicated.

Cool.

Actually it's trivial. There was an undocumented $TESTING variable that
already did exactly what -noreport was supposed to do.

Patch is attached. Option is now named -nosubmit, in line with
-submit-server and -submit-script.

Patch applied. Thanks!!!!

-Tanya

- ability to check out llvm-gcc or update llvm-gcc and build it before
running tests. In addition to using a prebuilt binary.

Does it need a prebuilt binary?
I have been suspecting so since it has been failing with BUILD ERROR for
me. I just haven't found the time to verify that yet.

I have yet to try building llvm-gcc from sources though. The
instructions seem to be clear enough, but it's still possible that I'll
hit a hidden snag.
Well, I'll try that anyway.

Thanks!, this will be very useful.

- ability to check out llvm-gcc or update llvm-gcc and build it before
running tests.

This seems to be a bit more complicated than I thought. There are
variations in the build process depending on whether it's a Darwin
system or not, installed gcc version, and presence or absence of
multilib extensions. I'm not sure that a simple-minded script can
reliably detect all the differences.
The alternative would be options and letting the user configure.

Writing a script would be a difficult task, since it would have to be
tested on as many architectures and configurations as possible, with the
developer having to rely on error reports to fix things (difficult, lots
of work, slow progress).
Options, on the other hand, would tend to perpetuate current entry
barriers.

I'm not too happy about either option, but probably a choice must be
made.
Which shall it be?

In addition to using a prebuilt binary.

This would be simple, of course.

Regards,
Jo

IIUC, all target specific variations are in llvm-gcc configure options. In the beginning it is OK to let user specify llvm-gcc configure options.

- ability to check out llvm-gcc or update llvm-gcc and build it
before
running tests.

This seems to be a bit more complicated than I thought. There are
variations in the build process depending on whether it's a Darwin
system or not, installed gcc version, and presence or absence of
multilib extensions. I'm not sure that a simple-minded script can
reliably detect all the differences.

IIUC, all target specific variations are in llvm-gcc configure
options. In the beginning it is OK to let user specify llvm-gcc
configure options.

I was going to suggest this as well. We should make it *really* simple in that you just specifiy the configure line options (environment variable?) and then we may need a couple extra building options for LLVM (ie. OPTIMIZE_OPTION=-O2).

-Tanya