PowerPC 64 build bots...

chandlerc · December 9, 2012, 10:19am

Hey Galina, Will;

I’ve been working to revive the PPC64 build bots, and succeeded, but not for the right reasons. There were still bootstrap assertion failures and other pretty blatant errors. Then we figured out why: the Clang bootstrapping build bots for Power7 are not actually running any of the Clang tests!

Could one of you tweak this build bot’s configuration to match the other bootstrap bot configurations that run both LLVM and Clang tests?

Is there a a Clang PPC64 build bot that doesn’t self-host so we can get faster turn around?

I think the ideal configuration is:

normal clang-test-only build bot
normal llvm-test-only build bot
bootstrap a new toolchain, and then run clang+llvm tests.

Thoughts?

d0k · December 9, 2012, 11:36am

This turns out to be a bug in our testing infrastructure (http://llvm.org/bugs/show_bug.cgi?id=13598). I added a workaround in r169695 and it seems that the bots are running tests now. The python that ships with fedora 17 seems to use a lot of address space for thread-local storage and exceeded the limit set by the makefile.

Another problem was that the bot reports a green status even though lit aborted. Could be a problem in the buildbot software somewhere. I attached a log from one of the last build runs so this doesn't get lost.

test.log (2.52 KB)

Ulrich_Weigand · December 10, 2012, 2:19pm

I've been working to revive the PPC64 build bots, and succeeded, but
not for the right reasons. There were still bootstrap assertion
failures and other pretty blatant errors. Then we figured out why:
the Clang bootstrapping build bots for Power7 are not actually
running any of the Clang tests!

Could one of you tweak this build bot's configuration to match the
other bootstrap bot configurations that run both LLVM and Clang tests?

Maybe I'm confused somehow, but I thought this one:
http://lab.llvm.org:8011/builders/clang-ppc64-elf-linux2
does bootstrap and then run both LLVM and Clang tests (successfully):
http://lab.llvm.org:8011/builders/clang-ppc64-elf-linux2/builds/775/steps/check-all_1/logs/stdio

The other Clang PPC64 build bot tries to run lnt.nightly-test:
http://lab.llvm.org:8011/builders/clang-ppc64-elf-linux
but fails (and has always been failing); one problem seems to
be that altivec is disabled for some reason? It's been on my
to-do list to check what's going on here ...

Is there a a Clang PPC64 build bot that *doesn't* self-host so we
can get faster turn around?

I think the ideal configuration is:

1) normal clang-test-only build bot
2) normal llvm-test-only build bot
3) bootstrap a new toolchain, and then run clang+llvm tests.

Besides the two bots mentioned above, there's also two LLVM bots:
http://lab.llvm.org:8011/builders/llvm-ppc64-linux1
http://lab.llvm.org:8011/builders/llvm-ppc64-linux2
which simply build and test LLVM only. (Not sure why there's
two of them, they seem to be doing the same thing. Will?)

Bye,
Ulrich

Will_Schmidt · December 10, 2012, 11:02pm

> I've been working to revive the PPC64 build bots, and succeeded, but
> not for the right reasons. There were still bootstrap assertion
> failures and other pretty blatant errors. Then we figured out why:
> the Clang bootstrapping build bots for Power7 are not actually
> running any of the Clang tests!
>
> Could one of you tweak this build bot's configuration to match the
> other bootstrap bot configurations that run both LLVM and Clang tests?

Hi,

So ignoring the ppc64-elf-linux2 bot (which is redundant wrt the
ppc64-elf-linux1 bot), Powerpc64 has three unique buildbots going; the
relevant 'factory' entries for them are so:

LLVMBuilder.getLLVMBuildFactory("ppc64-linux-gnu",

LNTBuilder.getLNTFactory(triple='ppc64-elf-linux1',

ClangBuilder.getClangBuildFactory(triple='ppc64-elf-linux',

I'm happy to change those or add another if desired, but will need
additional input on 'what'.

Maybe I'm confused somehow, but I thought this one:
http://lab.llvm.org:8011/builders/clang-ppc64-elf-linux2
does bootstrap and then run both LLVM and Clang tests (successfully):
http://lab.llvm.org:8011/builders/clang-ppc64-elf-linux2/builds/775/steps/check-all_1/logs/stdio

Correct, or at least that is the intent.

The other Clang PPC64 build bot tries to run lnt.nightly-test:
http://lab.llvm.org:8011/builders/clang-ppc64-elf-linux
but fails (and has always been failing); one problem seems to
be that altivec is disabled for some reason? It's been on my
to-do list to check what's going on here ...

Correct. This was the first clang buildbot that I set up, and it is
configured to do the LNT tests.

> Is there a a Clang PPC64 build bot that *doesn't* self-host so we
> can get faster turn around?
>
> I think the ideal configuration is:
>
> 1) normal clang-test-only build bot
> 2) normal llvm-test-only build bot
> 3) bootstrap a new toolchain, and then run clang+llvm tests.

Besides the two bots mentioned above, there's also two LLVM bots:
http://lab.llvm.org:8011/builders/llvm-ppc64-linux1
http://lab.llvm.org:8011/builders/llvm-ppc64-linux2
which simply build and test LLVM only. (Not sure why there's
two of them, they seem to be doing the same thing. Will?)

They are on different host systems, but are redundant with respect to
each other as far as the llvm testing goes. If having two complicates
things, I can disable the second.

Thanks,
-Will

d0k · December 11, 2012, 11:48am

I've been working to revive the PPC64 build bots, and succeeded, but
not for the right reasons. There were still bootstrap assertion
failures and other pretty blatant errors. Then we figured out why:
the Clang bootstrapping build bots for Power7 are not actually
running any of the Clang tests!

Could one of you tweak this build bot's configuration to match the
other bootstrap bot configurations that run both LLVM and Clang tests?

Maybe I'm confused somehow, but I thought this one:
http://lab.llvm.org:8011/builders/clang-ppc64-elf-linux2
does bootstrap and then run both LLVM and Clang tests (successfully):
http://lab.llvm.org:8011/builders/clang-ppc64-elf-linux2/builds/775/steps/check-all_1/logs/stdio

It does now, but failed to do so until r169695 due to a incompatibility of LLVM's makefiles and Fedora 17's version of python. It still reported a (false) green status, which is indeed confusing and probably a different bug in our buildbot infrastructure. The LLVM bots were unaffected, they're still running on Fedora 16.

Now the bot successfully runs all clang regression tests, which is great and very helpful by reporting new regressions early.

- Ben

Ulrich_Weigand · December 11, 2012, 12:45pm

> Maybe I'm confused somehow, but I thought this one:
> http://lab.llvm.org:8011/builders/clang-ppc64-elf-linux2
> does bootstrap and then run both LLVM and Clang tests (successfully):
> http://lab.llvm.org:8011/builders/clang-ppc64-elf-linux2/builds/
775/steps/check-all_1/logs/stdio

It does now, but failed to do so until r169695 due to a
incompatibility of LLVM's makefiles and Fedora 17's version of
python. It still reported a (false) green status, which is indeed
confusing and probably a different bug in our buildbot
infrastructure. The LLVM bots were unaffected, they're still running
on Fedora 16.

Ah, now I see the python/tls problems in runs earlier this month.

I'm still a bit confused because I distinctly remember that I saw
this bot go from red to green back last month, when I checked in
the last of a series of commits as r168316 (I was specifically
trying to make the bots go green at the time ...).

Is it possible that the python/tls problem was introduced at some
point in between, that is late November? Unfortunately, the build
bot logs don't appear to go back that far.

In any case, thanks for fixing the python issue!

Now the bot successfully runs all clang regression tests, which is
great and very helpful by reporting new regressions early.

Indeed. Now there's just the LNT bot still to make green

Bye,
Ulrich

Will_Schmidt · December 11, 2012, 2:19pm

> > Maybe I'm confused somehow, but I thought this one:
> > http://lab.llvm.org:8011/builders/clang-ppc64-elf-linux2
> > does bootstrap and then run both LLVM and Clang tests (successfully):
> > http://lab.llvm.org:8011/builders/clang-ppc64-elf-linux2/builds/
> 775/steps/check-all_1/logs/stdio
>
> It does now, but failed to do so until r169695 due to a
> incompatibility of LLVM's makefiles and Fedora 17's version of
> python. It still reported a (false) green status, which is indeed
> confusing and probably a different bug in our buildbot
> infrastructure. The LLVM bots were unaffected, they're still running
> on Fedora 16.

Ah, now I see the python/tls problems in runs earlier this month.

I'm still a bit confused because I distinctly remember that I saw
this bot go from red to green back last month, when I checked in
the last of a series of commits as r168316 (I was specifically
trying to make the bots go green at the time ...).

Is it possible that the python/tls problem was introduced at some
point in between, that is late November? Unfortunately, the build
bot logs don't appear to go back that far.

Yes... it was introduced by way of the buildbot host system. The
buildbot host was updated from F16 to F17 levels sometime in November.
That'll be precisely when the python incompatibility for the clang build
bots was introduced to that environment.

Thanks,
-Will

Ulrich_Weigand · December 11, 2012, 2:40pm

Ah, OK. That would indeed explain it.

Thanks,
Ulrich