Call for Help: Testing

Project Infrastructure LLVM Dev List Archives
1

There was a short discussion in the comments of PR6696 about Linux
testing.

The 2.7 release is slipping primarily because there are x86-32 problems to
fix (there is also a Darwin problem and some other issues).

We have two problems with our testing:

The OSU/Google buildbots aren't running test-suite, apparently. I didn't
realize that and as this is the public testing face for LLVM, I think this
needs to be fixed. I don't know if that's in Daniel's new testing plan or
not but that's a pretty large hole in what those outside Apple are able to
see regularly.

The second problem is the lack of x86-32 testing on Linux. This is only
going to get worse as x86-32 desktops and servers continue to disappear.

Pretty much the only way you'll get an x86-32 these days is by purchasing an
Atom netbook or laptop. Unfortunately, these kinds of devices don't suit
themselves well to testing as they frequently aren't connected to the 'net.

I have a netbook I can add to the testing pool. If enough other people do the
same perhaps we'll have at least a few up at any one time.

Another alternative is to add a buildbot recipe that builds x86-32 targets on
x86-64 machines. Is this already available? If so, it seems we should
already be running sufficient x86-32 testing. As we apparently don't have
enough such testing, I assume we don't have this kind of recipe available yet.
I think most Linux x86 distributions ship multilib compilers so this should
be possible without too much trouble.

Thoughts?

                                    -Dave

2

The second problem is the lack of x86-32 testing on Linux. This is only
going to get worse as x86-32 desktops and servers continue to disappear.

Pretty much the only way you'll get an x86-32 these days is by purchasing an
Atom netbook or laptop. Unfortunately, these kinds of devices don't suit
themselves well to testing as they frequently aren't connected to the 'net.

While x86-32 disappears, virtualization becomes ubiquitous along with x86-64. What about running your tests in x86-32 guests under KVM or Xen? That seems like it would solve everything on the infrastructure you already have. You can run any guest OS you want, too.

I have a netbook I can add to the testing pool. If enough other people do the
same perhaps we'll have at least a few up at any one time.

I've bothered the list with stupid questions about my toy project before and plan to do so much more in the future as my real LLVM project gets moving, so I ought to see if I can donate some cycles. I know nothing about your testing infrastructure, but I do have a 32-bit laptop that is only getting occasional use. How hard would it be to put it to use at least until you ship?

Dustin

3

> The second problem is the lack of x86-32 testing on Linux. This is only
> going to get worse as x86-32 desktops and servers continue to disappear.
>
> Pretty much the only way you'll get an x86-32 these days is by purchasing
> an Atom netbook or laptop. Unfortunately, these kinds of devices don't
> suit themselves well to testing as they frequently aren't connected to
> the 'net.

While x86-32 disappears, virtualization becomes ubiquitous along with
x86-64. What about running your tests in x86-32 guests under KVM or
Xen? That seems like it would solve everything on the infrastructure
you already have. You can run any guest OS you want, too.

That's an even better idea than running x86-32 on x86-64 because it
better emulates a whole 32-bit environment. I'll see if I can
set up such a beast on my box at home. Unfortunately, KVM+Qemu has
been pretty flaky for me.

> I have a netbook I can add to the testing pool. If enough other people
> do the same perhaps we'll have at least a few up at any one time.

I've bothered the list with stupid questions about my toy project before
and plan to do so much more in the future as my real LLVM project gets
moving, so I ought to see if I can donate some cycles. I know nothing
about your testing infrastructure, but I do have a 32-bit laptop that is
only getting occasional use. How hard would it be to put it to use at
least until you ship?

It's not hard. Basically, you set up a buildbot and then contact Daniel to
exchange unique identifiers and passwords that he enters into the buildbot
server and you set up on the client.

                                 -Dave

4

While lack of linux testing of the testsuite is a problem, this is not why the release is slipping. We need more people to fix bugs. It can't be the same people fixing the bugs for every release.

We need more people in the community to help and I can't see any way of sugar coating this message.

-Tanya

P.S. I'm not discounting the need for testing the testsuite on Linux. Its not 32 bit specific though.

5

Hi David,

The second problem is the lack of x86-32 testing on Linux. This is only
going to get worse as x86-32 desktops and servers continue to disappear.

this is not really a problem if you have root access on an x86-64 box. It
is quite easy to install a 32 bit chroot environment. That's what I'm using
on my laptop for testing x86-32. You really can't tell it's not the real thing.
The problem I have is that I don't have root access to the machines I'm using
for running testers, and the administrator doesn't want to set up such a 32 bit
environment.

Another alternative is to add a buildbot recipe that builds x86-32 targets on
x86-64 machines. Is this already available? If so, it seems we should
already be running sufficient x86-32 testing. As we apparently don't have
enough such testing, I assume we don't have this kind of recipe available yet.
I think most Linux x86 distributions ship multilib compilers so this should
be possible without too much trouble.

I am working on such a recipe for testing 32 bit dragonegg, but so far it fails
mysteriously and I didn't figure it out yet.

Ciao,

Duncan.

6

Hi David,

> The second problem is the lack of x86-32 testing on Linux. This is only
> going to get worse as x86-32 desktops and servers continue to disappear.

this is not really a problem if you have root access on an x86-64 box. It
is quite easy to install a 32 bit chroot environment. That's what I'm
using on my laptop for testing x86-32. You really can't tell it's not the
real thing. The problem I have is that I don't have root access to the
machines I'm using for running testers, and the administrator doesn't want
to set up such a 32 bit environment.

Yes, this is doable, but it's complicated. To get adequate testing we need
to make it as simple as possible for new users to participate. Either we
need a step-by-step recipe or we need a simpler approach. Either this or
virtualization would work but both are complicated to set up, raising the
bar to participate.

A chroot doesn't run a 32-bit kernel so virtualization would emulate the
32-bit environment slightly better. But I don't know if that's all that
significant in the end.

> Another alternative is to add a buildbot recipe that builds x86-32
> targets on x86-64 machines. Is this already available? If so, it seems
> we should already be running sufficient x86-32 testing. As we apparently
> don't have enough such testing, I assume we don't have this kind of
> recipe available yet. I think most Linux x86 distributions ship multilib
> compilers so this should be possible without too much trouble.

I am working on such a recipe for testing 32 bit dragonegg, but so far it
fails mysteriously and I didn't figure it out yet.

Great!

It strikes me that we also have an issue with gcc version. As demonstrated
by PR6616, compiler versions matter. It seems we will need to settle on a
standard compiler for each release to avoid some of the confusion that can
delay releases. What do the buildbots use? Unfortunately, this again makes
things more complicated as it requires testers to install specific versions of
compilers. Linux distributions don't always make that easy.

But one step at a time. Getting more 32-bit Linux testing is the first step.

                           -Dave

7

...Either this or
virtualization would work but both are complicated to set up, raising the
bar to participate.

Is it? I don't really know anything about it, but on a whim read just enough to know the name of the user-space manager program I needed, installed the tools on Fedora 12, created a virtual machine by following the on-screen directions, and installing from a Debian 32-bit .iso. It couldn't have been simpler without someone else actually using the mouse for me. :slight_smile:

It strikes me that we also have an issue with gcc version. As demonstrated
by PR6616, compiler versions matter. It seems we will need to settle on a
standard compiler for each release to avoid some of the confusion that can
delay releases. What do the buildbots use? Unfortunately, this again makes
things more complicated as it requires testers to install specific versions of
compilers. Linux distributions don't always make that easy.

Wouldn't the rest of the build environment also matter?

Another nice thing about virtual images is you can have many per actual machine, provided you go to the trouble of installing them. You can have one for the current version of every popular Linux distro and BSD in existence and run the ones you want to test that day. That ensures you get something close to the build environments people are using, anyway.

But one step at a time. Getting more 32-bit Linux testing is the first step.

I have a partner in crime that is much more of an admin than I am. I'm going to see if he has something to donate cycles from, which would be more useful than me doing it for several reasons. He's a partner in the real project I intend to rely on LLVM for, so it's a reasonable request.

I know he has a linuxified Playstation, so he might have other fun toys.

Dustin

8

I don't know enough yet to help there, sorry. I think anything I could fix without knowing LLVM internals would already be fixed. Maybe down the road....

Dustin

9

> ...Either this or
> virtualization would work but both are complicated to set up, raising the
> bar to participate.

Is it? I don't really know anything about it, but on a whim read just
enough to know the name of the user-space manager program I needed,
installed the tools on Fedora 12, created a virtual machine by following
the on-screen directions, and installing from a Debian 32-bit .iso. It
couldn't have been simpler without someone else actually using the mouse
for me. :slight_smile:

Which manager are you using? I had a heck of a time setting up virt-manager.
Perhaps Fedora does a better job integrating it than Debian. It wouldn't
surprise me. I'm glad it's easy somewhere. :slight_smile:

> It strikes me that we also have an issue with gcc version. As
> demonstrated by PR6616, compiler versions matter. It seems we will need
> to settle on a standard compiler for each release to avoid some of the
> confusion that can delay releases. What do the buildbots use?
> Unfortunately, this again makes things more complicated as it requires
> testers to install specific versions of compilers. Linux distributions
> don't always make that easy.

Wouldn't the rest of the build environment also matter?

To a point, yes, but the compiler is the biggest factor, I think. I
don't imagine libc would mess things up too often, although a libstdc++
difference (I suspect) did prevent me from seeing a build failure recently.

Another nice thing about virtual images is you can have many per actual
machine, provided you go to the trouble of installing them. You can

Right.

have one for the current version of every popular Linux distro and BSD
in existence and run the ones you want to test that day. That ensures
you get something close to the build environments people are using, anyway.

I agree. It would be good to have a list of build systems we want working
for each release, as a guide to testers on what they should set up if
possible. I suppose Tanya is the one to decide which platforms are
release critical.

I know he has a linuxified Playstation, so he might have other fun toys.

Tell him not to update to the latest firmware! :slight_smile:

                              -Dave

10

AFAICT, the only bugs left are Clang bugs and a Darwin llvm-gfortran bug.
The Linux bugs have all been triaged and either fixed, have what should
be a simple resolution or deemed not release-blocking:

Bug 6616 - gcc 4.2 problem, verified to work with 4.3 and 4.4
Bug 6660 - clang regressions
Bug 6696 - Just revert the offending patch
Bug 6778 - Darwin bug

                                   -Dave

11

Which manager are you using?

virt-manager. Not because I chose it intelligently from a survey from
alternatives, but because that's what was listed in the
directions-for-boneheads I used. Seems to be the standard RedHat offering.

I had a heck of a time setting up virt-manager. Perhaps Fedora does a
better job integrating it than Debian. It wouldn't surprise me. I'm
glad it's easy somewhere. :slight_smile:

Well, RedHat is really pushing virtualization on the server, and has decided to throw its lot in with KVM vice Xen. So it sort of makes sense that they're trying to make it Just Work(TM) in RHEL, and Fedora is their testbed.

I have gone back and forth between Fedora and Debian a couple of times. ATM I'm tempted to go back to Debian, due to some instability in Fedora. Debian definitely wins there. And Slackware, but the package tree is small and I'm too lazy to keep up to date on security patches manually.

I agree. It would be good to have a list of build systems we want
working for each release, as a guide to testers on what they should
set up if possible. I suppose Tanya is the one to decide which
platforms are release critical.

If kvm is scriptable, as I assume it is, you might be able to automate the ones you want.

I know he has a linuxified Playstation, so he might have other fun
toys.

Tell him not to update to the latest firmware! :slight_smile:

I suspect he knows. The project I hope to be asking questions about soon is that LLVM should make feasible a concept I have for pervasive, mostly automatic parallelism at the language level. So I'm glad to have a cell processor available.

Dustin

12

...Either this or
virtualization would work but both are complicated to set up, raising the
bar to participate.

Is it? I don't really know anything about it, but on a whim read just
enough to know the name of the user-space manager program I needed,
installed the tools on Fedora 12, created a virtual machine by following
the on-screen directions, and installing from a Debian 32-bit .iso. It
couldn't have been simpler without someone else actually using the mouse
for me. :slight_smile:

Which manager are you using? I had a heck of a time setting up virt-manager.
Perhaps Fedora does a better job integrating it than Debian. It wouldn't
surprise me. I'm glad it's easy somewhere. :slight_smile:

kvm seems to work fine lately on Debian:
kvm-img create foo.img 4G
kvm -hda foo.img -cdrom /path/to/release.iso -m 1024

It strikes me that we also have an issue with gcc version. As
demonstrated by PR6616, compiler versions matter. It seems we will need
to settle on a standard compiler for each release to avoid some of the
confusion that can delay releases. What do the buildbots use?
Unfortunately, this again makes things more complicated as it requires
testers to install specific versions of compilers. Linux distributions
don't always make that easy.

Wouldn't the rest of the build environment also matter?

To a point, yes, but the compiler is the biggest factor, I think. I
don't imagine libc would mess things up too often, although a libstdc++
difference (I suspect) did prevent me from seeing a build failure recently.

Another nice thing about virtual images is you can have many per actual
machine, provided you go to the trouble of installing them. You can

Right.

There are also some free build farms that you may use.
Besides the GCC compile farm, there is the OpenSUSE build service
(build.opensuse.org) that has an autobuilder for several Linux
distributions/architectures.
Its a bit complicated to automate though (you need to commit the sources
you want built, rather than having them pulled from somewhere), also it
wants a rpm build script (but you can run make check from the rpm build
script).
And if you care about Solaris, opencsw.org has a buildfarm where you may
request access
((http://www-mockup.opencsw.org/extend-it/signup/to-upstream-maintainers/)

IMHO the best way to get more widespread testing on Linux would be to
make llvm/clang able to fully build the kernel, (e)glibc, and other
essential software so that one can try to rebuild the full system with
clang.

Or at least have some wrappers that build using clang usually, and falls
back to gcc for not supported flags (like
-fno-toplevel-reorder/-fno-unit-at-a-time).

If that works then maybe there would be a distribution that offers an
easy way to rebuild your system (like Gentoo) that could be used to test
LLVM.

Best regards,
--Edwin

13

While lack of linux testing of the testsuite is a problem, this is
not why the release is slipping. We need more people to fix bugs. It
can't be the same people fixing the bugs for every release.

We need more people in the community to help and I can't see any way
of sugar coating this message.

AFAICT, the only bugs left are Clang bugs and a Darwin llvm-gfortran bug.
The Linux bugs have all been triaged and either fixed, have what should
be a simple resolution or deemed not release-blocking:

Bug 6616 - gcc 4.2 problem, verified to work with 4.3 and 4.4

Alright. Its tricky when you have multiple versions of gcc. There is no good solution to this, but thats fine.

Bug 6660 - clang regressions

These are only on Linux. So we still have some Linux regressions right now, but hopefully they are easy to fix.

Bug 6696 - Just revert the offending patch

Jim has sent me a patch. Getting code owner approval.

Bug 6778 - Darwin bug

Yes. Still in flux.

-Tanya

14

Yes, but this isn't the community to do it. My point is that it's difficult
for "the community" to help when there aren't any actual bugs for that
community to fix.

To say that "It can't be the same people fixing the bugs for every release"
discounts a lot of good, hard work put in by a lot of people. There wasn't a
lot for "the community" to do before late last week except run tests, which in
fact a number of people did.

Did I miss an announcement of PR6586 somewhere? The first I knew about it was
mid last week. Maybe I missed the posting on this mailing list about it.
It's been open for almost a month. Without knowing about that bug, it's
pretty tough for people to do anything about it.

                              -Dave

15

BTW, I will happily work on the Darwin bug as soon as Apple lets me install
Mac OS X on a virtual machine! :slight_smile:

                             -Dave

16

Hi, David

To say that "It can't be the same people fixing the bugs for every release"
discounts a lot of good, hard work put in by a lot of people. There wasn't a
lot for "the community" to do before late last week except run tests, which in
fact a number of people did.

Why? Prerelease 1 tarballs are available for at least 3 weeks. PR6623
was entered into bugzilla at March, 20, etc., etc., etc.

Did I miss an announcement of PR6586 somewhere? The first I knew about it was
mid last week. Maybe I missed the posting on this mailing list about it.
It's been open for almost a month. Without knowing about that bug, it's
pretty tough for people to do anything about it.

It's pretty strange to hear about this from you. I hoped everyone who
worked with llvm for some non-trivial period of time knows about
"master" bug habits in the LLVM bugzilla. Such bugs were created for
2.5 and 2.6 releases, so, one should definitely expect to see another
one for 2.7.

Also, I should note that master bug in question was *explicitly*
mentioned in Tanya's 2.7 prerelease 1 e-mail sent on March, 17.
Have you seen this e-mail?

17

Hi, David

> To say that "It can't be the same people fixing the bugs for every
> release" discounts a lot of good, hard work put in by a lot of people.
> There wasn't a lot for "the community" to do before late last week except
> run tests, which in fact a number of people did.

Why? Prerelease 1 tarballs are available for at least 3 weeks. PR6623
was entered into bugzilla at March, 20, etc., etc., etc.

But where was it posted to let people know about it?

> Did I miss an announcement of PR6586 somewhere? The first I knew about
> it was mid last week. Maybe I missed the posting on this mailing list
> about it. It's been open for almost a month. Without knowing about that
> bug, it's pretty tough for people to do anything about it.

It's pretty strange to hear about this from you. I hoped everyone who
worked with llvm for some non-trivial period of time knows about
"master" bug habits in the LLVM bugzilla. Such bugs were created for

Nope. I've seen master bugs with other projects, but I don't see
anything on the LLVM website to direct me. Again, to get volunteers
to help, it has to be as simple as possible for people to figure out
what needs to be done.

This is on all of us. I'm not pointing fingers at anyone in particular.

2.5 and 2.6 releases, so, one should definitely expect to see another
one for 2.7.

I hadn't seen those bugs. And even if I had, what about newer people?

Also, I should note that master bug in question was *explicitly*
mentioned in Tanya's 2.7 prerelease 1 e-mail sent on March, 17.
Have you seen this e-mail?

Hmm, I'm sure I got it. I must have missed it. I can't read the list
every day and after a few days things get pretty backed up.

I think it would be good to mention the bug number with every pre-release
related e-mail from the release manager. There's less chance of missing
an important message.

I'm just relating my experience. I got pretty frustrated seeing everyone's
work dismissed as inadequate.

                              -Dave

18

I don't see it in the gmane archives but that doesn't matter. Looking back
at those archives, I recall that I was in the middle of iterating on a
patch to add some debug capability to LLVM. I was focused on that and
could easily miss a bug reference like that. I wasn't focusing on the
2.7 release at the time, quite frankly.

People are going to miss e-mails. That's a fact of life.

Perhaps weekly status updates with tracking bug number after the release
branch is made would be helpful, along with daily updates as we get
closer to the scheduled release date.

I can't say I would have been able to devote loads of time to fixing bugs
at that point (I could have helped a bit, I'm sure) but having the information
out there and accessible (linked from the project home page, etc.) would help
many people to get involved.

                                  -Dave

19

Why? Prerelease 1 tarballs are available for at least 3 weeks. PR6623
was entered into bugzilla at March, 20, etc., etc., etc.

But where was it posted to let people know about it?

In pretty secret place! llvmdev & cfe-dev mailing lists.
See http://lists.cs.uiuc.edu/pipermail/llvmdev/2010-March/030270.html
for example

I think it would be good to mention the bug number with every pre-release
related e-mail from the release manager. There's less chance of missing
an important message.

It is. See http://lists.cs.uiuc.edu/pipermail/llvmdev/2010-March/030574.html
as an example.

20

Hi, David

To say that “It can’t be the same people fixing the bugs for every
release” discounts a lot of good, hard work put in by a lot of people.
There wasn’t a lot for “the community” to do before late last week except
run tests, which in fact a number of people did.

Why? Prerelease 1 tarballs are available for at least 3 weeks. PR6623
was entered into bugzilla at March, 20, etc., etc., etc.

But where was it posted to let people know about it?

It was posted to the dev mailing lists. In fact, that’s the exact same place that your email was posted. How can you claim that this is impossible or even hard to find, and yet send emails to it expecting people to find, read, and take action based on them? (Remember, your subject line was ‘Call for help’…)

Did I miss an announcement of PR6586 somewhere? The first I knew about
it was mid last week. Maybe I missed the posting on this mailing list
about it. It’s been open for almost a month. Without knowing about that
bug, it’s pretty tough for people to do anything about it.

It’s pretty strange to hear about this from you. I hoped everyone who
worked with llvm for some non-trivial period of time knows about
“master” bug habits in the LLVM bugzilla. Such bugs were created for

Nope. I’ve seen master bugs with other projects, but I don’t see
anything on the LLVM website to direct me.

Seriously? http://www.lmgtfy.com/?q=llvm+2.7+release+master+bug

1st link: the gmane archive you claim to have looked at, with the bug number in the snippet
4th link: the bugs mailing list email noting the creation of the bug, with all of its contents and a link to bugzilla

I hadn’t seen those bugs. And even if I had, what about newer people?

Also, I should note that master bug in question was explicitly
mentioned in Tanya’s 2.7 prerelease 1 e-mail sent on March, 17.
Have you seen this e-mail?

Hmm, I’m sure I got it. I must have missed it. I can’t read the list
every day and after a few days things get pretty backed up.

I’m not sure why you think burying links that are only relevant for a narrow window of time in a long-lived webpage is more discoverable than email. Also, unless you are subscribed to the commit list, the load from llvmdev is very modest.

I don’t see it in the gmane archives but that doesn’t matter.

I find this hard to believe. Did you really look? There are exactly two links in the email, one of them to the bug. Let’s use the fancy powers of search to help out:
http://article.gmane.org/gmane.comp.compilers.clang.devel/8084/match=release+master+bug

People are going to miss e-mails. That’s a fact of life.

Perhaps weekly status updates with tracking bug number after the release
branch is made would be helpful, along with daily updates as we get
closer to the scheduled release date.

The solution to people not reading email is not to send them more email.