Validation Update

I've been working with Daniel to get a test buildbot going to do validations.
I have a config set up here and will be running tests over the next week to
see if it works as expected. I expect various bugs to pop up.

Right now the buildbot is set up to build every N commits (N=10 for testing
purposes) and also build every Sunday at midnight. The buildbot does not
automatically tag successful validations. That will take some more work and
we don't want that automation quite yet anyway. We'll do it manually for a
while and then once we get the gist of it we can make it automatic.

One thing that would be nice to have is a commit hook on trunk (and eventually
release branches) that lets buildbot know when changes are submitted. As of
now, buildbot polls the svn server every 10 minutes, meaning it could miss
one of the "every 10" commits (revision mod 10 == 0). The buildbot manual
talks a bit about what the svn server needs to do here:

http://buildbot.net/repos/release/docs/buildbot.html#PBChangeSource

Could we get that set up in the llvm.org svn commit hook?

Eventually we'll want to migrate the buidlbot master to some publicly
accessible machine so others can contribute machines to do validations (for
various architectures, etc.). Where should this get hosted? Hosting at
llvm.org would be great but I'm not sure that's possible. Daniel has a
machine set up so that's a possibility too. Wherever it goes, I'll need an
account so I can muck around with the buildbot config as build machines are
added, targets change, etc. Eventually that account should migrate to
whomever is the overall validation coordinator (in the proposal I volunteered
myself to play that role for now).

                                        -Dave

Right now the buildbot is set up to build every N commits (N=10 for testing
purposes) and also build every Sunday at midnight. The buildbot does not
automatically tag successful validations. That will take some more work and
we don't want that automation quite yet anyway. We'll do it manually for a
while and then once we get the gist of it we can make it automatic.

How exactly do you plan to validate it? What point of reference will you be using as a start?

One thing that would be nice to have is a commit hook on trunk (and eventually
release branches) that lets buildbot know when changes are submitted. As of
now, buildbot polls the svn server every 10 minutes, meaning it could miss
one of the "every 10" commits (revision mod 10 == 0). The buildbot manual
talks a bit about what the svn server needs to do here:

http://buildbot.net/repos/release/docs/buildbot.html#PBChangeSource

Could we get that set up in the llvm.org svn commit hook?

What are the pros and cons of this? How much does it tax the server?

Eventually we'll want to migrate the buidlbot master to some publicly
accessible machine so others can contribute machines to do validations (for
various architectures, etc.). Where should this get hosted? Hosting at
llvm.org would be great but I'm not sure that's possible. Daniel has a
machine set up so that's a possibility too. Wherever it goes, I'll need an
account so I can muck around with the buildbot config as build machines are
added, targets change, etc. Eventually that account should migrate to
whomever is the overall validation coordinator (in the proposal I volunteered
myself to play that role for now).

The LLVM server is not the place to host this. Could Cray host a machine to do this?

-Tanya

> Right now the buildbot is set up to build every N commits (N=10 for
> testing purposes) and also build every Sunday at midnight. The buildbot
> does not automatically tag successful validations. That will take some
> more work and we don't want that automation quite yet anyway. We'll do
> it manually for a while and then once we get the gist of it we can make
> it automatic.

How exactly do you plan to validate it? What point of reference will you
be using as a start?

It's all in the proposal I posted a week or so ago.

> One thing that would be nice to have is a commit hook on trunk (and
> eventually release branches) that lets buildbot know when changes are
> submitted. As of now, buildbot polls the svn server every 10 minutes,
> meaning it could miss one of the "every 10" commits (revision mod 10 ==
> 0). The buildbot manual talks a bit about what the svn server needs to
> do here:
>
> http://buildbot.net/repos/release/docs/buildbot.html#PBChangeSource
>
> Could we get that set up in the llvm.org svn commit hook?

What are the pros and cons of this? How much does it tax the server?

It's an extra TCP connection out to the buildbot on every commit. I would
think that would be low cost compared to all the other processing that has to
go on.

> Eventually we'll want to migrate the buidlbot master to some publicly
> accessible machine so others can contribute machines to do validations
> (for various architectures, etc.). Where should this get hosted?
> Hosting at llvm.org would be great but I'm not sure that's possible.
> Daniel has a machine set up so that's a possibility too. Wherever it
> goes, I'll need an account so I can muck around with the buildbot config
> as build machines are added, targets change, etc. Eventually that
> account should migrate to whomever is the overall validation coordinator
> (in the proposal I volunteered myself to play that role for now).

The LLVM server is not the place to host this. Could Cray host a machine
to do this?

I can certainly ask. I'll let you know what I hear but be mindful that if
someone else steps up to coordinate validations, Cray isn't going to give them
access to an internal machine. That means we'd have to revisit this question
in the future to find another home. I'd rather put this somewhere that is
under control of the wider LLVM community. I think that's more in the
interest of the LLVM community.

I'd offer my personal machine as a host if I had any confidence my DSL would
stay up. :-/

                                                  -Dave

Right now the buildbot is set up to build every N commits (N=10 for
testing purposes) and also build every Sunday at midnight. The buildbot
does not automatically tag successful validations. That will take some
more work and we don't want that automation quite yet anyway. We'll do
it manually for a while and then once we get the gist of it we can make
it automatic.

How exactly do you plan to validate it? What point of reference will you
be using as a start?

It's all in the proposal I posted a week or so ago.

Yes, your proposal said that the community must agree on the set of tests used to validate it. You haven't said what the conclusion was. Also, in your proposal you did not specify what reference point you were going to use. Not everything in llvm-test passes. I will suggest that you use a release to get a starting reference point.

One thing that would be nice to have is a commit hook on trunk (and
eventually release branches) that lets buildbot know when changes are
submitted. As of now, buildbot polls the svn server every 10 minutes,
meaning it could miss one of the "every 10" commits (revision mod 10 ==
0). The buildbot manual talks a bit about what the svn server needs to
do here:

http://buildbot.net/repos/release/docs/buildbot.html#PBChangeSource

Could we get that set up in the llvm.org svn commit hook?

What are the pros and cons of this? How much does it tax the server?

It's an extra TCP connection out to the buildbot on every commit. I would
think that would be low cost compared to all the other processing that has to
go on.

I'll have to talk to some people about this before committing to getting this done. So ping me later.

Eventually we'll want to migrate the buidlbot master to some publicly
accessible machine so others can contribute machines to do validations
(for various architectures, etc.). Where should this get hosted?
Hosting at llvm.org would be great but I'm not sure that's possible.
Daniel has a machine set up so that's a possibility too. Wherever it
goes, I'll need an account so I can muck around with the buildbot config
as build machines are added, targets change, etc. Eventually that
account should migrate to whomever is the overall validation coordinator
(in the proposal I volunteered myself to play that role for now).

The LLVM server is not the place to host this. Could Cray host a machine
to do this?

I can certainly ask. I'll let you know what I hear but be mindful that if
someone else steps up to coordinate validations, Cray isn't going to give them
access to an internal machine. That means we'd have to revisit this question
in the future to find another home. I'd rather put this somewhere that is
under control of the wider LLVM community. I think that's more in the
interest of the LLVM community.

This is exactly why we can't do it at LLVM. We can't give arbitrary people accounts.

-Tanya

> It's all in the proposal I posted a week or so ago.

Yes, your proposal said that the community must agree on the set of tests
used to validate it. You haven't said what the conclusion was. Also, in

True enough. My assumption is we'll figure that out soon. I'll post a
separate thread to start the discussion.

your proposal you did not specify what reference point you were going to
use. Not everything in llvm-test passes. I will suggest that you use a
release to get a starting reference point.

I'd rather not have to track individual regressions between releases. I'd
rather just pick a set of tests that passes now and add to that as more things
start working.

>> What are the pros and cons of this? How much does it tax the server?
>
> It's an extra TCP connection out to the buildbot on every commit. I
> would think that would be low cost compared to all the other processing
> that has to go on.

I'll have to talk to some people about this before committing to getting
this done. So ping me later.

Ok, thanks.

> I can certainly ask. I'll let you know what I hear but be mindful that
> if someone else steps up to coordinate validations, Cray isn't going to
> give them access to an internal machine. That means we'd have to revisit
> this question in the future to find another home. I'd rather put this
> somewhere that is under control of the wider LLVM community. I think
> that's more in the interest of the LLVM community.

This is exactly why we can't do it at LLVM. We can't give arbitrary people
accounts.

Fair enough. Can anyone else help out here?

                                          -Dave

Fair enough. Can anyone else help out here?

The gcc compile farm perhaps? http://gcc.gnu.org/wiki/CompileFarm

Ciao,

Duncan.

> What are the pros and cons of this? How much does it tax the server?

It's an extra TCP connection out to the buildbot on every commit. I would
think that would be low cost compared to all the other processing that has
to go on.

There's also a python script to run that actually makes the connection, so
there's some overhead there as well.

> > Eventually we'll want to migrate the buidlbot master to some publicly
> > accessible machine so others can contribute machines to do validations
> > (for various architectures, etc.). Where should this get hosted?
> > Hosting at llvm.org would be great but I'm not sure that's possible.
> > Daniel has a machine set up so that's a possibility too. Wherever it
> > goes, I'll need an account so I can muck around with the buildbot
> > config as build machines are added, targets change, etc.

Actually, that's too pessimistic. All I really need is a way to have a
buildbot master installed and a way to get a config file installed (and
buildbot reloaded if necessary). That means at minimum the validation
coordinator would need someone inside to help set things up and make updates
on his or her behalf.

We're still checking into possibilities here at Cray. I'll let you all know
when I head something.

                                             -Dave