I have a buildbot operating to do validating builds of llvm up at
My DSL has been stable enough for the past few months for me to
feel comfortable hosting the buildbot there.
It's not yet sending messages to llvmdev. I want to do some more
testing of the setup before I turn it loose on everyone. But you can
go take a look to see how it operates.
-Dave
I have a buildbot operating to do validating builds of llvm up at
My DSL has been stable enough for the past few months for me to
feel comfortable hosting the buildbot there.
We had a discussion in the past on what validate means. Did you ever formalize that? It might be good if you posted (on your website?) what specific criteria you are using to declare a build validated. Or is this just a normal build bot?
It's not yet sending messages to llvmdev. I want to do some more
testing of the setup before I turn it loose on everyone. But you can
go take a look to see how it operates.
I don't think llvm-dev is the right place to be sending mail to. Maybe the testresults list? What mails do you plan to send and how frequent?
-Tanya
> I have a buildbot operating to do validating builds of llvm up at
>
> http://obbligato.org:8080
>
> My DSL has been stable enough for the past few months for me to
> feel comfortable hosting the buildbot there.We had a discussion in the past on what validate means. Did you ever
formalize that? It might be good if you posted (on your website?) what
specific criteria you are using to declare a build validated. Or is
this just a normal build bot?
We had a long discussion about this. I'll post some information but
the buildbot essentially does this:
- Build an LLVM without llvm-gcc
- Run LLVM tests
- Build llvm-gcc pointing to the newly-build LLVM
- Rebuild LLVM pointing to the newly-build llvm-gcc
- Run LLVM tests
- Run llvm-test
If everything passes for debug, release and paranoid
(--enable-expensive-checks) we'll consider LLVM validated
for that target.
> It's not yet sending messages to llvmdev. I want to do some more
> testing of the setup before I turn it loose on everyone. But you can
> go take a look to see how it operates.I don't think llvm-dev is the right place to be sending mail to. Maybe
the testresults list? What mails do you plan to send and how frequent?
The buildbot kicks off every 100 commits or so. There are three builds for
each target (the only buildslaves we have right now are for x86_64-linux and
i686-linux). Each one of those will send an e-mail.
I'm fine sending it to testresults if people pay attention. I know that I
don't read testresults regularly because there are a lot of test runs I
don't care about.
The whole point of the validation process is to identify bugs quickly so
they get fixed quickly and we keep llvm stable. It means people will
have to monitor it and react when stuff doesn't work.
-Dave
If you create a slave name and password for me, i'm happy to put one
of the ubuntu 8.04 8 core machines i have running it. They are
x86_64-linux
(you would need to add -j8, which can be done through properties easily)
Also, if you want to put the buildbot status with the other buildbots
we have on google1.osuosl.org, i'm happy to add your config to the
master
I have a buildbot operating to do validating builds of llvm up at
My DSL has been stable enough for the past few months for me to
feel comfortable hosting the buildbot there.We had a discussion in the past on what validate means. Did you ever
formalize that? It might be good if you posted (on your website?) what
specific criteria you are using to declare a build validated. Or is
this just a normal build bot?We had a long discussion about this. I'll post some information but
the buildbot essentially does this:- Build an LLVM without llvm-gcc
- Run LLVM tests
- Build llvm-gcc pointing to the newly-build LLVM
- Rebuild LLVM pointing to the newly-build llvm-gcc
- Run LLVM tests
- Run llvm-testIf everything passes for debug, release and paranoid
(--enable-expensive-checks) we'll consider LLVM validated
for that target.
As I mentioned before, I'm curious what reference point you are using to determine "pass" for llvm-test.
It's not yet sending messages to llvmdev. I want to do some more
testing of the setup before I turn it loose on everyone. But you can
go take a look to see how it operates.I don't think llvm-dev is the right place to be sending mail to. Maybe
the testresults list? What mails do you plan to send and how frequent?The buildbot kicks off every 100 commits or so. There are three builds for
each target (the only buildslaves we have right now are for x86_64-linux and
i686-linux). Each one of those will send an e-mail.
I'm fine sending it to testresults if people pay attention. I know that I
don't read testresults regularly because there are a lot of test runs I
don't care about.
I still don't think llvm-dev is the right place. We don't want that mailing list to get cluttered with buildbot test results. You can send to llvm-test and use filtering if you want to ignore the other results.
Thanks,
-Tanya
I have a buildbot operating to do validating builds of llvm up at
My DSL has been stable enough for the past few months for me to
feel comfortable hosting the buildbot there.We had a discussion in the past on what validate means. Did you ever
formalize that? It might be good if you posted (on your website?)
what
specific criteria you are using to declare a build validated. Or is
this just a normal build bot?We had a long discussion about this. I'll post some information but
the buildbot essentially does this:- Build an LLVM without llvm-gcc
- Run LLVM tests
- Build llvm-gcc pointing to the newly-build LLVM
- Rebuild LLVM pointing to the newly-build llvm-gcc
- Run LLVM tests
- Run llvm-testIf everything passes for debug, release and paranoid
(--enable-expensive-checks) we'll consider LLVM validated
for that target.As I mentioned before, I'm curious what reference point you are using
to determine "pass" for llvm-test.
Here's my idea for this:
- The first criteria is "does it compile and run without regressions from the last run?".
- The second criteria is "does it run significantly slower than the previous run.
- Defining what is "significantly slower" is the hard part here. Some statistical analysis needs to be done on the data; more than just simple ratios. (I'd like to see these analyses done on our nightly tests as well.) They can tell us if: A) the change is significant, and B) if we're gradually regressing over time.
The second criteria is *much* more difficult, of course.
It's not yet sending messages to llvmdev. I want to do some more
testing of the setup before I turn it loose on everyone. But you
can
go take a look to see how it operates.I don't think llvm-dev is the right place to be sending mail to.
Maybe
the testresults list? What mails do you plan to send and how
frequent?The buildbot kicks off every 100 commits or so. There are three
builds for
each target (the only buildslaves we have right now are for x86_64-
linux and
i686-linux). Each one of those will send an e-mail.
I'm fine sending it to testresults if people pay attention. I know
that I
don't read testresults regularly because there are a lot of test
runs I
don't care about.I still don't think llvm-dev is the right place. We don't want that
mailing list to get cluttered with buildbot test results. You can send
to llvm-test and use filtering if you want to ignore the other results.
I think that "testresults" is fine for now. That's where the other buildbots send their results.
-bw
Does that mean I wouldn't be running a buildbot master on my machine?
-Dave
I'm not sure performance regressions should be a requirement for validation.
Validation is aobut functional correctness. It gives developers confidence
that they can update to a particular revision without breaking anything.
Performance regression is important, no doubt, but I'd make it a condition
for release, not validation.
-Dave
Thinking about this some more, I could make a case that performance
regressions might fit into another type of validation - performance
validation. We could have two types of validation: functional and
performance. People could individually decide which they care about.
The key is to keep things simple for buildbot. We don't want to write a
bunch of complex test code in buildbot to do the performance
checks / statistical analysis Bill is talking about. Buildbot wants to
see PASS / FAIL / XFAIL, etc. So I think we'd need a nice DejaGNU
suite that does performance regression testing. To my knowledge, we
don't have one yet. So I'm going to leave it to others to develop such
a suite and if and when one pops up we can look at integrating it into
the buildbot validator.
We're all learning here and I know that as we go along we'll tweak the way
this works. I'm not worried about that. 
-Dave