Buildbot try builds to get pre-commit performance number

Hi,

as we got again more activity regarding performance testing and especially LNT, I wanted to get out an idea I had since a while, but where I neither had nor have the time or hardware to implement it.

The buildbot infrastructure provides so called 'try' builds, that allow a developer to submit patches they want to check for correctness or performance before actually committing them.

It would be nice to enable such a feature for our developers using our buildbot and LNT infrastructure. With the current -O3 builders, we should have enough hardware resources to draft a first version of this. So in case anybody is interested to try to set this up, that would be amazing and I am happy to discuss ideas here.

Also, with the LLVM Foundation being set up, my hope is that we get at some point a larger set of recent machines which will allow us to both increase the bandwidth for such a feature as well as the reliability of the performance results.

Cheers,
Tobias

Hi Tobi,

I like this idea very much, but I worry about possible abuses of web
input. Is this feature a patch upload form?

If they are, it'd help to have a git project (say on github, so we can
have more control), where those buildbots should pull the code from
(given specific branch names), so they can only build from an already
merged patchset and allow for more complex merges, too.

cheers,
--renato

http://docs.buildbot.net/0.8.1/try.html

We could probably also make the buildbot pull from a git repository. One of the issues could be that we currently have different SVN repositories that get checked out.

Cheers,
Tobias

Yes, that brings complexity on all building systems... I'm changing
one right now to accept it... :frowning:

Any change we do to this scheme is bound to create confusion, so I'd
recommend we use github repos and accept merges from people into
specific branches.

The script I'm changing has dependencies check, so if you ask to build
"clang" it'll checkout "llvm" first. If you ask for
"clang-tools-extra" it'll checkout "clang" first, and so on.

To deal with multi-repo changes, one could look into the dependencies
with the same branch name, and only if there isn't any, use "master".
So, if you change both clang and llvm, create a branch "tobi" on both,
and the bot will build both. If you only change Clang, the bot will
check out "tobi" on clang and "master" on llvm.

That's how I do my local git build...

cheers,
--renato

I really love this idea!

LNT has great functionality for A/B comparison runs, which can send you a nice report email. I have been discussing with a few people how we could make it easier to try out performance patches on random hardware.

I don’t know if using buildbot’s try feature would be compatible with the current buildbot pipeline?