New OS X and Jenkins based build cluster

I am happy to report our new Jenkins based build cluster is up and running:

http://lab.llvm.org:8080/green/

This cluster is a major hardware update. The cluster has 10 builders + the Jenkins master. All builders have SSDs and enough RAM to do LTO builds of clang.

Currently the cluster has:

  • a cmake + ninja incremental builder
  • a configure + make stage 1 builder
  • a configure + make stage 2 builder with LTO
  • a O3 LTO performance tracker

In the near future other jobs will be added:

  • LLDB
  • O0 -g performance tracker
  • libcxx tester
  • configure + make stage 3 build

When I am convinced things are working, I will enabled emails to developers, and the IRC bot.

This cluster is a replacement for the old Darwin buildbots here:

http://lab.llvm.org:8013/builders

I am happy to report our new Jenkins based build cluster is up and running:

http://lab.llvm.org:8080/green/

This cluster is a major hardware update. The cluster has 10 builders + the
Jenkins master. All builders have SSDs and enough RAM to do LTO builds of
clang.

Currently the cluster has:

   - a cmake + ninja incremental builder
   - a configure + make stage 1 builder
   - a configure + make stage 2 builder with LTO
   - a O3 LTO performance tracker

In the near future other jobs will be added:

   - LLDB
   - O0 -g performance tracker

Would this potentially include size tracking, too? (little harder to do if
it's self-hosting, since the newer compiler would have more/different code,
thus changing the size of debug info - so we might need some immutable code
to do this analysis on (even just a predefined version of LLVM would be an
option))

I am happy to report our new Jenkins based build cluster is up and running:

Nice!

This cluster is a major hardware update. The cluster has 10 builders + the
Jenkins master. All builders have SSDs and enough RAM to do LTO builds of
clang.

Out of curiosity, what is that amount of RAM at the moment?

Cheers.

Tim.

I am happy to report our new Jenkins based build cluster is up and running:

Nice!

This cluster is a major hardware update. The cluster has 10 builders + the
Jenkins master. All builders have SSDs and enough RAM to do LTO builds of
clang.

Out of curiosity, what is that amount of RAM at the moment?

16GB. 16GB seems to not swap for a LTO build with -j 2.

So, I’m really unhappy with the proliferation of build masters. I used to have a single webpage to go to and see whether I broke builders. I don’t even know how many I have now…

Is there a concrete plan to get back to a single build master? The Jenkins page you pointed me too seems completely OS X specific… =[

Out of the box, there is no size tracking on the NT performance suite. This is our standard performance test suite. We do have some code for reporting generated executable size in other parts of LNT though. Patches welcome to expand that to the NT test suite.

Hi Chris,

We're also using Jenkins for LLVM testing, but our setup is a lot more primitive than this! Do you plan on sharing the configuration scripts and set of plugins that you're using?

David

So, I'm really unhappy with the proliferation of build masters. I used to have a single webpage to go to and see whether I broke builders. I don't even know how many I have now....

While I agree that having a single page would be really nice; technically, the old buildbots that the new Jenkins system replaces (http://lab.llvm.org:8013/) never showed up on http://lab.llvm.org:8011/ either.

-- adrian

I can see why that would annoy you. The intent here was to beef up the OS X testing, which had become really weak. This was a lateral move, we replaced an old build master that was OS X with a new build master which is OS X.

At a technical level, there are lots of ways we could have a unified view over all of the CI systems. We need to sort out the who and what of project like that.

Yes! I should have mentioned this in the email. The code will go into zorg soon.

Chris,

What are the limitations of zorg/BuildBot that caused you to go with Jenkins
instead?

Jon

We had a bit of a discussion about that in this thread:

http://lists.cs.uiuc.edu/pipermail/llvmdev/2014-August/075708.html

Yes, I understand this didn't make anything worse, but it doesn't really
look like (yet) converging. It seems like it would maybe have been better
to discuss with folks whether Jenkins was the best way to go or instead the
existing zorg/buildbot infrastructure should be adapted to your needs. =[
Now the difference is larger and it looks like it will be harder to
converge again.

Would it be possible to archive installations and provide (registered)
developers access for fast bisection? Personally, I tend to need
bisections at least a hand full of times per month. With the hardware I
have at hand, that normally needs 3h+. Being able to do an initial pass
to cut it down e.g. to 32 revisions would greatly reduce the amount of
time as there is hopefully only one major API change in such a time
frame.

Joerg

There was such a discussion:
http://lists.cs.uiuc.edu/pipermail/llvmdev/2014-August/075708.html

Yea, I just missed it... and there doesn't really seem to be any path
toward convergence there either. =[

Yes, this is a great idea, and something I have already looked at a little bit. Two things we need: a good local bisection client, and some place to store the artifacts that is high bandwidth and available (I don’t think it is a good idea to host them from the lab itself). The compilers are 60mb-90mb each.