LLDB OS X buildbot/testbot details

Hi all,

The llvm.org Green Dragon (i.e. Jenkins-based) LLDB OS X buildbot/testbot has received some improvements today. The Jenkins build now uses the xUnit plugin to process xUnit-based test suite results, which are displayed more usefully on the “build and test” page here.

A few notes on some relevant details:

  • The summary page allows you to click the test details and see the pass/fail and timing info for each of the tests

  • If the test is in failure/error, the backtrace will show up in the details for the test method when you drill through the test results.

  • The test summary information that you would normally see in the output of a local test run is available in the bottom of the “Console Output” page for a given build. This can be useful as xUnit/JUnit only really has the concept of a pass/fail/error, but we also have timeouts, unexpected successes, exceptional exits, etc. that all have to get mapped to the JUnit pass/fail/error model. The JUnit test method run details try to capture this, but I find it useful from time to time to go to the consule output to see the “real” results. (Rerun info is only available on the Console Output).

  • The Jenkins jobs are set up in a 2-level job/sub-job structure. The link I sent above is the most useful one to look at because it does the build and test phase, and that covers 95% of what you want to look at. BUT, if you want to check what code is actually contained in the build and test run, you need to go to the parent job (will be listed under here, which lists the main job), then find its first sub-job called “Acquire Sources”. That will tell you which changes were synched for this job. (I may do something about this in the future since the workflow on this seems suboptimal, but this is how it is right now and is similar for other LLVM projects on Jenkins).

  • Right now the gtests are built, but not run. I’ll be addressing this as soon as Zachary and I straighten out a failing gtest on OS X. Those test results will just show up in the Console Output for the build and test phase, and will fail the build if they either fail to build or have a test run failure.

  • Currently the equivalent of the python test session directory is not collected and archived. I plan to remedy that in the future. I currently do archive the JUnit.xml output file from the XunitResultsFormatter output.

Let me know if you have any questions or feedback.