intermittent tests - moving to XFAIL rather than skip

Hey all,

I had been operating under the assumption that a test marked XFAIL (expected failure) that passes (which gets categorized as an “unexpected success”) would fail the test run. I have recently learned in the last few days that this is not true - the unexpected success does get captured in the logs and does show up as a ‘u’ in the test run, but it doesn’t cause the entire test process exit code to fail in the way that one or more failed tests would.

Given that, I’m going to flip the tests that were intermittent to be XFAIL instead of skip. This allows the tests to still run, allowing the code to run, and enabling us to catch (as an “Error”) if the test actually seg faults or something. All things being equal, I’d prefer to get notice this way over hiding the unexpected success. (Note we have a different issue in the test runner where we don’t always get something useful when we segfault, but I’ve already filed a bug on that).

I’d be in favor of doing more to track the unexpected successes — if we have tests marked as XFAIL that have since been fixed and always succeed, it would be good to flip them from XFAIL to normal test status that is expected to pass. But we can address that later, maybe with buildbots that can track those over time and start generating “potentially passing XFAILS” reports if they pass all of the last x runs (with some largish x).

I’ll go ahead and flip the intermittent skipped tests that I changed recently to XFAIL (generally on MacOSX and Linux) to adhere to this idea.

Done, r212762.

Hi Todd,

I noticed on the FreeBSD buildbot that the run reports a failure for
unexpected passes. I agree with your change though and will switch
the FreeBSD tests that are being skipped to XFAIL as well. If that
causes grief on the buildbots we can look at having them accept
unexpected passes.

Okay, thanks Ed!

Sorry for any churn on your buildbot there.

-Todd

Sorry for any churn on your buildbot there.

To be clear, I don't think you introduced any churn on the FreeBSD
buildbot with these changes. The issue already existed.

Oh - I was referring to having flipped intermittent failures (skipped before) to XFAIL, introducing some probable unexpected passes, which would then fail the FreeBSD build bot (maybe in the future) until you fiddle with the buildbot’s test run failure condition logic.

So it’s a “sorry for the hassle in the future” - in advance :wink:

Ah, I thought the change was just skipIfLinux -> expectedFailLinux.
If there are some cases that changed from a global skip to a global
xfail then yes they'll trip up the buildbot. Anyway I agree it's the
right change to make, and fiddling with the buildbot will be the right
course of action.

Great!