new stage2/stage3 comparison failures

Are any of the buildbots set up to validate the stage2/stage3
binaries on a 3 stage bootstrap of llvm/compiler-rt/libc++/polly? On
x86_64-apple-darwin using the fink packaging scripts of David Fang, I
have found that at some point after r241513 the following differences
appear...

UNRESOLVED DIFFS (stage2 vs. stage3):
  lib/Analysis/CMakeFiles/LLVMAnalysis.dir/DependenceAnalysis.cpp.o
  lib/AsmParser/CMakeFiles/LLVMAsmParser.dir/LLLexer.cpp.o
  lib/Bitcode/Reader/CMakeFiles/LLVMBitReader.dir/BitcodeReader.cpp.o
  lib/Bitcode/Writer/CMakeFiles/LLVMBitWriter.dir/BitcodeWriter.cpp.o
  lib/CodeGen/CMakeFiles/LLVMCodeGen.dir/AggressiveAntiDepBreaker.cpp.o
  lib/CodeGen/CMakeFiles/LLVMCodeGen.dir/Analysis.cpp.o
  lib/CodeGen/CMakeFiles/LLVMCodeGen.dir/LiveDebugVariables.cpp.o
  lib/CodeGen/CMakeFiles/LLVMCodeGen.dir/MachineModuleInfo.cpp.o
  lib/CodeGen/CMakeFiles/LLVMCodeGen.dir/MachineScheduler.cpp.o
  lib/CodeGen/CMakeFiles/LLVMCodeGen.dir/RegAllocGreedy.cpp.o
  lib/CodeGen/CMakeFiles/LLVMCodeGen.dir/RegAllocPBQP.cpp.o
  lib/CodeGen/CMakeFiles/LLVMCodeGen.dir/RegisterScavenging.cpp.o
  lib/CodeGen/CMakeFiles/LLVMCodeGen.dir/SplitKit.cpp.o
  lib/CodeGen/CMakeFiles/LLVMCodeGen.dir/StackColoring.cpp.o
  lib/CodeGen/CMakeFiles/LLVMCodeGen.dir/TargetLoweringObjectFileImpl.cpp.o
  lib/IR/CMakeFiles/LLVMCore.dir/AsmWriter.cpp.o
  lib/IR/CMakeFiles/LLVMCore.dir/DIBuilder.cpp.o
  lib/IR/CMakeFiles/LLVMCore.dir/Function.cpp.o
  lib/IR/CMakeFiles/LLVMCore.dir/InlineAsm.cpp.o
  lib/MC/CMakeFiles/LLVMMC.dir/ELFObjectWriter.cpp.o
  lib/ProfileData/CMakeFiles/LLVMProfileData.dir/InstrProfReader.cpp.o
  lib/Support/CMakeFiles/LLVMSupport.dir/APFloat.cpp.o
  lib/Support/CMakeFiles/LLVMSupport.dir/APInt.cpp.o
  lib/Support/CMakeFiles/LLVMSupport.dir/CommandLine.cpp.o
  lib/Support/CMakeFiles/LLVMSupport.dir/ConvertUTFWrapper.cpp.o
  lib/Support/CMakeFiles/LLVMSupport.dir/DAGDeltaAlgorithm.cpp.o
  lib/Support/CMakeFiles/LLVMSupport.dir/FoldingSet.cpp.o
  lib/Support/CMakeFiles/LLVMSupport.dir/ScaledNumber.cpp.o
  lib/Support/CMakeFiles/LLVMSupport.dir/Statistic.cpp.o
  lib/Support/CMakeFiles/LLVMSupport.dir/Timer.cpp.o
  lib/TableGen/CMakeFiles/LLVMTableGen.dir/Record.cpp.o
  lib/Target/ARM/MCTargetDesc/CMakeFiles/LLVMARMDesc.dir/ARMUnwindOpAsm.cpp.o
  lib/Transforms/Instrumentation/CMakeFiles/LLVMInstrumentation.dir/AddressSanitizer.cpp.o
  lib/Transforms/Scalar/CMakeFiles/LLVMScalarOpts.dir/EarlyCSE.cpp.o
  lib/Transforms/Scalar/CMakeFiles/LLVMScalarOpts.dir/LoopStrengthReduce.cpp.o
  lib/Transforms/Vectorize/CMakeFiles/LLVMVectorize.dir/LoopVectorize.cpp.o
  projects/compiler-rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_activation.cc.o
  projects/compiler-rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_allocator.cc.o
  projects/compiler-rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_debugging.cc.o
  projects/compiler-rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_fake_stack.cc.o
  projects/compiler-rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_flags.cc.o
  projects/compiler-rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_globals.cc.o
  projects/compiler-rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_interceptors.cc.o
  projects/compiler-rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_linux.cc.o
  projects/compiler-rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_mac.cc.o
  projects/compiler-rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_malloc_linux.cc.o
  projects/compiler-rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_malloc_mac.cc.o
  projects/compiler-rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_malloc_win.cc.o
  projects/compiler-rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_new_delete.cc.o
  projects/compiler-rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_poisoning.cc.o
  projects/compiler-rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_posix.cc.o
  projects/compiler-rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_report.cc.o
  projects/compiler-rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_rtl.cc.o
  projects/compiler-rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_stack.cc.o
  projects/compiler-rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_stats.cc.o
  projects/compiler-rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_suppressions.cc.o
  projects/compiler-rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_thread.cc.o
  projects/compiler-rt/lib/asan/CMakeFiles/RTAsan.osx.dir/asan_win.cc.o
  projects/compiler-rt/lib/interception/CMakeFiles/RTInterception.osx.dir/interception_linux.cc.o
  projects/compiler-rt/lib/interception/CMakeFiles/RTInterception.osx.dir/interception_mac.cc.o
  projects/compiler-rt/lib/interception/CMakeFiles/RTInterception.osx.dir/interception_type_test.cc.o
  projects/compiler-rt/lib/interception/CMakeFiles/RTInterception.osx.dir/interception_win.cc.o
  projects/compiler-rt/lib/lsan/CMakeFiles/RTLSanCommon.osx.dir/lsan_common.cc.o
  projects/compiler-rt/lib/lsan/CMakeFiles/RTLSanCommon.osx.dir/lsan_common_linux.cc.o
  projects/compiler-rt/lib/safestack/CMakeFiles/clang_rt.safestack_osx.dir/safestack.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_allocator.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_common.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_common_libcdep.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_coverage_libcdep.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_coverage_mapping_libcdep.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_deadlock_detector1.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_deadlock_detector2.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_flag_parser.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_flags.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_libc.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_libignore.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_linux.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_linux_libcdep.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_mac.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_persistent_allocator.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_platform_limits_linux.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_platform_limits_posix.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_posix.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_posix_libcdep.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_printf.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_procmaps_common.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_procmaps_freebsd.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_procmaps_linux.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_procmaps_mac.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_stackdepot.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_stacktrace.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_stacktrace_libcdep.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_stacktrace_printer.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_stoptheworld_linux_libcdep.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_suppressions.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_symbolizer.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_symbolizer_libbacktrace.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_symbolizer_libcdep.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_symbolizer_mac.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_symbolizer_posix_libcdep.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_symbolizer_process_libcdep.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_symbolizer_win.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_thread_registry.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_tls_get_addr.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_unwind_linux_libcdep.cc.o
  projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.osx.dir/sanitizer_win.cc.o
  projects/compiler-rt/lib/ubsan/CMakeFiles/RTUbsan.osx.dir/ubsan_diag.cc.o
  projects/compiler-rt/lib/ubsan/CMakeFiles/RTUbsan.osx.dir/ubsan_flags.cc.o
  projects/compiler-rt/lib/ubsan/CMakeFiles/RTUbsan.osx.dir/ubsan_handlers.cc.o
  projects/compiler-rt/lib/ubsan/CMakeFiles/RTUbsan.osx.dir/ubsan_handlers_cxx.cc.o
  projects/compiler-rt/lib/ubsan/CMakeFiles/RTUbsan.osx.dir/ubsan_init.cc.o
  projects/compiler-rt/lib/ubsan/CMakeFiles/RTUbsan.osx.dir/ubsan_type_hash.cc.o
  projects/compiler-rt/lib/ubsan/CMakeFiles/RTUbsan.osx.dir/ubsan_type_hash_itanium.cc.o
  projects/compiler-rt/lib/ubsan/CMakeFiles/RTUbsan.osx.dir/ubsan_type_hash_win.cc.o
  projects/compiler-rt/lib/ubsan/CMakeFiles/RTUbsan.osx.dir/ubsan_value.cc.o
  projects/compiler-rt/lib/ubsan/CMakeFiles/RTUbsan_standalone.osx.dir/ubsan_init_standalone.cc.o
  projects/openmp/runtime/src/CMakeFiles/omp.dir/kmp_version.c.o
  projects/openmp/runtime/src/kmp_i18n_default.inc
  projects/openmp/runtime/src/kmp_i18n_id.inc
  tools/clang/lib/AST/CMakeFiles/clangAST.dir/ASTContext.cpp.o
  tools/clang/lib/AST/CMakeFiles/clangAST.dir/ItaniumMangle.cpp.o
  tools/clang/lib/AST/CMakeFiles/clangAST.dir/MicrosoftMangle.cpp.o
  tools/clang/lib/Analysis/CMakeFiles/clangAnalysis.dir/ThreadSafety.cpp.o
  tools/clang/lib/Basic/CMakeFiles/clangBasic.dir/Diagnostic.cpp.o
  tools/clang/lib/CodeGen/CMakeFiles/clangCodeGen.dir/CGStmt.cpp.o
  tools/clang/lib/CodeGen/CMakeFiles/clangCodeGen.dir/MicrosoftCXXABI.cpp.o
  tools/clang/lib/Driver/CMakeFiles/clangDriver.dir/Tools.cpp.o
  tools/clang/lib/Frontend/Rewrite/CMakeFiles/clangRewriteFrontend.dir/RewriteModernObjC.cpp.o
  tools/clang/lib/Frontend/Rewrite/CMakeFiles/clangRewriteFrontend.dir/RewriteObjC.cpp.o
  tools/clang/lib/Lex/CMakeFiles/clangLex.dir/HeaderSearch.cpp.o
  tools/clang/lib/Sema/CMakeFiles/clangSema.dir/SemaOverload.cpp.o
  tools/clang/lib/Serialization/CMakeFiles/clangSerialization.dir/ASTReader.cpp.o
  tools/clang/lib/Serialization/CMakeFiles/clangSerialization.dir/ASTReaderDecl.cpp.o
  tools/clang/lib/Serialization/CMakeFiles/clangSerialization.dir/ASTWriter.cpp.o
  tools/clang/lib/Serialization/CMakeFiles/clangSerialization.dir/ASTWriterDecl.cpp.o
  tools/clang/lib/Serialization/CMakeFiles/clangSerialization.dir/ASTWriterStmt.cpp.o
  tools/clang/lib/Serialization/CMakeFiles/clangSerialization.dir/GlobalModuleIndex.cpp.o
  tools/clang/tools/extra/clang-tidy/CMakeFiles/clangTidy.dir/ClangTidy.cpp.o
  tools/clang/utils/TableGen/CMakeFiles/clang-tblgen.dir/ClangAttrEmitter.cpp.o
  tools/clang/utils/TableGen/CMakeFiles/clang-tblgen.dir/NeonEmitter.cpp.o
  tools/llvm-config/BuildVariables.inc
  tools/llvm-config/CMakeFiles/llvm-config.dir/llvm-config.cpp.o
  tools/llvm-readobj/CMakeFiles/llvm-readobj.dir/ARMAttributeParser.cpp.o
  tools/polly/lib/CMakeFiles/Polly.dir/Analysis/ScopInfo.cpp.o
  tools/polly/lib/CMakeFiles/Polly.dir/External/isl/imath/gmp_compat.c.o
  tools/polly/lib/CMakeFiles/Polly.dir/External/isl/isl_polynomial.c.o
  tools/polly/lib/CMakeFiles/Polly.dir/JSON/json_reader.cpp.o
  tools/polly/lib/CMakeFiles/Polly.dir/JSON/json_writer.cpp.o
  utils/TableGen/CMakeFiles/llvm-tblgen.dir/AsmMatcherEmitter.cpp.o
  utils/TableGen/CMakeFiles/llvm-tblgen.dir/AsmWriterEmitter.cpp.o
  utils/TableGen/CMakeFiles/llvm-tblgen.dir/AsmWriterInst.cpp.o
  utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeEmitterGen.cpp.o
  utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeGenDAGPatterns.cpp.o
  utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeGenRegisters.cpp.o
  utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeGenSchedule.cpp.o
  utils/TableGen/CMakeFiles/llvm-tblgen.dir/CodeGenTarget.cpp.o
  utils/TableGen/CMakeFiles/llvm-tblgen.dir/FixedLenDecoderEmitter.cpp.o
  utils/TableGen/CMakeFiles/llvm-tblgen.dir/InstrInfoEmitter.cpp.o
  utils/TableGen/CMakeFiles/llvm-tblgen.dir/RegisterInfoEmitter.cpp.o
/sw/src/fink.build/llvm37-3.7.0-1/llvm-3.7.0.src
######## 3-STAGE BOOTSTRAP of llvm/clang FAILED

It been hard to pinpoint that exact point of regression because there
have been so many other bootstrap failures introduced and resolved in
between that commit and current trunk. Hopefully one of the buildbots
has been observing these as well.
           Jack

Hi Jack,

r241673 introduced some non-determinism which could account for what you are seeing, and should have been fixed in r241809.

This was seen by at least one buildbot:
http://bb.pgr.jp/builders/clang-3stage-i686-linux/builds/2726

Thanks,
Silviu