PSA: debuginfo-tests workflow changing slightly

It looks like this broke green dragon:

http://green.lab.llvm.org/green/job/clang-stage1-configure-RA/40383/console

llvm-lit: /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/llvm/projects/libcxx/utils/libcxx/test/config.py:173: note: Adding environment variables: {'DYLD_LIBRARY_PATH': '/Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/clang-build/./lib', 'LIBCXX_FILESYSTEM_DYNAMIC_TEST_ROOT': '/Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/clang-build/projects/libcxx/test/filesystem/Output/dynamic_env'}

llvm-lit: /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/llvm/utils/lit/lit/llvm/config.py:332: note: using clang: /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/clang-build/bin/clang
llvm-lit: /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/llvm/utils/lit/lit/util.py:379: note: using SDKROOT: '/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk'

llvm-lit: /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/llvm/utils/lit/lit/TestingConfig.py:101: fatal: unable to parse config file '/Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/llvm/tools/clang/test/debuginfo-tests/lit.cfg.py', traceback: Traceback (most recent call last):
  File "/Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/llvm/utils/lit/lit/TestingConfig.py", line 88, in load_from_path
    exec(compile(data, path, 'exec'), cfg_globals, None)
  File "/Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/llvm/tools/clang/test/debuginfo-tests/lit.cfg.py", line 36, in <module>
    config.test_source_root = os.path.join(config.debuginfo_tests_src_root, 'tests')
AttributeError: TestingConfig instance has no attribute 'debuginfo_tests_src_root'

FAILED: CMakeFiles/check-all

-- adrian

I checked in a fix for that already, sorry for the trouble. I’m waiting for it to cycle

I checked in a fix for that already, sorry for the trouble. I’m waiting for it to cycle

awesome. Thanks!

– adrian

Wasn’t quite fixed, but it got a lot further this time. This time there was still an issue in the test_debuginfo.pl script regarding a hardcoded path to the llgdb.py script. I think I never encountered this locally because this codepath only happens on Darwin, and I was testing on Linux.

(As an aside, ugh… Perl…)

Regardless, this should be fixed in r317949, and hopefully that’s the last of the issues. I have to run for a couple of hours, but I can check on this again in a bit. But I strongly suspect it will be fixed now.

It looks like the bots are still red?

— Adrian

I might be missing something, but this doesn’t look like me?

http://green.lab.llvm.org/green/job/clang-stage1-configure-RA/40478/consoleFull#-42777206a1ca8a51-895e-46c6-af87-ce24fa4cd561

PASS: debuginfo-tests :: dbg-arg.c (34886 of 40729) PASS: debuginfo-tests :: ctor.cpp (34887 of 40729) PASS: debuginfo-tests :: ctor.cpp (34888 of 40729) PASS: debuginfo-tests :: aggregate-indirect-arg.cpp (34889 of 40729) PASS: debuginfo-tests :: aggregate-indirect-arg.cpp (34890 of 40729) PASS: debuginfo-tests :: dbg-arg.c (34891 of 40729) PASS: debuginfo-tests :: asan.c (34892 of 40729) PASS: debuginfo-tests :: asan.c (34893 of 40729) PASS: debuginfo-tests :: asan-blocks.c (34894 of 40729) PASS: debuginfo-tests :: asan-blocks.c (34895 of 40729) PASS: debuginfo-tests :: nested-struct.cpp (34896 of 40729) PASS: debuginfo-tests :: nested-struct.cpp (34897 of 40729) PASS: debuginfo-tests :: forward-declare-class.cpp (34898 of 40729) PASS: debuginfo-tests :: forward-declare-class.cpp (34899 of 40729) FAIL: debuginfo-tests :: foreach.m (34900 of 40729) ******************** TEST ‘debuginfo-tests :: foreach.m’ FAILED ******************** Script: – /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/clang-build/bin/clang --target=x86_64-apple-darwin15.6.0 -O0 -g /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/llvm/tools/clang/test/debuginfo-tests/tests/foreach.m -c -o /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/clang-build/tools/clang/test/debuginfo-tests/Output/foreach.m.tmp.o /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/clang-build/bin/clang --target=x86_64-apple-darwin15.6.0 /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/clang-build/tools/clang/test/debuginfo-tests/Output/foreach.m.tmp.o -o /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/clang-build/tools/clang/test/debuginfo-tests/Output/foreach.m.tmp.out -framework Foundation /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/llvm/utils/test_debuginfo.pl /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/llvm/tools/clang/test/debuginfo-tests/tests/foreach.m /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/clang-build/tools/clang/test/debuginfo-tests/Output/foreach.m.tmp.out – Exit Code: 1 Command Output (stdout): – Debugger output was: imported lldb from: “/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Versions/A/Resources/Python” error: foreach.m.tmp.out debug map object file ‘/Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/clang-build/tools/clang/test/debuginfo-tests/Output/foreach.m.tmp.o’ has changed (actual time is 0x5a0a2528, debug map time is 0x5a0a2526) since this executable was linked, file will be ignored > break 25 SBBreakpoint: id = 1, file = ‘’, line = 25, exact_match = 0, locations = 0 > r success > po thing = > quit – Command Output (stderr): – /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/llvm/tools/clang/test/debuginfo-tests/tests/foreach.m:11:11: error: expected string not found in input // CHECK: aaa ^ /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/clang-build/tools/clang/test/debuginfo-tests/Output/foreach.m.gdb.output:1:1: note: scanning from here imported lldb from: “/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Versions/A/Resources/Python” ^ /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/clang-build/tools/clang/test/debuginfo-tests/Output/foreach.m.gdb.output:2:211: note: possible intended match here error: foreach.m.tmp.out debug map object file ‘/Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/clang-build/tools/clang/test/debuginfo-tests/Output/foreach.m.tmp.o’ has changed (actual time is 0x5a0a2528, debug map time is 0x5a0a2526) since this executable was linked, file will be ignored

On the other hand this file hasn’t changed recently, but I have no way to test this as it uses the LLDB code path, which only runs on OSX.

The first build where a test fails with similar symptoms has just one blamelist entry:

http://green.lab.llvm.org/green/job/clang-stage1-configure-RA/40391/

  1. Update test_debuginfo.pl script to point to new tree location. (detailViewSVN
    by zturner
    – adrian

Since this is causing all of our internal CI to back up, could you please revert your two changes, so we can make sure that they were actually responsible, and can work on a fix for this? Let me know how we can help investigate this.

– adrian

Would you or someone else be able to help me figure out what’s wrong? I don’t have a Mac, so I can’t reproduce this. It’s strange that all the other tests pass

Yea I’m preparing a revert right now. Does it happen for you when you run debuginfo-tests locally?

Yes I can reproduce this locally. It looks like we are not passing an -isysroot (pointing to the SDK) to clang but it isn’t clear what lit magic would expand this.

– adrian

Ha! Found it. *Somebody* is setting an SDKROOT variable in the environment. Can you find the code that would do this?

— adrian

Yea I also just found it. Try adding this code in the bottom of debuginfo-tests/lit.cfg.py

lit.util.usePlatformSdkOnDarwin(config, lit_config)

I can confirm that that fixes the issue!

— adrian

Great! It’s close to the end of the day, so I’ll submit tomorrow to make sure everything has a chance to go fully green again to ensure I get failure emails if it breaks.

Just an update,

At this point I’ve submitted and reverted this probably 6+ times, and I don’t have time to be blocked by this anymore.

I’m looking into alternatives for now, including ways of creating a new top-level git repo such as https://git.llvm.org/git/codeview-tests.git/.

I wish there were another way, but at this point I’ve been working on appeasing this for close to 2 months and I need some control over the bits that determine whether I’m able to make forward progress.

If and when someone on the Apple side has some time to make this work with their build infrastructure, I will be happy to revisit.

Just an update,

At this point I’ve submitted and reverted this probably 6+ times, and I don’t have time to be blocked by this anymore.

I’m looking into alternatives for now, including ways of creating a new top-level git repo such as https://git.llvm.org/git/codeview-tests.git/.

FWIW, I don’t think that is a really great path forward…

I wish there were another way, but at this point I’ve been working on appeasing this for close to 2 months and I need some control over the bits that determine whether I’m able to make forward progress.

If and when someone on the Apple side has some time to make this work with their build infrastructure, I will be happy to revisit.

I think that the Apple folks need to either step up to fix the issue they have, explain exactly how Zach can fix it, or we need to land this and let it get cleaned up on Apple’s side. This seems like it is starting to hold up progress on what was a discussed and reasonable approach due to internal infrastructure issues that folks outside of Apple cannot reasonably help address.

In particular, Zach has created a way to minimize the impact on Apple’s CI system, and there are just bugs or challenges in getting that to work cleanly. It really seems like if Apple has a CI challenge here, they should step up to help address it and get this moving forward.

My two cents…

Hi Zackary,

Did you see my followup to you cfe-commits rollback:

http://lists.llvm.org/pipermail/cfe-commits/Week-of-Mon-20171120/210212.html

I think you can just remove the change to cfe/trunk/test/CMakeLists.txt and it should work just fine.

hth…
don

That change was added specifically to workaround a failure in a previous version of the commit. I think as chandler says it’s pretty much impossible for me to make any progress unless someone whose build bot is failing actually sits down, applies the patch, makes it work, and then sends me back the result.