libclang c API will not find c++ system headers

Hi

I wanted to sort out a bug/limitation in a color coding plugin for neovim (not maintained by me), and tried to set up a vanilla c++ project to find out how libclang works. This is what I've done:

1) Download prebuilt binaries of clang+llvm-8.0.0 for macOS. I later downloaded and tried 7.0.0 as well
2) Create command line tool project in Xcode
3) Link to libclang.dylib under Linked Frameworks and Libraries
4) Add /path/to/clang+llvm/lib under Library search paths
5) Add /path/to/clang+llvm/include under Header search paths
6) Add /path/to/clang+llvm/lib under runpath search paths
7) Create new c++ target with only the auto-generated code to test parsing
8) Build the new target from command line, and copy the clang invocation from the build log
9) Run regex replace on the clang invocation to be sure I have an exact copy of the command line args during build of the test target, as an array of c strings (const char *)
10) Call clang_parseTranslationUnit2 with those parameters, with and without argv[0] (the executable path).

This is my main implementation:

int main(int argc, const char * argv) {
CXIndex idx = clang_createIndex(0,1);
CXTranslationUnit tu;

CXErrorCode errorCode;

if\( \!\( errorCode = clang\_parseTranslationUnit2\( idx,
                                NULL,
                                options,
                                opt\_count,
                                NULL,
                                0,
                                0,
                                &tu\) \) \)

    std::cout << "Success\\n";
else
    std::cout << "error: " << errorCode << std::endl;

clang\_disposeTranslationUnit\(tu\);
clang\_disposeIndex\(idx\);
return 0;

}

When I try it on a c file, with system header includes, it produces the following output:
Success

With a c++ file, it produces the following output:
/Users/pbholmen/Projects/ColorCodingTest/Vanilla/main.cpp:9:10: fatal error: 'iostream' file not found
Success

This is the file it tries to parse:
#include <iostream>

int main(int argc, const char * argv) {
// insert code here...
std::cout << "Hello, World!\n";
return 0;
}

I have tried to invoke the clang binary inside the downloaded binaries folder, from the command line, compiling the same file with the exact same options as given to
clang_parseTranslationUnit2, and it compiles without issues.

I can post the list of command line options input to clang_parseTranslationUnit2 as well, but the message is getting long and I'd feel I'm spamming you with tons of Xcode-generated
options, but if you ask for it I'll post it as well. For argv[0] I tried both with the path to the Xcode-provided clang, the path to the dowloaded clang, and omitting the path. I tried to provide
the source file both as an argument to clang_parseTranslationUnit2, and embedded in my argv. Same result either way. I have not deleted any files or folders inside the downloaded prebuilt
binaries folder or relocated libclang.dylib.

Per

The per-compiled binaries available at http://releases.llvm.org are configured differently compared to the binaries shipped by Apple. You need to either manually (either on the command line or directly in your application) add these paths or run an extra installation step to install the headers in /usr/include where both binaries are looking. If you run Clang with the "-v" flag and compare the one shipped by Apple and the one provided at http://releases.llvm.org you will see the difference.

Thank you. I’ve tried running with the -v option now, and there is some
difference. This might indicate that third-party plugin and tool
developers should link to the Apple-provided libclang on macOS.

However, I’m not sure it’s needed, as I did make it work with the
precompiled binaries at LLVM Download Page simply by
calling clang_parseTranslationUnit2FullArgv, and providing the path to
the compiler executable in argv[0]. No extra steps necessary, no adding
paths, and no change of compiler options beyond those supplied by Xcode
(from the xcodebuild log).

I see some third-party developers of libclang-based plugins and tools
struggle with this, so I tried to do the same thing with a more complex
example and some Apple-specific system header includes that I know
vim plugin devs have struggled with, more specifically umbrella headers
in Apple Frameworks, and again it works smoothly right out of the box.
No errors reported, and I can traverse the AST with no problems.

I’ve seen plugin devs do all sorts of hacks to work around issues that

could have been solved this way, so I’d like to share a list of
what I did to make it work, in the faint hope that other third party
devs who run into the same problem read it:

  1. Make sure you load the libclang you expect, and not some arbitrary
    libclang the user might have in their system. Remember that you might
    link to one version at compile time, yet your tool or plugin loads a
    different version at runtime.

  2. Do not move libclang out of the clang+llvm distribution folder, or
    remove files that libclang expects to find there. It seems libclang
    expects to find certain files relative to the lib location.

  3. Use clang_parseTranslationUnit2FullArgv, and provide the path to the
    compiler executable in argv[0]. Interestingly, it made no difference in
    my case whether I used the one in the pre-compiled binaries folder
    downloaded at LLVM Download Page, or the one in
    Xcode’s toolchain.

I suggest tools and plugins don’t mangle with the command line options,

like adding paths or remove -isysroot etc. unless they have tried
everything above (and more), and it still doesn’t work. Doing so creates
all kinds of future compatibility issues, and breaks the invariant that
users of your tool should always be able to depend on: If it works on
the command line, with the same clang version, and exactly the same
compiler options, then it should also work with your tool or plug-in.

I don’t develop or maintain a cross-platform tool myself, so I can’t
guarantee that this invariant always holds for libclang itself,
but I’ve seen cases where devs of such tools think they have to find
workarounds when it could have been solved by the above steps, and
haven’t myself seen cases where it doesn’t hold.

Per

Thank you. I've tried running with the -v option now, and there is some
difference. This might indicate that third-party plugin and tool
developers should link to the Apple-provided libclang on macOS.

However, I'm not sure it's needed, as I did make it work with the
precompiled binaries at LLVM Download Page simply by
calling clang_parseTranslationUnit2FullArgv, and providing the path to
the compiler executable in argv[0]. No extra steps necessary, no adding
paths, and no change of compiler options beyond those supplied by Xcode
(from the xcodebuild log).

Hmm, passing the path to the compiler executable in argv[0] instead of the tool sounds interesting. I haven't thought about that before. But I don't want my tool to require to have the compiler installed. On macOS it might not matter, but on Linux and Windows it's less likely that Clang is installed.

I see some third-party developers of libclang-based plugins and tools
struggle with this, so I tried to do the same thing with a more complex
example and some Apple-specific system header includes that I know
vim plugin devs have struggled with, more specifically umbrella headers
in Apple Frameworks, and again it works smoothly right out of the box.
No errors reported, and I can traverse the AST with no problems.

I've seen plugin devs do all sorts of hacks to work around issues that
could have been solved this way, so I'd like to share a list of
what I did to make it work, in the faint hope that other third party
devs who run into the same problem read it:

1) Make sure you load the libclang you expect, and not some arbitrary
libclang the user might have in their system. Remember that you might
link to one version at compile time, yet your tool or plugin loads a
different version at runtime.

With my tool, I statically link libclang. I've got too many bug reports from users that have used an unsupported version of libclang.

2) Do not move libclang out of the clang+llvm distribution folder, or
remove files that libclang expects to find there. It seems libclang
expects to find certain files relative to the lib location.

3) Use clang_parseTranslationUnit2FullArgv, and provide the path to the
compiler executable in argv[0]. Interestingly, it made no difference in
my case whether I used the one in the pre-compiled binaries folder
downloaded at LLVM Download Page, or the one in
Xcode's toolchain.

I suggest tools and plugins don't mangle with the command line options,
like adding paths or remove -isysroot etc. unless they have tried
everything above (and more), and it still doesn't work. Doing so creates
all kinds of future compatibility issues, and breaks the invariant that
users of your tool should always be able to depend on: If it works on
the command line, with the same clang version, and exactly the same
compiler options, then it should also work with your tool or plug-in.

I don't develop or maintain a cross-platform tool myself, so I can't
guarantee that this invariant always holds for libclang itself,
but I've seen cases where devs of such tools think they have to find
workarounds when it could have been solved by the above steps, and
haven't myself seen cases where it doesn't hold.

I have the SDK installed in the /usr directory as well, so both the Apple provided compiler and the on at provided by llvm.org will find the SDK.