Default Headers with CompilerInstance

Hello everybody

I am now using clang API in order to alter some C sources using my custom ASTVisitor, and I am struggling to make my CompilerInstance object to know about the path of default system headers.

I have a protoype that uses the object CompilerInstance to initialize all the important stuffs (DiagnosticsEngine, Preprocessor ...) and calls "clang::ParseAST" at the end to use my own visitor.
This prototype works well for simple file that do not use any system headers (e.g. "stdlib.h"), but when I want to use these, I get errors like:

$ tsc.c:34:10: fatal error: 'stdlib.h' file not found
$ #include <stdlib.h>

If I set "Verbose = 1" to the HeaderSearchOptions of my CompilerInstance object, I can see that indeed, my search list is empty:

$ #include "..." search starts here:
$ End of search list.

I tried to add the system headers manually using "HeaderSearchOptions.Add" but (1) this is not portable and (2) anyway I can't make that work either.

Could you please explain me more in details how Clang does to find the default system headers ?
I attach the source file of my simple prototype.

Best regards,

- Antoine

prototype.cpp (13.9 KB)

See here for what’s needed to build a tool on top of clang:
http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Tooling/Tooling.cpp?revision=166508&view=markup

(the solution is to use the Driver, which handles finding of builtin and system includes)

Alternatively, you can just use the tooling library :slight_smile: (see http://clang.llvm.org/docs/LibTooling.html)

Cheers,
/Manuel

Hi,

See here for what's needed to build a tool on top of clang:
http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Tooling/Tooling.cpp?revision=166508&view=markup
(the solution is to use the Driver, which handles finding of builtin and
system includes)

Alternatively, you can just use the tooling library :slight_smile: (see
http://clang.llvm.org/docs/LibTooling.html)

None of these solutions relieves you from building your binary at the same place at Clang is installed, or duplicate the Clang headers along your tool installation so that they are in PATH_TO_BIN/../lib/...
Or did I miss something at some point?

    I tried to add the system headers manually using
    "HeaderSearchOptions.Add" but (1) this is not portable and (2)
    anyway I can't make that work either.

    Could you please explain me more in details how Clang does to find
    the default system headers ?
    I attach the source file of my simple prototype.

initialization it end up by looking the headers from the place the binary is installed.

Best,

Mehdi

Hi,

See here for what's needed to build a tool on top of clang:

http://llvm.org/viewvc/llvm-**project/cfe/trunk/lib/Tooling/**
Tooling.cpp?revision=166508&**view=markup<http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Tooling/Tooling.cpp?revision=166508&view=markup&gt;
(the solution is to use the Driver, which handles finding of builtin and
system includes)

Alternatively, you can just use the tooling library :slight_smile: (see
http://clang.llvm.org/docs/**LibTooling.html&lt;http://clang.llvm.org/docs/LibTooling.html&gt;
)

None of these solutions relieves you from building your binary at the same
place at Clang is installed, or duplicate the Clang headers along your tool
installation so that they are in PATH_TO_BIN/../lib/...
Or did I miss something at some point?

What you can do is slurp up the builtin headers as char*'s in your sources,
compile them into your binary, and use virtual files at runtime with
LibTooling (we're actually doing that for our internal mapreduce framework).

The problem with this approach, and the main reason Chandler has opposed
doing that for quite some time, is that when you see errors that reference
the builtin headers you'll not be able to open those files from your editor
(Chandler will jump in if I misrepresented his views :slight_smile:

Cheers,
/Manuel

Thank you for all your answers and sorry for the late reaction.

Let we avoid any misunderstanding. I can compile Clang and I can compile my tool without any issue.
The issue with systems headers occurs when I execute my tool, and when my tool parses a C file (or preprocesses it, with a "Preprocessor" object).

If I sum up your answers, there is no "magic" way that would make Clang API know about the path of my system headers. I have to "hard-code" them.

I find that a bit weird as when I compile Clang from the svn and use it as a frontend for LLVM to compile my files, it finds the headers without any intervention from myself.
Moreover, I've noticed that if I use libclang as defined in "Index.h" it also finds these. In this case however, I have to pass an argument to the function "clang_createTranslationUnitFromSourceFile" (at least "-Xclang", even without any further parameters is OK). However I've quitted with using it as seems no to feature any routines to parse the AST node by node, modify it and deserialize it.

I haven't tried the tooling library yet, does it feature such possibilities ?

Best regards,

- Antoine

Is -internal-isystem the flag that controls this? That seems to be what lit is using for %clang_cc1.

Jordan

Thank you for all your answers and sorry for the late reaction.

Let we avoid any misunderstanding. I can compile Clang and I can compile
my tool without any issue.
The issue with systems headers occurs when I execute my tool, and when my
tool parses a C file (or preprocesses it, with a "Preprocessor" object).

If I sum up your answers, there is no "magic" way that would make Clang
API know about the path of my system headers.

That is true.

I have to "hard-code" them.

That is not true :slight_smile: While not "magic", the way to do this is to use the
Driver (which is exactly what clang-the-compiler uses to find the system
and builtin includes).

I find that a bit weird as when I compile Clang from the svn and use it as
a frontend for LLVM to compile my files, it finds the headers without any
intervention from myself.
Moreover, I've noticed that if I use libclang as defined in "Index.h" it
also finds these. In this case however, I have to pass an argument to the
function "clang_createTranslationUnitFromSourceFile" (at least "-Xclang",
even without any further parameters is OK). However I've quitted with using
it as seems no to feature any routines to parse the AST node by node,
modify it and deserialize it.

I haven't tried the tooling library yet, does it feature such
possibilities ?

Yes. It uses the driver and a compilation database so it gets the exact
same flags your normal compile gets, which, combined, will lead to the
right headers being found in the right places.

Cheers,
/Manuel

Thank you for you answers:

@ Jordan

Is -internal-isystem the flag that controls this? That seems to be what lit is using for %clang_cc1.

Thank you. I looked up into Clang's sources and found many relevant references to this option, right into the Driver.

@ Manuel

That is not true :slight_smile: While not "magic", the way to do this is to use the Driver (which is exactly what clang-the-compiler uses to find the system and builtin includes).

That is right. The implementations of the Driver (and ToolChain) seem to feature some nice features I should use.

I guess that means I will have forget about any CompilerInstance object. Do you have any code snippet that uses the Driver ?
And by the way, will I still be able to use ASTParser() ?

Regards

- Antoine

Thank you for you answers:

@ Jordan
> Is -internal-isystem the flag that controls this? That seems to be what
lit is using for %clang_cc1.
Thank you. I looked up into Clang's sources and found many relevant
references to this option, right into the Driver.

@ Manuel
> That is not true :slight_smile: While not "magic", the way to do this is to use the
Driver (which is exactly what clang-the-compiler uses to find the system
and builtin includes).
That is right. The implementations of the Driver (and ToolChain) seem to
feature some nice features I should use.

I guess that means I will have forget about any CompilerInstance object.
Do you have any code snippet that uses the Driver ?
And by the way, will I still be able to use ASTParser() ?

See:
http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Tooling/Tooling.cpp?revision=166508&view=markup
The implementation of the Tooling library uses the drive exactly in the way
you'd need to use it - you can either copy that code, or use the Tooling
library :slight_smile:

Cheers,
/Manuel

See:
http://llvm.org/viewvc/llvm-project/cfe/trunk/lib/Tooling/Tooling.cpp?revision=166508&view=markup
The implementation of the Tooling library uses the drive exactly in the way you'd need to use it - you can either copy that code, or use the Tooling library :slight_smile:

Thank you ! I'll try to make sense of that.

- Antoine

I am now trying to use the libtooling. I get some errors about some missing JSON compilation database, but I'll figure out.

In the meantime I'll try to make my program work as soon as possible using the quick and dirty solution, and refine latter using the libTooling.

And I have a new question: where is the correct place to send arguments (like "-internal-isystem") to Clang's frontend ?
I've tried "CompilerInstance.makeDiagnostics()" but it looks like my arguments are not taken into account.

I attach my source (the main starts at line 387 and I create the diagnostics engine at line 437).

I've checked in the sources of LibTooling (as kindly pointed out by Manuel) and I could find any other location.... Maybe I messed up with the ordering of the function calls.

By the way, in order to add system headers, I've figured out how to use "HeaderSearchOptions.AddPath()" the correct way, but I get some errors in the file "xmmint.h" related to the width of vector. I think I should pass some extra argument arguments to the frontend...

FYI, the error I get is

In file included from tsc.c:43:
In file included from /home/trouve/boulot/2011_compiler/usr/lib/clang/3.2/include/xmmintrin.h:31:
/home/trouve/boulot/2011_compiler/usr/lib/clang/3.2/include/mmintrin.h:46:12: error: invalid conversion between vector type '__m64' and integer type 'int' of different size
    return (__m64)__builtin_ia32_vec_init_v2si(__i, 0);
           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/trouve/boulot/2011_compiler/usr/lib/clang/3.2/include/mmintrin.h:70:12: error: invalid conversion between vector type '__m64' and integer type 'int' of different size
    return (__m64)__builtin_ia32_packsswb((__v4hi)__m1, (__v4hi)__m2);
           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

(this continues with many similar errors)

- Antoine

prepareTSVC.cpp (18.1 KB)