LibTooling question

Hi CLANGers,

I've been playing with Clang's LibTooling classes closely following
the tutorial at http://clang.llvm.org/docs/LibTooling.html in order
to make use of clang as a frontend in our project.

However, I might need some assistence:

1. Include headers: Using the ClangTool class (like the example suggests)
   it cannot find its own headers. The documentation states:

   "Clang tools need their builtin headers and search for them the same
    way clang does. Thus, the default location to look for builtin headers
    is in a path $(dirname /path/to/tool)/../lib/clang/3.2/include relative
    to the tool binary."

   But apparently the passed pseudo argv[0] is clang-tool, and, hence, the
   invocation contains:

      -resource-dir ../lib/clang/3.1
      -internal-isystem ../lib/clang/3.1/include

   on Linux. (Windows seems to work.) It this intended behaviour?
   How can I set those paths? clang itself works. The ToolInvocation class
   as well, if I pass a reasonable path as argv[0].

2. I finally managed to run a SyntaxOnlyAction as well as PrintPreprocessedAction.
   But how can I compile a module to LLVM IR?

3. And is it possible to access the generated result (LLVM IR or preprocessed
   source code) without first writing to disk and re-reading.

Thanks in advance,
Mario

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi CLANGers,

I've been playing with Clang's LibTooling classes closely following
the tutorial at http://clang.llvm.org/docs/LibTooling.html in order
to make use of clang as a frontend in our project.

However, I might need some assistence:

1. Include headers: Using the ClangTool class (like the example suggests)
   it cannot find its own headers. The documentation states:

   "Clang tools need their builtin headers and search for them the same
    way clang does. Thus, the default location to look for builtin headers
    is in a path $(dirname /path/to/tool)/../lib/clang/3.2/include relative
    to the tool binary."

   But apparently the passed pseudo argv[0] is clang-tool, and, hence, the
   invocation contains:

That should only be the case if you use runToolOnCode, and not if you
use ClangTool.

Does the code live somewhere you can point me to?

      -resource-dir ../lib/clang/3.1
      -internal-isystem ../lib/clang/3.1/include

   on Linux. (Windows seems to work.) It this intended behaviour?
   How can I set those paths? clang itself works. The ToolInvocation class
   as well, if I pass a reasonable path as argv[0].

2. I finally managed to run a SyntaxOnlyAction as well as PrintPreprocessedAction.
   But how can I compile a module to LLVM IR?

3. And is it possible to access the generated result (LLVM IR or preprocessed
   source code) without first writing to disk and re-reading.

Regarding code generation I'm not sure :slight_smile: The tooling is definitely
built mainly with parsing in mind. I've looped in Nick for some
comments on what might be necessary to get your use case working...

Cheers,
/Manuel

Hi Manuel, Nick, CLANGers,

On Tue, Jul 31, 2012 at 10:37 AM, Mario Schwalbe <mario@se.inf.tu-dresden.de> wrote: I've been playing with Clang's LibTooling classes closely following the tutorial at http://clang.llvm.org/docs/LibTooling.html in order to make use of clang as a frontend in our project.

However, I might need some assistence:

1. Include headers: Using the ClangTool class (like the example suggests) it cannot find its own headers. The documentation states:

"Clang tools need their builtin headers and search for them the same way clang does. Thus, the default location to look for builtin headers is in a path $(dirname /path/to/tool)/../lib/clang/3.2/include relative to the tool binary."

But apparently the passed pseudo argv[0] is clang-tool, and, hence, the invocation contains:

That should only be the case if you use runToolOnCode, and not if you use ClangTool.

No. I never tried runToolOnCode().

Does the code live somewhere you can point me to?

Yes. http://clang.llvm.org/docs/LibTooling.html. Basically, it is the ClangCheck
example if you use a FixedCompilationDatabase passed via the command line.
Btw: It works on Windows because several -internal-isystem options point to
VC headers which contain stddef.h and co.

I got it to work now using the ToolInvocation class. I just wanted to point out
that imho this behaviour renders the ClangTool class somewhat useless on Linux
and others might fight the same problem as well...

2. I finally managed to run a SyntaxOnlyAction as well as PrintPreprocessedAction. But how can I compile a module to LLVM IR?

3. And is it possible to access the generated result (LLVM IR or preprocessed source code) without first writing to disk and re-reading.

Regarding code generation I'm not sure :slight_smile: The tooling is definitely built mainly with parsing in mind. I've looped in Nick for some comments on what might be necessary to get your use case working...

Never mind. I already found it by grepping the clang sources. :wink: The answer
are CodeGenActions like EmitLLVMAction and EmitBCAction. Although it seems like
I have to write to disk, since (a) the ToolInvocation class always generates an
- -o option, and (b) the derived CodeGenAction::takeModule() always returns NULL.
(I still don't know why.)

ciao,
Mario

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Manuel, Nick, CLANGers,

On Tue, Jul 31, 2012 at 10:37 AM, Mario Schwalbe <mario@se.inf.tu-dresden.de> wrote: I've been playing with Clang's LibTooling classes closely following the tutorial at http://clang.llvm.org/docs/LibTooling.html in order to make use of clang as a frontend in our project.

However, I might need some assistence:

1. Include headers: Using the ClangTool class (like the example suggests) it cannot find its own headers. The documentation states:

"Clang tools need their builtin headers and search for them the same way clang does. Thus, the default location to look for builtin headers is in a path $(dirname /path/to/tool)/../lib/clang/3.2/include relative to the tool binary."

But apparently the passed pseudo argv[0] is clang-tool, and, hence, the invocation contains:

That should only be the case if you use runToolOnCode, and not if you use ClangTool.

No. I never tried runToolOnCode().

Does the code live somewhere you can point me to?

Yes. http://clang.llvm.org/docs/LibTooling.html. Basically, it is the ClangCheck
example if you use a FixedCompilationDatabase passed via the command line.
Btw: It works on Windows because several -internal-isystem options point to
VC headers which contain stddef.h and co.

I need more context to reproduce. For me, with the latest clang-check
built from head, and the following setup:

$ find /tmp/c
/tmp/c
/tmp/c/bin
/tmp/c/bin/clang-check
/tmp/c/lib
/tmp/c/lib/clang
/tmp/c/lib/clang/3.2
/tmp/c/lib/clang/3.2/include
...
/tmp/c/lib/clang/3.2/include/stddef.h
...

$ clang-check t2.cc -- -c -v
Processing: /home/klimek/Snippets/t2.cc.
clang version 3.2 (trunk)
Target: x86_64-unknown-linux-gnu
Thread model: posix
clang Invocation:
"/tmp/c/bin/clang-check" "-cc1" "-triple" "x86_64-unknown-linux-gnu"
"-fsyntax-only" "-disable-free" "-main-file-name" "t2.cc"
"-mrelocation-model" "static" "-mdisable-fp-elim" "-fmath-errno"
"-masm-verbose" "-mconstructor-aliases" "-munwind-tables"
"-target-cpu" "x86-64" "-momit-leaf-frame-pointer" "-v"
"-resource-dir" "/tmp/c/bin/../lib/clang/3.2" "-fmodule-cache-path"
"/var/tmp/clang-module-cache" "-internal-isystem"
"/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../include/c++/4.6"
"-internal-isystem"
"/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../include/c++/4.6/x86_64-linux-gnu"
"-internal-isystem"
"/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../include/c++/4.6/backward"
"-internal-isystem" "/usr/local/include" "-internal-isystem"
"/tmp/c/bin/../lib/clang/3.2/include" "-internal-externc-isystem"
"/usr/include/x86_64-linux-gnu" "-internal-externc-isystem" "/include"
"-internal-externc-isystem" "/usr/include" "-fdeprecated-macro"
"-fdebug-compilation-dir" "/home/klimek/Snippets" "-ferror-limit" "19"
"-fmessage-length" "362" "-mstackrealign" "-fobjc-runtime=gcc"
"-fcxx-exceptions" "-fexceptions" "-fdiagnostics-show-option"
"-fcolor-diagnostics" "-x" "c++" "/home/klimek/Snippets/t2.cc"

clang -cc1 version 3.2 based upon LLVM 3.2svn default target
x86_64-unknown-linux-gnu
ignoring nonexistent directory "/include"
#include "..." search starts here:
#include <...> search starts here:
/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../include/c++/4.6
/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../include/c++/4.6/x86_64-linux-gnu
/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../include/c++/4.6/backward
/usr/local/include
/tmp/c/bin/../lib/clang/3.2/include
/usr/include/x86_64-linux-gnu
/usr/include
End of search list.

Which looks exactly like it should.

Can you post the output of calling your tool with -- -c -v?

Thanks,
/Manuel

Hi,

maybe I should have mentioned that I use the latest release version of LLVM and clang.

I need more context to reproduce. For me, with the latest clang-check built from head, and the following setup: [...] Which looks exactly like it should.

Can you post the output of calling your tool with -- -c -v?

$ /tmp/ClangCheck /tmp/TestProg.c -- -c -v
Processing: /tmp/TestProg.c.
clang version 3.1 (branches/release_31)
Target: x86_64-unknown-linux-gnu
Thread model: posix
clang Invocation:
"clang-tool" "-cc1" "-triple" "x86_64-unknown-linux-gnu" "-emit-obj" "-mrelax-all"
"-disable-free" "-main-file-name" "TestProg.c" "-mrelocation-model" "static"
"-mdisable-fp-elim" "-masm-verbose" "-mconstructor-aliases" "-munwind-tables"
"-target-cpu" "x86-64" "-target-linker-version" "2.22" "-momit-leaf-frame-pointer"
"-v" "-coverage-file" "TestProg.o" "-resource-dir" "../lib/clang/3.1" "-working-directory" "."
"-fmodule-cache-path" "/var/tmp/clang-module-cache" "-internal-isystem" "/usr/local/include"
"-internal-isystem" "../lib/clang/3.1/include" "-internal-externc-isystem"
"/usr/include/x86_64-linux-gnu" "-internal-externc-isystem" "/include"
"-internal-externc-isystem" "/usr/include" "-fdebug-compilation-dir" "/tmp" "-ferror-limit" "19"
"-fmessage-length" "100" "-mstackrealign" "-fgnu-runtime" "-fobjc-runtime-has-arc"
"-fobjc-runtime-has-weak" "-fobjc-fragile-abi" "-fdiagnostics-show-option" "-fcolor-diagnostics"
"-o" "TestProg.o" "-x" "c" "/tmp/TestProg.c"

clang -cc1 version 3.1 based upon LLVM 3.1 default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/usr/local/include"
ignoring nonexistent directory "../lib/clang/3.1/include"
ignoring nonexistent directory "/include"
#include "..." search starts here:
#include <...> search starts here:
/usr/include/x86_64-linux-gnu
/usr/include
End of search list.
In file included from /tmp/TestProg.c:1:
/usr/include/stdio.h:34:11: fatal error: 'stddef.h' file not found
# include <stddef.h>
          ^
1 error generated.
Error while processing /tmp/TestProg.c.

<div class="gmail_quote">On 1 August 2012 06:45, Manuel Klimek <span
dir="ltr">&lt;<a href="mailto:klimek@google.com"
target="_blank">klimek@google.com</a>&gt;</span> wrote:<br><blockquote
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex;"><div class="im">On Tue, Jul 31, 2012 at 10:37
AM, Mario Schwalbe<br>
&lt;<a href="mailto:mario@se.inf.tu-dresden.de">mario@se.inf.tu-dresden.de</a>&gt;
wrote:<br>
&gt; -----BEGIN PGP SIGNED MESSAGE-----<br>
&gt; Hash: SHA1<br>
&gt;<br>
&gt; Hi CLANGers,<br>
&gt;<br>
&gt; I've been playing with Clang's LibTooling classes closely following<br>
&gt; the tutorial at <a
href="http://clang.llvm.org/docs/LibTooling.html"
target="_blank">http://clang.llvm.org/docs/<wbr>LibTooling.html</a> in
order<br>
&gt; to make use of clang as a frontend in our project.<br>
&gt;<br>
&gt; However, I might need some assistence:<br>
&gt;<br>
&gt; 1. Include headers: Using the ClangTool class (like the example
suggests)<br>
&gt; &nbsp; &nbsp;it cannot find its own headers. The documentation states:<br>
&gt;<br>
&gt; &nbsp; &nbsp;"Clang tools need their builtin headers and search
for them the same<br>
&gt; &nbsp; &nbsp; way clang does. Thus, the default location to look
for builtin headers<br>
&gt; &nbsp; &nbsp; is in a path $(dirname
/path/to/tool)/../lib/clang/3.<wbr>2/include relative<br>
&gt; &nbsp; &nbsp; to the tool binary."<br>
&gt;<br>
&gt; &nbsp; &nbsp;But apparently the passed pseudo argv[0] is
clang-tool, and, hence, the<br>
&gt; &nbsp; &nbsp;invocation contains:<br>
<br>
</div>That should only be the case if you use runToolOnCode, and not if you<br>
use ClangTool.<br>
<br>
Does the code live somewhere you can point me to?<br>
<div class="im"><br>
&gt; &nbsp; &nbsp; &nbsp; -resource-dir ../lib/clang/3.1<br>
&gt; &nbsp; &nbsp; &nbsp; -internal-isystem ../lib/clang/3.1/include<br>
&gt;<br>
&gt; &nbsp; &nbsp;on Linux. (Windows seems to work.) It this intended
behaviour?<br>
&gt; &nbsp; &nbsp;How can I set those paths? clang itself works. The
ToolInvocation class<br>
&gt; &nbsp; &nbsp;as well, if I pass a reasonable path as argv[0].<br>
&gt;<br>
&gt; 2. I finally managed to run a SyntaxOnlyAction as well as
PrintPreprocessedAction.<br>
&gt; &nbsp; &nbsp;But how can I compile a module to LLVM IR?<br>
&gt;<br>
&gt; 3. And is it possible to access the generated result (LLVM IR or
preprocessed<br>
&gt; &nbsp; &nbsp;source code) without first writing to disk and re-reading.<br>
<br>
</div>Regarding code generation I'm not sure :slight_smile: The tooling is definitely<br>
built mainly with parsing in mind. I've looped in Nick for some<br>
comments on what might be necessary to get your use case
working...<br></blockquote><div><br></div><div>Yep, I've done this. I
create my own FrontendAction derived from clang::EmitLLVMOnlyAction.
&nbsp;The replacement EndSourceFileAction() calls
this-&gt;EmitLLVMOnlyAction::EndSourceFileAction(); and then
takeModule() returns the
llvm::Module.</div><div><br></div><div>Nick</div><div>&nbsp;</div><blockquote
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex;">Cheers,<br>
/Manuel<br>
<div class="HOEnZb"><div class="h5"><br>
&gt;<br>
&gt; Thanks in advance,<br>
&gt; Mario<br>
&gt; -----BEGIN PGP SIGNATURE-----<br>
&gt; Version: GnuPG v1.4.11 (GNU/Linux)<br>
&gt; Comment: Using GnuPG with Mozilla - <a
href="http://enigmail.mozdev.org/"
target="_blank">http://enigmail.mozdev.org/</a><br>
&gt;<br>
&gt; iQEcBAEBAgAGBQJQF5kiAAoJEDv0fP<wbr>6GapNt1j8IAInVS4J2446fYTpxJF+<wbr>uUjSe<br>
&gt; mV3rte0PyZ/fS1+70LfH+<wbr>etjjiP0PMhrvLKIvU5HazMZ2cn7ZpX<wbr>JXqopXQS/orcx<br>
&gt; hzwTrYzRehfVOzv0ykW0XwI5yKqaOg<wbr>+SKZtyevBSMMY7go+<wbr>x7CXZQw9rNncetI7L<br>
&gt; e4ITyzQmjQd/<wbr>Tal9OdWpDu2uB9Ma79QBmDYEm035jt<wbr>/twcEDATK91ZFgP5MSaCXQ<br>
&gt; DPwWMZhG01vkR5uoBCQX4gVeLXlMrJ<wbr>JtaSM21ZNv1jZ4cfJdqMqoVTh7wZ/<wbr>VKL1t<br>
&gt; jzvP0AyIWEg6VxI18do15KE2B7TKSz<wbr>q2vBpK0w7lgKbADS8qOZiQH+YzWHE+<wbr>Ahk=<br>
&gt; =PvUf<br>
&gt; -----END PGP SIGNATURE-----<br>
&gt; ______________________________<wbr>_________________<br>
&gt; cfe-dev mailing list<br>
&gt; <a href="mailto:cfe-dev@cs.uiuc.edu">cfe-dev@cs.uiuc.edu</a><br>
&gt; <a href="http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev"
target="_blank">http://lists.cs.uiuc.edu/<wbr>mailman/listinfo/cfe-dev</a><br>
</div></div></blockquote></div><br>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

maybe I should have mentioned that I use the latest release version of LLVM and clang.

That would explain it - I'd strongly suggest to use current trunk for
tooling - it's new enough that there are significant bugfixes.

Cheers,
/Manuel

Thanks for the tip. Works great now.

ciao,
Mario