Builtin headers

Hello all,

I’m using clang as a frontend to parse C and C++ code, which I convert to an internal format that my tool understand.

Since the beginning of the development, there was the problem including the builtin headers to the compilation which I kinda fix with a code to search for headers on the system.

The problem is that we ship the binary, including static and dynamic versions, which requires that the users to have clang installed on their machines. It’s OK for the dynamic version but for the static one it’s a bad solution.

So, right now I only found this two solutions:

  1. Either the user must have clang installed in their machines (which I can’t tell if there will be a problem about using headers from different versions of clang).

  2. Ship all the builtin headers, even for the static version, and hard code a path for them from the binary (like clang does).

Is it possible to implement a third solution: circumvent the requirement for all the builtin headers? I even tried to copy the builitin headers as virtualFiles, but it doesn’t work.

I’m using the following code to generate the AST:

bool llvm_languaget::parse(const std::string& path)
{
// Finish the compiler string
std::vectorstd::string compiler_string;
build_compiler_string(compiler_string);

clang::tooling::FixedCompilationDatabase Compilations(“./”, compiler_string);

std::vectorstd::string sources;
sources.push_back(“/esbmc_intrinsics.h”);
sources.push_back(path);

clang::tooling::ClangTool Tool(Compilations, sources);
Tool.mapVirtualFile(“/esbmc_intrinsics.h”, intrinsics);

Tool.buildASTs(ASTs);

// Use diagnostics to find errors, rather than the return code.
for (const auto &astunit : ASTs) {
if (astunit->getDiagnostics().hasErrorOccurred()) {
std::cerr << std::endl;
return true;
}
}

return false;
}

I already use VirtualFile to map some variables and function declarations but it doesn’t work if add, for example, the code from stddef.h and map to a virtual stddef.h.

Thank you,

Benjamin has added virtual fs capabilities that should enable us to be virtually overlay the builtin headers while still doing a normal header search (we are successfully doing that).

In my tool, DStep [1], I've embedded the headers in the executable itself. The tool is built using libclang. I'm using an array of "CXUnsavedFile" [2] for the internal headers which is passed when parsing the translation unit [3][4].

Not sure if that helps since you're using the C++ API.

[1] GitHub - jacob-carlborg/dstep: A tool for converting C and Objective-C headers to D modules

[2] https://github.com/jacob-carlborg/dstep/blob/master/clang/Compiler.d#L48

[3] https://github.com/jacob-carlborg/dstep/blob/master/dstep/driver/Application.d#L96

[4] https://github.com/jacob-carlborg/dstep/blob/master/clang/TranslationUnit.d#L35-L36

Hi Manuel and Benjamin,

Is this available on clang 3.8? (I’m building clang 3.8 now to try it out)

Is it too much trouble if you could provide some example? Do you simply add code from builtin headers content and map to a virtual file with the same name?

Thank you,

Hi Jacob,

I couldn’t get it working that way.

Thank you,

Hmm, strange. I just verified running my tool inside a docker container. I have three embedded internal Clang headers, it can find those but no other internal headers.

I'm not sure how the C++ interface works but I had to explicitly add the path to the virtual directory to the command line arguments. Basically what I have is a randomly generated string as the virtual directory, based on the root of the file system. The full path of the internal headers are the virtual directory plus the filename of the internal header. Example, the full path of "float.h" could be on Posix:

/94573920482/float.h

And on Windows:

C:\94573920482\float.h

The I add the virtual directory to the command line arguments:

-I/94573920482/

Not sure if any of that helps.

Hello Jacob,

It definitely helped! I finally was able to get it working (I’m using latest clang 3.8 from trunk).

Unfortunatelly, I couldn’t add it to an random folder. When I do something like:

compiler_string.push_back(“-I” + random_virtual_folder); // random_virtual_folder is the random folder, something like /yy4cBWDx/

clang presents the following output:

clang version 3.8.0 (git@github.com:llvm-mirror/clang.git 9c9d4e2a396b6f024fae416c078d59c9c19fa34d) (https://github.com/llvm-mirror/llvm.git 9be4dc8ab20a009ed5f24610888421ba84f8ec65)
Target: i386-unknown-linux-gnu
Thread model: posix
InstalledDir: /home/mramalho/esbmc/build/ansi-c
Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/3.4.6
Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/5.3.1
Selected GCC installation: /usr/lib/gcc/x86_64-redhat-linux/5.3.1
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: 32;@m32
clang Invocation:
“/home/mramalho/esbmc/build/ansi-c/c2goto” “-cc1” “-triple” “i386-unknown-linux-gnu” “-fsyntax-only” “-disable-free” “-disable-llvm-verifier” “-main-file-name” “esbmc_intrinsics.h” “-mrelocation-model” “static” “-mthread-model” “posix” “-mdisable-fp-elim” “-fmath-errno” “-masm-verbose” “-mconstructor-aliases” “-fuse-init-array” “-target-cpu” “pentium4” “-v” “-dwarf-column-info” “-debugger-tuning=gdb” “-resource-dir” “/home/mramalho/esbmc/build/ansi-c/…/lib64/clang/3.8.0” “-I” “/yy4cBWDx/” “-D” “pthread_join=pthread_join_noswitch” “-D” “pthread_mutex_lock=pthread_mutex_lock_noassert” “-D” “pthread_mutex_unlock=pthread_mutex_unlock_noassert” “-D” “pthread_cond_wait=pthread_cond_wait_nocheck” “-D” “__NO_CTYPE” “-D” “realloc=__ESBMC_realloc” “-internal-isystem” “/usr/local/include” “-internal-isystem” “/home/mramalho/esbmc/build/ansi-c/…/lib64/clang/3.8.0/include” “-internal-externc-isystem” “/include” “-internal-externc-isystem” “/usr/include” “-fdebug-compilation-dir” “/home/mramalho/esbmc/build/ansi-c” “-ferror-limit” “19” “-fmessage-length” “91” “-fobjc-runtime=gcc” “-fdiagnostics-show-option” “-fcolor-diagnostics” “-x” “c” “/esbmc_intrinsics.h”

ignoring nonexistent directory “/yy4cBWDx/”
ignoring nonexistent directory “/home/mramalho/esbmc/build/ansi-c/…/lib64/clang/3.8.0/include”
ignoring nonexistent directory “/include”
#include “…” search starts here:
#include <…> search starts here:
/usr/local/include
/usr/include
End of search list.

Bolded is the path being passed to the command line and being later ignored.

But, if I remove the random folder and add to “./”, it works correctly.

Thank you for the help.

Hi Manuel and Benjamin,

Is this available on clang 3.8? (I'm building clang 3.8 now to try it out)

Yep, this is new in 3.8.

Is it too much trouble if you could provide some example? Do you simply add
code from builtin headers content and map to a virtual file with the same
name?

The basic idea is to create a vfs::OverlayFileSystem with both a
vfs::RealFileSystem and a vfs::InMemoryFileSystem, and add the builtin
headers at a virtual path in the InMemoryFileSystem. Then initialize
your FileManager with the OverlayFileSystem and everything should
work.

There should be some example of this in-tree, just grep for InMemoryFileSystem.