Standalone tools

Hi all,

I'm one of the maintainers of IWYU, a tool for cleaning up #include
dependencies.

We've experimented with releasing IWYU binaries for a select number of
platforms, but people are having problems because Clang is looking for
headers relative to the executable. There have been reports that
moving IWYU into the Clang install directory makes things work, but
this is intrusive and a number of users don't even have Clang
installed.

Is there a strategy for how to package up a standalone Clang tool so
it can be distributed? Do/can we include the headers from
...lib/clang/3.5.0/include? Anything else necessary?

(Full disclosure: we're not using Tooling just yet, but I'm guessing
that doesn't change anything)

Thanks,
- Kim

I'm having the same problem. Currently I'm telling my users to separately install Clang. For the include files, it also work to copy them to the standard header locations.

It would be nice if the header files could be embedded in the tool (the executable).

Now that Clang has a VFS, this should not be hard to implement.

Dmitri

Hi Jacob,

I'm having the same problem. Currently I'm telling my users to separately
install Clang.

Do you deploy to Unix-like systems only? I'm guessing this works well
there, because everything is rooted in /usr/bin, right?

So:

  /usr/bin/your-tool
  /usr/bin/clang
  /usr/lib/clang/3.5.0/include

?

On Windows, the situation is a little different as everything installs
into C:\Program Files\LLVM, including the headers. So the only way to
have a tool find the headers would be to install it into LLVM's root
directory, which seems a little intrusive...

For the include files, it also work to copy them to the
standard header locations.

You mean the equivalent of /usr/lib/clang/3.5.0/include? If Clang was
installed afterwards, wouldn't they conflict?

It would be nice if the header files could be embedded in the tool (the
executable).

That sounds like a lovely idea.

The only obvious way I can see, given the current state, to achieve
some kind of isolation is to:
- install IWYU into /usr/bin/include-what-you-use
- install Clang headers into /usr/lib/include-what-you-use/<version>/include

For Windows, replace "/usr" with some install directory, e.g.
"C:\Program Files\IWYU"

This is assuming the headers is the only thing I need to worry about,
but for a syntax-only tool, it should be, right?

Thanks,
- Kim

Sorry, I lost the list in my reply, adding it back here.

How would that work?

Since which version of Clang is this available?

Now that Clang has a VFS, this should not be hard to implement.

How would that work?

Header file contents could be placed directly into the YAML file,
which would be embedded in the tool.

Since which version of Clang is this available?

VFS is available in trunk only.

Dmitri

Do you deploy to Unix-like systems only?

Yes, so far. I haven't been able to get my tool working on Windows yet.

I'm guessing this works well there, because everything is rooted in /usr/bin, right?

So:

   /usr/bin/your-tool
   /usr/bin/clang
   /usr/lib/clang/3.5.0/include

?

It's rather:

/any/path/the/user/chooses/my-tool
/usr/lib/libclang.so
/usr/include/stdarg.h
/usr/include/stddef.h

Alternatively they can put libclang next to my tool.

On Windows, the situation is a little different as everything installs
into C:\Program Files\LLVM, including the headers. So the only way to
have a tool find the headers would be to install it into LLVM's root
directory, which seems a little intrusive...

I haven't really explored this on Windows yet.

For the include files, it also work to copy them to the
standard header locations.

You mean the equivalent of /usr/lib/clang/3.5.0/include? If Clang was
installed afterwards, wouldn't they conflict?

I was actually thinking of /usr/include or any other path that is by default searched in by the compiler.

That sounds like a lovely idea.

The only obvious way I can see, given the current state, to achieve
some kind of isolation is to:
- install IWYU into /usr/bin/include-what-you-use
- install Clang headers into /usr/lib/include-what-you-use/<version>/include

Yes, seems to be the case.

For Windows, replace "/usr" with some install directory, e.g.
"C:\Program Files\IWYU"

This is assuming the headers is the only thing I need to worry about,
but for a syntax-only tool, it should be, right?

Yes, and the Clang libraries themselves. Unless you link statically.

YAML file? I think you need to explain a bit more. Or point to somewhere/some code I can read more. I have no idea what you're talking about :slight_smile:

Please take a look at tests in clang/test/VFS.

Dmitri

I have been doing that even before clang had a VFS.

The idea is using cmake to put the content of the file in the binary

Using this cmake script
https://github.com/woboq/moc-ng/blob/master/src/CMakeLists.txt#L57
and
https://github.com/woboq/moc-ng/blob/master/src/embedded_includes.h.in
This puts the file contents into the binary

Then I added a "-I/builtins" to the command line, and used
ToolInvocation::mapVirtualFile to expose the files:
http://code.woboq.org/mocng/src/main.cpp.html#354

Thanks, I'll have a look.

Cool, I'll have a look at that as well.

> It would be nice if the header files could be embedded in the tool (the
> executable).

Now that Clang has a VFS, this should not be hard to implement.

Note that I have suggested this before (multiple times), and the powers
that are (well, Chandler & Doug) have shot down the idea.

Hi Jacob,

>
> I'm having the same problem. Currently I'm telling my users to separately
> install Clang.

Do you deploy to Unix-like systems only? I'm guessing this works well
there, because everything is rooted in /usr/bin, right?

Btw, this is not the recommended way - the recommended way is to ship the
builtin headers with the tool (the builtin headers are basically versioned
with the parser).

I am not judging whether it is good or not. It just became easier than ever.

Dmitri

> > It would be nice if the header files could be embedded in the tool (the
> > executable).
>
> Now that Clang has a VFS, this should not be hard to implement.

I have been doing that even before clang had a VFS.

The idea is using cmake to put the content of the file in the binary

Using this cmake script
https://github.com/woboq/moc-ng/blob/master/src/CMakeLists.txt#L57
and
https://github.com/woboq/moc-ng/blob/master/src/embedded_includes.h.in
This puts the file contents into the binary

Then I added a "-I/builtins" to the command line, and used
ToolInvocation::mapVirtualFile to expose the files:
http://code.woboq.org/mocng/src/main.cpp.html#354

Btw, this is how we do it when we have tools that must run without a file
system.

>>
>> > It would be nice if the header files could be embedded in the tool
(the
>> > executable).
>>
>> Now that Clang has a VFS, this should not be hard to implement.
>
> Note that I have suggested this before (multiple times), and the powers
that
> are (well, Chandler & Doug) have shot down the idea.

I am not judging whether it is good or not. It just became easier than
ever.

Oh, it was always easy - the Tooling stuff always had access to the VFS
layer that has been there forever. Chandler objected on a principled basis
(how do we show error message that reference files that are not on disk,
among others)

FYI, for tools like clang_complete finding these headers causes regularly issues when file system paths change and YCM just includes some version of the headers in its own code, which seems to work but is to my understanding incorrect. So having those headers readily available seems from a user's perspective a good thing.

Cheers,
Tobias

Thanks all, I think I have enough to go on.

- Kim