Tips for optimising libclang

Clang Frontend
1

Hello List,

I was hoping someone could give me some tips on how to optimise
creation/reparsing of translation units via libclang as well as code
completion requests.

At the moment I'm getting over 1500 ms on average each time a TU is
created for a control source file I use and only marginally less when
it's reparsed. Code completion takes on average about 1000 ms (never
less). This on an i7 with 12GB of memory; system is amd64 Linux 3.5.0-27
and clang version 3.3 (trunk 170425).

The project's PCH includes can be viewed on pastebin [0] - STL and Boost
are among the includes.

Here's how I'm handling a source file with libclang:

1) Create index if it hasn't been created yet:

  index = clang_createIndex(0, 0);

2) Create TU:

  tu = clang_parseTranslationUnit(
    index,
    filename,
    parameters, [1]
    unsavedFiles, /* to keep it simple, I'm not using
          unsaved files at the moment */
    0,
    CXTranslationUnit_PrecompiledPreamble
                  > CXTranslationUnit_CacheCompletionResults);

3) When requesting code completion:

  results = clang_codeCompleteAt(
        tu,
        filename,
        line, col,
        unsavedFiles,
        0,
        CXCodeComplete_IncludeMacros);

Questions:

1) I haven't noticed any improvement in performance when adding more
source files (creating more TUs) that belong to the same project as the
control source file I mentioned above. Shouldn't it be the case that
libclang should reuse the outputs generated when parsing include files
shared by more than one source file?

1.1) If not, can it be done manually?

2) How can performance be improved? Before starting this project I was
hoping for (re-)parsing and code completion times not greater than
100ms.

[0] Includes in PCH - Pastebin.com
[1] -I/home/guedesm/prj/omnis/out
    -I/usr/include/c++/4.7 -I/usr/include/c++/4.7/x86_64-linux-gnu
    -I/usr/include/c++/4.7/backward
    -I/usr/lib/gcc/x86_64-linux-gnu/4.7/include
    -I/usr/local/include
    -I/usr/lib/gcc/x86_64-linux-gnu/4.7/include-fixed
    -I/usr/include/x86_64-linux-gnu
    -I/usr/include

2

Three points:

1)

When developing clang_complete, we noticed that clang_parseTranslationUnit was not enough to initialize the PrecompiledPreamble completely. Instead, we had to issue a reparse between the initial parse and the codeComplete command. I never found out if this was something we did wrong or something that libclang did not get right.

2)

Are you sure the file compiles without errors? If there are errors in the preamble, no caching will happen.

3)

Where are you code completing? The preamble works only if you don't code complete before or directly around the first set of include directives.

With clang_complete I see completion times of around 45 mseconds for larger boost based projects. So there seems to be a problem somewhere in this setup.

Tobias

3

Hi Tobias,

Three points:

1)

When developing clang_complete, we noticed that
clang_parseTranslationUnit was not enough to initialize the
PrecompiledPreamble completely. Instead, we had to issue a reparse
between the initial parse and the codeComplete command. I never found
out if this was something we did wrong or something that libclang did
not get right.

Strange. I'm seeing the exact same behaviour you describe. As you say,
only after parsing and then reparsing do subsequent TU reparsing calls
finish dramatically quicker. However, creating the preamble is now
taking ~2500ms BUT subsequent updates take in the order of 250-260ms.
Still a bit too high though.

What timings do you experience with clang_complete when reparsing
following an edit?

Many thanks for the invaluable info, Tobias!

4

This largely depends on the machine. On my (fast) laptop, the clang_complete example file in examples/boost.cpp takes 0.046 seconds.

It would be really great to investigate this issue to get rid of the need to reparse.

Tobi

5

After having looked into clang_complete (thanks for the tip, Tobias!) I
came to the conclusion that I'd coded my code completion logic wrong. I
was under the impression that before invoking clang_codeCompleteAt one
had to reparse the TU but it turns out that's not the case - all one has
to do is feed the source file contents in a CXUnsavedFile struct.

Thanks for your invaluable help!

6

Can you suggest a specific improvement to the documentation that would help
avoid this issue in the future?

-- Sean Silva

7

Well, I find that a difficult question to answer because my mother
tongue isn't English and so you could argue that the documentation is
clear enough and it was I who failed to understand it.

However, my personal opinion is that indeed it could do with some
improvements designed to remove (what I think are) ambiguities and to
make it easier for users who lack in-depth knowledge of clang, libclang
or compiler technologies in general, as is my case, to use (lib)clang
and develop. My personal opinion is the clearer the documentation is the
less noise (like this very thread) the core cfe-devs would (perhaps) get
in this mailing list; noise, which I'm sure is annoying - it took me
awhile before I finally convinced myself I should post here.

As requested, here's a few suggestions based on my (most) limited
knowledge of libclang:

* Creation of precompiled preamble

  There's no mention in the documentation about the issue that Tobias
raised in this thread; that the preamble is only created after the TU is
parsed a second time. IOW, the documentation should make it clear that
one must issue clang_parseTranslationUnit, followed by
clang_reparseTranslationUnit to ensure the preamble is created.

  BTW, no one has commented on whether this is a bug or a feature. I
take it it's the former?

* clang_reparseTranslationUnit

  The documentation doesn't make it clear that the TU is re-created (or
something to that effect) when reparsing a source file in which
preprocessor directives have been modified or added, which results in
parsing times identical to the time it takes to create the TU (first
parse).

* Code completion

  The documentation for the TU parameter of clang_codeCompleteAt states
that the "source files for this translation unit need not be completely
up-to-date (and the contents of those source files may be overridden via
unsaved_files)". As I stated at the start of my reply, English isn't my
first language, and so I may be at fault here. But it does seem like an
ambiguous statement as it doesn't explicitly state that in the case when
the source files have been edited since the last TU parse, the TU
doesn't need to be reparsed. Also, what happens if a preprocessor
directive has been added/modified in the source file in question and how
does that affect code completion?

* `Code' as a second language

  I think most people would agree that `code' is a programmers' second
language (if not first in some cases!). Why not complement the
documentation with more code examples to reinforce the textual
description? While I have come across some snippets of code in the
documentation, I believe certain core aspects of (lib)clang that lack
code examples would greatly benefit from them. This measure should
further help iron out whatever ambiguities linger in the text as well as
make it easier for non English speakers to understand the documentation.

Should the above prove useful in any way, I would be more than willing
to help in any way.

8

For me this looks like a bug or a behavior that is so surprising that it should be mentioned in the documentation. It would be really good if some clang autocompletion expert could comment on this.

Tobi