Libclang: Bad performance on reparse

Hi,
while testing kdevelop 5, I ran into some problems with the code completion.
I debugged it and tracked it down to the reparse step in libclang taking
5 seconds in an i7-5930K.

In my concrete example, I got a cpp file that consists of 173 line of code,
with minor template usage.
This program includes a big amount of headers, that use heavy templating.

The initial parse takes 13 seconds, which is fine. What makes me wonder is
the reparse step taking so long, as everything in the headers should be
cached by the PCH.

I prepared a git repo with an example, if you want to look at the code I used,
or try it yourself.

https://github.com/jmachowinski/clangTest

It should be noted, that I do not set CXTranslationUnit_Incomplete,
as kdevelop needs the flag to be NOT set. Setting the flag speeds up reparse
to 0.5 seconds.

Greetings
     Janosch

Debug output :

Parse start
Precompiling preamble: 8.2960 (100.0%) 0.2760 (100.0%) 8.5720 (100.0%) 8.5706 (100.0%)
Parsing ../test/tdot_asguard.cpp: 4.8320 (100.0%) 0.0200 (100.0%) 4.8520 (100.0%) 4.8499 (100.0%)
   ASTContext: expressions, declarations, and types: 254289451
   ASTContext: identifiers: 507904
   ASTContext: selectors: 4096
   ASTContext: side tables: 9053176
   Code completion: cached global results: 0
   SourceManager: content cache allocator: 86016
   SourceManager: malloc'ed memory buffers: 2070147
   SourceManager: mmap'ed memory buffers: 0
   SourceManager: data structures and tables: 124472904
   ExternalASTSource: malloc'ed memory buffers: 0
   ExternalASTSource: mmap'ed memory buffers: 188282552
   Preprocessor: malloc'ed memory: 306782
   Preprocessor: PreprocessingRecord: 838904
   Preprocessor: header search tables: 43
Parse took 13.4559
Cache global code completions for ../test/tdot_asguard.cpp: 0.1280 (100.0%) 0.1280 (100.0%) 0.1267 (100.0%)
Reparsing ../test/tdot_asguard.cpp: 5.0600 (100.0%) 0.0440 (100.0%) 5.1040 (100.0%) 5.1043 (100.0%)
   ASTContext: expressions, declarations, and types: 259532331
   ASTContext: identifiers: 2326528
   ASTContext: selectors: 4096
   ASTContext: side tables: 9151480
   Code completion: cached global results: 4063232
   SourceManager: content cache allocator: 111147
   SourceManager: malloc'ed memory buffers: 2082327
   SourceManager: mmap'ed memory buffers: 0
   SourceManager: data structures and tables: 124472904
   ExternalASTSource: malloc'ed memory buffers: 0
   ExternalASTSource: mmap'ed memory buffers: 188282552
   Preprocessor: malloc'ed memory: 8324702
   Preprocessor: PreprocessingRecord: 1775864
   Preprocessor: header search tables: 43
ReParse took 5.10452

Hi,
while testing kdevelop 5, I ran into some problems with the code completion.
I debugged it and tracked it down to the reparse step in libclang taking
5 seconds in an i7-5930K.

In my concrete example, I got a cpp file that consists of 173 line of code,
with minor template usage.
This program includes a big amount of headers, that use heavy templating.

Well, if the template usage in the headers is mostly dependent, PCH alone isn't going
to help much. We may be able to see that A<T> is using B<T> and C<T>, but we can't
actually instantiate any of that until we get a request for a specialization at a concrete
type, which we may not see until we get to the main source file.

There are a number of things that can be done on top of a PCH system to help move
some of that computation time into PCH generation. For example, the tool could remember
the set of types that the templates are instantiated at outside of the PCH and then
re-generate the PCH with eager instantiations. Also, of course, the tool may not actually
require those instantiations to be done at all; it may benefit from a great deal more laziness.
But all of this requires more logic from the tool, not just generating a PCH.

That may not be what's happening here, of course.

John.

Hi Janosch.

Thanks for providing a clear test-case. I’d be interested in following up on this, can you file a bug report?

Cheers,
Alex

Hi Alex,
just filed a bug report :
https://bugs.llvm.org/show_bug.cgi?id=34281
Greetings
     Janosch