Late Parsed Template support for PCH

Clang Frontend
#1

Hi All,

I’ve discovered a rather nasty issue with -fdelayed-template-parsing enabled. If this is used to build a PCH all of the stored template tokens are lost leading an incomplete AST when the PCH is used to create the complete TU.

Is this by design or a missing feature for the serialization code?

Unfortunately, due to a huge body of MSVC targeted code and a need to use PCH files to maintain acceptable performance I need to make this work seamlessly. The data that would need preserving is currently stored in the Parser and is a map of const FunctionDecl* to LateParsedTemplatedFunction*. The LateParsedTemplatedFunction is essentially a SmallVector of Tokens.

Before I dive in and attempt to add support for serializing this, is there anything I should know?

Cheers,
Will.

#2

Hi, I implemented the original -fdelayed-template-parsing mode and I simply did not consider PCH during the design/implementation. As it is currently this should output an error instead of creating an incomplete AST.

Feel free to add support for fdelayed-template-parsing + PCH.
I never looked at the PCH part of clang so I got no advice really.

#3

I would guess a missing feature. Problems can be demonstrated quite easily:

$ cat t.cpp
template<typename T>
T f(const T& t) {
     return t;
}

int main() {
     f(5);
}

$ clang --version
clang version 3.3 (tags/RELEASE_33/final)
Target: x86_64-unknown-linux-gnu
Thread model: posix

# No problems without -fdelayed-template-parsing...
$ clang -emit-ast t.cpp
$ clang -o t t.ast

# But with -fdelayed-template-parsing...
$ clang -emit-ast -fdelayed-template-parsing t.cpp
$ clang -o t t.ast
/tmp/t-YAfQmm.o: In function `main':
t.ast:(.text+0x14): undefined reference to `int f<int>(int const&)'
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Tom.

#4

It’s a missing feature. One will need to serialize the actual tokens for each of the late-parsed FunctionDecl’s, and be able to lazily deserialize and then parse the tokens when an actual definition is needed.

  • Doug
#5

Hi All,

I've discovered a rather nasty issue with -fdelayed-template-parsing
enabled. If this is used to build a PCH all of the stored template
tokens are lost leading an incomplete AST when the PCH is used to create
the complete TU.

Is this by design or a missing feature for the serialization code?

I would guess a missing feature. Problems can be demonstrated quite
easily:

$ cat t.cpp
template<typename T>
T f(const T& t) {
    return t;
}

int main() {
    f(5);
}

$ clang --version
clang version 3.3 (tags/RELEASE_33/final)
Target: x86_64-unknown-linux-gnu
Thread model: posix

# No problems without -fdelayed-template-parsing...
$ clang -emit-ast t.cpp
$ clang -o t t.ast

# But with -fdelayed-template-parsing...
$ clang -emit-ast -fdelayed-template-parsing t.cpp
$ clang -o t t.ast
/tmp/t-YAfQmm.o: In function `main':
t.ast:(.text+0x14): undefined reference to `int f<int>(int const&)'
clang: error: linker command failed with exit code 1 (use -v to see
invocation)

It's probably a good idea to paste this into a bug report (
http://llvm.org/bugs) to make sure it's not lost.

#6

Thanks for the feedback.

I’ve started work on supporting serialization and I’ve made reasonable progress so far. The need for lazy deserialization hadn’t escaped me but I was concentrating on just getting it up and running initially - as slightly slower would be better than completely broken. I can look into lazy deserialization but I may need a few pointers on the best approach and if it’s going to be a substantial job it might have to wait until a later patch.

  • Will.
#7

It’s already filed as

http://llvm.org/bugs/show_bug.cgi?id=9992

  • Doug