Tooling vs -fdelayed-template-parsing WAS: Clang-cl.exe and the VC++ preprocessor

Clang Frontend
1

We're struggling with delayed template parsing in our IWYU tool, so this
caught my eye.

This fragment:

  #include "tests/cxx/direct.h"

  template <typename T> void foo() {
      IndirectClass ic;
  }

(from https://code.google.com/p/include-what-you-use/issues/detail?id=129)

will have IWYU suggest removal of the #include in MSVC-compatible mode only
(because of -fdelayed-template-parsing). In 'normal' mode, the template
gets an AST built for it, and the IndirectClass use is properly detected.

We've considered always running IWYU with -fno-delayed-template-parsing,
but that would probably make parsing of Microsoft system headers fail
entirely.

Are we stuck here? Or is there a workaround for visiting template bodies
even under -fdelayed-template-parsing?

Clang-modernize seems to have run into the same problem, so this might
become more of a problem as tools start supporting Windows:
http://clang.llvm.org/extra/PassByValueTransform.html#note-about-delayed-template-parsing

So your approach sounds interesting, to treat delayed template parsing as
an error path. It would fix the vast majority of these RAV problems in
portable codebases, and code that relies on delayed parsing would still
compile, even if it would be silently missed in RAV.

- Kim

2

[...] I don't think we're getting a lot of value out of
-fdelayed-template-parsing. The only thing we get out of it is the ability
to do unqualified lookup of names that haven't been declared yet in the
global namespace. Because we still use our TreeTransform approach to
template instantiation, most of the parsing difficulty is handled by
-fms-compatibility hacks and not -fdelayed-template-parsing. The MS
compatibility hacks are actually better because they usually appear in
error codepaths, meaning they don't affect compilation performance and are
easy to diagnose with a warning, instead of -fdelayed-template-parsing
where we can't know if the user relies on it.

We're struggling with delayed template parsing in our IWYU tool, so this
caught my eye.

This fragment:

  #include "tests/cxx/direct.h"

  template <typename T> void foo() {
      IndirectClass ic;
  }

(from https://code.google.com/p/include-what-you-use/issues/detail?id=129)

will have IWYU suggest removal of the #include in MSVC-compatible mode
only (because of -fdelayed-template-parsing). In 'normal' mode, the
template gets an AST built for it, and the IndirectClass use is properly
detected.

We've considered always running IWYU with -fno-delayed-template-parsing,

but that would probably make parsing of Microsoft system headers fail
entirely.

Are we stuck here? Or is there a workaround for visiting template bodies
even under -fdelayed-template-parsing?

Hypothetically you could record every late parsed function decl
(FD->isLateTemplateParsed()) and call LateTemplateParser on it at the end
of the TU. This would get you the uninstantiated template AST, which is
what you want. I make no guarantees, though. :slight_smile:

3

Hi Reid,

4

I think parsing a late parsed function body has the side effect of making
it not late parsed, so it should never be visited twice.

5

That makes sense, and seems to be the case.

Thanks for your help!

- Kim

6

I'm guessing this introduces a risk for double-visiting FunctionDecls
that _do_ become instantiated; can I somehow check the FDs before
forcefully instantiating them? Not that I think it makes a big difference,
it just seems hygienic not to late-parse and revisit stuff that's already
been recorded.

I think parsing a late parsed function body has the side effect of making
it not late parsed, so it should never be visited twice.

That makes sense, and seems to be the case.

Can we somehow get this into the tooling infrastructure? I'd consider it a
bug if fdelayed-template-parsing leads to ASTs that don't have the
templates; from reading the docs, it seems like all it should do is parse
them at the end of the translation unit, or am I missing something?

Cheers,
/Manuel

7

Hi Manuel,

8

+richard

why are those not instantiated when a FrontendAction is run?

9

why are those not instantiated when a FrontendAction is run?

My choice of words was bad -- the template is not instantiated here, merely
parsed. We don't have any template arguments to give it.

We just (a few minutes ago, while experimenting with this) found a case
where late-parsing a template whose body contains undefined symbols would
generate an incomplete AST, e.g.

template<class T>
void foo(const T& t) {
  UndefinedSymbol::bar(t); // colon-colon
}

$ clang-check.exe -ast-dump incomplete.cpp -- -fno-delayed-template-parsing
TranslationUnitDecl 0xebe7e0 <<invalid sloc>> <invalid sloc>

-CXXRecordDecl 0xebeaa0 <<invalid sloc>> <invalid sloc> implicit class

type_info

-TypedefDecl 0xebeb20 <<invalid sloc>> <invalid sloc> implicit size_t

'unsigned int'

-TypedefDecl 0xebeb80 <<invalid sloc>> <invalid sloc> implicit

__builtin_va_list 'char *'
`-FunctionTemplateDecl 0xebed80 <incomplete.cpp:2:1, line:5:1> line:3:6 foo
  >-TemplateTypeParmDecl 0xebebb0 <line:2:10, col:16> col:16 class T
  `-FunctionDecl 0xebed10 <line:3:1, line:5:1> line:3:6 foo 'void (const T
&)'
    >-ParmVarDecl 0xebec80 <col:10, col:19> col:19 referenced t 'const T &'
    `-CompoundStmt 0xebee40 <col:22, line:5:1>
      `-CallExpr 0xebee20 <line:4:3, col:25> '<dependent type>'
        >-DependentScopeDeclRefExpr 0xebede0 <col:3, col:20> '<dependent

' lvalue

        `-DeclRefExpr 0xebee04 <col:24> 'const T' lvalue ParmVar 0xebec80
't' 'const T &'

This runs without error, while this:

template<class T>
void foo(const T& t) {
  UndefinedSymbol.bar(t); // dot
}

throws an error, as expected:

  incomplete.cpp:4:4: error: use of undeclared identifier 'UndefinedSymbol'
     UndefinedSymbol.bar(t);
     ^

I don't know if this is a general bug in the parser, but it'd be nice to
always get a diagnostic here.

- Kim

10

This is an -fms-compatibility hack, which is separate from delayed template
parsing. I'm surprised we don't have a -Wmicrosoft warning here, though.

11

Thanks, that explains it. I'm about to take off on a road-trip, or I'd
offer to add a warning. If you can give me a hint as to where to start I
may be able to get to it when we get back.

- Kim

12

I don't know where exactly this is happening, but we typically build
DependentDeclRefExprs when we have an undeclared id in a template in
-fms-compatibility mode. The name isn't actually dependent on template
arguments, but it allows us to delay lookup until template instantiation
time, which is basically what MSVC does.

13

Hi Reid,

14

There's a patch up for the other thing here:
http://reviews.llvm.org/D4854

If that's acceptable, I'd also like to add a -Wmicrosoft warning. I
already have a local patch that seems to trigger a warning in the
right places. Would you rather I put that into the same patch?

Thanks,
- Kim