c++ templates

I’m working on a set of reverse engineering tools (as in program analysis, not the illegal kind) that operate on template source code and have decided to use clang for its implementation. Imagine my surprise when I found out that clang doesn’t quite support templates. Rather than wait for somebody else to build it, I decided I would try to do this myself. How hard can it be :slight_smile:

Attached is a very (VERY) preliminary stab at the beginning of template parsing for clang. It’s not complete, it doesn’t do anything, there’s no AST stuff, I haven’t even considered scopes and name resolution, etc., etc. It just recognizes “template <…>”. I had hoped to spend some spare time, an hour or so a day (whenever possible) to continue working on this. Please bear in mind that a) I’ve never worked on a real compiler before and b) I probably don’t fully understand the requirements of what I’m working on.

I hope this is an adequate start.

Andrew Sutton
andrew.n.sutton@gmail.com

templates.diff (69.1 KB)

Andrew Sutton wrote:

Attached is a very (VERY) preliminary stab at the beginning of template parsing for clang. It's not complete, it doesn't do anything, there's no AST stuff, I haven't even considered scopes and name resolution, etc., etc. It just recognizes "template <...>". I had hoped to spend some spare time, an hour or so a day (whenever possible) to continue working on this. Please bear in mind that a) I've never worked on a real compiler before and b) I probably don't fully understand the requirements of what I'm working on.

I hate to be nitpicking instead of giving constructive comments, but a lot of the patch is just changes in whitespace. While I agree that all the trailing whitespace in the Clang source is a bit annoying, those changes make the patch very hard to read.

Sebastian

Attached is a very (VERY) preliminary stab at the beginning of template parsing for clang. It’s not complete, it doesn’t do anything, there’s no AST stuff, I haven’t even considered scopes and name resolution, etc., etc. It just recognizes “template <…>”. I had hoped to spend some spare time, an hour or so a day (whenever possible) to continue working on this. Please bear in mind that a) I’ve never worked on a real compiler before and b) I probably don’t fully understand the requirements of what I’m working on.

I hate to be nitpicking instead of giving constructive comments, but a lot of the patch is just changes in whitespace. While I agree that all the trailing whitespace in the Clang source is a bit annoying, those changes make the patch very hard to read.

I’ll lay that at the feet of my text editor. I guess I didn’t realize how much actually changed. Here’s the same diff - hopefully all the relevant parts - without the WS changes. I cut all the seemingly trivial changes by hand. Hopefully, I didn’t cut anything important.

Andrew Sutton
andrew.n.sutton@gmail.com

templates-2.diff (8.24 KB)

The benefit to zapping all of the trailing white space once is that we don't have to see the results of edits from folks that use editors that automatically trim the ends off like this.

Agreed; anyone up for doing a global trim trailing whitespace commit?

-Eli

I'd do the work, if you can convince Chris. :slight_smile: In the past, he didn't mind doing it on a file basis on files people touch, but preferred not to do the change wholesale.

The problem with this is it makes svn blame very difficult to use.

-Chris

Andrew Sutton wrote:

        Attached is a very (VERY) preliminary stab at the beginning of
        template parsing for clang. It's not complete, it doesn't do
        anything, there's no AST stuff, I haven't even considered
        scopes and name resolution, etc., etc. It just recognizes
        "template <...>". I had hoped to spend some spare time, an
        hour or so a day (whenever possible) to continue working on
        this. Please bear in mind that a) I've never worked on a real
        compiler before and b) I probably don't fully understand the
        requirements of what I'm working on.

Then you're just like me when I started on Clang :slight_smile: You'll learn, but you picked a particularly bad piece of C++ to start.

Although, parsing templates is the easy part. Parsing and expanding their use is what's really hard.

Your code looks good for a start. Of course, there's a lot missing, but what's there seems reasonable.

Sebastian

Welcome to Clang :slight_smile:

You certainly picked off a rather big piece of C++, eh? I have some
comments on your patch:

Index: test/Parser/cxx/temp/parse-err-1.cpp
Index: test/Parser/cxx/temp/parse-err-2.cpp
[etc]

Please collapse all of these parser tests into a single .cpp file
(say, test/Parser/cxx-template-decl.cpp). We like to minimize the
number of test files (within reason!) because it keeps testing fast.

Parser::DeclTy *Parser::ParseDeclaration(unsigned Context) {
   switch (Tok.getKind()) {
+ case tok::kw_export:
+ case tok::kw_template:
+ // TODO There's also the case of 'export template'. Can I look ahead?
+ return ParseTemplateDeclaration(Context);

You get look ahead with NextToken(). However, you don't need to do
that here. "export" only shows up in one place in the C++ grammar, and
always followed by "template".

Yep, please use [f]printf, or one of the other streams. <iostream> adds a static ctor to the file it is included in, so it is particularly evil. Here are some more thoughts on the matter:
http://llvm.org/docs/CodingStandards.html#ll_iostream

that said, it is probably better to just use raw_ostream with:
  llvm::outs() << foo << bar;

-Chris

Why doesn’t gmail automatically reply to the list?

know it’s just for debugging, but we’re moving away from using
iostreams for anything in Clang.

Yep, please use [f]printf, or one of the other streams. adds a static ctor to the file it is included in, so it is particularly evil. Here are some more thoughts on the matter:
http://llvm.org/docs/CodingStandards.html#ll_iostream

that said, it is probably better to just use raw_ostream with:
llvm::outs() << foo << bar;

Got it.

Andrew Sutton
andrew.n.sutton@gmail.com

* Chris Lattner:

I'd do the work, if you can convince Chris. :slight_smile: In the past, he
didn't mind doing it on a file basis on files people touch, but
preferred not to do the change wholesale.

The problem with this is it makes svn blame very difficult to use.

You could use Git's versions, which supports ignoring whitespace
changes. :sunglasses:

It shouldn't be too difficult to implement something similar for
Subversion. In fact, since Subversion doesn't store line diffs
(unlike CVS), it's probably just necessary to pass another option to
the internal diff utility.

Florian Weimer a écrit :

* Chris Lattner:

I'd do the work, if you can convince Chris. :slight_smile: In the past, he
didn't mind doing it on a file basis on files people touch, but
preferred not to do the change wholesale.

The problem with this is it makes svn blame very difficult to use.

It shouldn't be too difficult to implement something similar for
Subversion. In fact, since Subversion doesn't store line diffs
(unlike CVS), it's probably just necessary to pass another option to
the internal diff utility.

In fact, on windows, TortoiseSVN propose by default to ignore line ending change and whitespace change for blame. so it clearly can be implemented in the client.

  (and it show the superiority of windows tools over those of minor OS </troll>) more seriously, TortoiseSVN is a really good svn client.

svn already has that since 1.4.0 (that's over two years ago)

Attached is a very (VERY) preliminary stab at the beginning of template parsing for clang. It’s not complete, it doesn’t do anything, there’s no AST stuff, I haven’t even considered scopes and name resolution, etc., etc. It just recognizes “template <…>”. I had hoped to spend some spare time, an hour or so a day (whenever possible) to continue working on this. Please bear in mind that a) I’ve never worked on a real compiler before and b) I probably don’t fully understand the requirements of what I’m working on.

I hope this is an adequate start.

What should I be doing with the templates patch? Its growing steadily larger and I’ll probably get to a point soon where I have to step on a lot of code to get anything else built.

Andrew Sutton
andrew.n.sutton@gmail.com

Andrew Sutton wrote:

    Attached is a very (VERY) preliminary stab at the beginning of
    template parsing for clang. It's not complete, it doesn't do
    anything, there's no AST stuff, I haven't even considered scopes
    and name resolution, etc., etc. It just recognizes "template
    <...>". I had hoped to spend some spare time, an hour or so a day
    (whenever possible) to continue working on this. Please bear in
    mind that a) I've never worked on a real compiler before and b) I
    probably don't fully understand the requirements of what I'm
    working on.

    I hope this is an adequate start.

What should I be doing with the templates patch? Its growing steadily larger and I'll probably get to a point soon where I have to step on a lot of code to get anything else built.

When it's in a state where it has some closure (e.g. parsing declarations works, even if you can't actually use those parameters and sema never hears of it or treats it as non-templates), and you've written a few test cases, submit it for review. It has to do something useful (at least for testability), it should not crash the compiler under any circumstances.

I think a big discussion is in order before touching sema of templates anyway. We have to plan the template engine.

Sebastian

Andrew Sutton wrote:

    Attached is a very (VERY) preliminary stab at the beginning of
    template parsing for clang. It's not complete, it doesn't do
    anything, there's no AST stuff, I haven't even considered scopes
    and name resolution, etc., etc. It just recognizes "template
    <...>". I had hoped to spend some spare time, an hour or so a day
    (whenever possible) to continue working on this. Please bear in
    mind that a) I've never worked on a real compiler before and b) I
    probably don't fully understand the requirements of what I'm
    working on.

    I hope this is an adequate start.

What should I be doing with the templates patch? Its growing steadily
larger and I'll probably get to a point soon where I have to step on a
lot of code to get anything else built.

When it's in a state where it has some closure (e.g. parsing
declarations works, even if you can't actually use those parameters and
sema never hears of it or treats it as non-templates), and you've
written a few test cases, submit it for review. It has to do something
useful (at least for testability), it should not crash the compiler
under any circumstances.

Exactly. Then one of the main Clang developers will merge your patch
into Subversion. After a

I think a big discussion is in order before touching sema of templates
anyway. We have to plan the template engine.

Yes, that's a big chunk of design work. I expect that it'll happen
when a few people get together to come up with a coherent design and
prototype, then bring it to the larger Clang community for review and
ideas.

  - Doug

What should I be doing with the templates patch? Its growing steadily
larger and I’ll probably get to a point soon where I have to step on a
lot of code to get anything else built.

When it’s in a state where it has some closure (e.g. parsing
declarations works, even if you can’t actually use those parameters and
sema never hears of it or treats it as non-templates), and you’ve
written a few test cases, submit it for review. It has to do something
useful (at least for testability), it should not crash the compiler
under any circumstances.

That’s basically what I’m working towards. It’s coming a long nicely.

Exactly. Then one of the main Clang developers will merge your patch
into Subversion. After a

I think a big discussion is in order before touching sema of templates
anyway. We have to plan the template engine.

Yes, that’s a big chunk of design work. I expect that it’ll happen
when a few people get together to come up with a coherent design and
prototype, then bring it to the larger Clang community for review and
ideas.

That sounds like a good plan. Do you know of any good references on the design and implementation of template engines floating around (besides code?). I’m curious to see what other people have done.

Is there any way I can get write access to a branch? It would definitely help me out since I tend to work on 3-5 different operating systems, and propagating patches back and forth isn’t a lot of fun. It would also add a nice central point for other people interested in working on template stuff.

Andrew Sutton
andrew.n.sutton@gmail.com

Andrew Sutton wrote:

Is there any way I can get write access to a branch?

Chris might give you write access to the trunk. As I understand the development guidelines, we don't use branches. If you have to migrate code a lot, I recommend you simply overlay your local copy with a git or mercurial repository and use that for syncing.

Sebastian

Yep, I'm happy to give commit-after-approval access to just about anyone who has contributed a few patches. Please read and agree to this document and send me the requested info:
http://llvm.org/docs/DeveloperPolicy.html

Sebastian's right about branches, they are a definite no-go unless there is a very good reason for them.

-Chris