Question about ASTMatchers

If I match a varDecl() that is in a method of a template class I get a match for the template and instantiated templates

  • is there a way to tell the difference by matching some difference in the ancestors of the varDecl node?

I get a match for the translate::from_object a0(*args); varDecl in the method activate (see below) that has this type:
from_object<type-parameter-0-0, struct std::__1::integral_constant<_Bool, true> >

I also get a match that I think is from the instantiation TestFunctoid j(“test”) in tinyFunc - it has this type: “struct translate::from_object<int, std::true_type>”

There is a clear difference in the name - but I was looking for something that didn’t feel like a hack.

Are there differences in the AST class node for TestFunctoid when it is a template class vs when it is an instantiated template class?

#include <stdio.h>
#include <core/foundation.h>
#include <core/object.h>
#include <core/holder.h>

namespace asttooling {

template
class TestFunctoid : public core::Functoid {
public:

TestFunctoid(const string& name) : core::Functoid(name) {};
core::T_mv activate( core::const_ActivationFrame_spREF closedOverFrame, int numArgs, ArgArray args )
{
translate::from_object a0(*args);
printf( “Address of a0= %p\n”, &a0);
return Values0core::T_O();
}
};

void tinyFunc()
{
TestFunctoid j(“test”);
TinyStruct x(10);
tinyFunc(x);
printf(“Hi there, this is tinyFunc\n”);
}

};

Christian Schafmeister
Associate Professor
Chemistry Department
Temple University

In article <5C28B1F5-CDBF-4ABD-BF60-F70FB2AF6DFF@verizon.net>,
    Christian Schafmeister <chris.schaf@verizon.net> writes:

If I match a varDecl() that is in a method of a template class I get a
match for the template and instantiated templates
- is there a way to tell the difference by matching some difference in the
ancestors of the varDecl node?

Did you try using any of these:
- classTemplateDecl
- classTemplateSpecializationDecl
- unless(isTemplateInstantiation)

What does the ast dump say for your case? (I'm away from my clang
tools right now.)

clang -Xclang -ast-dump

#include <stdio.h>
#include <core/foundation.h>
#include <core/object.h>
#include <core/holder.h>

namespace asttooling {

    template<typename T>
    class TestFunctoid : public core::Functoid {
    public:

        TestFunctoid(const string& name) : core::Functoid(name) {};
        core::T_mv activate( core::const_ActivationFrame_spREF
closedOverFrame, int numArgs, ArgArray args )
        {
            translate::from_object<T> a0(*args);
            printf( "Address of a0= %p\n", &a0);
            return Values0<core::T_O>();
        }
    };

    void tinyFunc()
    {
        TestFunctoid<int> j("test");
        TinyStruct x(10);
        tinyFunc(x);

          ^^^^^^^^^^^^
tinyFunc wasn't declared as taking any arguments...

        printf("Hi there, this is tinyFunc\n");
    }

};

Skipping over everything from the include files, I got this from
ast-dump:

-NamespaceDecl 0x2d24a90 </tmp/foo.cpp:6:1, line:30:1> asttooling
>-ClassTemplateDecl 0x2d24c30 <line:8:5, line:20:5> TestFunctoid
> >-TemplateTypeParmDecl 0x2d24af0 <line:8:14, col:23> typename T
> >-CXXRecordDecl 0x2d24ba0 <line:9:5, line:20:5> class TestFunctoid definition invalid
> > >-CXXRecordDecl 0x2d24eb0 <line:9:5, col:11> class TestFunctoid
> > >-AccessSpecDecl 0x2d24f40 <line:10:5, col:11> public
> > >-CXXConstructorDecl 0x2d25090 <line:12:9, col:66> TestFunctoid<T> 'void (const int &)'
> > > >-ParmVarDecl 0x2d24fb0 <col:22, col:36> name 'const int &' invalid
> > > `-CompoundStmt 0x2d26458 <col:65, col:66> invalid
> > `-CXXMethodDecl 0x2d26370 <line:13:9, line:19:9> activate 'int (int, int, int)'
> > >-ParmVarDecl 0x2d25160 <line:13:30, line:14:1> closedOverFrame 'int' invalid
> > >-ParmVarDecl 0x2d26220 <col:18, col:22> numArgs 'int'
> > >-ParmVarDecl 0x2d26290 <col:31, col:40> args 'int' invalid
> > `-CompoundStmt 0x2d265d8 <line:15:9, line:19:9> invalid
> `-ClassTemplateSpecializationDecl 0x2d266f0 <line:8:5, line:20:5> class TestFunctoid definition
> >-TemplateArgument type 'int' invalid
> >-CXXRecordDecl 0x2d26a40 prev 0x2d266f0 <line:9:5, col:11> class TestFunctoid
> `-AccessSpecDecl 0x2d26ad0 <line:10:5, col:11> public
`-FunctionDecl 0x2d26640 <line:22:5, line:28:5> tinyFunc 'void (void)'
  `-CompoundStmt 0x2d26e38 <line:23:5, line:28:5>
    >-DeclStmt 0x2d26af8 <line:24:9, col:36>
    > `-VarDecl 0x2d268a0 <col:9, col:27> j 'TestFunctoid<int>':'class asttooling::TestFunctoid<int>' invalid
    >-DeclStmt 0x2d26bc8 <line:25:9, col:25>
    > `-VarDecl 0x2d26b20 <col:9, col:20> x 'int' invalid
    `-CallExpr 0x2d26df0 <line:27:9, col:46> 'int'
      >-ImplicitCastExpr 0x2d26dd8 <col:9> 'int (*)(const char *restrict, ...)' <FunctionToPointerDecay>
      > `-DeclRefExpr 0x2d26d50 <col:9> 'int (const char *restrict, ...)' lvalue Function 0x2d15500 'printf' 'int (const char *restrict, ...)'
      `-ImplicitCastExpr 0x2d26e20 <col:16> 'const char *' <ArrayToPointerDecay>
        `-StringLiteral 0x2d26d08 <col:16> 'const char [28]' lvalue "Hi there, this is tinyFunc\n"

`-EmptyDecl 0x2d26e70 <line:30:2>

hasAncestor(decl(anyOf(recordDecl(
isTemplateInstantiation()),
functionDecl(isTemplateInstantiation()))

should do what you want.

Manuel,

I completely missed those predicates - thank you!

How do I get the full name of an instantiated template recordDecl?

As in:
template class X;
class A;

X xa;

If a varDecl(…) matches on xa - how do I get the "X” name?

Best,

.Chris.

Christian Schafmeister
chris.schaf@verizon.net

Manuel,

I completely missed those predicates - thank you!

How do I get the full name of an instantiated template recordDecl?

As in:
template <class T> class X;
class A;

X<A> xa;

If a varDecl(…) matches on xa - how do I get the "X<A>” name?

Do you want to "get" it or "match" it. Assuming you want to "get" it, you'd
say something like varDecl(hasType(qualType().bind("t"))) (untested ;), and
then in the callback extract the QualType and call getAsString() on it.

In article <CAOsfVvnnO-6VRGLdzWuvp-Qb1qR61wAqJ_d8vHX-LgyWTa6u+g@mail.gmail.com>,
    Manuel Klimek <klimek@google.com> writes:

[...] Assuming you want to "get" it, you'd
say something like varDecl(hasType(qualType().bind("t"))) (untested ;), and
then in the callback extract the QualType and call getAsString() on it.

Thanks for these responses, Manuel! I'm still learning these matchers
and there's quite a bit to digest there. These suggestions have been
very instructive!

In article <
CAOsfVvnnO-6VRGLdzWuvp-Qb1qR61wAqJ_d8vHX-LgyWTa6u+g@mail.gmail.com>,
    Manuel Klimek <klimek@google.com> writes:

> [...] Assuming you want to "get" it, you'd
> say something like varDecl(hasType(qualType().bind("t"))) (untested ;),
and
> then in the callback extract the QualType and call getAsString() on it.

Thanks for these responses, Manuel! I'm still learning these matchers
and there's quite a bit to digest there. These suggestions have been
very instructive!

You might want to take a look at clang-query, which helps with
experimenting with matchers quickly (no compile needed, it uses the dynamic
matchers).

In article <CAOsfVvmovo0mC4dX=7Q9dWEno2BK6V6NiMxbkg0Nz9t+RUSjCg@mail.gmail.com>,
    Manuel Klimek <klimek@google.com> writes:

You might want to take a look at clang-query, which helps with
experimenting with matchers quickly (no compile needed, it uses the dynamic
matchers).

Is there documentation on this? I couldn't find anything except some
comments on the commit message.

I’m creating AST Matchers to match recordDecls and varDecls and match the types for the varDecls to recordDecls.
In many cases the varDecls are instantiating template classes and some of the template classes are specialized.

If I use this matcher:
recordDecl(isTemplateInstantiation(), isSameOrDerivedFrom(recordDecl(hasName(“BAR”)))).bind(“WHOLE”) —> FOO node
or

recordDecl(isExplicitTemplateSpecialization(), isSameOrDerivedFrom(recordDecl(hasName(“BAR”)))).bind(“WHOLE”) —> FOO node

How do I get the full name of the nodes that match and are bound to “WHOLE”?

node.getName() —> “FOO” not “FOO” or “FOO"

Here is the sample code:

#include <stdio.h>
#include <core/foundation.h>
#include <core/object.h>
#include <core/holder.h>

namespace asttooling {

class BAR {};

template class FOO : public BAR {};
class A {};

template<> class FOO : public BAR {};

void tinyFunc()
{
FOO
xa;
FOO xi;
printf(“Hi there, this is tinyFunc @%p %p\n”, &xa, &xi);
}

};

Christian Schafmeister
chris.schaf@verizon.net

I’m creating AST Matchers to match recordDecls and varDecls and match the
types for the varDecls to recordDecls.
In many cases the varDecls are instantiating template classes and some of
the template classes are specialized.

If I use this matcher:
recordDecl(isTemplateInstantiation(),
isSameOrDerivedFrom(recordDecl(hasName("BAR")))).bind(“WHOLE”) —> FOO<A>
node
or
recordDecl(isExplicitTemplateSpecialization(),
isSameOrDerivedFrom(recordDecl(hasName("BAR")))).bind(“WHOLE”) —>
FOO<int> node

How do I get the full name of the nodes that match and are bound to
“WHOLE”?

node.getName() —> “FOO” not “FOO<A>” or “FOO<int>"

I don't have a good answer. Generally, getName() (or getNameAsString())
will print the full declaration for anything that's a declaration context.
So if you match a member m in FOO, you'd get FOO<int>::m. This will be
expanded for everything but the last part.
I don't know whether there's a good way to get the expansion for
everything. I've never needed it, though; out of curiosity, what do you
need it for?

In article <CAOsfVvmovo0mC4dX=
7Q9dWEno2BK6V6NiMxbkg0Nz9t+RUSjCg@mail.gmail.com>,
    Manuel Klimek <klimek@google.com> writes:

> You might want to take a look at clang-query, which helps with
> experimenting with matchers quickly (no compile needed, it uses the
dynamic
> matchers).

Is there documentation on this? I couldn't find anything except some
comments on the commit message.

+peter, who wrote it.

Dear Manuel,

Thank you - after banging my head against the wall for a couple of days I realized that I needed to extract the template arguments from the ClassTemplateSpecializationDecl myself using ClassTemplateSpecializationDecl::getTemplateArgs

I’m writing a program that digs through the 165 C++ source files that is the C++ core of my Common Lisp system and analyzes classes and variables to construct the interface to a copying, generational garbage collector (the Memory-Pool-System (MPS) by Ravenbrook systems). I’m working on adding this garbage collector to my Common Lisp system and I don’t want to do it by hand because it would be a huge job to write and maintain. I have hundreds of classes/instantiated template classes and thousands of variable declarations that need to be matched to each other in the analysis.

For instance:

class Bar {…};
class Foo {
smart_ptr _bar;
};

void func() {
Foo a; ;; ← Nothing needs to be defined as a root because smart_ptr is on the stack and MPS will find it there.
Foo* b = new Foo(); ;; <— MPS needs to be informed that this is a root because the *b is on the heap and the smart_ptr b->_bar is on the heap.
}

So I need to analyze every class (matcher: recordDecl(…) ) like Foo and classify them whether they store pointers on the heap for the situation where an instance is stored on the stack or the heap. Then I use that info to construct the garbage collector interface or print warnings so that I can go in and by hand and fix to be compatible with the garbage collector.

I also need to identify every varDecl() and fieldDecl and match them up to their respective RecordDecl.

I’ve been struggling to construct a unique key for each RecordDecl given that I have CXXRecordDecls and instantiated ClassTemplateSpecializationDecls all over the place and I need to generate matching keys from the type of each varDecl.

I can’t use the AST node directly because I’m scanning 165 files and I can’t load all of the AST’s into memory at once - that requires tens of gigabytes and brings my machine to it’s knees. Once one file is processed, its AST nodes get destroyed so I need to generate keys based on the RecordDecl’s names.

I figured it out though - I wrote these functions (Common Lisp) that generate the name of each RecordDecl by checking what kind of RecordDecl it is and doing different things based on that. If it’s a ClassTemplateSpecializationDecl it gets the arguments using getTemplateArgs and builds a list of argument names and attaches it to the name of the template class.

(defun template-arg-as-string (template-arg)
(let* ((template-arg-kind (cast:get-kind template-arg)))
(case template-arg-kind
(ast-tooling:type
(cast:get-as-string (cast:get-as-type template-arg)))
(ast-tooling:integral
(format nil “INTEGRAL[~a]” (llvm:to-string (cast:get-as-integral template-arg) 10 t)))
(otherwise
(error “Add support for template-arg-as-string of kind: ~a” template-arg-kind)))))

(defun record-key (decl-node)
(or decl-node (error “There is a problem, the decl-node is nil”))
(case (type-of decl-node)
(cast:cxxrecord-decl
(format nil “class_~a” (cast:get-name decl-node)))
(cast:record-decl
(format nil “struct_~a” (cast:get-name decl-node)))
(cast:enum-decl
(format nil “enum_~a” (cast:get-name decl-node)))
(cast:class-template-specialization-decl
(let* ((decl-name (cast:get-name decl-node))
(template-args (cast:get-template-args decl-node))
(template-args-as-list (loop :for i :from 0 :below (cast:size template-args)
:collect (let* ((template-arg (cast:template-argument-list-get template-args i)))
(template-arg-as-string template-arg)))))
(format nil “class-template-specialization-decl_~a<~{~a~^,~}>” (cast:get-name decl-node) template-args-as-list)))
(cast:class-template-partial-specialization-decl
(let* ((decl-name (cast:get-name decl-node))
(template-args (cast:get-template-args decl-node))
(template-args-as-list (loop :for i :from 0 :below (cast:size template-args)
:collect (cast:get-as-string (cast:get-as-type (cast:template-argument-list-get template-args i))))))
(format nil “class-template-partial-specialization-decl_~a<~{~a~^,~}>” (cast:get-name decl-node) template-args-as-list)))
(otherwise
(format t “Add support for record-key for ~a get-name->~a~%” decl-node (cast:get-name decl-node))
(break “Check the decl-node”))))

Christian Schafmeister
chris.schaf@verizon.net

Sorry, but there's no real documentation yet. I hope to get to this soon. But
once you've launched the tool you can use the 'help' command to see the full
list of commands. You can also use the tab key at any time (including while
inputing a matcher) for context sensitive autocompletion.

Thanks,

Dear Manuel,

Thank you - after banging my head against the wall for a couple of days I
realized that I needed to extract the template arguments from the
ClassTemplateSpecializationDecl myself using
ClassTemplateSpecializationDecl::getTemplateArgs

Btw, I think a patch that factors out the code to print the fully qualified
name including the "expansion" of the last declcontext might be a good idea
to get into NamedDecl anyway.

I’m writing a program that digs through the 165 C++ source files that is
the C++ core of my Common Lisp system and analyzes classes and variables to
construct the interface to a copying, generational garbage collector (the
Memory-Pool-System (MPS) by Ravenbrook systems). I’m working on adding
this garbage collector to my Common Lisp system and I don’t want to do it
by hand because it would be a huge job to write and maintain. I have
hundreds of classes/instantiated template classes and thousands of variable
declarations that need to be matched to each other in the analysis.

For instance:

class Bar {…};
class Foo {
smart_ptr<Bar> _bar;
};

void func() {
    Foo a; ;; <- Nothing needs to be defined as a root because
smart_ptr<Bar> is on the stack and MPS will find it there.
    Foo* b = new Foo(); ;; <— MPS needs to be informed that this is a
root because the *b is on the heap and the smart_ptr<Bar> b->_bar is on the
heap.
}

So I need to analyze every class (matcher: recordDecl(…) ) like Foo and
classify them whether they store pointers on the heap for the situation
where an instance is stored on the stack or the heap. Then I use that info
to construct the garbage collector interface or print warnings so that I
can go in and by hand and fix to be compatible with the garbage collector.

I also need to identify every varDecl() and fieldDecl and match them up to
their respective RecordDecl.

I’ve been struggling to construct a unique key for each RecordDecl given
that I have CXXRecordDecls and instantiated
ClassTemplateSpecializationDecls all over the place and I need to generate
matching keys from the type of each varDecl.

There are multiple ways to create a key here:
a) use USRs (http://clang.llvm.org/doxygen/namespaceclang_1_1index.html);
if you want to have source code independent keys, I'd always prefer USRs
above creating some string representation yourself
b) use source locations expanded as string (that's actually what we do for
large scale refactorings since a couple of years now, and it works very
well)