Indexer Library and type references....

Hello,

I have a question about the indexer library and the index-test code…

Given code such as:


typedef int myType;

myType var1;

myType func1(myType *parm1)
{
return *parm1 / var1;
}


The indexer utility will correctly show references to parm1 within func1.

What I am interested in is providing the source location for “myType” on the initial typedef statement and then extracting all the references to myType such as var1, the function return type and parm1 type, as well as cast expression references…

At present I can do most of this by simply refering to the type assigned to the declaration but I am effectively building my own index rather than simply “findreferences” from the indexer library.

Is this functionality already in the indexer library and I am extremely dense? :slight_smile:


Also, can someone explain the subtle differences between a Source Location and an Instantiation Location as in:


<br>SourceLocation Loc = D->getLocation();<br>Loc = SM->getInstantiationLoc(Loc);<br><br>

Thanks,

Daniel Chapiesky





|

I'm pretty sure this is already in there. GIve it a try, if you don't get the result you expect, tell us what functions you're using the get the data and the people that know what functions you should be using instead will chime in.

Some details are here:
http://clang.llvm.org/docs/InternalsManual.html#SourceLocation

-Chris

Hi Daniel,

LOL… Thanks Argiris… I was pulling my hair out trying to divine if it was and that I was in fact incredibly dense.

The whole Clang layered library approach is absolute genius. Kudos. In two weeks, from initial download, to now, I have a better understanding of how Clang works than 5 years of digging in a certain other compiler’s internals… I was a big proponent of an xml output project for that compiler and yet with Clang, in the past two weeks, I have produced more detailed information than I was ever able to with that other project. Yay!

As for tracking types… What I am going to do is:

Let the indexer find all typedefs, classes, etc… which does produce output in the index-test --find-decls command… (reminder… I wanted --find-refs to work for types)

Let the indexer find all variables and functions
for each entity
get the QualType of the declaration
add the entity to the reference map of the given type

The only other issue would be a full AST walk looking for casts.

Thanks again and I will post my results soon.

Daniel

“In practice, the SourceLocation works together with the SourceManager class to encode two pieces of information about a location: it’s spelling location and it’s instantiation location. For most tokens, these will be the same. However, for a macro expansion (or tokens that came from a _Pragma directive) these will describe the location of the characters corresponding to the token and the location where the token was used (i.e. the macro instantiation point or the location of the _Pragma itself).”

So for someone, who might be working on rewriting code and was building a list of references, it would appear as if a number of duplicate references existed using simply the SourceLocation. This is fine for the rewriting bit, but for statistics on the actual number of references, etc…, the instantiation location provides the uniqueness needed…

Thanks.
Daniel

Type references in declarations are now tracked by the Indexer library.
Note, though, that type references in expressions are not tracked because type source information for exprs (like sizeof/alignof) is not preserved in the AST yet.

I’d appreciate any feedback that you may have!

-Argiris



— On Tue, 9/29/09, Argyris Kyrtzidis akyrtzi@gmail.com wrote:


> Type references in declarations are now tracked by the Indexer library.
> Note, though, that type references in expressions are not tracked because type source information for exprs (like sizeof/alignof) is not preserved in the AST yet.
>
> I’d appreciate any feedback that you may have!
>
> -Argiris

Argiris,

Yay! That is great news! I will get the latest svn today…

Still, for my purposes, every reference needs to be preserved. In order to do massive refactoring of the original source code such as shifting namespaces left or right, even the sizeof() would be needed… I will give some better feedback next week.

In the mean time, could not a sizeof() be treated like a cast in the AST?

Daniel





|

Type references in declarations are now tracked by the Indexer library.
Note, though, that type references in expressions are not tracked because type source information for exprs (like sizeof/alignof) is not preserved in the AST yet.

I'd appreciate any feedback that you may have!

-Argiris
Argiris,

Yay! That is great news! I will get the latest svn today...

Still, for my purposes, _every_ reference needs to be preserved. In order to do massive refactoring of the original source code such as shifting namespaces left or right,

Shifting namespaces sounds like a syntactic transformation rather than a transformation that needs to work on the AST.

even the sizeof() would be needed..... I will give some better feedback next week.

In the mean time, could not a sizeof() be treated like a cast in the AST?

Yes, it can. For every place that a type shows up in the expression or statement grammar, the corresponding Expr or Stmt node will need to maintain type location information (probably in a TypeSpecLoc). As far as I can tell, all of the infrastructure is there, but someone will need to go through and add this information for us to maintain all type source information for expressions and statements.

  - Doug



— On M

> Shifting namespaces sounds like a syntactic transformation rather than a transformation that needs to work on the AST.

Yes… It is syntactic… I am working on a refactoring tool (ala Netbeans Refactor/Rename) and I am using the Clang AST and indexer library to produce source locations and references not only against the primary project you were working on, but also within projects which depend upon the primary one…

For example… (this is a very simple example and not necessarily a good one at that…)

You have a large project which has been code reviewed and regression tested to within an inch of it’s life. It uses zlib 1.1.0 and does a fine job.

References to zlib are sprinkled throughout hundreds of modules and locations.

Now, a new version of zlib is available: 1.2.3 and you need it’s functionality for an additional module being created in the project.

By massively refactoring the source code of zlib 1.2.3 to a new namespace (not C++ , just plain vanilla C) so that you get something like zlib_123_deflate(), you will not namespace collisions between the old and new zlib when linking.

We have preserved our code reviewed source code and we have ensured that there is no weird stuff going on during the link cycle.

Anyway, preserving code reviewed code is the primary purpose of my refactor tool…

I will be making a beta available to the group in a couple of weeks with a cute wxwidgets front end.







> …but someone will need to go through and add this information for us to maintain all type source information for expressions and statements.
>
> - Doug

That will probably be me… :slight_smile:

Daniel



|

--- On M

Shifting namespaces sounds like a syntactic transformation rather than a transformation that needs to work on the AST.

Yes... It is syntactic... I am working on a refactoring tool (ala Netbeans Refactor/Rename) and I am using the Clang AST and indexer library to produce source locations and references not only against the primary project you were working on, but also within projects which depend upon the primary one...

For example... (this is a very simple example and not necessarily a good one at that...)

You have a large project which has been code reviewed and regression tested to within an inch of it's life. It uses zlib 1.1.0 and does a fine job.

References to zlib are sprinkled throughout hundreds of modules and locations.

Now, a new version of zlib is available: 1.2.3 and you need it's functionality for an additional module being created in the project.

By massively refactoring the source code of zlib 1.2.3 to a new namespace (not C++ , just plain vanilla C) so that you get something like zlib_123_deflate(), you will not namespace collisions between the old and new zlib when linking.

We have preserved our code reviewed source code and we have ensured that there is no weird stuff going on during the link cycle.

Anyway, preserving code reviewed code is the primary purpose of my refactor tool...

That makes sense. Clang AST + Indexer library were meant to support this kind of transformation; I hope it's working out for you.

I will be making a beta available to the group in a couple of weeks with a cute wxwidgets front end.

Cool.

...but someone will need to go through and add this information for us to maintain all type source information for expressions and statements.

    - Doug

That will probably be me... :slight_smile:

Great!

  - Doug