Source-to-source translation using LibTooling

Hello everybody,

I have successfully written a source-to-source translator using LibTooling, and I am now willing to go further and do some source-to-source translation.
However, I can't figure out how to output modified code with LibTooling.

I have used in the past the class "Rewriter", coupled with a "RewriterBuffer" to do this job, but without the LibTooling (I was manipulating "CompilerInstance" object directly).
However, a "RewriterBuffer" requires a file ID (that I used to get from a "SourceManager") to be created, and I don't have access to such information anymore, because I'm calling my tool using "Tool.run" from a custom "FrontEndAction".

I've spotted the example called "loop-convert" in the clang-tools-extra repository, that is using class derived from "RefactoringTool" to do source-to-source transformation. I kind of made sense of this code, but the way it works would require me to change my code deeply in order to fill the so-called replacement list.

Does someone has any experience on this matter in order to give some advices on the best approach I should adopt ?

Best regards,

- Antoine

Hello everybody,

I have successfully written a source-to-source translator using
LibTooling, and I am now willing to go further and do some source-to-source
translation.
However, I can't figure out how to output modified code with LibTooling.

I have used in the past the class "Rewriter", coupled with a
"RewriterBuffer" to do this job, but without the LibTooling (I was
manipulating "CompilerInstance" object directly).
However, a "RewriterBuffer" requires a file ID (that I used to get from a
"SourceManager") to be created, and I don't have access to such information
anymore, because I'm calling my tool using "Tool.run" from a custom
"FrontEndAction".

I've spotted the example called "loop-convert" in the clang-tools-extra
repository, that is using class derived from "RefactoringTool" to do
source-to-source transformation. I kind of made sense of this code, but the
way it works would require me to change my code deeply in order to fill the
so-called replacement list.

Does someone has any experience on this matter in order to give some
advices on the best approach I should adopt ?

I'm not 100% sure what you're looking for - as you said, the
RefactoringTool is what's meant to be used to write out changed code - I'm
open to changes to that interface if it helps your use case, but I don't
know your code, so it's hard for me to tell exactly why that would be a
deep change, or how to fit it into your project.

My intuition would be that at the point at which you would make changes vie
the Rewriter, you would instead fill the Replacements...

Cheers,
/Manuel

Hello everybody,

I have successfully written a source-to-source translator using LibTooling, and I am now willing to go further and do some source-to-source translation.
However, I can't figure out how to output modified code with LibTooling.

I have used in the past the class "Rewriter", coupled with a "RewriterBuffer" to do this job, but without the LibTooling (I was manipulating "CompilerInstance" object directly).
However, a "RewriterBuffer" requires a file ID (that I used to get from a "SourceManager") to be created, and I don't have access to such information anymore, because I'm calling my tool using "Tool.run" from a custom "FrontEndAction".

I've spotted the example called "loop-convert" in the clang-tools-extra repository, that is using class derived from "RefactoringTool" to do source-to-source transformation. I kind of made sense of this code, but the way it works would require me to change my code deeply in order to fill the so-called replacement list.

Does someone has any experience on this matter in order to give some advices on the best approach I should adopt ?

I'm not 100% sure what you're looking for - as you said, the RefactoringTool is what's meant to be used to write out changed code - I'm open to changes to that interface if it helps your use case, but I don't know your code, so it's hard for me to tell exactly why that would be a deep change, or how to fit it into your project.

My intuition would be that at the point at which you would make changes vie the Rewriter, you would instead fill the Replacements...

Cheers,
/Manuel

Thank you for your answer. Basically I would like to know what is the *standard* way to do source-to-source translation in Libtooling.
Should I understand that this is by using "RefactoringTool" ? Updating my code would require some effort for me to understand how the "Replacements" work, so I want to be sure that this is not to find out that there is a better way to do so.

To answer your question, I detect "for" statements; analyse the body of the loop to extract some metrics; and add a comment after the for with the values of the previously measured metrics. I used to use the "Rewriting" class to add this comment.

Regards,

- Antoine

>> Hello everybody,
>>
>> I have successfully written a source-to-source translator using
LibTooling, and I am now willing to go further and do some source-to-source
translation.
>> However, I can't figure out how to output modified code with LibTooling.
>>
>> I have used in the past the class "Rewriter", coupled with a
"RewriterBuffer" to do this job, but without the LibTooling (I was
manipulating "CompilerInstance" object directly).
>> However, a "RewriterBuffer" requires a file ID (that I used to get from
a "SourceManager") to be created, and I don't have access to such
information anymore, because I'm calling my tool using "Tool.run" from a
custom "FrontEndAction".
>>
>> I've spotted the example called "loop-convert" in the clang-tools-extra
repository, that is using class derived from "RefactoringTool" to do
source-to-source transformation. I kind of made sense of this code, but the
way it works would require me to change my code deeply in order to fill the
so-called replacement list.
>>
>> Does someone has any experience on this matter in order to give some
advices on the best approach I should adopt ?
>
> I'm not 100% sure what you're looking for - as you said, the
RefactoringTool is what's meant to be used to write out changed code - I'm
open to changes to that interface if it helps your use case, but I don't
know your code, so it's hard for me to tell exactly why that would be a
deep change, or how to fit it into your project.
>
> My intuition would be that at the point at which you would make changes
vie the Rewriter, you would instead fill the Replacements...
>
> Cheers,
> /Manuel

Thank you for your answer. Basically I would like to know what is the
*standard* way to do source-to-source translation in Libtooling.
Should I understand that this is by using "RefactoringTool" ? Updating my
code would require some effort for me to understand how the "Replacements"
work, so I want to be sure that this is not to find out that there is a
better way to do so.

To answer your question, I detect "for" statements; analyse the body of
the loop to extract some metrics; and add a comment after the for with the
values of the previously measured metrics. I used to use the "Rewriting"
class to add this comment.

Ah, yes, RefactoringTool is the intended "standard" way to do
source-to-source with libtooling. I think if you already use the Rewriter,
I'll expect it to actually be not hard to change to using the Replacements
:slight_smile:

Cheers,
/Manuel

>> Hello everybody,
>>
>> I have successfully written a source-to-source translator using LibTooling, and I am now willing to go further and do some source-to-source translation.
>> However, I can't figure out how to output modified code with LibTooling.
>>
>> I have used in the past the class "Rewriter", coupled with a "RewriterBuffer" to do this job, but without the LibTooling (I was manipulating "CompilerInstance" object directly).
>> However, a "RewriterBuffer" requires a file ID (that I used to get from a "SourceManager") to be created, and I don't have access to such information anymore, because I'm calling my tool using "Tool.run" from a custom "FrontEndAction".
>>
>> I've spotted the example called "loop-convert" in the clang-tools-extra repository, that is using class derived from "RefactoringTool" to do source-to-source transformation. I kind of made sense of this code, but the way it works would require me to change my code deeply in order to fill the so-called replacement list.
>>
>> Does someone has any experience on this matter in order to give some advices on the best approach I should adopt ?
>
> I'm not 100% sure what you're looking for - as you said, the RefactoringTool is what's meant to be used to write out changed code - I'm open to changes to that interface if it helps your use case, but I don't know your code, so it's hard for me to tell exactly why that would be a deep change, or how to fit it into your project.
>
> My intuition would be that at the point at which you would make changes vie the Rewriter, you would instead fill the Replacements...
>
> Cheers,
> /Manuel

Thank you for your answer. Basically I would like to know what is the *standard* way to do source-to-source translation in Libtooling.
Should I understand that this is by using "RefactoringTool" ? Updating my code would require some effort for me to understand how the "Replacements" work, so I want to be sure that this is not to find out that there is a better way to do so.

To answer your question, I detect "for" statements; analyse the body of the loop to extract some metrics; and add a comment after the for with the values of the previously measured metrics. I used to use the "Rewriting" class to add this comment.

Ah, yes, RefactoringTool is the intended "standard" way to do source-to-source with libtooling. I think if you already use the Rewriter, I'll expect it to actually be not hard to change to using the Replacements :slight_smile:

Cheers,
/Manuel

OK. Thank you for you fast answers.
I'll convert my code then, and kindly come back to you if I have any issue.

Best regards,

- Antoine

Dear,

Regarding the below topic, I am now trying to convert my tool using the RefactoringTool.

I am using a RecursiveASTVisitor, but I can't find any way (other than using a global variable) to pass the replacement list to my ASTVisitor.
Indeed, when I create my RecursiveASTVisitor object (inside ASTFrontendAction.CreateASTConsumer), I only have access to the Compiler object, not the tool object.
However, the only way I could find to get the replacement list was through RefactoringTool->getReplacements().

Did I miss something ?
Is there any way to make my RecursiveASTVisitor aware of the Replacement list ?

All the examples I could find on the repository use a MatchFinder with a custom callback that is aware of the replacement list.
Was the RefactoringTool API designed with this very specific use in mind ?

Regards,

- Antoine

Dear,

Regarding the below topic, I am now trying to convert my tool using the
RefactoringTool.

I am using a RecursiveASTVisitor, but I can't find any way (other than
using a global variable) to pass the replacement list to my ASTVisitor.
Indeed, when I create my RecursiveASTVisitor object (inside
ASTFrontendAction.CreateASTConsumer), I only have access to the Compiler
object, not the tool object.
However, the only way I could find to get the replacement list was through
RefactoringTool->getReplacements().

Did I miss something ?
Is there any way to make my RecursiveASTVisitor aware of the Replacement
list ?

All the examples I could find on the repository use a MatchFinder with a
custom callback that is aware of the replacement list.
Was the RefactoringTool API designed with this very specific use in mind ?

The Tool/RefactoringTool interface takes a tooling::FrontendActionFactory,
so you have two options (we have some convenience methods in Tooling.h
around line 66-100):

struct MyFrontendActionFactory : public FrontendActionFactory {
  MyFrontendActionFactory(tooling::Replacements* Replaces) :
Replaces(Replaces) {}
  virtual clang::FrontendAction *create() {
    // Give the replacements to your frontend action and from there to your
ASTConsumer
  }
  tooling::Replacements* Replaces;
};

Now with convenience methods you can also just do (note that you'll need
the MyASTConsumer part anyway):
struct MyASTConsumer : public ASTConsumer {
  MyASTConsumer(tooling::Replacements* Replaces); // otherwise like above
};
// This doesn't need to be an interface, simply having the method
newASTConsumer will be enough...
struct MyConsumerFactory {
  MyConsumerFactory(tooling::Replacements* Replaces);
  clang::ASTConsumer *newASTConsumer() { return new
MyASTConsumer(Replaces); }
  tooling::Replacements* Replaces;
};
and where you create the RefactoringTool:
RefactoringTool Tool(...);
MyConsumerFactory ConsumerFactory(Tool.getReplacements());
return Tool.runAndSave(newFrontendActionFactory(&ConsumerFactory));

Note that I didn't write out everything for the first version, as it's more
code :slight_smile: Also, I wrote all that without trying to compile it, so there will
be compile errors, but the principle should work...

Cheers,
/Manuel

Thank you so much for your comprehensive answer !
I got a bit lost with all these factories of factories, but it finally works !

  • Antoine