How to make clang compile the data from the RewriteBuffer

Clang Frontend
#1

Hi everyone,

I want to create a plugin that makes some changes on the parsed AST and that directly compiles the changes made on the AST. It is important that I do not generate a new file with the changes on the AST.
So, in other words, I want to change the AST in place, and then compile it in the same run, using a Clang Plugin.

I am using llvm 3.4.2.

By following the RecursiveASTVisitor example, I have written a clang plugin which modifies the parsed AST.

I am using the Rewriter class to ReplaceText inside the buffer.

I am also using the -add-plugin command (instead the “standard” -plugin command) when calling the plugin.

The rewriting part works excellently. However, when I continue the compilation, clang only compiles the original file, instead of the original file + changes.
Is there a way to tell clang/llvm to compile what I have written in the AST (using the Rewriter) without creating an intermediate file?

Thanks a lot for your help,
Bogdan

#2

Hi,
   There is a pending patch on cfe-commits about this "[PATCH] clang/Frontend/MultiplexConsumer.h". With it you could reorder the ASTConsumers in clang and get yours before codegen.
Vassil

#3

Hi Vassil,

Is the patch available for the 3.4.2 version of LLVM or must I migrate to the “current” version ?

Cheers,
Bogdan

Hi,
There is a pending patch on cfe-commits about this “[PATCH] clang/Frontend/MultiplexConsumer.h”. With it you could reorder the ASTConsumers in clang and get yours before codegen.
Vassil

Hi everyone,

I want to create a plugin that makes some changes on the parsed AST and that directly compiles the changes made on the AST. It is important that I do not generate a new file with the changes on the AST.
So, in other words, I want to change the AST in place, and then compile it in the same run, using a Clang Plugin.

I am using llvm 3.4.2.

By following the RecursiveASTVisitor example, I have written a clang plugin which modifies the parsed AST.

I am using the Rewriter class to ReplaceText inside the buffer.

I am also using the -add-plugin command (instead the “standard” -plugin command) when calling the plugin.

The rewriting part works excellently. However, when I continue the compilation, clang only compiles the original file, instead of the original file + changes.
Is there a way to tell clang/llvm to compile what I have written in the AST (using the Rewriter) without creating an intermediate file?

Thanks a lot for your help,
Bogdan

#4

Hi Bogdan,
   After applying the attached patch, I can do:

void MyPlugin::Initialize(ASTContext& Context) {
// We need to reorder the consumers in the MultiplexConsumer.
MultiplexConsumer& multiplex
= static_cast<MultiplexConsumer&>(m_CI.getASTConsumer());
std::vector<ASTConsumer*>& consumers = multiplex.getConsumers();
ASTConsumer* lastConsumer = consumers.back();
consumers.pop_back();
consumers.insert(consumers.begin(), lastConsumer);
}

This allows me to hook MyPlugin before clang's codegen.

Vassil

clang_multiplex_consumer.diff (576 Bytes)

#5

Hi Vassil,

thanks for the example and the patch!

I have applied both of them successfully, but somehow, re-ordering the consumers still did not give me the expected results. This means, when I call the plugin it executes correctly, but CodeGen still only uses the original file (without the changes made by the plugin).

Is there another step which I may have forgotten?

Cheers,
Bogdan

Hi Bogdan,
After applying the attached patch, I can do:

void MyPlugin::Initialize(ASTContext& Context) {

// We need to reorder the consumers in the MultiplexConsumer.
MultiplexConsumer& multiplex
= static_cast<MultiplexConsumer&>(m_CI.getASTConsumer());
std::vector<ASTConsumer*>& consumers = multiplex.getConsumers();
ASTConsumer* lastConsumer = consumers.back();
consumers.pop_back();
consumers.insert(consumers.begin(), lastConsumer);
}

This allows me to hook MyPlugin before clang’s codegen.

Vassil

Hi Vassil,

Is the patch available for the 3.4.2 version of LLVM or must I migrate to the “current” version ?

Cheers,
Bogdan

Hi,
There is a pending patch on cfe-commits about this “[PATCH] clang/Frontend/MultiplexConsumer.h”. With it you could reorder the ASTConsumers in clang and get yours before codegen.
Vassil

Hi everyone,

I want to create a plugin that makes some changes on the parsed AST and that directly compiles the changes made on the AST. It is important that I do not generate a new file with the changes on the AST.
So, in other words, I want to change the AST in place, and then compile it in the same run, using a Clang Plugin.

I am using llvm 3.4.2.

By following the RecursiveASTVisitor example, I have written a clang plugin which modifies the parsed AST.

I am using the Rewriter class to ReplaceText inside the buffer.

I am also using the -add-plugin command (instead the “standard” -plugin command) when calling the plugin.

The rewriting part works excellently. However, when I continue the compilation, clang only compiles the original file, instead of the original file + changes.
Is there a way to tell clang/llvm to compile what I have written in the AST (using the Rewriter) without creating an intermediate file?

Thanks a lot for your help,
Bogdan

#6

If you reorder the consumers in the initialization, it should work. Are you sure you have the right tests to check for the desired behaviour? I.e are you sure you are making a valid AST-transformation? Vassil

#7

That is a good question.

I have written a libTool with the same visitor and consumer.
I called my_Rewriter.ReplaceText() to make my AST-transformations.
I have output the RewriteBuffer to a new file and then I compiled it.
When I run the transformed source, it gives me the expected behavior. However, this requires me to explicitly create a new file with the changes.

If I do the same thing with the PluginASTAction (with or without re-ordering the Consumers), I only get the behavior from the original source file.

Hi Vassil,

thanks for the example and the patch!

I have applied both of them successfully, but somehow, re-ordering the consumers still did not give me the expected results. This means, when I call the plugin it executes correctly, but CodeGen still only uses the original file (without the changes made by the plugin).

Is there another step which I may have forgotten?

If you reorder the consumers in the initialization, it should work. Are you sure you have the right tests to check for the desired behaviour? I.e are you sure you are making a valid AST-transformation?
Vassil

#8

Hi Vassil,

I have attempted to debug the Rewriter and CodeGenerator myself and see what happens when compiling, but the amount of code is rather massive and will take some time.

Here is an running simplified example of what I am doing. Would mind running it and telling me whether it worked for you or not?

Cheers,
Bogdan

That is a good question.

I have written a libTool with the same visitor and consumer.
I called my_Rewriter.ReplaceText() to make my AST-transformations.
I have output the RewriteBuffer to a new file and then I compiled it.
When I run the transformed source, it gives me the expected behavior. However, this requires me to explicitly create a new file with the changes.

If I do the same thing with the PluginASTAction (with or without re-ordering the Consumers), I only get the behavior from the original source file.

Hi Vassil,

thanks for the example and the patch!

I have applied both of them successfully, but somehow, re-ordering the consumers still did not give me the expected results. This means, when I call the plugin it executes correctly, but CodeGen still only uses the original file (without the changes made by the plugin).

Is there another step which I may have forgotten?

If you reorder the consumers in the initialization, it should work. Are you sure you have the right tests to check for the desired behaviour? I.e are you sure you are making a valid AST-transformation?
Vassil

rewriter_ex.tar.gz (3.41 KB)

#9

Hi Bogdan,
   It seems that your visitor is called too late, i.e on HandleTranslationUnit. You should override HandleTopLevelDecl instead.
Cheers,
Vassil

#10

Hi Vassil,

I really wasn’t expecting this to work, but I tried it out anyway. Sadly, it still did not change anything.
With or without the Visitor, calling the HandleTopLevelDecl instead of HandleTranslationUnit did not give me the desired effect.

Cheers,
Bogdan

Hi Bogdan,
It seems that your visitor is called too late, i.e on HandleTranslationUnit. You should override HandleTopLevelDecl instead.
Cheers,
Vassil

Hi Vassil,

I have attempted to debug the Rewriter and CodeGenerator myself and see what happens when compiling, but the amount of code is rather massive and will take some time.

Here is an running simplified example of what I am doing. Would mind running it and telling me whether it worked for you or not?

Cheers,
Bogdan

That is a good question.

I have written a libTool with the same visitor and consumer.
I called my_Rewriter.ReplaceText() to make my AST-transformations.
I have output the RewriteBuffer to a new file and then I compiled it.
When I run the transformed source, it gives me the expected behavior. However, this requires me to explicitly create a new file with the changes.

If I do the same thing with the PluginASTAction (with or without re-ordering the Consumers), I only get the behavior from the original source file.

Hi Vassil,

thanks for the example and the patch!

I have applied both of them successfully, but somehow, re-ordering the consumers still did not give me the expected results. This means, when I call the plugin it executes correctly, but CodeGen still only uses the original file (without the changes made by the plugin).

Is there another step which I may have forgotten?

If you reorder the consumers in the initialization, it should work. Are you sure you have the right tests to check for the desired behaviour? I.e are you sure you are making a valid AST-transformation?
Vassil

#11

I don’t know. I haven’t used the Rewriter in such scenario. We transform the AST with regular Stmt or Decl Visitors. Vassil

#12

Do you happen to have a simple out of the box example which works for you?
It might still be something that I am doing wrong and it might not be related to the Consumers.

I would appreciate it very much! Thank you!

Bogdan

Hi Vassil,

I really wasn’t expecting this to work, but I tried it out anyway. Sadly, it still did not change anything.
With or without the Visitor, calling the HandleTopLevelDecl instead of HandleTranslationUnit did not give me the desired effect.

I don’t know. I haven’t used the Rewriter in such scenario. We transform the AST with regular Stmt or Decl Visitors.
Vassil

#13

I hope it helps. Vassil

#14

Thank you Vassil,

It is a very nice tool. Sadly, it differs slightly from my requirements, as you do not use the Rewriter at all.
I have been reading through your code and I noticed you you are Creating Declarations and adding them to the Decl Context.

I tried the same thing, and although I could successfully call Declaration->dump() an observe the changes, the resulting executable was still unaffected.

I also called Declaration->setDeclName() and directly overrode all members in a class. This should have at least given an error during initialization, but no, everything still compiled with the original code.

I think I am running out of options. :frowning:
Any suggestions?

Do you happen to have a simple out of the box example which works for you?
It might still be something that I am doing wrong and it might not be related to the Consumers.

I would appreciate it very much! Thank you!

https://github.com/vgvassilev/clad/tree/master/tools

I hope it helps.
Vassil