Is there a way to let the AST matcher run on multiple files in parallel?

Hi all,

Is there a way to let the AST matcher run on multiple files in parallel? I’m working on Mac. I have some old windows code which may slow down the compiling time (maybe for some other reasons I don’t know exactly). So this makes it necessary for me to analyze each translation unit in parallel.

At the moment I have only one mac which has 32 cores. I can live with that if it could be fully parallel.

Thanks!
Best regards,
Han

You can use
a) a compilation database
b) find . -name ‘*.cc’ |xargs -n1 -P32
depending on the output data you want to produce, you’ll need to pass an additional flag to your tool to get the shard number…

Hi Manuel,

Thank you for your answer. The goal is to do refactoring to the project and apply the replacements to the original files. For the option (a), could you describe more specifically how to do it? Thanks. I checked the run function of clang::tooling::ClangTool, but it’s just a for loop over all the source files.

Best regards,
Han

Those are not 2 options. You’ll want to have both :slight_smile:
You need the compilation database so the tool only needs the source file to run.

Hi Manuel,

That I have the compilation database already. :slight_smile:

Let’s say we have a simple case with only 2 source files (test1.cc and test2.cc). We also have the compilation database for these 2 files. And we have the AST matcher tool up and running. Normally I’d run ./matchertool.out test1.cc test2.cc to do the refactoring. To make the analyzing of the 2 files running in parallel, what shall we do next?

Thanks!
Best regards,
Han

Ok, so make sure your tool outputs the replacements that you want to apply in some format you can later slurp in and apply easily. Make the name of the output file depend on on the name of the translation unit you got at the command line.

Then you can do:
$ ls test1.cc test2.cc |xargs -n1 -P2 ./matchertool.out

Unfortunately currently I don’t think there’s a simpler way than that.

Hi Manuel,

It works! :slight_smile: Thank you very much!

Best regards,
Han

Hi Manuel,

I currently got some synchronization problem with this method. If a header file is included by multiple source files, the write operations to the header file are not synchronized. But if we use only 1 thread, it is synchronized.

Best regards,
Han

That’s why I wrote in my previous mail:
“make sure your tool outputs the replacements that you want to apply in some format you can later slurp in and apply easily.”
The “apply” part is going to be single threaded.

Thanks! :slight_smile: Let me see if I could generate some patch files then it would be easier for the synchronization.