A Few Questions Regarding The New Pass Manager, LTO, and LLD

LLVM Developers,

Hi!

There is a scenario where I want to do full program analysis, ideally after link time optimization is done and before assembly is emitted, when all symbols are in LLVM IR form and contained in a single module. But after some time browsing stackoverflow, lld documentation, and the mailing lists, I still have little clue how to do it… I am aware of the email sent by Y Liu and titled “how to add my own passes to LTO pass”. But it doesn’t seem to work for the new pass manager. A few questions:

  1. The legacy pass manager has an extension point, PassManagerBuilder::EP_FullLinkTimeOptimizationLast. Unfortunately, I can’t find a alternative in the new pass manager. The Bye example seems to suggest using PB.registerVectorizerStartEPCallback. I am not sure whether doing so will result in my module pass being ran at compile or link time?

  2. I wonder how does clang communicates passes that need to be run at lto to lld. The command line that I attempt to get lto running is clang -fexperimental-new-pass-manager -fpass-plugin=libmymodulepass.so -flto -fuse-ld=lld …. I straced the command for execv calls, and expected to see libmymodulepass.so to be passed to ld.lld as an argument, but found out it wasn’t. Could you shed some light on how clang and lld cope with each other during lto?

  3. Greping ld.lld --help reveals an interesting option: --lto-newpm-passes=<value>. Does the here mean built-in pass names like dce/mem2reg, or is it an option to insert our plugin like libmymodulepass.so?

I’ve looking into this for a while but haven’t made much progress…Any examples / command line on how to do this are greatly appreciated!

Best Wishes,

Chibin Zhang

2021.5.8

LLVM Developers,

Hi!

There is a scenario where I want to do full program analysis, ideally after link time optimization is done and before assembly is emitted, when all symbols are in LLVM IR form and contained in a single module. But after some time browsing stackoverflow, lld documentation, and the mailing lists, I still have little clue how to do it…

LLVM docs are useful but IMAO they only provide really high level guidelines. Therefore most of the time I just read the code. Most of the LLVM code has high quality comments.

I am aware of the email sent by Y Liu and titled “how to add my own passes to LTO pass”. But it doesn’t seem to work for the new pass manager. A few questions:

  1. The legacy pass manager has an extension point, PassManagerBuilder::EP_FullLinkTimeOptimizationLast. Unfortunately, I can’t find a alternative in the new pass manager. The Bye example seems to suggest using PB.registerVectorizerStartEPCallback. I am not sure whether doing so will result in my module pass being ran at compile or link time?

PassBuilder (https://llvm.org/doxygen/classllvm_1_1PassBuilder.html) is the right place to insert EP. If you’re building a pipeline for normal optimization (non-LTO) you can leverage registerOptimizerLastEPCallback. But I think right now you can’t insert any EP into LTO optimization (not pre-link one) pipeline, except peephole EP (registerPeepholeEPCallback). I don’t know the rationale behind this decision, maybe folks familiar with LTO can give you more details.
The callback registered with registerVectorizerStartEPC will only be executed by normal optimization pipeline (including O0)

  1. I wonder how does clang communicates passes that need to be run at lto to lld.

Most of the time they’re communicated via optimization level (i.e. O0 ~ O3…). Clang tells LLD what optimization level to be performed during LTO. LLD then (indirectly) asks PassBuilder to build the corresponding LTO optimization pipeline.

The command line that I attempt to get lto running is clang -fexperimental-new-pass-manager -fpass-plugin=libmymodulepass.so -flto -fuse-ld=lld …. I straced the command for execv calls, and expected to see libmymodulepass.so to be passed to ld.lld as an argument, but found out it wasn’t. Could you shed some light on how clang and lld cope with each other during lto?

I think neither LLD nor the gold plugin (llvm/tools/gold) can load (NewPM) PassPlugin into their LTO optimization pipeline. Your PassPlugin will eventually be loaded, but it’s unable to register your custom Pass into the LTO optimization pipeline.
The llvm-lto and llvm-lto2 command line tools do support PassPlugin though – but these tools are only intended for debug and development.

  1. Greping ld.lld --help reveals an interesting option: --lto-newpm-passes=<value>. Does the here mean built-in pass names like dce/mem2reg,

Yes, for all available names please refer to this file: https://github.com/llvm/llvm-project/blob/main/llvm/lib/Passes/PassRegistry.def

or is it an option to insert our plugin like libmymodulepass.so?

No, I don’t think LLD support PassPlugin now.

I’ve looking into this for a while but haven’t made much progress…Any examples / command line on how to do this are greatly appreciated!

The rule of thumb here is that Clang doesn’t explicitly tell LLD what Passes to run during LTO – it only tells LLD the optimization level (for LTO) which is delegated to PassBuilder to construct a pre-defined optimization pipeline accordingly.
And unfortunately I think there are little ways to customize LTO optimization pipeline via EP. But maybe this is just a missing feature, if that’s the case feel free to send a patch.

-Min

LLVM Developers,

Hi!

There is a scenario where I want to do full program analysis, ideally after link time optimization is done and before assembly is emitted, when all symbols are in LLVM IR form and contained in a single module. But after some time browsing stackoverflow, lld documentation, and the mailing lists, I still have little clue how to do it… I am aware of the email sent by Y Liu and titled “how to add my own passes to LTO pass”. But it doesn’t seem to work for the new pass manager. A few questions:

  1. The legacy pass manager has an extension point, PassManagerBuilder::EP_FullLinkTimeOptimizationLast. Unfortunately, I can’t find a alternative in the new pass manager. The Bye example seems to suggest using PB.registerVectorizerStartEPCallback. I am not sure whether doing so will result in my module pass being ran at compile or link time?

Feel free to create a patch to add a PassBuilder equivalent of EP_FullLinkTimeOptimizationLast.

  1. I wonder how does clang communicates passes that need to be run at lto to lld. The command line that I attempt to get lto running is clang -fexperimental-new-pass-manager -fpass-plugin=libmymodulepass.so -flto -fuse-ld=lld …. I straced the command for execv calls, and expected to see libmymodulepass.so to be passed to ld.lld as an argument, but found out it wasn’t. Could you shed some light on how clang and lld cope with each other during lto?

I don’t know pass plugins that well, but maybe llvm/examples/Bye/Bye.cpp and its surrounding infrastructure may help?