Until now I have used a module pass for some transformation logic of the input code when WebAssembly is targeted. I would now like to extend my analysis not just to a single module, but to the whole program that is grouped into one module (this is necessary for improvements to my analysis). I read that there is something called LTO passes that I can use. However, I can’t seem to find (any) documentation on how to create such a Pass (the .cpp file e.g.) and how to add it to whatever pass manager is used here. Any help? Any useful links?
Btw, I have used the legacy/old pass manager so far for this project, and would like to keep using that for now as well. I heard there is also a legacy and new LTO.
My analysis only applies to WebAssembly targets, so if I could only add the LTO pass for WebAssembly targets as well that would be great.
Thanks!
It is the same pass: zero difference.
LTO is implemented by having the linker merge multiple module into one and running the same passes.
Ok, you mean the same module pass? But do I have to add it differently to a (maybe different?) pass manager? Currently I am only doing:
//===----------------------------------------------------------------------===//
// The following functions are called from lib/CodeGen/Passes.cpp to modify
// the CodeGen pass sequence.
//===----------------------------------------------------------------------===//
void WebAssemblyPassConfig::addIRPasses() {
addPass(myCustomModulePass());
TargetPassConfig::addIRPasses();
}
in llvm/lib/Target/WebAssembly/WebAssemblyTargetMachine.cpp.
I read somewhere that I might have to extend llvm/tools/lto or llvm/lib/LTO, is that true? That would be totally fine, I’m compiling LLVM from source anyways (not adding the module pass using opt). If you could point me into the right direction, that would be great. Or, even better, do you have an example (either in LLVM itself or made by someone else) of how to add the LTO pass, which is apparently just a normal module pass, to the LTO pipeline/infrastructure?
This is a CodeGen pass I take it? The CodeGen passes for LTO backends are invoked fairly normally from codegen() in llvm/lib/LTO/LTOBackend.cpp (via addPassesToEmitFile) - similar to the way this is set up from clang for non-LTO compiles.
Still fairly new to LLVM, so not sure what to classify it as. I would say yes, it’s a CodeGen class, it’s a module pass that operates on LLVM IR and adds some instructions for example. Thanks for the help.
I take it you mean this from LTOBackend.cpp:
legacy::PassManager CodeGenPasses;
TargetLibraryInfoImpl TLII(Triple(Mod.getTargetTriple()));
CodeGenPasses.add(new TargetLibraryInfoWrapperPass(TLII));
CodeGenPasses.add(
createImmutableModuleSummaryIndexWrapperPass(&CombinedIndex));
if (Conf.PreCodeGenPassesHook)
Conf.PreCodeGenPassesHook(CodeGenPasses);
if (TM->addPassesToEmitFile(CodeGenPasses, *Stream->OS,
DwoOut ? &DwoOut->os() : nullptr,
Conf.CGFileType))
report_fatal_error("Failed to setup codegen");
CodeGenPasses.run(Mod);
Do you mean i need to duplicate this (in this codegen function) and then adapt to my module pass, or use the existing CodeGenPasses object and add my pass to that (so I would just need to add CodeGenPasses.add(new MyCustomModulePass()));?
I would check why it isn’t being invoked already via that call. I.e., addPassesToEmitFile calls addPassesToGenerateCode which calls addISelPasses which calls addIRPasses.
I think there my might be a misunderstanding. What do you mean with “why it isn’t being invoked already via that call”? I haven’t made any call yet. I’m actually asking which call to make. But I think I’ll try a simple CodeGenPasses.add(new MyCustomModulePass())); and see if that works
That was based on your earlier message that you are calling via WebAssemblyPassConfig::addIRPasses.
Ah ok, I see. But, as far as I understand, the WebassemblyPassConfig::addIRPasses call does not add the module pass as one that is run at LTO, but normally, i.e. individually on each module. So there has to be some step I have to take to register the module pass as an LTO module pass, and my assumption was that I have to do that in LTOBackend.cpp as you described.
There is no such thing as an “LTO” pass. Basically, once it gets to that call to codegen(), LTO has simply merged all modules into one huge monolithic module. Then the usual opt/codegen passes are run on the merged module. See also opt() in LTOBackend.cpp e.g. for where it invokes the IR optimization pipeline. And in codegen() it invokes the CodeGen passes via the TargetMachine addPassesToEmitFile on the merged LTO module the same way it would from clang for a non-LTO module.
So if the change you made was effective in getting your custom pass invoked for a non-LTO compile, I’m not sure why it wouldn’t work the same for an LTO compile.
If there is no such thing as an “LTO pass”, wouldn’t that mean that every module pass is run on the entire program “squashed” into one module during LTO? You differentiated between “non-LTO compile” and “LTO compile”. What does this mean? Do I have to pass specific clang options in order to include an LTO phase?
I guess my question is: what do you mean with “then the usual opt/codegen passes are run on the merged module”. Which passes exactly does this include? All passes that are also normally run on before LTO on each module individually? I naively thought that running a module pass again on the monolithic module would have some overhead, so doing that by default for all module passes seems unnecessary, so I thought one would have to specify exactly which module passes should be run at LTO.
Maybe my confusion also has something to do with the old vs legacy pass manager. The function lto::opt you mentioned looks like it only runs the the new PM passes with runNewPMPasses. I am not very familiar with the new PM (yet), I have only used the old/legacy PM for now. Do legacy PM passes even work with LTO?
Maybe a little more context on what I’m trying to do: This isn’t an optimization pass, but more of a transformation pass that inserts specific instructions (that improve memory safety for wasm) based on some analysis I do in the pass. My goal: the analysis can be a lot more powerful if it operates on the entire program (because it needs to identify methods that are defined outside of the compiled program). Therefore, I don’t want this module pass to be run at all normally on each module pre-LTO, I want this module pass to only be run once during LTO on the entire module. I acknowledge that this is probably not a standard use-case for using LTO in general, but I’m willing to have a somewhat “hacky” solution for now (our LLVM fork is more of a research project for now).
This also means I don’t want the pass to only be run at LTO for certain optimization levels (e.g. -O2, -O3). If possible, I would like it to always run, regardless of the optimization level. But it would be fine if this is not possible (e.g. then I could document that our memory safety protections only work for higher optimization levels).
Thank you very much for the help you’ve already provided!
This is exactly what I wrote above:
LLVM does not do anything specific to LTO. The only thing that happens in LTO (on top of what I wrote about the linker merging the modules) is that the linkers also tell if a symbol is exported or not (that is: we turns every function/globals that isn’t “exported” into a local linkage).
The only question is “how to get the pass to run”: at this point it is a “linker specific” question which determines how the “LTO plugin” is invoked and how the pass pipeline is setup. If you add your pass to WebAssemblyPassConfig::addIRPasses() it’ll necessary be called during codegen, assuming you built the LTO plugin with your modifications.
(which is what Teresa was referring to above “I would check why it isn’t being invoked already via that call. I.e., addPassesToEmitFile calls addPassesToGenerateCode which calls addISelPasses which calls addIRPasses”)
Now for Webassembly it may be a bit special since you may not go through the regular ELF linker: how does the linking phase work for you?
Ah ok, so “enabling LTO” is achieved by “building the LTO plugin with my modifications”. What exact modifications are you talking about? And how exactly do I need to the “build the LTO plugin”? Does this involve anything other than compiling LLVM normally? I guess I am not clear about what the “LTO plugin” even is. Do I need to enable/compile it?
Regarding the linker: We simply use clang --target=wasm64-unknown-wasi --sysroot .../wasi-libc/sysroot (so WASI as the sysroot), I think this means we use the default wasm-ld as the linker. Does this answer your question?
Anything…
Seems like you’re adding a pass and adding it to the codegen preparation phase of the Wasm backend.
It is linker specific, so there is no generic answer here… I believe lld for example does not even need a plugin, it’ll have native LLVM LTO support. But gold, bfd, or ld64 need a plugin.
If you take “any linker”, it does not include LLVM or even really know about it. So when you feed it with LLVM IR as input “object files”, it needs a “plugin” to be able to handle these “object files” and perform LTO.
Here is the doc for the Gold linker plugin for example: The LLVM gold plugin — LLVM 18.0.0git documentation
clang is a driver here, adding -v will show exactly how is the linker invoked.
I don’t know wasm-ld and I don’t know how it supports LTO (if it even does).
Try to add -flto=full to your clang invocations to see if it changes some linker flags to begin with?
Note also that for playing with LTO flow, you can always “emulate” everything using the tools provided by LLVM. You probably know opt to run arbitrary passes, but llvm-link is a tool that can “link” multiple input LLVM IR files into a single one, doing somehow the same job as the linker. There are a bunch of options that affect the behavior though (the linker knows which symbol should be exported, you may have to tell llvm-link, it can require some trial and error).
After you get a single merged module, opt can run any pass as usual (did I say already that there is nothing special to LTO?) and the resulting module can be compiled with clang (or llc) and later linked as usual.
My understanding is that wasm-ld is lld, so there is no plugin required. You would need to rebuild lld after making any change to the pipelines, such as the one you showed earlier with addIRPasses. lld links in and invokes LTO on the merged module.