Hello llvm-dev,
I would like to be able to customise the X86TargetMachine but I couldn't find any proper way to do this from code external to the LLVM codebase. For the IR passes there are RegisterPass and RegisterStandardPasses. Have I overlooked an existing mechanism for extending TargetMachine passes or is this not really supported?
As background, I'm working on runtime inlining of calls made via function pointers, including C++ virtual functions. This uses a normal compiler like clang with some additional decoration of the IR before running llc to get the ahead-of-time compiled machine code. The decorated code can then re-compile parts of the IR at runtime by calling into LLJIT.
I use some low-level mechanisms to pass information between decorated functions alongside the normal platform ABI, and this is where I need to manipulate the target machine behaviour. The end result is to store some data in the instruction stream preceding the function return address, and pass a context pointer in r14.
Currently I have a workaround by adding a whole new target via RegisterTargetMachine. The implementation does the additional processing in runOnMachineFunction and delegates as much as possible to the real X86 target obtained via TargetRegistry::lookupTarget. This is especially messy because many of the TargetPassConfig virtual functions that I delegate to X86PassConfig are protected (e.g. addPreRegAlloc) so it's clearly not a "proper" way of doing what I want. The code in question is in https://github.com/drti/drti/blob/a204564d74ac5f8ad6946b1c96e10c84a1ffb97e/passes/drti-target.cpp and a general description of the runtime inliner is at GitHub - drti/drti: Dynamic runtime inlining with LLVM
Can anyone suggest a better way to do the TargetMachine customization? If necessary I could contribute some LLVM code changes to add extension points but would appreciate some guidance on which way to approach it first.
Thanks,
Raoul.