I was trying to get debugging of JIT’ed code (linked with JITLink) working on macOS (among other things, the other big one being COFF support) and I found that the way to do that seems to be to create the appropriate llvm::orc::<>Platform. That in turn requires passing in a filesystem path to the ORC runtime archive, which is built as part of compiler_rt. Is there another way to provide those symbols? I realize that would require some refactoring, but passing a path to an archive file isn’t really an option for me so I’m willing to help make this work. I mainly was hoping to understand if there’s something in flight or something I’m missing.
Tagging you @lhames cause you might at least know who I should ping about this
@bzcheeseman I’d recommend embedding the archive within your binary. On Darwin you can use ld -sectcreate segname sectname file for this. On linux you could try one of the approaches in How to Embed Binary Data in Program Code (@weliveindetail – have you tried this? do you have a preferred approach?). I’m not sure the best way to do this on COFF, but I imagine it’s doable there too (@sunho – have you ever tried this?).
For COFF perspective, we also need to load bunch of msvc runtime libraries and they are specified as just file path as well. We’ll need to edit those to receive definition generator.
I’d recommend embedding the archive within your binary
So then I’d pass the path to my binary to the char *OrcRuntimePath that the platforms require in their create methods? Also I don’t think that’d work - it tries to create an object::Archive from the thing that is passed in, right? Or are you suggesting that I embed the binary and then write it to a temp file?
Out of curiosity, is there a reason we couldn’t define the symbols ourselves (or link them into the current binary) and pass them in as a JITDylib or something? As far as I can tell it’s a “because that’s how it’s currently done” thing more so than a “it must be done this way” thing?
From my perspective that’s a little more live-able just because it’s a platform library. The issue with orc_rt is that it’d need to be distributed with the compiler cause we can’t rely on the user having a build of CompilerRT
Ahh – @sunho’s right – the support for constructing a Platform from an archive is still up for review. (Good timing with your question by the way: I only started embedding the runtime a few weeks ago in one of my projects, which is why some of the groundwork hasn’t landed in-tree yet).
Out of curiosity, is there a reason we couldn’t define the symbols ourselves (or link them into the current binary) and pass them in as a JITDylib or something? As far as I can tell it’s a “because that’s how it’s currently done” thing more so than a “it must be done this way” thing?
Most of the ORC runtime should be pre-linkable in practice, and we want to refactor it to encourage more of that. However there are a few pieces that need to be JIT-linked. E.g. definitions of runtime functions that use JIT’d global state like dlopen, dlsym, and implementation details like the thread-local-storage entry points.
Hah good timing indeed. We’re probably working on almost the same thing
I think I understand…to paraphrase: we need to embed the code for dlopen and the like, but we can’t link it to the current binary because it’d be missing definitions or modify the process global state (as opposed to the JIT process global state), hence why we need to link it for the first time from within the JIT? So realistically, as long as we get the object file to JITLink somehow it doesn’t really matter how it got there - got it.
It’s more like you embed the runtime code that you will “JIT link” to JIT session, which is good to have when your JITed code run on remote computer (e.g. rasberry pi) but compiler’s running on main computer.
The only edition needed are on COFF runtime class inside ORC library. Just changing file path parameter inside constructor to Archive buffer or definition generator. Then, you can just give pointer to your embeded buffer of orc runtime library file to COFF runtime constructor.
I’d recommend embedding the archive within your binary. On Darwin you can use ld -sectcreate segname sectname file for this. On linux you could try one of the approaches in How to Embed Binary Data in Program Code @weliveindetail – have you tried this?
Not sure it’s the favorable approach here, but you can always embed sections manually via objcopy and linker script. I didn’t do it with the ORC runtime (yet), but with custom symtab and strtab sections for symbol lookup in embedded targets:
Whatever you’re comfortable with. From my perspective it’d be nice to have that landed so I’m totally up for helping fix any remaining issues so you can enjoy your vacation
