Add custom executable (output object) format

Hi, I’m creating a backend for my own custom virtual processor, and now that the plain text assembly output with llc works, I’d like to get clang to output my custom executable format. Thus my question: How would I go about doing this? Do I need a custom MCObjectWriter, MCObjectStreamer, etc., or changes to lld? Can I reuse the existing linker instead of writing my own? Ideally, I’d like to use the custom object format only for the final executable, not for intermediate object files, I don’t care about the format of these, as long as I can compile a program containing multiple source code files and produce the custom executable in the end. I don’t need support for dynamic libraries (or static libraries, for that matter).

I really hope someone can give me some hints, thanks!

LLVM supports several different object formats: ELF, COFF, MachO, and probably some variants. If you are happy with one of those formats for your relocatable object files, then the only compiler changes would be to have your target hook up the correct object file format.

If you want a custom executable format, I can see two options. One, try to leverage an existing linker to emit your custom format. Two, use an existing linker to output a standard format, then write a conversion tool. This means an extra post-link processing step, but might be simpler in the short term.

The newest member to the family is XCOFF.

HLSL circumvents the binary backend:

Thanks all!

So this first method would need changes in ./lld/, I guess? In this case, will (could) the intermediate object files still be ELF or something? (Or is there some more neutral format?) Not sure if LLD has some intermediate representation after it links the object files, such that this could be easily transformed into any executable.

Does LLD also support this? I can’t find it in ./lld/.

I see, because it wants to output something like LLVM IR I guess. In this case, would it be impossible to compile a project with multiple source (e.g. .c) files, because the linker is skipped, if I understand correctly? Also, I’d like to have absolute memory addresses (e.g. for globals) in the final executable, so I guess I’d need the linker for that, right?

If you want a custom object-file format, you will need to modify something. :slight_smile: I’m not really familiar with the internals of LLD, so I couldn’t say whether it would be better to have the compiler emit a new format and add a whole new “flavor” to LLD, or have the compiler emit an existing format and persuade LLD to turn that into a different output format. But if you want the final executable to be in a custom format, you will need to do one of those things.

IBM uses XCOFF for AIX. There are existing toolchains. They are just starting to add AIX support to LLVM.