[RFC] Forming a Massive Binaries Working Group in LLD

Subprojects LLD
1

Summary

We propose forming a new working group within the LLVM project dedicated to solving the challenges of linking massive binaries in LLD. The group would provide a platform for discussing cross-cutting concerns around relocation overflows, and modifications to the x86-64 ABI via code-models such as range-extension thunks.

Rationale

At Meta, our code-base continues to grow and we predominantly chose to statically link code. We are increasingly running into relocation overflows as a result as we’ve defaulted to the small code-model for performance. While the current x86-64 large code model prevents these overflows, it introduces performance penalties that make it a non-starter for our critical workloads.

We seek a solution that produces efficient instruction sequences when possible and employ other well known techniques (i.e. thunks) for when code breaches beyond certain limits – “pay as you go”.

Engineers from Meta and Google have already put forward to similar RFC to the x86-ABI community (please see matching threads for current discussion)

Aside:
If I were to be frank, the RFCs are similar enough but where they diverge, it looks like Google’s (Arthur) RFC has more current support. The difference doesn’t matter to me as much as they both attempt to solve the problem in a similar enough fashion so I’m in support of it as well.

Outcome

The outcome of the working group will be to build binaries of any size with performance penalty only for code that breaches 2GiB boundary. We believe that with post-link optimization (BOLT & Propeller), that we can keep the hottest code within this boundary.

In Scope

  • Changes to glibc to support multiple RELRO
  • Changes to coreutils to support multiple GOT
  • Changes to lld where necessary
  • Changes to llvm where necessary
  • Changes to the x86-64 ABI

Who is Involved So Far

So far, the conversation includes engineers from Meta and Google.

Organizational Details

  • Regular Meetings: We propose scheduling at a minimum, a monthly meeting for 1 hour to discuss designs, share prototype results, and track upstreaming progress.

  • Communications: We will keep public meeting notes and can use Discourse for major updates.

Call to action

Let me know if you would be interested in joining. I could not find much public discourse on binaries that breach 2GiB.

Looking forward to the discussion!

(I have modeled this post after RFC: Forming a Static Analysis Working Group in the Clang Ecosystem )

7 Likes
LLVM Weekly - #650, June 15th 2026
[RFC] Emit relocation failure statistics
LLVM Embedded Toolchains Working Group sync up
2

Thanks for starting this! As the person you discussed this with, obviously I’ll +1 this group. Hopefully other people are also interested (not hoping that people are running into issues, but if they are they’ll come and say what issues they have :P).

Monthly hour-long meetings sounds good to me; we can change the cadence if people end up wanting more/fewer meetings.

I’m not sure this necessarily belongs in LLD, there’s a lot more than LLD for all of this sort of work.

As an aside, we’ve been making some progress on x86-64, but we’re also looking into AArch64 which has some of the same problems, and likely some of the same solutions.

1 Like
3

Sounds awesome! This isn’t a priority for me at work anymore, so I can’t make time for it, but I’m invested in the outcome.

In one of your docs, you outline the multi-GOT solution, and I wanted to push on that a little bit. I think it’s more ABI-compatible and viable than most people seem to think it is.

I was wondering if this can be done entirely with only static linker changes. Assume the compiler emits small PIC code for all objects, which for x86, is the norm. Even static precompiled objects tend to use this model.

The GOT is… not that special, from a dynamic linker perspective. It’s just some memory with some dynamic relocations in the readwrite/relro segment. Nothing prevents the static linker from stamping out more than one sequence of 3-4 PT_LOAD segments with all the various protections:

  • readonly
  • text
  • readwrite
    • GOT1 is here, relro mprotects it on startup
    • regular rwdata
    • bss
  • readonly
  • text
  • readwrite
    • GOT2 is here, dynamic linker doesn’t know anything about it

It seems like LLD could partition the set of input objects today into two partitions: half in the first segments, half in the second set. Clearly, the cold ones go in the second half. One major downside is security. I’m told (but need a primary source) that glibc only supports one GNU_RELRO segment, so it will only mprotect one of your GOTs. However, you can compensate for this by hacking in your own startup hook that calls mprotect on the right memory.

The other downside, obviously, is static linker implementation burden. However, this is really not that different from range extension thunks, branch islands, and all that kind of stuff.

I admit, this idea is incomplete, there’s a lot left as an exercise to the reader, but I think it’s a promising direction. It’s a linker mode that you can set and forget, and it will always work, no matter how large your binary gets, since it’s basically leveraging the normal ELF x86_64 and arm small PIC shared library access patterns to build a single binary which is really a sequence of shared libraries rolled into one file.

I’ve been told this will break some untold number of assumptions in unwinders and other system libraries, but life is for the adventurous.

3 Likes
4

I’m happy to be involved on the AArch64 side.

I can help with lld and the AArch64 ABI, however I’ll have to exempt myself from any decisions that are x86_64 specific.

One of my colleagues in our GNU team has been doing some improvements in that area, such as introducing PIC support for the large code-model (via GOT accesses) and proposing a Medium code-model that makes more heavy use of the GOT abi-aa/sysvabi64/sysvabi64.rst at main · ARM-software/abi-aa · GitHub . The original large-code model was made on the assumption that it would be a relatively small program working on a really large amount of data, something like a HPC simulation problem. However this isn’t really appropriate for today’s use cases.

When LLD first implemented RELRO, I can remember checking to see if glibc could cope with more than one PT_GNU_RELRO program header, and it looked like it only has tracking information for one per RELRO segment per link map. It is possible that it has changed since then.

5

I think that’s still true of glibc. For CHERI compartmentalisation we’ve been experimenting with multiple GOTs per object (though one per “compartment” defined at link time as an additional input, not an arbitrary number decided based on warm fuzzy feelings in the linker about ranges, so we at least have a clear way of tracking “which GOTs need this symbol?” in the symbol flags, by pulling it out into a per-compartment flags field, which sounds awkward to do if you don’t have the input section grouping GOT-wise up-front). As part of that, we wanted multiple RELRO segments. In LLD we made it a limitation of one per compartment, not an arbitrary set as needed to span precisely all the relro sections and nothing else, so again that’s just per-compartment state rather than a vector. But as far as our runtime is concerned, there being multiple GOTs is entirely nothing to care about (PLT GOTs are more interesting, which we also make per-compartment; not hard to support but you do need to expect multiple dynamic entries for them). Multiple relro segment support we even just upstreamed to FreeBSD as it was a very boring change; given it’s not performance-critical code, we just re-iterate over the program headers rather than bother trying to save a list of all the segments (as would be the logical extension of saving the singular segment’s range): Making sure you're not a bot!. I do feel like one way or another implementations should support this, it’s not hard and is an obvious generalisation.

6

The problem with this approach is you end up having to partition by PC32 relocations, but there’s no guarantee there’ll be a valid partition especially with ThinLTO dso_local inference and a ton of input object files. Also I think from the static linker point of view this is more invasive than things like thunks, but unsure.

Ah good to know the background, but yeah we’d like to support arbitrary amounts of text.

This is still true today. We’re working on this right now.

Agreed, and thanks for the link.

7

Thanks for the proposal and the efforts so far!

At ByteDance, we have been dealing with issues related to very large C/C++ binaries for quite some time in production services. In the early stages, we explored a number of surrounding mitigations such as split DWARF, dead function and symbol elimination, and fine-tuning sanitizers (e.g., ASan) and coverage instrumentation to reduce overhead and binary size impact. However, over time we realized that these approaches mostly treat symptoms rather than addressing the root cause.

More recently, we have started to focus more on ABI-level and code model–level issues, and it has become increasingly clear that the problems we are seeing are systemic and require coordinated changes across the compiler and linker stack rather than isolated optimizations.

We are therefore very supportive of the idea of forming a dedicated working group for massive binaries in LLD. From our side, ByteDance would be very interested in contributing:

  • Real-world large-scale binary scenarios and workload characteristics from production services
  • Experiences and limitations we have observed from previous mitigation attempts (split DWARF, instrumentation tuning, symbol pruning, etc.)
  • Ongoing experiments around BOLT, which we believe have interesting overlaps with post-link optimization and layout constraints
  • Feedback on ABI/code model interactions that we have started to explore internally on x86/aarch64

We believe closer collaboration in this area would help surface realistic constraints early and guide more practical design choices.

Looking forward to participating in the working group and contributing upstream experience.

8

One thing I would like to see explored before committing to ABI changes: alias mappings of .data in a way that preserves copy-on-write semantics for fork. I’m not sure if this is possible on Linux today (if it is, it probably involves userfaultfd). If we can get that, then references to .data that are not (meaningfully) address-significant could keep using %rip-relative addressing. Only references that need the canonical address would have to go through the GOT. I expect that could help to reduce performance overhead. Maybe even to the degree that it’s possible to use one set of object files for the small and the new code model (if other optimizations are implemented as well).

9

For CHERI compartmentalisation we have done a limited form of this where we duplicate SHF_MERGE section slices into the sub-object (in the ELF sense) compartments that need them, since those are never address-significant (but even on CHERI are accessed using PC-relative addressing and therefore must be within the bounds of the compartment’s program counter, which are non-overlapping, and -ffunction-sections / -fdata-sections does not split them up into separate sections), and that already gets a bit awkward with having the same symbol resolve to different values depending on which input section asked for it. An upstream generalisation of this would surely be helpful for our use cases, but I wouldn’t like to say how easy it would be to achieve.

10

Great to see the interest from others in the community.
We’ve all been suffering in silence :slight_smile:

I guess next-step would be to try and see if there’s a common time we can all meet virtually.

I’ve had success with this platform to find a common time, let’s see if we can use it to find quorum:

Agenda items I see from this discussion:

  • Quick intro
  • Goal (make sure we are all on same page)
  • quick recap of two proposals from Google & Meta
  • where we are in the work?
  • review items , next steps

edit: I noticed the time range might not be wide enough to accommodate all timezones – oops. I can’t seem to edit it. If that’s a big problem, I’ll go ahead and remake it in a follow-up.

1 Like
11

Sorry for the late reply.
I was gone for a week taking part in https://tacosprint.org/

I reviewed the calendar and it looks like the consensus with 4/5 is ~5PM Pacific.
(@smithp35 I couldn’t find time that accomodated you as well :frowning: I’d love to have you involved as an lld maintainer so reach out and we can maybe figure something out )

I propose we meet July 8th 5 PM Pacific (July 9th 0 AM UTC)
I have created a google meet:

LLVM Massive Binaries Working Group
Wednesday, July 8 · 5:00 – 5:55pm
Time zone: America/Los_Angeles
Google Meet joining info
Video call link: Google Meet meeting
Or dial: ‪(US) +1 331-979-1841‬ PIN: ‪934 817 578‬#
More phone numbers: https://tel.meet/com-iwrx-xcx?pin=6827707153578
Or join via SIP: sip:6827707153578@metaea.onpexip.com

I have a calendar invite as well. If you direct message me your email, I can add you to it.

12

Not so much my wheelhouse anymore - but if it’s ever useful for me to throw in some of my experience, especially around DWARF, let me know - happy to document it, or hop in for a chat, etc.

1 Like
13

That’s 00:00 UTC at the moment given daylight savings (but 1am BST, which is the important thing for those in the UK)

14

It was difficult to find a time with worked for everyone. If needed I can do a separate meeting for those that can’t attend and bridge the notes.

image
image1850×426 24 KB

15

I can on occasion stay up to 1:00am, but wouldn’t want to make a habit of it. I can come along to the first one, but I’d want to make sure that the agenda was a good fit before coming again.

I don’t know how many others from European timezones are tentative about attending, but didn’t respond to the poll. Could be worth trying an earlier (for pacific time) session to see if anyone turns up. If there are enough people to make that worthwhile then there’s a possibility of alternating times or one offs.

16

I’m happy to change if we something becomes more available OR we can hold alternating times that prefer European timezone if you get value out of the meeting.

I can also 1:1 with you if you’d like to try and give you a recap of the meeting rather than staying up late.

@dblaikie please attend if you are interested and want to either fly-on-the wall or contribute.

I’m sure your DWARF knowledge would be useful (although I’m not sure at which point) since it suffers similar gaps at times.

17

Could you put this on the community calendar?

18

I just got to this late sorry – I just added calendar@llvm.org hopefully it works
(Along with all the other asks in that website).

Another thing to discuss in the meeting if we have time: [RFC] An opt-in CMake option for 64-bit Source Location

19

Thanks. Could I ask you edit this:

in your earlier post to “July 8th 5 PM Pacific (July 9th 0 AM UTC)” (or some suitable way of expressing midnight-between-8th-and-9th)? The current message still has the wrong UTC time relative to PDT, and moreover as it crosses past midnight it’s not on the 8th (1 AM BST on the 8th is now, not in 24 hours, and I managed to confuse myself rereading your earlier message).

20

done.

1 Like