[RFC] LLD: Preferring small code model COMDAT sections over large ones when mixing code models

Subprojects LLD
,
1

Related PR: #177298

Background

At Meta, we are experimenting with the medium code model to solve relocation overflow issues in large binaries. During this work, we encountered a problem when linking objects compiled with different code models (small vs. medium/large).

The Problem

When linking mixed code model objects, COMDAT groups may have both small and large (SHF_X86_64_LARGE) versions. Currently, LLD uses first-come-first-served selection, which can keep the large version even when small code model code references it. This causes relocation overflows because large sections can be placed beyond ±2GB from the code, exceeding the range of R_X86_64_PC32 relocations used by the small code model.

Proposed Solution

The submitted PR makes LLD prefer small COMDAT sections over large ones, ensuring compatibility when mixing code models. When a COMDAT with SHF_X86_64_LARGE
sections is already selected and a small version is encountered, the small version takes over.

Feedback Received

@MaskRay raised the following concerns:

In ELF,

GRP_COMDAT

indicates a section group subject to deduplication. While the specification doesn’t mandate which group prevails, all linkers consistently select the first one encountered. This patch fundamentally changes that behavior (guarantee).

This feels like an almost ODR violation scenario. The COMDAT contract is that all instances are identical and interchangeable. If they have different flags, they’re not truly identical.

The problem could potentially be addressed on the build system side—can you ensure small code model relocatable files are passed before large code model files?

@smithp35 provided additional context:

Arm’s proprietary linker doesn’t follow the first come first served model for COMDAT group selection. For various reasons, groups were ranked, such as the group with smallest size, highest optimization level, most use of architectural features, presence of optional metadata.

Our Perspective

  1. Build system solutions are fragile: Solving this problem reliably at the build system level is challenging and feels like a workaround. Large codebases with complex dependency graphs make it difficult to guarantee link order. It seems the linker should be able to handle any order.

  2. Trade-off between guarantees: Violating the “first COMDAT section wins” convention seems preferable to violating the “small code model cannot reference large sections” guarantee. The former is a linker implementation detail; the latter causes hard link failures.

  3. Precedent exists: As @smithp35 noted, there is precedent for linkers using more sophisticated COMDAT selection strategies beyond first-come-first-served.

  4. ODR considerations: While COMDAT sections with different flags aren’t strictly identical, they are semantically equivalent—the code/data content is the same, just compiled with different assumptions about addressing. The small version is strictly more compatible.

Looking forward to the community’s input on the best path forward.

cc: @WenleiHe, @fzakaria

1 Like
2

Can you explain what kind of sections you have that have both small and large ones? Without more info I am inclined to agree this feels like an ODR violation.

At Google we generally compile all code built from source with the same flags. If we do have precompiled binaries, we try to ensure it has a minimal C API, meaning no visible vague linkage globals/functions (most notably C++ STL) that can cause this exact issue. This should allow mixed small precompiled libraries and medium/large code built from source.

3

Here is an example:

 provider.cpp (medium code model - creates large section):
  template<typename T>
  struct Data {
      static T value;
  };

  template<typename T>
  T Data<T>::value = T{};

  template struct Data<int>;  // Explicit instantiation

  user.cpp (small code model - uses 32-bit relocation):
  template<typename T>
  struct Data {
      static T value;
  };

  int main() {
      return Data<int>::value;  // R_X86_64_32S relocation
  }

  Commands:
  # Compile with different code models
  clang++ -c -mcmodel=medium -mlarge-data-threshold=0 provider.cpp -o provider.o
  clang++ -c -mcmodel=small user.cpp -o user.o

  # Linker script to place large sections beyond 2GB
  cat > far.lds << 'EOF'
  SECTIONS {
    . = 0x400000;
    .text : { *(.text*) }
    .data : { *(.data*) }
    .bss : { *(.bss*) }
    . = 0x100000000;
    .ldata : { *(.ldata*) }
    .lbss : { *(.lbss*) }
  }
  EOF

  # Link - triggers overflow error
  ld.lld -e main -T far.lds -o test provider.o user.o

  Error:
  ld.lld: error: user.o:(function main: .text+0xe): relocation R_X86_64_32S out of range: 4294967296 is not in [-2147483648, 2147483647]; references 'Data<int>::value'
1 Like
4

sorry, I meant what kind of situation would you have provider.cpp and user.cpp built with different code models but linked into the same binary? for example, would you want to build cold code with the medium code model but hot code with the small code model?

5

We link non-trivial amount of pre-built third-party code. A lot of projects are not buckified yet.

6

To add a few more thoughts here:

The medium and large code models were designed to support mixing with small code model code. The medium code model allows large data while maintaining compatibility with small code model assumptions for code sections. I haven’t found any documentation stating that mixing code models requires restricting to a pure C API or avoiding COMDAT sections.

Also worth noting: the example I provided is valid, ODR-compliant source code. The only difference is the compilation flags used—the semantic content is identical. This is different from a true ODR violation where the source definitions differ.

Looking at the impact of this change:

Scenario With This Fix
Same code model for everything No change
Mixed code models, no conflicting COMDATs No change
Mixed code models, small sections encountered first No change
Mixed code models, large section selected over small Fixes the link failure

The proposed change only affects the one scenario that is already broken (relocation overflow). It’s deterministic—always prefer small—and improves compatibility without affecting any existing working builds.

The “first COMDAT wins” behavior is an implementation detail. Small code model relocations physically cannot reach large sections placed beyond ±2GB. Preferring small sections avoids a hard failure.

7

I’m by no means an expert, but I’m inclined to agree with @grigorypas: this isn’t an ODR violation, nor is it a COMDAT group violation, at least by a literal reading of the standard. Equally, there’s nothing in the standard about selection of COMDAT sections other than that one has to be selected.

The current ELF spec says this about GRP_COMDAT:

This is a COMDAT group. It may duplicate another COMDAT group in another object file, where duplication is defined as having the same group signature. In such cases, only one of the duplicate groups may be retained by the linker, and the members of the remaining groups must be discarded.

The duplication is purely defined based on the group signature, which is just a symbol name. The contents of the group are not required by the ELF standard to be identical by any means. Indeed, the lack of more requirements seems like a deliberate decision, not an oversight. A simple example for why differences between different COMDAT groups would be appropriate would be debug data could be present in one case, but not another. It’s hard to argue that this would be a violation of anything other than people’s debugging experience, if the one without debug data were picked.

This brings me to whether linkers should always pick the first COMDAT entry. Again, there is nothing in the GRP_COMDAT rules that says they must do this: it’s purely a convention. Up to now there hasn’t been a motivation to do something different, but it seems to me that if there is a motivation, we should be open to changing what we do, if it is practical to do so (noting in particular that if we choose not to because of practicality, we end up failing to link things in the use-case under discussion).

On the topic of ODR-violation, I cannot see anything in the rules on cppreference.com that require two templates (the typical cause of COMDATs) to be compiled with the same flags etc, as long as they have identical tokens. I acknowledge that things like small/large code model are outside the scope of the C++ standard. I haven’t got the time or expertise to look into things like the psABI for x86_64 to review, but it’s my understanding that they are intended to be compatible.

1 Like
8

While I wouldn’t call this an ODR violation, and is within the bounds of the ELF specification, I think it may x86_64 ABI impact. Essentially what does the ABI say about the combination of objects from the small and objects from the large code-models?

I’m not familar with the x86_64 ABI, only the AArch64 ABI. In the latter the combination of objects from the small code model and objects from the large code model is possible, but the result would have program size constraints of the small code model. This is similar to combining position-independent and position-dependent code, the result is position-dependent code.

This seems to be a case where the combination of, presumably limited numbers, of small code-model and large code model, ends up with the large-code model. However to guarantee this it imposes a requirement on a static linker to not mix small-code model code and large-code model data. Altering COMDAT selection rules is one way to achieve the requirement, I think there may be other ways too [1].

The reason I mention the x86_64 ABI is whether it guarantees this combination, or whether this what the AArch64 ABI calls quality of implementation, where LLD offers a guarantee that GNU ld doesn’t so programs may link on LLD, but fail on GNU ld.

As this is the x86_64 specific it isn’t my place to say whether this behaviour is the right thing or not. From an LLD implementation perspective I’m nervous of the additional implementation complexity due the old Arm implementation getting in the way of refactoring, having harder to predict selection behaviour, and a few unforeseen hard to track down bugs. Having an ABI require the behaviour is a strong argument in favour of making the change.

[1] As an aside, is there another way that this could be implemented without affecting COMDAT group selection? As I understand it, the problem occurs due to the introduction of, presumably data, sections with the SHF_X86_64_LARGE flag that have R_X86_64_PC32 relocations against them.

Prior to assignment of input sections to output sections, could we iterate over the relocations, if there’s a R_X86_64_PC32relocation to a section with SHF_X86_64_LARGEthen remove the SHF_X86_64_LARGEflag. Then those sections will get placed within range, assuming that’s possible given the size of the program.

If that worked, for me, while it may not perform as well, would be a cleaner more isolated change, that we could put behind an opt-in flag so users were aware of potential incompatibilities with other x86_64 linkers.

9

I would try to reframe this as a question of, under what conditions should we consider small, large, and medium code to be ABI compatible? During the development of medium code model support at Google, @aeubanks implemented the -mlarge-data-threshold=N flag, and our understanding at the time (see @jyknight 's comment on the original review) was that the threshold “is ABI”, meaning you can’t reasonably expect to link together objects compiled with different large data threshold settings. Linking together small / medium objects is sort of a degenerate case of that (large data threshold :infinity: ).

What we really wanted to deal with at the time was to enable linking in fragments of assembly coded to use the small code model (think ffmpeg and other codec libraries) and other third party SDKs with limited API surface area without too much disruption. I’ll admit, this was not as seamless as I had hoped. In some cases, we had to reach for the __attribute__((code_model("small"))) attribute to explicitly move globals into the small core of the ELF binary.

Perhaps what we can say is something like, if your ABI boundary consists of default-visibility code symbols (that would all use PLT/GOT relocations), you can mix small/medium/large code. This could be documented.

I worry that it’s not sufficient to have LLD patch up this COMDAT case. In the general case, if you refer to large data defined in a medium code model TU and referenced in a small code TU, that access pattern will be short. However, maybe that would only happen for hidden globals (-fvisibility=hidden). Interpret this as general caution that there are probably more ABI breaking issues when mixing these modes.

10

The example doesn’t demonstrate what you need. user.o doesn’t use COMDAT groups. The R_X86_64_32S relocation you see is due to -fno-pic

# you likely use a clang that defaults to -fno-pic for Linux
clang++ -c -mcmodel=small user.cpp -o user.o -fno-pic

R_X86_64_32S has a range of [0,2**31) (if we ignore the negative area), and is problematic for larger executables.

If we compile user.cpp with -fpie, there is no linker error.

11

This is similar to the discussions at https://groups.google.com/g/x86-64-abi/c/RsJDf06xMJ0/m/wLDbVhOyCAAJ

I am wondering if the “optimal” solution is to propose a new code-model so that less concern can be made about whether ABI compatibility is maintained.

New code-model: “large-ish”

Changes could include:

  • support sdata8 for various encodings that are conventionally set as sdata4
  • flag for COMDAT on the section group to specify ordering
  • support for multiple GOT
12

Reply to @MaskRay:

The example can be adapted to PIC/PIE as well. To demonstrate the COMDAT issue properly:

  1. Instantiate the template explicitly in both provider and user:

template struct Data<int>;
  1. Compile with -fpie (or -fPIC -fvisibility=hidden)

In practice, I encountered this problem with vtables in COMDAT sections.

Reply to @rnk:

I understand that the medium code model was designed with the expectation that all code would be compiled with the same code model and threshold parameter. However, I don’t see fundamental restrictions preventing mixing code models. The decision of whether to place data into small or large sections is local (done per module).

The only problems I see are:

  1. COMDAT sections (which this RFC addresses)

  2. Potentially using extern variables that are not referenced through GOT

I haven’t encountered the second case in practice, since pre-built code typically consists of third-party libraries that are self-contained and don’t rely on code we build from sources.

You’ve already demonstrated flexibility with mixing code models (inline assembly and third-party C APIs). It seems reasonable to extend this further to support broader use cases where pre-built third-party libraries (compiled with a different code model) need to be linked with application code. This is a practical scenario that developers may encounter, and solving for a couple of edge cases would make the toolchain more robust in these situations.

Reply to @smithp35:

The reason I mention the x86_64 ABI is whether it guarantees this combination, or whether this what the AArch64 ABI calls quality of implementation, where LLD offers a guarantee that GNU ld doesn’t so programs may link on LLD, but fail on GNU ld.

Why is this problematic? This would be backward compatible—LLD would be able to do what GNU ld does and more.

[1] As an aside, is there another way that this could be implemented without affecting COMDAT group selection?

We would also need to rename sections from .data... to .ldata... (or .bss/.lbss). We would still need to track when the linker chose to keep a large section and subsequently encounters a small section with the same signature. In terms of complexity, this seems comparable.

General Question:

We could guard this change behind a flag, allowing users to opt-in to this behavior. Would this approach be acceptable/preferable to the community?

13

It depends on how much value is being put on portability across toolchains. For a project that standardises on lld this is not a problem at all. For programs that don’t control their linker like LLVM, this could lead to seemingly random out of range relocation errors when moving a program developed on lld to GNU ld.

Without some kind of ABI guarantee, I think an opt-in flag would help in this situation, as in effect the program becomes dependent on LLD’s implementation.

14

For programs that don’t control their linker like LLVM, this could lead to seemingly random out of range relocation errors when moving a program developed on lld to GNU ld.

Putting aside the specifics of this issue, is it generally a problem if LLD allows a superset of links to succeed compared to binutils? Relaxing limitations seems like a great way for LLD to add value; and I’d kind of expect it to be opt-in to make LLD fail the same way to make sure projects were portable to both. This feels morally similar to old issues like number of significant characters in identifiers; it doesn’t seem like a good idea to constrain implementations to the lowest common denominator by default, but it could be an option for those willing to go the extra mile to ensure their program only requires the lowest common denominator.

15

I don’t think it is a general problem to have a superset of links to succeed. Sometimes this will happen by a quirk of a different layout decision, better optimisation etc. I do think it is beneficial that if we do have a significant behaviour difference, we document it, and at least see if we can implement an opt-in diagnostic for those wanting compatibility.

Just having the documentation helps. While it is unlikely that it will be ahead of time, it can help to explain why a link is failed and what to do about it.

I think –warn-backrefs — lld 23.0.0git documentation is an example of where there is a significant difference between GNU ld and lld (and some other linkers), that there is some documentation and an opt-in diagnostic.

1 Like
16

Can you edit your previous message to fix the code example?

Your suggested template struct Data<int>; does not work, either.

For user.cpp,

template<typename T>
struct Data {
    static T value;
};
template struct Data<int>;

int main() {
    return Data<int>::value;  // R_X86_64_32S relocation
}

doesn’t generate a COMDAT group. For this RFC to move forward, we need a concrete, self-contained reproduction script that demonstrates preferring the small code model COMDAT is the solution. If your goal is to ensure the small-model version is used, using explicit instantiation declarations in those specific object files seems like a much cleaner solution than changing LLD’s selection logic.

17

Fair enough. Here is an updated example:

// data.h                                                                                                                                                  
  #pragma once                                                                                                                                               
  #include <vector>                                                                                                                                          
                                                                                                                                                             
  template<typename T>                                                                                                                                       
  struct Cache {                                                                                                                                             
      static std::vector<T> data;                                                                                                                            
      static void add(T val) { data.push_back(val); }                                                                                                        
  };                                                                                                                                                         
                                                                                                                                                             
  template<typename T>                                                                                                                                       
  std::vector<T> Cache<T>::data;                                                                                                                             
                                                                                                                                                             
  // provider.cpp                                                                                                                                            
  #include "data.h"                                                                                                                                          
                                                                                                                                                             
  void init() {                                                                                                                                              
      Cache<int>::add(42);                                                                                                                                   
  }                                                                                                                                                          
                                                                                                                                                             
  // user.cpp                                                                                                                                                
  #include "data.h"                                                                                                                                          
                                                                                                                                                             
  int main() {                                                                                                                                               
      Cache<int>::add(1);                                                                                                                                    
      return Cache<int>::data.size();                                                                                                                        
  }                                                                                                                                                          
                                                                                                                                                             
  # far_pie.lds                                                                                                                                              
  SECTIONS {                                                                                                                                                 
    .text : { *(.text*) }                                                                                                                                    
    .rodata : { *(.rodata*) }                                                                                                                                
    .data.rel.ro : { *(.data.rel.ro*) }                                                                                                                      
    .dynamic : { *(.dynamic) }                                                                                                                               
    .got : { *(.got*) }                                                                                                                                      
    .data : { *(.data*) }                                                                                                                                    
    .bss : { *(.bss*) }                                                                                                                                      
    . = . + 0x100000000;                                                                                                                                     
    .ldata : { *(.ldata*) }                                                                                                                                  
    .lbss : { *(.lbss*) }                                                                                                                                    
  }                                                                                                                                                          
                                                                                                                                                             
  # Commands                                                                                                                                                 
  clang++ -c -fPIC -fvisibility=hidden -mcmodel=medium -mlarge-data-threshold=0 provider.cpp -o provider.o                                                   
  clang++ -c -fPIC -fvisibility=hidden -mcmodel=small user.cpp -o user.o                                                                                     
  clang++ -fuse-ld=lld -pie -Wl,-T,far_pie.lds -o test provider.o user.o                                                                                     
                                                                                                                                                             
  # Error                                                                                                                                                    
  ld.lld: error: user.o:(function main: .text+0x1c): relocation R_X86_64_PC32 out of range: 4294971456 is not in [-2147483648, 2147483647]; references       
  'Cache<int>::data'; R_X86_64_PC32 should not reference a section marked SHF_X86_64_LARGE                                                                   
  >>> referenced by user.cpp                                                                                                                                 
  >>> defined in provider.o

(I don’t seem to have the option to edit my previous post).

18

As already stated, the problem of referencing a symbol with small relocations but the symbol ends up being defined as large can be due to conflicting code models in COMDAT sections, or referencing an extern global without going through the GOT (e.g. __attribute__((visibility(“hidden”)))) from a small TU but the extern global is defined as large in another TU. There is no way to work around the latter aside from extreme hacks like the linker removing SHF_X86_64_LARGE and changing section names if there are any 32-bit relocations to the symbol. Even though the latter seems to not happen in practice, these two ways are similar enough that I’d be hesitant to standardize this in the x86-64 ABI. In other words, if we are going to do this I’d support this being an lld extension (whether or not under a flag) rather than in the spec.

Somewhat related, I had a draft patch a while ago to warn in lld when there was any 32-bit relocation referencing any large data but that never landed due to performance concerns.

I do wonder for how many globals you’re running into this issue. We’ve used __attribute__((model("small"))) in some cases to help with this sort of issue, although for handwritten assembly instead of the problem here. We’d also considered extending something like API Notes: Annotations Without Modifying Headers — Clang 23.0.0git documentation to allow marking C/C++ symbols with attributes via command line flags without modifying sources, although unsure how much work that is. Overall, if the number of globals you’re running into this issue with is small this is potentially a pragmatic alternative, but of course this is a lot less maintainable and won’t scale if you have lots of problematic globals.

19

Thanks for the detailed response, @aeubanks.

To answer your question about the scale of the issue: for a typical large binary at Meta, I’ve counted 300+ cases where small code model code references large sections.

When mixing code models, the most likely scenario is that small code model code comes from pre-built third-party libraries, where modifying source code or applying per-symbol annotations isn’t practical. This is exactly the case for us.

Regarding the hidden visibility scenario: this seems unlikely in practice. Pre-built third-party libraries are generally self-contained, so their hidden symbols would be defined within the same library and compiled with the same code model.

I’m fine with treating this as an lld extension behind an opt-in flag rather than pursuing ABI standardization. Happy to proceed with implementation if this sounds reasonable.

20

Thanks for fixing your code example.

I agree with aeubanks that this approach is not generic enough.
I have significant reservations about whether this justifies an upstream change.
The current argument describes a niche requirement specific to a particular build system/user, which might be better addressed via a local patch or a more robust fix at the source/build-system level (e.g., ensuring the preferred definition appears first in the link order).

Linkers should be driven by general ELF policies. Deviating from the regular COMDAT selection rule and arch-agnostic behavior to accommodate specific code-model mixing trouble for a user introduces a precedent that is difficult to maintain.

In addition, introducing special logic to determine symbol precedence is likely fundamentally incompatible with the parallel symbol resolution we are moving toward.

I suggest that you make such a change to your data.h: template<typename T> [[gnu::model("small")]] std::vector<T> Cache<T>::data;