[RFC] Improving compact x86-64 compact unwind descriptors

Project Infrastructure LLVM Dev List Archives
45

I’ve ported the Mach-O compact unwind format to ELF in a branch, establishing a baseline for an asynchronous unwinding extension.

% ~/Dev/object-file-size-analyzer/section_size.rb /tmp/out/custom-{fp,sframe,compact,fp-gcc,sframe-gcc}/bin/{llvm-mc,opt}
Filename                               |       .text size |        EH size |   .sframe size |  VM size | VM increase
---------------------------------------+------------------+----------------+----------------+----------+------------
/tmp/out/custom-fp/bin/llvm-mc         |  2120895 (23.5%) |  301528 (3.3%) |       0 (0.0%) |  9043221 |           -
/tmp/out/custom-sframe/bin/llvm-mc     |  2109231 (22.3%) |  367424 (3.9%) |  348041 (3.7%) |  9474085 |       +4.8%
/tmp/out/custom-compact/bin/llvm-mc    |  2109519 (24.4%) |  106288 (1.2%) |       0 (0.0%) |  8639637 |       -4.5%
/tmp/out/custom-fp-gcc/bin/llvm-mc     |  2744214 (29.2%) |  301836 (3.2%) |       0 (0.0%) |  9389677 |       +3.8%
/tmp/out/custom-sframe-gcc/bin/llvm-mc |  2705860 (27.7%) |  354292 (3.6%) |  356073 (3.6%) |  9780985 |       +8.2%
/tmp/out/custom-fp/bin/opt             | 38769545 (69.9%) | 3547688 (6.4%) |       0 (0.0%) | 55425217 |           -
/tmp/out/custom-sframe/bin/opt         | 38891295 (62.4%) | 4559644 (7.3%) | 4448874 (7.1%) | 62292133 |      +12.4%
/tmp/out/custom-compact/bin/opt        | 38898415 (74.8%) | 1200764 (2.3%) |       0 (0.0%) | 52020449 |       -6.1%
/tmp/out/custom-fp-gcc/bin/opt         | 54654215 (78.1%) | 3631196 (5.2%) |       0 (0.0%) | 70001373 |      +26.3%
/tmp/out/custom-sframe-gcc/bin/opt     | 53644895 (70.4%) | 4857364 (6.4%) | 5263676 (6.9%) | 76206149 |      +37.5%

Build configurations:

#!/bin/zsh
conf() {
  configure-llvm $1 -DCMAKE_EXE_LINKER_FLAGS='-fuse-ld=bfd -pie -Wl,-z,pack-relative-relocs' \
    -DCMAKE_SHARED_LINKER_FLAGS=-fuse-ld=bfd -DLLVM_ENABLE_UNWIND_TABLES=on -DLLVM_ENABLE_LLD=off ${@:2}
}

clang=(-DCMAKE_CXX_COMPILER=/tmp/Rel/bin/clang++ -DCMAKE_C_COMPILER=/tmp/Rel/bin/clang)
gcc=("-DCMAKE_C_COMPILER=$HOME/opt/gcc-15/bin/gcc" "-DCMAKE_CXX_COMPILER=$HOME/opt/gcc-15/bin/g++")

compact="-fomit-frame-pointer -momit-leaf-frame-pointer -B$HOME/opt/binutils/bin -mllvm -elf-compact-unwind -mllvm -x86-epilog-cfi=0"
fp="-fno-omit-frame-pointer -momit-leaf-frame-pointer -B$HOME/opt/binutils/bin -Wa,--gsframe=no"
sframe="-fomit-frame-pointer -momit-leaf-frame-pointer -B$HOME/opt/binutils/bin -Wa,--gsframe"

conf custom-compact -DCMAKE_{C,CXX}_FLAGS="$compact" ${clang[@]} \
  -DCMAKE_EXE_LINKER_FLAGS='-fuse-ld=lld -pie -Wl,-z,pack-relative-relocs' \
  -DCMAKE_SHARED_LINKER_FLAGS=-fuse-ld=lld

conf custom-fp -DCMAKE_{C,CXX}_FLAGS="-fno-integrated-as $fp" ${clang[@]}
conf custom-sframe -DCMAKE_{C,CXX}_FLAGS="-fno-integrated-as $sframe" ${clang[@]}

conf custom-fp-gcc -DCMAKE_{C,CXX}_FLAGS="$fp" ${gcc[@]}
conf custom-sframe-gcc -DCMAKE_{C,CXX}_FLAGS="$sframe" ${gcc[@]}

for i in compact fp sframe  fp-gcc sframe-gcc; do ninja -C /tmp/out/custom-$i llvm-mc opt; done

The /tmp/out/custom-compact build uses my llvm-project branch (http://github.com/MaskRay/llvm-project/tree/demo-unwind) that ports Mach-O compact unwind to ELF, allowing the majority of .eh_frame FDEs to replace CFI instructions with unwind descriptors.
Linker behavior:

  • Split FDEs into two groups: descriptor-based (augmentation ‘C’) and instruction-based
  • Generate .eh_frame_hdr version 2 with 12-byte table entries when compact FDEs are present: (pc_ptr, unwind_descriptor_or_fde_ptr). Compact FDEs described by .eh_frame_hdr inline are removed from the output .eh_frame section.

Note: .ARM.exidx and MIPS compact exception tables also describe unwind descriptors inline in a binary search index table.

FDEs not representable by compact unwind (e.g. shrink wrapping optimization) use the traditional CFI instructions (called DWARF escape in the Mach-O compact unwind information).

This implementation involves several components:

  • -mllvm -elf-compact-unwind: Emits .eh_frame CIEs with augmentation character ‘C’ and FDEs using unwind descriptors.
  • -mllvm -x86-epilog-cfi=0: Disables epilogue CFI for x86 (primarily implemented by D42848 in 2018, notably disabled for Darwin and Windows).
    Without this option most frames will not utilize unwind descriptors because the current Mach-O compact unwind implementation does not support popq %rbp; .cfi_def_cfa %rsp, 8; ret.
    I believe this is still fair as we expect to use a 8-byte descriptor, sufficient to describe epilogue CFI.
  • lld/ELF changes: FDEs are split into descriptor-based (augmentation ‘C’) and CFI-instruction-based groups. When compact FDEs are present, .eh_frame_hdr version 2 is generated with 12-byte table entries containing (pc_ptr, unwind_descriptor_or_fde_ptr). The PC pointer remains 4 bytes, while the 8-byte entry indicates either an unwind descriptor (odd value) or an FDE pointer (even value).

With the current implementation, 4937 out of 77648 FDEs (6.36%) require a DWARF escape, while the remaining FDEs can be replaced with unwind descriptors.

.eh_frame_hdr will become even smaller if we implement the two-level page table structure in Mach-O __unwind_info.

After I had implemented this, I then investigated the MIPS compact exception tables. I can now finalize the ‘in construction’ chapter of my blog post, https://maskray.me/blog/2020-11-08-stack-unwinding#mips-compact-exception-tables with the following notes:)

The specification is available at https://github.com/itanium-cxx-abi/cxx-abi/blob/main/MIPSCompactEH.pdf.

Binutils added support in 2015, though the GCC patch remains unmerged.

The following is based on my understanding of this format.

Compiler output. The directive .cfi_sections .eh_frame_entry instructs the assembler to emit index table entries to the .eh_frame_entry section.
.cfi_fde_data opcode1, ... betweens a pair of .cfi_startproc and .cfi_endproc describes the frame unwind opcodes where each opcode takes one byte.
The frame unwind opcodes describes the semantics of prologue instructions, similar to Windows ARM64 Frame Unwind Codes.

Assembler processing. The assembler generates a .eh_frame_entry.* section for each section with compact unwind information.
Each .eh_frame_entry is a pair of 4 bytes, where the first word is like the first word in a .eh_frame_hdr entry.
An .eh_frame_entry entry takes one of three forms:

  • Inline compact: (even pc, unwind_data). This form can be used when there are at most 3 opcodes (3 bytes) and no personality routine.
  • Out-of-line compact: (odd pc, even unwind_ptr) where unwind_ptr points to unwind data in the .gnu_extab section.
  • Legacy: (odd pc, odd legacy_unwind_ptr) where legacy_unwind_ptr points to the legacy .eh_frame section.

TODO: Describe .cfi_inline_lsda, which appears related to __gnu_compact_pr[1-3].

Linker processing. GNU ld concatenates .eh_frame_entry and .eh_frame_entry.* sections, sorting them by address.
The following internal linker script fragment adds a header before the entries:

.eh_frame_hdr   : { *(.eh_frame_hdr) *(.eh_frame_entry .eh_frame_entry.*) }

Although the section name remains the traditional .eh_frame_hdr, the version is set to 2.
The linker also defines the symbol __GNU_EH_FRAME_HDR to hold the .eh_frame_hdr address.

TODO: Describe LSDA representation, which is more compact than the traditional .gcc_except_table section.

While the current implementation seems synchronous only, extending it to asynchronous unwinding is natural.
https://inbox.sourceware.org/gcc-patches/55F0C4D5.6080507@redhat.com/ suggests that opcodes can be introduced to describe DW_CFA_remember_state/DW_CFA_restore_state.
TODO: This might be similar to END_C in Windows ARM64 for zero-length prologs that still have unwind state.

6 Likes
LLVM Weekly - #622, December 1st 2025