Summary
We propose upstreaming the Propeller profile conversion tool (provisionally named llvm-propeller) into llvm/tools. Propeller[1] is a post-link optimization (PLO) framework that improves application performance via precise function and basic block layout. While the Clang (-fbasic-block-sections) and LLD (-lto-basic-block-sections) components are already upstream, the critical tool for generating the Propeller profile remains external (google/llvm-propeller).
Integrating this tool will provide LLVM users with a complete, dependency-free, out-of-the-box solution for PLO. By leveraging a relinking-based approach, Propeller offers a scalable option designed for distributed and incremental build environments, serving as an alternative to binary-rewriting frameworks like BOLT[2].
Background & Motivation
Currently, using Propeller requires a disjointed workflow involving an external repository (google/llvm-propeller). Moving the tool to llvm/tools allows it to be distributed alongside standard tools like llvm-profgen in Profile Guided Optimizations (PGO) infra, drastically simplifying the user experience.
Proposed Architecture
We propose adding llvm-propeller to llvm/tools/. The llvm-propeller uses raw hardware performance profiles (Linux perf) and binary metadata to generate the build artifacts required for relinking and recompilation. The proposed tool functions as a standalone utility, adhering to the standard LLVM toolchain design:
Input
The llvm-propeller is designed with an abstract profile reader interface to take various types of execution profiles. The initial upstream version will support the text format output from Linux perf script for LBR and SPE branch data. Using perf script as input eliminates a dependency on the unstable binary perf.data format (and the libraries for parsing it), aligning llvm-propeller with other profile-driven tools like llvm-profgen and perf2bolt.
In the future, we plan to support other trace formats (e.g., Lauterbach TRACE32, SEGGER, ARM SPE, and Fuchsia profiles) within the LLVM ecosystem. An abstract reader interface will facilitate the integration of these formats as they become available, potentially via a shared library with llvm-profgen.
In addition to profile data, the tool accepts the original binary executable or object files, built with the -fbasic-block-address-map Clang flag.
Core Logic
llvm-propeller first parses execution paths from the hardware perf branch data generated by perf script. It then maps these addresses to LLVM Basic Blocks (BB) using the binary’s address map (SHT_LVM_BB_ADDR_MAP) section and constructs a Control Flow Graph (CFG) attributed with accurate edge weights. Finally, it makes whole-program optimization decisions, including basic block layout (e.g., Ext-TSP), function splitting, and path cloning.
Output
The llvm-propeller outputs two coordinated artifacts: cc_profile.txt, which drives compiler-side transformations (such as function splitting and path cloning), and ld_profile.txt, which instructs the linker (LLD) on the final optimal code block layout ordering.
Relation with Existing Tools
llvm-propeller is akin to the existing suite of LLVM performance tools:
llvm-profgen
Both llvm-profgen and llvm-propeller process hardware perf data, but they serve different optimization pipelines.
-
llvm-profgenconvertsperfsample profiles into a DebugInfo-based format consumed by CSSPGO and AutoFDO. Its primary goal is to provide the PGO profile for guiding compiler transformations such as function inlining, register allocation, and instruction expansion during the compilation phase. -
llvm-propellerperforms a whole-program analysis to reconstruct the CFG. It generates explicit directives (e.g.,cc_profile.txt,ld_profile.txt) that instruct the compiler and linker exactly how to layout basic blocks and functions in the final binary.
While llvm-profgen and llvm-propeller share the same input source (hardware perf data) and LLVM libraries (e.g., perf script), they produce fundamentally different outputs for different consumers. Keeping these tools separate allows each to evolve independently.
llvm-perfdata
llvm-profdata manipulates the PGO profile generated from various sources (e.g., instrumentation or llvm-profgen). It already serves as the primary tool for Instrumentation FDO (iFDO), AutoFDO (with the --sample flag), and MemProf. Its architecture is complex with different flags for each profiling method.
llvm-propeller, by contrast, generates linker-specific layout profiles from raw hardware perf data. Integrating this logic into llvm-profdata would further increase its complexity without any clear benefit. Therefore, a standalone tool is the reasonable choice to maintain separation tools between compiler-side PGO and linker-side layout generation.
BOLT
BOLT and Propeller are both post-link optimizers (PLO) in LLVM, but they serve different functional needs.
-
BOLT operates via binary rewriting. It takes a final linked binary, disassembles, optimizes, and rewrites a new binary. This approach is particularly useful in scenarios where source code or intermediate BUILD artifacts are unavailable, or in constrained build environments where recompilation is not feasible. BOLT also provides a flexible framework for prototyping whole-program optimizations.
-
Propeller operates via relinking. It uses the Propeller profile to drive the compiler and linker to reconstruct code. This approach is designed for distributed and incremental build systems, where leveraging build caching and parallelism is critical for scalability.
BOLT and Propeller are complementary. Integrating Propeller gives LLVM users the flexibility to choose the PLO strategy that best fits their infrastructure.
Dependency Removal
Although the google/llvm-propeller repository has a number of external dependencies, we are refactoring the codebase to depend on LLVM only.
Testing Strategy
Besides the standard unit and regression tests (using llvm-lit), we plan to provide a CMake configuration that facilitates a full bootstrap build of Clang using Propeller optimization. This allows developers to easily verify the end-to-end workflow from profile generation to linker optimization (BB layout, function splitting, and path cloning) on compatible hardware.
Integration Roadmap
We are committed to an upstreaming process that minimizes review workloads. Given the size (~10k LOC) of llvm-propeller, we strictly avoid a massive code dump. Instead, we will break the upsteaming into a series of small, logically independent, and reviewable patches. The initial patches will focus on the build system skeleton and core data structures, followed by the gradual upstream for the profile reader (refactoring shred logic from llvm-profgen), the optimization algorithms, and the profile writer.
References
[1] H Shen, K Pszeniczny, R Lavaee, S Kumar, S Tallam, XD Li, “Propeller: A Profile Guided, Relinking Optimizer for Warehouse-Scale Applications,” in Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS '21), 2021. https://dl.acm.org/doi/abs/10.1145/3575693.3575727
[2] M. Panchenko, R. Auler, B. Nell, and G. Ottoni, “BOLT: A Practical Binary Optimizer for Data Centers and Beyond,” in Proceedings of the 2019 IEEE/ACM International Symposium on Code Generation and Optimization (CGO '19), 2019. BOLT: A Practical Binary Optimizer for Data Centers and Beyond | IEEE Conference Publication | IEEE Xplore