Disclaimer: This document was generated with some help from Gemini and subsequently verified and proofread by me.
We propose adding a new ETMReader to llvm-profgen to support the decoding of ARM Embedded Trace Macrocell (ETM) bitstreams. This enables a hardware-assisted PGO workflow for ARM targets, initially focusing on 32-bit architectures, allowing developers to generate detailed profiles from deterministic hardware instruction streams.
Background
llvm-profgen currently transforms execution data into profiles usable by llvm-profdata. While it traditionally processes sampled data from the Linux perf tool (such as LBR and BRBE), it serves as the natural architectural home for hardware-assisted PGO workflows that leverage dedicated on-chip trace logic.
For many embedded systems:
These systems often run without an OS (bare-metal), which makes software-based sampling or perf-subsystem dependencies difficult to implement.
Instrumentation-based PGO might not be feasible due to strict code-size constraints, the impact of instrumentation overhead on real-time behavior, and the fact that resulting profile data can be large or difficult to exfiltrate from the target.
Hardware-assisted PGO (via ETM) provides an independent, deterministic alternative that captures the complete program flow without modifying the application binary or consuming CPU cycles for monitoring.
The Distinction
We refer to this mechanism as hardware-assisted PGO to reflect the detailed, deterministic nature of ETM data. While the end-to-end workflow resembles the standard AutoFDO pipeline, the underlying mechanism is different. AutoFDO traditionally implies a sampling mechanism (like Linux perf); however, because ETM provides a continuous, hardware-generated stream rather than a set of random samples, the term AutoFDO doesn’t fully capture the data quality provided. By using the term hardware-assisted PGO, we highlight the fine detail gained by using a dedicated hardware component to assist the compiler in understanding program behavior.
Proposed Changes
We propose integrating theCoreSight Decoding Library (OpenCSD) as an optional dependency to handle ETM. While this RFC focuses on M-profile (Cortex-M) cores, OpenCSD abstracts core-specific differences. This design ensures that we can add A-profile (Application-class) support in the future without any major changes to the decoder framework. Note that OpenCSD does not yet support certain features of later M-class cores, such as Branch Future instructions, so these can be integrated as the library matures.
New Input Format: We introduce the --etm flag to accept raw binary trace files. This allows llvm-profgen to process ETM data exported from common hardware debug probes (e.g., Lauterbach TRACE32 or SEGGER J-Trace).
ETMReader: We implement a specialized reader class that uses the OpenCSD library to decode raw ETM packets into an instruction stream. This reader acts as the bridge between the hardware bitstream and llvm-profgen’s internal infrastructure. It initializes the OpenCSD DecodeTree with the necessary protocol-specific decoders and packet processors. As the bitstream is fed into the library, the reader processes specific decoder outputs—such as branch executions and instruction range markers—to accurately reconstruct the original program flow.
Logic: The reader maps executable ELF segments to provide opcodes to the decoder, reconstructs the control flow, and aggregates instruction ranges into SampleCounters. Unlike traditional sampling, this approach provides a full reconstruction of the executed paths.
Users: Developers targeting ARM Cortex-M can generate detailed profiles in resource-constrained environments where instrumentation might not be an option. Furthermore, ETM provides precise branch information on Arm systems that lack the newer Branch Record Buffer Extension (BRBE). By enabling branch-aware profiling on a wider range of hardware, llvm-profgen allows developers to use advanced optimization workflows like BOLT and AutoFDO on existing Arm servers and embedded devices.
Accuracy: By using deterministic hardware logic rather than random samples, the tool might produce more accurate values than a sample based profile mechanism, but this comes at the cost of a much larger profiling data size.
Compatibility: The generated profiles remain compatible with the existing llvm-profdata and Clang’s sample based profile pipeline via -fprofile-sample-use.
Future Work
We have built an extensible architecture and may add support for Application-class (Cortex-A) ETM in the future if a specific use case arises.
Thank you for the proposal and a prototype, I think this is valuable. It’s worth emphasizing that the goal is to have a capability to parse raw ETM traces that are not captured/parsed by Linux perf.
Since you’ve mentioned BOLT, it would make sense to add ETM parser as a common library so BOLT can leverage it too (I thought about adding perf+ETM parser to address perf data conversion overheads with large traces).
One other thing to mention: since ETM traces can encode long instruction streams, it might be feasible to also add calling context tracking, similar to what’s envisioned by perf itrace flag g:
g synthesize a call chain (use with i or x)
This could provide context sensitivity and potentially enable CSSPGO (with limited context depth constrained by trace depth).
Thank you for adding the ETM support! Could you also share the ETM encoding format and specs? What instructions will be recorded in the branch/target pair? And is it able to record samples in different run mode so we can tell apart kernel code or user code etc.?
1) Common library: I have refactored the ETM parsing logic into a new ETMTraceDecoder component within LLVMProfileData and updated the PR. This allows BOLT to link against the decoder directly if needed.
2) Context sensitivity: Let me think about this a little bit and see how it could potentially be incorporated.
Thank you for the RFC. A quick disclaimer that I’m not an expert in trace.
I think the approach looks good, ETM is a good solution for trace in embedded systems when hardware is available. Using OpenCSD is also sensible as there are many versions of ETM and it can abstract away the differences.
IIRC Bolt is AArch64 only, so if the initial support is M-profile only it wouldn’t see any direct improvement from llvm-profgen until AArch64 support were added.
Thank you for adding the ETM support! Could you also share the ETM encoding format and specs?
And is it able to record samples in different run mode so we can tell apart kernel code or user code etc.?
Looking at the OpenCSD documentation it is possible to associate an address range with an exception level. In principle that could be used to distinguish between user and kernel code. I expect that might need more command-line options to drive OpenCSD.
I’ve left a couple of comments about Arm target detection in the prototype.
The ETM trace protocol tracks the current exception level, i.e. it tells you EL at the start of decode and on every change. So it shouldn’t ever be necessary to drive it off address ranges. (Similarly it tells you if you are in A32, T32 or A64 instruction state.)
On trace size - ETM traces can be large, especially when fully decoded, because they can run continuously. But they capture an enormous number of branches - if you run ETM trace for 1 second you might capture millions of branches. Its density as branches per byte of trace is on the order of a few bytes per branch with timing, closer to 1 or 2 bits per branch without timing. To get smaller trace sizes, there are several solutions - address range filttering (trace only within specified address ranges), address start/stop filtering (start or stop trace when hitting particular addresses) and trace strobing (periodically strobe the trace on and off). These filters can be used independently or in combination and operate within the ETM unit and need no software intervention other than initial setup, so they are compatible with embedded use where ETM config is from embedded code or an external debugger.
Another question about the trace file format. I believe Linux perf also have ETM support, but that seems to be reading the traces right off the ETB mapped RAM. I’m not too familliar with the OpenCSD tho. What kind of input format does it take? And what input format are we proposing here? Does it differ from the perf data? I mean if the input format is from vendored JTAG, would it be better to pre-process the ETM traces into a compatible “Stack + Branch/Targets” format like here: llvm-project/llvm/tools/llvm-profgen/PerfReader.cpp at main · llvm/llvm-project · GitHub ?
ETM creates a highly compressed binary trace stream. On some systems, streams from multiple ETMs can be multiplexed together, into a “formatted” stream. OpenCSD consumes either a formatted stream or an unformatted stream and decodes it into a series of branches. It doesn’t really matter where the stream comes from, whether it’s captured on-chip or via an external trace port.
In Linux perf, the kernel will handle capturing the data on-chip, either in an ETB or by routing the trace to RAM. It then passes the captured buffers up to userspace where “perf record” will put them as AUX records in perf.data. Later, “perf script” and other tools can post-process the raw data to get branch targets. This is done by putting the trace through OpenCSD, and also supplying OpenCSD with the binary images that it needs in order to reconstruct branch targets. This requires additional sideband information that perf has captured about process address space changes.
I.e. processing the ETM traces into branch targets is a post-processing step, not a pre-processing step.
Regarding context sensitivity:@aaupov I’ve given this some more thought. Reconstructing the calling context is a great suggestion and would definitely enhance the profile quality. However, to keep this initial implementation manageable and focused on basic instruction profiling, I’d prefer to implement context tracking as a future extension.