What embedded trace is, and what it is not
The vocabulary, the difference from debugging and profiling, and the one property that makes trace worth the trouble.
Trace is the least understood of the observation techniques, mostly because it is usually explained through a product.
In brief
Embedded instruction trace is a hardware facility that reports the processor’s execution - primarily its control flow - through a dedicated on-chip infrastructure, without adding instructions to the program under observation. The compressed stream leaves the chip through a trace port or an on-chip sink and is reconstructed against the program image. Its costs live in pins, bandwidth and infrastructure, not in the application code.
One sentence
A trace unit reports what the processor did, while it does it, without asking the processor to help.
Everything interesting follows from the second half of that sentence. The core does not execute extra instructions and the program does not know it is being watched; the physical cost lives in the trace infrastructure - pins, bandwidth and power - not in the application.
What it is not
Not a debugger. A debugger stops the target and reads its state. That is enormously useful and completely unsuitable for anything that must keep running
- an engine controller,
- a flight control loop,
- a motor drive.
Trace does not stop anything.
Not a profiler in the sampling sense. A sampling profiler interrupts periodically and asks where the program counter is. It builds a statistical picture. Trace is not statistical: it reports the actual sequence of executed instructions, in order.
Not logging. printf changes the program. It needs a working console, it
costs runtime, and it only tells you what someone thought to log in advance.
Trace needs no cooperation from the software, and it does not require you to
have guessed the question beforehand.
Not zero cost. The trace unit is on the chip whether you use it or not, but getting the data off the chip costs pins, board area and design attention. That is the real price, and it is paid at PCB design time.
The vocabulary
| Term | What it means |
|---|---|
| Trace source | The unit inside the chip that observes a core and emits trace. Arm® calls it ETM or ETE; Infineon® calls the surrounding solution MCDS; RISC-V® has its own encoders. |
| Instruction trace | The record of what was executed. This is the one that carries coverage and control flow. |
| Data trace | The record of memory accesses, or of selected variables. Far higher volume, usually filtered. |
| Funnel | The block that arbitrates several trace streams into one, preserving the source ID each stream already carries from its trace source. |
| Sink | Where the stream goes: an on-chip buffer, system memory, or straight out of the chip. |
| Trace port | The physical way out. Parallel pins, a high-speed serial link, or a system interface such as PCIe®. |
| Packet / message | The unit of encoding. Protocols differ in what they call these and what they put in them. |
| Sync point | A periodic full-address anchor. Without it, a decoder that joins mid-stream cannot know where it is. |
Instruction trace is compressed by construction
This surprises people the first time. The trace unit does not report every instruction. It reports the discontinuities - the points where the next instruction is not simply the following one.
Everything between two discontinuities is implied: the decoder holds the program image, so it can reconstruct the straight-line stretches itself. What has to be transmitted is which way each conditional branch went, and where each indirect branch landed.
That is why trace bandwidth does not scale with instruction rate but with branch rate, and why the interesting engineering question is always the same one: does the branch rate of your workload fit through your trace port?
Instruction-trace protocols encode enough information to reconstruct control flow according to their architecture and configuration. Direct branches and calls may be inferred from the program image rather than emitted as one packet per event. Exceptions, indirect transfers, filtering, synchronization and timing information are protocol-specific.
The trace stream is not a literal log of every executed instruction. It is a compressed observation that a decoder interprets together with the program image. The evidence claim therefore depends on protocol conformance, decoder correctness, image identity and trace continuity.
Where the bottleneck sits
The trace source observes execution inside the processor, but lossless delivery depends on the implemented source, buffering, configuration and downstream bandwidth. A system may report overflow, discard trace or apply backpressure according to its architecture. The validity record must therefore cover the complete path, not only the decoder.
The most common bottleneck is at the exit: the number of pins the board exposes, the clock they can be driven at, and the signal integrity of the routing. A part may support 16-bit parallel trace and your board may have room for four pins.
When more data is generated than fits through the port, what happens next depends on the configured policy. In a drop-on-overflow configuration, the trace unit signals an overflow and data is lost; other implementations may stall, backpressure, stop or suppress trace according to their architecture and configuration. Where data was dropped, a decoder can usually resynchronize afterwards, but the lost stretch is gone, and any analysis over that window is incomplete. In that case an overflow is not a warning to be tuned away later; it is a hole in your evidence.
Where the data can go
There is more than one architecture here, and the choice has consequences:
- On-chip buffer. No trace port and no application-code instrumentation, but the buffer is small and expensive in silicon - and trace generation, buffering and readout keep platform-specific residual effects.
- System memory as buffer. Plenty of capacity, no dedicated port - at the cost of bus and memory bandwidth taken from the application. That is intrusion, in the honest sense of the word.
- External buffer, captured through a dedicated port. No application-code instrumentation and no shared application-memory traffic, but the recording length is bounded by the buffer.
- Live processing. The stream is decoded and analyzed as it arrives, so storage stays bounded for the analyses that were defined up front. Observation is then not bounded by buffer size - as long as processing keeps up, and for the questions the pipeline was configured to answer.
The last two are the interesting pair, and the difference between them decides what kind of question you can ask. A fault that appears after six hours is not reachable with a buffer that holds milliseconds.
What passive trace does and does not change
| Property | Passive instruction trace |
|---|---|
| Application counters or logging calls | not required |
| Trace infrastructure configuration | required |
| Trace pins/link/buffer resources | configuration-dependent |
| Application memory writes for coverage state | not required |
| Target power and I/O effect | not assumed zero |
| Complete source-condition observability | not guaranteed |
| Lossless observation | must be verified |
Why bother
Because it reports on the executable that actually ran, without adding application-side code. Alternative methods change the executable, change the timing or answer statistically - each of which can be acceptable, and each of which needs its own argument.
For most software, that difference does not justify the effort. For software where a field failure is expensive - and that is a much larger category than “certified” - it does.
Background: why trace exists at all
Observation used to be free. When processors ran at tens of megahertz with external program memory, eavesdropping on the memory bus showed every fetched instruction address - following the program flow was trivial. Three architecture shifts closed that window:
- Caches moved instruction and data accesses next to the core, off the external bus.
- Internal memory - embedded devices integrate flash and RAM on-die for speed and cost, and accesses to it never appear outside the chip.
- Multicore - what little remains externally visible must also be attributed to the right control flow among truly parallel cores.
Trace units are the answer: dedicated on-chip circuits that make execution observable again. The reason they compress so aggressively is arithmetic, not taste. As a back-of-the-envelope estimate - an illustration, not a measurement: reporting a 32-bit instruction address plus the changing registers and flags naively costs on the order of 80 bits per clock cycle, which at 1 GHz would be roughly 80 Gbit/s of trace data per core. No economical interface carries that, so the protocols transmit only the discontinuities - which is the compression-by-construction described above.
Try it on silicon you already own
Trace support is not exotic. Intel® Processor Trace ships in modern Core and
Atom parts - the laptop on your desk very likely has it, and on Linux®
perf record -e intel_pt//u followed by perf script gives a first
instruction-accurate call trace with no extra hardware at all. The Raspberry
Pi 4’s BCM2711 exposes CoreSight™ trace, and most Cortex®-M/R/A, TriCore™
and PowerPC® boards used in industry bring a trace unit too. What varies -
and what decides whether the capability is usable for evidence rather than
curiosity - is whether the trace has a path off the chip:
the trace port is a board-design question, and
on-chip buffers limit the observation window to moments. That difference,
not the trace unit itself, is where the engineering happens.
Next: the trace protocols compared · why trace data is compressed
What this page establishes
- instruction trace is a hardware observation channel, separate from the program under observation;
- the on-chip chain - source, funnel, sink, port - and where the bandwidth limit usually sits;
- why the stream is compressed by construction and needs the program image for reconstruction;
- what passive observation changes on the target, and what it does not.
Limitations
Which trace facilities exist, and at what fidelity, is a property of the core and the SoC implementation - not of any tool. Whether a usable path off the chip exists is a board-design question: see the trace port and target trace feasibility.
Sources
The infrastructure described above is specified in:
- CoreSight™ Architecture Specification (Arm IHI 0029);
- Embedded Trace Macrocell Architecture Specification, ETMv4.0 to ETMv4.6 (Arm IHI 0064);
- IEEE-ISTO 5001-2012, the Nexus 5001 Forum standard;
- MIPI® System Trace Protocol;
- MIPI Parallel Trace Interface.
Apply this to your target
Send the exact processor, board, trace path, compiler and metric you need. We will review feasibility at configuration level rather than answering from the core-family name alone.
Which product carries this in practice - coverage, timing or control flow, and in which maturity - is on products and engagement paths.
Start a target feasibility review
Further reading: Hunting Heisenbugs while you sleep - agentic debugging with hardware trace