Embedded Trace is an integral part of nearly all modern processors. This whitepaper summarizes the essential facts about this powerful but still far too seldom used functional unit that application engineers, test engineers and project managers should know in order to test, optimize and debug a system efficiently.

This paper briefly discusses the problems associated with software-based instrumentation and how non-intrusive electronic probing was defeated by advancing computer architecture and system integration. We then identify embedded trace as the solution to this observability conundrum laying out the techniques that enable efficient and economically reasonable implementations for this innovative technology. We describe the challenges in bandwidth and volume that are faced by hopeful observers and backend applications highlighting the benefits of modern innovative online analysis capabilities. Finally, we provide a short overview over common physical trace interfaces and simple guidelines that ensure that your next system design is capable of leveraging this cutting-edge technology.

This whitepaper briefly examines common types and the nature of software anomalies. It explains how mistakes lead to observable anomalies and how these are differentiated into Bohrbugs and Mandelbugs according to their reproducibility. The principle of “scientific debugging” is explained. It is shown that the comprehensive observability of a system is a key capability for efficient debugging. Subsequently, the advantages and limitations of various existing and novel monitoring solutions such as printf()-debugging, start/stop-debugging, omniscient debugging, runtime verification and the novel CEDARtools^® approach are presented and discussed.

While multi-core processors offer more processing power than single-core architectures, they are more likely to produce hard-to-detect and concurrency-related
errors. This article presents a new technology that enables the measurement of the timing behavior of programs on multi-core architectures, the measurement of
the coverage achieved by test running on the target system, and the analysis of complex errors.

Original German version of this article @ Hanser Automotive 03/2020

This paper briefly examines common types and the nature of failures in complex, embedded multi-core environments. A novel tool CEDARtools ® is presented, overcoming these issues by processing hardware-generated trace in real-time, providing complex triggers and variable monitoring scopes, facilitating a post-mortem analysis. The approach permits preventive monitoring even before the program fails and allows mastering the evolving complexity in embedded development.

Proof of functional safety requires the collection of structural coverage information to confirm that the structural coverage is appropriate for the required safety level.
A new observation methodology based on processor traces provides the key means to gain non-intrusive insight into the execution of production code for multi-core SoCs. Trace data analysis must be automated to cope with the enormous bandwidth of trace data streams.
This new technique supports certification of test coverage and enables automated detection of emerging timing constraints. Most importantly, it allows structural tests to be executed on the basis of the production code without the need for software instrumentation.
The paper provides an overview of the applicable functional safety standards, explains the advantages of executing structural tests even at higher test levels, and gives practical hints for hardware and software architecture considerations for providing best observability for executing structural tests.

This paper presents an approach for rapidly adjustable embedded trace online monitoring of multi-core systems, called RETOM. Today, most commercial multi-core SoCs provide accurate runtime information through an embedded trace unit without affecting program execution. Available debugging solutions can use it to reconstruct the run offline, but usually for up to a few seconds only. RETOM employs a novel online reconstruction technique that makes the program run available outside the SoC and allows for evaluating a specification formulated in the stream-based specification language TeSSLa in real time. The necessary computing performance is provided by an FPGA-based event processing system. In contrast to other hardware-based runtime verification techniques, changing the specification requires no circuit synthesis and thus seconds rather than minutes or hours. Therefore, iterated testing and property adjustment during development and debugging becomes feasible while preserving the option of arbitrarily extending observation time, which may be necessary to detect rarely occurring errors. Experiments show the feasibility of the approach.