Evaluating coverage tools for integration-level evidence: five questions that separate the claims

Five questions for comparing coverage tools when the evidence must come from the optimized integration build - and the artifacts to ask every vendor for.

Published 2026-08-28Last technically reviewed 2026-08-28Evidence basis: Primary specifications, Engineering analysis

In brief

Coverage datasheets tend to read alike. Five questions separate them, and each one is about information a tool can actually have rather than about implementation quality: which metric the claim is about, at which optimization level it was measured, what happens to a condition the compiler turned into a conditional move, how unattributable object code is reported, and what remains in the denominator. They apply to one specific job - evidence at integration and system level, taken from the optimized build you ship; unit-level instrumentation is a different case and is not the subject here. A claim that survives all five can be shown rather than stated - there are three artifacts worth requesting from any vendor, ourselves included.

First, the scope - because it decides whether this page applies to you. Instrumented coverage at unit level is the established state of the art, and nothing here argues against it: at unit level a test-specific executable and instrumentation overhead are often acceptable, and detailed source-level feedback is exactly what you want - though target memory, timing or optimizer effects can still matter and should be checked where the test objective depends on them. The questions below concern a different job: coverage evidence at integration and system level, from the optimized build you ship. There, instrumentation usually - not always - loses its footing, because the added code changes the artifact and the conditions the test is meant to reproduce - test level and execution environment are separate attributes, and the cost of the added code can be measured.

If that is your job, most datasheets will still read alike. The differences show up in five questions - and none of them is unfair, because each one is about the information a tool can actually have, not about implementation quality. We answer all five for our own tooling on the linked pages.

The five questions

1. Which metric, exactly?

“Coverage from trace” can mean statement coverage, and statement coverage is the easy case. Ask which metric the claim is about. Condition and MC/DC evidence make demands that statement and branch coverage do not - the difference is not gradual, it is structural.

2. At which optimization level was that MC/DC number measured?

If the answer is -O0 or “moderate”, the follow-up question is whether that is the build you ship. If it is not, the evidence describes a different program, and the representativeness argument is back on your desk - why optimization decides this.

3. What happens to conditions that became a conditional move?

Under optimization, a source condition can become a conditional instruction with no branch at all. Many common instruction-trace configurations do not report the semantic result of a branchless conditional operation; they may show only that the instruction executed. A tool that reports such a condition as covered on that basis is reporting instruction execution, not a decision - the mechanism, with object code. Ask the vendor to identify the exact protocol element and configuration that supplies each condition outcome.

4. How is unattributable object code reported - and what does the source-level verdict rest on?

Optimization and code generation produce object code that cannot be assigned to a source obligation without ambiguity. The honest options are to flag it or to hand it to a review queue. The convenient option is to drop it - the residue question. And the question behind it applies to every metric, statement and branch included: a source-level figure is only as strong as the object-to-source mapping it rests on, and debug information under optimization is a weaker foundation than most reports assume. Ask which verdicts rest on debug information alone, and how that is disclosed. The gap is measured, not theoretical - our published real-scale comparison reports the attribution share alongside the coverage result. A reviewer will expect every obligation attributed or explicitly dispositioned.

5. What is in the denominator?

A headline percentage is only as honest as its denominator. Ask whether uncovered, unresolved, deactivated and excluded obligations are reported as distinct categories with counts - or whether some of them silently leave the denominator and improve the number - our denominator policy.

Where our own answers live

Question 2 and 3 are the reason CEDARtools.Anchor exists - how it closes the gap. Question 4 and 5 are reporting policy, described on the coverage technology page. Behind every source-level claim stands a verdict chain - from debug information or symbol table to the reported obligation - and that chain can be reviewed link by link.

Ask for artifacts, not adjectives

A claim that survives these five questions can be shown, not just stated. Three artifacts are reasonable to request from any vendor, including us:

a measurement record for a named target and toolchain configuration - ours are here;
the supported-target status for your exact configuration, not the core family name - documented per configuration;
a statement of the qualification and assurance boundary: which tool produces which verification result, and how an upstream error would be detected - ours is here.

Sources

The evaluation questions summarize the observability and mapping analysis on the linked knowledge pages, which cite the primary protocol and format documents. DO-178C / ED-12C is a licensed publication and is cited by section.

Apply this to your target

Send the exact processor, board, trace path, compiler and metric you need. We will review feasibility at configuration level rather than answering from the core-family name alone.

Start a target feasibility review