What an assessor may ask
Behind every checklist item there is a question about whether you know what your software does. Five of them, and what a good answer looks like.
Certification reviews have a reputation for being about paperwork. The paperwork is real, and it is a proxy: what is being established is whether the team knows what its software does and can show it.
Assessors and authorities may ask different questions depending on the standard, project and tool use. The questions below are a practical review checklist derived from common assurance concerns, not a statement of mandatory wording or a substitute for the applicable objectives.
Five questions recur. None of them is answered by a percentage.
1. “Where does this number come from?”
Not skepticism - process. A coverage figure is an output of a tool, applied to a build, driven by a test suite, in a configuration. All four are part of the answer.
A good answer names them: which build (compiler, flags, version), which test suite, which tool at which version, and which metric with which denominator. If any of those changed during the campaign, when and why.
A weak answer is a report with a percentage and a date.
2. “Is that the software you are delivering?”
The representativeness question, and the one that costs the most time when the answer is complicated.
Measuring the delivered binary removes one major representativeness question, but the answer still identifies the loaded artifact, target/configuration, test campaign, observation validity, mapping basis and tool-assurance boundary. If coverage was measured on an instrumented build, a host build, or with reduced optimization, then the answer is additionally an argument - and the assessor will want to know what could differ and why it does not matter here.
That argument is constructible. It is also a common and often expensive source of review effort in a structural coverage campaign, and the reason measuring on the shipped artifact is worth engineering effort.
3. “What did you do about the code that was not covered?”
The residue question, and the one that most distinguishes a mature process from a report generator.
A good answer first separates evidence problems - trace loss, excluded scope, unresolved mapping: unknown, not uncovered - from valid gaps, and then classifies every valid gap: shortcomings in the tests or procedures (and here are the new ones), inadequacies in the requirements (and here is the change), dead code (and here is the analysis), deactivated code (and here is why its activation is controlled). See dead and deactivated code.
A bad answer is a report with no gaps and nothing that makes the result inspectable. A no-gap report can be valid - but only the denominator, the exclusions, the unresolved mappings and the trace-validity record let a reviewer tell an unusually complete campaign from a tool that dropped what it could not classify.
4. “How do you know the tool is right?”
Tool qualification, and the answer depends on what the tool’s output is used for.
If the tool’s output can cause an error to go undetected - and a coverage tool’s can, by reporting something as covered that was not - then confidence in it has to be established. Either through qualification, or by an argument that the error would be caught elsewhere.
The specific version of this question that is worth preparing for: what does your tool depend on that you did not verify? For source-level verdicts derived from object-level measurement, the answer includes the compiler’s debug information, which is a chain worth thinking about - and, in daily work, less definite than it looks.
5. “Show me a case where the tests were wrong”
Not always asked in these words. What is being probed is whether the coverage activity ever changed anything.
A campaign that produced 100 % on the first run and no findings invites the question of whether it was measuring what it claimed. A campaign that found gaps, resolved them, found a dead branch that revealed a stale requirement, and documented all of it is telling a story that is easy to believe.
The pattern
All five questions have the same shape: can you connect the number to reality, and did you look at what it told you?
That is worth knowing when choosing tools and designing a process, because it reprioritizes things. A tool that produces a slightly lower percentage while making the residue visible and the method traceable is worth more in a review than one that produces a flattering number by construction.
We would rather build the first kind, which is why our reports name what they could not attribute instead of dropping it, and why the aggregation semantics are written down rather than left to the implementation.
Seven further questions worth preparing for, in the same spirit:
- How is the measured executable identified and related to the delivered artifact?
- How are trace loss, gaps and resynchronization reflected in the reported scope?
- Which obligations are observed, mapped, supplemented, unresolved or excluded?
- What evidence supports the compiler and debug-information mapping?
- Where is the tool-assurance boundary, including import interfaces?
- Which product, target and configuration versions were verified?
- What independent check can detect a wrong result from the measurement tool?
Sources
The objectives behind these questions are stated in:
- RTCA DO-178C / EUROCAE ED-12C and DO-330 / ED-215, cited by section (licensed publications).
The checklist above is reusable as it stands: walk your most recent coverage report through the five questions and the seven follow-ups before an assessor does, and note where a written answer is still missing.
Related: what the standards require · dead code and deactivated code