Menu

Searches titles, sections and text. ↑ ↓ to move, Enter to open, Esc to close. Opens with / or Ctrl+K.

    Contact

    The kinds of code coverage, and what each one proves

    Statement, line, branch, condition, MC/DC, function, call, object code. Eight metrics, one ladder, and a sharp difference in what each establishes.


    Coverage percentages get quoted as if they were one number. They are at least eight different numbers, measuring different things, with different failure modes.

    The ladder

    What none of the metrics establish: functional correctness, requirement completeness, correct test oracles, absence of data-only faults, or execution outside the valid observation and mapping scope. That shared boundary is spelled out once here so the entries can stay short.

    Function coverage. Was this function called at all? Coarse, cheap, and genuinely useful as a first pass: an uncalled function in a shipped binary is a question worth asking.

    Function call coverage. Was this call site exercised? A function called from five places, tested through one, has full function coverage and a gap.

    Object-code execution coverage. Records execution against obligations defined in the analyzed machine code. It is closest to the trace reconstruction and does not depend on source mapping for the basic observation. It does not by itself show which source requirement or construct the object code implements.

    Statement coverage. Did this statement execute? The most quoted metric and the weakest. A statement can execute with the decision above it having taken only one of its outcomes.

    Line coverage. Did this source line execute? Not the same as statement coverage: a line can hold several statements, and a statement can span lines. The distinction is usually harmless and occasionally the reason two tools disagree.

    Branch or decision coverage. Did each decision take both outcomes? Now you are testing the control flow rather than just touching it. This is where the metric starts to say something about the tests. Branch outcomes in the generated control flow and source-level decisions are related but not universally identical: optimization, switch lowering, compiler-generated control flow and multiple object branches for one source construct can change the mapping. The report states whether a value is object branch coverage or mapped source decision coverage.

    Condition coverage. Did each individual condition within a decision take both values? For if (a && b), that means both a and b were seen true and false.

    MC/DC. Did each condition independently affect the outcome of the decision? This is the strongest of the common structural criteria and the one required for the highest criticality levels. It is explained separately, because it deserves it.

    MC/DC
    each condition independently affected the outcome
    MC/DC...
    Condition
    each condition took both values
    Condition...
    Branch / decision
    both outcomes of the decision occurred
    Branch / decision...
    Statement / line
    this code ran at least once
    Statement / line...
    Metric and what it proves
    Metric and what it proves
    Depends on object-code structure
    Depends on object-code structure
    high
    high
    medium
    medium
    low
    low
    low
    low
    Each level includes the ones below it.
    Each level includes the ones below it.
    The rightmost column is the one that decides whether a metric survives compilation with optimization.

    What each one cannot tell you

    Every metric has a characteristic blind spot, and knowing them is more useful than knowing the definitions.

    MetricReaches 100 % while…
    Function…entire branches inside the function are untested
    Statement…one outcome of every decision was never taken
    Branch…individual conditions were never exercised independently
    Condition…the decision outcomes were not both produced (yes, really - it is possible)
    MC/DC…the code does the wrong thing correctly, in every tested combination

    That last row is the honest general caveat: structural coverage measures your tests, not your software. It tells you which code your test suite reached, not whether the code is right. A hundred percent of anything is a statement about test completeness with respect to structure, and nothing more.

    This is not a criticism of the metrics. It is the reason the standards ask for structural coverage in addition to requirements-based testing, never instead of it.

    Where the measurement gets hard

    The metrics above are defined on the source code. Coverage is measured on something that runs, which is object code. That gap is bridged either by instrumenting the source before compilation, or by observing the object code and mapping the result back.

    That bridge is built from debug information, and what it can and cannot carry decides how much weight it takes.

    The important thing about that choice is that it is not neutral with respect to the metric:

    • Function and call coverage survive the trip almost unharmed. A call is a call.
    • Statement and line coverage survive well, with one caveat: a statement can be optimized away entirely, and then “not covered” and “not present” have to be distinguished.
    • Branch, condition and MC/DC depend on the decision structure still existing in the object code - and that is exactly what an optimizer is entitled to remove.

    The higher you climb the ladder, the more the measurement depends on structure surviving compilation. This is the single most useful thing to understand about coverage on embedded targets, and it is the subject of its own article.

    Choosing a metric

    In practice the choice is made for you by a standard, or by the cost of failure.

    • If a failure is inconvenient, statement plus branch is a reasonable bar and cheap to reach.
    • If a failure is expensive, MC/DC on the code that ships is the meaningful target, and the difficulties above become your problem rather than a footnote.
    • If a standard applies, it tells you which metric at which criticality level - and, more subtly, what to do about the code your tests did not reach.

    Next: MC/DC, worked through

    Sources

    The external references in this article are: