Menu

Searches titles, sections and text. ↑ ↓ to move, Enter to open, Esc to close. Opens with / or Ctrl+K.

    Contact

    Have you ever stepped through -O3 code?

    A jumping cursor and a variable that is optimized out are not bugs. They are what a source-level coverage verdict rests on. Where is the evidence it is right?


    Set a breakpoint in an optimized build. Step.

    The cursor jumps backwards. It stops twice on the same line. It skips three lines that definitely execute. A variable you are watching reads <optimized out>. You step into a function and land in a different one that was inlined into it, then step again and land in the caller.

    Every experienced embedded developer has had this session. The usual reaction is to rebuild at -O0, or to give up and read the assembly.

    Now hold that experience next to the following claim: this build has 98 % MC/DC coverage at source level.

    The number was produced by mapping machine addresses back to source lines - through the same debug information that just told you the program was executing line 41 while it was demonstrably somewhere else.

    The mapping is not one-to-one, and the compiler says so

    At -O0 there is a clean correspondence between a source statement and a contiguous run of instructions. Optimization removes it, deliberately:

    • Inlining copies a function body into several callers. One source line now has instructions at many addresses, in different functions.
    • Code motion hoists a computation out of a loop. The instruction now runs at a point in the program where its source line is not being executed.
    • Merging and folding replace several source constructs with one instruction sequence.
    • Duplication - loop unrolling, tail duplication, jump threading - produces several copies of one source line, each with its own history.
    • Elimination deletes code the compiler proved unnecessary. Its source line has no instructions at all.

    These are not defects. They are why the optimized build is fast, and every one of them breaks the assumption that an address maps to the source line.

    What the compiler does when it cannot decide

    This is the part worth sitting with, and it is documented in the compiler’s own source. When LLVM merges two instructions that carry different source locations, it has to produce a location for the result. Its documentation states the rule verbatim:

    When two instructions are combined into a single instruction we also need to combine the original locations into a single location. When the locations are the same we can use either location. When they differ, we need a third location which is distinct from either. If they share a common scope, use this scope and compare the line/column pair of the locations with the common scope: if both match, keep the line and column; if only the line number matches, keep the line and set the column as 0; otherwise set line and column as 0. If they do not share a common scope the location is ambiguous and can’t be represented in a line entry. In this case, set line and column as 0 and use the scope of any location.

    • LLVM, include/llvm/IR/DebugInfoMetadata.h, documentation of DILocation::getMergedLocation

    Read the emphasized sentence again. The compiler is not making a best guess and recording it. It is stating that the question has no answer, and writing down a designated value that means no source line.

    That is the honest behavior, and it is the right engineering decision. It is also fatal to a claim built on top of it: the entry exists, it is well-formed, it parses - and it asserts nothing.

    The debug format provides that escape value on purpose. In the line number table, the line register is defined like this:

    An unsigned integer indicating a source line number. Lines are numbered beginning at 1. The compiler may emit the value 0 in cases where an instruction cannot be attributed to any source line.

    • DWARF Debugging Information Format Version 5, section 6.2.2, table 6.3

    The neighboring register is worth reading in the same breath. is_stmt is “a boolean indicating that the current instruction is a recommended breakpoint location” - one that is “intended to represent a line, a statement and/or a semantically distinct subpart of a statement”. Recommended. Intended to represent. The quotation marks around represent are the standard’s own.

    This is not sloppy drafting. It is a format being honest about the fact that after optimization there is often no such thing as the line for an address.

    A reproducible instance

    None of this requires a large program. Take a macro that expands into two nested if statements:

    void act(int);
    
    #define CHECK(x, y)      \
        if ((x) > 0) {       \
            if ((y) > 0) {   \
                act(1);      \
            } else {         \
                act(2);      \
            }                \
        }
    
    void h(int a, int b)
    {
        CHECK(a, b);
    }

    Compiled with clang 22.1.4 for AArch64 at -O2 -g, the emitted assembly carries its line table as .loc directives. The first comparison is attributed to line 14, column 5 - the macro invocation, not the macro body. The second comparison, and the conditional increment the compiler made out of the inner if/else, carry line 0 (excerpt):

        .loc    1 14 5 prologue_end     // ex3-macro-provenance.c:14:5
        cmp     w0, #1
        ...
        .loc    1 0 5 is_stmt 0         // ex3-macro-provenance.c:0:5
        cmp     w1, #1
        cinc    w0, w8, lt
        ...
        .loc    1 14 5 is_stmt 1        // ex3-macro-provenance.c:14:5

    Line 0 is the value the standard reserves for an instruction that “cannot be attributed to any source line” - the escape hatch quoted above, in use, in a fourteen-line program. Read the object file back with a disassembler and the whole block is annotated with line 14:

    0000000000000000 <h>:
    ; ex3-macro-provenance.c:14
           0:  cmp   w0, #0x1
           4:  b.lt  0x18 <h+0x18>
           8:  cmp   w1, #0x1
           c:  mov   w8, #0x1
          10:  cinc  w0, w8, lt
    ; ex3-macro-provenance.c:14
          14:  b     0x14 <h+0x14>        (relocation: act)
    ; ex3-macro-provenance.c:15
          18:  ret

    Two things happened here at once. The inner decision has no line of its own; and it has no branch of its own either - the compiler turned if (y > 0) act(1) else act(2) into a conditional increment that selects the argument. A tool that credits line 14 when the first comparison executes has said nothing false. It has also said nothing about the inner decision, which is the one a reviewer would ask about.

    Three consequences that show up in a report

    1. Attribution to the wrong line, silently

    A hoisted instruction carries the line it came from, which is not the line whose execution it now indicates. A coverage tool counting executions at that address credits a line that did not run in the sense the report implies.

    Nothing about this is detectable from the data. The entry is present and plausible.

    2. Coverage for a line that no longer exists

    If a decision was eliminated because the compiler proved it constant, the source line remains in your listing and has no instructions behind it. Depending on the tool, it is reported as uncovered - correct but misleading, since no test could ever cover it - or quietly dropped from the denominator, which is worse.

    Telling those apart is exactly the residue question that DO-178C section 6.4.4.3 asks about.

    3. Column information that is not always there

    The usual answer to multiple expressions on one line is DWARF column information. It helps - and it is not universal. In our own compatibility notes we recorded revisions of the same compiler family where column information is emitted for one target and not for another. A tool that relies on it inherits that variability without being able to detect it.

    So where is the evidence that DWARF is correct?

    This is the question the whole subject turns on, and it is almost never asked. A source-level coverage verdict on an optimized build rests on the debug information being right. Every argument for the verdict therefore contains, as an unstated premise, the claim the debug information is correct.

    Ask for the evidence behind that premise. There are two places it could come from, and neither one supplies it.

    The standard does not promise it

    You might expect the format specification to define what a compiler must emit, so that “correct” has a meaning and a producer could be tested against it. It explicitly declines to. The section is called “Permissive Rather Than Prescriptive”:

    The DWARF Standard specifies the meaning of DWARF descriptions. It does not specify in detail what a particular producer must generate for any source to object conversion. […] DWARF is permissive, allowing different producers to generate different descriptions for the same source to object conversion, and permitting different consumers to provide more or less functionality or information to the user. This may result in debugging information being larger or smaller, compilers or debuggers which are faster or slower, and more or less functional. These are described as differences in quality-of-implementation.

    • DWARF Debugging Information Format Version 5, section 1.3.12

    And, two paragraphs further, on the force of its own wording:

    Where the text says “should”, this is advice about best practice, but is not a requirement.

    Read that as a certification engineer rather than as a compiler engineer. Two compilers may describe the same translation differently, and both are conforming. Richer or poorer descriptions are not right or wrong, they are quality-of-implementation. There is therefore no conformance criterion against which a producer’s debug information could be qualified - not because nobody has written the test suite, but because the specification deliberately does not define the thing the test suite would check.

    For the compiler’s code generation the situation is the opposite: the semantics of the source language define what the output must do, which is why a code generator can be qualified at all.

    The one systematic study says the opposite of reliable

    If the specification does not settle it, measurement might. In 2020 a team from Georgia Tech and Apple built the first framework for systematically testing compiler-generated debug information on optimized code, and published it at PLDI. Their own framing of the state of the field:

    Almost all modern production software is compiled with optimization. […] However, little is known about the correctness of the debug information generated by optimizing compilers when debugging optimized code.

    The difficulty is that there is no oracle: on an optimized build there is nothing to compare the answer against. They constructed one - compile the same program unoptimized, take the value observed there as ground truth, and force the two builds to be comparable by inserting an unoptimizable barrier at the location under test.

    What came out of it:

    Confirmed bug reports against GCC and LLVM47 (21 GCC, 26 LLVM)
    Fixed at the time of writing11, about 23 %
    Affected the then-current development versionall of them
    Oldest still-live findingreached back to GCC 4.4, released about nine years earlier
    Optimization levels affectedGCC across -O1 to -O3, and -Og - the flag whose purpose is a better debugging experience - was not exempt; LLVM across -O1 to -O3
    Second compiler, three days of testing2 further confirmed bugs, in the Rust compiler

    The LLVM component with the most findings was the DWARF parser itself, and one report went to the DWARF standard committee as a request to clarify wording that two implementers had read differently.

    And then the sentence that matters most for anything built on top of this:

    It is interesting to note that the wrong variable information in both examples is only observable in a debugger. GCC and LLVM generate the correct optimized machine executables […] It demonstrates that the state-of-the-art optimizing compilers may generate wrong debug information for optimized code.

    • Li, Ding, Zhang, Italiano: Debug Information Validation for Optimized Code, PLDI 2020

    The binaries were correct. The programs did the right thing. Only the description of what they were doing was wrong - and nothing except a debugger, or a purpose-built oracle, would ever notice.

    What this does and does not show

    Being precise about the scope, because the argument is stronger without stretching it:

    • The study tests breakpoint locations and printed variable values, not coverage verdicts. It does not report a number for wrong line attribution in a coverage tool, and neither do we.
    • It tests two compilers plus one, not every toolchain, and certainly not whichever one you use.
    • Eleven fixes and an active response from maintainers say this is a healthy project, not a scandal. LLVM even has a dedicated routine that tries to reconstruct debug information when an instruction is deleted; the paper notes it is explicitly a best-effort function that cannot always succeed.

    What it does establish is the thing the premise needed and does not have: when someone finally built an oracle and looked, findings appeared immediately, in the current versions of both major compilers, at every optimization level - in an area where a defect changes program behavior by nothing at all.

    So when a tool description says the source-level mapping is reliable, the question is simply: on what evidence? The format does not define correctness, the compilers are not qualified for it, the tests do not exercise it, and the one group that measured it found bugs everywhere it looked.

    Why an assessor does not accept this

    Put yourself on the other side of the table. The question is not whether the tool is good. It is:

    How do you know that the source-level verdict is correct?

    For the compiler’s code generation, there is a clean answer: you verify the product. If code generation were wrong, the program would misbehave, and the test campaign would find it.

    For the compiler’s debug information, that answer does not exist. Wrong line tables change program behavior by exactly nothing. The binary passes every test with a thoroughly broken .debug_line section - including the very tests that produced the coverage report.

    So the reviewer is left with three questions and no comfortable answers:

    QuestionAnswer for code generationAnswer for debug information
    Is it verified by testing the product?yes, that is the standard argumentno - it is functionally silent
    Is there a defined correctness criterion?yes: the program’s semanticsno: the format is permissive by design (section 1.3.12), and has a value meaning unknown
    Does a failure announce itself?yes: wrong behaviorno: a wrong line entry looks like a right one

    This is not an argument that debug information is useless. It is excellent for what it was built for - a debugger, where a human is in the loop and notices the jumping cursor. It is an argument about what it can carry as evidence, and the answer is: less than a certification report assumes.

    What we concluded from it

    Our own patent application states the problem in its background section, and it is worth quoting because it is where the design came from:

    The control-flow reconstructed from the trace is at the machine level. For tasks that require a transformation to the source code level, this can be done using DWARF debug information. However, especially when optimizing compilations, it is usually not possible to make a precise assignment between the machine level and the source code level, because the compiler may have added additional code, removed code, transformed code (e.g., converted a control-flow into a data flow), or moved code.

    • the background section of our published international patent application; publication details are on the patents page

    The design decision that follows is a single sentence: the symbol table is authoritative, DWARF is advisory.

    The symbol table is different in kind. The linker consumes it to build the binary, so it is exercised by the same process that produces the artifact under test. A symbol resolves to an address or it does not - that criterion is defined and mechanically checkable. And when an anchor symbol is missing or has collided, that is visible in the table, which makes the failure mode detected rather than silent.

    Validated trace
    observation interval,
    validity checks passed
    Validated trace...
    Object-level coverage
    established on machine
    addresses
    Object-level coverage...
    &
    &
    &
    &
    Source attribution
    object → source mapping,
    rests on symbol table
    Source attribution...
    =
    =
    Symbol table
    (advisory)
    Symbol table...
    Source-level claim
    only as strong as the
    weakest link behind it
    Source-level claim...
    A serial AND chain: every link must hold - there is no averaging across links.
    Validated trace
    observation interval,
    validity checks passed
    Validated trace...
    Object-level coverage
    established on machine
    addresses
    Object-level coverage...
    Source attribution
    object → source mapping,
    rests on debug information
    ⚠ spec allows approximation
    under optimization
    Source attribution...
    &
    &
    =
    =
    &
    &
    DWARF
    (authoritative)
    DWARF...
    Source-level claim
    only as strong as the
    weakest link behind it
    Source-level claim...
    A link whose own specification declares it approximate caps the strength of the whole chain.

    Debug information is still used - to enrich source-level reporting, because it is genuinely good at that. It is simply not the thing the verdict rests on, and where the two disagree, the link-verified mapping wins.

    The mechanism is described in why the mapping breaks under optimization and on the CEDARtools.Anchor page.

    requirement R1 · function f()
    requirement R1 · function f()
    block 0x100..0x13F
    block 0x100..0x13F
    executed
    executed
    inlined helper g()
    inlined helper g()
    block 0x140..0x16F
    block 0x140..0x16F
    block 0x170..0x19F
    block 0x170..0x19F
    executed
    executed
    not executed in this run
    not executed in this run
    macro M in two functions
    macro M in two functions
    template instance h<T>()
    template instance h<T>()
    block 0x1A0..0x1CF
    block 0x1A0..0x1CF
    executed
    executed
    block 0x1D0..0x1EF
    compiler-generated, no source line
    block 0x1D0..0x1EF...
    Review queue
    unresolved mappings are listed, not dropped
    Review queue...
    Source constructs
    Source constructs
    Object-code blocks
    Object-code blocks
    Trace-observed execution
    Trace-observed execution
    1:1
    1:1
    Address ranges are schematic; the mapping classes are the ones described in the article.
    Address ranges are schematic; the mapping classes are...
    1:many
    1:many
    many:1
    many:1
    Every object-code block stays accounted for: mapped one-to-one, split, merged, or handed to the review queue - never silently dropped.

    What to ask, concretely

    If you take one question from this article into a tool evaluation, take the first one. It is the one that has no comfortable answer.

    1. Where is the evidence that the debug information is correct? Not that the tool reads it correctly - that the compiler wrote it correctly. Ask for the specification clause it conforms to, the qualification argument, or the measurement. All three are worth asking for precisely because none of them exists in the general case.
    2. At which optimization level was the source-level mapping validated?
    3. What does the tool do with an instruction whose debug location is line 0? Count it, drop it, or flag it - all three are defensible, and only one of them is what you want in an evidence report.
    4. What happens when debug information and symbol table disagree?
    5. How would a wrong attribution be detected? If the honest answer is “it would not be”, that is worth knowing before the review rather than during it.

    Related: DWARF as a chain of trust - the qualification argument in short form · what an assessor actually asks

    Sources, listed for reference and verification: the compiler passage is quoted from LLVM’s own header include/llvm/IR/DebugInfoMetadata.h. The specification passages are from the DWARF Debugging Information Format Version 5, sections 1.3.12 and 6.2.2. The study is Yuanbo Li, Shuo Ding, Qirun Zhang and Davide Italiano, Debug Information Validation for Optimized Code, PLDI 2020. A wrong citation in an article about wrong information would be a poor joke, so every quotation above is traceable to the primary document.