Generation is not verification
If the same actor writes the code and the tests that judge it, a green result means less than it looks. What has to sit outside the loop, and why.
This page presents an engineering argument for separating generation and acceptance; it does not claim that this architecture is an established industry standard.
Software generation by language models became good quickly. The part that did not keep pace is the ability to believe the result.
The self-grading problem
An AI coding assistant that writes both the implementation and the tests is being asked to mark its own exam. If it misunderstood the requirement, it misunderstands it consistently: the code implements the wrong thing, the test asserts the wrong thing, and the suite is green.
This is not a defect of any particular model. It is a property of the arrangement. Any actor that produces an artifact and also judges it has a degree of freedom that an independent judge does not.
Running on hardware does not remove the self-grading problem. A generator can compile, flash, exercise a real board and repair its own output. That is a valuable functional loop. It still does not establish that the declared acceptance criteria were satisfied independently.
The distinction is control of the verdict. The generator may propose implementation and test artifacts. The acceptance criteria, test oracles, measurement path and final pass, fail or unresolved decision must remain controlled outside the generator.
The same principle is why safety-oriented development frameworks commonly seek independence between development and verification activities - the required degree depends on the framework, lifecycle process and assurance level. A single assistant that creates both an artifact and the evidence used to accept it can reintroduce exactly the common-mode assumptions that this independence is meant to expose.
Where the consequence lands
For most software, a silent defect is an inconvenience. It gets found, it gets fixed, someone is annoyed.
The class of software this site is about is different: an engine controller, a pump, an infusion device, a rail interlocking. The failure is not annoying, it is expensive - and the fact that a test suite was green does not help afterwards.
That asymmetry, not any general skepticism about AI, is the reason verification deserves a separate mechanism.
What has to be outside
Three properties, and they follow from each other:
The verdict must not rely solely on the generator’s own judgment. Independence may be provided by a separate tool, organization, method or controlled review step, depending on the assurance objective. If the generator can influence the verdict, the verdict is a self-assessment.
The acceptance procedure must be reproducible. The same controlled inputs and configuration invoke the same checks, and measured variation is judged against predefined tolerances and decision rules. A deterministic calculation may repeat bit-for-bit; a physical measurement may not. A model asked “does this satisfy the requirement?” produces a plausible answer, not a reproducible procedure. Deterministic checks make the verdict reviewable - they do not make the requirements or the test oracle correct.
The evidence must be collected at the level relevant to the claim. Coverage on a host build of generated code says something about the host build. Where the delivered target executable is the subject of the claim, host-only evidence needs an additional representativeness argument - which is what trace-based measurement is for.
Put together: the model is treated as an untrusted code source, and the verification layer is separate, deterministic and reviewable. Treating model output as untrusted input to a deterministic verification workflow can reduce reliance on the model’s own judgment. Whether the model or the surrounding tools become relevant to a qualification argument depends on intended use, the credit taken for their output and the effectiveness of downstream error detection.
Choose the failure mode your gate exposes
A closed generate-and-verify loop has more outcomes than pass and fail - a correct, evidenced pass and a false rejection among them. Two undesirable failure modes matter most, and the design question is which one your gate exposes.
Silent false acceptance: the model produced something wrong, the check agreed, and it ships. Nobody learns anything until the field.
Visible non-convergence or false rejection: the loop does not close. Coverage does not reach the threshold, a test does not pass, a requirement has no evidence. Somebody looks at it.
The gate should bias unknown or unmet evidence toward a visible unresolved state while still permitting a correct, evidenced pass. Generation and verification should be separated because a generator should not be the sole judge of its own output. Deterministic external checks can make acceptance repeatable and reviewable, but their assurance is bounded by the requirements, test oracles, models, environments and metrics they encode. Independent review remains necessary for what the gate cannot observe.
What the gate cannot establish
The gate cannot establish the correctness or completeness of unstated requirements, an incorrect test oracle, incomplete acceptance criteria, common assumptions shared by generator and verifier, or system interactions that the verification environment does not represent. These remain controlled engineering and review responsibilities.
The measurement has an information limit
Passive branch trace on an optimized binary cannot recover a source-condition outcome that is absent from both the emitted trace and the recoverable object-level control flow. That is an information limit of that observation path, not a universal limit of every coverage tool. Other methods may add trace features, use verified metadata, instrument or supplement the build, or provide static/formal evidence; each introduces its own assurance and representativeness conditions. The mechanism is described in why the mapping breaks.
And the loop has a clock: it iterates at the speed of the measurement. A coverage answer that takes hours makes every iteration take hours; whether the answer can keep up is decided by how fast the trace can be decoded.
The supplementation approach that answers it is the subject of a published international patent application. The application is in the international phase: examination and grant occur only after entry into the national and regional phases. Publication details and our granted patent families are on the patents page.
What follows is a requirement rather than a slogan: evidence used to accept generated code must either be collected on the delivered target artifact for the claimed property or be accompanied by an explicit, reviewable representativeness or equivalence argument. Optimization-dependent structure and observability must be addressed rather than assumed to transfer.
Honest reporting is part of the mechanism
One more property, easy to overlook and load-bearing: what cannot be measured must be reported as not measured.
A verification layer that reports an unmeasurable item as 0 % is pessimistic and noisy. One that reports it as 100 % is dangerous. One that reports it as “not measured, here is why” preserves the reviewer’s ability to judge - which is the entire purpose of the exercise.
Apply the independence questions above to your own acceptance gate: who controls the criteria, the oracles and the measurement path - and what does the gate report when evidence is missing?
Related: CEDARtools.Forge · what an assessor actually asks