test_benches_can_fail

Prove the benches can fail.

A bench that always passes and a bench that checks nothing are indistinguishable from a green suite, and this project has now shipped several of the second kind: a scan bench that raised before comparing and so never compared, a verilator install that could not build any timing bench, and a device check whose field was read from a snapshot that never populated it. Every one of those was found by hand or by an adversarial reviewer. None was found by the suite, because a suite only reports what a test asserts, and none of them asserted that the check ran.

The check here is mutation. Take a tile whose bench passes, change its SystemVerilog so it computes the wrong answer, and require the bench to FAIL. Together with the existing pass-on-clean-source tests that is a two-sided proof: the bench distinguishes a correct implementation from an incorrect one, which is the only property that makes a green bench mean anything.

Mutations are chosen to be semantic and small — an off-by-one in a count, an inverted comparison — so a bench that survives one is not merely insensitive to a wholesale rewrite. They are applied to a COPY; the registry’s source is never touched.

Attributes

_COCOTB_BENCH_PREFIX
_MUTATIONS

Classes

_Mutation

One tile, and a textual change to its HDL that makes it wrong.

Functions

_mutated_sources(mutation, tmp_path) → tuple[pathlib.Path, ...]	Every source the tile needs, with the named one mutated in a copy.
test_a_bench_fails_when_its_tile_is_made_wrong(...) → None	The bench must reject an implementation that computes the wrong

Module Contents

test_benches_can_fail._COCOTB_BENCH_PREFIX = 'dau_core.tests.cocotb_benches'

class test_benches_can_fail._Mutation

One tile, and a textual change to its HDL that makes it wrong.

core_name: str

find: str

replace: str

what_it_breaks: str

parameters: dict | None = None

test_benches_can_fail._MUTATIONS

test_benches_can_fail._mutated_sources(mutation: _Mutation, tmp_path: pathlib.Path) → tuple[pathlib.Path, ...]

Every source the tile needs, with the named one mutated in a copy.

test_benches_can_fail.test_a_bench_fails_when_its_tile_is_made_wrong(mutation: _Mutation, tmp_path: pathlib.Path) → None

The bench must reject an implementation that computes the wrong answer. If it passes here it is not comparing anything, and its green result elsewhere means nothing.