mirror of
https://github.com/deepseek-ai/deepseek-harness
synced 2026-08-15 21:04:50 +00:00
e6fad266a6
Define the in-file RFC contract in docs/rfc/README.md § The file format: the header block (`# RFC: <title>` plus a dateless Status enum cross-checked against the lifecycle folder), the per-lifecycle body skeleton (a Problem opener everywhere; Proposal/Alternatives considered/ Acceptance criteria/Risks in proposed/; present-tense Decision/ Consequences with proposal-era headings banned in implemented/; the frozen proposal shape in rejected/), and a mandatory Alternatives considered section with a date-fenced grandfather comment for pre-format RFCs whose alternatives are not reconstructible from the record. Enforce it with a new doc-sync gate, scripts/verify-rfc-format.ts, and normalize all 112 RFCs to it: ~15 Status-line spellings collapse to the enum, 29 Context openers become Problem, the 39 legacy-format XXX debt markers are resolved and banned from reappearing, proposal-era sections in implemented RFCs are rewritten to shipped reality (including the web/fs/subagent seam RFCs' migration plans and test checklists, closing the doc-tiers deferred-work item on the web seam), every RFC gains an Alternatives considered section or the grandfather comment, and the bilingual pair is re-mirrored and re-recorded. Move the generated index tables out of README.md into a fully generated docs/rfc/INDEX.md — gen-rfc-index now writes the whole file, and verify-rfc-classification checks its freshness and rejects index-shaped rows in the curated README — which makes room for the format contract to live in the README front door instead of a separate FORMAT.md. The decision record, and the first RFC written in the new format, is docs/rfc/implemented/process/2026-07-05-uniform-rfc-format.md.
35 lines
2.1 KiB
Markdown
35 lines
2.1 KiB
Markdown
# RFC: Mutation testing as the coverage counterweight
|
|
|
|
Status: proposed
|
|
|
|
## Problem
|
|
|
|
The per-file 100% coverage gate ([the quality-gates decision](../../implemented/process/2026-06-11-quality-gates.md)) proves every line *executes* under test — not that any assertion would notice if the line were wrong. Under agent-written tests, coverage pressure can produce execution-without-assertion. Mutation testing measures what coverage cannot: whether the suite *kills* deliberately injected bugs.
|
|
|
|
## Proposal
|
|
|
|
Stryker (`@stryker-mutator/vitest-runner`) over `packages/*/src`:
|
|
|
|
- **PR-scoped incremental runs** (changed files only) as a CI job — fast enough to gate merges once tuned.
|
|
- **Nightly full runs** with a tracked mutation score; start by recording, then set the threshold at the observed baseline and ratchet upward (same policy as coverage: thresholds only ever tighten).
|
|
- Surviving mutants are work items: an agent picks a survivor, writes the killing test, repeats — a well-shaped autonomous loop.
|
|
- Equivalent mutants (provably behavior-preserving) get annotated exclusions with reasons, mirroring the `/* v8 ignore */` policy.
|
|
|
|
## Plan
|
|
|
|
1. Add Stryker config scoped to one package (llm — smallest, most algorithmic) and measure runtime.
|
|
2. Expand to all packages; record baseline scores in the config.
|
|
3. Wire the nightly job; add the incremental PR job once runtime is acceptable.
|
|
|
|
## Acceptance criteria
|
|
|
|
- A Stryker config runs over `packages/*/src` with the vitest runner; a nightly job records the mutation score, and a ratcheting threshold fails the run when the score drops below the recorded baseline.
|
|
- PR-scoped incremental runs gate merges once runtime is acceptable — or are explicitly kept nightly-only, with that outcome recorded here.
|
|
- Equivalent mutants carry annotated exclusions with reasons, mirroring the `/* v8 ignore */` policy.
|
|
|
|
## Risks
|
|
|
|
Runtime: mutation testing is expensive; per-file 100% coverage helps (every mutant is at least reached). If PR-scoped runs stay too slow, keep them nightly-only and rely on the score ratchet.
|
|
|
|
<!-- rfc-format: alternatives-not-recorded (pre-format RFC) -->
|