mirror of https://github.com/deepseek-ai/deepseek-harness synced 2026-08-15 21:04:50 +00:00

Files

Tianyi Cui e6fad266a6 docs(rfc): define and enforce a uniform RFC format; adopt it across the corpus

Define the in-file RFC contract in docs/rfc/README.md § The file format:
the header block (`# RFC: <title>` plus a dateless Status enum
cross-checked against the lifecycle folder), the per-lifecycle body
skeleton (a Problem opener everywhere; Proposal/Alternatives considered/
Acceptance criteria/Risks in proposed/; present-tense Decision/
Consequences with proposal-era headings banned in implemented/; the
frozen proposal shape in rejected/), and a mandatory Alternatives
considered section with a date-fenced grandfather comment for pre-format
RFCs whose alternatives are not reconstructible from the record.

Enforce it with a new doc-sync gate, scripts/verify-rfc-format.ts, and
normalize all 112 RFCs to it: ~15 Status-line spellings collapse to the
enum, 29 Context openers become Problem, the 39 legacy-format XXX debt
markers are resolved and banned from reappearing, proposal-era sections
in implemented RFCs are rewritten to shipped reality (including the
web/fs/subagent seam RFCs' migration plans and test checklists, closing
the doc-tiers deferred-work item on the web seam), every RFC gains an
Alternatives considered section or the grandfather comment, and the
bilingual pair is re-mirrored and re-recorded.

Move the generated index tables out of README.md into a fully generated
docs/rfc/INDEX.md — gen-rfc-index now writes the whole file, and
verify-rfc-classification checks its freshness and rejects index-shaped
rows in the curated README — which makes room for the format contract to
live in the README front door instead of a separate FORMAT.md.

The decision record, and the first RFC written in the new format, is
docs/rfc/implemented/process/2026-07-05-uniform-rfc-format.md.

2026-07-05 22:58:25 +08:00

3.4 KiB

Raw Blame History

RFC: Property-based testing for protocol-shaped code

Status: implemented

Merges the original proposal and the decision record for one topic. It found a real BlockAssembler duplicate-block-end bug on first run.

Problem

Example-based tests pin the cases we thought of. The harness's core is protocol-shaped — chunk streams, event logs, schema conversion, inbox scheduling — where the input space is combinatorial and the interesting bugs live in interleavings nobody wrote an example for. The motivating evidence: a block-assembly ordering bug once survived 100% line coverage of the happy paths. Per-file 100% coverage proves every line ran, not that every interleaving is correct.

Decision

Adopt fast-check (a root devDependency) with one tests/properties.spec.ts per protocol-shaped package, generators tuned for realistic-but-adversarial inputs (not uniform noise) and numRuns kept so the suite stays well under ~10s locally. Failures print a reproducible seed. (The original proposal also sketched a nightly CI job running 100× the iterations; that was not shipped — the property suite runs only in the normal push/pull_request CI, and a scheduled high-iteration job remains possible future work.)

dsh-llm / BlockAssembler: arbitrary chunk streams (valid + malformed: duplicate indices, stragglers, missing block-start). Invariants: blocks() count ≤ distinct indices seen; re-assembly idempotent (blocks() is stable across repeated calls and message().content mirrors it); blocks() never throws and yields only valid content-block tags; finish reflects the last finish chunk, defaulting to {kind:'stop'} when none arrives.
dsh-session: arbitrary event logs. Invariants: deriveMessages deterministic; replay-from-seed identical; seq strictly monotonic; non-message events never affect derived history; derived content is decoupled from the log.
dsh-tools: arbitrary SchemaSpec. Invariants: JSON Schema required equals the required:true keys at every level; conversion total; and the composition with runtime arg validation — generated args satisfying a spec pass validateArgs, and targeted corruptions (dropped required key, non-object top level) are rejected. This closes the validator/InferArgs drift risk.
dsh-agent-loop: arbitrary send schedules against a never-exhausting adapter, driven through the agent/status settle signal (no wall-clock sleeps). Invariants: no message lost; turn numbers strictly increase; status transitions stay on the legal machine.

Consequences

Generator quality is the value lever — the generators bias toward small index pools and short strings so collisions and interleavings are common.
It already paid off: the BlockAssembler stream found a real bug — a duplicate block-end at the same index overwrote an already-flushed block, so the streamed prefix disagreed with final blocks(). Fixed (first close wins, matching the existing straggler rule) with a dedicated regression test.
A property flake from a timeout is a finding, not something to retry away. The loop properties are deterministic by construction (settle on agent/status), so a hang is a real defect.
Property tests supplement, not replace, the example tests that pin specific branches for the 100%-coverage gate.

3.4 KiB Raw Blame History Unescape Escape

RFC: Property-based testing for protocol-shaped code

Problem

Decision

Consequences

3.4 KiB

Raw Blame History