mirror of
https://github.com/deepseek-ai/deepseek-harness
synced 2026-08-15 21:04:50 +00:00
2f6d3b8539
Adds fast-check + one tests/properties.spec.ts per protocol-shaped package (llm/BlockAssembler, session, tools/schema DSL, agent-loop scheduling). The tools suite includes the RFC 001<->005 composition property (generated args satisfying a spec pass validateArgs), closing the validator/InferArgs drift risk from ADR 0011. Loop properties are deterministic (settle on agent/status, no sleeps). The BlockAssembler suite found a real bug on first run: a duplicate block-end at the same index overwrote an already-flushed block, so the streamed prefix disagreed with final blocks(). Fixed (first close wins, matching the existing straggler rule) + regression test. Graduates RFC 001 -> ADR 0013.
25 lines
2.2 KiB
Markdown
25 lines
2.2 KiB
Markdown
# RFC 001: Property-based testing for protocol-shaped code
|
||
|
||
Status: implemented — see [ADR 0013](../adr/0013-property-based-testing.md). (It found a real BlockAssembler duplicate-`block-end` bug on first run.)
|
||
|
||
## Problem
|
||
|
||
Example-based tests pin the cases we thought of. The harness's core is protocol-shaped — chunk streams, event logs, schema conversion — where the input space is combinatorial and the interesting bugs live in interleavings nobody wrote an example for (the `streamBlocks` ordering bug survived 100% line coverage of the happy paths).
|
||
|
||
## Proposal
|
||
|
||
Adopt fast-check (vitest integration) with generators for our vocabulary:
|
||
|
||
- **BlockAssembler**: arbitrary chunk sequences (valid and malformed — duplicate indices, stragglers after block-end, missing block-start). Invariants: `flushReady() + flushRemaining() ≡ blocks()` in order; `streamBlocks ≡ generate().message.content`; memory bounded (partials map size ≤ distinct indices); idempotent re-assembly.
|
||
- **Session**: arbitrary event logs (seeded generators over SessionEventMap). Invariants: `deriveMessages` deterministic; replay-from-seed produces identical derivation; seq strictly monotonic; derived history unaffected by non-message events.
|
||
- **Schema DSL**: arbitrary SchemaSpecs. Invariants: generated JSON Schema's `required` array equals the `required: true` keys at every nesting level; conversion is total (never throws); generated args satisfying `InferArgs` validate against the generated schema (once RFC 005's validator exists — the two RFCs compose).
|
||
- **Inbox/loop**: arbitrary send/steer/abort schedules against a scripted adapter. Invariants: no message lost (every send/steer appears in the log exactly once), turn numbers strictly increase, status transitions follow idle→running→idle/disposed.
|
||
|
||
## Plan
|
||
|
||
One `tests/properties.spec.ts` per package; fast-check as devDependency; numRuns tuned so the suite stays under ~10s locally, with a nightly CI job running 100× the iterations. Failures persist their seed in the report so agents can reproduce deterministically.
|
||
|
||
## Risks
|
||
|
||
Generator quality determines value — invest in generators that produce *realistic-but-adversarial* streams, not uniform noise. Property flake from timeouts must be treated as a finding, not retried away.
|