mirror of
https://github.com/deepseek-ai/deepseek-harness
synced 2026-08-15 21:04:50 +00:00
7702a33531
Applies the documentation standard to its biggest offender. Every rule survives as one to three lines plus a link to its durable home; the stories, duplicate statements, and re-narrations go: - Situational clusters evict to new homes: docs/testing.md (tiers, with-key policy, real-over-mock, world-verification, real-entry-path guards), docs/defensive-patterns.md (the bug-class rules), and docs/cookbook/responding-to-pr-review-on-a-stack.md (the stacked-PR review procedure). - Doc-authoring rules consolidate into docs/AGENTS.md § Writing rules (current-state-never-history, md-wrap, ts-block compilation, @mode, catalog same-change, pair same-change). - packages/README.md drops to the group table + the extension-vs-bundle dependency rule; the hand ASCII graph yields to the generated module-graph.md; group READMEs are the canonical per-package map. - packages/AGENTS.md keeps only its packages-specific rules (export shape, ctx.get, real-Loader coverage); examples/AGENTS.md repoints its with-key-policy link; rfc/README.md loses a narrated-history aside; dsh-code-review / dsh-find-simplifications / verify-md-wrap references follow the moved content. - Budget manifest ratchets: AGENTS.md 8200 -> 1500 (now 1,495 words), packages/README.md 1900 -> 600, packages/AGENTS.md 600 -> 450; the two new eviction docs join the budget set (testing 800, defensive 550); docs/AGENTS.md raises 1000 -> 1250 for the absorbed writing rules (the one justified increase). The doc-tiers RFC's deferred list prunes the two items this change ships.
34 lines
5.0 KiB
Markdown
34 lines
5.0 KiB
Markdown
# Testing policy
|
|
|
|
How this repo tests, tier by tier, and the rules that keep a green suite meaning something. Commands live in the root [AGENTS.md](../AGENTS.md) § Commands; the RFCs linked per tier carry the design rationale.
|
|
|
|
## Tiers
|
|
|
|
- **Unit** (`pnpm run test`): vitest, colocated at `packages/<group>/<pkg>/tests/*.spec.ts`. Every registry gets an HMR-safety test (dispose the contributing fiber, assert cleanup). Excessive tests are welcome — err toward covering edge cases, error paths, event ordering, and concurrency races; review findings get regression tests (see `packages/core/agent-loop/tests/review-fixes.spec.ts`).
|
|
- **Coverage gate** (`pnpm run test:coverage`): the gating run, per-file 100% on `packages/*/*/src`. An uncovered line is often dead code the gate is correctly flagging for deletion, not a missing test to bolt on. Line coverage is necessary, never sufficient — it proves lines ran, not that the feature works as shipped.
|
|
- **Real-API e2e** (`pnpm run test:e2e`): with-key tests against the live DeepSeek API; self-skip without `DEEPSEEK_API_KEY` so keyless CI stays green ([real-API e2e RFC](rfc/implemented/testing/2026-06-19-real-api-e2e-ci.md)).
|
|
- **Snapshot** (`pnpm run test:snapshot`): boots the real example subprocess, replays a recorded session keyless, diffs normalized stdout + the re-persisted log against committed goldens ([snapshot RFC](rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md)). Re-record with `pnpm run test:snapshot:record`; reviewing the golden diff is part of the review.
|
|
|
|
## The with-key policy: inference is cheap here
|
|
|
|
We are DeepSeek — do not ration real-API tests. A no-key test proves the plumbing; only a with-key run proves the agent works against a real model. Write many: real prompts that write files, multi-turn conversations, tool use, cancellation mid-stream. Cheapest and highest-value are **smoke tests** that boot the real example, send one real prompt, and check the world — they catch the "green unit tests, broken product" class that mocks structurally cannot ([postmortem 0001](postmortem/0001-acp-default-export-drops-inject.md)). The self-skip exists only so secretless CI and keyless contributors aren't blocked; it is not a cost signal. Every example ships both a keyless and a with-key smoke ([examples/AGENTS.md](../examples/AGENTS.md)).
|
|
|
|
## Prefer the real implementation over a mock
|
|
|
|
Mock only the genuinely expensive or non-deterministic boundary (the LLM adapter, the network, the clock); keep everything downstream real. A hand-rolled stand-in proves the bridge moves bytes, not that the shipping tool behaves as asserted — the two drift while the test stays green. Example: bridge tool-call tests run the scripted mock MODEL but the real tool + real executor (`makeBridgeHarness({ withBash: true })` plugs `dsh-bash-local` + `dsh-tool-bash` and runs an actual `echo`).
|
|
|
|
## Verify the world, not the self-report
|
|
|
|
An e2e assertion re-runs the command or re-reads the file externally; a keyword probe on the agent's own output lets a cheating agent pass. Assert untouched files are byte-identical. e2e tests own their resources: create the harness in the test, dispose in `afterEach` (even on failure/retry/timeout); shared fixtures live in a plain `tests/harness.ts`, never another `*.e2e.ts` (importing a spec re-registers its `describe` and duplicates real API calls).
|
|
|
|
## Test the real entry path
|
|
|
|
- A plugin shipped via `cordis.yml` needs at least one test through the REAL Loader path: hand-built `ctx.plugin({...})` mounts bypass `unwrapExports` and cannot catch a broken export shape ([postmortem 0001](postmortem/0001-acp-default-export-drops-inject.md); export-shape rules in [packages/AGENTS.md](../packages/AGENTS.md)).
|
|
- A guard only guards if the regression actually fails it. For a plugin without `inject` (bundle/composition plugins), a Loader smoke stays green under a broken export shape — add an explicit `expect('default' in mod).toBe(false)` plus an `unwrapExports` round-trip assertion, and prove it: introduce the regression, watch red, revert.
|
|
- "Real entry path" means the published artifact: the package `bin` points at built `lib/bin.js` under plain `node`, which tsx masks (settle races, module resolution, a swallowed load failure exiting 0). Keep the built-bin smokes green (`packages/ui/*/tests/built-bin.e2e.ts`), and assert a genuinely-missing config exits non-zero.
|
|
- An e2e that spawns an example from a temp cwd sets `TSX_TSCONFIG_PATH` to the repo-root tsconfig, or it silently falls back to stale built `lib/` ([examples/AGENTS.md](../examples/AGENTS.md)).
|
|
|
|
## When a snapshot test is required
|
|
|
|
Any change affecting the editor-facing transcript or end-to-end agent UX — the ACP bridge, the loop's observable output, tool presentation — adds or updates a scenario under `examples/acp-agent/tests/snapshots/` (or states in the PR why none applies). New capability seams, lifecycle shapes, or transcript surfaces name their coverage at every tier at plan time and verify the harness can express it — a harness gap is scheduled work, not a mid-build surprise.
|