DeepSeek/deepseek-harness
1
0
Fork 0
mirror of https://github.com/deepseek-ai/deepseek-harness synced 2026-08-15 21:04:50 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/master' into worktree/acp-automation-protocol

Browse Source 
# Conflicts:
#	.agents/notes/implemented/architecture/2026-06-14-session-persistence.md
#	.agents/notes/implemented/architecture/2026-06-20-package-hierarchy.md
#	.agents/notes/implemented/architecture/2026-07-02-tool-render-intent-union.md
#	.agents/notes/implemented/feature/2026-06-14-acp-agent-client-protocol.md
#	.agents/notes/implemented/feature/2026-06-14-acp-multi-session.md
#	.agents/notes/implemented/feature/2026-06-18-acp-terminal-and-tool-rendering.md
#	.agents/notes/implemented/feature/2026-07-19-model-facing-goal-tools.i18n.yaml
#	.agents/notes/implemented/feature/2026-07-19-model-facing-goal-tools.md
#	.agents/notes/implemented/feature/2026-07-19-model-facing-goal-tools.zh.md
#	.agents/notes/implemented/simplification/2026-07-04-trim-acp-bridge-unreachable-surface.md
#	docs/architecture.i18n.yaml
#	docs/cookbook/extension-cookbook.i18n.yaml
#	docs/cookbook/extension-cookbook.md
#	docs/cookbook/extension-cookbook.zh.md
#	docs/core-data-structures/approval.md
#	docs/core-data-structures/user-interaction.md
#	docs/event-producer-consumer.md
#	docs/persistence-catalog.md
#	docs/testing.md
#	docs/tool-catalog.md
#	examples/acp-agent/tests/fixtures/live-mode-switching-2026-07-07.session.jsonl
#	examples/acp-agent/tests/snapshots/cordis-inspect-jsdoc/stdout.expected.jsonl
#	examples/acp-agent/tests/snapshots/permission-switching/session.jsonl
#	packages/goal/tool-goal/README.md
#	packages/ui/acp/README.md
#	packages/ui/acp/acp-feature-support.md
#	packages/ui/acp/src/index.ts
#	packages/ui/acp/tests/bridge.spec.ts
#	packages/ui/acp/tests/dispose.spec.ts
#	packages/ui/acp/tests/edges.spec.ts
#	packages/ui/acp/tests/turns.spec.ts
This commit is contained in:
Tianyi Cui
2026-07-24 23:43:10 +08:00
parent d01474d1d0 b97b1b4a3f
commit 440384fdb4
829 changed files with 23607 additions and 1815 deletions
Download Patch File Download Diff File Expand all files Collapse all files

10
docs/testing.md
View File

@@ -1,21 +1,23 @@
# Testing policy
English | [中文](testing.zh.md)
How this repo tests, tier by tier, and the rules that keep a green suite meaningful. Commands live in root [AGENTS.md](../AGENTS.md); linked Agent Notes carry the rationale.
## Tiers
- **Unit** (`pnpm run test`): vitest over `packages|examples/*/tests/**/*.spec.ts`, colocated with what they test. Every registry gets an HMR-safety test (dispose the contributing fiber, assert cleanup). Prefer edge cases, error paths, event ordering, concurrency races, and permanent contract regressions (see `packages/core/agent-loop/tests/contract-regressions.spec.ts`).
- **Unit** (`pnpm run test`): vitest over package and example specs under their `tests/**` directories plus repository script specs under `scripts/**/*.spec.ts`; tests stay with the code area they exercise. Every registry gets an HMR-safety test (dispose the contributing fiber, assert cleanup). Prefer edge cases, error paths, event ordering, concurrency races, and permanent contract regressions (see `packages/core/agent-loop/tests/contract-regressions.spec.ts`).
- **Coverage gate** (`pnpm run test:coverage`): the gating run, per-file 100% on `packages/*/*/src`. An uncovered line is often dead code the gate is correctly flagging for deletion, not a missing test to bolt on. Line coverage is necessary, never sufficient — it proves lines ran, not that the feature works as shipped.
- **Real-API e2e** (`pnpm run test:e2e`): with-key tests against live provider APIs — the DeepSeek model plus provider-specific smokes that gate on their own keys (`EXA_API_KEY`, `PERPLEXITY_API_KEY`, …); each suite self-skips without its key so keyless CI stays green ([real-API e2e Agent Note](../.agents/notes/implemented/testing/2026-06-19-real-api-e2e-ci.md)).
- **Snapshot** (`pnpm run test:snapshot`): transport-specific keyless expected outputs cover protocol or presentation behavior, while persisted logs pin assembled backend behavior. ACP suites boot the real automation server subprocess, replay a recorded session, and diff normalized JSON-RPC plus the re-persisted log ([ACP snapshot Agent Note](../.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md)); the headless suite independently pins `stream-json` through its real one-shot subprocess. TUI completed journeys replay recorded primary/child JSONL through the real agent loop and tools before projecting ANSI into semantic terminal-state expected outputs; package-local snapshots retain transient renderer states, and a real PTY conversation covers the process boundary ([TUI snapshot Agent Note](../.agents/notes/implemented/testing/2026-07-18-tui-terminal-state-snapshots.md)). Use `pnpm run test:snapshot:record` when a model transcript must change and `pnpm run test:snapshot:refresh` when committed replay input remains correct; review every JSONL and expected-output diff. System-prompt/tool-schema content is pinned by one ACP scenario (`text-turn`) and tokenized in every other fixture, so a prompt or schema edit churns one committed line ([pinned-header Agent Note](../.agents/notes/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
- **Snapshot** (`pnpm run test:snapshot`): keyless expected outputs cover external behavior — transport contracts and presentation, while persisted logs pin assembled backend behavior. ACP boots the real automation-server example, replays a recorded session, and diffs normalized JSON-RPC plus the re-persisted log ([ACP snapshot Agent Note](../.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md)); headless pins `stream-json` through its real one-shot process. TUI journeys replay primary/child JSONL through the real loop and tools, then project ANSI into semantic terminal-state outputs; package snapshots retain transient states and a real PTY covers the process boundary ([TUI snapshot Agent Note](../.agents/notes/implemented/testing/2026-07-18-tui-terminal-state-snapshots.md)). Use `pnpm run test:snapshot:record` when a model transcript changes and `pnpm run test:snapshot:refresh` when replay input remains valid; review every JSONL and expected-output diff. One ACP scenario (`text-turn`) pins full system-prompt/tool-schema content; other fixtures tokenize it so an edit churns one line ([pinned-header Agent Note](../.agents/notes/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
## The with-key policy: inference is cheap here
We are DeepSeek — do not ration real-API tests. A no-key test proves plumbing; only a with-key run proves the agent works against a real model. Write many: file-writing prompts, multi-turn conversations, tool use, cancellation mid-stream. Highest-value are **smoke tests** that boot the real example, send one real prompt, and check the world — they catch the "green unit tests, broken product" class that mocks structurally cannot ([postmortem 0001](postmortem/0001-acp-default-export-drops-inject.md)). The self-skip exists only so secretless CI and keyless contributors aren't blocked; it is not a cost signal. Every example ships both a keyless smoke and a with-key smoke ([examples/AGENTS.md](../examples/AGENTS.md)).
We are DeepSeek — do not ration real-API tests. A no-key test proves plumbing; only a with-key run proves the agent works against a real model. Cover file-writing prompts, multi-turn conversations, tool use, and mid-stream cancellation. Highest-value are **smoke tests** that boot the real example, send one prompt, and check the world — they catch the "green unit tests, broken product" class that mocks cannot ([postmortem 0001](postmortem/0001-acp-default-export-drops-inject.md)). Self-skip keeps secretless CI and keyless contributors unblocked; it is not a cost signal. Every example ships keyless and with-key smokes ([examples/AGENTS.md](../examples/AGENTS.md)).
## Prefer the real implementation over a mock
Mock only the genuinely expensive or non-deterministic boundary (the LLM adapter, the network, the clock); keep everything downstream real. A hand-rolled stand-in proves the bridge moves bytes, not that the shipping tool behaves as asserted — the two drift while the test stays green. Example: bridge tool-call tests run the scripted mock MODEL but the real tool + real executor (`makeBridgeHarness({ withBash: true })` plugs `dsh-bash-local` + `dsh-tool-bash` and runs an actual `echo`).
Mock only the expensive or non-deterministic boundary (LLM adapter, network, clock); keep everything downstream real. A hand-rolled stand-in proves the bridge moves bytes, not that the shipping tool behaves as asserted. Bridge tool-call tests use the scripted mock model with the real tool and executor: `makeBridgeHarness({ withBash: true })` plugs in `dsh-bash-local` and `dsh-tool-bash`, then runs `echo`.
Recovery tests separate pre/post-chunk failures by step and prove failed chunks derive no message or tool side effect. Cover exhaustion, cancellation, policy composition, persistence, status, wire counts, transport-closing idle timeouts, and shipping Loader composition.