mirror of
https://github.com/deepseek-ai/deepseek-harness
synced 2026-08-15 21:04:50 +00:00
Merge branch 'master' into codex/skill-system
This commit is contained in:
Yichen Jiang
GitHub
@@ -1,17 +1,17 @@
|
||||
# Testing policy
|
||||
|
||||
How this repo tests, tier by tier, and the rules that keep a green suite meaning something. Commands live in the root [AGENTS.md](../AGENTS.md) § Commands; the RFCs linked per tier carry the design rationale.
|
||||
How this repo tests, tier by tier, and the rules that keep a green suite meaningful. Commands live in root [AGENTS.md](../AGENTS.md); linked RFCs carry the rationale.
|
||||
|
||||
## Tiers
|
||||
|
||||
- **Unit** (`pnpm run test`): vitest over `packages|examples/*/tests/**/*.spec.ts`, colocated with what they test. Every registry gets an HMR-safety test (dispose the contributing fiber, assert cleanup). Excessive tests are welcome — err toward covering edge cases, error paths, event ordering, and concurrency races; review findings get regression tests (see `packages/core/agent-loop/tests/review-fixes.spec.ts`).
|
||||
- **Unit** (`pnpm run test`): vitest over `packages|examples/*/tests/**/*.spec.ts`, colocated with what they test. Every registry gets an HMR-safety test (dispose the contributing fiber, assert cleanup). Prefer edge cases, error paths, event ordering, and concurrency races; review findings get regression tests (see `packages/core/agent-loop/tests/review-fixes.spec.ts`).
|
||||
- **Coverage gate** (`pnpm run test:coverage`): the gating run, per-file 100% on `packages/*/*/src`. An uncovered line is often dead code the gate is correctly flagging for deletion, not a missing test to bolt on. Line coverage is necessary, never sufficient — it proves lines ran, not that the feature works as shipped.
|
||||
- **Real-API e2e** (`pnpm run test:e2e`): with-key tests against live provider APIs — the DeepSeek model plus provider-specific smokes that gate on their own keys (`EXA_API_KEY`, `PERPLEXITY_API_KEY`, …); each suite self-skips without its key so keyless CI stays green ([real-API e2e RFC](rfc/implemented/testing/2026-06-19-real-api-e2e-ci.md)).
|
||||
- **Snapshot** (`pnpm run test:snapshot`): boots the real example subprocess, replays a recorded session keyless, diffs normalized stdout + the re-persisted log against committed goldens ([snapshot RFC](rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md)). Re-record with `pnpm run test:snapshot:record`; reviewing the golden diff is part of the review. System-prompt/tool-schema content is pinned by ONE scenario (`text-turn`) and tokenized in every other fixture, so a prompt or schema edit churns one committed line ([pinned-header RFC](rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
|
||||
- **Snapshot** (`pnpm run test:snapshot`): boots the real example subprocess, replays a recorded session keyless, diffs normalized stdout + the re-persisted log against committed goldens ([snapshot RFC](rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md)). Use `pnpm run test:snapshot:record` when the model transcript should change; use `pnpm run test:snapshot:refresh` when the committed transcript is still the right mock LLM input and replay goldens need keyless rewrite. Review the golden diff. System-prompt/tool-schema content is pinned by ONE scenario (`text-turn`) and tokenized in every other fixture, so a prompt or schema edit churns one committed line ([pinned-header RFC](rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
|
||||
|
||||
## The with-key policy: inference is cheap here
|
||||
|
||||
We are DeepSeek — do not ration real-API tests. A no-key test proves the plumbing; only a with-key run proves the agent works against a real model. Write many: real prompts that write files, multi-turn conversations, tool use, cancellation mid-stream. Cheapest and highest-value are **smoke tests** that boot the real example, send one real prompt, and check the world — they catch the "green unit tests, broken product" class that mocks structurally cannot ([postmortem 0001](postmortem/0001-acp-default-export-drops-inject.md)). The self-skip exists only so secretless CI and keyless contributors aren't blocked; it is not a cost signal. Every example ships a keyless smoke and — unless keyless-by-nature — a with-key smoke ([examples/AGENTS.md](../examples/AGENTS.md)).
|
||||
We are DeepSeek — do not ration real-API tests. A no-key test proves plumbing; only a with-key run proves the agent works against a real model. Write many: file-writing prompts, multi-turn conversations, tool use, cancellation mid-stream. Highest-value are **smoke tests** that boot the real example, send one real prompt, and check the world — they catch the "green unit tests, broken product" class that mocks structurally cannot ([postmortem 0001](postmortem/0001-acp-default-export-drops-inject.md)). The self-skip exists only so secretless CI and keyless contributors aren't blocked; it is not a cost signal. Every example ships a keyless smoke and — unless keyless-by-nature — a with-key smoke ([examples/AGENTS.md](../examples/AGENTS.md)).
|
||||
|
||||
## Prefer the real implementation over a mock
|
||||
|
||||
|
||||
@@ -22,6 +22,7 @@ export default tseslint.config(
|
||||
'**/lib/**',
|
||||
'**/node_modules/**',
|
||||
'**/.sessions/**',
|
||||
'.claude/**', // harness-local state (worktrees, skills) — other checkouts, not this one's sources
|
||||
'**/.doc-typecheck-*/**',
|
||||
'vendor/**', // vendored source keeps upstream style and idioms
|
||||
'**/*.js',
|
||||
@@ -38,7 +39,14 @@ export default tseslint.config(
|
||||
],
|
||||
languageOptions: {
|
||||
parserOptions: {
|
||||
project: ['./packages/*/*/tsconfig.json', './tsconfig.json'],
|
||||
// One shared tsserver-style project service instead of 60+ standalone
|
||||
// per-package programs: the old `project` glob built every package's
|
||||
// full dependency closure (sibling sources via the dev `paths` map +
|
||||
// the vendored Cordis stack) as its own program and kept them all
|
||||
// resident — ~5 GB peak, an OOM past node's default heap. The service
|
||||
// resolves each file to its nearest owning tsconfig and shares the
|
||||
// graph.
|
||||
projectService: true,
|
||||
tsconfigRootDir: import.meta.dirname,
|
||||
},
|
||||
},
|
||||
@@ -89,7 +97,9 @@ export default tseslint.config(
|
||||
],
|
||||
languageOptions: {
|
||||
parserOptions: {
|
||||
project: ['./tsconfig.json'],
|
||||
// Same shared project service as the src block: test files resolve
|
||||
// through the root tsconfig (its include covers every tests/ tree).
|
||||
projectService: true,
|
||||
tsconfigRootDir: import.meta.dirname,
|
||||
},
|
||||
},
|
||||
|
||||
@@ -33,7 +33,7 @@ The editor sets each session's `cwd` to the project it opens; both the agent's b
|
||||
|
||||
## Snapshot tests (record-once / replay-deterministic)
|
||||
|
||||
This example is the home of the harness's **snapshot tests** — they boot this server as a real subprocess, drive it with a deterministic input script, and diff its normalized output against committed golden files. The model is made deterministic by `@deepseek-ai/dsh-llm-replay`, a function/namespace plugin that installs an `llm/stream` waterfall listener and short-circuits it, serving model streams reconstructed from a recorded **session JSONL** fixture (`<scenario>/session.jsonl`) — so replay needs no API key. The fixture IS the persisted session log: its `assistant/chunk` events carry every `StreamChunk`, so grouping them by `(turn, step)` reconstructs each `stream()` call (one model call per loop step). Recording is therefore "run the real agent once and harvest the `.jsonl`". The two failure modes not expressible as logged chunks — a pure throw before any chunk, and cancel/hang — use an optional `<scenario>/replay.override.json` sidecar (a `ReplayEntry[]` that replaces the derived script). A scenario that needs the agent to operate on existing files ships an optional `<scenario>/workspace/` directory — the harness copies its contents into the temp cwd before the run (see `workspace-edit`). See [the ACP snapshot tests RFC](../../docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md) for the full design.
|
||||
This example is the home of the harness's **snapshot tests** — they boot this server as a real subprocess, drive it with a deterministic input script, and diff its normalized output against committed golden files. The model is made deterministic by `@deepseek-ai/dsh-llm-replay`, a function/namespace plugin that installs an `llm/stream` waterfall listener and short-circuits it, serving model streams reconstructed from a recorded **session JSONL** fixture (`<scenario>/session.jsonl`) — so replay needs no API key. The fixture IS the persisted session log: its `assistant/chunk` events carry every `StreamChunk`, so grouping them by `(turn, step)` reconstructs each `stream()` call (one model call per loop step). Recording is therefore "run the real agent once and harvest the `.jsonl`"; use `pnpm run test:snapshot:record` when the model transcript itself should change, and `pnpm run test:snapshot:refresh` when the committed model transcript is still the right mock input and only the current replay output/goldens need to be rewritten. The two failure modes not expressible as logged chunks — a pure throw before any chunk, and cancel/hang — use an optional `<scenario>/replay.override.json` sidecar (a `ReplayEntry[]` that replaces the derived script). A scenario that needs the agent to operate on existing files ships an optional `<scenario>/workspace/` directory — the harness copies its contents into the temp cwd before the run (see `workspace-edit`). See [the ACP snapshot tests RFC](../../docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md) for the full design.
|
||||
|
||||
## MVP limitations
|
||||
|
||||
|
||||
@@ -1,15 +1,16 @@
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import { dirname, join } from 'node:path'
|
||||
import { defineAcpSnapshotSuite, type Scenario } from '@deepseek-ai/dsh-acp-snapshot'
|
||||
import { defineAcpSnapshotSuite, type Scenario, type SnapshotSuiteOptions } from '@deepseek-ai/dsh-acp-snapshot'
|
||||
|
||||
/**
|
||||
* The acp-agent example's snapshot suite: the scenario table for
|
||||
* `dsh-acp-snapshot`'s suite factory, which owns every compare/guard mechanic
|
||||
* (golden + re-persisted-log diffs, record write-back, the pinned-header
|
||||
* (golden + re-persisted-log diffs, record/refresh write-back, the pinned-header
|
||||
* uniformity guard, the fixture guards). Fixtures live under `snapshots/<name>/`;
|
||||
* `pnpm run test:snapshot:record` re-records the `recorded` scenarios against
|
||||
* the real API. See the package README (packages/support/acp-snapshot) and the
|
||||
* snapshot RFC, docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md.
|
||||
* `pnpm run test:snapshot:record` re-records model transcripts against the real
|
||||
* API; `pnpm run test:snapshot:refresh` rewrites current replay goldens keyless.
|
||||
* See the package README (packages/support/acp-snapshot) and the snapshot RFC,
|
||||
* docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md.
|
||||
*/
|
||||
|
||||
// The dsh-acp-agent bin (the demo:acp entry), this example's cordis.yml, and
|
||||
@@ -26,6 +27,21 @@ const AGENT = {
|
||||
const CODE_MODE_CONFIG = fileURLToPath(new URL('../code-mode.cordis.yml', import.meta.url))
|
||||
const BOTH_MODE_CONFIG = fileURLToPath(new URL('../both-mode.cordis.yml', import.meta.url))
|
||||
|
||||
function snapshotModeFromEnv(value: string | undefined): SnapshotSuiteOptions['mode'] {
|
||||
switch (value) {
|
||||
case undefined:
|
||||
case '':
|
||||
case 'replay':
|
||||
return 'replay'
|
||||
case 'record':
|
||||
return 'record'
|
||||
case 'refresh':
|
||||
return 'refresh'
|
||||
default:
|
||||
throw new Error(`unknown DSH_SNAPSHOT mode: ${value}`)
|
||||
}
|
||||
}
|
||||
|
||||
const SCENARIOS: Scenario[] = [
|
||||
{ name: 'handshake', hasModelTurn: false, recorded: false },
|
||||
{ name: 'reject-extra-dirs', hasModelTurn: false, recorded: false },
|
||||
@@ -112,5 +128,5 @@ defineAcpSnapshotSuite({
|
||||
agent: AGENT,
|
||||
snapshotsDir: join(dirname(fileURLToPath(import.meta.url)), 'snapshots'),
|
||||
scenarios: SCENARIOS,
|
||||
mode: process.env.DSH_SNAPSHOT === 'record' ? 'record' : 'replay',
|
||||
mode: snapshotModeFromEnv(process.env.DSH_SNAPSHOT),
|
||||
})
|
||||
|
||||
@@ -22,6 +22,7 @@
|
||||
"test:e2e": "vitest run --config vitest.e2e.config.ts",
|
||||
"test:snapshot": "vitest run --config vitest.snapshot.config.ts",
|
||||
"test:snapshot:record": "DSH_SNAPSHOT=record vitest run --config vitest.snapshot.config.ts --update",
|
||||
"test:snapshot:refresh": "DSH_SNAPSHOT=refresh vitest run --config vitest.snapshot.config.ts",
|
||||
"check:ci": "tsx scripts/run-gates.ts ci-primary",
|
||||
"check:ci:static": "tsx scripts/run-gates.ts ci-static",
|
||||
"check:ci:lint": "tsx scripts/run-gates.ts ci-lint",
|
||||
|
||||
@@ -6,7 +6,7 @@ Three layers, importable separately:
|
||||
|
||||
- **`runScenario` (harness)** — boots the real agent bin as a subprocess via tsx (unbuilt, Loader path), drives it over ACP JSON-RPC stdio from a deterministic `input.json` script, tees raw stdout for the golden + purity check, and harvests every persisted session JSONL (parent + subagent children, primary-first) after a graceful stdin-EOF shutdown. Parameterized by `AgentUnderTest` (`binScript`, `configPath`, `tsconfigPath` — absolute paths; the subprocess cwd is a temp dir outside the repo).
|
||||
- **Normalizers** — pure functions turning the two captured surfaces into stable text: `normalizeStdout` (JSON-RPC ids → first-seen sequence; UUIDs/cwd → tokens; doubles as the stdout-purity check), `normalizeSessionLog` (times zeroed, `seq` kept), and the composable `scrubRequestHeaders` (header bulk → `{{system}}`/`{{tools}}`, structure kept — [pinned-header RFC](../../../docs/rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
|
||||
- **`defineAcpSnapshotSuite` (factory)** — registers the whole describe/it tree for a scenario table: per-scenario golden + re-persisted-log compares, record-mode fixture write-back, the per-header-class pin with its live uniformity guard, and the fixture guard block (no orphan scenario dirs, required files present, exactly one pin per class, pinning fixtures well-formed, non-pinning fixtures header-scrubbed). Must be called at vitest collection time.
|
||||
- **`defineAcpSnapshotSuite` (factory)** — registers the whole describe/it tree for a scenario table: per-scenario golden + re-persisted-log compares, record/refresh fixture write-back, the per-header-class pin with its live uniformity guard, and the fixture guard block (no orphan scenario dirs, required files present, exactly one pin per class, pinning fixtures well-formed, non-pinning fixtures header-scrubbed). Must be called at vitest collection time.
|
||||
|
||||
A consuming `*.snapshot.ts` is the scenario table plus one factory call:
|
||||
|
||||
@@ -27,12 +27,16 @@ defineAcpSnapshotSuite({
|
||||
},
|
||||
snapshotsDir: join(dirname(fileURLToPath(import.meta.url)), 'snapshots'),
|
||||
scenarios: SCENARIOS, // exactly one entry per header class sets pinsHeader
|
||||
mode: process.env.DSH_SNAPSHOT === 'record' ? 'record' : 'replay',
|
||||
mode: process.env.DSH_SNAPSHOT === 'record'
|
||||
? 'record'
|
||||
: process.env.DSH_SNAPSHOT === 'refresh'
|
||||
? 'refresh'
|
||||
: 'replay',
|
||||
})
|
||||
```
|
||||
|
||||
A scenario booting a differently-composed tree sets its own `configPath` (an overlay whose basename still ends in `cordis.yml`, so the bin's replay swap finds the sibling `*cordis.snapshot.yml`) and, when that composition changes the request header, its own `headerClass` with its own pinning scenario — the acp-agent example's Code Mode scenarios are the template.
|
||||
|
||||
The example also ships a `cordis.snapshot.yml` replay overlay next to its `cordis.yml` (the bin swaps them under `DSH_SNAPSHOT=replay` — [single-source replay config RFC](../../../docs/rfc/implemented/testing/2026-07-04-single-source-acp-replay-config.md)); replay fixtures are served by [`dsh-llm-replay`](../llm-replay/README.md), which this package points at via the `DSH_SNAPSHOT_*` env vars it sets on the child. Fixture roles, record/replay semantics, and scenario-table fields are documented on `Scenario` and in the [snapshot RFC](../../../docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md).
|
||||
The example also ships a `cordis.snapshot.yml` replay overlay next to its `cordis.yml` (the bin swaps them under `DSH_SNAPSHOT=replay` — [single-source replay config RFC](../../../docs/rfc/implemented/testing/2026-07-04-single-source-acp-replay-config.md)); replay fixtures are served by [`dsh-llm-replay`](../llm-replay/README.md), which this package points at via the `DSH_SNAPSHOT_*` env vars it sets on the child. `pnpm run test:snapshot:record` calls the live LLM and rewrites the recorded scenarios' model fixtures; `pnpm run test:snapshot:refresh` stays keyless, runs the replay overlay, and rewrites stdout plus comparable session-log goldens from the committed model scripts. Fixture roles, record/replay/refresh semantics, and scenario-table fields are documented on `Scenario` and in the [snapshot RFC](../../../docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md).
|
||||
|
||||
Constraints: `suite.ts` imports vitest, so the package is importable only inside a vitest run (the harness and normalizers have no such dependency but ship from the same entry). ACP-specific by design — the harness speaks the SDK's `ClientSideConnection`. Permission round-trips are scriptable: `InputScript.permissionAnswers` is a FIFO queue of option-kind selections (`allow_once`, `reject_once`, …) the client maps to the agent-issued `optionId` at answer time; an absent or exhausted queue answers `cancelled`, and a kind the request never offered rejects the run (the agent is answered `cancelled`, so a tolerant agent cannot absorb the scenario bug).
|
||||
|
||||
@@ -23,8 +23,11 @@
|
||||
*
|
||||
* `pnpm run test:snapshot:record` (DSH_SNAPSHOT=record + -u) re-records the
|
||||
* `session.jsonl` fixtures against the real API and refreshes the stdout golden
|
||||
* in one pass; the caller resolves that env into {@link SnapshotSuiteOptions}
|
||||
* (env reading stays at the suite edge, not in this library).
|
||||
* in one pass. `pnpm run test:snapshot:refresh` (DSH_SNAPSHOT=refresh) instead
|
||||
* replays the committed model scripts keylessly and writes the current stdout
|
||||
* + persisted-log goldens back without calling a live LLM. The caller resolves
|
||||
* that env into {@link SnapshotSuiteOptions} (env reading stays at the suite
|
||||
* edge, not in this library).
|
||||
*
|
||||
* @module @deepseek-ai/dsh-acp-snapshot/suite
|
||||
*/
|
||||
@@ -124,12 +127,13 @@ export interface SnapshotSuiteOptions {
|
||||
/** The scenario table; exactly one entry must set `pinsHeader`. */
|
||||
scenarios: Scenario[]
|
||||
/**
|
||||
* `replay` (keyless, the default tier) or `record` (live API; re-records the
|
||||
* `recorded` scenarios' fixtures and refreshes the vitest goldens under
|
||||
* `--update`). The caller derives this from `$DSH_SNAPSHOT` — env reading
|
||||
* stays outside this library.
|
||||
* `replay` (keyless, the default tier), `record` (live API; re-records the
|
||||
* `recorded` scenarios' fixtures and refreshes the Vitest goldens under
|
||||
* `--update`), or `refresh` (keyless replay that rewrites stdout goldens and
|
||||
* comparable session fixtures from the replay run). The caller derives this
|
||||
* from `$DSH_SNAPSHOT` — env reading stays outside this library.
|
||||
*/
|
||||
mode: 'replay' | 'record'
|
||||
mode: 'replay' | 'record' | 'refresh'
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -201,6 +205,86 @@ export function headerDeltaCount(rawLog: string): number {
|
||||
.length
|
||||
}
|
||||
|
||||
/** A literal string replacement used to carry an existing fixture's volatile value into a refreshed log. */
|
||||
export interface FixtureReplacement {
|
||||
/** The fresh replay-run value to replace. */
|
||||
from: string
|
||||
/** The existing fixture value to keep. */
|
||||
to: string
|
||||
}
|
||||
|
||||
function parseJsonlRecords(text: string): Record<string, unknown>[] {
|
||||
return text.split('\n')
|
||||
.filter(line => line.trim().length > 0)
|
||||
.map(line => JSON.parse(line) as Record<string, unknown>)
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the cross-log id/cwd replacements used by refresh write-back.
|
||||
*
|
||||
* @param logs The freshly harvested logs, in fixture order.
|
||||
* @param fixtures The existing fixture contents, in matching order.
|
||||
* @returns Literal replacements from fresh volatile values to the fixture's old values.
|
||||
*/
|
||||
export function refreshFixtureReplacements(logs: HarvestedLog[], fixtures: string[]): FixtureReplacement[] {
|
||||
const replacements: FixtureReplacement[] = []
|
||||
for (let i = 0; i < logs.length; i++) {
|
||||
const fresh = parseJsonlRecords((logs[i] as HarvestedLog).content)[0]
|
||||
const existing = parseJsonlRecords(fixtures[i] ?? '')[0]
|
||||
for (const field of ['id', 'cwd'] as const) {
|
||||
const from = fresh?.[field]
|
||||
const to = existing?.[field]
|
||||
if (typeof from === 'string' && typeof to === 'string' && from.length > 0 && from !== to) {
|
||||
replacements.push({ from, to })
|
||||
}
|
||||
}
|
||||
}
|
||||
return replacements
|
||||
}
|
||||
|
||||
function preserveFixtureVolatiles(record: Record<string, unknown>, existing: Record<string, unknown> | undefined): void {
|
||||
if (existing === undefined || existing.type !== record.type) return
|
||||
if (record.type === 'session') {
|
||||
for (const field of ['id', 'createdAt', 'cwd', 'parentSession'] as const) {
|
||||
if (field in record && field in existing) record[field] = existing[field]
|
||||
}
|
||||
return
|
||||
}
|
||||
if ('time' in record && 'time' in existing) record.time = existing.time
|
||||
if (record.type !== 'hook/result') return
|
||||
const data = record.data
|
||||
const existingData = existing.data
|
||||
if (
|
||||
data !== null && typeof data === 'object'
|
||||
&& existingData !== null && typeof existingData === 'object'
|
||||
&& 'durationMs' in data && 'durationMs' in existingData
|
||||
) {
|
||||
(data as Record<string, unknown>).durationMs = (existingData as Record<string, unknown>).durationMs
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Rewrite a fresh replay-produced log so repeated refreshes do not churn
|
||||
* volatile fixture fields. Meaningful event payloads come from `fresh`; the
|
||||
* existing fixture lends session ids, cwd, creation times, event times, and
|
||||
* hook durations where the record shape still matches.
|
||||
*
|
||||
* @param fresh The newly harvested session JSONL.
|
||||
* @param existing The committed fixture JSONL being refreshed.
|
||||
* @param replacements Cross-log literal replacements from {@link refreshFixtureReplacements}.
|
||||
* @returns The stabilized JSONL content to write back.
|
||||
*/
|
||||
export function stabilizeRefreshLog(fresh: string, existing: string, replacements: FixtureReplacement[]): string {
|
||||
let stable = fresh
|
||||
for (const { from, to } of replacements) stable = stable.split(from).join(to)
|
||||
const existingRecords = parseJsonlRecords(existing)
|
||||
const records = parseJsonlRecords(stable)
|
||||
for (let i = 0; i < records.length; i++) {
|
||||
preserveFixtureVolatiles(records[i] as Record<string, unknown>, existingRecords[i])
|
||||
}
|
||||
return records.map(record => JSON.stringify(record)).join('\n') + '\n'
|
||||
}
|
||||
|
||||
/**
|
||||
* Register the suite: one `describe` per scenario (the golden/log compares and
|
||||
* the header-uniformity guard) plus the fixture guard block (no orphan
|
||||
@@ -215,6 +299,8 @@ export function headerDeltaCount(rawLog: string): number {
|
||||
export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
|
||||
const { agent, snapshotsDir, scenarios, mode } = options
|
||||
const RECORDING = mode === 'record'
|
||||
const REFRESHING = mode === 'refresh'
|
||||
const childMode: 'replay' | 'record' = RECORDING ? 'record' : 'replay'
|
||||
|
||||
/** The class a scenario's header composition belongs to (see {@link Scenario.headerClass}). */
|
||||
const classOf = (scenario: Scenario): string => scenario.headerClass ?? 'default'
|
||||
@@ -238,15 +324,18 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
|
||||
describe(`snapshot: ${scenario.name}`, () => {
|
||||
// In RECORD mode, only re-run the `recorded` (live-API) scenarios; the
|
||||
// `authored` ones (sidecar-driven errors/cancel) are never re-recorded.
|
||||
// REFRESH mode is replay-backed and deterministic, so it runs every
|
||||
// scenario and rewrites the comparable fixtures from that replay run.
|
||||
it.skipIf(RECORDING && !scenario.recorded)('matches the goldens', async () => {
|
||||
const dir = join(snapshotsDir, scenario.name)
|
||||
const input = JSON.parse(await readFile(join(dir, 'input.json'), 'utf8')) as InputScript
|
||||
const overrideFile = join(dir, 'replay.override.json')
|
||||
const workspaceDir = join(dir, 'workspace')
|
||||
const childSessions = scenario.childSessions ?? 0
|
||||
const comparesLog = scenario.comparesLog ?? scenario.hasModelTurn
|
||||
const result = await runScenario(input, {
|
||||
agent,
|
||||
mode,
|
||||
mode: childMode,
|
||||
fixtureFile: join(dir, 'session.jsonl'),
|
||||
...existsSync(overrideFile) ? { overrideFile } : {},
|
||||
// In REPLAY, forward the recorded child fixtures so each subagent session
|
||||
@@ -271,30 +360,47 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
|
||||
}
|
||||
|
||||
// RECORD mode (recorded model scenarios only): persist the freshly-harvested
|
||||
// logs back to their fixtures — the primary to session.jsonl, each child to
|
||||
// session.<n>.jsonl in harvest order. `--update` refreshes the Vitest
|
||||
// goldens but NOT these fixtures, so write them here. A non-pinning
|
||||
// scenario's fixtures are written header-scrubbed, so a re-record can
|
||||
// never smuggle the full prompt/schema content back into every fixture.
|
||||
// live logs back to their fixtures. REFRESH mode does the same from a
|
||||
// keyless replay run for every comparable log, including authored
|
||||
// scenarios that live record deliberately skips. The primary goes to
|
||||
// session.jsonl, each child to session.<n>.jsonl in harvest order. A
|
||||
// non-pinning scenario's fixtures are written header-scrubbed, so a
|
||||
// re-record/refresh can never smuggle the full prompt/schema content
|
||||
// back into every fixture.
|
||||
const scrub = scenario.pinsHeader === true
|
||||
? (log: string): string => log
|
||||
: scrubRequestHeaders
|
||||
if (RECORDING && scenario.recorded && scenario.hasModelTurn) {
|
||||
expect(result.sessionLogs.length, 'record produced no session log to harvest').toBeGreaterThan(0)
|
||||
const fixtureFiles = ['session.jsonl', ...Array.from({ length: childSessions }, (_, i) => `session.${i + 1}.jsonl`)]
|
||||
const existingFixtures = REFRESHING
|
||||
? await Promise.all(fixtureFiles.map(file => readFile(join(dir, file), 'utf8')))
|
||||
: []
|
||||
const replacements = REFRESHING ? refreshFixtureReplacements(result.sessionLogs, existingFixtures) : []
|
||||
const writesSessionFixtures = (RECORDING && scenario.recorded && scenario.hasModelTurn)
|
||||
|| (REFRESHING && comparesLog)
|
||||
if (writesSessionFixtures) {
|
||||
expect(result.sessionLogs.length, `${mode} produced no session log to harvest`).toBeGreaterThan(0)
|
||||
expect(result.sessionLogs.length, `expected ${childSessions + 1} session logs (parent + children)`)
|
||||
.toBe(childSessions + 1)
|
||||
await writeFile(join(dir, 'session.jsonl'), scrub((result.sessionLogs[0] as HarvestedLog).content))
|
||||
const primary = (result.sessionLogs[0] as HarvestedLog).content
|
||||
await writeFile(join(dir, 'session.jsonl'), scrub(
|
||||
REFRESHING ? stabilizeRefreshLog(primary, existingFixtures[0] as string, replacements) : primary,
|
||||
))
|
||||
for (let i = 1; i < result.sessionLogs.length; i++) {
|
||||
await writeFile(join(dir, `session.${i}.jsonl`), scrub((result.sessionLogs[i] as HarvestedLog).content))
|
||||
const child = (result.sessionLogs[i] as HarvestedLog).content
|
||||
await writeFile(join(dir, `session.${i}.jsonl`), scrub(
|
||||
REFRESHING ? stabilizeRefreshLog(child, existingFixtures[i] as string, replacements) : child,
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
await expect(normalizeStdout(result.rawStdout, ctx))
|
||||
.toMatchFileSnapshot(join(dir, 'stdout.golden.jsonl'))
|
||||
const stdout = normalizeStdout(result.rawStdout, ctx)
|
||||
if (REFRESHING) {
|
||||
await writeFile(join(dir, 'stdout.golden.jsonl'), stdout)
|
||||
}
|
||||
await expect(stdout).toMatchFileSnapshot(join(dir, 'stdout.golden.jsonl'))
|
||||
|
||||
// A model turn always produces a log worth comparing; a hook scenario can
|
||||
// produce one without a model turn (a `rejected` turn carrying `hook/*`).
|
||||
const comparesLog = scenario.comparesLog ?? scenario.hasModelTurn
|
||||
if (comparesLog) {
|
||||
// The harvested logs (primary-first) must match their committed fixtures
|
||||
// 1:1. Each side passes through normalizeSessionLog, scrubbed against ITS
|
||||
@@ -307,7 +413,6 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void {
|
||||
// reason, and config, but not its bulk content (pinned once, in the
|
||||
// `pinsHeader` scenario).
|
||||
expect(result.sessionLogs.length, 'this scenario must persist a session log').toBe(childSessions + 1)
|
||||
const fixtureFiles = ['session.jsonl', ...Array.from({ length: childSessions }, (_, i) => `session.${i + 1}.jsonl`)]
|
||||
for (let i = 0; i < fixtureFiles.length; i++) {
|
||||
const harvested = scrub((result.sessionLogs[i] as HarvestedLog).content)
|
||||
const fixture = scrub(await readFile(join(dir, fixtureFiles[i] as string), 'utf8'))
|
||||
|
||||
@@ -1,11 +1,18 @@
|
||||
import { cpSync, mkdtempSync } from 'node:fs'
|
||||
import { cpSync, mkdtempSync, readFileSync, writeFileSync } from 'node:fs'
|
||||
import { rm } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import { afterAll, describe, expect, it } from 'vitest'
|
||||
import { defineAcpSnapshotSuite, type Scenario } from '../src/index.ts'
|
||||
import { childFixturePaths, fixtureContext, headerDeltaCount, normalizedHeaders } from '../src/suite.ts'
|
||||
import { defineAcpSnapshotSuite, type HarvestedLog, type Scenario } from '../src/index.ts'
|
||||
import {
|
||||
childFixturePaths,
|
||||
fixtureContext,
|
||||
headerDeltaCount,
|
||||
normalizedHeaders,
|
||||
refreshFixtureReplacements,
|
||||
stabilizeRefreshLog,
|
||||
} from '../src/suite.ts'
|
||||
|
||||
/**
|
||||
* Unit tests for the suite factory, by running it: two synthetic suites over
|
||||
@@ -55,16 +62,40 @@ const RECORD_SCENARIOS: Scenario[] = [
|
||||
{ name: 'rec-skip', hasModelTurn: true, recorded: false, overridden: true },
|
||||
]
|
||||
|
||||
// Record mode mutates its snapshots dir, so run it on a throwaway copy —
|
||||
// except under the documented bootstrap knob, which regenerates the committed
|
||||
// fixtures/goldens in place.
|
||||
// Record/refresh modes mutate their snapshots dir, so run them on throwaway
|
||||
// copies — except record's documented bootstrap knob, which regenerates the
|
||||
// committed record fixtures/goldens in place.
|
||||
const BOOTSTRAP = process.env.ACP_SNAPSHOT_SPEC_BOOTSTRAP === '1'
|
||||
const recordDir = BOOTSTRAP ? RECORD_SRC : mkdtempSync(join(tmpdir(), 'acp-snap-record-suite-'))
|
||||
if (!BOOTSTRAP) cpSync(RECORD_SRC, recordDir, { recursive: true })
|
||||
const refreshDir = mkdtempSync(join(tmpdir(), 'acp-snap-refresh-suite-'))
|
||||
cpSync(REPLAY_DIR, refreshDir, { recursive: true })
|
||||
staleRefreshFixtures(refreshDir)
|
||||
afterAll(async () => {
|
||||
if (!BOOTSTRAP) await rm(recordDir, { recursive: true, force: true })
|
||||
await rm(refreshDir, { recursive: true, force: true })
|
||||
})
|
||||
|
||||
function staleRefreshFixtures(dir: string): void {
|
||||
writeFileSync(join(dir, 'plain-turn', 'stdout.golden.jsonl'), 'stale stdout\n')
|
||||
|
||||
const plainBehaviorFile = join(dir, 'plain-turn', 'behavior.json')
|
||||
const plainBehavior = JSON.parse(readFileSync(plainBehaviorFile, 'utf8')) as Record<string, unknown>
|
||||
plainBehavior.echoEnv = true
|
||||
writeFileSync(plainBehaviorFile, `${JSON.stringify(plainBehavior, null, 2)}\n`)
|
||||
|
||||
writeFileSync(join(dir, 'blocked-log', 'session.jsonl'), [
|
||||
'{"type":"session","id":"99999999-8888-4777-8666-555555555555","createdAt":13,"cwd":"/rec/blocked-cwd"}',
|
||||
'{"type":"hook/result","seq":1,"time":13,"data":{"decision":"stale","durationMs":99}}',
|
||||
'',
|
||||
].join('\n'))
|
||||
writeFileSync(join(dir, 'authored-error', 'session.jsonl'), [
|
||||
'{"type":"session","id":"77777777-8888-4777-8666-555555555555","createdAt":13,"cwd":"/rec/error-cwd"}',
|
||||
'{"type":"turn/end","seq":1,"time":9,"data":{"error":"stale"}}',
|
||||
'',
|
||||
].join('\n'))
|
||||
}
|
||||
|
||||
describe('defineAcpSnapshotSuite: replay mode', () => {
|
||||
defineAcpSnapshotSuite({ agent: AGENT, snapshotsDir: REPLAY_DIR, scenarios: REPLAY_SCENARIOS, mode: 'replay' })
|
||||
})
|
||||
@@ -75,6 +106,27 @@ describe('defineAcpSnapshotSuite: record mode', () => {
|
||||
defineAcpSnapshotSuite({ agent: AGENT, snapshotsDir: recordDir, scenarios: RECORD_SCENARIOS, mode: 'record' })
|
||||
})
|
||||
|
||||
describe('defineAcpSnapshotSuite: refresh mode', () => {
|
||||
defineAcpSnapshotSuite({ agent: AGENT, snapshotsDir: refreshDir, scenarios: REPLAY_SCENARIOS, mode: 'refresh' })
|
||||
})
|
||||
|
||||
describe('defineAcpSnapshotSuite: refresh write-back', () => {
|
||||
it('rewrites stdout and comparable logs from a replay-mode child run', () => {
|
||||
const stdout = readFileSync(join(refreshDir, 'plain-turn', 'stdout.golden.jsonl'), 'utf8')
|
||||
expect(stdout).not.toContain('stale stdout')
|
||||
expect(stdout).toContain('env:{\\"mode\\":\\"replay\\"')
|
||||
expect(stdout).not.toContain('\\"mode\\":\\"refresh\\"')
|
||||
|
||||
const blocked = readFileSync(join(refreshDir, 'blocked-log', 'session.jsonl'), 'utf8')
|
||||
expect(blocked).toContain('"decision":"block"')
|
||||
expect(blocked).not.toContain('"decision":"stale"')
|
||||
|
||||
const authored = readFileSync(join(refreshDir, 'authored-error', 'session.jsonl'), 'utf8')
|
||||
expect(authored).toContain('"error":"model exploded"')
|
||||
expect(authored).not.toContain('"error":"stale"')
|
||||
})
|
||||
})
|
||||
|
||||
describe('defineAcpSnapshotSuite: registration contract', () => {
|
||||
it("throws when a scenario's header class has no pinning scenario", () => {
|
||||
expect(() => {
|
||||
@@ -175,3 +227,56 @@ describe('headerDeltaCount', () => {
|
||||
expect(headerDeltaCount(`${other}\n`)).toBe(0)
|
||||
})
|
||||
})
|
||||
|
||||
describe('refreshFixtureReplacements', () => {
|
||||
it('maps fresh ids and cwd values to the existing fixture values, skipping non-replacements', () => {
|
||||
const log = (content: string): HarvestedLog => ({ id: 'diagnostic', createdAt: 1, content })
|
||||
const logs = [
|
||||
log('{"type":"session","id":"","cwd":"/same"}\n'),
|
||||
log('{"type":"session","id":"new-parent","cwd":"/new"}\n'),
|
||||
log('{"type":"session","id":"new-child","cwd":"/new"}\n'),
|
||||
]
|
||||
const fixtures = [
|
||||
'{"type":"session","id":"","cwd":"/same"}\n',
|
||||
'{"type":"session","id":"old-parent","cwd":"/old"}\n',
|
||||
]
|
||||
expect(refreshFixtureReplacements(logs, fixtures)).toEqual([
|
||||
{ from: 'new-parent', to: 'old-parent' },
|
||||
{ from: '/new', to: '/old' },
|
||||
])
|
||||
})
|
||||
})
|
||||
|
||||
describe('stabilizeRefreshLog', () => {
|
||||
it('keeps volatile fixture fields while preserving fresh meaningful payloads', () => {
|
||||
const fresh = [
|
||||
'{"type":"session","id":"new-child","createdAt":200,"cwd":"/new","parentSession":"new-parent","seedLength":1}',
|
||||
'{"type":"hook/result","seq":1,"time":22,"data":{"decision":"block","durationMs":37}}',
|
||||
'{"type":"turn/end","seq":2,"time":33,"data":{"error":"fresh error"}}',
|
||||
'{"type":"tool/result","seq":3,"time":44,"data":{"text":"new-parent in /new"}}',
|
||||
'{"type":"hook/result","seq":4,"time":55,"data":{"decision":"allow","durationMs":5}}',
|
||||
'',
|
||||
].join('\n')
|
||||
const existing = [
|
||||
'{"type":"session","id":"old-child","createdAt":100,"cwd":"/old","parentSession":"old-parent","seedLength":5}',
|
||||
'{"type":"hook/result","seq":1,"time":11,"data":{"decision":"stale","durationMs":99}}',
|
||||
'{"type":"turn/end","seq":2,"data":{"error":"stale"}}',
|
||||
'{"type":"assistant/message","seq":3,"time":12,"data":{"text":"different type"}}',
|
||||
'{"type":"hook/result","seq":4,"time":13,"data":{"decision":"stale"}}',
|
||||
'',
|
||||
].join('\n')
|
||||
|
||||
expect(stabilizeRefreshLog(fresh, existing, [
|
||||
{ from: 'new-parent', to: 'old-parent' },
|
||||
{ from: 'new-child', to: 'old-child' },
|
||||
{ from: '/new', to: '/old' },
|
||||
])).toBe([
|
||||
'{"type":"session","id":"old-child","createdAt":100,"cwd":"/old","parentSession":"old-parent","seedLength":1}',
|
||||
'{"type":"hook/result","seq":1,"time":11,"data":{"decision":"block","durationMs":99}}',
|
||||
'{"type":"turn/end","seq":2,"time":33,"data":{"error":"fresh error"}}',
|
||||
'{"type":"tool/result","seq":3,"time":44,"data":{"text":"old-parent in /old"}}',
|
||||
'{"type":"hook/result","seq":4,"time":13,"data":{"decision":"allow","durationMs":5}}',
|
||||
'',
|
||||
].join('\n'))
|
||||
})
|
||||
})
|
||||
|
||||
@@ -7,10 +7,14 @@ import { defineConfig } from 'vitest/config'
|
||||
// normalized stdout transcript + re-persisted log against committed goldens.
|
||||
// `pnpm run test:snapshot:record` (DSH_SNAPSHOT=record + -u) re-records the
|
||||
// fixtures against the real API and refreshes the goldens.
|
||||
// `pnpm run test:snapshot:refresh` (DSH_SNAPSHOT=refresh) stays keyless: it
|
||||
// replays the committed model scripts and writes the current stdout/log goldens
|
||||
// without calling the live LLM.
|
||||
//
|
||||
// Replay loads no .env (it must never reach the network — a recorded fixture
|
||||
// drives the model). Record reads DEEPSEEK_API_KEY from the env or a gitignored
|
||||
// repo-root .env, so a contributor with a key only in .env can still record.
|
||||
// drives the model), and refresh uses that same keyless replay path. Record
|
||||
// reads DEEPSEEK_API_KEY from the env or a gitignored repo-root .env, so a
|
||||
// contributor with a key only in .env can still record.
|
||||
if (process.env.DSH_SNAPSHOT === 'record') {
|
||||
try {
|
||||
process.loadEnvFile(new URL('.env', import.meta.url).pathname)
|
||||
|
||||
Reference in New Issue
Block a user