From 9d2cf8ce826ee82c2af13f25de12bdf1920dda7e Mon Sep 17 00:00:00 2001 From: Tianyi Cui <53024+tianyicui@users.noreply.github.com> Date: Fri, 10 Jul 2026 00:48:27 +0800 Subject: [PATCH 1/4] Add keyless snapshot refresh mode --- docs/testing.md | 8 +- examples/acp-agent/README.md | 2 +- examples/acp-agent/tests/acp.snapshot.ts | 28 +++- package.json | 1 + packages/support/acp-snapshot/README.md | 10 +- packages/support/acp-snapshot/src/suite.ts | 147 +++++++++++++++--- .../support/acp-snapshot/tests/suite.spec.ts | 117 +++++++++++++- vitest.snapshot.config.ts | 8 +- 8 files changed, 278 insertions(+), 43 deletions(-) diff --git a/docs/testing.md b/docs/testing.md index a4a77c9d0c..62989a1096 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -1,17 +1,17 @@ # Testing policy -How this repo tests, tier by tier, and the rules that keep a green suite meaning something. Commands live in the root [AGENTS.md](../AGENTS.md) § Commands; the RFCs linked per tier carry the design rationale. +How this repo tests, tier by tier, and the rules that keep a green suite meaningful. Commands live in root [AGENTS.md](../AGENTS.md); linked RFCs carry the rationale. ## Tiers -- **Unit** (`pnpm run test`): vitest over `packages|examples/*/tests/**/*.spec.ts`, colocated with what they test. Every registry gets an HMR-safety test (dispose the contributing fiber, assert cleanup). Excessive tests are welcome — err toward covering edge cases, error paths, event ordering, and concurrency races; review findings get regression tests (see `packages/core/agent-loop/tests/review-fixes.spec.ts`). +- **Unit** (`pnpm run test`): vitest over `packages|examples/*/tests/**/*.spec.ts`, colocated with what they test. Every registry gets an HMR-safety test (dispose the contributing fiber, assert cleanup). Prefer edge cases, error paths, event ordering, and concurrency races; review findings get regression tests (see `packages/core/agent-loop/tests/review-fixes.spec.ts`). - **Coverage gate** (`pnpm run test:coverage`): the gating run, per-file 100% on `packages/*/*/src`. An uncovered line is often dead code the gate is correctly flagging for deletion, not a missing test to bolt on. Line coverage is necessary, never sufficient — it proves lines ran, not that the feature works as shipped. - **Real-API e2e** (`pnpm run test:e2e`): with-key tests against live provider APIs — the DeepSeek model plus provider-specific smokes that gate on their own keys (`EXA_API_KEY`, `PERPLEXITY_API_KEY`, …); each suite self-skips without its key so keyless CI stays green ([real-API e2e RFC](rfc/implemented/testing/2026-06-19-real-api-e2e-ci.md)). -- **Snapshot** (`pnpm run test:snapshot`): boots the real example subprocess, replays a recorded session keyless, diffs normalized stdout + the re-persisted log against committed goldens ([snapshot RFC](rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md)). Re-record with `pnpm run test:snapshot:record`; reviewing the golden diff is part of the review. System-prompt/tool-schema content is pinned by ONE scenario (`text-turn`) and tokenized in every other fixture, so a prompt or schema edit churns one committed line ([pinned-header RFC](rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)). +- **Snapshot** (`pnpm run test:snapshot`): boots the real example subprocess, replays a recorded session keyless, diffs normalized stdout + the re-persisted log against committed goldens ([snapshot RFC](rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md)). Use `pnpm run test:snapshot:record` when the model transcript should change; use `pnpm run test:snapshot:refresh` when the committed transcript is still the right mock LLM input and replay goldens need keyless rewrite. Review the golden diff. System-prompt/tool-schema content is pinned by ONE scenario (`text-turn`) and tokenized in every other fixture, so a prompt or schema edit churns one committed line ([pinned-header RFC](rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)). ## The with-key policy: inference is cheap here -We are DeepSeek — do not ration real-API tests. A no-key test proves the plumbing; only a with-key run proves the agent works against a real model. Write many: real prompts that write files, multi-turn conversations, tool use, cancellation mid-stream. Cheapest and highest-value are **smoke tests** that boot the real example, send one real prompt, and check the world — they catch the "green unit tests, broken product" class that mocks structurally cannot ([postmortem 0001](postmortem/0001-acp-default-export-drops-inject.md)). The self-skip exists only so secretless CI and keyless contributors aren't blocked; it is not a cost signal. Every example ships a keyless smoke and — unless keyless-by-nature — a with-key smoke ([examples/AGENTS.md](../examples/AGENTS.md)). +We are DeepSeek — do not ration real-API tests. A no-key test proves plumbing; only a with-key run proves the agent works against a real model. Write many: file-writing prompts, multi-turn conversations, tool use, cancellation mid-stream. Highest-value are **smoke tests** that boot the real example, send one real prompt, and check the world — they catch the "green unit tests, broken product" class that mocks structurally cannot ([postmortem 0001](postmortem/0001-acp-default-export-drops-inject.md)). The self-skip exists only so secretless CI and keyless contributors aren't blocked; it is not a cost signal. Every example ships a keyless smoke and — unless keyless-by-nature — a with-key smoke ([examples/AGENTS.md](../examples/AGENTS.md)). ## Prefer the real implementation over a mock diff --git a/examples/acp-agent/README.md b/examples/acp-agent/README.md index 5b3c936651..85cb185c82 100644 --- a/examples/acp-agent/README.md +++ b/examples/acp-agent/README.md @@ -33,7 +33,7 @@ The editor sets each session's `cwd` to the project it opens; both the agent's b ## Snapshot tests (record-once / replay-deterministic) -This example is the home of the harness's **snapshot tests** — they boot this server as a real subprocess, drive it with a deterministic input script, and diff its normalized output against committed golden files. The model is made deterministic by `@deepseek-ai/dsh-llm-replay`, a function/namespace plugin that installs an `llm/stream` waterfall listener and short-circuits it, serving model streams reconstructed from a recorded **session JSONL** fixture (`/session.jsonl`) — so replay needs no API key. The fixture IS the persisted session log: its `assistant/chunk` events carry every `StreamChunk`, so grouping them by `(turn, step)` reconstructs each `stream()` call (one model call per loop step). Recording is therefore "run the real agent once and harvest the `.jsonl`". The two failure modes not expressible as logged chunks — a pure throw before any chunk, and cancel/hang — use an optional `/replay.override.json` sidecar (a `ReplayEntry[]` that replaces the derived script). A scenario that needs the agent to operate on existing files ships an optional `/workspace/` directory — the harness copies its contents into the temp cwd before the run (see `workspace-edit`). See [the ACP snapshot tests RFC](../../docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md) for the full design. +This example is the home of the harness's **snapshot tests** — they boot this server as a real subprocess, drive it with a deterministic input script, and diff its normalized output against committed golden files. The model is made deterministic by `@deepseek-ai/dsh-llm-replay`, a function/namespace plugin that installs an `llm/stream` waterfall listener and short-circuits it, serving model streams reconstructed from a recorded **session JSONL** fixture (`/session.jsonl`) — so replay needs no API key. The fixture IS the persisted session log: its `assistant/chunk` events carry every `StreamChunk`, so grouping them by `(turn, step)` reconstructs each `stream()` call (one model call per loop step). Recording is therefore "run the real agent once and harvest the `.jsonl`"; use `pnpm run test:snapshot:record` when the model transcript itself should change, and `pnpm run test:snapshot:refresh` when the committed model transcript is still the right mock input and only the current replay output/goldens need to be rewritten. The two failure modes not expressible as logged chunks — a pure throw before any chunk, and cancel/hang — use an optional `/replay.override.json` sidecar (a `ReplayEntry[]` that replaces the derived script). A scenario that needs the agent to operate on existing files ships an optional `/workspace/` directory — the harness copies its contents into the temp cwd before the run (see `workspace-edit`). See [the ACP snapshot tests RFC](../../docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md) for the full design. ## MVP limitations diff --git a/examples/acp-agent/tests/acp.snapshot.ts b/examples/acp-agent/tests/acp.snapshot.ts index d45ddf8591..7102d8ce88 100644 --- a/examples/acp-agent/tests/acp.snapshot.ts +++ b/examples/acp-agent/tests/acp.snapshot.ts @@ -1,15 +1,16 @@ import { fileURLToPath } from 'node:url' import { dirname, join } from 'node:path' -import { defineAcpSnapshotSuite, type Scenario } from '@deepseek-ai/dsh-acp-snapshot' +import { defineAcpSnapshotSuite, type Scenario, type SnapshotSuiteOptions } from '@deepseek-ai/dsh-acp-snapshot' /** * The acp-agent example's snapshot suite: the scenario table for * `dsh-acp-snapshot`'s suite factory, which owns every compare/guard mechanic - * (golden + re-persisted-log diffs, record write-back, the pinned-header + * (golden + re-persisted-log diffs, record/refresh write-back, the pinned-header * uniformity guard, the fixture guards). Fixtures live under `snapshots//`; - * `pnpm run test:snapshot:record` re-records the `recorded` scenarios against - * the real API. See the package README (packages/support/acp-snapshot) and the - * snapshot RFC, docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md. + * `pnpm run test:snapshot:record` re-records model transcripts against the real + * API; `pnpm run test:snapshot:refresh` rewrites current replay goldens keyless. + * See the package README (packages/support/acp-snapshot) and the snapshot RFC, + * docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md. */ // The dsh-acp-agent bin (the demo:acp entry), this example's cordis.yml, and @@ -26,6 +27,21 @@ const AGENT = { const CODE_MODE_CONFIG = fileURLToPath(new URL('../code-mode.cordis.yml', import.meta.url)) const BOTH_MODE_CONFIG = fileURLToPath(new URL('../both-mode.cordis.yml', import.meta.url)) +function snapshotModeFromEnv(value: string | undefined): SnapshotSuiteOptions['mode'] { + switch (value) { + case undefined: + case '': + case 'replay': + return 'replay' + case 'record': + return 'record' + case 'refresh': + return 'refresh' + default: + throw new Error(`unknown DSH_SNAPSHOT mode: ${value}`) + } +} + const SCENARIOS: Scenario[] = [ { name: 'handshake', hasModelTurn: false, recorded: false }, { name: 'reject-extra-dirs', hasModelTurn: false, recorded: false }, @@ -111,5 +127,5 @@ defineAcpSnapshotSuite({ agent: AGENT, snapshotsDir: join(dirname(fileURLToPath(import.meta.url)), 'snapshots'), scenarios: SCENARIOS, - mode: process.env.DSH_SNAPSHOT === 'record' ? 'record' : 'replay', + mode: snapshotModeFromEnv(process.env.DSH_SNAPSHOT), }) diff --git a/package.json b/package.json index c4cc307893..7f068c7f19 100644 --- a/package.json +++ b/package.json @@ -22,6 +22,7 @@ "test:e2e": "vitest run --config vitest.e2e.config.ts", "test:snapshot": "vitest run --config vitest.snapshot.config.ts", "test:snapshot:record": "DSH_SNAPSHOT=record vitest run --config vitest.snapshot.config.ts --update", + "test:snapshot:refresh": "DSH_SNAPSHOT=refresh vitest run --config vitest.snapshot.config.ts", "check:ci": "tsx scripts/run-gates.ts ci-primary", "check:ci:static": "tsx scripts/run-gates.ts ci-static", "check:ci:lint": "tsx scripts/run-gates.ts ci-lint", diff --git a/packages/support/acp-snapshot/README.md b/packages/support/acp-snapshot/README.md index 209b1a81cb..8ff2fe1413 100644 --- a/packages/support/acp-snapshot/README.md +++ b/packages/support/acp-snapshot/README.md @@ -6,7 +6,7 @@ Three layers, importable separately: - **`runScenario` (harness)** — boots the real agent bin as a subprocess via tsx (unbuilt, Loader path), drives it over ACP JSON-RPC stdio from a deterministic `input.json` script, tees raw stdout for the golden + purity check, and harvests every persisted session JSONL (parent + subagent children, primary-first) after a graceful stdin-EOF shutdown. Parameterized by `AgentUnderTest` (`binScript`, `configPath`, `tsconfigPath` — absolute paths; the subprocess cwd is a temp dir outside the repo). - **Normalizers** — pure functions turning the two captured surfaces into stable text: `normalizeStdout` (JSON-RPC ids → first-seen sequence; UUIDs/cwd → tokens; doubles as the stdout-purity check), `normalizeSessionLog` (times zeroed, `seq` kept), and the composable `scrubRequestHeaders` (header bulk → `{{system}}`/`{{tools}}`, structure kept — [pinned-header RFC](../../../docs/rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)). -- **`defineAcpSnapshotSuite` (factory)** — registers the whole describe/it tree for a scenario table: per-scenario golden + re-persisted-log compares, record-mode fixture write-back, the per-header-class pin with its live uniformity guard, and the fixture guard block (no orphan scenario dirs, required files present, exactly one pin per class, pinning fixtures well-formed, non-pinning fixtures header-scrubbed). Must be called at vitest collection time. +- **`defineAcpSnapshotSuite` (factory)** — registers the whole describe/it tree for a scenario table: per-scenario golden + re-persisted-log compares, record/refresh fixture write-back, the per-header-class pin with its live uniformity guard, and the fixture guard block (no orphan scenario dirs, required files present, exactly one pin per class, pinning fixtures well-formed, non-pinning fixtures header-scrubbed). Must be called at vitest collection time. A consuming `*.snapshot.ts` is the scenario table plus one factory call: @@ -27,12 +27,16 @@ defineAcpSnapshotSuite({ }, snapshotsDir: join(dirname(fileURLToPath(import.meta.url)), 'snapshots'), scenarios: SCENARIOS, // exactly one entry per header class sets pinsHeader - mode: process.env.DSH_SNAPSHOT === 'record' ? 'record' : 'replay', + mode: process.env.DSH_SNAPSHOT === 'record' + ? 'record' + : process.env.DSH_SNAPSHOT === 'refresh' + ? 'refresh' + : 'replay', }) ``` A scenario booting a differently-composed tree sets its own `configPath` (an overlay whose basename still ends in `cordis.yml`, so the bin's replay swap finds the sibling `*cordis.snapshot.yml`) and, when that composition changes the request header, its own `headerClass` with its own pinning scenario — the acp-agent example's Code Mode scenarios are the template. -The example also ships a `cordis.snapshot.yml` replay overlay next to its `cordis.yml` (the bin swaps them under `DSH_SNAPSHOT=replay` — [single-source replay config RFC](../../../docs/rfc/implemented/testing/2026-07-04-single-source-acp-replay-config.md)); replay fixtures are served by [`dsh-llm-replay`](../llm-replay/README.md), which this package points at via the `DSH_SNAPSHOT_*` env vars it sets on the child. Fixture roles, record/replay semantics, and scenario-table fields are documented on `Scenario` and in the [snapshot RFC](../../../docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md). +The example also ships a `cordis.snapshot.yml` replay overlay next to its `cordis.yml` (the bin swaps them under `DSH_SNAPSHOT=replay` — [single-source replay config RFC](../../../docs/rfc/implemented/testing/2026-07-04-single-source-acp-replay-config.md)); replay fixtures are served by [`dsh-llm-replay`](../llm-replay/README.md), which this package points at via the `DSH_SNAPSHOT_*` env vars it sets on the child. `pnpm run test:snapshot:record` calls the live LLM and rewrites the recorded scenarios' model fixtures; `pnpm run test:snapshot:refresh` stays keyless, runs the replay overlay, and rewrites stdout plus comparable session-log goldens from the committed model scripts. Fixture roles, record/replay/refresh semantics, and scenario-table fields are documented on `Scenario` and in the [snapshot RFC](../../../docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md). Constraints: `suite.ts` imports vitest, so the package is importable only inside a vitest run (the harness and normalizers have no such dependency but ship from the same entry). ACP-specific by design — the harness speaks the SDK's `ClientSideConnection`. Permission round-trips are scriptable: `InputScript.permissionAnswers` is a FIFO queue of option-kind selections (`allow_once`, `reject_once`, …) the client maps to the agent-issued `optionId` at answer time; an absent or exhausted queue answers `cancelled`, and a kind the request never offered rejects the run (the agent is answered `cancelled`, so a tolerant agent cannot absorb the scenario bug). diff --git a/packages/support/acp-snapshot/src/suite.ts b/packages/support/acp-snapshot/src/suite.ts index 53e05d7876..f97cb82c59 100644 --- a/packages/support/acp-snapshot/src/suite.ts +++ b/packages/support/acp-snapshot/src/suite.ts @@ -23,8 +23,11 @@ * * `pnpm run test:snapshot:record` (DSH_SNAPSHOT=record + -u) re-records the * `session.jsonl` fixtures against the real API and refreshes the stdout golden - * in one pass; the caller resolves that env into {@link SnapshotSuiteOptions} - * (env reading stays at the suite edge, not in this library). + * in one pass. `pnpm run test:snapshot:refresh` (DSH_SNAPSHOT=refresh) instead + * replays the committed model scripts keylessly and writes the current stdout + * + persisted-log goldens back without calling a live LLM. The caller resolves + * that env into {@link SnapshotSuiteOptions} (env reading stays at the suite + * edge, not in this library). * * @module @deepseek-ai/dsh-acp-snapshot/suite */ @@ -124,12 +127,13 @@ export interface SnapshotSuiteOptions { /** The scenario table; exactly one entry must set `pinsHeader`. */ scenarios: Scenario[] /** - * `replay` (keyless, the default tier) or `record` (live API; re-records the - * `recorded` scenarios' fixtures and refreshes the vitest goldens under - * `--update`). The caller derives this from `$DSH_SNAPSHOT` — env reading - * stays outside this library. + * `replay` (keyless, the default tier), `record` (live API; re-records the + * `recorded` scenarios' fixtures and refreshes the Vitest goldens under + * `--update`), or `refresh` (keyless replay that rewrites stdout goldens and + * comparable session fixtures from the replay run). The caller derives this + * from `$DSH_SNAPSHOT` — env reading stays outside this library. */ - mode: 'replay' | 'record' + mode: 'replay' | 'record' | 'refresh' } /** @@ -201,6 +205,86 @@ export function headerDeltaCount(rawLog: string): number { .length } +/** A literal string replacement used to carry an existing fixture's volatile value into a refreshed log. */ +export interface FixtureReplacement { + /** The fresh replay-run value to replace. */ + from: string + /** The existing fixture value to keep. */ + to: string +} + +function parseJsonlRecords(text: string): Record[] { + return text.split('\n') + .filter(line => line.trim().length > 0) + .map(line => JSON.parse(line) as Record) +} + +/** + * Build the cross-log id/cwd replacements used by refresh write-back. + * + * @param logs The freshly harvested logs, in fixture order. + * @param fixtures The existing fixture contents, in matching order. + * @returns Literal replacements from fresh volatile values to the fixture's old values. + */ +export function refreshFixtureReplacements(logs: HarvestedLog[], fixtures: string[]): FixtureReplacement[] { + const replacements: FixtureReplacement[] = [] + for (let i = 0; i < logs.length; i++) { + const fresh = parseJsonlRecords((logs[i] as HarvestedLog).content)[0] + const existing = parseJsonlRecords(fixtures[i] ?? '')[0] + for (const field of ['id', 'cwd'] as const) { + const from = fresh?.[field] + const to = existing?.[field] + if (typeof from === 'string' && typeof to === 'string' && from.length > 0 && from !== to) { + replacements.push({ from, to }) + } + } + } + return replacements +} + +function preserveFixtureVolatiles(record: Record, existing: Record | undefined): void { + if (existing === undefined || existing.type !== record.type) return + if (record.type === 'session') { + for (const field of ['id', 'createdAt', 'cwd', 'parentSession', 'seedLength'] as const) { + if (field in record && field in existing) record[field] = existing[field] + } + return + } + if ('time' in record && 'time' in existing) record.time = existing.time + if (record.type !== 'hook/result') return + const data = record.data + const existingData = existing.data + if ( + data !== null && typeof data === 'object' + && existingData !== null && typeof existingData === 'object' + && 'durationMs' in data && 'durationMs' in existingData + ) { + (data as Record).durationMs = (existingData as Record).durationMs + } +} + +/** + * Rewrite a fresh replay-produced log so repeated refreshes do not churn + * volatile fixture fields. Meaningful event payloads come from `fresh`; the + * existing fixture lends session ids, cwd, creation times, event times, and + * hook durations where the record shape still matches. + * + * @param fresh The newly harvested session JSONL. + * @param existing The committed fixture JSONL being refreshed. + * @param replacements Cross-log literal replacements from {@link refreshFixtureReplacements}. + * @returns The stabilized JSONL content to write back. + */ +export function stabilizeRefreshLog(fresh: string, existing: string, replacements: FixtureReplacement[]): string { + let stable = fresh + for (const { from, to } of replacements) stable = stable.split(from).join(to) + const existingRecords = parseJsonlRecords(existing) + const records = parseJsonlRecords(stable) + for (let i = 0; i < records.length; i++) { + preserveFixtureVolatiles(records[i] as Record, existingRecords[i]) + } + return records.map(record => JSON.stringify(record)).join('\n') + '\n' +} + /** * Register the suite: one `describe` per scenario (the golden/log compares and * the header-uniformity guard) plus the fixture guard block (no orphan @@ -215,6 +299,8 @@ export function headerDeltaCount(rawLog: string): number { export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void { const { agent, snapshotsDir, scenarios, mode } = options const RECORDING = mode === 'record' + const REFRESHING = mode === 'refresh' + const childMode: 'replay' | 'record' = RECORDING ? 'record' : 'replay' /** The class a scenario's header composition belongs to (see {@link Scenario.headerClass}). */ const classOf = (scenario: Scenario): string => scenario.headerClass ?? 'default' @@ -238,15 +324,18 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void { describe(`snapshot: ${scenario.name}`, () => { // In RECORD mode, only re-run the `recorded` (live-API) scenarios; the // `authored` ones (sidecar-driven errors/cancel) are never re-recorded. + // REFRESH mode is replay-backed and deterministic, so it runs every + // scenario and rewrites the comparable fixtures from that replay run. it.skipIf(RECORDING && !scenario.recorded)('matches the goldens', async () => { const dir = join(snapshotsDir, scenario.name) const input = JSON.parse(await readFile(join(dir, 'input.json'), 'utf8')) as InputScript const overrideFile = join(dir, 'replay.override.json') const workspaceDir = join(dir, 'workspace') const childSessions = scenario.childSessions ?? 0 + const comparesLog = scenario.comparesLog ?? scenario.hasModelTurn const result = await runScenario(input, { agent, - mode, + mode: childMode, fixtureFile: join(dir, 'session.jsonl'), ...existsSync(overrideFile) ? { overrideFile } : {}, // In REPLAY, forward the recorded child fixtures so each subagent session @@ -271,30 +360,47 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void { } // RECORD mode (recorded model scenarios only): persist the freshly-harvested - // logs back to their fixtures — the primary to session.jsonl, each child to - // session..jsonl in harvest order. `--update` refreshes the Vitest - // goldens but NOT these fixtures, so write them here. A non-pinning - // scenario's fixtures are written header-scrubbed, so a re-record can - // never smuggle the full prompt/schema content back into every fixture. + // live logs back to their fixtures. REFRESH mode does the same from a + // keyless replay run for every comparable log, including authored + // scenarios that live record deliberately skips. The primary goes to + // session.jsonl, each child to session..jsonl in harvest order. A + // non-pinning scenario's fixtures are written header-scrubbed, so a + // re-record/refresh can never smuggle the full prompt/schema content + // back into every fixture. const scrub = scenario.pinsHeader === true ? (log: string): string => log : scrubRequestHeaders - if (RECORDING && scenario.recorded && scenario.hasModelTurn) { - expect(result.sessionLogs.length, 'record produced no session log to harvest').toBeGreaterThan(0) + const fixtureFiles = ['session.jsonl', ...Array.from({ length: childSessions }, (_, i) => `session.${i + 1}.jsonl`)] + const existingFixtures = REFRESHING + ? await Promise.all(fixtureFiles.map(file => readFile(join(dir, file), 'utf8'))) + : [] + const replacements = REFRESHING ? refreshFixtureReplacements(result.sessionLogs, existingFixtures) : [] + const writesSessionFixtures = (RECORDING && scenario.recorded && scenario.hasModelTurn) + || (REFRESHING && comparesLog) + if (writesSessionFixtures) { + expect(result.sessionLogs.length, `${mode} produced no session log to harvest`).toBeGreaterThan(0) expect(result.sessionLogs.length, `expected ${childSessions + 1} session logs (parent + children)`) .toBe(childSessions + 1) - await writeFile(join(dir, 'session.jsonl'), scrub((result.sessionLogs[0] as HarvestedLog).content)) + const primary = (result.sessionLogs[0] as HarvestedLog).content + await writeFile(join(dir, 'session.jsonl'), scrub( + REFRESHING ? stabilizeRefreshLog(primary, existingFixtures[0] as string, replacements) : primary, + )) for (let i = 1; i < result.sessionLogs.length; i++) { - await writeFile(join(dir, `session.${i}.jsonl`), scrub((result.sessionLogs[i] as HarvestedLog).content)) + const child = (result.sessionLogs[i] as HarvestedLog).content + await writeFile(join(dir, `session.${i}.jsonl`), scrub( + REFRESHING ? stabilizeRefreshLog(child, existingFixtures[i] as string, replacements) : child, + )) } } - await expect(normalizeStdout(result.rawStdout, ctx)) - .toMatchFileSnapshot(join(dir, 'stdout.golden.jsonl')) + const stdout = normalizeStdout(result.rawStdout, ctx) + if (REFRESHING) { + await writeFile(join(dir, 'stdout.golden.jsonl'), stdout) + } + await expect(stdout).toMatchFileSnapshot(join(dir, 'stdout.golden.jsonl')) // A model turn always produces a log worth comparing; a hook scenario can // produce one without a model turn (a `rejected` turn carrying `hook/*`). - const comparesLog = scenario.comparesLog ?? scenario.hasModelTurn if (comparesLog) { // The harvested logs (primary-first) must match their committed fixtures // 1:1. Each side passes through normalizeSessionLog, scrubbed against ITS @@ -307,7 +413,6 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void { // reason, and config, but not its bulk content (pinned once, in the // `pinsHeader` scenario). expect(result.sessionLogs.length, 'this scenario must persist a session log').toBe(childSessions + 1) - const fixtureFiles = ['session.jsonl', ...Array.from({ length: childSessions }, (_, i) => `session.${i + 1}.jsonl`)] for (let i = 0; i < fixtureFiles.length; i++) { const harvested = scrub((result.sessionLogs[i] as HarvestedLog).content) const fixture = scrub(await readFile(join(dir, fixtureFiles[i] as string), 'utf8')) diff --git a/packages/support/acp-snapshot/tests/suite.spec.ts b/packages/support/acp-snapshot/tests/suite.spec.ts index 0f47010a19..8e79e367f2 100644 --- a/packages/support/acp-snapshot/tests/suite.spec.ts +++ b/packages/support/acp-snapshot/tests/suite.spec.ts @@ -1,11 +1,18 @@ -import { cpSync, mkdtempSync } from 'node:fs' +import { cpSync, mkdtempSync, readFileSync, writeFileSync } from 'node:fs' import { rm } from 'node:fs/promises' import { tmpdir } from 'node:os' import { join } from 'node:path' import { fileURLToPath } from 'node:url' import { afterAll, describe, expect, it } from 'vitest' -import { defineAcpSnapshotSuite, type Scenario } from '../src/index.ts' -import { childFixturePaths, fixtureContext, headerDeltaCount, normalizedHeaders } from '../src/suite.ts' +import { defineAcpSnapshotSuite, type HarvestedLog, type Scenario } from '../src/index.ts' +import { + childFixturePaths, + fixtureContext, + headerDeltaCount, + normalizedHeaders, + refreshFixtureReplacements, + stabilizeRefreshLog, +} from '../src/suite.ts' /** * Unit tests for the suite factory, by running it: two synthetic suites over @@ -55,16 +62,40 @@ const RECORD_SCENARIOS: Scenario[] = [ { name: 'rec-skip', hasModelTurn: true, recorded: false, overridden: true }, ] -// Record mode mutates its snapshots dir, so run it on a throwaway copy — -// except under the documented bootstrap knob, which regenerates the committed -// fixtures/goldens in place. +// Record/refresh modes mutate their snapshots dir, so run them on throwaway +// copies — except record's documented bootstrap knob, which regenerates the +// committed record fixtures/goldens in place. const BOOTSTRAP = process.env.ACP_SNAPSHOT_SPEC_BOOTSTRAP === '1' const recordDir = BOOTSTRAP ? RECORD_SRC : mkdtempSync(join(tmpdir(), 'acp-snap-record-suite-')) if (!BOOTSTRAP) cpSync(RECORD_SRC, recordDir, { recursive: true }) +const refreshDir = mkdtempSync(join(tmpdir(), 'acp-snap-refresh-suite-')) +cpSync(REPLAY_DIR, refreshDir, { recursive: true }) +staleRefreshFixtures(refreshDir) afterAll(async () => { if (!BOOTSTRAP) await rm(recordDir, { recursive: true, force: true }) + await rm(refreshDir, { recursive: true, force: true }) }) +function staleRefreshFixtures(dir: string): void { + writeFileSync(join(dir, 'plain-turn', 'stdout.golden.jsonl'), 'stale stdout\n') + + const plainBehaviorFile = join(dir, 'plain-turn', 'behavior.json') + const plainBehavior = JSON.parse(readFileSync(plainBehaviorFile, 'utf8')) as Record + plainBehavior.echoEnv = true + writeFileSync(plainBehaviorFile, `${JSON.stringify(plainBehavior, null, 2)}\n`) + + writeFileSync(join(dir, 'blocked-log', 'session.jsonl'), [ + '{"type":"session","id":"99999999-8888-4777-8666-555555555555","createdAt":13,"cwd":"/rec/blocked-cwd"}', + '{"type":"hook/result","seq":1,"time":13,"data":{"decision":"stale","durationMs":99}}', + '', + ].join('\n')) + writeFileSync(join(dir, 'authored-error', 'session.jsonl'), [ + '{"type":"session","id":"77777777-8888-4777-8666-555555555555","createdAt":13,"cwd":"/rec/error-cwd"}', + '{"type":"turn/end","seq":1,"time":9,"data":{"error":"stale"}}', + '', + ].join('\n')) +} + describe('defineAcpSnapshotSuite: replay mode', () => { defineAcpSnapshotSuite({ agent: AGENT, snapshotsDir: REPLAY_DIR, scenarios: REPLAY_SCENARIOS, mode: 'replay' }) }) @@ -75,6 +106,27 @@ describe('defineAcpSnapshotSuite: record mode', () => { defineAcpSnapshotSuite({ agent: AGENT, snapshotsDir: recordDir, scenarios: RECORD_SCENARIOS, mode: 'record' }) }) +describe('defineAcpSnapshotSuite: refresh mode', () => { + defineAcpSnapshotSuite({ agent: AGENT, snapshotsDir: refreshDir, scenarios: REPLAY_SCENARIOS, mode: 'refresh' }) +}) + +describe('defineAcpSnapshotSuite: refresh write-back', () => { + it('rewrites stdout and comparable logs from a replay-mode child run', () => { + const stdout = readFileSync(join(refreshDir, 'plain-turn', 'stdout.golden.jsonl'), 'utf8') + expect(stdout).not.toContain('stale stdout') + expect(stdout).toContain('env:{\\"mode\\":\\"replay\\"') + expect(stdout).not.toContain('\\"mode\\":\\"refresh\\"') + + const blocked = readFileSync(join(refreshDir, 'blocked-log', 'session.jsonl'), 'utf8') + expect(blocked).toContain('"decision":"block"') + expect(blocked).not.toContain('"decision":"stale"') + + const authored = readFileSync(join(refreshDir, 'authored-error', 'session.jsonl'), 'utf8') + expect(authored).toContain('"error":"model exploded"') + expect(authored).not.toContain('"error":"stale"') + }) +}) + describe('defineAcpSnapshotSuite: registration contract', () => { it("throws when a scenario's header class has no pinning scenario", () => { expect(() => { @@ -175,3 +227,56 @@ describe('headerDeltaCount', () => { expect(headerDeltaCount(`${other}\n`)).toBe(0) }) }) + +describe('refreshFixtureReplacements', () => { + it('maps fresh ids and cwd values to the existing fixture values, skipping non-replacements', () => { + const log = (content: string): HarvestedLog => ({ id: 'diagnostic', createdAt: 1, content }) + const logs = [ + log('{"type":"session","id":"","cwd":"/same"}\n'), + log('{"type":"session","id":"new-parent","cwd":"/new"}\n'), + log('{"type":"session","id":"new-child","cwd":"/new"}\n'), + ] + const fixtures = [ + '{"type":"session","id":"","cwd":"/same"}\n', + '{"type":"session","id":"old-parent","cwd":"/old"}\n', + ] + expect(refreshFixtureReplacements(logs, fixtures)).toEqual([ + { from: 'new-parent', to: 'old-parent' }, + { from: '/new', to: '/old' }, + ]) + }) +}) + +describe('stabilizeRefreshLog', () => { + it('keeps volatile fixture fields while preserving fresh meaningful payloads', () => { + const fresh = [ + '{"type":"session","id":"new-child","createdAt":200,"cwd":"/new","parentSession":"new-parent","seedLength":1}', + '{"type":"hook/result","seq":1,"time":22,"data":{"decision":"block","durationMs":37}}', + '{"type":"turn/end","seq":2,"time":33,"data":{"error":"fresh error"}}', + '{"type":"tool/result","seq":3,"time":44,"data":{"text":"new-parent in /new"}}', + '{"type":"hook/result","seq":4,"time":55,"data":{"decision":"allow","durationMs":5}}', + '', + ].join('\n') + const existing = [ + '{"type":"session","id":"old-child","createdAt":100,"cwd":"/old","parentSession":"old-parent","seedLength":5}', + '{"type":"hook/result","seq":1,"time":11,"data":{"decision":"stale","durationMs":99}}', + '{"type":"turn/end","seq":2,"data":{"error":"stale"}}', + '{"type":"assistant/message","seq":3,"time":12,"data":{"text":"different type"}}', + '{"type":"hook/result","seq":4,"time":13,"data":{"decision":"stale"}}', + '', + ].join('\n') + + expect(stabilizeRefreshLog(fresh, existing, [ + { from: 'new-parent', to: 'old-parent' }, + { from: 'new-child', to: 'old-child' }, + { from: '/new', to: '/old' }, + ])).toBe([ + '{"type":"session","id":"old-child","createdAt":100,"cwd":"/old","parentSession":"old-parent","seedLength":5}', + '{"type":"hook/result","seq":1,"time":11,"data":{"decision":"block","durationMs":99}}', + '{"type":"turn/end","seq":2,"time":33,"data":{"error":"fresh error"}}', + '{"type":"tool/result","seq":3,"time":44,"data":{"text":"old-parent in /old"}}', + '{"type":"hook/result","seq":4,"time":13,"data":{"decision":"allow","durationMs":5}}', + '', + ].join('\n')) + }) +}) diff --git a/vitest.snapshot.config.ts b/vitest.snapshot.config.ts index ecc8d911aa..d0a8ce6a54 100644 --- a/vitest.snapshot.config.ts +++ b/vitest.snapshot.config.ts @@ -7,10 +7,14 @@ import { defineConfig } from 'vitest/config' // normalized stdout transcript + re-persisted log against committed goldens. // `pnpm run test:snapshot:record` (DSH_SNAPSHOT=record + -u) re-records the // fixtures against the real API and refreshes the goldens. +// `pnpm run test:snapshot:refresh` (DSH_SNAPSHOT=refresh) stays keyless: it +// replays the committed model scripts and writes the current stdout/log goldens +// without calling the live LLM. // // Replay loads no .env (it must never reach the network — a recorded fixture -// drives the model). Record reads DEEPSEEK_API_KEY from the env or a gitignored -// repo-root .env, so a contributor with a key only in .env can still record. +// drives the model), and refresh uses that same keyless replay path. Record +// reads DEEPSEEK_API_KEY from the env or a gitignored repo-root .env, so a +// contributor with a key only in .env can still record. if (process.env.DSH_SNAPSHOT === 'record') { try { process.loadEnvFile(new URL('.env', import.meta.url).pathname) From 5f7177a0a983b0eb2733605026ee0d9bb4dcdb53 Mon Sep 17 00:00:00 2001 From: Tianyi Cui <53024+tianyicui@users.noreply.github.com> Date: Fri, 10 Jul 2026 00:55:47 +0800 Subject: [PATCH 2/4] Stabilize workflow worker coverage wait --- .../tests/workflow-workerthread.spec.ts | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/packages/workflow/workflow-workerthread/tests/workflow-workerthread.spec.ts b/packages/workflow/workflow-workerthread/tests/workflow-workerthread.spec.ts index ecaeed61f6..8d623e8a6f 100644 --- a/packages/workflow/workflow-workerthread/tests/workflow-workerthread.spec.ts +++ b/packages/workflow/workflow-workerthread/tests/workflow-workerthread.spec.ts @@ -17,6 +17,8 @@ function fakeParent(): Agent { /** The vm-context escape hatch, spelled once: real Worker tests use it to make the WORKER misbehave. */ const ESCAPE = "globalThis.constructor.constructor('return process')()" +/** Linux coverage workers can take longer than Vitest's default waitFor timeout to start executing script. */ +const WORKER_CHILD_START_TIMEOUT_MS = 3_000 /** One controllable child run: the test (or auto mode) settles it. */ interface ControlledRun { @@ -572,7 +574,7 @@ describe('dsh-workflow-workerthread', () => { `), parent: fakeParent(), }) - await vi.waitFor(() => { expect(starts).toBe(1) }) + await vi.waitFor(() => { expect(starts).toBe(1) }, { timeout: WORKER_CHILD_START_TIMEOUT_MS }) handle.cancel('stop now') await vi.waitFor(() => { expect(cancelled).toEqual(['stop now']) }, { timeout: 800 }) // The wedged worker's own completion loses to the in-flight cancel. @@ -602,7 +604,7 @@ describe('dsh-workflow-workerthread', () => { `), parent, }) - await vi.waitFor(() => { expect(provider.runs.length).toBe(1) }) + await vi.waitFor(() => { expect(provider.runs.length).toBe(1) }, { timeout: WORKER_CHILD_START_TIMEOUT_MS }) const before = Date.now() await handle.dispose() // Bounded by the grace (plus the terminate), never by the 1.5s spin. From af242dbae56df6b4092c3dd026e12a1682045256 Mon Sep 17 00:00:00 2001 From: Tianyi Cui <53024+tianyicui@users.noreply.github.com> Date: Fri, 10 Jul 2026 01:04:06 +0800 Subject: [PATCH 3/4] Preserve snapshot refresh seed boundaries --- packages/support/acp-snapshot/src/suite.ts | 2 +- packages/support/acp-snapshot/tests/suite.spec.ts | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/packages/support/acp-snapshot/src/suite.ts b/packages/support/acp-snapshot/src/suite.ts index f97cb82c59..27de939d3d 100644 --- a/packages/support/acp-snapshot/src/suite.ts +++ b/packages/support/acp-snapshot/src/suite.ts @@ -245,7 +245,7 @@ export function refreshFixtureReplacements(logs: HarvestedLog[], fixtures: strin function preserveFixtureVolatiles(record: Record, existing: Record | undefined): void { if (existing === undefined || existing.type !== record.type) return if (record.type === 'session') { - for (const field of ['id', 'createdAt', 'cwd', 'parentSession', 'seedLength'] as const) { + for (const field of ['id', 'createdAt', 'cwd', 'parentSession'] as const) { if (field in record && field in existing) record[field] = existing[field] } return diff --git a/packages/support/acp-snapshot/tests/suite.spec.ts b/packages/support/acp-snapshot/tests/suite.spec.ts index 8e79e367f2..ae01190c01 100644 --- a/packages/support/acp-snapshot/tests/suite.spec.ts +++ b/packages/support/acp-snapshot/tests/suite.spec.ts @@ -271,7 +271,7 @@ describe('stabilizeRefreshLog', () => { { from: 'new-child', to: 'old-child' }, { from: '/new', to: '/old' }, ])).toBe([ - '{"type":"session","id":"old-child","createdAt":100,"cwd":"/old","parentSession":"old-parent","seedLength":5}', + '{"type":"session","id":"old-child","createdAt":100,"cwd":"/old","parentSession":"old-parent","seedLength":1}', '{"type":"hook/result","seq":1,"time":11,"data":{"decision":"block","durationMs":99}}', '{"type":"turn/end","seq":2,"time":33,"data":{"error":"fresh error"}}', '{"type":"tool/result","seq":3,"time":44,"data":{"text":"old-parent in /old"}}', From 53f6a959e2f9721ac676a91bbda58dfd7121a1d9 Mon Sep 17 00:00:00 2001 From: kingwl Date: Fri, 10 Jul 2026 21:15:48 +0800 Subject: [PATCH 4/4] chore(lint): one shared project service; ignore harness-local .claude state MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two independent multipliers were pushing bare `pnpm run lint` past node's default heap: 1. parserOptions.project expanded to every package tsconfig plus the root one — each matched config built its OWN ts.Program, and the dev paths map pulls sibling package SOURCES (plus the vendored Cordis stack) into every such program, so resident memory grew as sum-of-closures, not repo size: ~4.6 GB peak for 425 repo files. projectService shares one tsserver-style graph: ~2.0 GB peak, ~28 s → ~14 s wall. 2. `eslint .` traversed .claude/ harness-local state — stale worktree checkouts there carry tens of thousands of additional .ts files (whole-repo copies), roughly tripling the work again even under the project service. Other checkouts are not this one's sources; ignore them like node_modules. (#169 carries the identical ignore line inside its chain; the hunks dedupe on its next rebase.) Type-aware rules verified live under the service: a floating-promise probe still trips no-floating-promises. --- eslint.config.mjs | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/eslint.config.mjs b/eslint.config.mjs index cbb696bccb..c62d3e9739 100644 --- a/eslint.config.mjs +++ b/eslint.config.mjs @@ -22,6 +22,7 @@ export default tseslint.config( '**/lib/**', '**/node_modules/**', '**/.sessions/**', + '.claude/**', // harness-local state (worktrees, skills) — other checkouts, not this one's sources '**/.doc-typecheck-*/**', 'vendor/**', // vendored source keeps upstream style and idioms '**/*.js', @@ -38,7 +39,14 @@ export default tseslint.config( ], languageOptions: { parserOptions: { - project: ['./packages/*/*/tsconfig.json', './tsconfig.json'], + // One shared tsserver-style project service instead of 60+ standalone + // per-package programs: the old `project` glob built every package's + // full dependency closure (sibling sources via the dev `paths` map + + // the vendored Cordis stack) as its own program and kept them all + // resident — ~5 GB peak, an OOM past node's default heap. The service + // resolves each file to its nearest owning tsconfig and shares the + // graph. + projectService: true, tsconfigRootDir: import.meta.dirname, }, }, @@ -89,7 +97,9 @@ export default tseslint.config( ], languageOptions: { parserOptions: { - project: ['./tsconfig.json'], + // Same shared project service as the src block: test files resolve + // through the root tsconfig (its include covers every tests/ tree). + projectService: true, tsconfigRootDir: import.meta.dirname, }, },