From 9d2cf8ce826ee82c2af13f25de12bdf1920dda7e Mon Sep 17 00:00:00 2001 From: Tianyi Cui <53024+tianyicui@users.noreply.github.com> Date: Fri, 10 Jul 2026 00:48:27 +0800 Subject: [PATCH 1/7] Add keyless snapshot refresh mode --- docs/testing.md | 8 +- examples/acp-agent/README.md | 2 +- examples/acp-agent/tests/acp.snapshot.ts | 28 +++- package.json | 1 + packages/support/acp-snapshot/README.md | 10 +- packages/support/acp-snapshot/src/suite.ts | 147 +++++++++++++++--- .../support/acp-snapshot/tests/suite.spec.ts | 117 +++++++++++++- vitest.snapshot.config.ts | 8 +- 8 files changed, 278 insertions(+), 43 deletions(-) diff --git a/docs/testing.md b/docs/testing.md index a4a77c9d0c..62989a1096 100644 --- a/docs/testing.md +++ b/docs/testing.md @@ -1,17 +1,17 @@ # Testing policy -How this repo tests, tier by tier, and the rules that keep a green suite meaning something. Commands live in the root [AGENTS.md](../AGENTS.md) § Commands; the RFCs linked per tier carry the design rationale. +How this repo tests, tier by tier, and the rules that keep a green suite meaningful. Commands live in root [AGENTS.md](../AGENTS.md); linked RFCs carry the rationale. ## Tiers -- **Unit** (`pnpm run test`): vitest over `packages|examples/*/tests/**/*.spec.ts`, colocated with what they test. Every registry gets an HMR-safety test (dispose the contributing fiber, assert cleanup). Excessive tests are welcome — err toward covering edge cases, error paths, event ordering, and concurrency races; review findings get regression tests (see `packages/core/agent-loop/tests/review-fixes.spec.ts`). +- **Unit** (`pnpm run test`): vitest over `packages|examples/*/tests/**/*.spec.ts`, colocated with what they test. Every registry gets an HMR-safety test (dispose the contributing fiber, assert cleanup). Prefer edge cases, error paths, event ordering, and concurrency races; review findings get regression tests (see `packages/core/agent-loop/tests/review-fixes.spec.ts`). - **Coverage gate** (`pnpm run test:coverage`): the gating run, per-file 100% on `packages/*/*/src`. An uncovered line is often dead code the gate is correctly flagging for deletion, not a missing test to bolt on. Line coverage is necessary, never sufficient — it proves lines ran, not that the feature works as shipped. - **Real-API e2e** (`pnpm run test:e2e`): with-key tests against live provider APIs — the DeepSeek model plus provider-specific smokes that gate on their own keys (`EXA_API_KEY`, `PERPLEXITY_API_KEY`, …); each suite self-skips without its key so keyless CI stays green ([real-API e2e RFC](rfc/implemented/testing/2026-06-19-real-api-e2e-ci.md)). -- **Snapshot** (`pnpm run test:snapshot`): boots the real example subprocess, replays a recorded session keyless, diffs normalized stdout + the re-persisted log against committed goldens ([snapshot RFC](rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md)). Re-record with `pnpm run test:snapshot:record`; reviewing the golden diff is part of the review. System-prompt/tool-schema content is pinned by ONE scenario (`text-turn`) and tokenized in every other fixture, so a prompt or schema edit churns one committed line ([pinned-header RFC](rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)). +- **Snapshot** (`pnpm run test:snapshot`): boots the real example subprocess, replays a recorded session keyless, diffs normalized stdout + the re-persisted log against committed goldens ([snapshot RFC](rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md)). Use `pnpm run test:snapshot:record` when the model transcript should change; use `pnpm run test:snapshot:refresh` when the committed transcript is still the right mock LLM input and replay goldens need keyless rewrite. Review the golden diff. System-prompt/tool-schema content is pinned by ONE scenario (`text-turn`) and tokenized in every other fixture, so a prompt or schema edit churns one committed line ([pinned-header RFC](rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)). ## The with-key policy: inference is cheap here -We are DeepSeek — do not ration real-API tests. A no-key test proves the plumbing; only a with-key run proves the agent works against a real model. Write many: real prompts that write files, multi-turn conversations, tool use, cancellation mid-stream. Cheapest and highest-value are **smoke tests** that boot the real example, send one real prompt, and check the world — they catch the "green unit tests, broken product" class that mocks structurally cannot ([postmortem 0001](postmortem/0001-acp-default-export-drops-inject.md)). The self-skip exists only so secretless CI and keyless contributors aren't blocked; it is not a cost signal. Every example ships a keyless smoke and — unless keyless-by-nature — a with-key smoke ([examples/AGENTS.md](../examples/AGENTS.md)). +We are DeepSeek — do not ration real-API tests. A no-key test proves plumbing; only a with-key run proves the agent works against a real model. Write many: file-writing prompts, multi-turn conversations, tool use, cancellation mid-stream. Highest-value are **smoke tests** that boot the real example, send one real prompt, and check the world — they catch the "green unit tests, broken product" class that mocks structurally cannot ([postmortem 0001](postmortem/0001-acp-default-export-drops-inject.md)). The self-skip exists only so secretless CI and keyless contributors aren't blocked; it is not a cost signal. Every example ships a keyless smoke and — unless keyless-by-nature — a with-key smoke ([examples/AGENTS.md](../examples/AGENTS.md)). ## Prefer the real implementation over a mock diff --git a/examples/acp-agent/README.md b/examples/acp-agent/README.md index 5b3c936651..85cb185c82 100644 --- a/examples/acp-agent/README.md +++ b/examples/acp-agent/README.md @@ -33,7 +33,7 @@ The editor sets each session's `cwd` to the project it opens; both the agent's b ## Snapshot tests (record-once / replay-deterministic) -This example is the home of the harness's **snapshot tests** — they boot this server as a real subprocess, drive it with a deterministic input script, and diff its normalized output against committed golden files. The model is made deterministic by `@deepseek-ai/dsh-llm-replay`, a function/namespace plugin that installs an `llm/stream` waterfall listener and short-circuits it, serving model streams reconstructed from a recorded **session JSONL** fixture (`/session.jsonl`) — so replay needs no API key. The fixture IS the persisted session log: its `assistant/chunk` events carry every `StreamChunk`, so grouping them by `(turn, step)` reconstructs each `stream()` call (one model call per loop step). Recording is therefore "run the real agent once and harvest the `.jsonl`". The two failure modes not expressible as logged chunks — a pure throw before any chunk, and cancel/hang — use an optional `/replay.override.json` sidecar (a `ReplayEntry[]` that replaces the derived script). A scenario that needs the agent to operate on existing files ships an optional `/workspace/` directory — the harness copies its contents into the temp cwd before the run (see `workspace-edit`). See [the ACP snapshot tests RFC](../../docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md) for the full design. +This example is the home of the harness's **snapshot tests** — they boot this server as a real subprocess, drive it with a deterministic input script, and diff its normalized output against committed golden files. The model is made deterministic by `@deepseek-ai/dsh-llm-replay`, a function/namespace plugin that installs an `llm/stream` waterfall listener and short-circuits it, serving model streams reconstructed from a recorded **session JSONL** fixture (`/session.jsonl`) — so replay needs no API key. The fixture IS the persisted session log: its `assistant/chunk` events carry every `StreamChunk`, so grouping them by `(turn, step)` reconstructs each `stream()` call (one model call per loop step). Recording is therefore "run the real agent once and harvest the `.jsonl`"; use `pnpm run test:snapshot:record` when the model transcript itself should change, and `pnpm run test:snapshot:refresh` when the committed model transcript is still the right mock input and only the current replay output/goldens need to be rewritten. The two failure modes not expressible as logged chunks — a pure throw before any chunk, and cancel/hang — use an optional `/replay.override.json` sidecar (a `ReplayEntry[]` that replaces the derived script). A scenario that needs the agent to operate on existing files ships an optional `/workspace/` directory — the harness copies its contents into the temp cwd before the run (see `workspace-edit`). See [the ACP snapshot tests RFC](../../docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md) for the full design. ## MVP limitations diff --git a/examples/acp-agent/tests/acp.snapshot.ts b/examples/acp-agent/tests/acp.snapshot.ts index d45ddf8591..7102d8ce88 100644 --- a/examples/acp-agent/tests/acp.snapshot.ts +++ b/examples/acp-agent/tests/acp.snapshot.ts @@ -1,15 +1,16 @@ import { fileURLToPath } from 'node:url' import { dirname, join } from 'node:path' -import { defineAcpSnapshotSuite, type Scenario } from '@deepseek-ai/dsh-acp-snapshot' +import { defineAcpSnapshotSuite, type Scenario, type SnapshotSuiteOptions } from '@deepseek-ai/dsh-acp-snapshot' /** * The acp-agent example's snapshot suite: the scenario table for * `dsh-acp-snapshot`'s suite factory, which owns every compare/guard mechanic - * (golden + re-persisted-log diffs, record write-back, the pinned-header + * (golden + re-persisted-log diffs, record/refresh write-back, the pinned-header * uniformity guard, the fixture guards). Fixtures live under `snapshots//`; - * `pnpm run test:snapshot:record` re-records the `recorded` scenarios against - * the real API. See the package README (packages/support/acp-snapshot) and the - * snapshot RFC, docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md. + * `pnpm run test:snapshot:record` re-records model transcripts against the real + * API; `pnpm run test:snapshot:refresh` rewrites current replay goldens keyless. + * See the package README (packages/support/acp-snapshot) and the snapshot RFC, + * docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md. */ // The dsh-acp-agent bin (the demo:acp entry), this example's cordis.yml, and @@ -26,6 +27,21 @@ const AGENT = { const CODE_MODE_CONFIG = fileURLToPath(new URL('../code-mode.cordis.yml', import.meta.url)) const BOTH_MODE_CONFIG = fileURLToPath(new URL('../both-mode.cordis.yml', import.meta.url)) +function snapshotModeFromEnv(value: string | undefined): SnapshotSuiteOptions['mode'] { + switch (value) { + case undefined: + case '': + case 'replay': + return 'replay' + case 'record': + return 'record' + case 'refresh': + return 'refresh' + default: + throw new Error(`unknown DSH_SNAPSHOT mode: ${value}`) + } +} + const SCENARIOS: Scenario[] = [ { name: 'handshake', hasModelTurn: false, recorded: false }, { name: 'reject-extra-dirs', hasModelTurn: false, recorded: false }, @@ -111,5 +127,5 @@ defineAcpSnapshotSuite({ agent: AGENT, snapshotsDir: join(dirname(fileURLToPath(import.meta.url)), 'snapshots'), scenarios: SCENARIOS, - mode: process.env.DSH_SNAPSHOT === 'record' ? 'record' : 'replay', + mode: snapshotModeFromEnv(process.env.DSH_SNAPSHOT), }) diff --git a/package.json b/package.json index c4cc307893..7f068c7f19 100644 --- a/package.json +++ b/package.json @@ -22,6 +22,7 @@ "test:e2e": "vitest run --config vitest.e2e.config.ts", "test:snapshot": "vitest run --config vitest.snapshot.config.ts", "test:snapshot:record": "DSH_SNAPSHOT=record vitest run --config vitest.snapshot.config.ts --update", + "test:snapshot:refresh": "DSH_SNAPSHOT=refresh vitest run --config vitest.snapshot.config.ts", "check:ci": "tsx scripts/run-gates.ts ci-primary", "check:ci:static": "tsx scripts/run-gates.ts ci-static", "check:ci:lint": "tsx scripts/run-gates.ts ci-lint", diff --git a/packages/support/acp-snapshot/README.md b/packages/support/acp-snapshot/README.md index 209b1a81cb..8ff2fe1413 100644 --- a/packages/support/acp-snapshot/README.md +++ b/packages/support/acp-snapshot/README.md @@ -6,7 +6,7 @@ Three layers, importable separately: - **`runScenario` (harness)** — boots the real agent bin as a subprocess via tsx (unbuilt, Loader path), drives it over ACP JSON-RPC stdio from a deterministic `input.json` script, tees raw stdout for the golden + purity check, and harvests every persisted session JSONL (parent + subagent children, primary-first) after a graceful stdin-EOF shutdown. Parameterized by `AgentUnderTest` (`binScript`, `configPath`, `tsconfigPath` — absolute paths; the subprocess cwd is a temp dir outside the repo). - **Normalizers** — pure functions turning the two captured surfaces into stable text: `normalizeStdout` (JSON-RPC ids → first-seen sequence; UUIDs/cwd → tokens; doubles as the stdout-purity check), `normalizeSessionLog` (times zeroed, `seq` kept), and the composable `scrubRequestHeaders` (header bulk → `{{system}}`/`{{tools}}`, structure kept — [pinned-header RFC](../../../docs/rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)). -- **`defineAcpSnapshotSuite` (factory)** — registers the whole describe/it tree for a scenario table: per-scenario golden + re-persisted-log compares, record-mode fixture write-back, the per-header-class pin with its live uniformity guard, and the fixture guard block (no orphan scenario dirs, required files present, exactly one pin per class, pinning fixtures well-formed, non-pinning fixtures header-scrubbed). Must be called at vitest collection time. +- **`defineAcpSnapshotSuite` (factory)** — registers the whole describe/it tree for a scenario table: per-scenario golden + re-persisted-log compares, record/refresh fixture write-back, the per-header-class pin with its live uniformity guard, and the fixture guard block (no orphan scenario dirs, required files present, exactly one pin per class, pinning fixtures well-formed, non-pinning fixtures header-scrubbed). Must be called at vitest collection time. A consuming `*.snapshot.ts` is the scenario table plus one factory call: @@ -27,12 +27,16 @@ defineAcpSnapshotSuite({ }, snapshotsDir: join(dirname(fileURLToPath(import.meta.url)), 'snapshots'), scenarios: SCENARIOS, // exactly one entry per header class sets pinsHeader - mode: process.env.DSH_SNAPSHOT === 'record' ? 'record' : 'replay', + mode: process.env.DSH_SNAPSHOT === 'record' + ? 'record' + : process.env.DSH_SNAPSHOT === 'refresh' + ? 'refresh' + : 'replay', }) ``` A scenario booting a differently-composed tree sets its own `configPath` (an overlay whose basename still ends in `cordis.yml`, so the bin's replay swap finds the sibling `*cordis.snapshot.yml`) and, when that composition changes the request header, its own `headerClass` with its own pinning scenario — the acp-agent example's Code Mode scenarios are the template. -The example also ships a `cordis.snapshot.yml` replay overlay next to its `cordis.yml` (the bin swaps them under `DSH_SNAPSHOT=replay` — [single-source replay config RFC](../../../docs/rfc/implemented/testing/2026-07-04-single-source-acp-replay-config.md)); replay fixtures are served by [`dsh-llm-replay`](../llm-replay/README.md), which this package points at via the `DSH_SNAPSHOT_*` env vars it sets on the child. Fixture roles, record/replay semantics, and scenario-table fields are documented on `Scenario` and in the [snapshot RFC](../../../docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md). +The example also ships a `cordis.snapshot.yml` replay overlay next to its `cordis.yml` (the bin swaps them under `DSH_SNAPSHOT=replay` — [single-source replay config RFC](../../../docs/rfc/implemented/testing/2026-07-04-single-source-acp-replay-config.md)); replay fixtures are served by [`dsh-llm-replay`](../llm-replay/README.md), which this package points at via the `DSH_SNAPSHOT_*` env vars it sets on the child. `pnpm run test:snapshot:record` calls the live LLM and rewrites the recorded scenarios' model fixtures; `pnpm run test:snapshot:refresh` stays keyless, runs the replay overlay, and rewrites stdout plus comparable session-log goldens from the committed model scripts. Fixture roles, record/replay/refresh semantics, and scenario-table fields are documented on `Scenario` and in the [snapshot RFC](../../../docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md). Constraints: `suite.ts` imports vitest, so the package is importable only inside a vitest run (the harness and normalizers have no such dependency but ship from the same entry). ACP-specific by design — the harness speaks the SDK's `ClientSideConnection`. Permission round-trips are scriptable: `InputScript.permissionAnswers` is a FIFO queue of option-kind selections (`allow_once`, `reject_once`, …) the client maps to the agent-issued `optionId` at answer time; an absent or exhausted queue answers `cancelled`, and a kind the request never offered rejects the run (the agent is answered `cancelled`, so a tolerant agent cannot absorb the scenario bug). diff --git a/packages/support/acp-snapshot/src/suite.ts b/packages/support/acp-snapshot/src/suite.ts index 53e05d7876..f97cb82c59 100644 --- a/packages/support/acp-snapshot/src/suite.ts +++ b/packages/support/acp-snapshot/src/suite.ts @@ -23,8 +23,11 @@ * * `pnpm run test:snapshot:record` (DSH_SNAPSHOT=record + -u) re-records the * `session.jsonl` fixtures against the real API and refreshes the stdout golden - * in one pass; the caller resolves that env into {@link SnapshotSuiteOptions} - * (env reading stays at the suite edge, not in this library). + * in one pass. `pnpm run test:snapshot:refresh` (DSH_SNAPSHOT=refresh) instead + * replays the committed model scripts keylessly and writes the current stdout + * + persisted-log goldens back without calling a live LLM. The caller resolves + * that env into {@link SnapshotSuiteOptions} (env reading stays at the suite + * edge, not in this library). * * @module @deepseek-ai/dsh-acp-snapshot/suite */ @@ -124,12 +127,13 @@ export interface SnapshotSuiteOptions { /** The scenario table; exactly one entry must set `pinsHeader`. */ scenarios: Scenario[] /** - * `replay` (keyless, the default tier) or `record` (live API; re-records the - * `recorded` scenarios' fixtures and refreshes the vitest goldens under - * `--update`). The caller derives this from `$DSH_SNAPSHOT` — env reading - * stays outside this library. + * `replay` (keyless, the default tier), `record` (live API; re-records the + * `recorded` scenarios' fixtures and refreshes the Vitest goldens under + * `--update`), or `refresh` (keyless replay that rewrites stdout goldens and + * comparable session fixtures from the replay run). The caller derives this + * from `$DSH_SNAPSHOT` — env reading stays outside this library. */ - mode: 'replay' | 'record' + mode: 'replay' | 'record' | 'refresh' } /** @@ -201,6 +205,86 @@ export function headerDeltaCount(rawLog: string): number { .length } +/** A literal string replacement used to carry an existing fixture's volatile value into a refreshed log. */ +export interface FixtureReplacement { + /** The fresh replay-run value to replace. */ + from: string + /** The existing fixture value to keep. */ + to: string +} + +function parseJsonlRecords(text: string): Record[] { + return text.split('\n') + .filter(line => line.trim().length > 0) + .map(line => JSON.parse(line) as Record) +} + +/** + * Build the cross-log id/cwd replacements used by refresh write-back. + * + * @param logs The freshly harvested logs, in fixture order. + * @param fixtures The existing fixture contents, in matching order. + * @returns Literal replacements from fresh volatile values to the fixture's old values. + */ +export function refreshFixtureReplacements(logs: HarvestedLog[], fixtures: string[]): FixtureReplacement[] { + const replacements: FixtureReplacement[] = [] + for (let i = 0; i < logs.length; i++) { + const fresh = parseJsonlRecords((logs[i] as HarvestedLog).content)[0] + const existing = parseJsonlRecords(fixtures[i] ?? '')[0] + for (const field of ['id', 'cwd'] as const) { + const from = fresh?.[field] + const to = existing?.[field] + if (typeof from === 'string' && typeof to === 'string' && from.length > 0 && from !== to) { + replacements.push({ from, to }) + } + } + } + return replacements +} + +function preserveFixtureVolatiles(record: Record, existing: Record | undefined): void { + if (existing === undefined || existing.type !== record.type) return + if (record.type === 'session') { + for (const field of ['id', 'createdAt', 'cwd', 'parentSession', 'seedLength'] as const) { + if (field in record && field in existing) record[field] = existing[field] + } + return + } + if ('time' in record && 'time' in existing) record.time = existing.time + if (record.type !== 'hook/result') return + const data = record.data + const existingData = existing.data + if ( + data !== null && typeof data === 'object' + && existingData !== null && typeof existingData === 'object' + && 'durationMs' in data && 'durationMs' in existingData + ) { + (data as Record).durationMs = (existingData as Record).durationMs + } +} + +/** + * Rewrite a fresh replay-produced log so repeated refreshes do not churn + * volatile fixture fields. Meaningful event payloads come from `fresh`; the + * existing fixture lends session ids, cwd, creation times, event times, and + * hook durations where the record shape still matches. + * + * @param fresh The newly harvested session JSONL. + * @param existing The committed fixture JSONL being refreshed. + * @param replacements Cross-log literal replacements from {@link refreshFixtureReplacements}. + * @returns The stabilized JSONL content to write back. + */ +export function stabilizeRefreshLog(fresh: string, existing: string, replacements: FixtureReplacement[]): string { + let stable = fresh + for (const { from, to } of replacements) stable = stable.split(from).join(to) + const existingRecords = parseJsonlRecords(existing) + const records = parseJsonlRecords(stable) + for (let i = 0; i < records.length; i++) { + preserveFixtureVolatiles(records[i] as Record, existingRecords[i]) + } + return records.map(record => JSON.stringify(record)).join('\n') + '\n' +} + /** * Register the suite: one `describe` per scenario (the golden/log compares and * the header-uniformity guard) plus the fixture guard block (no orphan @@ -215,6 +299,8 @@ export function headerDeltaCount(rawLog: string): number { export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void { const { agent, snapshotsDir, scenarios, mode } = options const RECORDING = mode === 'record' + const REFRESHING = mode === 'refresh' + const childMode: 'replay' | 'record' = RECORDING ? 'record' : 'replay' /** The class a scenario's header composition belongs to (see {@link Scenario.headerClass}). */ const classOf = (scenario: Scenario): string => scenario.headerClass ?? 'default' @@ -238,15 +324,18 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void { describe(`snapshot: ${scenario.name}`, () => { // In RECORD mode, only re-run the `recorded` (live-API) scenarios; the // `authored` ones (sidecar-driven errors/cancel) are never re-recorded. + // REFRESH mode is replay-backed and deterministic, so it runs every + // scenario and rewrites the comparable fixtures from that replay run. it.skipIf(RECORDING && !scenario.recorded)('matches the goldens', async () => { const dir = join(snapshotsDir, scenario.name) const input = JSON.parse(await readFile(join(dir, 'input.json'), 'utf8')) as InputScript const overrideFile = join(dir, 'replay.override.json') const workspaceDir = join(dir, 'workspace') const childSessions = scenario.childSessions ?? 0 + const comparesLog = scenario.comparesLog ?? scenario.hasModelTurn const result = await runScenario(input, { agent, - mode, + mode: childMode, fixtureFile: join(dir, 'session.jsonl'), ...existsSync(overrideFile) ? { overrideFile } : {}, // In REPLAY, forward the recorded child fixtures so each subagent session @@ -271,30 +360,47 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void { } // RECORD mode (recorded model scenarios only): persist the freshly-harvested - // logs back to their fixtures — the primary to session.jsonl, each child to - // session..jsonl in harvest order. `--update` refreshes the Vitest - // goldens but NOT these fixtures, so write them here. A non-pinning - // scenario's fixtures are written header-scrubbed, so a re-record can - // never smuggle the full prompt/schema content back into every fixture. + // live logs back to their fixtures. REFRESH mode does the same from a + // keyless replay run for every comparable log, including authored + // scenarios that live record deliberately skips. The primary goes to + // session.jsonl, each child to session..jsonl in harvest order. A + // non-pinning scenario's fixtures are written header-scrubbed, so a + // re-record/refresh can never smuggle the full prompt/schema content + // back into every fixture. const scrub = scenario.pinsHeader === true ? (log: string): string => log : scrubRequestHeaders - if (RECORDING && scenario.recorded && scenario.hasModelTurn) { - expect(result.sessionLogs.length, 'record produced no session log to harvest').toBeGreaterThan(0) + const fixtureFiles = ['session.jsonl', ...Array.from({ length: childSessions }, (_, i) => `session.${i + 1}.jsonl`)] + const existingFixtures = REFRESHING + ? await Promise.all(fixtureFiles.map(file => readFile(join(dir, file), 'utf8'))) + : [] + const replacements = REFRESHING ? refreshFixtureReplacements(result.sessionLogs, existingFixtures) : [] + const writesSessionFixtures = (RECORDING && scenario.recorded && scenario.hasModelTurn) + || (REFRESHING && comparesLog) + if (writesSessionFixtures) { + expect(result.sessionLogs.length, `${mode} produced no session log to harvest`).toBeGreaterThan(0) expect(result.sessionLogs.length, `expected ${childSessions + 1} session logs (parent + children)`) .toBe(childSessions + 1) - await writeFile(join(dir, 'session.jsonl'), scrub((result.sessionLogs[0] as HarvestedLog).content)) + const primary = (result.sessionLogs[0] as HarvestedLog).content + await writeFile(join(dir, 'session.jsonl'), scrub( + REFRESHING ? stabilizeRefreshLog(primary, existingFixtures[0] as string, replacements) : primary, + )) for (let i = 1; i < result.sessionLogs.length; i++) { - await writeFile(join(dir, `session.${i}.jsonl`), scrub((result.sessionLogs[i] as HarvestedLog).content)) + const child = (result.sessionLogs[i] as HarvestedLog).content + await writeFile(join(dir, `session.${i}.jsonl`), scrub( + REFRESHING ? stabilizeRefreshLog(child, existingFixtures[i] as string, replacements) : child, + )) } } - await expect(normalizeStdout(result.rawStdout, ctx)) - .toMatchFileSnapshot(join(dir, 'stdout.golden.jsonl')) + const stdout = normalizeStdout(result.rawStdout, ctx) + if (REFRESHING) { + await writeFile(join(dir, 'stdout.golden.jsonl'), stdout) + } + await expect(stdout).toMatchFileSnapshot(join(dir, 'stdout.golden.jsonl')) // A model turn always produces a log worth comparing; a hook scenario can // produce one without a model turn (a `rejected` turn carrying `hook/*`). - const comparesLog = scenario.comparesLog ?? scenario.hasModelTurn if (comparesLog) { // The harvested logs (primary-first) must match their committed fixtures // 1:1. Each side passes through normalizeSessionLog, scrubbed against ITS @@ -307,7 +413,6 @@ export function defineAcpSnapshotSuite(options: SnapshotSuiteOptions): void { // reason, and config, but not its bulk content (pinned once, in the // `pinsHeader` scenario). expect(result.sessionLogs.length, 'this scenario must persist a session log').toBe(childSessions + 1) - const fixtureFiles = ['session.jsonl', ...Array.from({ length: childSessions }, (_, i) => `session.${i + 1}.jsonl`)] for (let i = 0; i < fixtureFiles.length; i++) { const harvested = scrub((result.sessionLogs[i] as HarvestedLog).content) const fixture = scrub(await readFile(join(dir, fixtureFiles[i] as string), 'utf8')) diff --git a/packages/support/acp-snapshot/tests/suite.spec.ts b/packages/support/acp-snapshot/tests/suite.spec.ts index 0f47010a19..8e79e367f2 100644 --- a/packages/support/acp-snapshot/tests/suite.spec.ts +++ b/packages/support/acp-snapshot/tests/suite.spec.ts @@ -1,11 +1,18 @@ -import { cpSync, mkdtempSync } from 'node:fs' +import { cpSync, mkdtempSync, readFileSync, writeFileSync } from 'node:fs' import { rm } from 'node:fs/promises' import { tmpdir } from 'node:os' import { join } from 'node:path' import { fileURLToPath } from 'node:url' import { afterAll, describe, expect, it } from 'vitest' -import { defineAcpSnapshotSuite, type Scenario } from '../src/index.ts' -import { childFixturePaths, fixtureContext, headerDeltaCount, normalizedHeaders } from '../src/suite.ts' +import { defineAcpSnapshotSuite, type HarvestedLog, type Scenario } from '../src/index.ts' +import { + childFixturePaths, + fixtureContext, + headerDeltaCount, + normalizedHeaders, + refreshFixtureReplacements, + stabilizeRefreshLog, +} from '../src/suite.ts' /** * Unit tests for the suite factory, by running it: two synthetic suites over @@ -55,16 +62,40 @@ const RECORD_SCENARIOS: Scenario[] = [ { name: 'rec-skip', hasModelTurn: true, recorded: false, overridden: true }, ] -// Record mode mutates its snapshots dir, so run it on a throwaway copy — -// except under the documented bootstrap knob, which regenerates the committed -// fixtures/goldens in place. +// Record/refresh modes mutate their snapshots dir, so run them on throwaway +// copies — except record's documented bootstrap knob, which regenerates the +// committed record fixtures/goldens in place. const BOOTSTRAP = process.env.ACP_SNAPSHOT_SPEC_BOOTSTRAP === '1' const recordDir = BOOTSTRAP ? RECORD_SRC : mkdtempSync(join(tmpdir(), 'acp-snap-record-suite-')) if (!BOOTSTRAP) cpSync(RECORD_SRC, recordDir, { recursive: true }) +const refreshDir = mkdtempSync(join(tmpdir(), 'acp-snap-refresh-suite-')) +cpSync(REPLAY_DIR, refreshDir, { recursive: true }) +staleRefreshFixtures(refreshDir) afterAll(async () => { if (!BOOTSTRAP) await rm(recordDir, { recursive: true, force: true }) + await rm(refreshDir, { recursive: true, force: true }) }) +function staleRefreshFixtures(dir: string): void { + writeFileSync(join(dir, 'plain-turn', 'stdout.golden.jsonl'), 'stale stdout\n') + + const plainBehaviorFile = join(dir, 'plain-turn', 'behavior.json') + const plainBehavior = JSON.parse(readFileSync(plainBehaviorFile, 'utf8')) as Record + plainBehavior.echoEnv = true + writeFileSync(plainBehaviorFile, `${JSON.stringify(plainBehavior, null, 2)}\n`) + + writeFileSync(join(dir, 'blocked-log', 'session.jsonl'), [ + '{"type":"session","id":"99999999-8888-4777-8666-555555555555","createdAt":13,"cwd":"/rec/blocked-cwd"}', + '{"type":"hook/result","seq":1,"time":13,"data":{"decision":"stale","durationMs":99}}', + '', + ].join('\n')) + writeFileSync(join(dir, 'authored-error', 'session.jsonl'), [ + '{"type":"session","id":"77777777-8888-4777-8666-555555555555","createdAt":13,"cwd":"/rec/error-cwd"}', + '{"type":"turn/end","seq":1,"time":9,"data":{"error":"stale"}}', + '', + ].join('\n')) +} + describe('defineAcpSnapshotSuite: replay mode', () => { defineAcpSnapshotSuite({ agent: AGENT, snapshotsDir: REPLAY_DIR, scenarios: REPLAY_SCENARIOS, mode: 'replay' }) }) @@ -75,6 +106,27 @@ describe('defineAcpSnapshotSuite: record mode', () => { defineAcpSnapshotSuite({ agent: AGENT, snapshotsDir: recordDir, scenarios: RECORD_SCENARIOS, mode: 'record' }) }) +describe('defineAcpSnapshotSuite: refresh mode', () => { + defineAcpSnapshotSuite({ agent: AGENT, snapshotsDir: refreshDir, scenarios: REPLAY_SCENARIOS, mode: 'refresh' }) +}) + +describe('defineAcpSnapshotSuite: refresh write-back', () => { + it('rewrites stdout and comparable logs from a replay-mode child run', () => { + const stdout = readFileSync(join(refreshDir, 'plain-turn', 'stdout.golden.jsonl'), 'utf8') + expect(stdout).not.toContain('stale stdout') + expect(stdout).toContain('env:{\\"mode\\":\\"replay\\"') + expect(stdout).not.toContain('\\"mode\\":\\"refresh\\"') + + const blocked = readFileSync(join(refreshDir, 'blocked-log', 'session.jsonl'), 'utf8') + expect(blocked).toContain('"decision":"block"') + expect(blocked).not.toContain('"decision":"stale"') + + const authored = readFileSync(join(refreshDir, 'authored-error', 'session.jsonl'), 'utf8') + expect(authored).toContain('"error":"model exploded"') + expect(authored).not.toContain('"error":"stale"') + }) +}) + describe('defineAcpSnapshotSuite: registration contract', () => { it("throws when a scenario's header class has no pinning scenario", () => { expect(() => { @@ -175,3 +227,56 @@ describe('headerDeltaCount', () => { expect(headerDeltaCount(`${other}\n`)).toBe(0) }) }) + +describe('refreshFixtureReplacements', () => { + it('maps fresh ids and cwd values to the existing fixture values, skipping non-replacements', () => { + const log = (content: string): HarvestedLog => ({ id: 'diagnostic', createdAt: 1, content }) + const logs = [ + log('{"type":"session","id":"","cwd":"/same"}\n'), + log('{"type":"session","id":"new-parent","cwd":"/new"}\n'), + log('{"type":"session","id":"new-child","cwd":"/new"}\n'), + ] + const fixtures = [ + '{"type":"session","id":"","cwd":"/same"}\n', + '{"type":"session","id":"old-parent","cwd":"/old"}\n', + ] + expect(refreshFixtureReplacements(logs, fixtures)).toEqual([ + { from: 'new-parent', to: 'old-parent' }, + { from: '/new', to: '/old' }, + ]) + }) +}) + +describe('stabilizeRefreshLog', () => { + it('keeps volatile fixture fields while preserving fresh meaningful payloads', () => { + const fresh = [ + '{"type":"session","id":"new-child","createdAt":200,"cwd":"/new","parentSession":"new-parent","seedLength":1}', + '{"type":"hook/result","seq":1,"time":22,"data":{"decision":"block","durationMs":37}}', + '{"type":"turn/end","seq":2,"time":33,"data":{"error":"fresh error"}}', + '{"type":"tool/result","seq":3,"time":44,"data":{"text":"new-parent in /new"}}', + '{"type":"hook/result","seq":4,"time":55,"data":{"decision":"allow","durationMs":5}}', + '', + ].join('\n') + const existing = [ + '{"type":"session","id":"old-child","createdAt":100,"cwd":"/old","parentSession":"old-parent","seedLength":5}', + '{"type":"hook/result","seq":1,"time":11,"data":{"decision":"stale","durationMs":99}}', + '{"type":"turn/end","seq":2,"data":{"error":"stale"}}', + '{"type":"assistant/message","seq":3,"time":12,"data":{"text":"different type"}}', + '{"type":"hook/result","seq":4,"time":13,"data":{"decision":"stale"}}', + '', + ].join('\n') + + expect(stabilizeRefreshLog(fresh, existing, [ + { from: 'new-parent', to: 'old-parent' }, + { from: 'new-child', to: 'old-child' }, + { from: '/new', to: '/old' }, + ])).toBe([ + '{"type":"session","id":"old-child","createdAt":100,"cwd":"/old","parentSession":"old-parent","seedLength":5}', + '{"type":"hook/result","seq":1,"time":11,"data":{"decision":"block","durationMs":99}}', + '{"type":"turn/end","seq":2,"time":33,"data":{"error":"fresh error"}}', + '{"type":"tool/result","seq":3,"time":44,"data":{"text":"old-parent in /old"}}', + '{"type":"hook/result","seq":4,"time":13,"data":{"decision":"allow","durationMs":5}}', + '', + ].join('\n')) + }) +}) diff --git a/vitest.snapshot.config.ts b/vitest.snapshot.config.ts index ecc8d911aa..d0a8ce6a54 100644 --- a/vitest.snapshot.config.ts +++ b/vitest.snapshot.config.ts @@ -7,10 +7,14 @@ import { defineConfig } from 'vitest/config' // normalized stdout transcript + re-persisted log against committed goldens. // `pnpm run test:snapshot:record` (DSH_SNAPSHOT=record + -u) re-records the // fixtures against the real API and refreshes the goldens. +// `pnpm run test:snapshot:refresh` (DSH_SNAPSHOT=refresh) stays keyless: it +// replays the committed model scripts and writes the current stdout/log goldens +// without calling the live LLM. // // Replay loads no .env (it must never reach the network — a recorded fixture -// drives the model). Record reads DEEPSEEK_API_KEY from the env or a gitignored -// repo-root .env, so a contributor with a key only in .env can still record. +// drives the model), and refresh uses that same keyless replay path. Record +// reads DEEPSEEK_API_KEY from the env or a gitignored repo-root .env, so a +// contributor with a key only in .env can still record. if (process.env.DSH_SNAPSHOT === 'record') { try { process.loadEnvFile(new URL('.env', import.meta.url).pathname) From 5f7177a0a983b0eb2733605026ee0d9bb4dcdb53 Mon Sep 17 00:00:00 2001 From: Tianyi Cui <53024+tianyicui@users.noreply.github.com> Date: Fri, 10 Jul 2026 00:55:47 +0800 Subject: [PATCH 2/7] Stabilize workflow worker coverage wait --- .../tests/workflow-workerthread.spec.ts | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/packages/workflow/workflow-workerthread/tests/workflow-workerthread.spec.ts b/packages/workflow/workflow-workerthread/tests/workflow-workerthread.spec.ts index ecaeed61f6..8d623e8a6f 100644 --- a/packages/workflow/workflow-workerthread/tests/workflow-workerthread.spec.ts +++ b/packages/workflow/workflow-workerthread/tests/workflow-workerthread.spec.ts @@ -17,6 +17,8 @@ function fakeParent(): Agent { /** The vm-context escape hatch, spelled once: real Worker tests use it to make the WORKER misbehave. */ const ESCAPE = "globalThis.constructor.constructor('return process')()" +/** Linux coverage workers can take longer than Vitest's default waitFor timeout to start executing script. */ +const WORKER_CHILD_START_TIMEOUT_MS = 3_000 /** One controllable child run: the test (or auto mode) settles it. */ interface ControlledRun { @@ -572,7 +574,7 @@ describe('dsh-workflow-workerthread', () => { `), parent: fakeParent(), }) - await vi.waitFor(() => { expect(starts).toBe(1) }) + await vi.waitFor(() => { expect(starts).toBe(1) }, { timeout: WORKER_CHILD_START_TIMEOUT_MS }) handle.cancel('stop now') await vi.waitFor(() => { expect(cancelled).toEqual(['stop now']) }, { timeout: 800 }) // The wedged worker's own completion loses to the in-flight cancel. @@ -602,7 +604,7 @@ describe('dsh-workflow-workerthread', () => { `), parent, }) - await vi.waitFor(() => { expect(provider.runs.length).toBe(1) }) + await vi.waitFor(() => { expect(provider.runs.length).toBe(1) }, { timeout: WORKER_CHILD_START_TIMEOUT_MS }) const before = Date.now() await handle.dispose() // Bounded by the grace (plus the terminate), never by the 1.5s spin. From af242dbae56df6b4092c3dd026e12a1682045256 Mon Sep 17 00:00:00 2001 From: Tianyi Cui <53024+tianyicui@users.noreply.github.com> Date: Fri, 10 Jul 2026 01:04:06 +0800 Subject: [PATCH 3/7] Preserve snapshot refresh seed boundaries --- packages/support/acp-snapshot/src/suite.ts | 2 +- packages/support/acp-snapshot/tests/suite.spec.ts | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/packages/support/acp-snapshot/src/suite.ts b/packages/support/acp-snapshot/src/suite.ts index f97cb82c59..27de939d3d 100644 --- a/packages/support/acp-snapshot/src/suite.ts +++ b/packages/support/acp-snapshot/src/suite.ts @@ -245,7 +245,7 @@ export function refreshFixtureReplacements(logs: HarvestedLog[], fixtures: strin function preserveFixtureVolatiles(record: Record, existing: Record | undefined): void { if (existing === undefined || existing.type !== record.type) return if (record.type === 'session') { - for (const field of ['id', 'createdAt', 'cwd', 'parentSession', 'seedLength'] as const) { + for (const field of ['id', 'createdAt', 'cwd', 'parentSession'] as const) { if (field in record && field in existing) record[field] = existing[field] } return diff --git a/packages/support/acp-snapshot/tests/suite.spec.ts b/packages/support/acp-snapshot/tests/suite.spec.ts index 8e79e367f2..ae01190c01 100644 --- a/packages/support/acp-snapshot/tests/suite.spec.ts +++ b/packages/support/acp-snapshot/tests/suite.spec.ts @@ -271,7 +271,7 @@ describe('stabilizeRefreshLog', () => { { from: 'new-child', to: 'old-child' }, { from: '/new', to: '/old' }, ])).toBe([ - '{"type":"session","id":"old-child","createdAt":100,"cwd":"/old","parentSession":"old-parent","seedLength":5}', + '{"type":"session","id":"old-child","createdAt":100,"cwd":"/old","parentSession":"old-parent","seedLength":1}', '{"type":"hook/result","seq":1,"time":11,"data":{"decision":"block","durationMs":99}}', '{"type":"turn/end","seq":2,"time":33,"data":{"error":"fresh error"}}', '{"type":"tool/result","seq":3,"time":44,"data":{"text":"old-parent in /old"}}', From 62627d7625deee296b564598c3463634c71b4623 Mon Sep 17 00:00:00 2001 From: pku-xht Date: Fri, 10 Jul 2026 16:42:20 +0800 Subject: [PATCH 4/7] fix(subagent-acp): contain onError sink exceptions to keep result from rejecting MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit spec.onError is a caller-supplied callback boundary, but the flattening catch invoked it unguarded: a throwing sink rejected the whole async result, breaking the seam's "result never rejects on a child-level failure" contract (and docs/defensive-patterns.md's contain-callback-exceptions rule). The sink's own throw is now swallowed — named as such — while the original child failure still settles as stopReason 'error'. Regression test drives a spawn failure through a throwing sink and asserts result resolves. Same defect as the codex backend's, fixed there on PR #240; this is the symmetric fix for the already-merged ACP backend. --- packages/subagent/subagent-acp/src/run.ts | 9 +++++++- .../subagent-acp/tests/subagent-acp.spec.ts | 22 +++++++++++++++++++ 2 files changed, 30 insertions(+), 1 deletion(-) diff --git a/packages/subagent/subagent-acp/src/run.ts b/packages/subagent/subagent-acp/src/run.ts index a9fefba27c..9d5b17cb9a 100644 --- a/packages/subagent/subagent-acp/src/run.ts +++ b/packages/subagent/subagent-acp/src/run.ts @@ -89,6 +89,7 @@ export interface AcpRunSpec { * (the seam contract forbids `result` rejecting). The driver calls this with * the original error and the chosen stop reason so the fault is preserved * rather than silently lost; the provider wires it to `ctx.logger.warn`. + * A throw from the sink itself is contained — it cannot reject `result`. * Optional — omitted in a unit test that asserts the stop reason directly. */ onError?: (error: Error, stopReason: SubagentStopReason) => void @@ -336,7 +337,13 @@ export function startAcpRun(request: SubagentStartRequest, spec: AcpRunSpec): Su // (initialize/newSession/prompt transport/RPC errors, or ENOENT), not a // local bug. Flatten to `error` and surface the original via onError so a // real fault is preserved rather than silently lost. - spec.onError?.(toError(error), 'error') + try { + spec.onError?.(toError(error), 'error') + } catch { + // Swallows only the caller-supplied sink's OWN throw: an unguarded + // sink exception would reject `result` and break the contract above. + // The child-level failure being reported still settles as `error`. + } return { output: collectOutput(), stopReason: 'error' } } })() diff --git a/packages/subagent/subagent-acp/tests/subagent-acp.spec.ts b/packages/subagent/subagent-acp/tests/subagent-acp.spec.ts index 92c025077a..4d8a586c63 100644 --- a/packages/subagent/subagent-acp/tests/subagent-acp.spec.ts +++ b/packages/subagent/subagent-acp/tests/subagent-acp.spec.ts @@ -483,6 +483,28 @@ describe('dsh-subagent-acp', () => { await run.dispose() }) + it('resolves error (never rejects) even when the onError sink itself throws', async () => { + // onError is a caller-supplied callback boundary: its own exception must be + // contained, or it would reject `result` and break the seam's "result never + // rejects" contract that the flattening above exists to uphold. + const run = startAcpRun( + { prompt: [{ type: 'text', text: 'p' }], parent: fakeParent }, + { + command: '/nonexistent/acp-agent-binary', + args: [], + cwd: process.cwd(), + permission: 'reject', + env: {}, + disposeEofGraceMs: DEFAULT_DISPOSE_EOF_GRACE_MS, + disposeGraceMs: DEFAULT_DISPOSE_GRACE_MS, + onError: () => { throw new Error('sink boom') }, + }, + ) + const result = await run.result + expect(result.stopReason).toBe('error') + await run.dispose() + }) + it('settles aborted when the child crashes (tears the pipe) AFTER a cancel', async () => { // The child hangs, we cancel, and instead of answering the child exits hard // — the pending prompt RPC rejects. With a cancel already requested, the From 64b4e2ed2db6d5a27c1266b5c8bd97c8532929b1 Mon Sep 17 00:00:00 2001 From: pku-xht Date: Fri, 10 Jul 2026 16:43:41 +0800 Subject: [PATCH 5/7] test(workflow-workerthread): flake-proof the lifecycle spec's waits under CI load MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The spec's 16 vi.waitFor sites used the 1s default timeout to wait for worker-thread startup and child registration — CPU-bound work that blows past 1s on a contended runner. The CI coverage lane (4 vitest workers plus suites that spawn real subprocesses) hit this 3 times across 4 recent PR runs, each a different subset of the cancellation/worker-death tests, each green on rerun. Every wait now goes through a shared helper with a 10s bound, and the file sets a 30s test timeout to make room for it. The one deliberately tight wait keeps its 800ms bound through the helper's override — it proves the host (not the wedged worker's later loop turn) delivered the cancel, so a generous bound would erase what it tests. No behavior under test changed. --- .../tests/workflow-workerthread.spec.ts | 52 +++++++++++++------ 1 file changed, 36 insertions(+), 16 deletions(-) diff --git a/packages/workflow/workflow-workerthread/tests/workflow-workerthread.spec.ts b/packages/workflow/workflow-workerthread/tests/workflow-workerthread.spec.ts index ecaeed61f6..175a82a7b9 100644 --- a/packages/workflow/workflow-workerthread/tests/workflow-workerthread.spec.ts +++ b/packages/workflow/workflow-workerthread/tests/workflow-workerthread.spec.ts @@ -15,6 +15,26 @@ function fakeParent(): Agent { return { id: AgentId('workflow-parent'), options: {} } as unknown as Agent } +// Worker-thread startup is CPU-bound (a fresh thread compiles the runtime on +// every start): on a contended CI runner it regularly blows past vitest's 5s +// default test timeout, observed repeatedly on the coverage lane. +vi.setConfig({ testTimeout: 30_000 }) + +/** + * `vi.waitFor` with a contention-proof timeout: the 1s default flaked + * repeatedly on the CI coverage lane, where worker-thread cold start competes + * with three sibling vitest workers for CPU. Every wait in this file is for + * something that WILL happen (a worker starting, a child registering) — a + * generous bound only removes the flake, it cannot mask a genuine hang (the + * file-wide test timeout above still fences those). + * @param assertion - retried until it stops throwing or the timeout elapses. + * @param timeout - override for a wait that must stay deliberately tight. + * @returns resolves when the assertion passes. + */ +function waitFor(assertion: () => void, timeout = 10_000): Promise { + return vi.waitFor(assertion, { timeout, interval: 50 }) +} + /** The vm-context escape hatch, spelled once: real Worker tests use it to make the WORKER misbehave. */ const ESCAPE = "globalThis.constructor.constructor('return process')()" @@ -316,7 +336,7 @@ describe('dsh-workflow-workerthread', () => { const runEnds: WorkflowResultInfo[] = [] ctx.on('workflow/end', (_info, result) => { runEnds.push(result) }) const handle = ctx.workflows.start({ ...scripted("return await agent('long job')"), parent }) - await vi.waitFor(() => { expect(provider.runs.length).toBe(1) }) + await waitFor(() => { expect(provider.runs.length).toBe(1) }) handle.cancel('user stopped it') const result = await handle.result expect(result.stopReason).toBe('cancelled') @@ -357,7 +377,7 @@ describe('dsh-workflow-workerthread', () => { const controller = new AbortController() const second = ctx.workflows.start({ ...scripted("return await agent('job')"), parent, signal: controller.signal }) - await vi.waitFor(() => { expect(provider.runs.length).toBe(1) }) + await waitFor(() => { expect(provider.runs.length).toBe(1) }) controller.abort() expect((await second.result).stopReason).toBe('cancelled') await second.dispose() @@ -400,7 +420,7 @@ describe('dsh-workflow-workerthread', () => { `), parent, }) - await vi.waitFor(() => { expect(narration).toContain('started') }) + await waitFor(() => { expect(narration).toContain('started') }) handle.cancel('raced the completion') const result = await handle.result expect(result.stopReason).toBe('cancelled') @@ -480,7 +500,7 @@ describe('dsh-workflow-workerthread', () => { }) const result = await handle.result expect(result.stopReason).toBe('completed') - await vi.waitFor(() => { expect(provider.runs.length).toBe(1) }) + await waitFor(() => { expect(provider.runs.length).toBe(1) }) await handle.dispose() // Not a waitFor: by the time dispose() returns, the slow child disposal // must already be complete (host-side registry quiescence). @@ -526,7 +546,7 @@ describe('dsh-workflow-workerthread', () => { expect(result.stopReason).toBe('completed') // BEFORE dispose(): the settlement itself must have aborted the signal — // without it this child would stay live until dispose's terminate. - await vi.waitFor(() => { expect(aborted).toEqual(['workflow settled']) }) + await waitFor(() => { expect(aborted).toEqual(['workflow settled']) }) await handle.dispose() }) @@ -572,9 +592,9 @@ describe('dsh-workflow-workerthread', () => { `), parent: fakeParent(), }) - await vi.waitFor(() => { expect(starts).toBe(1) }) + await waitFor(() => { expect(starts).toBe(1) }) handle.cancel('stop now') - await vi.waitFor(() => { expect(cancelled).toEqual(['stop now']) }, { timeout: 800 }) + await waitFor(() => { expect(cancelled).toEqual(['stop now']) }, 800) // The wedged worker's own completion loses to the in-flight cancel. const result = await handle.result expect(result.stopReason).toBe('cancelled') @@ -602,7 +622,7 @@ describe('dsh-workflow-workerthread', () => { `), parent, }) - await vi.waitFor(() => { expect(provider.runs.length).toBe(1) }) + await waitFor(() => { expect(provider.runs.length).toBe(1) }) const before = Date.now() await handle.dispose() // Bounded by the grace (plus the terminate), never by the 1.5s spin. @@ -625,7 +645,7 @@ describe('dsh-workflow-workerthread', () => { `), parent, }) - await vi.waitFor(() => { expect(provider.runs.length).toBe(1) }) + await waitFor(() => { expect(provider.runs.length).toBe(1) }) const handleDispose = handle.dispose() const result = await handle.result // The script itself settled (the wrapper's own dispose RPC found the @@ -663,7 +683,7 @@ describe('dsh-workflow-workerthread', () => { `), parent, }) - await vi.waitFor(() => { expect(order.filter(entry => entry.startsWith('start:')).length).toBe(2) }) + await waitFor(() => { expect(order.filter(entry => entry.startsWith('start:')).length).toBe(2) }) const fast = provider.runs.find(run => (run.request.prompt[0] as { text?: string }).text === 'fast')! fast.settle(text('fast done')) handle.cancel('stop now') @@ -694,7 +714,7 @@ describe('dsh-workflow-workerthread', () => { ...scripted("await parallel([() => agent('a'), () => agent('b')])\nreturn 'unreachable'"), parent, }) - await vi.waitFor(() => { expect(provider.runs.length).toBe(2) }) + await waitFor(() => { expect(provider.runs.length).toBe(2) }) handle.cancel('user stop') const result = await handle.result expect(result.stopReason).toBe('cancelled') @@ -749,7 +769,7 @@ describe('dsh-workflow-workerthread', () => { // A worker death is a stop reason like any other: workflow/end fires // with the error outcome — for a bus observer it is the only obituary. expect(runEnds).toEqual([{ stopReason: 'error', error: result.error, agentsStarted: 1 }]) - await vi.waitFor(() => { expect(cancelled.length).toBe(1) }) + await waitFor(() => { expect(cancelled.length).toBe(1) }) await handle.dispose() }, 15_000) @@ -770,7 +790,7 @@ describe('dsh-workflow-workerthread', () => { expect(result.stopReason).toBe('error') expect(result.error).toContain('worker blew up') // The reap wound the stray child down (cancel + a CLEAN dispose). - await vi.waitFor(() => { + await waitFor(() => { expect(provider.runs.length).toBe(1) expect(provider.runs[0]!.disposed).toBe(true) }) @@ -802,7 +822,7 @@ describe('dsh-workflow-workerthread', () => { `), parent, }) - await vi.waitFor(() => { expect(order.filter(entry => entry.startsWith('start:')).length).toBe(2) }) + await waitFor(() => { expect(order.filter(entry => entry.startsWith('start:')).length).toBe(2) }) const fast = provider.runs.find(run => (run.request.prompt[0] as { text?: string }).text === 'fast')! fast.settle(text('fast done')) const result = await handle.result @@ -837,7 +857,7 @@ describe('dsh-workflow-workerthread', () => { const result = await handle.result expect(result.stopReason).toBe('error') expect(result.error).toContain('exit code 5') - await vi.waitFor(() => { expect(provider.runs[0]!.disposed).toBe(true) }) + await waitFor(() => { expect(provider.runs[0]!.disposed).toBe(true) }) await handle.dispose() }, 15_000) @@ -855,7 +875,7 @@ describe('dsh-workflow-workerthread', () => { }) const logs: string[] = [] ctx.on('workflow/log', (_info, message) => { logs.push(message) }) - await vi.waitFor(() => { expect(logs).toContain('armed') }) + await waitFor(() => { expect(logs).toContain('armed') }) handle.cancel('stop it') // The grace is deliberately huge: only the worker's own death (exit 3, // unreachable by the cancel — the script ignores hooks) settles this. From dd2f37b80fa1e69403268f02bd64b4576daa01f8 Mon Sep 17 00:00:00 2001 From: pku-xht Date: Fri, 10 Jul 2026 20:48:14 +0800 Subject: [PATCH 6/7] fix(workflow-workerthread): tighten post-result promptness waits back down MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review follow-up: the blanket 10s default correctly targets worker-thread cold-start races (starting, first-script-line, async child-registration messages — genuinely CPU-bound under CI contention), but four waits assert something different — that the HOST reacted PROMPTLY to an event that already happened (a settled result, an observed worker death). Those had no cold-start left to wait on, so the generous default just widened the window a real regression could hide in. Verified by injecting a 6s delay into the settle-reap's abort call: the un-overridden helper's test still passed in ~6s. The same mutation now fails in ~1s with the explicit 1000ms override restored on all four sites (the abort-on-settle test's own assertion, the two worker-death cancel/dispose reap checks, and the dispose-ack-race check). The other 12 waits keep the 10s default — they run BEFORE a result is awaited, waiting on the worker to actually start rather than on a host reaction. Doc comment corrected to describe the split instead of claiming every wait is a cold-start race. --- .../tests/workflow-workerthread.spec.ts | 38 +++++++++++++------ 1 file changed, 27 insertions(+), 11 deletions(-) diff --git a/packages/workflow/workflow-workerthread/tests/workflow-workerthread.spec.ts b/packages/workflow/workflow-workerthread/tests/workflow-workerthread.spec.ts index 175a82a7b9..75a0b5116e 100644 --- a/packages/workflow/workflow-workerthread/tests/workflow-workerthread.spec.ts +++ b/packages/workflow/workflow-workerthread/tests/workflow-workerthread.spec.ts @@ -21,12 +21,18 @@ function fakeParent(): Agent { vi.setConfig({ testTimeout: 30_000 }) /** - * `vi.waitFor` with a contention-proof timeout: the 1s default flaked - * repeatedly on the CI coverage lane, where worker-thread cold start competes - * with three sibling vitest workers for CPU. Every wait in this file is for - * something that WILL happen (a worker starting, a child registering) — a - * generous bound only removes the flake, it cannot mask a genuine hang (the - * file-wide test timeout above still fences those). + * `vi.waitFor` with a contention-proof default timeout: the 1s default + * flaked repeatedly on the CI coverage lane, where worker-thread cold start + * (CPU-bound — a fresh thread compiles the runtime) competes with three + * sibling vitest workers for CPU. The 10s default is for exactly those + * races — waiting for a worker to start, run its first script line, or + * deliver an async child-registration message to the host. It is NOT for a + * wait that asserts the HOST reacted PROMPTLY to something that already + * happened (a settled result, an observed worker death): those keep an + * explicit tight override below, or the generous default would silently + * accept a multi-second regression in host-side reap latency as passing + * (proven by injecting a 6s delay into one such reap and watching the + * un-overridden version of this helper still pass in ~6s). * @param assertion - retried until it stops throwing or the timeout elapses. * @param timeout - override for a wait that must stay deliberately tight. * @returns resolves when the assertion passes. @@ -545,8 +551,11 @@ describe('dsh-workflow-workerthread', () => { const result = await handle.result expect(result.stopReason).toBe('completed') // BEFORE dispose(): the settlement itself must have aborted the signal — - // without it this child would stay live until dispose's terminate. - await waitFor(() => { expect(aborted).toEqual(['workflow settled']) }) + // without it this child would stay live until dispose's terminate. This + // is a HOST-PROMPTNESS claim, not a cold-start race — a tight explicit + // bound (unlike the file default) so a multi-second reap regression + // cannot pass by outlasting the wait. + await waitFor(() => { expect(aborted).toEqual(['workflow settled']) }, 1000) await handle.dispose() }) @@ -769,7 +778,9 @@ describe('dsh-workflow-workerthread', () => { // A worker death is a stop reason like any other: workflow/end fires // with the error outcome — for a bus observer it is the only obituary. expect(runEnds).toEqual([{ stopReason: 'error', error: result.error, agentsStarted: 1 }]) - await waitFor(() => { expect(cancelled.length).toBe(1) }) + // Result already settled — this is the reap's promptness, not a + // cold-start race; tight explicit bound (see the helper's doc comment). + await waitFor(() => { expect(cancelled.length).toBe(1) }, 1000) await handle.dispose() }, 15_000) @@ -790,10 +801,12 @@ describe('dsh-workflow-workerthread', () => { expect(result.stopReason).toBe('error') expect(result.error).toContain('worker blew up') // The reap wound the stray child down (cancel + a CLEAN dispose). + // Result already settled — this is the reap's promptness, not a + // cold-start race; tight explicit bound (see the helper's doc comment). await waitFor(() => { expect(provider.runs.length).toBe(1) expect(provider.runs[0]!.disposed).toBe(true) - }) + }, 1000) await handle.dispose() }, 15_000) @@ -857,7 +870,10 @@ describe('dsh-workflow-workerthread', () => { const result = await handle.result expect(result.stopReason).toBe('error') expect(result.error).toContain('exit code 5') - await waitFor(() => { expect(provider.runs[0]!.disposed).toBe(true) }) + // Result already settled — this is the reap's promptness (bounded + // above the mock's fixed 300ms dispose delay, not a cold-start race); + // tight explicit bound (see the helper's doc comment). + await waitFor(() => { expect(provider.runs[0]!.disposed).toBe(true) }, 1000) await handle.dispose() }, 15_000) From 53f6a959e2f9721ac676a91bbda58dfd7121a1d9 Mon Sep 17 00:00:00 2001 From: kingwl Date: Fri, 10 Jul 2026 21:15:48 +0800 Subject: [PATCH 7/7] chore(lint): one shared project service; ignore harness-local .claude state MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two independent multipliers were pushing bare `pnpm run lint` past node's default heap: 1. parserOptions.project expanded to every package tsconfig plus the root one — each matched config built its OWN ts.Program, and the dev paths map pulls sibling package SOURCES (plus the vendored Cordis stack) into every such program, so resident memory grew as sum-of-closures, not repo size: ~4.6 GB peak for 425 repo files. projectService shares one tsserver-style graph: ~2.0 GB peak, ~28 s → ~14 s wall. 2. `eslint .` traversed .claude/ harness-local state — stale worktree checkouts there carry tens of thousands of additional .ts files (whole-repo copies), roughly tripling the work again even under the project service. Other checkouts are not this one's sources; ignore them like node_modules. (#169 carries the identical ignore line inside its chain; the hunks dedupe on its next rebase.) Type-aware rules verified live under the service: a floating-promise probe still trips no-floating-promises. --- eslint.config.mjs | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/eslint.config.mjs b/eslint.config.mjs index cbb696bccb..c62d3e9739 100644 --- a/eslint.config.mjs +++ b/eslint.config.mjs @@ -22,6 +22,7 @@ export default tseslint.config( '**/lib/**', '**/node_modules/**', '**/.sessions/**', + '.claude/**', // harness-local state (worktrees, skills) — other checkouts, not this one's sources '**/.doc-typecheck-*/**', 'vendor/**', // vendored source keeps upstream style and idioms '**/*.js', @@ -38,7 +39,14 @@ export default tseslint.config( ], languageOptions: { parserOptions: { - project: ['./packages/*/*/tsconfig.json', './tsconfig.json'], + // One shared tsserver-style project service instead of 60+ standalone + // per-package programs: the old `project` glob built every package's + // full dependency closure (sibling sources via the dev `paths` map + + // the vendored Cordis stack) as its own program and kept them all + // resident — ~5 GB peak, an OOM past node's default heap. The service + // resolves each file to its nearest owning tsconfig and shares the + // graph. + projectService: true, tsconfigRootDir: import.meta.dirname, }, }, @@ -89,7 +97,9 @@ export default tseslint.config( ], languageOptions: { parserOptions: { - project: ['./tsconfig.json'], + // Same shared project service as the src block: test files resolve + // through the root tsconfig (its include covers every tests/ tree). + projectService: true, tsconfigRootDir: import.meta.dirname, }, },