Merge remote-tracking branch 'origin/master' into session-fork

This commit is contained in:
Hypatia May
2026-07-07 09:21:01 +08:00
43 changed files with 362 additions and 48 deletions

View File

@@ -160,6 +160,7 @@ Generated by `pnpm run gen-rfc-index` from the RFC tree — never edit by hand;
| [Per-session snapshot replay for nested agents](implemented/testing/2026-06-22-subagent-snapshot-replay.md) | 2026-06-22 |
| [Hook snapshot matrix — end-to-end goldens for both bridges](implemented/testing/2026-07-04-hook-snapshot-matrix.md) | 2026-07-04 |
| [Single-source the acp-agent replay config](implemented/testing/2026-07-04-single-source-acp-replay-config.md) | 2026-07-04 |
| [Pin request-header content in one snapshot scenario](implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md) | 2026-07-06 |
## Rejected

View File

@@ -51,7 +51,7 @@ The ACP server app loads `@deepseek-ai/dsh-llm-deepseek`, whose `apply` throws w
A snapshot run asserts **two** normalized surfaces, because the harness's external surfaces are distinct:
1. The **stdout transcript** — the framed `session/update` JSON-RPC the editor sees. Catches regressions in the ACP bridge's event→update translation (`streamSessionEventUpdate`). Compared against a committed `stdout.golden.jsonl`.
2. The **re-persisted session JSONL** — the log the replay run itself persists, compared against the scenario's `session.jsonl`. Catches regressions in the loop, tool dispatch, and turn/step structure that never surface on stdout. There is no separate session golden: `session.jsonl` is BOTH the replay source (recorded scenarios) and the expected produced log. Both sides pass through `normalizeSessionLog` before comparing — the fixture is raw-harvested (its own real session id / cwd / timestamps) and the replay output has fresh ones, so each is scrubbed against ITS OWN volatile values (the fixture's read from its header line) and the comparison is on normalized form. For an authored override scenario the same `session.jsonl` holds the expected produced log; `replay.override.json` drives the model, and `llm-replay` ignores the fixture for model chunks when an override exists, so committing the expected log there does not affect replay.
2. The **re-persisted session JSONL** — the log the replay run itself persists, compared against the scenario's `session.jsonl`. Catches regressions in the loop, tool dispatch, and turn/step structure that never surface on stdout. There is no separate session golden: `session.jsonl` is BOTH the replay source (recorded scenarios) and the expected produced log. Both sides pass through `normalizeSessionLog` before comparing — the fixture is raw-harvested (its own real session id / cwd / timestamps) and the replay output has fresh ones, so each is scrubbed against ITS OWN volatile values (the fixture's read from its header line) and the comparison is on normalized form. Request-header CONTENT (the composed system prompt + tool schemas) is additionally scrubbed to `{{system}}`/`{{tools}}` tokens on both sides — in the stored fixtures too — for every scenario except the one that pins it ([pinned-header RFC](2026-07-06-pin-request-header-content-in-one-scenario.md)). For an authored override scenario the same `session.jsonl` holds the expected produced log; `replay.override.json` drives the model, and `llm-replay` ignores the fixture for model chunks when an override exists, so committing the expected log there does not affect replay.
The two are genuinely additive: stdout is the bridge's *lossy projection* of the log (it drops `assistant/message.usage`, `step/*`, exact `seq`/`time`, and renders tool I/O differently), so a loop/tool/turn-structure regression can change the JSONL while leaving the stdout projection identical, and a bridge-translation regression can change stdout while the JSONL is untouched. Asserting the JSONL equality also echoes the proposed [universal replay fixture](../../proposed/testing/2026-06-11-deterministic-and-stress-testing.md) idea.

View File

@@ -0,0 +1,32 @@
# RFC: Pin request-header content in one snapshot scenario
Status: implemented
## Problem
Every model-driving ACP snapshot fixture (`session.jsonl`) embedded the full composed system prompt and the complete tool-schema list in its `request/header` event — roughly 8 KB on one line, per fixture. That content is identical across the suite (byte-identical tool list everywhere, including subagent children; identical prompt modulo each recording's temp cwd), so any change touching a tool description or a system-prompt line had to update every fixture: re-record everything against the live API (churning model responses and stdout goldens along the way) or hand-edit ~35 giant header lines. Introducing the dynamic-workflows feature — one new tool plus one prompt paragraph — rewrote every snapshot fixture in the repo, burying the behavioral diff a reviewer should be reading.
## Decision
Exactly one scenario — `text-turn`, flagged `pinsHeader` in `acp.snapshot.ts` — commits and compares the full request-header content. Every other fixture stores and compares that content as stable tokens via the pure normalizer `scrubRequestHeaders` in `snapshot-normalize.ts`: a `request/header` event's `header.system` becomes `"{{system}}"` and `header.tools` becomes `"{{tools}}"`; a `request/header-delta` keeps its structural facts — the system delta's `keepStart`/`keepEnd` line positions with one `{{system}}` token per inserted line, the tools delta's added/removed/changed tool names — and tokenizes only the bulk (prompt text, schema bodies), so two different deltas still compare different. The scrub is composed in front of `normalizeSessionLog` on BOTH sides of a non-pinning scenario's log compare and applied to the harvested logs record mode writes, so a re-record cannot smuggle the content back. Absent fields stay absent — WHETHER a header carried a prompt or tools is behavior and stays visible — and `config`/`reason` stay verbatim: a model swap churns every fixture by design (it invalidates the recorded responses), while a prompt or schema edit churns none of them (replay derives model behavior exclusively from `assistant/chunk` events and never reads header content — see `dsh-llm-replay`).
A system-prompt or tool-schema change therefore lands as exactly one committed-fixture diff — the pinned `text-turn` header line — updated by hand or by re-recording that one scenario (`pnpm run test:snapshot:record` with `-t text-turn`).
Guards make the split self-enforcing. On disk (fixture meta-tests): every non-pinning `session*.jsonl` must be a fixed point of `scrubRequestHeaders` (unscrubbed content crept in — apply the scrub), the pinning scenario's fixture must NOT be one (the pin lost its content), and exactly one scenario must pin. Live (every non-pinning scenario run): each `request/header` the run produces — parent, spawn child, fork child, initial or resume — must equal the pinned fixture's header after both sides normalize their own volatile values, and no `request/header-delta` may appear at all (a mid-run header change diverges from the pin by construction, and its content would be invisible under the scrub), so the single-pin premise is asserted rather than assumed.
One pin covers the whole suite because every session — parent, spawn child, fork child — composes the identical tool list and the identical prompt modulo cwd, and the uniformity guard fails the suite the moment that stops holding. If header composition ever becomes session-dependent by design (a restricted subagent toolset, say), the divergent shape gets its own pinning scenario.
## Alternatives considered
- **Re-record or hand-edit every fixture per change** — the status quo; the churn this RFC removes.
- **Scrub at compare time only, keeping fixtures raw** — the compares go green without fixture edits, but every committed fixture then carries a permanently stale copy of the prompt and schemas: dead weight that misleads readers and still rewrites wholesale on the next re-record. Storing the tokens keeps the fixture honest about what it does and does not pin.
- **Scrub everywhere, pin nowhere** — loses the only end-to-end record of the composed header as actually sent (prompt assembly, registered-tool order, full schemas). The generated tool catalog documents each tool in isolation; only a real fixture pins the composed set.
- **Slim the session log itself (log a content digest, store the header elsewhere)** — violates the reconstructability contract: the product log must reproduce each request bit-for-bit ([reconstructable-requests RFC](../architecture/2026-07-05-reconstructable-requests.md)). Header bulk is a test-artifact concern, solved in test normalization; the live log is untouched.
## Verification
All 37 snapshot scenarios replay green with the scrubbed fixtures (the committed fixtures were rewritten once through `scrubRequestHeaders` itself; `text-turn` untouched). The fixed-point, pin-retains-content, exactly-one-pin, live header-uniformity, and no-unpinned-delta guards run inside the suite, and `scrubRequestHeaders` has unit coverage for both header event types, delta structure preservation (line positions, insert arity, tool names), absent-field preservation, config/reason retention, byte-for-byte pass-through of other lines, and idempotence.
## Consequences
A tool-description or system-prompt change churns one committed fixture line instead of every fixture in the suite, so snapshot diffs read as behavior again, and ~270 KB of duplicated header bytes leave the repo. The cost: non-pinning fixtures no longer display header content, so reading one shows tokens where the prompt and schemas were — the pinned `text-turn` fixture is the place to look, and the live uniformity guard guarantees it speaks for every session in the suite. A header change surfaces as a suite-wide test failure whose fix is the one pinned line, rather than as ~35 fixture rewrites.

View File

@@ -7,7 +7,7 @@ How this repo tests, tier by tier, and the rules that keep a green suite meaning
- **Unit** (`pnpm run test`): vitest over `packages|examples/*/tests/**/*.spec.ts`, colocated with what they test. Every registry gets an HMR-safety test (dispose the contributing fiber, assert cleanup). Excessive tests are welcome — err toward covering edge cases, error paths, event ordering, and concurrency races; review findings get regression tests (see `packages/core/agent-loop/tests/review-fixes.spec.ts`).
- **Coverage gate** (`pnpm run test:coverage`): the gating run, per-file 100% on `packages/*/*/src`. An uncovered line is often dead code the gate is correctly flagging for deletion, not a missing test to bolt on. Line coverage is necessary, never sufficient — it proves lines ran, not that the feature works as shipped.
- **Real-API e2e** (`pnpm run test:e2e`): with-key tests against live provider APIs — the DeepSeek model plus provider-specific smokes that gate on their own keys (`EXA_API_KEY`, `PERPLEXITY_API_KEY`, …); each suite self-skips without its key so keyless CI stays green ([real-API e2e RFC](rfc/implemented/testing/2026-06-19-real-api-e2e-ci.md)).
- **Snapshot** (`pnpm run test:snapshot`): boots the real example subprocess, replays a recorded session keyless, diffs normalized stdout + the re-persisted log against committed goldens ([snapshot RFC](rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md)). Re-record with `pnpm run test:snapshot:record`; reviewing the golden diff is part of the review.
- **Snapshot** (`pnpm run test:snapshot`): boots the real example subprocess, replays a recorded session keyless, diffs normalized stdout + the re-persisted log against committed goldens ([snapshot RFC](rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md)). Re-record with `pnpm run test:snapshot:record`; reviewing the golden diff is part of the review. System-prompt/tool-schema content is pinned by ONE scenario (`text-turn`) and tokenized in every other fixture, so a prompt or schema edit churns one committed line ([pinned-header RFC](rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
## The with-key policy: inference is cheap here

View File

@@ -4,7 +4,7 @@ import { fileURLToPath } from 'node:url'
import { dirname, join } from 'node:path'
import { describe, expect, it } from 'vitest'
import { type HarvestedLog, type InputScript, runScenario } from './snapshot-harness.ts'
import { type NormalizeContext, normalizeSessionLog, normalizeStdout } from './snapshot-normalize.ts'
import { type NormalizeContext, normalizeSessionLog, normalizeStdout, scrubRequestHeaders } from './snapshot-normalize.ts'
/**
* ACP snapshot tests (REPLAY by default, keyless). Each scenario under
@@ -16,6 +16,16 @@ import { type NormalizeContext, normalizeSessionLog, normalizeStdout } from './s
* golden: the fixture doubles as the replay source (recorded scenarios) and the
* expected produced log (both sides normalized before comparing).
*
* Request-header content (the composed system prompt + tool schemas riding on
* `request/header` events) is pinned by exactly ONE scenario — the one with
* `pinsHeader` — and scrubbed to `{{system}}`/`{{tools}}` tokens in every
* other fixture and compare, so a prompt or tool-schema edit churns one
* committed line instead of every fixture. A per-run uniformity guard keeps
* the single pin sound: every live header must equal the pinned one, and no
* header-delta may appear outside the pinning scenario (see the
* pinned-header RFC,
* docs/rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md).
*
* `pnpm run test:snapshot:record` (DSH_SNAPSHOT=record + -u) re-records the
* `session.jsonl` fixtures against the real API and refreshes the stdout golden
* in one pass.
@@ -55,12 +65,31 @@ interface Scenario {
* harvested child logs back to those files. Defaults to 0.
*/
childSessions?: number
/**
* Whether THIS scenario's fixtures keep the full request-header content (the
* composed system prompt and tool schema list on `request/header` /
* `request/header-delta` events) and compare it verbatim. Exactly one
* scenario pins it; every other scenario stores and compares that content as
* `{{system}}`/`{{tools}}` tokens ({@link scrubRequestHeaders}), so a system
* prompt or tool-schema change shows up as ONE committed-fixture diff, not
* one per scenario. One pin suffices because header composition is
* suite-uniform (parent, spawn child, and fork child all compose the same
* prompt-modulo-cwd and the same tools) — and that premise is ASSERTED, not
* assumed: every non-pinning run's live headers must equal the pinned
* fixture's (normalized), so a session-dependent header (say, a restricted
* subagent toolset) fails loud until it gets its own pinning scenario.
* Defaults to false.
*/
pinsHeader?: boolean
}
const SCENARIOS: Scenario[] = [
{ name: 'handshake', hasModelTurn: false, recorded: false },
{ name: 'reject-extra-dirs', hasModelTurn: false, recorded: false },
{ name: 'text-turn', hasModelTurn: true, recorded: true },
// text-turn is the pinned-header scenario: the minimal single text turn,
// whose fixture is the ONE place the full system prompt + tool schemas are
// committed and compared verbatim.
{ name: 'text-turn', hasModelTurn: true, recorded: true, pinsHeader: true },
{ name: 'tool-call-turn', hasModelTurn: true, recorded: true },
{ name: 'fs-terminal-card', hasModelTurn: true, recorded: true },
{ name: 'todo-plan', hasModelTurn: true, recorded: true },
@@ -119,6 +148,10 @@ const SCENARIOS: Scenario[] = [
{ name: 'hook-codex-stop-continue', hasModelTurn: true, recorded: true },
]
/** The single header-pinning scenario. Guarded here (and by a meta-test) so the pin cannot silently vanish. */
const pinningScenario = SCENARIOS.find(s => s.pinsHeader === true)
if (pinningScenario === undefined) throw new Error('acp.snapshot: no scenario pins the request-header content')
/** The sibling child-fixture paths for a scenario (`session.1.jsonl` …). */
function childFixturePaths(dir: string, childSessions: number): string[] {
return Array.from({ length: childSessions }, (_, i) => join(dir, `session.${i + 1}.jsonl`))
@@ -146,6 +179,30 @@ function fixtureContext(fixture: string): NormalizeContext {
}
}
/**
* The `data.header` payload of every `request/header` event in a session
* JSONL, in log order, with the log's volatile values scrubbed first
* ({@link normalizeSessionLog}) so headers harvested from different runs —
* each embedding its own temp cwd in the composed prompt — compare on equal
* footing.
*/
function normalizedHeaders(rawLog: string, ctx: NormalizeContext): unknown[] {
return normalizeSessionLog(rawLog, ctx)
.split('\n')
.filter(line => line.trim().length > 0)
.map(line => JSON.parse(line) as { type?: unknown; data?: { header?: unknown } })
.filter(record => record.type === 'request/header')
.map(record => record.data?.header)
}
/** Count the `request/header-delta` events in a session JSONL. */
function headerDeltaCount(rawLog: string): number {
return rawLog.split('\n')
.filter(line => line.trim().length > 0)
.filter(line => (JSON.parse(line) as { type?: unknown }).type === 'request/header-delta')
.length
}
for (const scenario of SCENARIOS) {
describe(`snapshot: ${scenario.name}`, () => {
// In RECORD mode, only re-run the `recorded` (live-API) scenarios; the
@@ -181,14 +238,19 @@ for (const scenario of SCENARIOS) {
// RECORD mode (recorded model scenarios only): persist the freshly-harvested
// logs back to their fixtures — the primary to session.jsonl, each child to
// session.<n>.jsonl in harvest order. `--update` refreshes the Vitest
// goldens but NOT these fixtures, so write them here.
// goldens but NOT these fixtures, so write them here. A non-pinning
// scenario's fixtures are written header-scrubbed, so a re-record can
// never smuggle the full prompt/schema content back into every fixture.
const scrub = scenario.pinsHeader === true
? (log: string): string => log
: scrubRequestHeaders
if (RECORDING && scenario.recorded && scenario.hasModelTurn) {
expect(result.sessionLogs.length, 'record produced no session log to harvest').toBeGreaterThan(0)
expect(result.sessionLogs.length, `expected ${childSessions + 1} session logs (parent + children)`)
.toBe(childSessions + 1)
await writeFile(join(dir, 'session.jsonl'), (result.sessionLogs[0] as HarvestedLog).content)
await writeFile(join(dir, 'session.jsonl'), scrub((result.sessionLogs[0] as HarvestedLog).content))
for (let i = 1; i < result.sessionLogs.length; i++) {
await writeFile(join(dir, `session.${i}.jsonl`), (result.sessionLogs[i] as HarvestedLog).content)
await writeFile(join(dir, `session.${i}.jsonl`), scrub((result.sessionLogs[i] as HarvestedLog).content))
}
}
@@ -203,15 +265,49 @@ for (const scenario of SCENARIOS) {
// 1:1. Each side passes through normalizeSessionLog, scrubbed against ITS
// OWN volatile values — the live run's via `ctx`, the committed fixture's
// via its own header (a committed file cannot share the live run's ids).
// Unless this scenario pins the header, both sides ALSO pass through
// scrubRequestHeaders: the live log carries the real prompt/schemas, the
// fixture carries the `{{system}}`/`{{tools}}` tokens, and the scrub is
// idempotent — so the compare checks the header's presence, position,
// reason, and config, but not its bulk content (pinned once, in the
// `pinsHeader` scenario).
expect(result.sessionLogs.length, 'this scenario must persist a session log').toBe(childSessions + 1)
const fixtureFiles = ['session.jsonl', ...Array.from({ length: childSessions }, (_, i) => `session.${i + 1}.jsonl`)]
for (let i = 0; i < fixtureFiles.length; i++) {
const harvested = (result.sessionLogs[i] as HarvestedLog).content
const fixture = await readFile(join(dir, fixtureFiles[i] as string), 'utf8')
const harvested = scrub((result.sessionLogs[i] as HarvestedLog).content)
const fixture = scrub(await readFile(join(dir, fixtureFiles[i] as string), 'utf8'))
expect(normalizeSessionLog(harvested, ctx), `${fixtureFiles[i]} mismatch`)
.toEqual(normalizeSessionLog(fixture, fixtureContext(fixture)))
}
}
// Header-uniformity guard: the single pin is sound only while every
// session in the suite composes the SAME header and keeps it for the
// whole run. Assert both halves live. (1) Every request/header the run
// produced (parent, spawn child, fork child, initial or resume) must
// equal the pinned fixture's header after each side is normalized
// against its own volatile values. (2) No request/header-delta may
// appear at all — a mid-run header change diverges from the pin by
// construction, and its content would be invisible under the scrub. If
// either fails, either the header changed (update the pin: re-record or
// hand-edit the pinning scenario's fixture) or composition became
// session-dependent by design (give the divergent shape its own
// pinning scenario).
if (scenario.pinsHeader !== true) {
const pinnedFixture = await readFile(join(SNAPSHOTS_DIR, pinningScenario.name, 'session.jsonl'), 'utf8')
const pinned = normalizedHeaders(pinnedFixture, fixtureContext(pinnedFixture))
expect(pinned.length, `the pinning fixture (${pinningScenario.name}) must carry exactly one request/header`)
.toBe(1)
for (const log of result.sessionLogs) {
expect(headerDeltaCount(log.content), `session ${log.id}: a request/header-delta in a non-pinning scenario`)
.toBe(0)
const headers = normalizedHeaders(log.content, ctx)
for (const [k, header] of headers.entries()) {
expect(header, `session ${log.id}: request/header #${k + 1} diverged from the pinned (${pinningScenario.name}) header`)
.toEqual(pinned[0])
}
}
}
})
})
}
@@ -253,4 +349,37 @@ describe('snapshot fixtures', () => {
}
}
})
it('exactly one scenario pins the request-header content', () => {
// Zero pins would drop the prompt/schema surface from the suite entirely;
// two would split it. The single pin is the design (pinned-header RFC).
expect(SCENARIOS.filter(s => s.pinsHeader === true).map(s => s.name)).toEqual(['text-turn'])
})
it('committed fixtures carry request-header content ONLY in the pinning scenario', async () => {
// The whole point of the pin: a system-prompt or tool-schema change must
// churn exactly one committed line. A non-pinning fixture that carries the
// full header (a hand-recorded file, or a header line hand-edited out of
// its canonical JSON form) silently reopens the suite-wide churn, so fail
// loud here: every non-pinning session*.jsonl must be a fixed point of
// scrubRequestHeaders (apply the scrub to fix a violation), and the
// pinning scenario's fixtures must NOT be (their content IS the pin).
for (const scenario of SCENARIOS) {
const dir = join(SNAPSHOTS_DIR, scenario.name)
const files = [
'session.jsonl',
...Array.from({ length: scenario.childSessions ?? 0 }, (_, i) => `session.${i + 1}.jsonl`),
]
for (const file of files) {
const fixture = await readFile(join(dir, file), 'utf8')
if (scenario.pinsHeader === true) {
expect(scrubRequestHeaders(fixture), `${scenario.name}/${file} must PIN the full header content`)
.not.toEqual(fixture)
} else {
expect(scrubRequestHeaders(fixture), `${scenario.name}/${file} carries unscrubbed header content`)
.toEqual(fixture)
}
}
}
})
})

View File

@@ -1,5 +1,5 @@
import { describe, expect, it } from 'vitest'
import { type NormalizeContext, normalizeSessionLog, normalizeStdout } from '../tests/snapshot-normalize.ts'
import { type NormalizeContext, normalizeSessionLog, normalizeStdout, scrubRequestHeaders } from '../tests/snapshot-normalize.ts'
/**
* Unit tests for the pure snapshot normalizers. Live as a *.spec.ts (runs in
@@ -108,3 +108,78 @@ describe('normalizeSessionLog', () => {
expect(out).toContain('"durationMs":88')
})
})
describe('scrubRequestHeaders', () => {
const headerLine = JSON.stringify({ type: 'session', version: 0, id: 's', createdAt: 1, cwd: '/w' })
const headerEvent = (header: object) =>
JSON.stringify({ type: 'request/header', seq: 3, time: 9, data: { header, reason: 'initial' } })
it('replaces header system and tools with tokens, keeping config and reason', () => {
const ev = headerEvent({
config: { model: 'm' },
system: 'You are an agent.\nBe brief.',
tools: [{ name: 'read', description: 'Read a file.', parameters: { type: 'object' } }],
})
const out = scrubRequestHeaders(`${headerLine}\n${ev}\n`)
expect(out).toContain('"system":"{{system}}"')
expect(out).toContain('"tools":"{{tools}}"')
expect(out).toContain('"config":{"model":"m"}')
expect(out).toContain('"reason":"initial"')
expect(out).not.toContain('You are an agent')
expect(out).not.toContain('Read a file')
})
it('keeps an absent system/tools absent (presence is behavior)', () => {
const out = scrubRequestHeaders(`${headerLine}\n${headerEvent({ config: { model: 'm' } })}\n`)
expect(out).not.toContain('{{system}}')
expect(out).not.toContain('{{tools}}')
})
it('scrubs a header-delta system payload but keeps its line positions and arity', () => {
const delta = JSON.stringify({
type: 'request/header-delta', seq: 8, time: 9,
data: { system: { keepStart: 1, keepEnd: 4, insert: ['leaked prompt line', 'second line'] }, config: { model: 'm2' } },
})
const out = scrubRequestHeaders(`${headerLine}\n${delta}\n`)
// One token PER inserted line: the edit's position AND extent survive.
expect(out).toContain('"insert":["{{system}}","{{system}}"]')
expect(out).toContain('"keepStart":1')
expect(out).toContain('"keepEnd":4')
expect(out).toContain('"config":{"model":"m2"}')
expect(out).not.toContain('leaked prompt line')
expect(out).not.toContain('{{tools}}') // no tools delta → none invented
})
it('scrubs a header-delta tools payload but keeps the added/removed/changed names', () => {
const delta = JSON.stringify({
type: 'request/header-delta', seq: 8, time: 9,
data: {
tools: {
added: [{ name: 'grep', description: 'Search files.', parameters: { type: 'object' } }],
removed: ['bash_kill'],
changed: [{ name: 'read', description: 'Read v2.', parameters: { type: 'object' } }],
},
},
})
const out = scrubRequestHeaders(`${headerLine}\n${delta}\n`)
// WHICH tools changed is behavior and survives; their bulk does not.
expect(out).toContain('"added":[{"name":"grep","description":"{{tools}}","parameters":"{{tools}}"}]')
expect(out).toContain('"removed":["bash_kill"]')
expect(out).toContain('"changed":[{"name":"read","description":"{{tools}}","parameters":"{{tools}}"}]')
expect(out).not.toContain('Search files')
expect(out).not.toContain('Read v2')
})
it('passes every other line through byte-for-byte and is idempotent', () => {
const other = JSON.stringify({ type: 'assistant/chunk', seq: 4, time: 9, data: { turn: 1, step: 1, chunk: { type: 'text-delta', index: 0, text: 'hi' } } })
const delta = JSON.stringify({
type: 'request/header-delta', seq: 8, time: 9,
data: { system: { keepStart: 0, keepEnd: 0, insert: ['x'] }, tools: { added: [{ name: 't', description: 'd', parameters: {} }], removed: [], changed: [] } },
})
const raw = `${headerLine}\n${headerEvent({ config: { model: 'm' }, system: 's', tools: [] })}\n${delta}\n${other}\n`
const once = scrubRequestHeaders(raw)
expect(once.split('\n')[0]).toBe(headerLine)
expect(once.split('\n')[3]).toBe(other)
expect(scrubRequestHeaders(once)).toBe(once)
})
})

View File

@@ -12,11 +12,21 @@
* `durationMs` (wall-clock hook runtime) → 0. NOT scrubbed: the log's `seq`
* (deterministic — `seq = log.length`, part of the event-log contract).
*
* A separate, composable normalizer — {@link scrubRequestHeaders} — replaces
* the bulky request-header CONTENT (the composed system prompt and the tool
* schema list) with `{{system}}`/`{{tools}}` tokens. It is deliberately NOT
* folded into {@link normalizeSessionLog}: the one header-pinning scenario
* compares that content verbatim, every other scenario composes the scrub in
* (the `pinsHeader` flag in acp.snapshot.ts; see the pinned-header RFC,
* docs/rfc/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md).
*
* See docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md.
*/
const SESSION_ID = '{{sessionId}}'
const CWD = '{{cwd}}'
const SYSTEM = '{{system}}'
const TOOLS = '{{tools}}'
/** A UUID v4 string, the shape `randomUUID()` produces for session ids. */
const UUID_RE = /[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/gi
@@ -110,3 +120,66 @@ export function normalizeSessionLog(rawLog: string, ctx: NormalizeContext): stri
})
return records.map(r => JSON.stringify(r)).join('\n') + '\n'
}
/**
* Replace request-header CONTENT in a session JSONL with stable tokens,
* keeping its structure: a `request/header` event's `data.header.system` →
* `{{system}}` and `data.header.tools` → `{{tools}}`; a
* `request/header-delta` event keeps every structural fact — the system
* delta's `keepStart`/`keepEnd` line positions and inserted-line COUNT (one
* `{{system}}` token per inserted line), the tools delta's
* added/removed/changed tool NAMES — and tokenizes only the bulk (prompt
* text; each added/changed schema's fields other than `name` → `{{tools}}`),
* so two different deltas still compare different.
* Absent fields stay absent — WHETHER a header carried a system prompt or
* tools is behavior and stays visible; `config` and `reason` are small and
* stable, so they stay verbatim (a model swap churns every fixture by design
* — it invalidates the recorded responses; a prompt/schema edit churns none —
* replay never reads this content, see dsh-llm-replay).
*
* Only lines with something to scrub are re-serialized; every other line
* passes through byte-for-byte, so the transform is idempotent and applying
* it to an already-scrubbed fixture is a no-op — the on-disk-fixtures guard
* in acp.snapshot.ts relies on exactly that.
*/
export function scrubRequestHeaders(rawLog: string): string {
const lines = rawLog.split('\n')
const out = lines.map((line) => {
if (line.trim().length === 0) return line
const record = JSON.parse(line) as Record<string, unknown>
const data = record.data as Record<string, unknown> | null | undefined
if (data === null || typeof data !== 'object') return line
if (record.type === 'request/header') {
const header = data.header as Record<string, unknown> | null | undefined
if (header === null || typeof header !== 'object') return line
if (!('system' in header) && !('tools' in header)) return line
if ('system' in header) header.system = SYSTEM
if ('tools' in header) header.tools = TOOLS
return JSON.stringify(record)
}
if (record.type === 'request/header-delta') {
let touched = false
const system = data.system as Record<string, unknown> | null | undefined
if (system !== null && typeof system === 'object' && Array.isArray(system.insert)) {
system.insert = system.insert.map(() => SYSTEM)
touched = true
}
const tools = data.tools as Record<string, unknown> | null | undefined
if (tools !== null && typeof tools === 'object') {
if (Array.isArray(tools.added)) { tools.added = tools.added.map(scrubToolSchema); touched = true }
if (Array.isArray(tools.changed)) { tools.changed = tools.changed.map(scrubToolSchema); touched = true }
}
return touched ? JSON.stringify(record) : line
}
return line
})
return out.join('\n')
}
/** Tokenize one tool schema's bulk (description, parameters, anything else), keeping its identifying `name`. */
function scrubToolSchema(tool: unknown): unknown {
if (tool === null || typeof tool !== 'object' || Array.isArray(tool)) return tool
const out: Record<string, unknown> = {}
for (const [k, v] of Object.entries(tool)) out[k] = k === 'name' ? v : TOOLS
return out
}

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

File diff suppressed because one or more lines are too long

View File

@@ -6,7 +6,7 @@ Its consumer is the ACP snapshot harness in `examples/acp-agent`, which loads th
## How the fixture works
The fixture IS the persisted session log (`<scenario>/session.jsonl`). Its `assistant/chunk` events carry every `StreamChunk`, so grouping them by `(turn, step)` reconstructs each `stream()` call's chunk sequence (one model call per loop step). Recording is therefore "run the real agent once and harvest the `.jsonl`", done by the snapshot harness — this plugin does not record.
The fixture IS the persisted session log (`<scenario>/session.jsonl`). Its `assistant/chunk` events carry every `StreamChunk`, so grouping them by `(turn, step)` reconstructs each `stream()` call's chunk sequence (one model call per loop step). Recording is therefore "run the real agent once and harvest the `.jsonl`", done by the snapshot harness — this plugin does not record. A fixture may carry its `request/header` content tokenized to `{{system}}`/`{{tools}}` (the harness pins that content in one scenario and scrubs the rest); replay is indifferent — derivation reads only `assistant/chunk` events and the line-0 session header.
Two failure modes are not reconstructable from `assistant/chunk` alone — a pure throw before any chunk (e.g. an HTTP 401, where the log holds only a `turn/end {error}` and no chunks) and a cancel/hang (timing, not chunk content). A scenario that needs those supplies an optional sidecar (`<scenario>/replay.override.json`: a `ReplayEntry[]`) that REPLACES the derived script.

View File

@@ -12,7 +12,11 @@
* `(turn, step)` reconstructs each `stream()` call's chunk sequence (one model
* call per loop step — see packages/core/agent-loop/src/loop.ts). Recording is
* therefore "run the real agent once and harvest the `.jsonl`", done by the
* snapshot harness — this plugin does not record.
* snapshot harness — this plugin does not record. A fixture may carry its
* `request/header` content tokenized to `{{system}}`/`{{tools}}` (the harness
* pins that content in one scenario and scrubs the rest); replay is
* indifferent — derivation reads ONLY `assistant/chunk` events and the line-0
* session header.
*
* A NESTED-agent scenario records more than one log: the parent plus one per
* in-process subagent (each subagent runs as its own {@link Session} on the same