Adds .github/workflows/e2e.yml, which runs `pnpm run test:e2e` against the
external DeepSeek API (https://api.deepseek.com) using a DEEPSEEK_API_KEY repo
secret. ci.yml stays keyless/forkable; this is a separate, secret-consuming
workflow that fills the gap of nothing in CI exercising the with-key suites.
- Triggers: workflow_dispatch + push to main/master + nightly schedule +
pull_request. A job-level `if:` skips untrusted PRs (forks + Dependabot, both
keyless), keying the Dependabot test on the PR author (pull_request.user.login)
not github.actor. A job-level skip reports as success, so this is safe as a
required check.
- Unconditional preflight hard-fails on a missing secret so the self-skipping
suite can't report a false green when the key is misconfigured.
- Secret scoped to the preflight + e2e steps only; permissions: contents: read;
DEEPSEEK_BASE_URL pinned to the external API; single Node 24; timeout 45m;
cancel-in-progress only for PR runs.
Plan converged with Codex (gpt-5.5:xhigh) over 3 review rounds.
Rename the concrete Agent class to make its ReAct-style reasoning loop
explicit in the name. Package name, default-export plugin (`AgentLoop`),
and the `ctx.agentLoop` service key are unchanged.
Master's "fix(acp): align prompt and workspace contracts" made session/new
reject a non-empty additionalDirectories / mcpServers (widening the workspace
scope is unimplemented). Add a `reject-extra-dirs` scenario + a
`newSessionExpectError` input op that pins this editor-facing contract: the
bridge answers with `-32602 Invalid params: additionalDirectories is not
supported`. Keyless, deterministic, no model call.
(session/load replay — the other new master behavior — needs a two-phase
seed-then-load harness and is left for a focused follow-up.)
Master's "fix(acp): align prompt and workspace contracts" changed the editor-
facing transcript — `user/message` → `user_message_chunk` is now emitted during
session/load replay ONLY, not live streaming, so a live prompt no longer echoes
the user message back. The snapshot tier caught this (6 goldens shifted); this
re-records the four recorded scenarios against the API and re-accepts the two
authored goldens so they reflect the merged behavior. Full suite green and
deterministic; this is the tier working as designed.
Establishes the standard way to give a snapshot scenario a non-empty starting
workspace: an optional `<scenario>/workspace/` directory whose contents the
harness copies into the temp cwd before the run (for both record and replay),
so the agent's bash tools see the seeded files. The cwd is normalized in the
goldens, so seeded paths stay stable.
The new `workspace-edit` scenario demonstrates the full read→write→verify cycle
on a seeded file: it ships `workspace/greeting.txt` ("hello"), prompts the agent
to append a WORLD line and cat it back. The recorded log captures the real bash
edits (`echo WORLD >> greeting.txt`, then `cat` showing `hello\nWORLD`), and it
replays deterministically with no key.
Also hardens runScenario teardown (Codex review): workspace seeding and spawn
now run inside the try whose finally removes both temp dirs, so a seeding/spawn
failure can't leak them. Documents the convention in the RFC + example README.
The snapshot replay config duplicated most of base.yml + the acp tail just to
swap llm-deepseek → llm-replay. Factor the shared pieces:
- examples/base-core.yml: the providerless provider/tool core (llm, sessions,
system-prompt, tools, agents, invariants, bash-local, tool-bash). base.yml is
now base-core + the llm-deepseek adapter; the snapshot replay config is
base-core + llm-replay. The replay config no longer hand-copies the core.
- examples/acp-agent/acp-tail.yml: agent-loop (no pre-created agents) +
persistence + the ACP bridge/system-prompt, shared by cordis.yml and the
replay config so the three acp-agent configs can't drift. Its persistence root
is `$DSH_SNAPSHOT_SESSIONS_ROOT ?? ./.sessions`.
- Deleted cordis.snapshot-record.yml: recording now reuses the normal cordis.yml
(real adapter), with the harness redirecting the persistence root via env.
start.ts maps DSH_SNAPSHOT=record → cordis.yml.
Verified: snapshot replay 8/8 keyless; record path works through cordis.yml;
ACP e2e no-key boot green through the doubly-nested include (cordis.yml →
base.yml → base-core.yml); coding-agent boots clean; all gates pass.
The goldens now mirror the shape of the surfaces they capture — one compact
JSON record per line — matching the wire (NDJSON stdout) and disk (JSONL
session log) formats, renamed *.golden.jsonl. They stay grep/jq-able and
faithful to what the agent emits, where the prior pretty-printed .txt was a
reformatted representation. Both normalizers drop the 2-space indent; the
normalizer spec asserts the compact form. All 11 goldens regenerated; replay
remains deterministic (8/8 across runs).
`pnpm run test:snapshot -- -t <name>` silently runs ALL scenarios: pnpm
forwards `--` literally and vitest treats everything after it as positional
filename filters, so `-t` is ignored. The working form drops the separator —
`pnpm run test:snapshot -t <name>` (and `-u` likewise).
Holistic-review fixes for integration gaps the per-commit reviews missed:
- CI now runs `pnpm run test:snapshot` (a step after the coverage gate). It was
wired into pre-push but not .github/workflows/ci.yml, so the RFC/AGENTS claim
that snapshot replay runs in the default PR gate was only half-true — CI is
the real gate.
- vitest.snapshot.config.ts loads the repo .env ONLY when DSH_SNAPSHOT=record.
Loading it unconditionally contradicted the replay safety story (replay must
never reach the network), and runScenario forwards process.env to the child.
Non-ENOENT load errors now surface instead of being swallowed.
- start.ts: the graceful-shutdown comment said "RECORD runs" but the path
applies to both snapshot modes (replay also closes stdin → dispose → exit).
- docs/development.md: list the new pre-push snapshot job and the CI snapshot
gate.
Documents the snapshot tier and makes its use a convention. AGENTS.md gains the
test:snapshot / test:snapshot:record commands (and corrects the now-stale
`pnpm run test` include comment — the unit suite picks up examples/*/tests too),
plus a Conventions bullet: a change affecting the editor-facing transcript or
end-to-end agent UX needs a snapshot test (or an explicit note why none
applies); a pure internal refactor is exempt. The dsh-code-review skill gains a
matching reviewer-only check — review the golden diff itself, since a changed
*.golden is a behavior change in disguise.
Adds the first cut of snapshot scenarios, each asserting a normalized stdout
transcript golden and (for model turns) a re-persisted session-log golden:
- text-turn, tool-call-turn, multi-turn: RECORDED against the real API — the
committed session.jsonl is a genuine harvested log; replay derives the model
script from it and reproduces deterministically with no key. tool-call-turn
exercises the real bash executor (echo SNAPSHOT_OK → tool/call + tool/result
+ a post-tool answer step).
- error-finish, cancel: AUTHORED via a replay.override.json sidecar (the live
API can't be coaxed into a deterministic 401 or mid-stream cancel). error-
finish replays a {kind:throw} 401 → the bridge answers the prompt with a
JSON-RPC error and the log records turn/end{kind:error}; cancel replays a
{kind:hang} → stopReason:cancelled.
Two input-DSL ops support these: promptExpectError (awaits the prompt, asserts
it rejects — the editor's view of a failed turn — and swallows it) and
promptAndCancel (dispatches the prompt unawaited, waits until the client
OBSERVES the streamed agent_message_chunk, then cancels — pinning frame order
so the cancel transcript is deterministic; fixes a flake Codex caught where the
late chunk and the cancelled response could interleave either way).
Scenarios carry a `recorded` flag so test:snapshot:record only re-runs the
live-API ones. reasoning/max-tokens scenarios are deferred (hard to force
deterministically from the live model). Per docs/rfc/implemented/2026-06-19.
Adds the snapshot-test harness and the keyless replay pipeline end-to-end.
- snapshot-harness.ts: boots the real acp-agent subprocess via the cordis
Loader (preserving TSX_TSCONFIG_PATH so unbuilt dsh-* imports resolve from a
temp cwd), tees raw stdout into an SDK ClientSideConnection, interprets a
per-scenario input.json DSL (initialize / newSession capturing the random
sessionId / prompt / cancel), closes stdin to trigger graceful shutdown, and
harvests the persisted session.jsonl. Failure-safe: a finally block SIGKILLs
a live child, awaits its exit, and removes both temp dirs even on a thrown
step or harvest. Raw bytes are buffered and decoded once (no multibyte split).
- snapshot-normalize.ts (+ spec): two pure normalizers (stdout frames + session
JSONL) scrub cwd, session ids / UUIDs, and JSON-RPC ids, and zero time /
createdAt — but keep `seq` (deterministic by contract). normalizeStdout throws
on a non-JSON line (the stdout-purity check).
- start.ts: selects cordis.snapshot.yml (replay, providerless) or
cordis.snapshot-record.yml (record, real adapter) from DSH_SNAPSHOT, skips
.env in replay, and disposes the ctx on stdin end so persistence flushes
before exit (harvest-after-flush, not on the prompt response).
- acp.snapshot.ts: asserts the normalized stdout golden (and, for model
scenarios, the re-persisted JSONL golden) via toMatchFileSnapshot; record mode
writes the harvested log back to the scenario fixture; an orphan-fixture guard
fails on an unregistered scenario dir.
- handshake scenario: initialize + session/new (no model call; a header-only
session.jsonl, since session/new persists no events).
- vitest.snapshot.config.ts, test:snapshot / test:snapshot:record scripts, a
pre-push snapshot job, and the knip entry.
Incorporates Codex review: record-fixture writeback, failure-safe teardown,
seq-not-scrubbed, harvest-after-flush. Per docs/rfc/implemented/2026-06-19.
Per a design revision, the per-scenario snapshot fixture becomes EXACTLY the
persisted session JSONL (<scenario>/session.jsonl) rather than a hand-authored
llm.json. The log already holds all LLM behavior (assistant/chunk carries every
StreamChunk) AND all harness behavior (tool/call, tool/result, turn/*, usage),
so one artifact drives replay and doubles as a behavioral golden.
llm-replay becomes replay-only (the record-tee is removed; recording is now
"run the real agent once and harvest the .jsonl", done by the harness in a
later commit). deriveReplayScript(events) groups assistant/chunk by (turn,step)
in log order — exact because the loop makes one ctx.llm.stream() call per step
and tags each chunk with the current (turn,step). The two failure modes the log
can't express (a thrown stream — no terminal finish; cancel/hang — timing) use
an optional replay.override.json sidecar.
Hardens against a Codex review finding: a derived group is only valid if it
ends in a `finish` chunk. A group without one is the fingerprint of a thrown
stream() and is NOT silently replayed as a clean stop — deriveReplayScript
throws, naming the (turn,step), so a missing sidecar override fails loud.
Updates the unit tests (parse/derive/load helpers, sidecar override, finish-
terminated grouping, HMR), the example README, and the RFC prose to the JSONL
format. Two goldens (stdout transcript + re-persisted JSONL) and the harness
wiring land in the next commit.
Introduces examples/acp-agent/src/llm-replay.ts, a function/namespace plugin
that installs a single llm/stream waterfall listener. In record mode it tees
the real model's StreamChunks into a per-scenario llm.json (flushed atomically
after EACH stream, since the snapshot subprocess is SIGKILLed and start.ts has
no disposal path). In replay mode it short-circuits the waterfall and serves
recorded streams back positionally — the Nth stream() call gets the Nth entry —
so a snapshot test can drive the real agent with no API key.
Each fixture entry is a discriminated record {chunks|throw|hang} so it can
replay BOTH branches of the LLM failure contract (throw from stream() vs a
finish-error chunk) plus cancellation. A throw entry carries the prefix chunks
emitted before the throw, replayed before the error, so a mid-stream failure
(partial output then STREAM_CLOSED) reproduces what the loop saw live.
Fail-loud on a missing or exhausted fixture (never a silent skip). Unit tests
drive the real LlmService waterfall (record tee, ordered replay, the three
entry kinds, partial-then-throw, fail-loud, event-driven abort, HMR-safety).
Broadens the unit vitest include to examples/*/tests and registers the plugin
+ snapshot tests as knip entries. Per docs/rfc/implemented/2026-06-19.
Records the decision to add a third test tier: snapshot tests that boot the
real acp-agent subprocess over ACP stdio, record the LLM's streamed responses
once against the real API, then replay them deterministically so the full
stdout transcript can be diffed against a committed golden — keyless in CI.
Captures the design choices hardened in a Codex (xhigh) review: record at the
provider-neutral llm/stream waterfall; a discriminated fixture entry schema
(chunks/throw/hang) that honors both LLM failure branches; positional replay
with a one-in-flight-stream constraint; per-stream atomic fixture flush (the
subprocess is SIGKILLed, so dispose-time flush would never run); a providerless
replay config; normalize-then-snapshot parsed frames; normalization over an OS
sandbox now with the rootless bwrap/sandbox-exec tier reserved via the
BashExecutor capability seam. Cross-links the proposed determinism RFC
(complementary: internal history invariant vs external protocol contract).
Merging after clean merge from master + clean Codex CLI review. Reclassifies 4 non-vendor TODOs that were tagged too aggressively:
- XXX(stateful-shell): design reference, current spawn-per-call works fine
- XXX(tool-bash-owner-hmr) (3x): HMR-only dev issue, acceptable today
All other TODOs kept as-is. No vendor changes.
The ToolCallPresentation / ToolResultPresentation / ToolTerminal shapes grew
incrementally and the responsibility split is now muddy (overlapping call/result
terminal fields, the bridge stitching content + terminal + rawInput per call).
Flag it as a release-blocking FIXME to redesign around a tool's render INTENT
(a tagged union over card kinds) and pin it in an RFC before more tools/UIs
depend on the current bag-of-optionals.
Review nit: the KNOWN RESIDUAL comment named only the [exit code: N] case, but
the same end-of-string spoof applies to [killed by signal: SIG]. Reword to cover
both markers.
Codex + an independent review pass found three real defects in the prior commit:
1. parseExitStatus could misreport a SUCCESSFUL command as a failure: a clean
exit 0 appends no marker, so output ending in "[exit code: 5]" (no trailing
newline) was read as the marker. Anchor the parse to a LEADING newline —
renderResult always inserts one before a real marker, so a body that merely
ends in marker-like text no longer matches. A narrow residual (a clean exit 0
whose final line is exactly the marker) is inherent to the replay-only-sees-
text design and documented; the complete fix (a structured exit on the event)
is the RFC's named escape hatch.
2. A run_in_background start and an isError result were rendered as exited
terminal cards with a false exit-0 pill. A background start returns a task-id
ack (not a streamed terminal) and is no longer marked terminal; an isError
result (spawn failure / abort) carries no exit pill.
3. The terminal capability was re-read live on the result path, so a second
initialize between a call and its result could desync them (orphan
terminal_output or clobbered card). Snapshot the capability per session at
creation (SessionRecord.terminalEnabled) so call and result always agree.
Also reword the reference-parity claim: keeping the description as a content
block in terminal mode is a DELIBERATE divergence (claude-agent-acp drops it).
Tests added for each; with-key e2e still green.
Match claude-agent-acp / codex-acp: the bash tool_call title IS the command
(an execute card hides rawInput), the model description rides as a content
text block above the card, and the completed card carries an exit-status pill
via _meta.terminal_exit.
Bridge fixes found in review of the prior terminal-card commit:
- tool_call_update.content is OMITTED in terminal mode (an ACP update.content
REPLACES the call's content collection in Zed, so the fenced ```console block
would clobber the terminal content block).
- terminal.output preserves RAW newlines (terminal renderers rely on exact
bytes); only the fenced fallback trims trailing blank lines.
- a relative workdir is resolved against the session cwd for the card header,
matching where the command actually ran.
- result-side terminal output is gated on the pending call having registered a
terminal (no orphan _meta.terminal_output for a terminal Zed never made).
The exit pill is recovered by parsing renderResult's status markers (the pure
presentResult seam sees only content blocks); a round-trip test pins the parse
to the marker emission. Neutral ToolTerminal gains exitCode/signal; widened
ToolCallPresentation with a content block. Docs (RFC + 3 READMEs) updated;
with-key e2e verifies the card + exit pill against the real model.
When the client advertises clientCapabilities._meta.terminal_output (Zed), a
bash tool call now renders as a real TERMINAL card — a cwd header + the command
+ its output — instead of the plain ```console text block. Keeps agent-side
dsh-bash execution; rejects the spec's client-side terminal/create (which would
bypass sandbox/env-scrub/ownership/cwd). Matches what claude-agent-acp and
codex-acp do; wire contract verified against Zed's source.
- dsh-tools: a provider-neutral ToolTerminal shape ({ cwd?, output? }) on
ToolCallPresentation/ToolResultPresentation — a tool asks "render me as a
terminal"; no ACP types leak in.
- dsh-tool-bash: bash presentCall marks terminal (cwd from an explicit absolute
workdir, else left for the bridge to fill from the session cwd); presentResult
carries the output alongside the ```console fallback.
- dsh-acp: initialize reads/remembers the _meta.terminal_output capability;
streamSessionEventUpdate maps a terminal presentation to
content:[{type:'terminal',terminalId}] + _meta.terminal_info on the call and
_meta.terminal_output on the update WHEN capable — else the unchanged text
path. terminalId is the callId; cwd defaults to the session header. The pure
translator gained a TerminalRendering {enabled,cwd} param (off by default).
Tests via the REAL tool-bash + bash-local: capability ON -> terminal content +
_meta; OFF -> no _meta (text path). The with-key e2e adds a real-model terminal
card case (echo over ACP with the capability on). 773 tests, 100% coverage.
The exit-status pill (_meta.terminal_exit), live streaming
(_meta.terminal_output_delta), and command classification are RFC follow-ups.
Records the verified design before implementing: keep dsh-bash agent-side
execution and render Zed's terminal tool-call card via the `_meta` convention
(terminal_info/terminal_output/terminal_exit), capability-gated on
clientCapabilities._meta.terminal_output, with the ```console text block as the
no-capability fallback. Rejects the spec's client-side terminal/create path (it
would bypass dsh-bash's sandbox/env-scrub/ownership/cwd). Studied
claude-agent-acp, codex-acp, and Zed's renderer to ground the wire contract.
Live streaming and command classification are noted as separate follow-ups.
- bash presentCall title is now "description — command" (e.g. "List files in
src — ls -la src"). An execute-kind ACP card HIDES rawInput (Zed renders it
only for non-terminal tools), so the command must ride in the always-visible
title to be seen — matching how claude-agent-acp/codex-acp title execute
tools. The command stays in rawInput too for non-execute UIs that show it.
- Rework the acp tool-call presentation tests (turns + load replay) to drive the
REAL dsh-tool-bash + dsh-bash-local via a new makeBridgeHarness({ withBash })
option, running an actual `echo` — instead of an inline fake bash tool. The
mock MODEL still scripts the call (deterministic, no key), but the tool and
executor are real, so the test verifies the shipping presentCall/presentResult.
- AGENTS.md: add the principle "prefer the REAL implementation over a mock/
stand-in in tests" (mock only the expensive/non-deterministic boundary).
- RFC (proposed): the ACP terminal sub-protocol + command classification — the
capability-gated rich rendering (live cwd-header terminal card, classify a
`cat` as a read / `grep` as a search) that the reference adapters do; the
fenced ```console text block stays the no-capability baseline. Studied
codex-acp, claude-agent-acp, and Zed's renderer to ground it.
- schemas() builds the model-facing ToolSchema by EXPLICIT allowlist
({name, description, parameters, strict?}) instead of stripping `execute` —
presentCall/presentResult are functions that must never leak into a model
request, and an allowlist can't drift when a new ToolDefinition member lands.
- session/load replay uses a THROWAWAY ToolPresenter, not record.presenter, so
a historical interrupted-mid-tool turn (tool/call with no tool/result) can't
leave stale in-flight state on the live presenter that serves later events.
- ToolPresenter.call/result contain a throwing presentCall/presentResult: log
via an onError sink and fall back to the generic presentation, so a buggy
display callback can never fail a live turn or a load replay.
- acp README inject list now includes `tools`.
- remove a stray blank line at EOF (git diff --check gate).
Regressions added: schemas() drops presenter callbacks (+ keeps `strict`);
session/load replays a tool call with the tool-owned presentation; a throwing
presenter is contained (direct + through the real bridge) with and without an
onError sink.
