Adds docs/rfc/implemented/2026-06-19-real-api-e2e-ci.md covering the rationale
for running the real-API e2e suite in a separate secret-consuming workflow, the
fork/Dependabot/secret threat model, the residual exposure of the pull_request
trigger, and what changes when the repo goes public. Indexes it in the RFC
README.
Also adds a SECURITY comment on the pull_request trigger forbidding a switch to
pull_request_target (an untrusted-code-with-secrets leak vector, especially once
public), pointing at the RFC.
- Rewrite the snapshot-test RFC's replay-plugin section to state current
reality directly (the plugin is the @deepseek-ai/dsh-llm-replay package,
under the coverage gate) instead of keeping the old example-local text with a
"superseded" note bolted on.
- Add docs/rfc/implemented/AGENTS.md (+ CLAUDE.md symlink): an implemented RFC
must be kept current with what actually shipped — update paths/names/structure
in the same change that moves the code, in place, not as an append-only
changelog of its own drift. A reversal of the DECISION is still a new RFC.
- Reconcile docs/rfc/README.md: the "never edited into a different decision"
rule now distinguishes tracking where a decision lives (required) from
flipping the decision (forbidden), and the implemented/ bullet points at the
new convention.
Round 1 of Codex review on the extraction PR.
- (B) EOF-exit race: the 200ms flush-then-exit setTimeout was untracked, so a
fiber/HMR dispose within that window could not cancel it and the process
would still exit. Track the handle and clear it in the disposer; coalesce
re-entrant maybeExit() calls onto the one pending timer. Regression tests for
both (dispose-within-window cancels; repeated idle schedules once).
- (B/doc) ui-stdio rendering is global, not scoped by config.agent (faithful to
the original copies — agent scopes only input + the EOF-exit gate). Corrected
the README + Config JSDoc, which overclaimed "drive and render".
- (C) createStdioChat is exported and driven directly by tests/programmatic
callers that bypass schemastery validation, so default welcome/agent in the
helper (?? 'ready.'/'main') instead of trusting the cast. Test for empty config.
- (A/doc) docs/rfc/.../acp-snapshot-tests.md asserted the replay plugin
deliberately stays in examples/ ("don't split preemptively") — now false since
this PR packages it. Added a superseding note with the why (coverage gate).
All gates green: typecheck, lint, test:coverage (891, 100%), doc-sync,
test:e2e (6 keyless pass), test:snapshot unaffected.
Adds .github/workflows/e2e.yml, which runs `pnpm run test:e2e` against the
external DeepSeek API (https://api.deepseek.com) using a DEEPSEEK_API_KEY repo
secret. ci.yml stays keyless/forkable; this is a separate, secret-consuming
workflow that fills the gap of nothing in CI exercising the with-key suites.
- Triggers: workflow_dispatch + push to main/master + nightly schedule +
pull_request. A job-level `if:` skips untrusted PRs (forks + Dependabot, both
keyless), keying the Dependabot test on the PR author (pull_request.user.login)
not github.actor. A job-level skip reports as success, so this is safe as a
required check.
- Unconditional preflight hard-fails on a missing secret so the self-skipping
suite can't report a false green when the key is misconfigured.
- Secret scoped to the preflight + e2e steps only; permissions: contents: read;
DEEPSEEK_BASE_URL pinned to the external API; single Node 24; timeout 45m;
cancel-in-progress only for PR runs.
Plan converged with Codex (gpt-5.5:xhigh) over 3 review rounds.
Logic that lived under examples/ was outside the per-file 100% coverage
gate (examples/ are not workspaces) and, in the stdio-UI case, duplicated
across two examples. Move it into packages/ so it is gated and de-duped.
- packages/ui-stdio (new): unify the two diverged stdio-chat.ts copies into
one @deepseek-ai/dsh-ui-stdio plugin (welcome/agent Config). A test-only
I/O seam (createStdioChat(ctx, config, runtime)) keeps process streams out
of the serializable config and makes every render/EOF/disposal branch
unit-testable. Per-file 100%. echo/coding cordis.yml now load the package;
both src/stdio-chat.ts deleted.
- packages/llm-replay (new): move examples/acp-agent/src/llm-replay.ts (+ its
spec) here so its derive/parse/replay branches fall under the coverage gate.
cordis.snapshot.yml + README rewired to the package name; added apply/env
/assertNever/abort tests to reach per-file 100%.
- examples/{echo,coding}-agent: keyless Loader-path e2e smokes that boot the
real cordis.yml (no key) — the guard a hand-mounted unit test cannot be for
the unwrapExports/export-shape class (postmortem 0001). examples/AGENTS.md
codifies the keyless+with-key smoke convention (keyless-by-nature exception
for echo-agent).
- AGENTS.md: a scoped, removal-triggered pre-release stance (foundation over
blast radius). packages/README.md: new rows + a FIXME to later regroup ALL
packages into a hierarchy. Wiring: tsconfig paths/refs, publint, knip,
module-graph.
Verified: typecheck, lint, test:coverage (887 tests, 100%), build, hygiene,
doc-sync, test:snapshot (10), test:e2e (6 keyless pass, with-key self-skip).
Rename the concrete Agent class to make its ReAct-style reasoning loop
explicit in the name. Package name, default-export plugin (`AgentLoop`),
and the `ctx.agentLoop` service key are unchanged.
Master's "fix(acp): align prompt and workspace contracts" made session/new
reject a non-empty additionalDirectories / mcpServers (widening the workspace
scope is unimplemented). Add a `reject-extra-dirs` scenario + a
`newSessionExpectError` input op that pins this editor-facing contract: the
bridge answers with `-32602 Invalid params: additionalDirectories is not
supported`. Keyless, deterministic, no model call.
(session/load replay — the other new master behavior — needs a two-phase
seed-then-load harness and is left for a focused follow-up.)
Master's "fix(acp): align prompt and workspace contracts" changed the editor-
facing transcript — `user/message` → `user_message_chunk` is now emitted during
session/load replay ONLY, not live streaming, so a live prompt no longer echoes
the user message back. The snapshot tier caught this (6 goldens shifted); this
re-records the four recorded scenarios against the API and re-accepts the two
authored goldens so they reflect the merged behavior. Full suite green and
deterministic; this is the tier working as designed.
Establishes the standard way to give a snapshot scenario a non-empty starting
workspace: an optional `<scenario>/workspace/` directory whose contents the
harness copies into the temp cwd before the run (for both record and replay),
so the agent's bash tools see the seeded files. The cwd is normalized in the
goldens, so seeded paths stay stable.
The new `workspace-edit` scenario demonstrates the full read→write→verify cycle
on a seeded file: it ships `workspace/greeting.txt` ("hello"), prompts the agent
to append a WORLD line and cat it back. The recorded log captures the real bash
edits (`echo WORLD >> greeting.txt`, then `cat` showing `hello\nWORLD`), and it
replays deterministically with no key.
Also hardens runScenario teardown (Codex review): workspace seeding and spawn
now run inside the try whose finally removes both temp dirs, so a seeding/spawn
failure can't leak them. Documents the convention in the RFC + example README.
The snapshot replay config duplicated most of base.yml + the acp tail just to
swap llm-deepseek → llm-replay. Factor the shared pieces:
- examples/base-core.yml: the providerless provider/tool core (llm, sessions,
system-prompt, tools, agents, invariants, bash-local, tool-bash). base.yml is
now base-core + the llm-deepseek adapter; the snapshot replay config is
base-core + llm-replay. The replay config no longer hand-copies the core.
- examples/acp-agent/acp-tail.yml: agent-loop (no pre-created agents) +
persistence + the ACP bridge/system-prompt, shared by cordis.yml and the
replay config so the three acp-agent configs can't drift. Its persistence root
is `$DSH_SNAPSHOT_SESSIONS_ROOT ?? ./.sessions`.
- Deleted cordis.snapshot-record.yml: recording now reuses the normal cordis.yml
(real adapter), with the harness redirecting the persistence root via env.
start.ts maps DSH_SNAPSHOT=record → cordis.yml.
Verified: snapshot replay 8/8 keyless; record path works through cordis.yml;
ACP e2e no-key boot green through the doubly-nested include (cordis.yml →
base.yml → base-core.yml); coding-agent boots clean; all gates pass.
The goldens now mirror the shape of the surfaces they capture — one compact
JSON record per line — matching the wire (NDJSON stdout) and disk (JSONL
session log) formats, renamed *.golden.jsonl. They stay grep/jq-able and
faithful to what the agent emits, where the prior pretty-printed .txt was a
reformatted representation. Both normalizers drop the 2-space indent; the
normalizer spec asserts the compact form. All 11 goldens regenerated; replay
remains deterministic (8/8 across runs).
`pnpm run test:snapshot -- -t <name>` silently runs ALL scenarios: pnpm
forwards `--` literally and vitest treats everything after it as positional
filename filters, so `-t` is ignored. The working form drops the separator —
`pnpm run test:snapshot -t <name>` (and `-u` likewise).
Holistic-review fixes for integration gaps the per-commit reviews missed:
- CI now runs `pnpm run test:snapshot` (a step after the coverage gate). It was
wired into pre-push but not .github/workflows/ci.yml, so the RFC/AGENTS claim
that snapshot replay runs in the default PR gate was only half-true — CI is
the real gate.
- vitest.snapshot.config.ts loads the repo .env ONLY when DSH_SNAPSHOT=record.
Loading it unconditionally contradicted the replay safety story (replay must
never reach the network), and runScenario forwards process.env to the child.
Non-ENOENT load errors now surface instead of being swallowed.
- start.ts: the graceful-shutdown comment said "RECORD runs" but the path
applies to both snapshot modes (replay also closes stdin → dispose → exit).
- docs/development.md: list the new pre-push snapshot job and the CI snapshot
gate.
Documents the snapshot tier and makes its use a convention. AGENTS.md gains the
test:snapshot / test:snapshot:record commands (and corrects the now-stale
`pnpm run test` include comment — the unit suite picks up examples/*/tests too),
plus a Conventions bullet: a change affecting the editor-facing transcript or
end-to-end agent UX needs a snapshot test (or an explicit note why none
applies); a pure internal refactor is exempt. The dsh-code-review skill gains a
matching reviewer-only check — review the golden diff itself, since a changed
*.golden is a behavior change in disguise.
Adds the first cut of snapshot scenarios, each asserting a normalized stdout
transcript golden and (for model turns) a re-persisted session-log golden:
- text-turn, tool-call-turn, multi-turn: RECORDED against the real API — the
committed session.jsonl is a genuine harvested log; replay derives the model
script from it and reproduces deterministically with no key. tool-call-turn
exercises the real bash executor (echo SNAPSHOT_OK → tool/call + tool/result
+ a post-tool answer step).
- error-finish, cancel: AUTHORED via a replay.override.json sidecar (the live
API can't be coaxed into a deterministic 401 or mid-stream cancel). error-
finish replays a {kind:throw} 401 → the bridge answers the prompt with a
JSON-RPC error and the log records turn/end{kind:error}; cancel replays a
{kind:hang} → stopReason:cancelled.
Two input-DSL ops support these: promptExpectError (awaits the prompt, asserts
it rejects — the editor's view of a failed turn — and swallows it) and
promptAndCancel (dispatches the prompt unawaited, waits until the client
OBSERVES the streamed agent_message_chunk, then cancels — pinning frame order
so the cancel transcript is deterministic; fixes a flake Codex caught where the
late chunk and the cancelled response could interleave either way).
Scenarios carry a `recorded` flag so test:snapshot:record only re-runs the
live-API ones. reasoning/max-tokens scenarios are deferred (hard to force
deterministically from the live model). Per docs/rfc/implemented/2026-06-19.
Adds the snapshot-test harness and the keyless replay pipeline end-to-end.
- snapshot-harness.ts: boots the real acp-agent subprocess via the cordis
Loader (preserving TSX_TSCONFIG_PATH so unbuilt dsh-* imports resolve from a
temp cwd), tees raw stdout into an SDK ClientSideConnection, interprets a
per-scenario input.json DSL (initialize / newSession capturing the random
sessionId / prompt / cancel), closes stdin to trigger graceful shutdown, and
harvests the persisted session.jsonl. Failure-safe: a finally block SIGKILLs
a live child, awaits its exit, and removes both temp dirs even on a thrown
step or harvest. Raw bytes are buffered and decoded once (no multibyte split).
- snapshot-normalize.ts (+ spec): two pure normalizers (stdout frames + session
JSONL) scrub cwd, session ids / UUIDs, and JSON-RPC ids, and zero time /
createdAt — but keep `seq` (deterministic by contract). normalizeStdout throws
on a non-JSON line (the stdout-purity check).
- start.ts: selects cordis.snapshot.yml (replay, providerless) or
cordis.snapshot-record.yml (record, real adapter) from DSH_SNAPSHOT, skips
.env in replay, and disposes the ctx on stdin end so persistence flushes
before exit (harvest-after-flush, not on the prompt response).
- acp.snapshot.ts: asserts the normalized stdout golden (and, for model
scenarios, the re-persisted JSONL golden) via toMatchFileSnapshot; record mode
writes the harvested log back to the scenario fixture; an orphan-fixture guard
fails on an unregistered scenario dir.
- handshake scenario: initialize + session/new (no model call; a header-only
session.jsonl, since session/new persists no events).
- vitest.snapshot.config.ts, test:snapshot / test:snapshot:record scripts, a
pre-push snapshot job, and the knip entry.
Incorporates Codex review: record-fixture writeback, failure-safe teardown,
seq-not-scrubbed, harvest-after-flush. Per docs/rfc/implemented/2026-06-19.
Per a design revision, the per-scenario snapshot fixture becomes EXACTLY the
persisted session JSONL (<scenario>/session.jsonl) rather than a hand-authored
llm.json. The log already holds all LLM behavior (assistant/chunk carries every
StreamChunk) AND all harness behavior (tool/call, tool/result, turn/*, usage),
so one artifact drives replay and doubles as a behavioral golden.
llm-replay becomes replay-only (the record-tee is removed; recording is now
"run the real agent once and harvest the .jsonl", done by the harness in a
later commit). deriveReplayScript(events) groups assistant/chunk by (turn,step)
in log order — exact because the loop makes one ctx.llm.stream() call per step
and tags each chunk with the current (turn,step). The two failure modes the log
can't express (a thrown stream — no terminal finish; cancel/hang — timing) use
an optional replay.override.json sidecar.
Hardens against a Codex review finding: a derived group is only valid if it
ends in a `finish` chunk. A group without one is the fingerprint of a thrown
stream() and is NOT silently replayed as a clean stop — deriveReplayScript
throws, naming the (turn,step), so a missing sidecar override fails loud.
Updates the unit tests (parse/derive/load helpers, sidecar override, finish-
terminated grouping, HMR), the example README, and the RFC prose to the JSONL
format. Two goldens (stdout transcript + re-persisted JSONL) and the harness
wiring land in the next commit.
Introduces examples/acp-agent/src/llm-replay.ts, a function/namespace plugin
that installs a single llm/stream waterfall listener. In record mode it tees
the real model's StreamChunks into a per-scenario llm.json (flushed atomically
after EACH stream, since the snapshot subprocess is SIGKILLed and start.ts has
no disposal path). In replay mode it short-circuits the waterfall and serves
recorded streams back positionally — the Nth stream() call gets the Nth entry —
so a snapshot test can drive the real agent with no API key.
Each fixture entry is a discriminated record {chunks|throw|hang} so it can
replay BOTH branches of the LLM failure contract (throw from stream() vs a
finish-error chunk) plus cancellation. A throw entry carries the prefix chunks
emitted before the throw, replayed before the error, so a mid-stream failure
(partial output then STREAM_CLOSED) reproduces what the loop saw live.
Fail-loud on a missing or exhausted fixture (never a silent skip). Unit tests
drive the real LlmService waterfall (record tee, ordered replay, the three
entry kinds, partial-then-throw, fail-loud, event-driven abort, HMR-safety).
Broadens the unit vitest include to examples/*/tests and registers the plugin
+ snapshot tests as knip entries. Per docs/rfc/implemented/2026-06-19.
Records the decision to add a third test tier: snapshot tests that boot the
real acp-agent subprocess over ACP stdio, record the LLM's streamed responses
once against the real API, then replay them deterministically so the full
stdout transcript can be diffed against a committed golden — keyless in CI.
Captures the design choices hardened in a Codex (xhigh) review: record at the
provider-neutral llm/stream waterfall; a discriminated fixture entry schema
(chunks/throw/hang) that honors both LLM failure branches; positional replay
with a one-in-flight-stream constraint; per-stream atomic fixture flush (the
subprocess is SIGKILLed, so dispose-time flush would never run); a providerless
replay config; normalize-then-snapshot parsed frames; normalization over an OS
sandbox now with the rootless bwrap/sandbox-exec tier reserved via the
BashExecutor capability seam. Cross-links the proposed determinism RFC
(complementary: internal history invariant vs external protocol contract).
