mirror of
https://github.com/deepseek-ai/deepseek-harness
synced 2026-08-15 21:04:50 +00:00
Adds the first cut of snapshot scenarios, each asserting a normalized stdout
transcript golden and (for model turns) a re-persisted session-log golden:
- text-turn, tool-call-turn, multi-turn: RECORDED against the real API — the
committed session.jsonl is a genuine harvested log; replay derives the model
script from it and reproduces deterministically with no key. tool-call-turn
exercises the real bash executor (echo SNAPSHOT_OK → tool/call + tool/result
+ a post-tool answer step).
- error-finish, cancel: AUTHORED via a replay.override.json sidecar (the live
API can't be coaxed into a deterministic 401 or mid-stream cancel). error-
finish replays a {kind:throw} 401 → the bridge answers the prompt with a
JSON-RPC error and the log records turn/end{kind:error}; cancel replays a
{kind:hang} → stopReason:cancelled.
Two input-DSL ops support these: promptExpectError (awaits the prompt, asserts
it rejects — the editor's view of a failed turn — and swallows it) and
promptAndCancel (dispatches the prompt unawaited, waits until the client
OBSERVES the streamed agent_message_chunk, then cancels — pinning frame order
so the cancel transcript is deterministic; fixes a flake Codex caught where the
late chunk and the cancelled response could interleave either way).
Scenarios carry a `recorded` flag so test:snapshot:record only re-runs the
live-API ones. reasoning/max-tokens scenarios are deferred (hard to force
deterministically from the live model). Per docs/rfc/implemented/2026-06-19.
Examples
Runnable demos (not workspaces) that showcase how the harness is wired.
echo-agent
A mock model + echo tool + stdio UI + JSONL persistence demo. Demonstrates:
- Loading plugins from a
cordis.ymlvia@cordisjs/plugin-loader+@cordisjs/plugin-include - Registering a mock
LlmAdapter(streaming scripted responses) - Registering a tool via
ctx.tools.register() - Persisting session events to JSONL via the
session/event+session/flushpattern - A minimal stdio UI consuming
agent/stream-chunkand session events
Run with: pnpm run demo:echo
When prompted, type "echo " to trigger a tool call round-trip.
coding-agent
The real thing: DeepSeek V4 + the bash tool suite + stdio chat + JSONL persistence, wired from cordis.yml. Where echo-agent proves the skeleton with mocks, this is a usable coding assistant.
Run with: pnpm run demo:coding (needs DEEPSEEK_API_KEY in the environment or a gitignored repo-root .env). See coding-agent/README.md for details.