mirror of
https://github.com/deepseek-ai/deepseek-harness
synced 2026-08-15 21:04:50 +00:00
1fd9e2a9ab
Every session.jsonl fixture embedded the full composed system prompt and
complete tool-schema list in its request/header event (~8 KB on one line,
identical across the suite), so any prompt or tool-schema edit forced a
re-record or hand-edit of every fixture — see the dynamic-workflows PR for
the churn pattern this removes.
Now exactly one scenario (text-turn, flagged pinsHeader) commits and
compares that content verbatim; every other fixture stores and compares it
as {{system}}/{{tools}} tokens via the new pure scrubRequestHeaders
normalizer (applied to both compare sides and to record-mode writes, so a
re-record cannot reintroduce the content). request/header-delta payloads
are scrubbed the same way; config/reason stay verbatim — a model swap
SHOULD churn every fixture, a prompt edit should not. Replay is unaffected:
script derivation reads only assistant/chunk events.
Fixture meta-guards enforce the split: non-pinning fixtures must be fixed
points of the scrub, the pinning fixture must not be, and exactly one
scenario pins. Committed fixtures migrated through the same function.
Docs: pinned-header RFC (implemented/testing), base snapshot RFC + testing
policy + llm-replay module doc/README updated.
8 lines
819 B
JSON
8 lines
819 B
JSON
{"type":"session","version":0,"id":"{{sessionId}}","createdAt":0,"cwd":"{{cwd}}"}
|
|
{"type":"turn/start","seq":0,"time":0,"data":{"turn":1,"trigger":{"kind":"message","source":{"kind":"user"}}}}
|
|
{"type":"user/message","seq":1,"time":0,"data":{"content":[{"type":"text","text":"This prompt triggers a recorded provider error."}],"source":{"kind":"user"}},"surfaceOp":"append"}
|
|
{"type":"step/start","seq":2,"time":0,"data":{"turn":1,"step":1}}
|
|
{"type":"request/header","seq":3,"time":0,"data":{"header":{"config":{"model":"deepseek-v4-flash"},"system":"{{system}}","tools":"{{tools}}"},"reason":"initial"}}
|
|
{"type":"step/end","seq":4,"time":0,"data":{"turn":1,"step":1}}
|
|
{"type":"turn/end","seq":5,"time":0,"data":{"turn":1,"reason":{"kind":"error","step":1,"message":"simulated provider error (HTTP 401)","code":"AUTH"}}}
|