deepseek-harness/docs/adr/0018-session-persistence.md

ADR 0018: Session persistence as an abstract service over the existing SessionEvent

Status: accepted (2026-06-15)

Context

Sessions lived only in memory. The example session-jsonl.ts plugin (duplicated byte-for-byte in both examples) was write-only telemetry: it buffered session/event and appended JSON lines, with no read/replay path, no crash-safety (no fsync, no atomic write, a fire-and-forget dispose drain), no listing, and no format versioning. Nothing could rehydrate a past session from disk into a live agent, so durable resume ("continue yesterday's task"), durable forking, and the ACP session/load method (RFC 010) were all impossible.

The event-sourced model makes the append-only log the single source of truth and derives LLM history from it. Persistence had to stay faithful to that: persist the existing SessionEvent directly, with no parallel "persisted message" type that the log is converted to and from. The backend also had to be swappable — a file store now, a database store later — behind one interface.

Decision

Persistence is an abstract capability seam (ADR 0009, the dsh-bash template), not loop or core logic:

1. Interface (dsh-session-persistence, ctx.sessionPersistence) — an abstract SessionPersistence service: create/append/load/list/has/delete/update. Its persisted unit IS the existing SessionEvent ({ type, seq, time, data }), reused verbatim — no conversion type.
2. Implementation (dsh-session-persistence-jsonl) — an append-only JSONL log per session (a SessionHeader line then one SessionEvent per line, verbatim including assistant/chunk) plus an atomic .summary.json sidecar for the mutable SessionSummary.

Key choices recorded here because they are durable, contested, and surprising:

• The canonical durable log persists every SessionEvent verbatim, including assistant/chunk. deriveMessages() skips chunks, and a chunk-filtered rollout (Codex's policy.rs) is tempting — but seq = log.length and the load-validation events[i].seq === i require a contiguous log; filtering chunks out would leave holes and break both the contract and resume. A chunk-filtered projection is possible later as a derived view with its own renumbering, but it is NOT the canonical log.
• Append-only; a crashed turn is closed, never truncated. Committed events — those at or below a flushed turn/end — are never rewritten. The loop only flushes at turn/end, so a crash can leave a durable log whose final turn never closed: real, fully-written events sit after the last turn/end. A single turn can be huge in a long-horizon task (many steps, large tool output spanning a long autonomous run), so discarding the interrupted turn would silently destroy a large amount of real work — truncating a turn is wrong. Instead, on reload load PRESERVES those events and CLOSES the orphaned turn by durably appending the minimal synthetic boundary events: a step/end if a step was still open, then a turn/end carrying the merge-extensible { kind: 'interrupted' } reason (a marker that records the turn was cut short by a crash, not completed by the model — no loop ever emits it). load returns the balanced log, so a resumed session is immediately usable. The ONLY thing discarded is a never-fully-written torn tail fragment — a final record whose bytes (JSONL) or row were never completely flushed; that fragment is not a valid event and is dropped before the synthetic closers are written. A parse error or seq gap in the COMMITTED region (at or before the last real turn/end) is genuine corruption and makes the session unloadable.
• File backend canonical, DB backend a drop-in. SessionEvent maps 1:1 onto a row (session_id, seq, type, time, data) — append is INSERT (in a transaction asserting the contiguous-seq contract), load is SELECT … ORDER BY seq. A future dsh-session-persistence-sqlite is a SessionPersistence subclass with no interface change (opencode runs this exact shape on SQLite/WAL).
• Metadata is out-of-log. Format version, cwd, and lineage are storage concerns, not replayable conversation state, so they live in a SessionMeta (SessionHeader & SessionSummary) owned by dsh-session and attached to a Session via a new readonly session.header — never in SessionEventMap, never reaching deriveMessages(). The alternative (a merge-extensible session/meta event as log line 0) was rejected: an in-log event would ride along with a seeded/forked session for free, but metadata is not replayable state, so the explicit out-of-log header seam is the cleaner cost.
• load returns a resumable event log, not just bytes. load(sessionId) yields the SessionMeta plus the committed SessionEvent[] (through the last complete turn/end), shaped so a caller can reconstruct a live session with the loaded events as seed (so lastTurnNumber/deriveMessages continue) on the SAME session id. The agent-facing create/resume factory that consumes this is a separate seam (a follow-up on ctx.agents); the persistence layer deliberately stops at the load primitive and does NOT reach into the loop. The agent-loop does NOT hard-inject sessionPersistence (that would pend non-persistent demos forever), so any resume path built on this rejects with a clear error when the backend is absent.

Format versioning: the header carries a version; load rejects an unknown version (no v1 migration). Stated honestly: append-only + flush is robust to partial trailing writes (tolerated on load) but not to fsync-less power loss mid-line; a DB/WAL backend is the stronger option later.

Consequences

Two new packages and the metadata seam in dsh-session (session.header, the create(id?, options?) signature). Bought: durable resume/fork, a read/replay path, crash tolerance, and the foundation RFC 010's session/load needs — all over the existing event-sourced log, with the backend swappable behind one interface. The reusable runPersistenceContract suite holds every backend to the same append-only / contiguous-seq / lazy-materialization / serializability semantics. This completes ADR 0003's deferred "real persistence backend" and resolves its TODO(review) on the event vocabulary: persisting the log freezes its shape, and the assistant/chunk fidelity question is answered above (persist verbatim).