deepseek-harness/.agents/notes/implemented/architecture/2026-06-14-session-persistence.md

Agent Note: Session persistence as an abstract service over the existing SessionEvent

Status: implemented

English | 中文

Problem

Sessions lived only in memory. The example session-jsonl.ts plugin (duplicated byte-for-byte in both examples) was write-only telemetry: it buffered session/event and appended JSON lines, with no read/replay path, no crash-safety (no fsync, no atomic write, a fire-and-forget dispose drain), no listing, and no format versioning. Nothing could rehydrate a past session from disk into a live agent, so durable resume, durable forking, and host-side session browsing were all impossible.

The event-sourced model makes the append-only log the single source of truth and derives LLM history from it. Persistence had to stay faithful to that: persist the existing SessionEvent directly, with no parallel "persisted message" type that the log is converted to and from. The backend also had to be swappable — a file store now, a database store later — behind one interface.

Decision

Persistence is a capability seam with an abstract Service Definition (capability seams, the dsh-bash template), not loop or core logic:

1. Interface (dsh-session-persistence, ctx.sessionPersistence) — an abstract SessionPersistence service: locate/create/append/prepare/load/inspect/readFrom/list/listSnapshots. Its persisted unit IS the existing SessionEvent ({ type, seq, time, data }), reused verbatim — no conversion type.
2. Implementation (dsh-session-persistence-jsonl) — an append-only logical JSONL log per session: a SessionHeader line followed by storage records that losslessly represent the contiguous SessionEvent stream. Eligible assistant/chunk delta runs use packed rows by default; checksummed Zstandard frames are the default physical encoding, with raw lines configurable.

Key choices recorded here because they are durable, contested, and surprising:

• The canonical durable log persists every SessionEvent losslessly, including assistant/chunk. JSONL storage may encode a consecutive delta run as one packed row, but logical readers reconstruct the exact event boundaries, sequence numbers, and timestamps. deriveMessages() skips chunks, and a chunk-filtered rollout (Codex's policy.rs) is tempting — but seq = log.length and validation of events[i].seq === i require a contiguous logical log; filtering chunks out would leave holes and break both the contract and resume. A chunk-filtered projection is possible later as a derived view with its own renumbering, but it is NOT the canonical log.
• Append-only; a crashed turn is closed, never truncated. Flushed events are never rewritten. The semantic checkpoint policy drains the request before model dispatch, a recorded top-level call before tool dispatch, and the complete response/result batch after a step; the loop drains the final turn boundary. Because one interrupted turn may contain substantial valid work, cold inspection preserves its contiguous, parseable events and adds risk-classified error results for unanswered assistant calls, a missing step/end, and turn/end with { kind: 'interrupted' } to the in-memory logical view. prepare or load commits those closers before returning a recoverable view; the synthetic results keep resumed provider transcripts valid. Only an incomplete final record is discarded during committed repair; a parse error or sequence gap at or before the last real turn/end is corruption and makes the session unloadable.
• File backend canonical, DB backend a proven drop-in. SessionEvent maps 1:1 onto a row (session_id, seq, type, time, data) — append is INSERT (in a transaction asserting the contiguous-seq contract), and reads use SELECT … ORDER BY seq. dsh-session-persistence-sqlite is exactly this: a SessionPersistence subclass with no interface change (opencode runs this exact shape on SQLite/WAL), and it passes the same runPersistenceContract suite as the JSONL backend — so the contract holds both backends to identical semantics (lazy materialization, logical interrupted-turn closure, single committed repair, contiguous-seq), expressed once over file bytes and once over rows. Its database carries a dedicated application id and monotonic schema version. A pristine file creates all tables and stamps both header values in one transaction; an unversioned file with any user-defined schema object or application identity, a foreign current-version identity, and every non-current version reject before journal-mode mutation.
• Metadata is out-of-log. Format version, cwd, and lineage are storage concerns, not replayable conversation state, so they live in a SessionHeader owned by dsh-session and attached to a Session via a new readonly session.header — never in SessionEventMap, never reaching deriveMessages(). createdAt is non-negative safe-integer Unix epoch milliseconds: live creation and persistence registration reject fractional values, JSONL validates the decoded header, and SQLite stores it in a strict INTEGER column. The alternative (a merge-extensible session/meta event as log line 0) was rejected: an in-log event would ride along with a seeded/forked session for free, but metadata is not replayable state, so the explicit out-of-log header boundary is the cleaner cost. (The header was originally split into an immutable SessionHeader plus a mutable SessionSummary whose union was SessionMeta; the mutable summary was later removed as dead state — see Drop the mutable session summary.)
• ctx.agents.create() and ctx.agents.resume() are async factories; resume additionally crosses the persistence boundary. ctx.agents.resume({ resumeSessionId }) obtains the exact unpublished Session through ctx.sessionPersistence.prepare(), publishes it under the persisted id, and continues its projections. The Session preparation decision owns reuse between history inspection and resume. The agent-loop does NOT hard-inject sessionPersistence (that would pend non-persistent demos forever); resume rejects with a clear error when it is absent.

Alternatives considered

Each key choice above records its rejected alternative where the choice is stated: a chunk-filtered canonical log (Codex's policy.rs shape) — breaks the contiguous-seq contract; truncating a crashed turn — silently destroys a long autonomous run's real work; an in-log session/meta event as line 0 — metadata is not replayable state; finite fractional createdAt values — have no producer and diverge from integer Unix-millisecond storage and query columns; adopting a non-pristine unversioned SQLite file — can overwrite unrelated objects or identity; hard-injecting sessionPersistence into the loop — would pend non-persistent demos forever.

Format versioning: the header carries a version; cold reads reject any non-current version. The pre-release session format stays pinned at SESSION_FORMAT_VERSION = 0 and carries no broad compatibility promise, while the coordinator may own an explicit narrow import upgrade when persisted user data requires it (pre-identity message recovery). Stated honestly: append-only + flush is robust to partial trailing writes (tolerated during cold preparation) but not to fsync-less power loss mid-line; a DB/WAL backend is the stronger option later.

Consequences

Two new packages and the metadata contract in dsh-session (session.header, the create(id?, options?) signature). Bought: durable resume/fork, a read/replay path, crash tolerance, and host-side session access over the existing event-sourced log, with the backend swappable behind one interface. The reusable runPersistenceContract suite holds every backend to the same append-only, contiguous-seq, lazy-materialization, logical-recovery, integer-metadata, and serializability semantics. Persisting the full logical log also settles event fidelity: every assistant/chunk survives exactly even when JSONL packs several into one storage row. SQLite initialization either commits its complete owned schema and header identity or leaves no partial schema to strand on the next open.