RFC: Every LLM request is reconstructable from the session log

Status: implemented

Problem

Two gaps shared one root. First, provider KV caching (DeepSeek context caching) is prefix-based — a request pays full price only for the tokens after the longest stored prefix it matches — yet nothing in the request pipeline stated, checked, or measured prefix stability: every registered PromptSection happened to be static, the tool set happened not to change mid-session, no listener happened to rewrite requests. A single time-interpolating section would have silently multiplied context cost, and no test or metric would have moved. Second, and deeper: the session log — the system's single source of truth — could not actually answer what the model saw. It recorded every message but never the system prompt, the tool schemas, or even which model; the mutable agent/request waterfall handed listeners the whole GenerateOptions to rewrite per call; replay equivalence was therefore a property of the plugin population, not of the design.

The reference shape for the happy path is MiniCode's LLMClient: a stateful conversation client, appended to — never rebuilt — as the conversation advances, resetting only when the system prompt, tool set, or compaction genuinely changes what the model must see. The design question this RFC answers is how to get that discipline without giving up event-sourcing.

Decision

The principle

Model-visible ⟺ logged. Anything that reaches a model request must be recorded in the session log. The checkable consequence: every conversation request the loop sends is a pure function of the session log — anyone holding the log reconstructs it byte-for-byte. Scope, stated precisely: the guarantee covers the loop-built GenerateOptions; provider wire bytes follow from it because both adapters' serialization is a pure per-message function at a pinned code version; direct one-shots (compaction's summarize call) log their envelope scalars (compact/summary.{model, maxTokens}) and their input is deterministic code over the logged region — reconstructable from log + code, outside the invariant by the unfrozen-request marker.

Prefix-cache stability is corollary #1, not the headline: an append-only log projected by a per-node pure function yields requests that are append-extensions of their predecessors whenever the header is unchanged — stability is emergent, not managed. Byte-exact audit/replay is corollary #2; resume and fork with attributable drift is corollary #3.

The mechanism

Messages. Session.deriveMessages() is cached: each surface entry is projected exactly once, when first seen, through the public per-event function deriveEventMessage(event); a surface rewrite (a compaction replace — SurfaceManager.replaceGeneration) rebuilds. Callers get a fresh array per call over shared, deep-frozen messages: mutating logged history through a projection is unrepresentable (it throws), replacing the old clone-per-call isolation. External reconstructors fold the same public function over a log prefix, so no two paths can disagree.

The header. The request's non-history half — EpochHeader: call config (LlmCallConfig: model + sampling scalars), rendered system prompt, assembled tool schemas, and the session prefix (messagePrefix, below) — is logged session state in canonical form (empty system/tools/prefix ≡ absent). One log-only, turn-enclosed event carries it: request/header, always a full snapshot. Each loop instance appends one on its first request ('initial' when the log has none, 'resume' otherwise, even when nothing changed: the boundary itself is a recorded fact and cross-restart drift becomes attributable); a later request whose canonical header differs appends another with reason 'change'. foldRequestHeader reconstructs by selecting the latest snapshot, and the live session tracks that fold with the same lazy cursor as the message cache. Legacy v0 logs containing the removed delta representation are rejected at seed and persistence-load boundaries rather than partially replayed.

The loop, transmission-stateless. Per step: render assembly (every step — value comparison needs no change-signal discipline, and a section that varies per step surfaces as a logged header event per step instead of a silent bust) → on the instance's FIRST step only, the agent/session-prefix waterfall — request-ONLY messages fronting the entire derived history (a frozen empty seed, contributions returned as an extension of next(); the home for session-stable openers that must NOT become history — a skills catalog, an AGENTS.md digest), deep-frozen and cached on the instance so reuse is structural and the prefix cannot drift mid-session — → agent/pre-step, carrying the composed prefix (compaction's surface mutations land before derivation, and its pressure gate counts the prefix this instance will actually send — never a previous instance's logged one, which could under-gate a resumed/forked instance whose contributor grew) → messages snapshot, then step/start appended as the next operation in the same synchronous frame → seed the call config (first request of the instance: from AgentOptions, so explicit options always beat the logged baseline — fork model-overrides and resume reconfiguration stay correct; afterwards: from the folded header) → the agent/request waterfall, re-typed (agent, turn, step, config: LlmCallConfig, next) → LlmCallConfig — a frozen seed and a returned replacement are ALL a listener shapes; durable content flows through the log channels (inject(), steering, prompt-submit additionalContext, sections via system-prompt/assemble) — → the header event the request owes the log, carrying the prefix as messagePrefix (no session event carries it, so the header is its only durable record; resume = a new instance = a recompose, anchored by its 'resume' snapshot) → build GenerateOptions from messagePrefix + snapshot + header, deep-freeze (deepFreeze exempts the AbortSignal, the one live control channel — freezing one breaks AbortController.abort()), dispatch. The loop's per-instance bookkeeping is one boolean plus the cached prefix: whether this instance has logged its anchoring snapshot, and what it composed.

The reconstruction boundary is step/start, unconditionally. A step's messages are the derivation over events[0..stepStartSeq). Because the snapshot precedes the step/start append in the same synchronous frame, an agent.inject() from an agent/request listener or any concurrent task lands after the boundary and joins the NEXT request. session/event is observe-only during publication: a reentrant append is rejected until the current callback list drains, preventing nested event delivery from overtaking the event being observed. agent/pre-step is the sanctioned seam for content that must affect the CURRENT request. A step's header for reconstruction is the latest request/header at or after its step/start (before the first response event), or the fold carried forward when the request header is unchanged.

Enforcement. Dev-mode (dsh-invariants), on llm/stream: a frozen request with a live sessionId — the loop-built marker; hand-built one-shots are unfrozen and skipped — must carry messages deep-equal to the folded header's messagePrefix followed by the boundary derivation — the derivation rebuilt through a FRESH Session over events[0..stepStartSeq) so the live cache cannot vouch for itself — and header fields equal to foldRequestHeader over the log. There is no divergence allowance and nothing to allow: no seam can put unlogged content into a request — the agent/session-prefix seam's product enters only because the header event records it first. prepend: true only defends against the replay adapter's short-circuit (an append-registered listener); two prepended listeners have no defined mutual order in cordis, so correctness rests on the seq-bounded fold, never on listener timing. Measurement stays lean: the with-key e2e (request-cache.e2e.ts) proves usage.cacheReadTokens > 0 on every request after the first against the live API, and per-step usage in the log is the production observable — a header event or compaction shows up as a cache-read collapse on the next step.

The MiniCode shape: adopted, with the provenance arrow inverted

What survives from LLMClient: the conversation is maintained, not rebuilt — one projection per message, ever; requests advance append-only; resets happen only for a system-prompt/tool change, a config change, or compaction, each now a logged fact. What is deliberately inverted: MiniCode's client is the source of truth and its event stream derives from client appends (on_event(MessageAdded)), which suits an advisory event stream. Here the log is contractual — persistence, crash recovery, fork seeding, transcript rendering, and the snapshot harness all replay it — and it carries strictly more than a message list (turn/step boundaries, raw chunk streams, tool-call pairing, provenance, log-only records), so a message-list client cannot generate it. The arrow therefore points log → client: the conversation state IS the log plus two cached folds inside Session (messages, header), and the "client" the loop talks to is the session itself. What the inversion buys over the original: the reconstruction is checkable against an independent record on every request — MiniCode's client has nothing to check itself against.

Alternatives considered

Client as source of truth (literal MiniCode): a second operative truth beside the log — the two drift and nothing notices; see the section above.
A stateful transmission client mirroring the log (a PromptPrefix class holding committed/open message zones with an append/editTail/reset vocabulary, the log pushed into it per event): behaviorally equivalent on the happy path, but it duplicates conversation state outside the session, needs transactional rollback around listener seams, keeps an unlogged content-shaping surface (editTail) whose divergence the invariant must specially allow, and still cannot answer "what header did the model see" from the log. Dissolving it into the session's own caches plus logged header events made every one of those problems unrepresentable instead of guarded. (PR #162 is the archaeology of this alternative, three designs deep.)
Per-call request scalars (a freely mutable config handed to each agent/request dispatch): a listener flips the model per call with zero accounting, silently abandoning the provider cache this design exists to protect. Config is per-conversation logged state; the waterfall proposes, the log records.
Detect-and-report (compare consecutive requests, warn on divergence): catches violations after the fact; a violating request is still constructible and ships. Rejected for interface-level unrepresentability.
Event-driven assembly (re-render only on change signals): a missed-signal bug class — a tool registered mid-session emits tools/change, not system-prompt/change, and a third-party provider may emit nothing. Per-step render + value compare is robust with zero signal discipline.
A custom header-delta codec (system line edits, name-keyed tool edits, whole config/prefix replacements): it reduced repeated header bytes but duplicated state across codec types, diff/apply machinery, and fallback handling. Full changed snapshots preserve reconstructability with one representation; compression remains available if measured logs justify it.
Narrative changed-field lists on header snapshots: derivable by diffing consecutive snapshots — one home per fact. Snapshots keep a reason because an instance boundary versus an in-instance change is not derivable from data alone.

Consequences

A request that is not explained by the log cannot be constructed by accident — not by the loop, not by a listener; mutating a built request throws; every header change is a durable, diffable log event.
Choosing between the advisory channels is a change-frequency decision, and the design makes the stable one structural: an agent/session-prefix contribution is composed once per loop instance and reused verbatim, so it extends the cacheable prefix at zero marginal cost and CANNOT bust the provider cache mid-session; content that changes mid-session flows through the append-only history channels — agent.inject(), a tools/post-execute decision's additionalContext, prompt-submit additionalContext — each a durable context/message paid once and prefix-cached thereafter, at the price of accumulating in history and the log. Route session-frozen openers to the prefix and change notices to the history channels; a per-step request-only tail slot was deliberately dropped (no consumer, and a durable append covers every current update pattern).
What still costs full price at the provider is inherent and logged: compaction (its compact/* events and surface replacement), a real prompt/tool/config change (request/header with reason 'change'), or a process boundary with drift ('resume' snapshot differing from its predecessor). The provider's own reasoning-content exclusion is managed server-side.
The step/start-listener behavior change (above) is the one observable semantics change for plugins; agent/pre-step is the current-request seam.
Tool-result trimming (planned) needs no new mechanism: a logged single-entry surface replace (start === end) carrying a trimmed tool/result under the same callId — compaction-family, replay-correct, cache-bust batched by the same pressure logic.
Session logs grow one request/header snapshot per loop instance plus full snapshots on real changes. This spends more bytes than a custom delta codec but stays small beside chunk-heavy logs and leaves one replay representation. SESSION_FORMAT_VERSION stays 0; a legacy v0 delta event is rejected rather than migrated.
Snapshot goldens changed once (every transcript gains its header events); the fs-writing fixtures are stored in the normalized authored form with cwd-relative tool arguments, because replay only round-trips cwd-independent argument paths.
FIXME(call-config-shape): revisit LlmCallConfig's exact field set — which fields are genuinely epoch-level for cache purposes (model certainly; the sampling scalars sit there out of caution), and where provider-specific extras (reasoning options, extra body params) belong when an adapter needs them.

15 KiB Raw Blame History