diff --git a/docs/architecture.md b/docs/architecture.md index fd21648dd2..e1b60a5331 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -136,7 +136,7 @@ The session log is the source of truth. `deriveMessages()` projects session even **Model-visible ⟺ logged**: the log reconstructs every request — messages at `step/start` fronted by the header's session prefix, headers by folding `request/header` — and dev invariants assert this ([reconstructability RFC](rfc/implemented/architecture/2026-07-05-reconstructable-requests.md)). -Durability is a plugin concern. Persistence backends buffer synchronous `session/event` notifications and the loop awaits a turn-end checkpoint before moving on. The `SessionPersistence` seam stores `SessionEvent` directly, with metadata in `SessionHeader`; JSONL and SQLite share one contract suite. +Durability is a plugin concern. Backends buffer synchronous `session/event` notifications; the loop awaits a turn-end checkpoint. `SessionPersistence` stores `SessionEvent` directly and metadata in `SessionHeader`; JSONL defaults to checksummed Zstandard, with SQLite under one contract. ### Model Content diff --git a/docs/core-data-structures/persistence.md b/docs/core-data-structures/persistence.md index 72ebcc5852..19fc547b04 100644 --- a/docs/core-data-structures/persistence.md +++ b/docs/core-data-structures/persistence.md @@ -95,7 +95,7 @@ Replay/fork is therefore `ctx.sessions.create(id, { seed: seedEvents })`; resumi Both implement the same abstract `SessionPersistence` (locate/create/append/load/list over `SessionEvent`) and pass `runPersistenceContract`, proving the seam is genuinely backend-agnostic: -- **[dsh-session-persistence-jsonl](../../packages/session-persistence/session-persistence-jsonl)** — an append-only JSONL log per session with crash-safe atomic writes, the interrupted-turn crash recovery above, and a read/replay path. +- **[dsh-session-persistence-jsonl](../../packages/session-persistence/session-persistence-jsonl)** — an append-only logical JSONL log per session, stored as checksummed concatenated Zstandard frames by default or raw lines by configuration, with crash-safe atomic writes, interrupted-turn recovery, and a read/replay path. - **[dsh-session-persistence-sqlite](../../packages/session-persistence/session-persistence-sqlite)** — `node:sqlite`, one row per `SessionEvent`. The row shape `(session_id, seq, type, time, data, source_event_seqs, surface_op)` maps 1:1 onto the event, including optional surface metadata, so there is no parallel persisted schema to keep in sync. Multiple backends sharing one on-disk session coordinate writes through the [shared persistence write-coordinator](../rfc/implemented/architecture/2026-06-18-shared-persistence-write-coordinator.md). diff --git a/docs/i18n/terminology.md b/docs/i18n/terminology.md index f11a1ff19e..0037ba9cfa 100644 --- a/docs/i18n/terminology.md +++ b/docs/i18n/terminology.md @@ -64,6 +64,7 @@ | waterfall | waterfall | waterfall(瀑布式事件) | | | | wheel | wheel 包 | | | Python 打包格式 | | worktree | worktree | | | git 工作区概念 | +| Zstandard | Zstandard | | | RFC 8878 compression format; `zstd` remains a code value. | ## 双语类(中英文文本各自使用中英文) diff --git a/docs/rfc/INDEX.md b/docs/rfc/INDEX.md index 65afa473fc..564b0383ca 100644 --- a/docs/rfc/INDEX.md +++ b/docs/rfc/INDEX.md @@ -168,6 +168,7 @@ Generated by `pnpm run gen-rfc-index` from the RFC tree — never edit by hand; | [Initiating Agent scope over AsyncLocalStorage](implemented/architecture/2026-07-15-agent-initiator-scope.md) | 2026-07-15 | | [Advisory LLM catalogs and per-session ACP model selection](implemented/architecture/2026-07-15-llm-model-catalog-and-acp-selection.md) | 2026-07-15 | | [Replay token meter service](implemented/architecture/2026-07-15-replay-token-meter-service.md) | 2026-07-15 | +| [Zstandard JSONL session logs](implemented/architecture/2026-07-19-zstandard-jsonl-session-logs.md) | 2026-07-19 | ### Process diff --git a/docs/rfc/implemented/architecture/2026-06-14-session-persistence.md b/docs/rfc/implemented/architecture/2026-06-14-session-persistence.md index 3bd3c3fdf1..61f186805c 100644 --- a/docs/rfc/implemented/architecture/2026-06-14-session-persistence.md +++ b/docs/rfc/implemented/architecture/2026-06-14-session-persistence.md @@ -13,7 +13,7 @@ The [event-sourced model](2026-06-11-event-sourced-sessions.md) makes the append Persistence is an abstract **capability seam** ([capability seams](2026-06-13-capability-seams.md), the `dsh-bash` template), not loop or core logic: 1. **Interface** (`dsh-session-persistence`, `ctx.sessionPersistence`) — an abstract `SessionPersistence` service: `create`/`append`/`load`/`list`. Its persisted unit IS the existing `SessionEvent` (`{ type, seq, time, data }`), reused verbatim — no conversion type. -2. **Implementation** (`dsh-session-persistence-jsonl`) — an append-only JSONL log per session (a `SessionHeader` line then one `SessionEvent` per line, verbatim **including `assistant/chunk`**). +2. **Implementation** (`dsh-session-persistence-jsonl`) — an append-only logical JSONL log per session (a `SessionHeader` line then one `SessionEvent` per line, verbatim **including `assistant/chunk`**), encoded as [checksummed Zstandard frames by default](2026-07-19-zstandard-jsonl-session-logs.md) or raw lines by configuration. Key choices recorded here because they are durable, contested, and surprising: diff --git a/docs/rfc/implemented/architecture/2026-06-18-shared-persistence-write-coordinator.md b/docs/rfc/implemented/architecture/2026-06-18-shared-persistence-write-coordinator.md index fef6ad034c..f69426dcac 100644 --- a/docs/rfc/implemented/architecture/2026-06-18-shared-persistence-write-coordinator.md +++ b/docs/rfc/implemented/architecture/2026-06-18-shared-persistence-write-coordinator.md @@ -28,7 +28,7 @@ Six methods (five required + an optional lifecycle hook) — the only seam betwe ### The opaque torn marker -The single design choice that keeps the seam clean: the crash-repair "where is the torn tail" token is OPAQUE to the coordinator. The coordinator computes the synthetic closers (it owns `interruptedTurnClosers` from `dsh-session`), but it only ever tests `tornMarker !== undefined` and passes the value straight back to `commitRepair` — it never inspects it. Each backend picks its own marker type: JSONL uses the byte offset to truncate to, SQLite the seq to delete from (both happen to be `number`). The JSONL backend folds its `committedBytes < buffer.byteLength` comparison INSIDE the hook so the returned marker is already `number | undefined`; without that fold the coordinator would have to know about byte lengths. +The single design choice that keeps the seam clean: the crash-repair "where is the torn tail" token is OPAQUE to the coordinator. The coordinator computes the synthetic closers (it owns `interruptedTurnClosers` from `dsh-session`), but it only ever tests `tornMarker !== undefined` and passes the value straight back to `commitRepair` — it never inspects it. Each backend picks its own marker type: JSONL carries the byte offset to truncate to plus any complete events decoded from an incomplete final frame, while SQLite carries the seq to delete from. The coordinator therefore knows neither byte lengths nor frame recovery state. ## Testing diff --git a/docs/rfc/implemented/architecture/2026-07-19-zstandard-jsonl-session-logs.i18n.yaml b/docs/rfc/implemented/architecture/2026-07-19-zstandard-jsonl-session-logs.i18n.yaml new file mode 100644 index 0000000000..35d0c29831 --- /dev/null +++ b/docs/rfc/implemented/architecture/2026-07-19-zstandard-jsonl-session-logs.i18n.yaml @@ -0,0 +1,6 @@ +# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each +# side as of the last confirmed-consistent state. Both languages carry equal authority; +# after editing either side, bring the other along and re-record with: +# pnpm run verify-translation-pairing --write +2026-07-19-zstandard-jsonl-session-logs.md: 5c33b4a888e34d2f47ec5ede126c96a7b45f7fda +2026-07-19-zstandard-jsonl-session-logs.zh.md: 7dfa72e8e7064fde0fd5d7d026ab9b02e70287b7 diff --git a/docs/rfc/implemented/architecture/2026-07-19-zstandard-jsonl-session-logs.md b/docs/rfc/implemented/architecture/2026-07-19-zstandard-jsonl-session-logs.md new file mode 100644 index 0000000000..5c33b4a888 --- /dev/null +++ b/docs/rfc/implemented/architecture/2026-07-19-zstandard-jsonl-session-logs.md @@ -0,0 +1,57 @@ +# RFC: Zstandard JSONL session logs + +Status: implemented + +English | [中文](2026-07-19-zstandard-jsonl-session-logs.zh.md) + +## Problem + +The JSONL persistence backend keeps every `SessionEvent` verbatim, including high-volume `assistant/chunk` records. Raw text makes logs inspectable but spends storage and I/O on repeated JSON keys and model text. Compression must retain the existing append/fsync commit boundary, collision-safe first materialization, crash repair, and metadata-only listing; rewriting a whole compressed file after every turn would discard those properties. + +The encoding also has to remain explicit at the deployment boundary. Snapshot fixtures and external line readers require raw JSONL, while a backend cannot safely guess between compressed and raw artifacts in one root or silently migrate pre-release session data. + +## Decision + +### Configuration and suffix ownership + +`dsh-session-persistence-jsonl` accepts `compression?: 'zstd' | 'none'` and explicitly resolves omission to `'zstd'`. Zstandard artifacts end in `.jsonl.zstd`; `'none'` retains the original newline-delimited UTF-8 `.jsonl` representation. `SessionLocation.kind` remains `'jsonl'`, because both encodings carry the same logical record format, and `SESSION_FORMAT_VERSION` remains `0` under the repository's pre-release reject-without-migration policy. + +Each persistence root belongs to one encoding. A one-time discovery preflight rejects any opposite suffix, and targeted load, live-adoption, listing, and materialization paths repeat the relevant suffix check after an initially empty preflight. The error names the incompatible artifact and directs the deployment to the matching configuration or a separate root. There is no migration, dual read, dual write, or extension-based fallback. + +### Frame and write path + +The compressed artifact is a standard concatenation of independent [Zstandard frames](https://datatracker.ietf.org/doc/html/rfc8878): one checksummed frame containing exactly the header line, followed by one checksummed frame for every durable append batch. Normal loop batches are turn commits, so frame boundaries preserve the existing persistence checkpoint without making the storage layer depend on turn event types. + +Compression uses Node's built-in [`zstdCompress` and `zstdDecompress`](https://nodejs.org/download/release/v22.19.0/docs/api/zlib.html), available at the repository's Node 22.19 floor. The backend enables `ZSTD_c_checksumFlag`, otherwise accepts Node's defaults, and exposes neither a compression-level knob nor a new dependency. The API is marked experimental by Node, so the Node 22.19, 24, and 26 compatibility gate exercises the exact helper. + +First materialization compresses the two initial frames before opening the temporary file, then keeps the existing write, file `fsync`, collision-safe hard-link publication, and directory `fsync` sequence. Later batches are compressed before opening the destination and appended at EOF. A caught write or file-sync failure truncates to the prior byte length, syncs the rollback, and rethrows so the coordinator can retry the unchanged batch. + +### Read, listing, and crash recovery + +A frame-boundary scanner reads the standard magic, variable header fields, block headers and payload sizes, and optional checksum trailer. It does not interpret compressed blocks. Complete frames are decompressed independently and sequentially, which validates their checksums, and their plaintext is passed to the existing JSONL scanner. A checksum/decompression failure in any complete frame, a malformed complete-frame JSONL tail, or invalid frame structure is corruption and rejects. + +Listing reads in bounded chunks only until the first complete frame is available, validates and decompresses that header frame, and never reads an event frame. The dedicated header frame therefore preserves metadata-only listing even for very large session logs. + +EOF inside the final frame is a recoverable torn tail. Node's decoder is given the available frame prefix; every complete newline-terminated event it emits is retained. Repair truncates from that frame's starting byte and appends one new checksummed frame containing the recovered complete events followed by the coordinator's synthetic tool, step, and turn closers. If the tear occurs before any complete event is decodable, repair drops the partial frame and retains all prior complete frames. + +### Consumers and verification + +The CLI, ACP, and stdio app bundles expose symmetric `persistenceCompression` pass-through configuration. Snapshot recording and replay compositions select `'none'` explicitly because committed fixtures are raw JSONL inputs to replay and normalization; ordinary runtime compositions use the compressed default. + +The shared persistence and coordinator contracts run against both encodings. Backend tests cover standard framing and checksum interoperability, header-only listing, append rollback, encoding mismatch rejection, complete-frame corruption, and final-frame tears through headers, blocks, and checksum trailers. Default runtime, built-bin, headless, ACP, and Python smokes assert the compressed suffix and Zstandard magic or decode the header; raw-content tests opt out explicitly. + +## Alternatives considered + +- **One frame per JSONL record** — rejected because it multiplies frame headers and checksums for high-volume chunk events and makes a physical boundary unrelated to the durable append batch. +- **Rewrite one whole compressed stream after every append** — rejected because cost grows with log size and replacement would give up append/fsync rollback and the established collision-safe materialization mechanics. +- **Use a streaming compressor across appends** — rejected because an interrupted encoder state does not leave independently checksummed append units, complicating bounded listing and frame-start repair. +- **Add an external native Zstandard dependency** — rejected because the supported Node floor already provides the required codec; another native artifact would enlarge installation and executable-packaging risk without adding a required behavior. +- **Expose compression level or keep raw JSONL as the default** — rejected because there is no deployment evidence for a second tuning policy, while `'none'` preserves the line-readable path for fixtures and integrations that need it. + +## Consequences + +- Ordinary session roots store `.jsonl.zstd` and retain append-only, fsync, rollback, and interrupted-turn recovery semantics. +- Raw JSONL remains a deliberate configuration, but changing encoding requires a fresh/separate root or selecting the mode that matches existing artifacts. +- One frame per durable batch adds bounded framing/checksum overhead and allows header-only listing plus repair from an exact append boundary. +- External tools must understand concatenated Zstandard frames or consume raw-mode artifacts; generic one-shot Node decompression reads only the first independent frame, so backend reads walk frames explicitly. +- The implementation depends on Node's experimental built-in Zstandard API without an npm dependency; the supported-version compatibility gate makes drift visible. diff --git a/docs/rfc/implemented/architecture/2026-07-19-zstandard-jsonl-session-logs.zh.md b/docs/rfc/implemented/architecture/2026-07-19-zstandard-jsonl-session-logs.zh.md new file mode 100644 index 0000000000..7dfa72e8e7 --- /dev/null +++ b/docs/rfc/implemented/architecture/2026-07-19-zstandard-jsonl-session-logs.zh.md @@ -0,0 +1,57 @@ +# RFC:Zstandard JSONL 会话日志 + +Status: implemented + +[English](2026-07-19-zstandard-jsonl-session-logs.md) | 中文 + +## 问题 + +JSONL 持久化后端会逐字保留每个 `SessionEvent`,其中包括数量庞大的 `assistant/chunk` 记录。原始文本便于检查,但重复的 JSON 键和模型文本会增加存储与 I/O 开销。压缩编码必须保留既有的 append/fsync 提交边界、首次物化时的无冲突发布、崩溃修复以及仅元数据列举;如果每轮都重写整个压缩文件,就会失去这些属性。 + +编码还必须在部署边界上保持显式。快照 fixture 与外部逐行读取器需要原始 JSONL,而后端无法在同一根目录中安全猜测压缩产物与原始产物,也不能静默迁移预发布会话数据。 + +## 决策 + +### 配置与后缀归属 + +`dsh-session-persistence-jsonl` 接受 `compression?: 'zstd' | 'none'`,并将省略值显式解析为 `'zstd'`。Zstandard 产物使用 `.jsonl.zstd` 后缀;`'none'` 保留原有的换行分隔 UTF-8 `.jsonl` 表示。`SessionLocation.kind` 仍为 `'jsonl'`,因为两种编码承载同一逻辑记录格式;按照仓库的预发布拒绝且不迁移策略,`SESSION_FORMAT_VERSION` 仍为 `0`。 + +每个持久化根目录只归属于一种编码。一次性的发现预检会拒绝任何相反后缀,而针对性的加载、活跃采用、列举与物化路径会在最初空目录预检之后再次执行对应后缀检查。错误会指出不兼容产物,并要求部署选择匹配配置或单独根目录。系统不提供迁移、双重读取、双重写入或基于扩展名的兜底。 + +### 帧与写入路径 + +压缩产物是标准独立 [Zstandard 帧](https://datatracker.ietf.org/doc/html/rfc8878)的串联:第一个带校验和的帧只包含头部行,后续每个持久追加批次各占一个带校验和的帧。正常 agent loop 批次就是轮次提交,因此帧边界保留既有持久化检查点,同时不让存储层依赖轮次事件类型。 + +压缩使用 Node 内置的 [`zstdCompress` 与 `zstdDecompress`](https://nodejs.org/download/release/v22.19.0/docs/api/zlib.html),仓库最低支持的 Node 22.19 已提供这些 API。后端启用 `ZSTD_c_checksumFlag`,其余采用 Node 默认值,不公开压缩级别调节项,也不增加依赖。Node 将该 API 标记为实验性,因此 Node 22.19、24 与 26 兼容性门禁会执行同一个辅助实现。 + +首次物化会在打开临时文件之前压缩两个初始帧,然后保留既有的写入、文件 `fsync`、避免冲突的硬链接发布与目录 `fsync` 顺序。后续批次也会先压缩,再打开目标并在 EOF 追加。捕获到写入或文件同步失败时,后端会截断到原有字节长度,同步回滚结果,再重新抛出错误,让协调器重试未变化的批次。 + +### 读取、列举与崩溃恢复 + +帧边界扫描器会读取标准魔数、可变头字段、块头与负载长度,以及可选校验和尾部,但不会解释压缩块。后端独立且按顺序解压完整帧,由此验证各帧校验和,再把明文交给既有 JSONL 扫描器。任何完整帧的校验和或解压失败、完整帧中畸形的 JSONL 尾部,或者无效帧结构都属于损坏并拒绝加载。 + +列举只按有界分片读取到第一个完整帧可用为止,验证并解压该头部帧,绝不读取事件帧。因此,即使会话日志很大,专用头部帧仍能维持仅元数据列举。 + +最终帧内部遇到 EOF 属于可恢复的撕裂尾部。后端把已有帧前缀交给 Node 解码器,并保留其产出的每个完整、以换行结束的事件。修复从该帧起始字节截断,再追加一个新的带校验和帧,其中依次包含恢复出的完整事件,以及协调器生成的工具、步骤与轮次闭合事件。如果撕裂位置尚不足以解码任何完整事件,修复会丢弃该不完整帧并保留此前全部完整帧。 + +### 消费方与验证 + +CLI、ACP 与 stdio 应用包公开对称的 `persistenceCompression` 透传配置。快照录制与回放组合显式选择 `'none'`,因为提交的 fixture 是回放与规范化过程使用的原始 JSONL 输入;普通运行时组合使用压缩默认值。 + +共享持久化契约与协调器契约会针对两种编码运行。后端测试覆盖标准帧与校验和互操作性、仅头部列举、追加回滚、编码不匹配拒绝、完整帧损坏,以及横跨头部、块和校验和尾部的最终帧撕裂。默认运行时、构建后二进制、headless、ACP 与 Python 冒烟测试会断言压缩后缀与 Zstandard 魔数,或解码头部;读取原始内容的测试则显式退出压缩。 + +## 考虑过的替代方案 + +- **每条 JSONL 记录一个帧**——不予采纳,因为它会让大量分片事件各自承担帧头与校验和开销,并让物理边界脱离持久追加批次。 +- **每次追加都重写一个完整压缩流**——不予采纳,因为成本会随日志大小增长,而且替换操作会放弃追加/fsync 回滚和既有的无冲突物化机制。 +- **跨追加使用流式压缩器**——不予采纳,因为编码器状态中断后不会留下可独立校验的追加单元,从而使有界列举与按帧起点修复更复杂。 +- **增加外部原生 Zstandard 依赖**——不予采纳,因为受支持的 Node 最低版本已经提供所需编解码器;另一个原生产物会增加安装与可执行文件打包风险,却不增加必需行为。 +- **公开压缩级别或继续默认使用原始 JSONL**——不予采纳,因为没有部署证据支持第二种调节策略,而 `'none'` 已为需要逐行读取的 fixture 与集成保留路径。 + +## 后果 + +- 普通会话根目录存储 `.jsonl.zstd`,并保留仅追加、fsync、回滚与中断轮次恢复语义。 +- 原始 JSONL 仍是显式配置,但切换编码需要使用全新或单独根目录,或者选择与既有产物匹配的模式。 +- 每个持久批次一个帧会增加有界的帧与校验和开销,同时支持仅头部列举和从精确追加边界开始修复。 +- 外部工具必须理解串联的 Zstandard 帧,或者消费原始模式产物;Node 通用的一次性解压只读取第一个独立帧,因此后端读取会显式遍历各帧。 +- 实现依赖 Node 的实验性内置 Zstandard API,但不增加 NPM 依赖;受支持版本兼容性门禁会暴露 API 漂移。 diff --git a/docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md b/docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md index 2f7c0dbd8e..697d916a99 100644 --- a/docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md +++ b/docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md @@ -40,7 +40,7 @@ Replay is positional and therefore permits only one in-flight model stream per s ### Recording harvests the log; keyless replay needs a providerless config -Recording runs the scenario with the real `llm-deepseek` adapter and the JSONL persistence backend, then copies the produced `.jsonl` into the scenario dir. Per-event appends are durable, but the harness shuts the subprocess down gracefully (close stdin → `await ctx.dispose()`) before harvesting so the final events are flushed. `llm-replay` itself does no recording — it is replay-only. +Recording runs the scenario with the real `llm-deepseek` adapter and the JSONL persistence backend configured with `persistenceCompression: 'none'`, then copies the produced `.jsonl` into the scenario dir. The explicit raw mode keeps committed replay fixtures line-readable while ordinary deployments use the backend's compressed default. Per-event appends are durable, but the harness shuts the subprocess down gracefully (close stdin → `await ctx.dispose()`) before harvesting so the final events are flushed. `llm-replay` itself does no recording — it is replay-only. Replay uses a `cordis.snapshot.yml` overlay that replaces the real adapter with `llm-replay` while retaining the live composition. Recording uses the ordinary config and a harness-supplied persistence root. Replay mode skips `.env` loading, so a stray API key cannot trigger a live call. See the [single-source config RFC](2026-07-04-single-source-acp-replay-config.md).