Merge remote-tracking branch 'origin/master' into codex/skill-system

# Conflicts:
#	docs/event-producer-consumer.md
This commit is contained in:
Yichen Jiang
2026-07-06 17:11:35 +08:00
124 changed files with 7426 additions and 6716 deletions

View File

@@ -11,7 +11,7 @@ Compaction extends [`SessionEventMap`](session.md) with three event types via de
| Event | Payload | Role |
|---|---|---|
| `compact/start` | `{ turn }` | acquires the log-recorded lock |
| `compact/summary` | `{ summary, shadowedRange, shadowedSeqs, shadowedTokenCount }` | provenance: the summary blocks, the shadowed surface-boundary pair (`start`/`end` seqs — a position span, not a numeric interval), the shadowed seqs in surface order, and the estimated token count |
| `compact/summary` | `{ summary, shadowedRange, shadowedSeqs, shadowedTokenCount, model, maxTokens? }` | provenance: the summary blocks, the shadowed surface-boundary pair (`start`/`end` seqs — a position span, not a numeric interval), the shadowed seqs in surface order, the estimated token count, and the summarize call's envelope (`model`, plus its generation cap when one applied) — logged so the one-shot request is reconstructable from log + code (the reconstructability RFC) |
| `compact/end` | `{ turn, error? }` | releases the lock (`error` set when summarization threw) |
The lock brackets the **whole** operation: `compact/start` is appended first, then summarization, the `compact/summary` provenance record, and the `user/message` replacement all land, and only then `compact/end`. Releasing the lock last turns a crash mid-operation into a detectable orphaned lock (a `compact/start` with no matching `compact/end`) rather than a `compact/end` that falsely claims compaction finished.
@@ -50,6 +50,6 @@ interface CompactionResult {
## The service
`CompactService` (`ctx.compact`, abstract — defined in [`packages/compact/compact/src/index.ts`](../../packages/compact/compact/src/index.ts)) declares two abstract methods: `compactIfNeeded(agent, turn, step, fullSystemPrompt, signal)` checks token pressure and compacts an older range if the history is too large (returning `null` when nothing needs it), and `compactRegion(session, start, end, agent, turn, step, signal?)` forcibly summarizes surface nodes `[start, end]` into a single replacement node. `compactIfNeeded`'s parameters are all required — the loop's `agent/pre-step` checkpoint supplies the agent, lifecycle context, assembled `fullSystemPrompt`, and turn `signal`. A backend summarizing via `ctx.llm.stream()` must forward `signal` into the call's `GenerateOptions.signal`, so an abort or dispose tears down the in-flight summarization. The entire strategy — token estimation, retention policy, event sequencing, summarization — is a HOW decision owned by the implementation.
`CompactService` (`ctx.compact`, abstract — defined in [`packages/compact/compact/src/index.ts`](../../packages/compact/compact/src/index.ts)) declares two abstract methods: `compactIfNeeded(agent, fullSystemPrompt, signal)` checks token pressure and compacts an older range if the history is too large (returning `null` when nothing needs it), and `compactRegion(session, start, end, agent, signal?)` forcibly summarizes surface nodes `[start, end]` into a single replacement node. `compactIfNeeded`'s parameters are all required — the loop's `agent/pre-step` checkpoint supplies the agent, the assembled `fullSystemPrompt`, and the turn `signal`. A backend summarizing via `ctx.llm.stream()` must forward `signal` into the call's `GenerateOptions.signal`, so an abort or dispose tears down the in-flight summarization. The entire strategy — token estimation, retention policy, event sequencing, summarization — is a HOW decision owned by the implementation.
Auto-compaction runs on the serial `agent/pre-step` loop seam (fired once per step, after `turn/start` and BEFORE the step opens and its request history is derived), not the `agent/request` waterfall: compaction mutates the session surface in place — with its log-only `compact/*` records landing cleanly outside any step — and the loop derives the request from the already-compacted surface. Retention is turn-agnostic — the only structural guard is tool-pairing balance (a compacted region's edges are balanced cuts on the surface, so it never splits a step's tool-calls from their results), so a single runaway turn that alone exceeds the window compacts its own early closed steps rather than being retained verbatim. The backend that ships this (`dsh-compact-basic`) documents the retention walk, summary shrink validation, bounded re-compaction, and the crash/recoverable failure taxonomy.

View File

@@ -28,6 +28,8 @@ Everything else is documented on a **sub-page**, not here. The rule that draws t
> Type definitions on this page are pasted **verbatim** from source and drift-checked by `pnpm run verify-type-equiv` (see [development.md](../development.md#documenting-types-verbatim-ts-type-equiv)). Inline JSDoc is omitted for readability; follow the source link for the full contracts.
FIXME(catalog-verbs): the drift gate covers only the nouns (the pasted type shapes); every method surface on these pages is hand-written prose. core-data-structures should probably also generate the *verbs* — the public methods of the cataloged classes — so a signature change cannot silently outdate the catalog.
## The `…Map → derived-union` pattern
Almost every extensible sum type in the harness follows one shape: an interface keyed by a discriminant tag (the `…Map`), from which the union is derived with `keyof`. Plugins add variants by **declaration merging** — no edit to the owning package.
@@ -184,6 +186,21 @@ interface ToolSchema {
The model-facing `ToolSchema` is the wire shape; the registered `ToolDefinition` that produces it (schema + `execute`) is on [tools.md](tools.md).
### The request envelope: `LlmCallConfig` and the logged header
Requests are built by the loop, not shaped per call: the non-content half of a request — the `EpochHeader`: this call configuration plus the rendered system prompt and assembled tool schemas — is logged session state (`request/header` snapshot and delta events, [session.md](session.md#the-request-header-events-requestheader-and-requestheader-delta)), so every conversation request is a pure function of the session log ([reconstructability RFC](../rfc/implemented/architecture/2026-07-05-reconstructable-requests.md)). The `agent/request` waterfall receives a frozen `LlmCallConfig` seed and a listener returns a replacement to switch model or sampling — the loop logs whatever the request actually uses. Loop-built requests arrive at `llm/stream` deep-frozen; mutation throws.
FIXME(call-config-shape): revisit the exact definition of this type — which fields are genuinely epoch-level for cache purposes (`model` certainly; the sampling scalars sit here out of caution), and where provider-specific extras (reasoning options, extra body params) belong when an adapter needs them.
```ts type-equiv
interface LlmCallConfig {
model: string
temperature?: number
maxTokens?: number
stop?: string[]
}
```
## Sessions
A `Session` is an **append-only log** of typed `SessionEvent`s — the single source of truth. The LLM message history is *derived* from the log (`deriveMessages()`), not stored separately. The event vocabulary derives from `SessionEventMap`:
@@ -212,7 +229,7 @@ type SessionEvent<T extends SessionEventType = SessionEventType> = {
}[T]
```
The thirteen event variants (`turn/start`, `turn/end`, `step/start`, `step/end`, `user/message`, `prompt/blocked`, `context/message`, `assistant/chunk`, `assistant/message`, `tool/call`, `tool/result`, `steering/message`, `todo/write`), the `deriveMessages()` projection rules, the `TurnTrigger`/`TurnEndReason` reasons, and the turn-enclosure invariant are on **[session.md](session.md)**. How the log is made durable — the `SessionPersistence` seam, JSONL/SQLite backends, the `session/flush` checkpoint, crash recovery, and `SessionHeader` — is on **[persistence.md](persistence.md)**.
The fifteen event variants (`turn/start`, `turn/end`, `step/start`, `step/end`, `user/message`, `prompt/blocked`, `context/message`, `assistant/chunk`, `assistant/message`, `tool/call`, `tool/result`, `steering/message`, `todo/write`, `request/header`, `request/header-delta`), the `deriveMessages()` projection rules, the `TurnTrigger`/`TurnEndReason` reasons, and the turn-enclosure invariant are on **[session.md](session.md)**. How the log is made durable — the `SessionPersistence` seam, JSONL/SQLite backends, the `session/flush` checkpoint, crash recovery, and `SessionHeader` — is on **[persistence.md](persistence.md)**.
## The agent handle

View File

@@ -60,6 +60,30 @@ interface SessionEventMap {
* cordis-catalog row.
*/
'todo/write': { todos: TodoItem[] }
/**
* Full snapshot of the {@link EpochHeader} the NEXT request is built under,
* with the {@link RequestHeaderReason} it was recorded whole. Appended by
* the loop inside the step, before dispatch, on a loop instance's first
* request-building step (`'initial'`/`'resume'`) or when a delta failed its
* round-trip guard (`'fallback'`); always records what the request actually
* used, post-`agent/request`. Anchors the header fold: reconstruction reads
* the latest snapshot and applies the deltas after it. NOT a
* {@link SurfaceEventType}: it produces no LLM message — it is the request
* envelope, logged so every request is a pure function of the session log
* (the reconstructability RFC).
*/
'request/header': { header: EpochHeader; reason: RequestHeaderReason }
/**
* Amendment to the folded {@link EpochHeader}: at least one of a
* {@link SystemDelta}, a {@link ToolsDelta}, or a whole replacement
* {@link LlmCallConfig} (four scalars — not worth diffing). Appended by the
* loop inside the step, before dispatch, when the header for this request
* differs from the fold of the log so far; the writer verifies
* `applyHeaderDelta(previous, delta)` reproduces the new header exactly and
* falls back to a `'fallback'` `request/header` snapshot when it cannot, so
* a logged delta ALWAYS round-trips. NOT a {@link SurfaceEventType}.
*/
'request/header-delta': { system?: SystemDelta; tools?: ToolsDelta; config?: LlmCallConfig }
}
```
@@ -74,6 +98,23 @@ export interface TodoItem {
}
```
### The request header events: `request/header` and `request/header-delta`
The request envelope — the `EpochHeader` (call config + rendered system prompt + assembled tool schemas) — is logged session state, so every conversation request is a pure function of the log (the reconstructability RFC). A `request/header` snapshot (reason `'initial' | 'resume' | 'fallback'`) anchors the fold at conversation birth, process boundaries, and delta-encoding fallbacks; `request/header-delta` events amend it mid-run. `foldRequestHeader(events)` reconstructs the header any request was built under; the writer round-trip-verifies every delta before logging it, so a well-formed log always folds. Neither is a `SurfaceEventType` — they produce no LLM message.
```ts type-equiv
export interface EpochHeader {
/** The conversation's call configuration (model + sampling scalars). */
config: LlmCallConfig
/** Rendered system prompt text; absent for a system-less request. */
system?: string
/** Assembled tool schemas; absent for a tool-less request. */
tools?: ToolSchema[]
}
```
Canonical form: an empty system prompt and an empty tool list are ABSENT fields, matching how requests are built. The delta payloads (`SystemDelta` — a common-prefix/suffix line trim; `ToolsDelta` — name-keyed added/removed/changed) live beside the events in [`packages/core/session/src/types.ts`](../../packages/core/session/src/types.ts).
## `SessionEvent<T>` — one log entry
A proper discriminated union over `type` (not independent `type`/`data` unions), so `switch (event.type)` narrows `event.data` without casts. `seq` is the monotonic position in the log (`seq = log.length`); `time` is epoch ms.
@@ -148,9 +189,9 @@ export interface SurfaceNode {
}
```
## Derived history: `deriveMessages()`
## Derived history: `deriveMessages()` and `deriveEventMessage()`
`Session.deriveMessages()` projects the event log into the `Message[]` the model sees. The projection rules:
`Session.deriveMessages()` projects the event log into the `Message[]` the model sees — cached (each surface node projected once, when first seen; a surface rewrite rebuilds) and frozen (a fresh array per call over shared, deep-frozen messages, so mutating logged history through a projection is unrepresentable). `deriveEventMessage(event)` is the per-node pure function the fold applies — public so external reconstructors and the dev invariant project a log prefix with exactly the same rules and cannot disagree with the cache. The projection rules:
- `user/message` → a user message.
- `assistant/message` → an assistant message. Raw `assistant/chunk` events are replay/UI data and are **skipped** in derivation (the assembled message is authoritative). An **empty-content** `assistant/message` is also skipped — a max-tokens step cut off with no content still records an `assistant/message` to host its `usage`, but a content-less assistant turn must not enter the provider transcript.

View File

@@ -86,6 +86,6 @@ interface Config {
## Prompt and tool contract
`ctx.skills.renderModelListing()` returns a `## Skills` fragment wrapped in `<available_skills>`. Descriptions and `whenToUse` are whitespace-normalized, length-capped, and XML-escaped before rendering. The listing is appended to the same `GenerateOptions.system` string by the `agent/request` waterfall, after the base system prompt is assembled.
`ctx.skills.renderModelListing()` returns a `## Skills` fragment wrapped in `<available_skills>`. Descriptions and `whenToUse` are whitespace-normalized, length-capped, and XML-escaped before rendering. The listing is appended as a late `system-prompt/assemble` section for the calling agent, so it remains cwd-sensitive while still flowing through the reconstructable system-prompt path.
The model-facing `skill({ name })` tool validates the kebab-case name, loads the complete definition for the calling agent cwd, rejects unknown or `disableModelInvocation` skills, and returns a `<skill_content name="...">` block with the body plus base-directory and relative-path guidance. The tool result is the only v1 path that exposes full skill instructions to the model.