deepseek-harness/docs/core-data-structures/llm-streaming.md

LLM Streaming

The wire-level streaming vocabulary of dsh-llm. core.md introduces StreamChunk, Message, and ContentBlock; this page owns the full chunk protocol, the adapter contract every adapter must obey, and the shared assembler.

Source: packages/llm/llm/src/types.ts

StreamChunk — the raw protocol

A streaming response interleaves several typed blocks (text, reasoning, multiple tool calls). index ties each delta to its block; block-end carries the fully-assembled ContentBlock so consumers don't have to re-assemble deltas themselves. It is a closed discriminated union — a switch over type ends with assertNever, so adding a variant breaks compilation at every consumer that must handle it.

type StreamChunk =
  | { type: 'block-start'; index: number; blockType: ContentBlockType }
  | { type: 'text-delta'; index: number; text: string }
  | { type: 'reasoning-delta'; index: number; text: string }
  | { type: 'tool-call-delta'; index: number; id: CallId; name?: string; argumentsDelta: string }
  | { type: 'block-end'; index: number; block: ContentBlock }
  | { type: 'usage'; usage: TokenUsage }
  | {
    type: 'finish'
    reason: FinishReason
    /** Adapter-private lossless-JSON state for replaying a successful response. */
    replayState?: unknown
  }

The adapter contract

Every adapter MUST obey these, and every consumer may rely on them:

• usage before finish, nothing after finish. Defer both to the provider's end-of-stream marker so a trailing usage-only chunk can't violate the ordering.
• Tool-call arguments stay raw JSON strings end-to-end. Partial fragments stream via argumentsDelta; a provider that hands back parsed objects re-stringifies at block-end.
• Two sanctioned error paths. A failure may either THROW from stream() (transport/protocol errors) or end the stream with finish {kind:'error'|'aborted'} (provider in-band errors, for adapters that can't throw mid-stream). Consumers must handle both. The agent loop closes the failed step and offers either form to agent/request-error; absent recovery it becomes a turn error, and no normal completed assistant message is logged for that request.
• Context overflow has one canonical code. Both DeepSeek adapters classify explicit provider detail through isContextWindowExceededError() and surface CONTEXT_WINDOW_EXCEEDED, whether the failure arrives as a thrown HTTP LlmError or an in-band finish error. Consumers route on the code, never provider text.
• Every provider HTTP request carries the app-attribution header. Adapters send attributionHeaders() (below) - the User-Agent baseline - and prove it with a wire-level test (mock server asserting the received header, or the library's header hook for a library-backed adapter).
• Replay state is adapter-owned. A successful finish may carry lossless-JSON state needed to reconstruct a native provider response. The loop stores it with the assembled assistant message unless an agent/step-result listener rewrote the content. On a later request, LlmService passes the state only when the historical provider and target provider are currently registered to the exact same adapter instance. That adapter validates the state and owns any cross-model or cross-provider conversion; other adapters receive the provider-neutral content and provenance without the private state.

This contract was pinned down by two deliberately independent implementations: dsh-llm-deepseek (hand-rolled fetch/SSE) and dsh-llm-pi-ai (a generic multi-provider adapter through @earendil-works/pi-ai). The library-backed adapter cannot throw mid-stream, so it exercises the finish-chunk error path the hand-rolled one might not.

AppIdentity — app attribution

The static public application identity every adapter sends to providers (packages/llm/llm/src/attribution.ts). attributionHeaders(identity?) maps it to the standard User-Agent header only; OpenRouter-specific app attribution headers are intentionally not supported by this contract. The default APP_IDENTITY sources its version from the package manifest; every field is a public product fact - no secrets, paths, session ids, or per-user identifiers, and nothing per-request may influence the values. Rationale: Mandatory User-Agent attribution.

interface AppIdentity {
  product: string
  version: string
  url: string
}

TokenUsage

Per-call token accounting. Counts are disjoint: inputTokens is uncached input only; cached input is reported separately, and billed input is the sum of the three. Adapters whose providers fold cache hits into a single prompt total (DeepSeek's prompt_tokens) subtract them back out. reasoningTokens, when present, is informational detail already included in outputTokens; totals must not add it again.

interface TokenUsage {
  inputTokens: number
  outputTokens: number
  cacheReadTokens?: number
  cacheWriteTokens?: number
  reasoningTokens?: number
}

BlockAssembler

BlockAssembler (packages/llm/llm/src/assembler.ts) is the single shared implementation that folds a StreamChunk stream back into ContentBlocks, usage, finish reason, and replay state. The loop logs the raw chunks while feeding the same chunks through an assembler, then stores the assembled assistant content with its provider/model provenance. A consumer that needs the assembled result without re-implementing the fold uses this.

The seam

LlmAdapter is the provider seam: subclass, implement stream(), and register one adapter instance with ctx.llm.registerAdapter(providers, adapter). GenerateOptions.provider selects the registered adapter; GenerateOptions.model is passed to that adapter and need not be registered at lifecycle start. Duplicate provider routes fail atomically. Optional providerInfo() and asynchronous listModels() methods feed LlmService.listProviders() / listModels() with detached selector metadata. That catalog is advisory rather than a request whitelist: the adapter remains authoritative and may accept unlisted model ids. Adapter lookup happens at the terminal continuation of the llm/stream waterfall, so a listener may short-circuit the call or route a mutable one-shot request before lookup. The block-start / block-end index correlation and the assembler together mean an adapter only has to emit well-formed chunks — block reassembly is not each adapter's problem. The consumer surface (ctx.llm.stream()) and the llm/stream waterfall are described in architecture.md § Content blocks and streaming.

ContentBlockType (the key set the index-correlated blocks carry) derives from ContentBlockMap:

interface ContentBlockMap {
  'text': TextBlock
  'reasoning': ReasoningBlock
  'tool-call': ToolCallBlock
  'tool-result': ToolResultBlock
}

See core.md § Content blocks and messages for the block interfaces.