mirror of
https://github.com/deepseek-ai/deepseek-harness
synced 2026-08-15 21:04:50 +00:00
2745879132
ImageBlock had no production producer and every consumer dropped it: the deepseek serializer skipped it, the pi-ai converter skipped it as unrepresentable, the ACP bridge neither advertises image prompt capability nor forwards image blocks, and compact-basic charged a flat 85-token estimate and rendered an [image] placeholder. A block constructed today would silently vanish from the wire — the vocabulary advertised a capability no path honors, the silent-data-loss shape the defensive patterns warn against. The only constructors were tests pinning the skip/estimate branches. Remove ImageBlock and its ContentBlockMap entry (its cache?: CacheHint field leaves with it; CacheHint itself and the other two cache? fields are out of scope). compact-basic loses its explicit image estimate and placeholder arms (the merge-extensible default arms absorb the case); the deepseek serializer, pi-ai converter, and ACP codec already handled image in their default arms, so only their image-naming comments change. The codec's inbound rejection of ACP-protocol image prompt content stays — that guards wire content a client can send regardless of our vocabulary. Tests that constructed harness image blocks to pin the removed branches are dropped (the 85-token estimate pin) or retargeted onto plugin-added block types / other non-text blocks, which the surviving default arms own. Docs, the type-equiv pastes, and the content-block vocabulary RFC's block list and multimodal-home consequence are updated in the same change; the RFC moves to implemented/ and the index is regenerated. A real multimodal feature reintroduces image via declaration merging together with the adapter mapping, ACP advertisement, and compaction pricing that honor it.
66 lines
4.3 KiB
Markdown
66 lines
4.3 KiB
Markdown
# LLM Streaming
|
|
|
|
The wire-level streaming vocabulary of [dsh-llm](../../packages/llm/llm). [core.md](core.md) introduces `StreamChunk`, `Message`, and `ContentBlock`; this page owns the full chunk protocol, the adapter contract every adapter must obey, and the shared assembler.
|
|
|
|
Source: [`packages/llm/llm/src/types.ts`](../../packages/llm/llm/src/types.ts)
|
|
|
|
## `StreamChunk` — the raw protocol
|
|
|
|
A streaming response interleaves several typed blocks (text, reasoning, multiple tool calls). `index` ties each delta to its block; `block-end` carries the fully-assembled `ContentBlock` so consumers don't have to re-assemble deltas themselves. It is a **closed** discriminated union — a `switch` over `type` ends with `assertNever`, so adding a variant breaks compilation at every consumer that must handle it.
|
|
|
|
```ts type-equiv
|
|
type StreamChunk =
|
|
| { type: 'block-start'; index: number; blockType: ContentBlockType }
|
|
| { type: 'text-delta'; index: number; text: string }
|
|
| { type: 'reasoning-delta'; index: number; text: string }
|
|
| { type: 'tool-call-delta'; index: number; id: CallId; name?: string; argumentsDelta: string }
|
|
| { type: 'block-end'; index: number; block: ContentBlock }
|
|
| { type: 'usage'; usage: TokenUsage }
|
|
| { type: 'finish'; reason: FinishReason }
|
|
```
|
|
|
|
## The adapter contract
|
|
|
|
Every adapter MUST obey these, and every consumer may rely on them:
|
|
|
|
- **`usage` before `finish`, nothing after `finish`.** Defer both to the provider's end-of-stream marker so a trailing usage-only chunk can't violate the ordering.
|
|
- **Tool-call `arguments` stay raw JSON strings end-to-end.** Partial fragments stream via `argumentsDelta`; a provider that hands back parsed objects re-stringifies at `block-end`.
|
|
- **Two sanctioned error paths.** A failure may either THROW from `stream()` (transport/protocol errors) **or** end the stream with `finish {kind:'error'|'aborted'}` (provider in-band errors, for adapters that can't throw mid-stream). Consumers must handle *both*. The agent loop translates a finish-error/aborted into a turn error — it never logs a normal completed assistant message for a failed step.
|
|
|
|
This contract is why two adapters exist as a deliberate pair: `dsh-llm-deepseek` (hand-rolled fetch/SSE) and `dsh-llm-pi-ai` (the same endpoint through `@earendil-works/pi-ai`). Two independent internals over one contract is what pinned the protocol down — the library-backed adapter can't throw mid-stream, so it exercises the finish-chunk error path the hand-rolled one might not.
|
|
|
|
## `TokenUsage`
|
|
|
|
Per-call token accounting. Counts are **disjoint**: `inputTokens` is uncached input only; cached input is reported separately, and billed input is the sum of the three. Adapters whose providers fold cache hits into a single prompt total (DeepSeek's `prompt_tokens`) subtract them back out.
|
|
|
|
```ts type-equiv
|
|
interface TokenUsage {
|
|
inputTokens: number
|
|
outputTokens: number
|
|
cacheReadTokens?: number
|
|
cacheWriteTokens?: number
|
|
reasoningTokens?: number
|
|
}
|
|
```
|
|
|
|
## `BlockAssembler`
|
|
|
|
`BlockAssembler` ([`packages/llm/llm/src/assembler.ts`](../../packages/llm/llm/src/assembler.ts)) is the single shared implementation that folds a `StreamChunk` stream back into `ContentBlock`s and a final `Message`. The loop logs the raw chunks (for replay fidelity) while feeding the same chunks through an assembler — so the canonical log keeps token-level detail and the derived message is rebuilt deterministically. A consumer that needs the assembled result without re-implementing the fold uses this.
|
|
|
|
## The seam
|
|
|
|
`LlmAdapter` is the provider seam: subclass, implement `stream()`, register with `ctx.llm.registerAdapter(models, adapter)`. The `block-start` / `block-end` `index` correlation and the assembler together mean an adapter only has to emit well-formed chunks — block reassembly is not each adapter's problem. The consumer surface (`ctx.llm.stream()`) and the `llm/stream` waterfall are described in [architecture.md § The vocabulary](../architecture.md#the-vocabulary-dsh-llm).
|
|
|
|
`ContentBlockType` (the key set the `index`-correlated blocks carry) derives from `ContentBlockMap`:
|
|
|
|
```ts type-equiv
|
|
interface ContentBlockMap {
|
|
'text': TextBlock
|
|
'reasoning': ReasoningBlock
|
|
'tool-call': ToolCallBlock
|
|
'tool-result': ToolResultBlock
|
|
}
|
|
```
|
|
|
|
See [core.md § Content blocks and messages](core.md#content-blocks-and-messages) for the block interfaces.
|