deepseek-harness/docs/cookbook/adding-an-llm-adapter.md

Cookbook: adding an LLM adapter

English | 中文

How to connect a new model provider. Reference implementations: packages/llm/llm-deepseek (hand-rolled HTTP/SSE) and packages/llm/llm-pi-ai (wrapping an LLM library). Read the StreamChunk doc in packages/llm/llm/src/types.ts first — it records the protocol conventions both adapters were verified against.

The shape

class MyAdapter extends LlmAdapter {
  async * stream(options: GenerateOptions): AsyncIterable<StreamChunk> { … }
}

export const name = 'llm-myprovider'
export const inject = ['llm']
export const Config: z<Config> = z.object({ apiKey: z.string(), … })

export function apply(ctx: Context, config: Config) {
  ctx.llm.registerAdapter(['my-provider'], new MyAdapter(…))
}

Registration is effect-based (HMR-safe); one adapter per provider route — duplicates throw, and multi-route registration is all-or-nothing. options.provider selects the adapter and options.model is the provider model id, so a dynamic catalog adapter can serve new models without lifecycle reconfiguration. Secrets are cordis-native: schemastery Config with env fallbacks, fed from cordis.yml via !!js process.env.MY_KEY. Never read ad-hoc key files in code.

Protocol obligations (the contract two implementations verified)

• Emit usage BEFORE finish; emit NOTHING after finish. The robust way: buffer finish/usage until the provider's end-of-stream marker, then flush (handles providers that send trailing usage-only chunks).
• Tool-call arguments are RAW JSON strings end-to-end; stream fragments as argumentsDelta. If your provider hands back parsed objects, re-stringify at block-end.
• Allocate block indexes in first-seen stream order; reuse the index for every delta of the same block.
• Errors have exactly two sanctioned paths: THROW from stream() (transport and protocol failures — use LlmError with a stable code), or end the stream with finish {kind: 'error' | 'aborted'} (provider in-band failures). Consumers handle both; pick per failure class and document it.
• Honor options.signal (pass it to fetch / your SDK).
• A GenerateOptions field your provider cannot honor (e.g. a stop list on a provider without stop sequences): throw LlmError(..., 'UNSUPPORTED') rather than silently dropping it.
• If the provider requires response ids, signatures, or other native metadata on follow-up calls, emit the minimal lossless-JSON projection as finish.replayState. Validate it when rebuilding history. LlmService passes it only when the historical provider route and target provider route are currently owned by the exact same adapter instance; your adapter decides whether same-model, cross-model, or cross-provider restoration is legal. Never infer native replay from provider/model names alone when state is absent.

Provider-specific request knobs (thinking modes, effort levels) belong in the ADAPTER's Config, not in GenerateOptions — the core vocabulary stays provider-neutral.

Structure that worked

Split the adapter into testable stages (llm-deepseek's layout): wire types (types.ts, coverage-exempt) → request serializer → SSE/transport parser → chunk-translation state machine → a thin adapter class wiring them. Each stage gets its own unit suite.

Testing

• Unit: mock the provider, not the harness. A scripted node:http server speaking the provider's wire format covers happy paths, every error status, malformed payloads, premature closes, and aborts — no network, and it drives the 100% per-file coverage gate. Works for SDK-backed adapters too (point the SDK's baseURL at the mock).
• Hostile framing tests. Split stream payloads at arbitrary byte positions (including mid-UTF-8) — real networks do.
• E2E: tests/*.e2e.ts under pnpm run test:e2e, gated with describe.skipIf(!process.env.MY_KEY) so CI (no secrets) stays green. Cover representative model/provider/API families and every provider mode you map, a tool-call round trip INCLUDING the follow-up turn with results in history, and loose assertions only (substring/structure, bounded maxTokens — real models are nondeterministic).
• Register the e2e file pattern in knip.json (per-workspace entry override) or knip flags it unused.