LLM adapters

English | 中文

This guide connects a new LLM provider to Harness.

Overview

An LLM adapter extends LlmAdapter and implements stream(), translating Harness's provider-neutral request into a provider API call and translating the response back into Harness chunks.

Minimal implementation

import type { Context } from 'cordis'
import Schema from 'schemastery'
import { LlmAdapter, type GenerateOptions, type StreamChunk } from '@deepseek-ai/dsh-llm'

class MyAdapter extends LlmAdapter {
  private apiKey: string

  constructor(apiKey: string) {
    super()
    this.apiKey = apiKey
  }

  async *stream(options: GenerateOptions): AsyncIterable<StreamChunk> {
    // 1. Convert options.messages to the provider format.
    // 2. Call the streaming API.
    // 3. Convert the response into StreamChunk values.
  }
}

export interface Config {
  apiKey: string
  models: string[]
}

export const Config: Schema<Config> = Schema.object({
  apiKey: Schema.string().required(),
  models: Schema.array(Schema.string()).required(),
})

export const name = 'my-llm-adapter'
export const inject = ['llm']

export function apply(ctx: Context, config: Config) {
  const adapter = new MyAdapter(config.apiKey)
  ctx.llm.registerAdapter(config.models, adapter)
}

StreamChunk protocol

stream() yields chunks using this protocol:

import { CallId, type StreamChunk } from '@deepseek-ai/dsh-llm'

async function* exampleChunks(): AsyncIterable<StreamChunk> {
  // 1. Start each content block with block-start.
  yield { type: 'block-start', index: 0, blockType: 'text' }

  // 2. Stream text through text-delta.
  yield { type: 'text-delta', index: 0, text: 'Hello' }
  yield { type: 'text-delta', index: 0, text: ' world' }

  // 3. End each content block with block-end and the complete block.
  yield {
    type: 'block-end',
    index: 0,
    block: { type: 'text', text: 'Hello world' },
  }

  // 4. Tool-call block.
  yield { type: 'block-start', index: 1, blockType: 'tool-call' }
  yield {
    type: 'tool-call-delta',
    index: 1,
    id: CallId('call-123'),
    name: 'bash',
    argumentsDelta: '{"command":"ls"}',
  }
  yield {
    type: 'block-end',
    index: 1,
    block: {
      type: 'tool-call',
      id: CallId('call-123'),
      name: 'bash',
      arguments: '{"command":"ls"}',
    },
  }

  // 5. Token usage.
  yield { type: 'usage', usage: { inputTokens: 100, outputTokens: 50 } }

  // 6. Finish reason.
  yield { type: 'finish', reason: { kind: 'stop' } }
  // Alternatively, { kind: 'tool-calls' } requests tool execution.
}

Key rules

Every block-start has a matching block-end.
index increases from 0 and identifies content-block order.
A tool-call-delta carries raw JSON text in argumentsDelta, either all at once or over multiple chunks.
finish is the final chunk.
Emit usage before finish.

GenerateOptions

stream() receives the exported GenerateOptions type. It includes the model, adapter-owned reasoning-effort id, conversation history, system prompt, tool schemas, generation parameters, stop sequences, and abort signal; treat the TypeScript type exported by @deepseek-ai/dsh-llm as authoritative. Map supported fields to the provider API. If the provider cannot honor a field, throw LlmError with a stable code instead of silently dropping it.

Override resolveModelReasoning(provider, model, signal?) when an exact model exposes selectable reasoning strengths. Return ordered opaque ids and display names plus an optional configured default; do not promote provider names into a core enum. Honor the optional signal for asynchronous lookup so cancellation and disposal reach quiescence. The service validates the metadata and rejects unsupported explicit values before stream(). Returning undefined means that model has no selectable reasoning-effort capability.

Register an adapter

ctx.llm.registerAdapter(['model-name-1', 'model-name-2'], adapter)

The first argument lists the model names handled by the adapter. If cordis.yml selects model: model-name-1, the service routes that request to this adapter.

Use it from cordis.yml

- id: my-llm
  name: './src/my-llm-adapter.ts'
  config:
    apiKey: !!js process.env.MY_API_KEY
    models:
      - my-model-v1
      - my-model-v2

- id: tui-agent
  name: '@deepseek-ai/dsh-tui-demo'
  config:
    provider: my-llm
    model: my-model-v1  # References the model registered above.
    workspaceContext: false

Reference implementations

The repository contains complete implementations:

packages/llm/llm-deepseek/ — DeepSeek API adapter using the OpenAI-compatible format
packages/llm/llm-pi-ai/ — Pi AI adapter using a different API format

Compare the two shipped adapters to see the same harness contract implemented over different provider SDKs.

Error handling

Adapters throw transport and protocol failures as LlmError values with stable codes. The agent loop preserves the error and code for diagnostics and policy; it does not convert an ordinary Error automatically. Every provider HTTP request must also merge attributionHeaders() and forward options.signal.

import {
  attributionHeaders,
  LlmAdapter,
  LlmError,
  type GenerateOptions,
  type StreamChunk,
} from '@deepseek-ai/dsh-llm'

class HttpAdapter extends LlmAdapter {
  constructor(private readonly endpoint: string) {
    super()
  }

  async *stream(options: GenerateOptions): AsyncIterable<StreamChunk> {
    const response = await fetch(this.endpoint, {
      method: 'POST',
      headers: {
        'content-type': 'application/json',
        ...attributionHeaders(),
      },
      body: JSON.stringify({ model: options.model, messages: options.messages }),
      ...options.signal ? { signal: options.signal } : {},
    })
    if (!response.ok) {
      throw new LlmError(`Provider API error: ${response.status}`, 'PROVIDER_HTTP_ERROR')
    }
    // A real adapter parses the response and emits the complete chunk sequence.
    yield { type: 'finish', reason: { kind: 'stop' } }
  }
}

6.0 KiB Raw Blame History