diff --git a/docs/cordis-catalog/services.md b/docs/cordis-catalog/services.md index b858cb6260..e384bf3ad0 100644 --- a/docs/cordis-catalog/services.md +++ b/docs/cordis-catalog/services.md @@ -144,7 +144,7 @@ stream(options: GenerateOptions): AsyncIterable Types: [GenerateOptions](../core-data-structures/core.md) · [StreamChunk](../core-data-structures/llm-streaming.md) -Source: [`packages/llm/llm/src/index.ts:96`](../../packages/llm/llm/src/index.ts) +Source: [`packages/llm/llm/src/index.ts:94`](../../packages/llm/llm/src/index.ts) ## `ctx.permission` — `PermissionService` diff --git a/docs/rfc/implemented/testing/2026-06-11-property-based-testing.md b/docs/rfc/implemented/testing/2026-06-11-property-based-testing.md index 67404ab49a..db7a9c8a79 100644 --- a/docs/rfc/implemented/testing/2026-06-11-property-based-testing.md +++ b/docs/rfc/implemented/testing/2026-06-11-property-based-testing.md @@ -20,7 +20,7 @@ Adopt `fast-check` (a root devDependency) with one `tests/properties.spec.ts` pe ## Consequences - Generator quality is the value lever — the generators bias toward small index pools and short strings so collisions and interleavings are common. -- **It already paid off:** the BlockAssembler stream found a real bug — a duplicate `block-end` at the same index overwrote an already-flushed block, so the streamed prefix disagreed with final `blocks()`. Fixed (first close wins, matching the existing straggler rule) with a dedicated regression test. +- **It already paid off:** the BlockAssembler stream found a real bug — a duplicate `block-end` at the same index rewrote a completed block. Fixed (first close wins, matching the existing straggler rule) with a dedicated regression test. - A property flake from a timeout is a finding, not something to retry away. The loop properties are deterministic by construction (settle on `agent/status`), so a hang is a real defect. - Property tests supplement, not replace, the example tests that pin specific branches for the 100%-coverage gate. diff --git a/docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md b/docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md index 5f05c92317..238f2044ad 100644 --- a/docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md +++ b/docs/rfc/implemented/testing/2026-06-19-acp-snapshot-tests.md @@ -28,7 +28,7 @@ Each scenario's `session.jsonl` is harvested from a real run. `assistant/chunk` ``` { kind: 'chunks', chunks: StreamChunk[] } -| { kind: 'throw', chunks: StreamChunk[], message: string, code: string, status?: number } +| { kind: 'throw', chunks: StreamChunk[], message: string, code: string } | { kind: 'hang' } ``` diff --git a/docs/rfc/implemented/testing/2026-06-20-remove-redundant-snapshot-log-goldens.md b/docs/rfc/implemented/testing/2026-06-20-remove-redundant-snapshot-log-goldens.md index 45348cfd2e..5c9e1014b8 100644 --- a/docs/rfc/implemented/testing/2026-06-20-remove-redundant-snapshot-log-goldens.md +++ b/docs/rfc/implemented/testing/2026-06-20-remove-redundant-snapshot-log-goldens.md @@ -6,7 +6,7 @@ Status: implemented Model-driving ACP snapshot scenarios ship both `session.jsonl` and `session.golden.jsonl`. For normal recorded scenarios, `session.jsonl` is the replay fixture harvested from a real run, and the replay test normalizes the newly persisted log and compares it to `session.golden.jsonl`. In the current fixtures, the normalized recorded log and normalized golden are identical for the ordinary recorded scenarios. -Authored override scenarios (`error-finish`, `cancel`) currently use `replay.override.json` to drive model behavior and keep `session.jsonl` as a minimal dummy fixture, while `session.golden.jsonl` holds the expected persisted log. The override file is a JSON array of `ReplayEntry` objects: `{ "kind": "chunks", "chunks": StreamChunk[] }`, `{ "kind": "throw", "chunks": StreamChunk[], "message": string, "code": string, "status"?: number }`, or `{ "kind": "hang" }`. That split is also unnecessary: when an override sidecar exists, `llm-replay` replaces the derived script and does not need `session.jsonl` for model chunks, so `session.jsonl` can still be the expected session-log artifact for the scenario. +Authored override scenarios (`error-finish`, `cancel`) currently use `replay.override.json` to drive model behavior and keep `session.jsonl` as a minimal dummy fixture, while `session.golden.jsonl` holds the expected persisted log. The override file is a JSON array of `ReplayEntry` objects: `{ "kind": "chunks", "chunks": StreamChunk[] }`, `{ "kind": "throw", "chunks": StreamChunk[], "message": string, "code": string }`, or `{ "kind": "hang" }`. That split is also unnecessary: when an override sidecar exists, `llm-replay` replaces the derived script and does not need `session.jsonl` for model chunks, so `session.jsonl` can still be the expected session-log artifact for the scenario. ## Decision diff --git a/examples/acp-agent/tests/snapshots/error-finish/replay.override.json b/examples/acp-agent/tests/snapshots/error-finish/replay.override.json index eea32f25ca..cfa0d84227 100644 --- a/examples/acp-agent/tests/snapshots/error-finish/replay.override.json +++ b/examples/acp-agent/tests/snapshots/error-finish/replay.override.json @@ -1,3 +1,3 @@ [ - { "kind": "throw", "chunks": [], "message": "simulated provider error (HTTP 401)", "code": "AUTH", "status": 401 } + { "kind": "throw", "chunks": [], "message": "simulated provider error (HTTP 401)", "code": "AUTH" } ] diff --git a/packages/llm/llm-deepseek/src/adapter.ts b/packages/llm/llm-deepseek/src/adapter.ts index ece053c6b3..5fa01ce503 100644 --- a/packages/llm/llm-deepseek/src/adapter.ts +++ b/packages/llm/llm-deepseek/src/adapter.ts @@ -98,10 +98,10 @@ export class DeepSeekAdapter extends LlmAdapter { const parsed = await response.json() as WireError if (parsed.error?.message) message = parsed.error.message } catch { - // Only swallow error-body parsing: status and code are already captured, - // so malformed gateway JSON must not mask the actionable HTTP failure. + // Only swallow error-body parsing: the stable code and status-line message + // are already captured, so malformed gateway JSON must not mask the failure. } - throw new LlmError(message, code, response.status) + throw new LlmError(message, code) } if (!response.body) { throw new LlmError('DeepSeek API returned no response body', 'EMPTY_RESPONSE') diff --git a/packages/llm/llm-deepseek/tests/adapter.spec.ts b/packages/llm/llm-deepseek/tests/adapter.spec.ts index c0a2f8f482..d1fc163651 100644 --- a/packages/llm/llm-deepseek/tests/adapter.spec.ts +++ b/packages/llm/llm-deepseek/tests/adapter.spec.ts @@ -159,7 +159,7 @@ describe('DeepSeekAdapter against a mock server', () => { status, body: JSON.stringify({ error: { message: `failed with ${status}`, type: 't', code: 'c' } }), } - const server = await mockServer([behavior, behavior, behavior]) + const server = await mockServer([behavior, behavior]) const ctx = await harness(server.url) await expect(assemble(ctx,{ model: 'deepseek-v4-flash', messages: [] })) .rejects.toThrow(`failed with ${status}`) @@ -167,11 +167,6 @@ describe('DeepSeekAdapter against a mock server', () => { assemble(ctx,{ model: 'deepseek-v4-flash', messages: [] }) .catch((error: unknown) => (error as LlmError).code), ).resolves.toBe(code) - // The numeric HTTP status is carried on the error for explicit handling. - await expect( - assemble(ctx,{ model: 'deepseek-v4-flash', messages: [] }) - .catch((error: unknown) => (error as LlmError).status), - ).resolves.toBe(status) }) it('keeps the status-line message for JSON error bodies without a message', async () => { diff --git a/packages/llm/llm-pi-ai/README.md b/packages/llm/llm-pi-ai/README.md index bb63d82010..22369dc187 100644 --- a/packages/llm/llm-pi-ai/README.md +++ b/packages/llm/llm-pi-ai/README.md @@ -76,4 +76,4 @@ Unit tests use pi-ai catalog models redirected to local mock servers and cover p - **Catalog membership is required** — custom model ids that are absent from the installed pi-ai catalog fail with `UNKNOWN_MODEL`, even when a provider profile supplies a custom endpoint. - **`GenerateOptions.stop` is unsupported** — pi-ai's common stream options cannot guarantee stop-sequence behavior across providers, so the adapter rejects the field. - **In-history `system` messages use pi-ai's common context conversion** — provider-specific placement follows pi-ai rather than a harness-owned wire override. -- **`LlmError.status` is unavailable for in-stream failures** — pi-ai error events do not expose a stable HTTP status across providers. +- **Provider HTTP status is unavailable** — pi-ai error events do not expose a stable HTTP status across providers; failures expose only stable harness error codes. diff --git a/packages/llm/llm/README.md b/packages/llm/llm/README.md index a67859ed19..039844040e 100644 --- a/packages/llm/llm/README.md +++ b/packages/llm/llm/README.md @@ -45,7 +45,7 @@ Every product adapter sends application identity on provider HTTP requests. `att - `LlmAdapter` — abstract base class for provider adapters. The only required method is `stream()`. - `BlockAssembler` — incrementally assembles raw chunks into complete content blocks and an assistant message. The agent loop feeds it raw chunks (logging them for replay) while reading the assembled blocks/message for history. - `HarnessError` — base class for the harness error taxonomy: a stable `code` string (distinct from the human `message`) plus `cause` chaining. Lives here, in the leaf package every other imports, so a single base is shared without a new dependency edge. Per-package errors (`LlmError`, `ToolArgsError`, `InvariantError`, …) extend it. `isHarnessError(value)` narrows at seams. -- `LlmError` — extends `HarnessError`; `code` string (`NO_ADAPTER`, `DUPLICATE_ADAPTER`, and adapter codes like `AUTH`/`RATE_LIMIT`) plus an optional numeric `status` when the failure came from a non-2xx provider response. +- `LlmError` — extends `HarnessError`; its stable `code` string (`NO_ADAPTER`, `DUPLICATE_ADAPTER`, and adapter codes like `AUTH`/`RATE_LIMIT`) is the programmatic failure contract. ### Real adapters diff --git a/packages/llm/llm/src/assembler.ts b/packages/llm/llm/src/assembler.ts index 8780718c7a..a721721fb6 100644 --- a/packages/llm/llm/src/assembler.ts +++ b/packages/llm/llm/src/assembler.ts @@ -39,12 +39,10 @@ export class BlockAssembler { private _replayState: unknown = undefined /** - * Feed one chunk. Returns the completed block when the chunk closes one - * (an explicit `block-end`), otherwise undefined. + * Feed one chunk into the assembly state. * @param chunk - the next raw chunk, in stream order. - * @returns the authoritative block from the first `block-end` at its index; undefined for every other chunk. */ - push(chunk: StreamChunk): ContentBlock | undefined { + push(chunk: StreamChunk): void { switch (chunk.type) { case 'block-start': { if (!this.partials.has(chunk.index)) { @@ -78,7 +76,7 @@ export class BlockAssembler { // and the final assembled block in agreement. if (partial.block) return partial.block = chunk.block - return chunk.block + return } case 'usage': { this._usage = chunk.usage diff --git a/packages/llm/llm/src/index.ts b/packages/llm/llm/src/index.ts index 26e9e3b289..020012a1da 100644 --- a/packages/llm/llm/src/index.ts +++ b/packages/llm/llm/src/index.ts @@ -43,12 +43,10 @@ declare module 'cordis' { /** * Typed error for LLM-related failures. Extends {@link HarnessError}, so the - * `code` string (e.g. `AUTH`, `RATE_LIMIT`, `NO_ADAPTER`) is shared taxonomy; - * `status` carries the HTTP status when the error originated from a non-2xx - * provider response (absent for protocol/usage errors that have no HTTP status). + * `code` string (e.g. `AUTH`, `RATE_LIMIT`, `NO_ADAPTER`) is shared taxonomy. */ export class LlmError extends HarnessError { - constructor(message: string, code: string, public status?: number, options?: ErrorOptions) { + constructor(message: string, code: string, options?: ErrorOptions) { super(message, code, options) this.name = 'LlmError' } diff --git a/packages/llm/llm/tests/assembler.spec.ts b/packages/llm/llm/tests/assembler.spec.ts index 5612e93cb4..f7a5028d14 100644 --- a/packages/llm/llm/tests/assembler.spec.ts +++ b/packages/llm/llm/tests/assembler.spec.ts @@ -29,12 +29,12 @@ describe('BlockAssembler', () => { expect(assembler.message().role).toBe('assistant') }) - it('returns the completed block from push() on block-end', () => { + it('records the completed block from block-end', () => { const assembler = new BlockAssembler() - expect(assembler.push({ type: 'block-start', index: 0, blockType: 'text' })).toBeUndefined() - expect(assembler.push({ type: 'text-delta', index: 0, text: 'hi' })).toBeUndefined() - const block = assembler.push({ type: 'block-end', index: 0, block: { type: 'text', text: 'hi' } }) - expect(block).toEqual({ type: 'text', text: 'hi' }) + assembler.push({ type: 'block-start', index: 0, blockType: 'text' }) + assembler.push({ type: 'text-delta', index: 0, text: 'hi' }) + assembler.push({ type: 'block-end', index: 0, block: { type: 'text', text: 'hi' } }) + expect(assembler.blocks()).toEqual([{ type: 'text', text: 'hi' }]) }) it('tolerates deltas without explicit block-start/end', () => { @@ -57,8 +57,8 @@ describe('BlockAssembler', () => { // push a delta first to guarantee the partial exists assembler.push({ type: 'text-delta', index: 0, text: 'hi' }) // block-end's ensure() must find the existing partial (the second branch path) - const block = assembler.push({ type: 'block-end', index: 0, block: { type: 'text', text: 'hi' } }) - expect(block).toEqual({ type: 'text', text: 'hi' }) + assembler.push({ type: 'block-end', index: 0, block: { type: 'text', text: 'hi' } }) + expect(assembler.blocks()).toEqual([{ type: 'text', text: 'hi' }]) }) it('throws from assemble() when a partial has an unhandled blockType', () => { @@ -128,7 +128,7 @@ describe('assertNever', () => { it('BlockAssembler.push rejects chunks outside the closed StreamChunk union', () => { const assembler = new BlockAssembler() - expect(() => assembler.push({ type: 'rogue-chunk' } as unknown as StreamChunk)) + expect(() => { assembler.push({ type: 'rogue-chunk' } as unknown as StreamChunk) }) .toThrow('unreachable variant in BlockAssembler.push') }) }) @@ -140,26 +140,8 @@ describe('BlockAssembler duplicate-close contract', () => { { type: 'block-end', index: 0, block: { type: 'reasoning', text: 'first' } }, { type: 'block-end', index: 0, block: { type: 'text', text: 'second' } }, ] - const streaming = new BlockAssembler() - const closed = [] - for (const chunk of chunks) { - const block = streaming.push(chunk) - if (block) closed.push(block) - } - - const oneShot = new BlockAssembler() - for (const chunk of chunks) oneShot.push(chunk) - - expect(closed).toEqual([{ type: 'reasoning', text: 'first' }]) - expect(oneShot.blocks()).toEqual([{ type: 'reasoning', text: 'first' }]) - expect(closed).toEqual(oneShot.blocks()) - }) - - it('push returns undefined for a duplicate block-end (it closed nothing)', () => { - const a = new BlockAssembler() - expect(a.push({ type: 'block-end', index: 0, block: { type: 'text', text: 'x' } })) - .toEqual({ type: 'text', text: 'x' }) - expect(a.push({ type: 'block-end', index: 0, block: { type: 'text', text: 'y' } })) - .toBeUndefined() + const assembler = new BlockAssembler() + for (const chunk of chunks) assembler.push(chunk) + expect(assembler.blocks()).toEqual([{ type: 'reasoning', text: 'first' }]) }) }) diff --git a/packages/llm/llm/tests/service.spec.ts b/packages/llm/llm/tests/service.spec.ts index 83429ba0cd..c9a4d2936b 100644 --- a/packages/llm/llm/tests/service.spec.ts +++ b/packages/llm/llm/tests/service.spec.ts @@ -255,11 +255,12 @@ describe('LlmService', () => { it('LlmError extends the shared HarnessError base', async () => { const { HarnessError, isHarnessError } = await import('@deepseek-ai/dsh-llm') - const err = new LlmError('boom', 'AUTH', 401) + const cause = new Error('root cause') + const err = new LlmError('boom', 'AUTH', { cause }) expect(err).toBeInstanceOf(HarnessError) expect(isHarnessError(err)).toBe(true) expect(err.code).toBe('AUTH') - expect(err.status).toBe(401) + expect(err.cause).toBe(cause) }) it('HarnessError carries a code, names itself by subclass, and chains cause', async () => { diff --git a/packages/support/llm-replay/src/index.ts b/packages/support/llm-replay/src/index.ts index 51e799114f..3299905a07 100644 --- a/packages/support/llm-replay/src/index.ts +++ b/packages/support/llm-replay/src/index.ts @@ -20,7 +20,7 @@ import { LlmAdapter, LlmError, assertNever } from '@deepseek-ai/dsh-llm' */ export type ReplayEntry = | { kind: 'chunks'; chunks: StreamChunk[] } - | { kind: 'throw'; chunks: StreamChunk[]; message: string; code: string; status?: number } + | { kind: 'throw'; chunks: StreamChunk[]; message: string; code: string } | { kind: 'hang' } /** One model exposed by a replay-only provider catalog. */ @@ -283,7 +283,7 @@ async function* replayEntry(entry: ReplayEntry, signal: AbortSignal | undefined) if (signal?.aborted) throw new Error('aborted') yield chunk } - throw new LlmError(entry.message, entry.code, entry.status) + throw new LlmError(entry.message, entry.code) case 'hang': // Replay a stream that stalls until cancelled (mirrors MockAdapter): one // chunk, then wait for abort and surface it as the consumer expects. diff --git a/packages/support/llm-replay/tests/llm-replay.spec.ts b/packages/support/llm-replay/tests/llm-replay.spec.ts index c9bd545b36..59ffc485db 100644 --- a/packages/support/llm-replay/tests/llm-replay.spec.ts +++ b/packages/support/llm-replay/tests/llm-replay.spec.ts @@ -175,7 +175,7 @@ describe('loadReplayScript', () => { it('uses the sidecar override when present, ignoring the JSONL', () => { writeFileSync(file, sessionJsonl([]), 'utf8') const overrideFile = join(dir, 'replay.override.json') - const override: ReplayEntry[] = [{ kind: 'throw', chunks: [], message: '401', code: 'AUTH', status: 401 }] + const override: ReplayEntry[] = [{ kind: 'throw', chunks: [], message: '401', code: 'AUTH' }] writeFileSync(overrideFile, JSON.stringify(override), 'utf8') expect(loadReplayScript({ file, overrideFile })).toEqual(override) }) @@ -265,12 +265,12 @@ describe('installLlmReplay (through the real LlmService)', () => { expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).toEqual(second) }) - it('replays a sidecar throw-entry as an LlmError with code/status, after its prefix chunks', async () => { + it('replays a sidecar throw-entry as an LlmError with its stable code, after its prefix chunks', async () => { writeFileSync(file, sessionJsonl([]), 'utf8') const overrideFile = join(dir, 'replay.override.json') const partial: StreamChunk[] = [{ type: 'block-start', index: 0, blockType: 'text' }] writeFileSync(overrideFile, JSON.stringify([ - { kind: 'throw', chunks: partial, message: 'unauthorized', code: 'AUTH', status: 401 }, + { kind: 'throw', chunks: partial, message: 'unauthorized', code: 'AUTH' }, ]), 'utf8') const ctx = new Context() await ctx.plugin(LlmService) @@ -279,7 +279,7 @@ describe('installLlmReplay (through the real LlmService)', () => { const seen: StreamChunk[] = [] await expect((async () => { for await (const c of ctx.llm.stream({ provider: 'm', model: 'm', messages: [] })) seen.push(c) - })()).rejects.toMatchObject({ message: 'unauthorized', code: 'AUTH', status: 401 }) + })()).rejects.toMatchObject({ message: 'unauthorized', code: 'AUTH' }) expect(seen).toEqual(partial) }) @@ -384,7 +384,7 @@ describe('installLlmReplay (through the real LlmService)', () => { const overrideFile = join(dir, 'replay.override.json') const partial: StreamChunk[] = [{ type: 'block-start', index: 0, blockType: 'text' }] writeFileSync(overrideFile, JSON.stringify([ - { kind: 'throw', chunks: partial, message: 'unauthorized', code: 'AUTH', status: 401 }, + { kind: 'throw', chunks: partial, message: 'unauthorized', code: 'AUTH' }, ]), 'utf8') const ctx = new Context() await ctx.plugin(LlmService) diff --git a/website/zh-CN/api/harness/llm.md b/website/zh-CN/api/harness/llm.md index 200e926d15..a70d5bd478 100644 --- a/website/zh-CN/api/harness/llm.md +++ b/website/zh-CN/api/harness/llm.md @@ -6,7 +6,7 @@ The abstract `llm` service: an adapter registry plus a streaming model-call surface, interceptable via the `llm/stream` waterfall. -[Source](https://github.com/deepseek-harness/deepseek-harness/blob/master/packages/llm/llm/src/index.ts#L96) +[Source](https://github.com/deepseek-harness/deepseek-harness/blob/master/packages/llm/llm/src/index.ts#L94) ### ctx.llm.registerAdapter(providers, adapter) @@ -21,7 +21,7 @@ Register an adapter for the given provider routes. Throws `LlmError` with code ` **Returns** the disposer that unregisters all of them. -[Source](https://github.com/deepseek-harness/deepseek-harness/blob/master/packages/llm/llm/src/index.ts#L111) +[Source](https://github.com/deepseek-harness/deepseek-harness/blob/master/packages/llm/llm/src/index.ts#L109) ### ctx.llm.listProviders() @@ -33,7 +33,7 @@ Describe provider routes with a registered adapter. **Returns** detached provider metadata in registration order. -[Source](https://github.com/deepseek-harness/deepseek-harness/blob/master/packages/llm/llm/src/index.ts#L142) +[Source](https://github.com/deepseek-harness/deepseek-harness/blob/master/packages/llm/llm/src/index.ts#L140) ### ctx.llm.listModels(provider) @@ -47,7 +47,7 @@ Discover models advertised by one registered provider. Catalog membership is adv **Returns** detached model metadata in adapter-preferred order. -[Source](https://github.com/deepseek-harness/deepseek-harness/blob/master/packages/llm/llm/src/index.ts#L152) +[Source](https://github.com/deepseek-harness/deepseek-harness/blob/master/packages/llm/llm/src/index.ts#L150) ### ctx.llm.stream(options) @@ -61,4 +61,4 @@ Stream one model call as raw chunks (token-level deltas). Throws `LlmError` with **Returns** the chunk stream, possibly wrapped by `llm/stream` listeners. -[Source](https://github.com/deepseek-harness/deepseek-harness/blob/master/packages/llm/llm/src/index.ts#L210) +[Source](https://github.com/deepseek-harness/deepseek-harness/blob/master/packages/llm/llm/src/index.ts#L208)