mirror of https://github.com/deepseek-ai/deepseek-harness synced 2026-08-15 21:04:50 +00:00

Files

Tianyi Cui 825b57aff9 feat(llm): structured error taxonomy with a shared HarnessError base (RFC 005 pt 2)

Introduce HarnessError in dsh-llm (the leaf package): a stable machine-routable
code distinct from the message, cause chaining, name from the subclass, plus
isHarnessError. LlmError, ToolArgsError, and InvariantError now extend it.

Tool failures carry the structure end-to-end: ToolExecutionResult gains
error: { name, code } (populated from a thrown HarnessError), and the loop
forwards it onto the tool/result session event (which gained the same optional
field) for retry/sandbox plugins and replay. The loop's toError wraps non-Error
throws in a HarnessError(code: UNKNOWN, cause) instead of a bare Error.

Landed last and in isolation so it's a pure upgrade over the plain Error+code
the earlier PRs used — independently revertible. Graduates RFC 005 pt 2 ->
ADR 0015; RFC 005 now fully implemented.

2026-06-14 01:07:28 +08:00

2.7 KiB

Raw Blame History

RFC 005: Runtime validation at the model boundary, error taxonomy, dev-mode invariants

Status: implemented — part 1 (arg validation) → ADR 0011; part 3 (dev invariants) → ADR 0012; part 2 (error taxonomy) → ADR 0015

Problem

Three gaps where compile-time guarantees stop:

Tool args are model-generated JSON — defineTool's InferArgs<S> claim is only as true as the model's output. Today a malformed call reaches execute untyped-in-practice.
Tool errors flatten to a text block; name/code/stack are lost, so future sandbox/retry plugins can't distinguish ENOENT from EACCES, and the model gets less actionable feedback than it could.
Loop ordering invariants (seq monotonicity, step/turn event nesting, turn-number continuity) are asserted only where tests look.

Proposal

Schema validation in defineTool: before execute, validate parsed args against the SchemaSpec (the converter already encodes the structure — a small interpreter walks it: presence of required keys, primitive type checks, enum membership, recursion into objects/arrays). On mismatch, return an isError ToolExecutionResult describing the violation — the model can self-correct. Raw-registered tools (MCP) keep validating their own input.
Structured error taxonomy: per-package error classes extending a common HarnessError (name, code, cause chaining). ToolExecutionResult gains optional error: { name, code } alongside the model-facing text. The loop's errorData consumes it; session error events carry the code. This also properly fixes the non-Error-throw message degradation found in review.
Dev-mode invariants: a dsh-invariants debug plugin (everything is a plugin — it's just listeners) asserting, when enabled: session seq strictly increases; step/start precedes its chunks; turn/start/turn/end pair and nest; tool/call has a matching tool/result; status transitions are legal. Enabled in tests and the demo; off in production. Doubles as executable documentation of the event contract. (As implemented, the tool rule is one-directional — a tool/result requires a prior tool/call, but NOT the converse: a throwing tools/execute waterfall ends a step with no result. See ADR 0012.)

Plan

2 first (taxonomy is a dependency of 1's error shape), then 1, then 3. Property tests (RFC 001) then close the loop: generated args ↔ validator ↔ InferArgs agreement.

Risks

Validator/InferArgs drift — covered by the RFC 001 composition property. Validation cost per call is negligible next to a model call.

2.7 KiB Raw Blame History