mirror of
https://github.com/deepseek-ai/deepseek-harness
synced 2026-08-15 21:04:50 +00:00
Merge remote-tracking branch 'origin/master' into codex/skill-system
# Conflicts: # docs/architecture.md # docs/config-catalog.md # docs/module-graph.md # docs/rfc/INDEX.md # examples/acp-agent/tests/snapshots/text-turn/session.jsonl # packages/core/agent-core/src/index.ts # packages/core/tools/tests/gen-tool-catalog.spec.ts # packages/support/acp-snapshot/src/suite.ts # packages/ui/acp-agent/src/index.ts
This commit is contained in:
137
docs/rfc/implemented/feature/2026-06-15-code-mode.md
Normal file
137
docs/rfc/implemented/feature/2026-06-15-code-mode.md
Normal file
@@ -0,0 +1,137 @@
|
||||
# RFC: Code Mode — the model writes TypeScript against the tool registry
|
||||
|
||||
Status: implemented
|
||||
|
||||
## Problem
|
||||
|
||||
Today the agent loop advertises every registered tool to the model as a native JSON-schema function definition. `ToolRegistry` contributes its schemas to the system-prompt assembly, the assembly's `tools` land on the wire (and in the logged request header), the model invokes one `tool-call` block per step, and the loop dispatches each call through `ctx.tools.execute()` **sequentially** (parallel tool execution is an explicit open TODO in `dsh-tools` and [docs/architecture.md](../../../architecture.md)), with **every** intermediate `tool-result` re-entering the model's context on the next request.
|
||||
|
||||
For multi-step tool work this is token-heavy and serial. The model cannot compose tools — loop over a result set, branch on an intermediate value, fan out, post-process — without a full model round-trip per call, and each round-trip drags the entire intermediate result back into context whether the model needs it or not.
|
||||
|
||||
Cloudflare's [Code Mode](https://blog.cloudflare.com/code-mode/) proposes an alternative grounded in a simple observation: LLMs are better at writing code than at emitting tool calls, because they have seen millions of lines of real code and comparatively few contrived tool-calling traces. Instead of one tool call per step, the model writes a TypeScript program against a generated API over the tools, the program executes in a sandboxed runtime, and the model curates what comes back — only what it prints or returns — instead of every intermediate result.
|
||||
|
||||
An earlier draft of this RFC designed Code Mode as an add-on consumer plugin with zero core changes, deferring the execution substrate to a follow-up. Both constraints are dropped here, deliberately. First, the harness is pre-release and optimizes for the correct foundation over blast radius: tool presentation is the registry's own concern, and bolting a second presentation onto it from outside means transforming the registry's contribution after the fact — a waterfall listener whose correctness depends on listener ordering, which fights the [reconstructable-requests](../../implemented/architecture/2026-07-05-reconstructable-requests.md) design instead of riding it (that refactor removed request mutation from `agent/request`, the seam the old draft relied on). Second, the substrate question is answerable now: a Node `worker_threads` runtime gives real containment — separate isolate, empty environment, heap caps, and a `terminate()` that reliably stops a hot synchronous loop — where the old draft's `node:vm` stub had none of those, and it fits the harness's existing trust model (§Trust posture) without a hardening follow-up.
|
||||
|
||||
## Decision
|
||||
|
||||
Three decisions, each elaborated in its own section below:
|
||||
|
||||
1. **Code Mode is a first-class presentation mode of `ToolRegistry`** (`dsh-tools`), selected by a validated `mode` config: `'native'` (today's behavior, the default), `'code'` (the wire carries exactly one tool, `run_code`, plus a generated SDK `.d.ts` in the system prompt), or `'both'` (native schemas *and* `run_code` + SDK). The registry's existing tool-schema provider contributes whatever the mode dictates, so the wire tool list is shaped at its source — no interception, no listener-ordering caveats — and the logged request header records it for free.
|
||||
2. **Code execution is a capability seam** — a new group `packages/code-runtime/` with the interface package `@deepseek-ai/dsh-code-runtime` owning `ctx.codeRuntime` ([capability seams](../../implemented/architecture/2026-06-13-capability-seams.md); consumer = `dsh-tools`, with core-consumes-a-seam precedent in `agent-loop` → `dsh-llm`). The runtime knows nothing about tools: it is handed a program and named async bindings, runs the program, and reports `{ value, logs, error? }`. Language and substrate are backend properties, so a future Python or container backend is a new implementation package, not a redesign.
|
||||
3. **The shipped implementation is `@deepseek-ai/dsh-code-runtime-worker`**: one fresh Node worker thread per run, executing the model's TypeScript after type-strip, with bindings bridged over the message port, an empty environment, configurable heap/output/time caps, and hard termination. Its trust posture is bash-equivalent by design — no unsafe-acknowledgement flags — because the harness already ships `dsh-bash-local`, which executes arbitrary model-written shell commands with strictly *more* ambient authority.
|
||||
|
||||
### The registry owns the mode
|
||||
|
||||
`ToolRegistry` gains a schemastery-validated config (`static Config`), its first: `mode: 'native' | 'code' | 'both'`, default `'native'`. A deployment flips it from `cordis.yml` (`tools: { mode: code }`) — no code edit, per the no-hardcoded-tunables convention.
|
||||
|
||||
**Wire tool list = the registry's contribution.** The registry already feeds the assembly through `ctx.systemPrompt.tools(() => this.schemas())`; the provider becomes mode-aware: `'native'` contributes all schemas (unchanged), `'code'` contributes only `run_code`'s schema, `'both'` contributes all schemas plus `run_code`. Because [`PromptAssembly.tools` is the single source the loop's request header snapshots](../../../../packages/core/system-prompt/src/index.ts), the collapse is automatically logged and reconstructable — model-visible ⟺ logged holds with zero new mechanism. Scope of the guarantee, stated honestly: the mode governs the **registry's** contribution, and the registry is the only shipped schema source — but `systemPrompt.tools()` is a public multi-provider API and the `system-prompt/assemble` waterfall may transform the assembly, so a deployment that wires a second direct provider (or a mutating listener) owns what it adds, exactly as in native mode. Those are deliberate acts; what the design eliminates is the *accidental* leak the old draft worried about — a listener-ordering race around an after-the-fact collapse — and the shipped-configuration invariant (`'code'` ⇒ assembled tools exactly `[run_code]`) is pinned by tests and, like every request, by the logged header.
|
||||
|
||||
**Interaction with `toolOrder`, stated up front:** a configured `systemPrompt.toolOrder` naming native tools rejects every assembly under `mode: 'code'` (those names are no longer contributed), by the existing fail-loud rule for unlisted names. This is correct behavior, not a bug: a deployment switching modes updates its order config or drops it.
|
||||
|
||||
**The SDK prompt section.** Under `'code'` and `'both'` the registry registers one lazy prompt section (`tools:sdk`, in the 100–199 tool-guidance order band) whose thunk regenerates, at each assembly, a TypeScript declaration of every registered tool except `run_code` itself, plus fixed usage instructions. The thunk reads the live store and emits tools in lexicographic name order, so its output is deterministic and stable across steps — an unchanged tool set produces byte-identical text (prefix-cache-friendly; a mid-session registration surfaces as one logged header delta, exactly like a native-mode tool change).
|
||||
|
||||
**Codegen.** A pure `jsonSchemaToTs(schema)` module inside `dsh-tools` (sibling of `json-schema.ts` — `schemas()` and the SDK are two projections of the same store) maps the JSON-Schema subset the `defineTool` DSL emits (object/string/number/boolean/array, `properties`, `required`, string `enum` → literal union, nested objects, array `items`, `description` → JSDoc) to a TS type literal. It is **total**: any construct outside that subset (`$ref`, `oneOf`/`anyOf`, `integer`, future MCP shapes, …) degrades to `unknown` without throwing. Because `ToolSchema.name` is an arbitrary string, the SDK is declared as one object constant — `declare const tools: { "some-mcp-tool"(args: …): Promise<string>; bash(args: …): Promise<string>; … }` — quoted keys make every name reachable with no sanitization or alias-collision logic. Typing is advisory (the runtime executes type-stripped JS); the instructions say so.
|
||||
|
||||
### The run_code tool and the dispatch bridge
|
||||
|
||||
Under `'code'` and `'both'` the registry registers `run_code` in itself as an ordinary tool — one required parameter `{ code: string }` — so the unchanged loop dispatches it through the normal pipeline and `tools/pre-execute` / `tools/post-execute` gate it like any other call (a permission plugin can inspect the program text before it runs). Its `execute(args, exec)`:
|
||||
|
||||
1. **Builds the bindings**: the bridge owns a **run-scoped `AbortController`** whose signal follows `exec.signal` (an outer cancel propagates in) and which the bridge itself fires the moment the run settles for any reason — completion, program exception, `computeMs`/`maxWallMs` expiry, worker exit. For every registered tool except `run_code`, the binding is an async function that (a) checks the run signal before and after (throwing stops the program — necessary because `ctx.tools.execute()` converts errors to `isError` data), (b) **JSON-normalizes the argument** — a `JSON.parse(JSON.stringify(args))` round-trip, rejecting that one call with a descriptive `Error` when the value does not survive (`BigInt`, circular structures) — because the seam's structured-clone boundary is wider than JSON while the session log accepts only JSON: normalizing BEFORE dispatch makes the dispatched form and the logged form the same JSON value by construction, so an executed sub-call can never fail at logging time, (c) awaits its turn on the **per-run serialization queue** (below), (d) calls `this.execute({ callId, name, arguments, agent: exec.agent, signal: runSignal })` with a deterministic sub-id `` CallId(`${exec.callId}:code:${n}`) `` — the run signal, not the bare outer one, so a budget expiry aborts an in-flight sub-tool (`bash-local` kills on its spec signal) instead of orphaning it, (e) appends a `tool/code-dispatch` session event, and (f) maps the result: success → the text-block contents joined as a `string` (non-text blocks become placeholders, an MVP limitation), `isError` → **the binding rejects** with an `Error` carrying the result text. Rejection is the deliberate model-facing contract — real code signals failure by throwing, `try/catch` and `Promise.all` short-circuiting behave as every model has seen them behave — where the old draft's `{ output, isError }` envelope made error handling a bespoke convention.
|
||||
2. **Runs the program**: `ctx.codeRuntime.run({ program: args.code, bindings: [{ global: 'tools', functions }], signal: exec.signal })`.
|
||||
3. **Surfaces the outcome — after reaching quiescence.** When `ctx.codeRuntime.run()` resolves, the bridge fires the run-scoped abort (cancelling any in-flight sub-dispatch and abandoning queued-unstarted ones), then **awaits the dispatch queue's drain before returning**, per the dispose-to-quiescence rule in [defensive patterns](../../../defensive-patterns.md): an aborted in-flight sub-call still settles and logs its `isError` `tool/code-dispatch` event *inside* the open turn, and nothing can append after `run_code` returns. A successful run then returns one text block — the captured console/stdout output followed by the rendered return value (if any) — plus a `meta` payload (capped logs, dispatch count) for presentation. A run with `result.error` throws a `CodeRunFailedError extends HarnessError` (`code: 'CODE_RUN_FAILED'`, message = the error kind and text plus captured logs so the model can self-correct); the registry's existing catch turns it into a structured `isError` result.
|
||||
|
||||
**Sub-call `additionalContext` is suppressed, deliberately.** A `tools/post-execute` hook may attach `additionalContext` to a call; for loop-dispatched calls the loop buffers those and appends each as a `context/message` only after the step's `tool/result`s, preserving call/result adjacency. A sub-dispatch result's `additionalContext` has no such safe outlet from inside a running `run_code`: injecting immediately would land a `context/message` between the parent's `tool/call` and its `tool/result` (breaking the adjacency the buffering exists to protect), and `PostToolDecision.additionalContext` is singular where a program may produce many. The MVP therefore drops sub-call `additionalContext`, pinned by a test and stated in the hooks bridge's docs; the follow-up (a plural context channel or loop-level sub-dispatch buffering) is deferred until a real hook needs it through Code Mode.
|
||||
|
||||
**Concurrency: serialized, enforced by the binding.** The bindings are async, so a model writing `Promise.all([tools.a(…), tools.b(…)])` starts both immediately — concurrent dispatch would be the *default*, while the tool contract still carries no concurrency-safety metadata (the open parallel-execution TODO). Each `run_code` invocation therefore owns a dispatch queue and every binding call chains onto it, so even `Promise.all` executes the underlying `ctx.tools.execute()` calls one at a time in submission order; when the run settles, queued-but-unstarted dispatches are abandoned. Lifting this per-tool once tools can declare themselves concurrency-safe is deferred work, same as before.
|
||||
|
||||
**Presentation.** `run_code`'s render intent is decided here per the [render-intent RFC](../../implemented/architecture/2026-07-02-tool-render-intent-union.md): `presentCall` → a `generic` card, `kind: 'execute'`, title `Run code`, `rawInput` = the program text; `presentResult` → a `generic` card whose content is the captured output (from `meta`). Not a `terminal` card: that card's semantics are "a shell command in a working directory", which a program is not.
|
||||
|
||||
### Observability: `tool/code-dispatch`
|
||||
|
||||
Each sub-dispatch appends one session event, declared by `dsh-tools` via `SessionEventMap` declaration merging (the map is merge-extensible for exactly this; `todo/write` is the log-only precedent): `tool/code-dispatch` with `{ parentCallId, subCallId, name, arguments, isError, resultSummary }` — `arguments` being the bridge's JSON-normalized value, the very one dispatched, so the append cannot fail on payload shape. It is log-only — `deriveEventMessage()` ignores unknown event types by design, so sub-calls never re-enter model context — but persistence and UIs get every call. As a log event it carries JSDoc prose but **no `@mode` tag** (that vocabulary belongs to cordis bus events; the persistence-catalog generator hard-errors on one) and lands in the regenerated `docs/persistence-catalog.md`; appends happen inside `run_code`'s execution, so the turn-enclosure invariant is satisfied by construction. A `run_code` execution arriving without `exec.agent` (the loop always supplies it; direct programmatic calls may not) still runs and simply skips event logging, exactly as the `ToolExecution` contract allows.
|
||||
|
||||
### The code-runtime seam
|
||||
|
||||
`packages/code-runtime/code-runtime/` — `@deepseek-ai/dsh-code-runtime`, depending only on `cordis`. An abstract `CodeRuntime extends Service` (`super(ctx, 'codeRuntime')`) plus the vocabulary:
|
||||
|
||||
- `CodeRunRequest = { program: string; bindings: CodeBindingNamespace[]; signal?: AbortSignal }`
|
||||
- `CodeBindingNamespace = { global: string; functions: Record<string, (args: unknown) => Promise<unknown>> }` — the runtime exposes each namespace as a global object of async functions inside the program; binding arguments and resolutions must be structured-cloneable (a runtime may cross a serialization boundary; ours does).
|
||||
- `CodeRunResult = { value?: unknown; logs: CodeLogEntry[]; error?: CodeRunFailure }` — an error is a field on a resolved result, never a rejection of `run()`.
|
||||
- `CodeLogEntry = { source: 'console' | 'stdout' | 'stderr'; level?: 'log' | 'info' | 'warn' | 'error' | 'debug'; text: string }`
|
||||
- `CodeRunFailure = { kind: 'exception' | 'timeout' | 'abort' | 'worker-exit'; message: string }` — orthogonal outcomes reported independently per [defensive patterns](../../../defensive-patterns.md); a timed-out run is not an exception, an abort is not a timeout.
|
||||
- Two readonly backend descriptors, informational not gating: `language` (what the program must be written in — `'typescript'` for the shipped backend; a Python backend would say so, and pair with its own SDK generator on the presentation side) and `isolation` (`'worker-thread'` for the shipped backend; `'process'`, `'container'`, … for future ones). `dsh-tools` requires `language === 'typescript'` in the MVP — its codegen emits TS — and fails the assembly loudly otherwise, the same misconfiguration idiom as `toolOrder` violations (as when `mode` is non-native with no `ctx.codeRuntime` loaded at all).
|
||||
|
||||
Per explicit-over-implicit at seams, the request spells out everything the runtime acts on; defaulting (timeouts, caps) is the implementation's validated config, never a hidden `??` inside `run()`. Consumption uses the loop's established optional-backend idiom: cordis has no optional injection — every `inject` entry gates activation — so a static `inject` on the registry would hold `ctx.tools` (and every tool plugin behind it) hostage to a code runtime existing even under `mode: 'native'`; instead the registry reads `ctx.get('codeRuntime')` at use time, exactly as `agent-loop` consumes `sessionPersistence`, with absence failing loud in the provider thunk as above. The seam split is justified by real planned divergence on both axes — substrate (worker now; container/microVM later) and language (the Python/AssemblyScript direction sketched in the earlier draft survives as future work) — not by speculation: `dsh-tools` consumes the interface today and tests against a trivial in-repo fake, exactly the interface/implementation/consumer shape of the bash template.
|
||||
|
||||
### The worker-thread runtime
|
||||
|
||||
`@deepseek-ai/dsh-code-runtime-worker`, the second package of the `packages/code-runtime/` group. Per `run()`:
|
||||
|
||||
1. **Type-strip host-side** with Node's built-in `stripTypeScriptTypes` (`node:module`; present across the repo's whole engines range, `^22.19.0 || >=24.0.0`, and position-preserving, so runtime error line numbers match the model's source). Strip-only mode rejects non-erasable syntax (`enum`, namespaces) — that rejection returns as `error.kind: 'exception'` with Node's message, the SDK instructions say "erasable TypeScript only", and the model self-corrects like any other program error. A syntax-level failure never spawns a worker.
|
||||
2. **Spawn one fresh `Worker` per run** from the package's own bootstrap module: `env: {}` (truly empty — stronger than the scrubbed-env rule for spawned commands), `resourceLimits` from config, `stdout`/`stderr` captured into `logs` rather than inherited. No pooling and no cross-run state: a program's world dies with its worker, which keeps runs reconstructable from the log alone and makes state bleed unrepresentable.
|
||||
3. **Execute** in the bootstrap: the stripped program becomes the body of an `AsyncFunction` whose parameters are the binding globals and a capturing `console` shim, so top-level `await` and `return` work and the program's completion value is the run's `value` (structured-cloneable values cross as-is; anything else is replaced by its `util.inspect` rendering, documented).
|
||||
4. **Bridge bindings over the message port**: each binding function in the worker posts `{ id, global, name, args }` and awaits the reply; the host validates the name against the request's bindings, invokes, and replies `{ id, ok, value }` or `{ id, ok: false, message }` (a host-side binding rejection becomes a program-side rejection). The worker-side namespace objects are built null-prototype via `defineProperty`, so a binding named `__proto__`, `constructor`, or `toString` is an ordinary own property, not a prototype collision. Unknown names, duplicate ids, and post-settlement messages are rejected or ignored — the port protocol assumes a hostile peer, because the peer runs model code.
|
||||
5. **Enforce caps — two independent budgets, because the peer is hostile.** The compute budget (`computeMs`) meters the worker's *measured busy time* via `worker.performance.eventLoopUtilization()` polling — not host-side "is an RPC pending" bookkeeping, which a program defeats by firing an un-awaited call at a slow tool and then spinning hot while the host thinks it is waiting. Measured busy time cannot be gamed: a hot loop accrues it whether or not a dispatch is in flight, and a program genuinely awaiting a slow tool accrues none, so a long-running `bash` sub-call still does not kill an innocent run. The wall ceiling (`maxWallMs`) never pauses for anything and backstops what busy-time cannot see (a program awaiting a promise nobody will resolve). Budget expiry, `signal` abort, and run completion all funnel into `worker.terminate()`, which ends hot synchronous loops too (measured; this was `node:vm`'s unfixable gap); the failure reports which budget fired. Heap overflow surfaces as the worker's OOM exit → `error.kind: 'worker-exit'`. Log and value sizes are capped by config, truncation marked in-band. All caps are validated config fields with defaults (`computeMs: 60_000`, `maxWallMs: 600_000`, `maxLogBytes: 65_536`, `maxValueBytes: 32_768`, `maxOldGenerationSizeMb: 512`), changeable from `cordis.yml`.
|
||||
6. **Dispose to quiescence**: the service's own disposal terminates in-flight workers and *awaits* their exits before resolving, per [defensive patterns](../../../defensive-patterns.md).
|
||||
|
||||
### Trust posture
|
||||
|
||||
The worker runtime is **containment, not a security boundary**, and the RFC says so without ceremony. Model code in the worker can reach Node globals — `fetch`, `process` (with an empty env), dynamic `import()` of built-ins — so a deliberately adversarial program has ambient authority comparable to what the harness's own `bash` tool already grants every model turn: `dsh-bash-local` runs arbitrary model-written commands with the host filesystem, network, and a scrubbed-but-populated environment. One asymmetry runs the other way and is stated plainly: `worker.terminate()` ends the thread, not OS processes a program may have spawned via `node:child_process` — weaker than `bash-local`'s process-group kill for direct children (equivalent for double-forked daemons, which survive both); the wall-clock ceiling bounds the worker itself, and orphan cleanup is the same deployment-level concern it already is for bash. Code Mode is gated where bash is gated — `tools/pre-execute`, where permission/sandbox plugins veto or approve the program before it runs — and adds containment bash does not have: empty env, heap caps, hard termination of the program itself, a separate isolate. The earlier draft's two-flag unsafe ceremony (`{ unsafe: true }` constructor + `allowUnsafeRuntime`) existed for a `node:vm` stub with *no* containment and is dropped with it; demanding scarier flags for the better-contained executor than for bash would be posture theater. A deployment that needs a hard boundary (untrusted multi-tenant input) needs it for bash too; that is a future `isolation: 'container'` backend, and the `isolation` descriptor exists so such a deployment can tell backends apart.
|
||||
|
||||
### What the model sees
|
||||
|
||||
The `tools:sdk` section carries the `.d.ts` plus fixed instructions: the program is the body of an async TypeScript function (erasable syntax only — no `enum`/namespaces; type annotations are advisory); call tools as `await tools.name(args)` (quoted access for exotic names); a failed tool call **rejects** with an `Error` carrying the tool's error text — catch it to handle and continue; calls run **sequentially** even under `Promise.all`; emit results via `return` and/or `console.log`, and only that curated output returns to the context — intermediate tool results never do. That last line is the payoff the whole design serves: output-side context cost becomes the model's own editorial decision. On the input side the `.d.ts` is not free — for a large tool surface it can rival the native JSON schemas it replaces (and `'both'` pays for the two side by side) — but it is prefix-stable, so provider prefix caching amortizes it; the win is workload-dependent and the RFC claims no more.
|
||||
|
||||
## Consequences
|
||||
|
||||
The design shipped as four stacked changes — this RFC, the `dsh-code-runtime` interface package, the `dsh-code-runtime-worker` backend, and the `dsh-tools` integration — each gates-green with docs in the same change; review fixes landed on the change that introduced them and merged down.
|
||||
|
||||
What exists now:
|
||||
|
||||
- **The seam**: `packages/code-runtime/` — `@deepseek-ai/dsh-code-runtime` (abstract `CodeRuntime`, the vocabulary above, `ctx.codeRuntime`) and `@deepseek-ai/dsh-code-runtime-worker` (the worker-thread backend, every cap a validated config field). Rows in the service map, capability-seams graph, config catalog, and cordis catalog.
|
||||
- **The registry surface**: `ToolRegistry`'s first config (`mode`), the mode-aware wire contribution, the `tools:sdk` section, `jsonSchemaToTs`/`renderToolsSdk` (exported), `run_code` + the dispatch bridge + `CodeRunFailedError`, and the `tool/code-dispatch` log event (declaration-merged into `SessionEventMap`, regenerated into the persistence catalog; `run_code` in the tool catalog).
|
||||
- **The composed surface**: the `tools` config forwards through `agent-core` and both app packages (`stdio-agent`, `acp-agent`); `demo:code-mode` boots each UI example's `code-mode.cordis.yml` overlay (the worker runtime + `mode: 'code'` over the base tree); the adding-a-tool cookbook states that a registered tool is reachable from programs for free, and the tool-pipeline doc shows sub-dispatches re-entering both waterfalls.
|
||||
- **Interactions inherited by deployments**: a `toolOrder` naming native tools rejects every assembly under `'code'` (update or drop the order config when switching modes); sub-call `additionalContext` is dropped by the bridge (a plural context channel is deferred until a real hook needs it through Code Mode); sub-dispatch stays serialized until tools can declare concurrency safety — the same metadata the native parallel-dispatch TODO waits on.
|
||||
|
||||
## Testing
|
||||
|
||||
What the suites pin, per tier:
|
||||
|
||||
- **Unit — worker runtime** (real workers, no mocks): output/value capture and log-source attribution; error kinds (exception incl. non-erasable syntax, abort, worker-exit under OOM); the two budgets from both sides (a hot loop behind an un-awaited pending dispatch dies at `computeMs` busy time; a program idling on a slow binding outlives `computeMs` and dies only at `maxWallMs`); binding-bridge hostility (junk/forged port traffic incl. non-object messages and forged `log`/`done` cap bypass attempts, unknown names, duplicate ids, post-settlement replies, `__proto__`/`constructor`/`toString` binding names); structured-clone fallback and cap truncation; `env` emptiness verified from inside a program; dispose-awaits-exit. A real-load-path e2e runs the BUILT package under plain `node` so the worker entry resolves both unbuilt (tsx) and built — the published-artifact guard from [docs/testing.md](../../../testing.md).
|
||||
- **Unit — registry integration**: the codegen table (DSL subset, quoted names, `unknown` degradation, byte-identical determinism); provider contribution per mode (`'native'` unchanged, `'code'` exactly `[run_code]`, `'both'` all + `run_code`); `toolOrder × mode` rejection; missing-runtime / wrong-language loud failures; serialization non-overlap (a probe tool records enter/exit under `Promise.all`); abort aborting the in-flight sub-dispatch and abandoning queued ones; binding rejection on `isError` and on JSON-unrepresentable arguments; `CodeRunFailedError` → structured `isError` carrying kind + logs; `tool/code-dispatch` payloads (JSON-normalized arguments identical to what dispatched); `deriveMessages()` ignoring the event; sub-call `additionalContext` suppression; HMR safety (disposing the registry removes the tool and the section).
|
||||
- **e2e (with-key, self-skips)**: a real model under `mode: 'code'` composes two bash calls in one program (`examples/coding-agent/tests/code-mode.e2e.ts`) — every logged `request/header` carries exactly `[run_code]`, the dispatch events land under the parent call, the file the program wrote exists, and the final answer is the curated output.
|
||||
- **Snapshot (keyless replay)**: goldens for a `run_code` turn under `'code'` and `'both'` (`code-mode-turn`, `both-mode-turn`), each its own header-pinning class — the SDK section text, the collapsed header tool list, the dispatch events, and the result card are committed and replayed.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
**An add-on consumer plugin, zero core changes (the previous draft of this RFC).** Rejected on both halves. The wire-collapse half aged out from under it: it targeted the `agent/request` waterfall, which [reconstructable requests](../../implemented/architecture/2026-07-05-reconstructable-requests.md) has since re-typed to call-config-only, and the surviving alternative — transforming the assembly a waterfall listener receives — is strictly worse than contributing the right list in the first place (transformation must undo `toolOrder` canonicalization it cannot see the config for, and its correctness depends on where it sits in a listener chain). The deeper reason is ownership: which tools the model is offered, in which representation, is the registry's single concern — `schemas()` for function calling and the SDK for Code Mode are two projections of one store, and splitting the second projection into a satellite package would preserve a boundary the domain does not have.
|
||||
|
||||
**`node:vm` as the reference runtime, hardening deferred (also the previous draft).** Rejected: `node:vm` is not isolation (prototype-chain escapes reach the host realm), cannot interrupt a hot loop, and forced the draft into a two-flag unsafe ceremony plus a mandatory follow-up RFC. The worker thread delivers the missing properties now — separate isolate, empty env, `resourceLimits`, reliable `terminate()` (all verified by probe before this revision) — at bash-equivalent trust, so the reference implementation and the production one are the same package and the ceremony dissolves.
|
||||
|
||||
**Result elision / summarization over native tool-calling.** Addresses only the context-bloat half of the problem: trimming old `tool-result`s (now cheap to add as a logged surface replace, per the reconstructable-requests consequences) still pays one model round-trip per call and cannot express loops, branches, or joins. Complementary, not competing; it can layer under Code Mode for residual native calls.
|
||||
|
||||
**Parallel native dispatch in the loop.** The other answer to round-trip cost; still valid future work (the open TODO), still blocked on concurrency-safety metadata, and still no composition — it parallelizes calls the model already decided on in one step. Code Mode's serialized-queue decision keeps the two compatible: when the metadata lands, both native parallel dispatch and per-tool binding parallelism unlock together.
|
||||
|
||||
**Always-exclusive (Cloudflare-faithful, no mode).** Rejected for this SDK's primary consumer: a coding agent's bread-and-butter single calls (`bash`, `read`, `edit`) are already ideal as native calls, and forcing every edit through a program taxes the common case. The mode config keeps the faithful form (`'code'`) one line away without imposing it.
|
||||
|
||||
**Per-tool visibility tiers (this tool native, that tool code-only).** Deferred again, knowingly: it needs per-tool metadata and a presentation split that `'native' | 'code' | 'both'` does not, and every learning it depends on (how models actually split usage under `'both'`) arrives only after this ships.
|
||||
|
||||
**Sanitized identifier aliases in the SDK** (`my-tool` → `my_tool`, Cloudflare's approach). Rejected: quoted keys on a `declare const` make every name reachable with zero alias-collision logic; models handle `tools["my-tool"](…)` fine.
|
||||
|
||||
**A REPL-style persistent kernel** (state survives across `run_code` calls). Rejected for the MVP: cross-call state would be invisible to the session log, breaking the reconstructability guarantee that every request is a pure function of the log; fresh-per-run keeps it. A kernel-style backend remains expressible behind the seam later, with its own logging story.
|
||||
|
||||
## Risks
|
||||
|
||||
**The worker is not a hard security boundary.** Deliberate and documented (§Trust posture): posture equals the existing bash tool, containment exceeds it, gating uses the same seams. Deployments needing more need a future `isolation: 'container'` backend — tracked as the seam's designed extension, not a TODO on this design.
|
||||
|
||||
**`stripTypeScriptTypes` is marked experimental.** It is the same engine (amaro/swc) behind Node's own native `.ts` execution, exposed as an API across this repo's whole engines range. Mitigations: the runtime's unit suite pins the behaviors relied on (position preservation, erasable-only rejection message shape loosely), the call sits behind one private function, and `amaro`/`sucrase` are drop-in replacements if the API shifts. The erasable-only subset is a model-facing contract line, and the error path is a working feedback loop, not a dead end.
|
||||
|
||||
**Prompt cost of the SDK, especially under `'both'`.** The `.d.ts` can rival the native schemas it complements; `'both'` carries two representations. Prefix stability + provider caching amortize per-session cost; the mode is per-deployment; the RFC makes no unconditional-savings claim. Measured guidance (when to prefer which mode) is explicitly post-ship learning.
|
||||
|
||||
**Registry scope growth.** `dsh-tools` absorbs codegen, a tool, a bridge, and an event. Contained by module boundaries inside the package (`ts-types.ts`, `code-mode.ts` beside `schema.ts`/`json-schema.ts`/`presentation.ts`) and by the seam: everything substrate-shaped lives behind `ctx.codeRuntime`.
|
||||
|
||||
**Structured-clone limits at the binding boundary.** The seam's clone boundary admits values JSON does not (`Date`, `Map`, `BigInt`), and the session log accepts only JSON — left unhandled, a sub-call could execute and then fail at `tool/code-dispatch` append time. Closed by the bridge's JSON-normalization step (§ the dispatch bridge): what does not survive the round-trip rejects that binding call before dispatch, so every executed sub-call is loggable by construction. The seam itself keeps the wider structured-clone contract (it is about the port, and stated so a future binding producer cannot discover it in production); consumers with stricter payload needs enforce them at their own boundary, as the bridge does. Non-text sub-result content is reduced to placeholders — a known MVP limitation, recorded in the SDK instructions.
|
||||
|
||||
**Serialized-only sub-dispatch.** `Promise.all` gains no wall-clock parallelism yet, only fewer round-trips; models may over-expect. The instructions state it; lifting it is tied to the same concurrency-safety metadata the native parallel-dispatch TODO needs.
|
||||
|
||||
**Budget metering reads the event loop, not a flag.** Busy-time polling (`eventLoopUtilization()`) is coarser than an exact CPU meter — a budget expires up to one poll interval late — and its correctness claim ("a pending dispatch cannot pause it") is load-bearing against a hostile program. Both sides are unit-tested (hot loop with a pending decoy dispatch dies at `computeMs`; idle-on-slow-binding survives to `maxWallMs`), and the poll interval is an internal constant, not config — nothing a deployment could mis-tune into a bypass.
|
||||
66
docs/rfc/implemented/feature/2026-07-05-dynamic-workflows.md
Normal file
66
docs/rfc/implemented/feature/2026-07-05-dynamic-workflows.md
Normal file
@@ -0,0 +1,66 @@
|
||||
# RFC: Dynamic workflows — a script-driven multi-agent orchestration seam
|
||||
|
||||
Status: implemented
|
||||
|
||||
## Problem
|
||||
|
||||
The harness can delegate ONE task to ONE child (`dsh-tool-subagent`), but work that fans out across many independent pieces — an audit over many files, a migration, multi-angle research, adversarial verification of findings — forces the model to orchestrate turn by turn: every intermediate result lands in the parent context, the plan lives nowhere durable, and coordination costs a model round-trip per step. Claude Code ships this capability as [dynamic workflows](https://code.claude.com/docs/en/workflows): the model writes a JavaScript orchestration script, a runtime executes it, and the script — not the conversation — holds the loop, the branching, and the intermediate results.
|
||||
|
||||
## Decision
|
||||
|
||||
A workflow capability family at `packages/workflow/` in the bash seam shape (interface / implementation / consumer), plus the structured-output foundation it needs on the subagent seam.
|
||||
|
||||
### The script contract (Claude Code-compatible)
|
||||
|
||||
A workflow call is two parts: a `meta` JSON parameter (the identity block — `name`, `description`, optional `whenToUse`/`phases`; the field vocabulary matches Claude Code's meta block) and a `script` — a plain-JS body with top-level `await`, ending in `return <json-value>`. Meta is DATA, never code: the engine shape-validates it and evaluates no script text to obtain it (a body still opening with a CC-style `export const meta` statement is rejected with a pointed message). The body sees exactly: `agent(prompt, {label, phase, schema, model})`, `parallel(thunks)`, `pipeline(items, ...stages)` (NO cross-stage barrier; `(prev, item, index)` callbacks), `phase(title)`, `log(message)`, and `args`. CC semantics are preserved where they matter to script authors: a failed child resolves `null` (scripts `.filter(Boolean)`); an ordinary stage throw nulls the ITEM and skips its remaining stages. CC's determinism bans (`Date.now()`/`Math.random()`/argless `new Date()` throwing) are NOT enforced — they exist for CC's journaling/resume, which this cut defers — so a CC-authored BODY runs unchanged (its meta header moves into the parameter) while scripts written here may freely read the clock.
|
||||
|
||||
One deliberate strictness DIVERGENCE from CC: hook misuse — unknown or deferred options (`effort`/`isolation`/`agentType`), malformed arguments, schemas outside the supported subset, tripped caps, seam start failures — throws a `WorkflowError` with `fatal: true`, and the combinators RE-THROW fatal errors instead of nulling the item. Without this, a typo'd option dissolves into a `null` indistinguishable from a child failure — the accepted-then-ignored failure mode this repo bans. One addition: the tool's `args` parameter is a JSON OBJECT (a bare list is wrapped as a field) so the wire schema stays honest.
|
||||
|
||||
### The seam (dsh-workflow)
|
||||
|
||||
`ctx.workflows` is an abstract `WorkflowService` in the bash shape — one engine per context, no named-provider registry (engines are deployment swaps, not co-residents). `start(request)` throws synchronously for a script that cannot begin; a returned `WorkflowRun`'s `result` NEVER rejects (failures resolve as `stopReason: 'error' | 'cancelled'`). The `workflow/*` events are observe-only emits carrying DATA SNAPSHOTS (id + meta; `workflow/end` omits the result value), per-listener contained, mirroring `subagent/start`/`subagent/end` — control stays with the run's holder. Vocabulary details: [core-data-structures/workflow.md](../../../core-data-structures/workflow.md).
|
||||
|
||||
### The engine (dsh-workflow-workerthread): one worker thread per run
|
||||
|
||||
**Trust premise (governs every engine decision below)**: workflow scripts are MODEL-WRITTEN — the same trust level as the model's existing bash access — so the engine defends against BUGGY scripts, never hostile ones. In scope: `result` never rejects, no unhandled rejections from dropped hook promises, loud rejection of values JSON cannot carry, fatal-vs-null hook discipline, cancellation that always frees the caller. Out of scope, deliberately: adversarial values (throwing/spinning accessors, proxies with hostile traps, prototype forgery, `prepareStackTrace` hijack) AND Node-API escape from the script's context — the vm context shares object machinery with its surrounding realm, so a script can reach the `Function` constructor (`globalThis.constructor.constructor`) and from it `process` and every Node builtin; the absent globals are API surface, not containment, and a worker thread is NOT a security boundary (an escapee holds process-wide privileges — Node's permission model is per-process). Worker-side code MAY run script code while reading script values, and that is accepted: a synchronous spin costs the script its OWN thread (terminated at the post-cancel grace), never the host loop, so containing error VALUES would be cost without a threat model. Genuine sandboxing (isolated-vm, a separate process) remains an engine swap behind the seam, not incremental defenses here.
|
||||
|
||||
**Why node:worker_threads**: one run = one worker thread, no pooling — a run is heavyweight (many children), so thread spin-up (~tens of ms) is noise. The script runs in a vm context INSIDE the worker, keeping the script-visible surface exactly the hook contract above (a bare worker realm would leak `setTimeout`/`fetch`/`process` as accidental API), and every `agent()` bridges to `ctx.subagents` by message-port RPC — children are I/O-bound LLM loops and stay on the host loop; the thread isolates the SCRIPT, the only part that can spin. What the thread buys: `start()` never blocks the host (an in-process engine runs the initial synchronous slice inline and cannot kill a spin past the first await — it could only ABANDON such a script, leaving the spin on the host loop), the post-cancel grace ends in a REAL `worker.terminate()`, and the value boundary is serialization by construction. isolated-vm was rejected for actual sandboxing: maintenance mode, `--no-node-snapshot` on EVERY consumer process (including published bins) on Node ≥ 20, node-gyp source-build fallback. Key mechanics (details in the package README): meta shape-validation and a body pre-parse stay HOST-side (preserving the seam's synchronous throws), a ready→go handshake keeps a run cancelled before start from ever executing the body, `cancel()` drives both child-cancel channels host-side (the shared request signal AND each child's explicit `cancel()` — a wedged worker cannot relay its own cancel RPCs), a host-side child registry backs worker-death reaping and `dispose()` quiescence, the wire protocol is enum-keyed payload maps private to the package, and on a termination path `agentsStarted` degrades to the host-observed count. Coverage puts the worker-side session on an in-process `MessageChannel` (real-Worker code is invisible to main-process v8) and proves the built `lib/worker.js` — a second tsdown entry, sanctioned in the workspace-constraints gate by the `"./worker"` subpath export — under plain node in the built-bin smoke gate.
|
||||
|
||||
**Meta as data, never evaluated**: the meta block reaches the seam as a plain JSON request field (the tool's schema-validated `meta` parameter) and the engine only shape-validates it, every violation named. This is a host-isolation invariant, not a convenience: evaluating a meta literal host-side — even one contractually "pure", in an empty timed vm context — hands script-controlled getters a host stack with no timeout the moment the result is READ, defeating the exact spin isolation the worker thread buys.
|
||||
|
||||
**Value boundary**: values leaving the script (meta, hook options, schemas, the return value) go through `materializeFromRealm` — a plain recursive walk that rejects loud everything JSON cannot carry (exotic prototypes, functions, symbols, cycles, sparse arrays, non-finite numbers, nested `undefined`), copying via `Object.defineProperty` so a `"__proto__"` key becomes a data property, never a prototype mutation; getters are read ordinarily and their RESULT crosses (a throwing read fails loud) — which is also what makes every later postMessage hop total. Values entering the realm (`args`, `agent()` results, hook promises and failures, combinator arrays) are handed over directly as worker-realm values — the script is trusted, so outer prototypes are not a leak; `args` rides the `workerData` structured clone (the caller-isolation copy) and is cloned once more so a script scribbling on it cannot mutate the session's init object. Hook failures are `WorkflowError`s built OUTSIDE the script's context: the combinators recognize fatality by `instanceof` against the engine's own class (unforgeable from the script), and the script-visible consequence — in-script `instanceof Error` is `false` for hook errors; branch on `e.name`/`e.code` — is documented in the engine README. Realm functions (stages, thunks) are called, never materialized. Thrown script values are rendered by a total renderer (stack → message → `String()`, fixed label if rendering throws), so `result` cannot reject. Caps (`maxConcurrentAgents` auto = `min(16, max(1, availableParallelism() - 2))`, `maxTotalAgents` 1000, `maxItemsPerCall` 4096) and timeouts are validated Config, not literals.
|
||||
|
||||
### The consumer (dsh-tool-workflow)
|
||||
|
||||
A `workflow` tool mirroring `dsh-tool-subagent`'s synchronous shape: start, await, `try/finally` dispose, abort-bridge `exec.signal`, non-`completed` → `isError`. Render intent: a `generic` card titled by the call's `meta.name` parameter (presentation is a pure function of args). The tool description IS the model-facing authoring spec. The usage policy ships with the tool as its own `tool:<toolName>` prompt section (explicit-ask-only guidance — tool guidance lives in tool plugins, never in the deployment persona); the harness has no ultracode-style effort gate.
|
||||
|
||||
### The foundation: structured output on the subagent seam
|
||||
|
||||
`agent({schema})` needs `SubagentStartRequest.outputSchema` to actually work; it was vocabulary without an implementation (`outputSchema: false` everywhere). Implemented in `dsh-subagent-inprocess` for both in-process backends: a globally registered `structured_output` capture tool whose per-child schema is enforced by a `prepend: true` `system-prompt/assemble` listener doing FINAL-ASSEMBLY enforcement (post-processing `await next()` — cooperative mutation would not survive a downstream listener returning a replacement assembly; the calling instruction rides as a trailing prompt section, since `AgentOptions` carries no per-agent prompt field, and the loop logs the result as the step's `request/header`, keeping the injection reconstructable), a `prepend: true` `agent/turn-continuation` veto after capture (no wasted extra model step) plus a `tools/pre-execute` deny for calls arriving after the capture (terminal within the step, not only at its end), and validation-retry in-turn via `ToolArgsError`. The schema is `structuredClone`d at `start()` (caller mutation cannot drift enforcement). Deliberately NO re-prompt: a child that finishes cleanly without calling the tool settles `error` to the parent. Lifetime is refcounted by backends (plugin lifetime) AND live runs (start → settle). The seam's `outputSchema` type became the raw JSON-Schema SUBSET (`StructuredOutputSchema` in dsh-tools: single-string `type`, `properties`/`required`/`additionalProperties`, `items`, scalar `enum`/`const`; anything unenforced is rejected loud) — the schema travels verbatim to the model as the forced tool's parameters, so the wire format, not the author DSL, is the right vocabulary.
|
||||
|
||||
## Deferred (documented non-goals of this cut)
|
||||
|
||||
- **Background collection** (start tool → run id → completion notice → collect), designed alongside bash/subagent background unification.
|
||||
- **Journaling + resume** (`resumeFromRunId`, cached agent() prefixes) — implementing it reintroduces CC's determinism bans as a script-contract tightening (scripts may read the clock today).
|
||||
- **Saved/bundled workflows** (a `.deepseek/workflows/` registry, slash-command surface) and **script persistence to a run directory** (the tool-call event already records the script durably).
|
||||
- **Nested `workflow()`**, **token `budget`**, and the `effort`/`isolation`/`agentType` agent options (each rejects loud with a message naming it deferred).
|
||||
- **An overall run wall-clock timeout** — cancellation always frees the caller (result settles within the grace), so a cap on total run time is a policy knob for the background redesign, not a correctness need here.
|
||||
- **Engine hardening beyond worker threads**: an isolated-vm or separate-process engine behind the same seam (actual sandboxing; memory limits).
|
||||
- **ACP progress UI** over the `workflow/*` events (a `/workflows`-style view); the events exist for it.
|
||||
- **ACP-backend structured output** and **`toolFilter`** (both still capability-gated `false`).
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Hostile-value containment in the host** (trap-free proxy rejection, accessor-never-invoked descriptor walks, realm-side pre-rendering of thrown values, realm-built promises/arrays/error clones with structural fatal recognition): an earlier revision built all of it, and review showed the cost was real while the threat model was not — every one of those defenses guards against an author the premise already trusts. Removed in favor of the plain boundary above; the thread boundary makes such machinery redundant anyway (serialization by construction).
|
||||
- **In-process `node:vm` execution** (the first cut of this RFC shipped it): mechanically simplest — no RPC, no thread — but `start()` blocks the caller for the script's initial synchronous slice, a synchronous spin past the first await cannot be killed in-process (the vm `timeout` covers only that first slice), and `dispose()` could only ABANDON an unsettling script, leaving the spin on the host loop. Superseded by the worker-thread engine, which keeps the same vm-context script surface while unblocking the host and making termination real.
|
||||
- **Background execution as the default** (CC's shape): deferred; foreground-synchronous matches `dsh-tool-subagent`'s cut, and background semantics should be designed ONCE across bash/subagent/workflow rather than per-tool.
|
||||
- **Workflow-layer JSON parsing for `agent({schema})`**: duplicating a seam concern at one consumer while the seam's capability flag stayed dishonestly `false`.
|
||||
- **Meta embedded in the script as `export const meta = {...}`** (CC's exact format; the first cut shipped it): keeps scripts self-contained and CC scripts drop-in, but obtaining meta means evaluating model-written text on the HOST — the shipped extractor ran the literal in an empty timed vm context, yet reading the RESULT still executed script-controlled getters on the host stack outside any timeout, re-opening the host-spin hole the worker thread exists to close. A JSON parameter deletes the scanner, the evaluation, and the hole outright; the cost is that a CC script's meta header must move into the parameter (the body stays drop-in).
|
||||
- **`SchemaSpec` as the outputSchema type**: the author-facing DSL cannot express what arrives as data and cannot be validated against without conversion loss.
|
||||
- **A schema-object library (zod, or the repo's schemastery) for the structured-output subset**: the schema is wire data — plain JSON that crosses the vm realm boundary in `agent({schema})` and lands verbatim in the forced tool's parameters — exactly where live schema objects cannot sit; consuming raw JSON Schema at runtime would need a third-party converter on top (zod core only emits JSON Schema, not the reverse), and it would put a second schema language beside schemastery's config role.
|
||||
- **ajv for value validation**: it validates FULL JSON Schema, so the subset gate — the module's actual point, since every accepted keyword must be one the harness enforces — would remain hand-written regardless; it compiles validators through `new Function`; and it would be dsh-tools' first runtime dependency, all to replace the ~70-line value walker while the path-qualified, every-violation error reporting stays custom either way.
|
||||
- **Provider JSON mode (`response_format: {type: json_object}`) instead of the forced capture tool**: the official API guarantees valid JSON, not schema-conforming JSON (no `json_schema` type; the docs' own guidance is to validate client-side, with the schema riding in the prompt), so both walkers survive untouched and only the capture-tool mechanics could go — at the cost of tools during a structured child's run (whether `response_format` composes with tool calling is undocumented), the in-turn validation retry (`ToolArgsError` keeps recovery inside the turn; a JSON-mode empty body — a documented failure mode — ends the turn, and the only recovery is the re-prompt loop this design rejects), and a new per-adapter `LlmCallConfig` surface. The accepted upgrade path is strict TOOL schemas (provider-side constrained decoding on tool parameters) when available: the same forced tool and subset gate, with the gate narrowed to the provider's strict subset.
|
||||
|
||||
## Consequences
|
||||
|
||||
The harness gains CC-compatible script orchestration: fan-out plans live in a rerunnable artifact instead of the parent context, and the structured-output half of the subagent seam is now real (the vocabulary stopped lying about `outputSchema`). What it cost, all bounded by the trust premise: a worker thread per run (~tens-of-ms spin-up), every hook crossing a message port as RPC, and a termination-path `agentsStarted` that degrades to the host-observed count; in exchange `start()` never blocks the host, a post-cancel grace ends in a real `worker.terminate()`, and the value boundary is serialization by construction. A worker thread is still NOT a security boundary — scripts share the model's trust level, and actual sandboxing names its exit (the isolated-vm/separate-process engine swap behind the seam). The fatal-vs-null strictness divergence from CC means a CC-authored script that RELIES on option typos dissolving to `null` behaves differently here — judged worth it to keep the repo's no-accepted-then-ignored rule. Consumers must hold the run handle for control (`cancel`/`dispose`); observers get data snapshots only, so no listener can extend a run's lifetime or corrupt another's view.
|
||||
Reference in New Issue
Block a user