deepseek-harness/docs/rfc/implemented/feature/2026-07-05-dynamic-workflows.md

RFC: Dynamic workflows — a script-driven multi-agent orchestration seam

Status: implemented

Problem

The harness can delegate ONE task to ONE child (dsh-tool-subagent), but work that fans out across many independent pieces — an audit over many files, a migration, multi-angle research, adversarial verification of findings — forces the model to orchestrate turn by turn: every intermediate result lands in the parent context, the plan lives nowhere durable, and coordination costs a model round-trip per step. Claude Code ships this capability as dynamic workflows: the model writes a JavaScript orchestration script, a runtime executes it, and the script — not the conversation — holds the loop, the branching, and the intermediate results.

Decision

A workflow capability family at packages/workflow/ in the bash seam shape (interface / implementation / consumer), plus the structured-output foundation it needs on the subagent seam.

The script contract (Claude Code-compatible)

A workflow call is two parts: a meta JSON parameter (the identity block — name, description, optional whenToUse/phases; the field vocabulary matches Claude Code's meta block) and a script — a plain-JS body with top-level await, ending in return <json-value>. Meta is DATA, never code: the engine shape-validates it and evaluates no script text to obtain it (a body still opening with a CC-style export const meta statement is rejected with a pointed message). The body sees exactly: agent(prompt, {label, phase, schema, model}), parallel(thunks), pipeline(items, ...stages) (NO cross-stage barrier; (prev, item, index) callbacks), phase(title), log(message), and args. CC semantics are preserved where they matter to script authors: a failed child resolves null (scripts .filter(Boolean)); an ordinary stage throw nulls the ITEM and skips its remaining stages. CC's determinism bans (Date.now()/Math.random()/argless new Date() throwing) are NOT enforced — they exist for CC's journaling/resume, which this cut defers — so a CC-authored BODY runs unchanged (its meta header moves into the parameter) while scripts written here may freely read the clock.

One deliberate strictness DIVERGENCE from CC: hook misuse — unknown or deferred options (effort/isolation/agentType), malformed arguments, schemas outside the supported subset, tripped caps, seam start failures — throws a WorkflowError with fatal: true, and the combinators RE-THROW fatal errors instead of nulling the item. Without this, a typo'd option dissolves into a null indistinguishable from a child failure — the accepted-then-ignored failure mode this repo bans. One addition: the tool's args parameter is a JSON OBJECT (a bare list is wrapped as a field) so the wire schema stays honest.

The seam (dsh-workflow)

ctx.workflows is an abstract WorkflowService in the bash shape — one engine per context, no named-provider registry (engines are deployment swaps, not co-residents). start(request) throws synchronously for a script that cannot begin; a returned WorkflowRun's result NEVER rejects (failures resolve as stopReason: 'error' | 'cancelled'). The workflow/* events are observe-only emits carrying DATA SNAPSHOTS (id + meta; workflow/end omits the result value), per-listener contained, mirroring subagent/start/subagent/end — control stays with the run's holder. Vocabulary details: core-data-structures/workflow.md.

The engine (dsh-workflow-workerthread): one worker thread per run

Trust premise (governs every engine decision below): workflow scripts are MODEL-WRITTEN — the same trust level as the model's existing bash access — so the engine defends against BUGGY scripts, never hostile ones. In scope: result never rejects, no unhandled rejections from dropped hook promises, loud rejection of values JSON cannot carry, fatal-vs-null hook discipline, cancellation that always frees the caller. Out of scope, deliberately: adversarial values (throwing/spinning accessors, proxies with hostile traps, prototype forgery, prepareStackTrace hijack) AND Node-API escape from the script's context — the vm context shares object machinery with its surrounding realm, so a script can reach the Function constructor (globalThis.constructor.constructor) and from it process and every Node builtin; the absent globals are API surface, not containment, and a worker thread is NOT a security boundary (an escapee holds process-wide privileges — Node's permission model is per-process). Worker-side code MAY run script code while reading script values, and that is accepted: a synchronous spin costs the script its OWN thread (terminated at the post-cancel grace), never the host loop, so containing error VALUES would be cost without a threat model. Genuine sandboxing (isolated-vm, a separate process) remains an engine swap behind the seam, not incremental defenses here.

Why node:worker_threads: one run = one worker thread, no pooling — a run is heavyweight (many children), so thread spin-up (~tens of ms) is noise. The script runs in a vm context INSIDE the worker, keeping the script-visible surface exactly the hook contract above (a bare worker realm would leak setTimeout/fetch/process as accidental API), and every agent() bridges to ctx.subagents by message-port RPC — children are I/O-bound LLM loops and stay on the host loop; the thread isolates the SCRIPT, the only part that can spin. What the thread buys: start() never blocks the host (an in-process engine runs the initial synchronous slice inline and cannot kill a spin past the first await — it could only ABANDON such a script, leaving the spin on the host loop), the post-cancel grace ends in a REAL worker.terminate(), and the value boundary is serialization by construction. isolated-vm was rejected for actual sandboxing: maintenance mode, --no-node-snapshot on EVERY consumer process (including published bins) on Node ≥ 20, node-gyp source-build fallback. Key mechanics (details in the package README): meta shape-validation and a body pre-parse stay HOST-side (preserving the seam's synchronous throws), a ready→go handshake keeps a run cancelled before start from ever executing the body, cancel() drives both child-cancel channels host-side (the shared request signal AND each child's explicit cancel() — a wedged worker cannot relay its own cancel RPCs), a host-side child registry backs worker-death reaping and dispose() quiescence, the wire protocol is enum-keyed payload maps private to the package, and on a termination path agentsStarted degrades to the host-observed count. Coverage puts the worker-side session on an in-process MessageChannel (real-Worker code is invisible to main-process v8) and proves the built lib/worker.js — a second tsdown entry, sanctioned in the workspace-constraints gate by the "./worker" subpath export — under plain node in the built-bin smoke gate.

Meta as data, never evaluated: the meta block reaches the seam as a plain JSON request field (the tool's schema-validated meta parameter) and the engine only shape-validates it, every violation named. This is a host-isolation invariant, not a convenience: evaluating a meta literal host-side — even one contractually "pure", in an empty timed vm context — hands script-controlled getters a host stack with no timeout the moment the result is READ, defeating the exact spin isolation the worker thread buys.

Value boundary: values leaving the script (meta, hook options, schemas, the return value) go through materializeFromRealm — a plain recursive walk that rejects loud everything JSON cannot carry (exotic prototypes, functions, symbols, cycles, sparse arrays, non-finite numbers, nested undefined), copying via Object.defineProperty so a "__proto__" key becomes a data property, never a prototype mutation; getters are read ordinarily and their RESULT crosses (a throwing read fails loud) — which is also what makes every later postMessage hop total. Values entering the realm (args, agent() results, hook promises and failures, combinator arrays) are handed over directly as worker-realm values — the script is trusted, so outer prototypes are not a leak; args rides the workerData structured clone (the caller-isolation copy) and is cloned once more so a script scribbling on it cannot mutate the session's init object. Hook failures are WorkflowErrors built OUTSIDE the script's context: the combinators recognize fatality by instanceof against the engine's own class (unforgeable from the script), and the script-visible consequence — in-script instanceof Error is false for hook errors; branch on e.name/e.code — is documented in the engine README. Realm functions (stages, thunks) are called, never materialized. Thrown script values are rendered by a total renderer (stack → message → String(), fixed label if rendering throws), so result cannot reject. Caps (maxConcurrentAgents auto = min(16, max(1, availableParallelism() - 2)), maxTotalAgents 1000, maxItemsPerCall 4096) and timeouts are validated Config, not literals.

The consumer (dsh-tool-workflow)

A workflow tool mirroring dsh-tool-subagent's synchronous shape: start, await, try/finally dispose, abort-bridge exec.signal, non-completed → isError. Render intent: a generic card titled by the call's meta.name parameter (presentation is a pure function of args). The tool description IS the model-facing authoring spec. The usage policy ships with the tool as its own tool:<toolName> prompt section (explicit-ask-only guidance — tool guidance lives in tool plugins, never in the deployment persona); the harness has no ultracode-style effort gate.

The foundation: structured output on the subagent seam

agent({schema}) needs SubagentStartRequest.outputSchema to actually work; it was vocabulary without an implementation (outputSchema: false everywhere). Implemented in dsh-subagent-inprocess for both in-process backends: a globally registered structured_output capture tool whose per-child schema is enforced by a prepend: true system-prompt/assemble listener doing FINAL-ASSEMBLY enforcement (post-processing await next() — cooperative mutation would not survive a downstream listener returning a replacement assembly; the calling instruction rides as a trailing prompt section, since AgentOptions carries no per-agent prompt field, and the loop logs the result as the step's request/header, keeping the injection reconstructable), a prepend: true agent/turn-continuation veto after capture (no wasted extra model step) plus a tools/pre-execute deny for calls arriving after the capture (terminal within the step, not only at its end), and validation-retry in-turn via ToolArgsError. The schema is structuredCloned at start() (caller mutation cannot drift enforcement). Deliberately NO re-prompt: a child that finishes cleanly without calling the tool settles error to the parent. Lifetime is refcounted by backends (plugin lifetime) AND live runs (start → settle). The seam's outputSchema type became the raw JSON-Schema SUBSET (StructuredOutputSchema in dsh-tools: single-string type, properties/required/additionalProperties, items, scalar enum/const; anything unenforced is rejected loud) — the schema travels verbatim to the model as the forced tool's parameters, so the wire format, not the author DSL, is the right vocabulary.

Deferred (documented non-goals of this cut)

• Background collection (start tool → run id → completion notice → collect), designed alongside bash/subagent background unification.
• Journaling + resume (resumeFromRunId, cached agent() prefixes) — implementing it reintroduces CC's determinism bans as a script-contract tightening (scripts may read the clock today).
• Saved/bundled workflows (a .deepseek/workflows/ registry, slash-command surface) and script persistence to a run directory (the tool-call event already records the script durably).
• Nested workflow(), token budget, and the effort/isolation/agentType agent options (each rejects loud with a message naming it deferred).
• An overall run wall-clock timeout — cancellation always frees the caller (result settles within the grace), so a cap on total run time is a policy knob for the background redesign, not a correctness need here.
• Engine hardening beyond worker threads: an isolated-vm or separate-process engine behind the same seam (actual sandboxing; memory limits).
• ACP progress UI over the workflow/* events (a /workflows-style view); the events exist for it.
• ACP-backend structured output and toolFilter (both still capability-gated false).

Alternatives considered

• Hostile-value containment in the host (trap-free proxy rejection, accessor-never-invoked descriptor walks, realm-side pre-rendering of thrown values, realm-built promises/arrays/error clones with structural fatal recognition): an earlier revision built all of it, and review showed the cost was real while the threat model was not — every one of those defenses guards against an author the premise already trusts. Removed in favor of the plain boundary above; the thread boundary makes such machinery redundant anyway (serialization by construction).
• In-process node:vm execution (the first cut of this RFC shipped it): mechanically simplest — no RPC, no thread — but start() blocks the caller for the script's initial synchronous slice, a synchronous spin past the first await cannot be killed in-process (the vm timeout covers only that first slice), and dispose() could only ABANDON an unsettling script, leaving the spin on the host loop. Superseded by the worker-thread engine, which keeps the same vm-context script surface while unblocking the host and making termination real.
• Background execution as the default (CC's shape): deferred; foreground-synchronous matches dsh-tool-subagent's cut, and background semantics should be designed ONCE across bash/subagent/workflow rather than per-tool.
• Workflow-layer JSON parsing for agent({schema}): duplicating a seam concern at one consumer while the seam's capability flag stayed dishonestly false.
• Meta embedded in the script as export const meta = {...} (CC's exact format; the first cut shipped it): keeps scripts self-contained and CC scripts drop-in, but obtaining meta means evaluating model-written text on the HOST — the shipped extractor ran the literal in an empty timed vm context, yet reading the RESULT still executed script-controlled getters on the host stack outside any timeout, re-opening the host-spin hole the worker thread exists to close. A JSON parameter deletes the scanner, the evaluation, and the hole outright; the cost is that a CC script's meta header must move into the parameter (the body stays drop-in).
• SchemaSpec as the outputSchema type: the author-facing DSL cannot express what arrives as data and cannot be validated against without conversion loss.
• A schema-object library (zod, or the repo's schemastery) for the structured-output subset: the schema is wire data — plain JSON that crosses the vm realm boundary in agent({schema}) and lands verbatim in the forced tool's parameters — exactly where live schema objects cannot sit; consuming raw JSON Schema at runtime would need a third-party converter on top (zod core only emits JSON Schema, not the reverse), and it would put a second schema language beside schemastery's config role.
• ajv for value validation: it validates FULL JSON Schema, so the subset gate — the module's actual point, since every accepted keyword must be one the harness enforces — would remain hand-written regardless; it compiles validators through new Function; and it would be dsh-tools' first runtime dependency, all to replace the ~70-line value walker while the path-qualified, every-violation error reporting stays custom either way.
• Provider JSON mode (response_format: {type: json_object}) instead of the forced capture tool: the official API guarantees valid JSON, not schema-conforming JSON (no json_schema type; the docs' own guidance is to validate client-side, with the schema riding in the prompt), so both walkers survive untouched and only the capture-tool mechanics could go — at the cost of tools during a structured child's run (whether response_format composes with tool calling is undocumented), the in-turn validation retry (ToolArgsError keeps recovery inside the turn; a JSON-mode empty body — a documented failure mode — ends the turn, and the only recovery is the re-prompt loop this design rejects), and a new per-adapter LlmCallConfig surface. The accepted upgrade path is strict TOOL schemas (provider-side constrained decoding on tool parameters) when available: the same forced tool and subset gate, with the gate narrowed to the provider's strict subset.

Consequences

The harness gains CC-compatible script orchestration: fan-out plans live in a rerunnable artifact instead of the parent context, and the structured-output half of the subagent seam is now real (the vocabulary stopped lying about outputSchema). What it cost, all bounded by the trust premise: a worker thread per run (~tens-of-ms spin-up), every hook crossing a message port as RPC, and a termination-path agentsStarted that degrades to the host-observed count; in exchange start() never blocks the host, a post-cancel grace ends in a real worker.terminate(), and the value boundary is serialization by construction. A worker thread is still NOT a security boundary — scripts share the model's trust level, and actual sandboxing names its exit (the isolated-vm/separate-process engine swap behind the seam). The fatal-vs-null strictness divergence from CC means a CC-authored script that RELIES on option typos dissolving to null behaves differently here — judged worth it to keep the repo's no-accepted-then-ignored rule. Consumers must hold the run handle for control (cancel/dispose); observers get data snapshots only, so no listener can extend a run's lifetime or corrupt another's view.