RFC: Explicit model-facing tool order

Status: implemented

Problem

The order of the tool list a model call carries — request/header.tools on the session log and GenerateOptions.tools on the wire — was an emergent artifact: the tool registry returns schemas in registration order, the system-prompt assembly concatenates providers in registration order, and the loop logged and dispatched the result verbatim. Registration order is plugin load order, and plugin load order is a race: the cordis loader imports every cordis.yml entry concurrently, so which tool plugin registers first depends on module-import completion timing. The plugin dependency relation cannot rescue this — it is a partial order under which independent tool plugins (e.g. tool-subagent vs tool-todo) are incomparable, so both interleavings are legal linearizations. This stopped being theoretical when a CI runner resolved the race differently from every recording machine: snapshot goldens pinned one permutation of request/header.tools, the node 22.18 CI leg produced the other, and 5/5 snapshot tests failed on a diff that was pure array reordering. Tool order is part of the request bytes (prompt-cache stability, potentially model behavior) and, since the reconstructability contract, part of the durable session log — it must be a decision, not a residue.

Decision

The system-prompt assembly owns the canonical model-facing tool order, exactly where it already owns section order. toolOrder?: string[] on dsh-system-prompt is the optional explicit policy:

A listed tool that is registered takes its listed position.
A listed name with no registered tool is a configuration error. Shape errors (rest entry missing or duplicate names) fail from the service constructor; an unregistered name rejects every assemble() — the earliest moment the registered tool set exists to check against (tool plugins register after the service constructs), and the only universal one (registrations can change at any time; cordis has no "all plugins loaded" event). Under the shipped loop the first turn fails before any model request — see the consequences below for the exact blast radius.
A registered tool absent from the list is inserted at the '<unlisted-tools>' rest entry (TOOL_ORDER_REST), in lexicographic name order among the other unlisted tools.
No collected tool may use TOOL_ORDER_REST as its ToolSchema.name; the assembly rejects that reserved name before ordering.
The list must contain the rest entry exactly once and no duplicate names.
When toolOrder is unset, the canonical order is plain lexicographic name order (code-unit comparison, locale-independent), so determinism requires no configuration.

The policy is applied where the list is born: assemble(), before the system-prompt/assemble waterfall. The assembly canonicalizes the tools it collects from providers the same way it sorts sections by their order field — on the initial assembly, killing the registration-order entropy at its source. The waterfall therefore starts from one deterministic list; when a listener leaves that order intact, the loop's EpochHeader, the request/header event, the deep-frozen request, and the dev invariant's cross-check inherit it with no new loop change.

Scope is deliberately narrow: this fixes the REGISTRATION-ORDER race, not plugin behavior. A system-prompt/assemble listener may still add, remove, or rearrange tools — same as it may edit sections after their sort — and owns the determinism of what it emits; the waterfall contract already demands deterministic listeners (the reconstructability invariant would catch a listener that diverges between build and replay).

Config plumbing follows the persona precedent, and toolOrder sits beside it: the app configs (dsh-stdio-agent, dsh-acp-agent) accept the key and forward it through dsh-agent-core (whose schema is the intersection of the owners' schemas) to the SystemPrompt child. One schemastery footnote is load-bearing: a schemastery array defaults to [], but an omitted toolOrder must stay ABSENT (= lexicographic) rather than become an explicitly-configured empty list (invalid — it lacks the rest entry), so every schema on the chain forces the default to undefined.

Alternatives considered

Registration order (the status quo) — a concurrent-import race, host-dependent (the CI flake above), invisible in review.
A linearization of the plugin dependency graph — the relation is partial and independent tool plugins are incomparable; the flake happened with the partial order fully satisfied.
Per-plugin weight on each tool contribution — scatters the order across plugins yet still needs a global numbering convention nobody owns (the section order bands show that coordination cost being paid by hand).
Sorting in ToolRegistry.schemas() (the registry layer) — equally deterministic, but the registry is a membership store consumed by more than the assembly; ordering is a prompt-composition concern, and the assembly already owns the composition policy for sections.
A LlmService config + orderTools() method the loop calls before logging the header — works, but adds a public service method and a loop edit solely to apply a policy at a distance; every future request composer must remember the call. Canonicalizing where the list is born makes an unordered list unrepresentable, with zero new surface.
Normalizing inside llm.stream() — runs after the header event is logged (the flake survives) and rebuilds the deep-frozen envelope, silently disarming the reconstruction invariant.
An exhaustive list (no rest entry) — every newly loaded tool plugin would break boot; the mandatory rest entry keeps unlisted tools deterministic and their position explicit.
A boot-time validation pass (a SystemPrompt.assertToolOrderSatisfied() called by dsh-app-boot after loader.await()) — would turn the misconfiguration into a startup death instead of a first-turn failure, but costs a public service method plus a structural coupling from the generic boot glue to one service, and cannot replace the assembly-time check anyway (embedded callers never run app boot; registrations change after boot). No existing event can host the check either: cordis v4 has no ready-like event, loader/entry-init/internal/status fire mid-load (racy against tool registration, the very entropy this RFC kills), and the agent lifecycle events are no earlier than the assembly. One enforcement point at assemble() was judged worth the later failure moment.

Consequences

Every registry-built assembly starts with a deterministic tool order on every host; absent an expert listener that deliberately changes it, every request/header event and model request inherits that order. The CI-vs-local registration-order flip is structurally gone, and the default is lexicographic.
The initial PromptAssembly.tools is canonical, so waterfall listeners start from the model-facing order; provider registration order is observable nowhere before that cooperative seam.
The snapshot suite's single pinned request-header fixture (text-turn) carries the new canonical tool order; every other ACP snapshot keeps the header bulk scrubbed as {{system}}/{{tools}}, per the pinned-header design.
A pure tool reordering between steps is logged like any other header change: a full request/header snapshot with reason 'change'. Stable canonical order prevents registration timing from creating such changes in the ordinary path.
The toolOrder key rides the app → agent-core → SystemPrompt forwarding chain, so deployments set it next to persona in the app config; dsh-llm and the agent loop are untouched.
A misspelled or unloaded tool name in toolOrder fails the turn at prompt assembly, not the boot: the loop assembles inside the turn (after turn/start, before step/start), so the rejection reaches the turn's outer catch — the turn closes balanced with an error reason carrying the message, agent/error mirrors it, no step opens, no request/header is logged, no request reaches the adapter, and the agent returns to idle. Every turn fails identically until the config is fixed; the process itself stays up (matching the repo rule that explicit config references must not be silently ignored — the enforcement point is the assembly because no earlier universal moment exists).
A tool provider that returns the reserved rest-entry name has the same prompt-assembly failure shape as an unknown listed name. This keeps the sentinel from becoming an ambiguous real tool and preserves the "never drops a tool" ordering contract.

Testing

Unit tests on dsh-system-prompt pin the ordering semantics (lexicographic default, listed/rest placement, unknown-name rejection at assembly, reserved tool-name rejection, stable handling of shared names, provider-order independence), the pre-waterfall contract (listeners observe the canonical list; a listener-appended tool is not re-sorted), and each invalid-list rejection at load. Loop-level tests assert the request/header fold carries the canonical order for scrambled registration orders (identical across permutations), that a configured toolOrder reaches both the logged header and the dispatched deep-frozen request, that the frozen loop-built envelope survives to the adapter, and that an unregistered toolOrder name fails the turn with a balanced error turn/end, an agent/error, no step, no logged header, and no dispatched request. Forwarding is asserted at every level that exposes the key (dsh-agent-core, dsh-stdio-agent, dsh-acp-agent). The snapshot tier replays all scenarios while only the pinned text-turn header carries the full canonical tool list; non-pinning fixtures continue to compare through {{tools}}.

9.7 KiB Raw Blame History