diff --git a/.agents/notes/implemented/feature/2026-07-07-mcp-client-plugin.i18n.yaml b/.agents/notes/implemented/feature/2026-07-07-mcp-client-plugin.i18n.yaml index ca61a3783c..678ae169cf 100644 --- a/.agents/notes/implemented/feature/2026-07-07-mcp-client-plugin.i18n.yaml +++ b/.agents/notes/implemented/feature/2026-07-07-mcp-client-plugin.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write .agents/notes/implemented/feature/2026-07-07-mcp-client-plugin.md -2026-07-07-mcp-client-plugin.md: 756a5c4dc9f1152ecb1955d93b8dc47fcf07c661 -2026-07-07-mcp-client-plugin.zh.md: 9f45203c479379087c5a19eccce3ab7a339d4ed0 +2026-07-07-mcp-client-plugin.md: 077d978d8815d759574f89ed524ab3bc8c24a267 +2026-07-07-mcp-client-plugin.zh.md: 8f58c9359ca8447717cc353f97f1333370fa6fda diff --git a/.agents/notes/implemented/feature/2026-07-07-mcp-client-plugin.md b/.agents/notes/implemented/feature/2026-07-07-mcp-client-plugin.md index 756a5c4dc9..077d978d88 100644 --- a/.agents/notes/implemented/feature/2026-07-07-mcp-client-plugin.md +++ b/.agents/notes/implemented/feature/2026-07-07-mcp-client-plugin.md @@ -153,13 +153,7 @@ Build the child environment from the subprocess seam's shared `scrubbedParentEnv ### Disconnection / crash -No auto-reconnect. If the MCP server process exits or the transport closes: - -1. The effect disposes → all registered tools are unregistered (fiber-scoped disposers). -2. Subsequent model calls to those tools → `ToolNotFoundError` → `isError: true`. -3. Recovery: user edits `cordis.yml` (triggers HMR reload) or restarts the harness. - -This matches the ACP subagent pattern: "crash = terminal, report error, clean up, don't retry." +A per-instance connection supervisor reconnects automatically after a lost connection with bounded exponential backoff and a per-outage attempt budget, re-running discovery on success; exhaustion unregisters the server's tools and stops until reload. The [auto-reconnect Agent Note](2026-08-06-mcp-client-auto-reconnect.md) owns that decision, including the `reconnect` config block and the `reconnect.enabled: false` opt-out that restores manual HMR/restart recovery. ## Alternatives considered @@ -173,7 +167,7 @@ Rejected. There is no foreseeable alternative MCP client implementation — MCP ### Auto-reconnect with exponential backoff -Rejected for v1. Adds complexity (partial-availability state where tools are registered but temporarily non-functional), and stdio process crashes usually indicate a configuration problem that retrying won't fix. HMR already provides the manual recovery path. Can be added as a future `reconnect: boolean` config if needed. +Rejected for v1: it added a partial-availability state (tools registered but temporarily non-functional), and stdio crashes often indicate configuration problems retrying cannot fix; HMR was the recovery path. Operational feedback reversed the deferral — the [auto-reconnect Agent Note](2026-08-06-mcp-client-auto-reconnect.md) implements it with a bounded per-outage budget and an opt-out. ### Bridge Resources and Prompts @@ -211,4 +205,4 @@ Coverage is named per tier; each behavior lives at the cheapest tier that can ex - **MCP SDK stability**: the `@modelcontextprotocol/sdk` is still evolving; breaking changes require updating the bridge. The version is pinned, and the SDK is widely adopted (Claude Desktop, Cursor, VS Code) so breaking changes are unlikely to be silent. - **Tool schema quality**: MCP servers may expose poorly-described tools (vague descriptions, incomplete JSON schemas). The harness passes them through as-is — garbage-in-garbage-out; that is the server author's responsibility, not the bridge's. - **Stdio process management**: a misbehaving MCP server that ignores signals could wedge dispose. The Cordis fiber disposal has bounded quiescence; a stuck transport eventually times out at the framework level. -- Crash recovery is manual (HMR edit or restart) — accepted for v1; a `reconnect` config remains open as future work. +- Crash recovery is automatic within the [reconnect budget](2026-08-06-mcp-client-auto-reconnect.md); manual reload remains the path after exhaustion or with `reconnect.enabled: false`. diff --git a/.agents/notes/implemented/feature/2026-07-07-mcp-client-plugin.zh.md b/.agents/notes/implemented/feature/2026-07-07-mcp-client-plugin.zh.md index 9f45203c47..8f58c9359c 100644 --- a/.agents/notes/implemented/feature/2026-07-07-mcp-client-plugin.zh.md +++ b/.agents/notes/implemented/feature/2026-07-07-mcp-client-plugin.zh.md @@ -153,13 +153,7 @@ MCP 仅保证工具名在[单个服务器内](https://modelcontextprotocol.io/sp ### 断连 / 崩溃 -不自动重连。如果 MCP 服务器进程退出或传输层关闭： - -1. effect dispose → 所有已注册工具被注销（fiber 作用域的 disposer）。 -2. 后续模型对这些工具的调用 → `ToolNotFoundError` → `isError: true`。 -3. 恢复：用户编辑 `cordis.yml`（触发 HMR 重载）或重启 harness。 - -这与 ACP subagent 模式一致：「崩溃即终态，报告错误，清理资源，不重试。」 +每个实例的连接监督器在连接丢失后以有界指数退避和单次故障尝试预算自动重连，成功后重新执行发现流程；尝试耗尽则注销该服务器的工具并停止，直到重新加载。[自动重连 Agent Note](2026-08-06-mcp-client-auto-reconnect.md) 拥有该决策，包括 `reconnect` 配置块和恢复手动 HMR/重启恢复的 `reconnect.enabled: false` opt-out。 ## 曾考虑的替代方案 @@ -173,7 +167,7 @@ MCP 仅保证工具名在[单个服务器内](https://modelcontextprotocol.io/sp ### 指数退避自动重连 -v1 否决。引入复杂性（工具已注册但暂时不可用的部分可用状态），且 stdio 进程崩溃通常表明配置问题，重试无法修复。HMR 已提供手动恢复路径。如有需要，可在未来作为 `reconnect: boolean` 配置项添加。 +v1 否决：引入了部分可用状态（工具已注册但暂时不可用），且 stdio 崩溃往往表明配置问题，重试无法修复；HMR 曾是恢复路径。运营反馈扭转了该延期决定——[自动重连 Agent Note](2026-08-06-mcp-client-auto-reconnect.md) 以有界的单次故障预算和 opt-out 实现了自动重连。 ### 桥接 Resources 和 Prompts @@ -211,4 +205,4 @@ v1 否决。它能防止跨服务器冲突，但无法将 MCP 注册与原生 ha - **MCP SDK 稳定性**：`@modelcontextprotocol/sdk` 仍在演进中；破坏性变更需要更新桥接。版本已固定，且该 SDK 被广泛采用（Claude Desktop、Cursor、VS Code），因此破坏性变更不太可能悄然发生。 - **工具 schema 质量**：MCP 服务器可能暴露描述不佳的工具（模糊的描述、不完整的 JSON Schema）。harness 原样透传——垃圾进垃圾出；这是服务器作者的责任，不是桥接的。 - **Stdio 进程管理**：行为异常的 MCP 服务器如果忽略信号，可能卡住 dispose。Cordis fiber 的 dispose 具有有界的完全停稳过程；卡住的传输层最终会在框架层面超时。 -- 崩溃恢复是手动的（HMR 编辑或重启）——v1 已接受；`reconnect` 配置作为未来工作保持开放。 +- 崩溃恢复在[重连预算](2026-08-06-mcp-client-auto-reconnect.md)内自动进行；耗尽后或配置 `reconnect.enabled: false` 时回退为手动重新加载。 diff --git a/.agents/notes/implemented/feature/2026-08-06-mcp-client-auto-reconnect.i18n.yaml b/.agents/notes/implemented/feature/2026-08-06-mcp-client-auto-reconnect.i18n.yaml new file mode 100644 index 0000000000..d9ec007b53 --- /dev/null +++ b/.agents/notes/implemented/feature/2026-08-06-mcp-client-auto-reconnect.i18n.yaml @@ -0,0 +1,6 @@ +# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each +# side as of the last confirmed-consistent state. Both languages carry equal authority; +# after editing either side, bring the other along and re-record with: +# pnpm run verify-translation-pairing --write .agents/notes/implemented/feature/2026-08-06-mcp-client-auto-reconnect.md +2026-08-06-mcp-client-auto-reconnect.md: b187f5de71a10ca3121d817f383a98669246c81f +2026-08-06-mcp-client-auto-reconnect.zh.md: a4ec6897c34f2142ceb95abecd13d189c4a95624 diff --git a/.agents/notes/implemented/feature/2026-08-06-mcp-client-auto-reconnect.md b/.agents/notes/implemented/feature/2026-08-06-mcp-client-auto-reconnect.md new file mode 100644 index 0000000000..b187f5de71 --- /dev/null +++ b/.agents/notes/implemented/feature/2026-08-06-mcp-client-auto-reconnect.md @@ -0,0 +1,48 @@ +# Agent Note: MCP client auto-reconnect with bounded backoff + +Status: implemented + +English | [中文](2026-08-06-mcp-client-auto-reconnect.zh.md) + +## Problem + +The [MCP client](2026-07-07-mcp-client-plugin.md) connected once at plugin load. When a stdio server crashed or was killed, its registered tools stayed visible but every call failed with `Not connected` until a human edited the config (HMR) or restarted the Host — v1 explicitly deferred reconnection. Long-running hosts (ACP automation, web) cannot be bounced because a child process died, and for stdio the harness composition is the only party that can respawn it. External feedback escalated this as a real operational gap (issue #1746). + +## Decision + +`packages/mcp/mcp-client/src/connection.ts` owns a per-instance connection supervisor; `apply()` shrinks to config resolution plus two effects (the `serverName` reservation and the supervisor's lifecycle). The supervisor owns the client/transport generations, the live tool registrations, and the reconnect loop. + +**Trigger.** The supervisor arms `client.onclose` per generation. The SDK fires it when the stdio child exits, so a crash is observed without polling. `StreamableHTTPClientTransport` fires `onclose` only for deliberate closes — it owns its internal SSE-stream recovery and surfaces request failures per call — so HTTP servers are effectively outside supervisor restarts; the package README records that limitation. + +**Generations without interleaving.** Each attempt builds a fresh transport and `Client` (the SDK binds a Protocol to one transport for life). One per-supervisor queue serializes every `syncTools` call — initial syncs and `list_changed` re-syncs across all generations — and an `isCurrent` fence makes stale generations inert, so no two syncs can interleave the dispose-previous/register-next swap (which would double-dispose one generation and leak another). The queue also closes a pre-existing race where two rapid `list_changed` notifications re-synced concurrently. Failure signals are idempotent per generation: a connect rejection racing its own transport close schedules exactly one retry. + +**Bounded backoff with an outage budget.** Delays double from `initialDelayMs` up to `maxDelayMs`. One outage shares `maxAttempts` consecutive failed attempts; exhaustion unregisters the server's tools, logs at error level, and stops until disposal or reload. A connection that survives past the stability window — `maxDelayMs`, derived rather than a fifth tunable, as the longest configured backoff spacing — resets the budget, so an occasionally-crashing server recovers indefinitely while a crash loop whose connects briefly succeed cannot launder its budget into a restart storm. + +**Config and resolution.** Both transports accept `reconnect { enabled, initialDelayMs, maxDelayMs, maxAttempts }` with schemastery defaults (on, 500ms, 30s, 10). `resolveReconnectPolicy()` is the explicit resolve step: it re-judges every bound and cross-field constraint because programmatic construction may bypass Schemastery, and misconfiguration fails the plugin instance at load. + +**Observable states.** Reconnecting logs at warn with attempt count and delay, recovery at info, final failure and disabled-loss at error. During an outage the last good generation stays registered and calls against it fail — deterministic public names mean a recovered unchanged tool list reproduces identical definitions, keeping the model-visible schema prefix stable instead of flapping. With `reconnect.enabled: false` a lost connection keeps the v1 manual-recovery behavior. + +**Disposal.** Dispose flips the fence, cancels any pending timer, closes the current client, then awaits the in-flight attempt and the sync queue before unregistering — quiescence, not just a request to stop. The reconnect timer is unref'd so a waiting backoff never holds a finishing process open. + +## Alternatives considered + +**Consecutive-failure counter that resets on every successful connect.** Rejected: a crash-looping server whose connects briefly succeed would reset the budget each cycle and restart forever — exactly the restart storm the failure cap exists to prevent. The uptime-gated reset distinguishes a recovered server from a looping one without new configuration. + +**Reuse one SDK `Client` across reconnects.** The Protocol clears its transport on close and can technically connect again, but the SDK's own guidance is one connection per Protocol instance, and reuse carries notification handlers and negotiated capability state across server incarnations. A fresh `Client` per generation plus the `isCurrent` fence is unambiguous. + +**Unregister tools immediately on disconnect, re-register on recovery.** Rejected: a transient outage would flap the model-visible tool list (two schema-prefix invalidations per crash) for no information gain; failing calls already signal the outage, and the swap on recovery is atomic per generation. Tools are unregistered at final failure so a permanently dead server does not leak permanently broken tools. + +**Route Streamable HTTP request failures into the supervisor.** Rejected for now: the HTTP transport already reconnects its SSE stream with its own backoff, per-request errors do not imply a dead server, and there is no child process the harness could respawn. Transport close stays the single trigger. + +**Restart through Loader/HMR machinery instead of an in-plugin supervisor.** Rejected: the Loader owns config-driven recomposition, not runtime health. A plugin restarting itself through the Loader would conflate config generations with connection generations and lose the per-outage budget. + +## Testing + +Unit (`tests/reconnect.spec.ts`, mocked SDK): recovery swaps generations without duplication or leaks and serves post-recovery calls, the failure cap unregisters tools and stops, dispose cancels a pending backoff and quiesces an in-flight sync, a close after dispose schedules nothing, disabled mode keeps the v1 behavior, the stability window resets the budget while a crash loop exhausts it, double failure signals schedule one retry, stale generations and handlers are inert, and `resolveReconnectPolicy` rejects each invalid bound. E2E (`tests/mcp-client.e2e.ts`, keyless): the fixture server gained a `crash` tool that replies then exits; real-process tests prove a stdio crash recovers end to end and that unloading the plugin mid-outage stops reconnection promptly. Snapshot: deliberately none, per the original note's rationale — reconnection adds no new presentation shape, and a snapshot composition spawning a crashing server would make replays timing-dependent. + +## Consequences + +- A crashed stdio MCP server recovers without human intervention: bounded backoff, re-discovery, atomic generation swap. Default policy retries an outage for roughly 2.5 minutes before giving up. +- Connection state is genuinely more intricate than connect-once — the partial-availability window v1 avoided now exists (registered tools failing during an outage), concentrated in one module with the invariants named. +- `reconnect` is new config surface on both transports, and the stability window is deliberately derived from `maxDelayMs`; making it independently tunable is a compatible future change. +- After final failure or with reconnect disabled, the plugin stays loaded with no (or failing) tools until reload — deliberate and logged, so a chronically broken server cannot restart forever. diff --git a/.agents/notes/implemented/feature/2026-08-06-mcp-client-auto-reconnect.zh.md b/.agents/notes/implemented/feature/2026-08-06-mcp-client-auto-reconnect.zh.md new file mode 100644 index 0000000000..a4ec6897c3 --- /dev/null +++ b/.agents/notes/implemented/feature/2026-08-06-mcp-client-auto-reconnect.zh.md @@ -0,0 +1,48 @@ +# Agent Note: MCP client auto-reconnect with bounded backoff + +Status: implemented + +[English](2026-08-06-mcp-client-auto-reconnect.md) | 中文 + +## 问题 + +[MCP 客户端](2026-07-07-mcp-client-plugin.md)在插件加载时仅连接一次。stdio 服务器崩溃或被终止后，其已注册的工具仍然可见，但每次调用均以 `Not connected` 失败，直到人工编辑配置触发 HMR（热模块替换）重载，或重启 Host——v1 明确推迟了重连机制。长时间运行的 Host（ACP 自动化、Web）不能因为子进程死亡就被重启；而对于 stdio 传输，harness 组合层是唯一能重新拉起子进程的一方。外部反馈将此升级为真实的运维缺口（issue #1746）。 + +## 决策 + +`packages/mcp/mcp-client/src/connection.ts` 拥有一个逐实例的连接监督器；`apply()` 收缩为配置解析加两个副作用（`serverName` 预留和监督器的生命周期）。监督器负责管理 client/transport 代、活跃的工具注册以及重连循环。 + +**触发条件。** 监督器在每一代上挂载 `client.onclose`。SDK 在 stdio 子进程退出时触发该回调，因此崩溃无需轮询即可感知。`StreamableHTTPClientTransport` 仅在主动关闭时触发 `onclose`——它内部拥有自己的 SSE（Server-Sent Events）流恢复机制，并将请求失败以逐调用方式暴露——因此 HTTP 服务器实际上不在监督器的重启范围内；包 README 记录了该限制。 + +**代隔离，无交错。** 每次尝试构建一个全新的 transport 和 `Client`（SDK 将一个 Protocol 绑定到一个 transport 上终身使用）。每个监督器内部有一个队列将所有 `syncTools` 调用串行化——跨所有代的初始同步和 `list_changed` 再同步——`isCurrent` 栅栏使过时的代变为惰性，从而确保不会有两次同步交错执行 dispose 上一代/注册下一代的切换（否则会对同一代执行两次 dispose 并泄漏另一代）。该队列还消除了一个先前存在的竞态：两次快速的 `list_changed` 通知同时触发重新同步。失败信号按代幂等：一次连接拒绝与其自身 transport 关闭竞态时，仅调度恰好一次重试。 + +**有界退避与故障预算。** 延迟从 `initialDelayMs` 起逐次翻倍，上限为 `maxDelayMs`。一次故障期间共享 `maxAttempts` 次连续失败尝试的预算；耗尽后注销该服务器的工具、以 error 级别记录日志并停止，直到 dispose 或重新加载。连接在存活超过稳定窗口——即 `maxDelayMs`，作为最长退避间隔从配置推导得出而非作为第五个独立调参项——之后重置预算；因此偶尔崩溃的服务器可无限恢复，而连接短暂成功后立即再次崩溃的循环无法将其预算洗白为重启风暴。 + +**配置与解析。** 两种传输均接受 `reconnect { enabled, initialDelayMs, maxDelayMs, maxAttempts }` 配置，Schemastery 默认值为（启用、500ms、30s、10）。`resolveReconnectPolicy()` 是显式的解析步骤：它重新校验每个边界值和跨字段约束，因为程序化构造可能绕过 Schemastery，配置错误在加载时即令插件实例失败。 + +**可观测状态。** 重连中以 warn 级别记录尝试次数和延迟，恢复以 info 级别记录，最终失败和禁用状态下的断连以 error 级别记录。故障期间，上一个正常代保持注册，对其工具的调用返回失败——确定性公开名称意味着恢复后未变化的工具列表会复现相同的定义，保持模型可见 schema 前缀稳定而非反复抖动。设置 `reconnect.enabled: false` 后，断连保持 v1 的手动恢复行为。 + +**资源释放。** dispose 翻转栅栏、取消待执行的定时器、关闭当前 client，然后等待正在进行的尝试和同步队列完成后再注销工具——完全停稳，而非仅发出停止请求。重连定时器使用 unref，因此等待中的退避不会阻止进程正常退出。 + +## 曾考虑的替代方案 + +**连续失败计数器，每次成功连接即重置。** 否决：连接短暂成功后立即崩溃的循环服务器会在每个周期重置预算并永远重启——恰恰是失败上限旨在防止的重启风暴。基于运行时间的重置能区分已恢复的服务器与循环崩溃的服务器，无需新增配置。 + +**跨重连复用同一个 SDK `Client`。** Protocol 在关闭时清除其 transport，技术上可以再次连接，但 SDK 自身的指导方针是每个 Protocol 实例对应一次连接，且复用会将通知处理器和已协商的能力状态带入新的服务器实例。每代创建全新 `Client` 加 `isCurrent` 栅栏的方式无歧义。 + +**断连时立即注销工具，恢复时重新注册。** 否决：短暂故障会使模型可见工具列表抖动（每次崩溃触发两次 schema 前缀失效），而无任何信息增益；失败的调用已足以标示故障，恢复时的切换按代原子执行。工具仅在最终失败时注销，确保永久死亡的服务器不会泄漏永久失效的工具。 + +**将 Streamable HTTP 请求失败路由到监督器。** 暂不采纳：HTTP 传输已使用自己的退避机制重连其 SSE 流，逐请求错误并不意味着服务器已死，且 harness 没有可重新拉起的子进程。transport 关闭仍是唯一触发条件。 + +**通过 Loader/HMR 机制重启，而非使用插件内监督器。** 否决：Loader 负责配置驱动的重组合，而非运行时健康管理。插件通过 Loader 重启自身会混淆配置代与连接代，并丢失逐故障预算。 + +## 测试 + +单元测试（`tests/reconnect.spec.ts`，mock SDK）：恢复在不产生重复或泄漏的前提下切换代并服务恢复后的调用、失败上限注销工具并停止、dispose 取消待执行的退避并使进行中的同步完全停稳、dispose 后的关闭不调度任何操作、禁用模式保持 v1 行为、稳定窗口重置预算而崩溃循环耗尽预算、双重失败信号仅调度一次重试、过时的代和处理器为惰性、`resolveReconnectPolicy` 拒绝每个无效边界值。E2E（`tests/mcp-client.e2e.ts`，无需密钥）：fixture 服务器新增了一个 `crash` 工具（先回复再退出）；真实进程测试证明 stdio 崩溃端到端恢复，以及在故障期间卸载插件能立即停止重连。快照：刻意不做，原因与原 Agent Note 相同——重连不引入新的展示形态，而在快照组合中 spawn 崩溃服务器会使回放依赖时序。 + +## 后果 + +- 崩溃的 stdio MCP 服务器无需人工干预即可恢复：有界退避、重新发现、原子代切换。默认策略对一次故障大约重试 2.5 分钟后放弃。 +- 连接状态确实比一次性连接更复杂——v1 刻意回避的部分可用窗口现已存在（故障期间已注册工具返回失败），集中在一个模块中并命名了所有不变式。 +- `reconnect` 是两种传输上的新配置表面，稳定窗口刻意从 `maxDelayMs` 推导；将其设为独立可调参数是兼容的未来变更。 +- 最终失败后或禁用重连时，插件保持加载状态但无（或失败的）工具，直到重新加载——行为是刻意的且有日志记录，确保长期故障的服务器不能永远重启。 diff --git a/docs/config-catalog.i18n.yaml b/docs/config-catalog.i18n.yaml index 09c961ef69..6a90ab01d5 100644 --- a/docs/config-catalog.i18n.yaml +++ b/docs/config-catalog.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write docs/config-catalog.md -config-catalog.md: 51c6ae46eeca1279390c9d9315a6161edd2de618 -config-catalog.zh.md: dc93f5b4b55b07c52c58405ba4793c2c6eca28df +config-catalog.md: 76f1d0344d3e172416af36eff27bd4cb8544ed4a +config-catalog.zh.md: d5752a1922d8b8b2e70f055ec05117ed20c445bf diff --git a/docs/config-catalog.md b/docs/config-catalog.md index 51c6ae46ee..76f1d0344d 100644 --- a/docs/config-catalog.md +++ b/docs/config-catalog.md @@ -1069,6 +1069,8 @@ export interface StdioConfig { toolCallTimeoutMs: number /** Fail plugin activation when the initial connection or tool synchronization fails. */ failOnStartupError: boolean + /** Automatic reconnect policy after a lost connection; omission uses the defaults. */ + reconnect?: ReconnectConfig } /** Config for connecting to an MCP server over Streamable HTTP (SSE). */ @@ -1089,10 +1091,24 @@ export interface StreamableHttpConfig { toolCallTimeoutMs: number /** Fail plugin activation when the initial connection or tool synchronization fails. */ failOnStartupError: boolean + /** Automatic reconnect policy after a lost connection; omission uses the defaults. */ + reconnect?: ReconnectConfig +} + +/** Automatic reconnect policy for one MCP server connection. */ +export interface ReconnectConfig { + /** Reconnect automatically after a lost connection (default true). */ + enabled?: boolean + /** First reconnect delay in milliseconds; doubles per consecutive failed attempt (default 500). */ + initialDelayMs?: number + /** Backoff ceiling in milliseconds; also the uptime after which the attempt budget resets (default 30000). */ + maxDelayMs?: number + /** Consecutive failed attempts per outage before giving up for good (default 10). */ + maxAttempts?: number } ``` -Source: [`packages/mcp/mcp-client/src/index.ts:100`](../packages/mcp/mcp-client/src/index.ts) +Source: [`packages/mcp/mcp-client/src/index.ts:104`](../packages/mcp/mcp-client/src/index.ts) ## `@deepseek-ai/dsh-permission` diff --git a/docs/config-catalog.zh.md b/docs/config-catalog.zh.md index dc93f5b4b5..d5752a1922 100644 --- a/docs/config-catalog.zh.md +++ b/docs/config-catalog.zh.md @@ -1071,6 +1071,8 @@ export interface StdioConfig { toolCallTimeoutMs: number /** Fail plugin activation when the initial connection or tool synchronization fails. */ failOnStartupError: boolean + /** Automatic reconnect policy after a lost connection; omission uses the defaults. */ + reconnect?: ReconnectConfig } /** Config for connecting to an MCP server over Streamable HTTP (SSE). */ @@ -1091,6 +1093,20 @@ export interface StreamableHttpConfig { toolCallTimeoutMs: number /** Fail plugin activation when the initial connection or tool synchronization fails. */ failOnStartupError: boolean + /** Automatic reconnect policy after a lost connection; omission uses the defaults. */ + reconnect?: ReconnectConfig +} + +/** Automatic reconnect policy for one MCP server connection. */ +export interface ReconnectConfig { + /** Reconnect automatically after a lost connection (default true). */ + enabled?: boolean + /** First reconnect delay in milliseconds; doubles per consecutive failed attempt (default 500). */ + initialDelayMs?: number + /** Backoff ceiling in milliseconds; also the uptime after which the attempt budget resets (default 30000). */ + maxDelayMs?: number + /** Consecutive failed attempts per outage before giving up for good (default 10). */ + maxAttempts?: number } ``` diff --git a/docs/module-graph.i18n.yaml b/docs/module-graph.i18n.yaml index d478f79cd1..349fadadec 100644 --- a/docs/module-graph.i18n.yaml +++ b/docs/module-graph.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write docs/module-graph.md -module-graph.md: 6763ca6e84a5cc2e5e776a56e5cfc20b710017aa -module-graph.zh.md: 339345b39a8225c34ecf0ba73efff34d4b940046 +module-graph.md: 78beee5f9311ae4c75a9ac291d1c7882914bfb8f +module-graph.zh.md: 6b71b0997ce542d53285cd815277556dba46b353 diff --git a/docs/module-graph.md b/docs/module-graph.md index 6763ca6e84..78beee5f93 100644 --- a/docs/module-graph.md +++ b/docs/module-graph.md @@ -920,6 +920,7 @@ flowchart TD pkg_mcp_client --> pkg_invariants pkg_mcp_client --> pkg_llm pkg_mcp_client --> pkg_subprocess + pkg_mcp_client --> pkg_timeout pkg_mcp_client --> pkg_tools pkg_tool_bash_persistent --> pkg_agent pkg_tool_bash_persistent --> pkg_invariants @@ -1403,7 +1404,7 @@ flowchart TD | [`timeout-policy`](../packages/guard/timeout-policy) | `guard` | [`invariants`](../packages/support/invariants), [`llm`](../packages/llm/llm), [`timeout`](../packages/util/timeout), [`tools`](../packages/core/tools) | | [`tool-ask-user`](../packages/interaction/tool-ask-user) | `interaction` | [`agent`](../packages/core/agent), [`invariants`](../packages/support/invariants), [`tools`](../packages/core/tools), [`user-interaction`](../packages/interaction/user-interaction) | | [`tool-lsp`](../packages/lsp/tool-lsp) | `lsp` | [`invariants`](../packages/support/invariants), [`llm`](../packages/llm/llm), [`lsp`](../packages/lsp/lsp), [`system-prompt`](../packages/core/system-prompt), [`timeout`](../packages/util/timeout), [`tools`](../packages/core/tools) | -| [`mcp-client`](../packages/mcp/mcp-client) | `mcp` | [`invariants`](../packages/support/invariants), [`llm`](../packages/llm/llm), [`subprocess`](../packages/subprocess/subprocess), [`tools`](../packages/core/tools) | +| [`mcp-client`](../packages/mcp/mcp-client) | `mcp` | [`invariants`](../packages/support/invariants), [`llm`](../packages/llm/llm), [`subprocess`](../packages/subprocess/subprocess), [`timeout`](../packages/util/timeout), [`tools`](../packages/core/tools) | | [`tool-bash-persistent`](../packages/pty/tool-bash-persistent) | `pty` | [`agent`](../packages/core/agent), [`invariants`](../packages/support/invariants), [`pty`](../packages/pty/pty), [`timeout`](../packages/util/timeout), [`tools`](../packages/core/tools) | | [`tool-pty`](../packages/pty/tool-pty) | `pty` | [`agent`](../packages/core/agent), [`invariants`](../packages/support/invariants), [`llm`](../packages/llm/llm), [`pty`](../packages/pty/pty), [`retention`](../packages/util/retention), [`system-prompt`](../packages/core/system-prompt), [`tasks`](../packages/tasks/tasks), [`tools`](../packages/core/tools) | | [`tool-cordis`](../packages/self-modification/tool-cordis) | `self-modification` | [`invariants`](../packages/support/invariants), [`scope`](../packages/core/scope), [`tools`](../packages/core/tools) | diff --git a/docs/module-graph.zh.md b/docs/module-graph.zh.md index 339345b39a..6b71b0997c 100644 --- a/docs/module-graph.zh.md +++ b/docs/module-graph.zh.md @@ -922,6 +922,7 @@ flowchart TD pkg_mcp_client --> pkg_invariants pkg_mcp_client --> pkg_llm pkg_mcp_client --> pkg_subprocess + pkg_mcp_client --> pkg_timeout pkg_mcp_client --> pkg_tools pkg_tool_bash_persistent --> pkg_agent pkg_tool_bash_persistent --> pkg_invariants @@ -1405,7 +1406,7 @@ flowchart TD | [`timeout-policy`](../packages/guard/timeout-policy) | `guard` | [`invariants`](../packages/support/invariants), [`llm`](../packages/llm/llm), [`timeout`](../packages/util/timeout), [`tools`](../packages/core/tools) | | [`tool-ask-user`](../packages/interaction/tool-ask-user) | `interaction` | [`agent`](../packages/core/agent), [`invariants`](../packages/support/invariants), [`tools`](../packages/core/tools), [`user-interaction`](../packages/interaction/user-interaction) | | [`tool-lsp`](../packages/lsp/tool-lsp) | `lsp` | [`invariants`](../packages/support/invariants), [`llm`](../packages/llm/llm), [`lsp`](../packages/lsp/lsp), [`system-prompt`](../packages/core/system-prompt), [`timeout`](../packages/util/timeout), [`tools`](../packages/core/tools) | -| [`mcp-client`](../packages/mcp/mcp-client) | `mcp` | [`invariants`](../packages/support/invariants), [`llm`](../packages/llm/llm), [`subprocess`](../packages/subprocess/subprocess), [`tools`](../packages/core/tools) | +| [`mcp-client`](../packages/mcp/mcp-client) | `mcp` | [`invariants`](../packages/support/invariants), [`llm`](../packages/llm/llm), [`subprocess`](../packages/subprocess/subprocess), [`timeout`](../packages/util/timeout), [`tools`](../packages/core/tools) | | [`tool-bash-persistent`](../packages/pty/tool-bash-persistent) | `pty` | [`agent`](../packages/core/agent), [`invariants`](../packages/support/invariants), [`pty`](../packages/pty/pty), [`timeout`](../packages/util/timeout), [`tools`](../packages/core/tools) | | [`tool-pty`](../packages/pty/tool-pty) | `pty` | [`agent`](../packages/core/agent), [`invariants`](../packages/support/invariants), [`llm`](../packages/llm/llm), [`pty`](../packages/pty/pty), [`retention`](../packages/util/retention), [`system-prompt`](../packages/core/system-prompt), [`tasks`](../packages/tasks/tasks), [`tools`](../packages/core/tools) | | [`tool-cordis`](../packages/self-modification/tool-cordis) | `self-modification` | [`invariants`](../packages/support/invariants), [`scope`](../packages/core/scope), [`tools`](../packages/core/tools) | diff --git a/packages/mcp/mcp-client/README.i18n.yaml b/packages/mcp/mcp-client/README.i18n.yaml index 9568c08454..c66d8ffba8 100644 --- a/packages/mcp/mcp-client/README.i18n.yaml +++ b/packages/mcp/mcp-client/README.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write packages/mcp/mcp-client/README.md -README.md: 76d1271f6f7a3e9c959bdcf5e969906f25563c56 -README.zh.md: b2da1119af3a8d52761a5040059e7a9d922aa567 +README.md: 97ac9c173fc2b848f524e8c0fdd93eca072567af +README.zh.md: 1b5b5c523e0a477db30f97a748651dbe7e6992ea diff --git a/packages/mcp/mcp-client/README.md b/packages/mcp/mcp-client/README.md index 76d1271f6f..97ac9c173f 100644 --- a/packages/mcp/mcp-client/README.md +++ b/packages/mcp/mcp-client/README.md @@ -45,6 +45,10 @@ The model sees `mcp__github__create_issue`, `mcp__web__search`, … — the same | `headers` | http | no | Extra headers (e.g. auth tokens) | | `toolCallTimeoutMs` | both | no | Timeout per `callTool` invocation (default 60000) | | `failOnStartupError` | both | no | Reject plugin activation when initial connection or tool synchronization fails (default `false`) | +| `reconnect.enabled` | both | no | Reconnect automatically after a lost connection (default `true`) | +| `reconnect.initialDelayMs` | both | no | First reconnect delay in ms; doubles per consecutive failed attempt (default 500) | +| `reconnect.maxDelayMs` | both | no | Backoff ceiling in ms; also the uptime after which the attempt budget resets (default 30000) | +| `reconnect.maxAttempts` | both | no | Consecutive failed attempts per outage before giving up for good (default 10) | ## Tool naming @@ -62,7 +66,9 @@ Every MCP tool has two names: the raw MCP name (sent on the wire in `tools/call` - Tool execute: `client.callTool({ name: rawName, arguments }, { signal })` with timeout + abort support—the public name is never sent to the server. - Canonical success is `{ content: JsonValue[], structuredContent? }`; complete JSON MCP blocks survive for programmatic callers. A supported advertised `outputSchema` validates `structuredContent`; unsupported schema vocabulary falls back to unconstrained `JsonValue`. - Native/model rendering keeps the existing text projection: text blocks join with newlines while image, audio, resource, and unsupported blocks become placeholders. -- On disconnect/crash: no auto-reconnect. Registered tools remain until plugin disposal or a successful re-sync, and calls can fail against the closed transport; reload with HMR or restart the Host to reconnect. +- On disconnect/crash: the supervisor restarts the original server config with exponential backoff (`reconnect.initialDelayMs` doubling up to `reconnect.maxDelayMs`) and re-runs discovery on success — the recovered generation replaces the previous one, so tools neither duplicate nor leak. During the outage the last good generation stays registered; calls against it fail until recovery. +- Reconnection is budgeted per outage: after `reconnect.maxAttempts` consecutive failures the server's tools are unregistered and reconnection stops until an HMR reload or Host restart. A connection that survives past `maxDelayMs` resets the budget, so an occasionally-crashing server recovers indefinitely while a crash-looping one — even with briefly successful connects — still exhausts the cap instead of restarting forever. +- Reconnect states are user-visible in logs: reconnecting (warn, with attempt count and delay), recovered (info), final failure and disabled-loss (error). Disposal cancels any pending reconnect. With `reconnect.enabled: false`, a lost connection keeps tools registered but failing until a reload — the manual-recovery behavior. ## Services consumed @@ -76,7 +82,7 @@ Every MCP tool has two names: the raw MCP name (sent on the wire in `tools/call` #### What the model sees -After initial discovery succeeds, each advertised MCP tool appears as a native tool named `mcp____` (or its deterministic normalized form), with the server-provided description and input schema. A successful re-sync replaces the generation; plugin disposal removes it. +After initial discovery succeeds, each advertised MCP tool appears as a native tool named `mcp____` (or its deterministic normalized form), with the server-provided description and input schema. A successful re-sync — including the one after an automatic reconnect — replaces the generation; plugin disposal or an exhausted reconnect budget removes it. #### Token effect @@ -84,7 +90,7 @@ Data-dependent schema cost is paid on every request while the tools are register #### KV Cache effect -Prefix-stable while the discovered tool set and schemas are unchanged. A re-sync that adds, removes, renames, or changes a tool replaces definitions and may invalidate reuse from the first changed schema token. +Prefix-stable while the discovered tool set and schemas are unchanged. A re-sync that adds, removes, renames, or changes a tool replaces definitions and may invalidate reuse from the first changed schema token; a reconnect that recovers an unchanged list reproduces identical definitions and stays prefix-stable. ### Tool-call history and results @@ -104,6 +110,6 @@ Append-only; newly visible content follows the reusable request prefix and does - **Tools are the only bridged MCP capability** — Resources and Prompts have no harness consumption surface and are deferred. - **Startup timeout is inherited from the MCP SDK** — DSH does not yet expose a connection/discovery timeout. Each initialize or paginated `tools/list` request uses the SDK's 60-second default, so an unresponsive server or cursor chain can delay both activation and teardown while the initial synchronization settles. -- **Crash recovery is manual** — transport closure does not auto-reconnect; registered tools can remain visible but fail against the closed transport until an HMR reload or Host restart. +- **Reconnect triggers on transport close** — a crashed stdio child fires it; Streamable HTTP failures surface per request and through the SDK transport's own SSE-stream recovery, so an unreachable HTTP server is retried per call rather than respawned by the supervisor. - **Native non-text rendering is lossy** — image, audio, and resource payloads become placeholders in model context even though the execution-local canonical value preserves their JSON blocks. Richer Native multimedia projection is deferred. - **Unsupported MCP output schemas are not enforced** — `structuredContent` falls back to `JsonValue` when the advertised schema uses vocabulary outside the harness subset. diff --git a/packages/mcp/mcp-client/README.zh.md b/packages/mcp/mcp-client/README.zh.md index b2da1119af..1b5b5c523e 100644 --- a/packages/mcp/mcp-client/README.zh.md +++ b/packages/mcp/mcp-client/README.zh.md @@ -45,6 +45,10 @@ MCP 客户端桥接插件：连接外部 [Model Context Protocol](https://modelc | `headers` | http | 否 | 额外标头（例如认证 token） | | `toolCallTimeoutMs` | 两者 | 否 | 每次 `callTool` 调用的超时（默认 60000） | | `failOnStartupError` | 两者 | 否 | 初始连接或工具同步失败时拒绝插件激活（默认 `false`） | +| `reconnect.enabled` | 两者 | 否 | 连接丢失后自动重新连接（默认 `true`） | +| `reconnect.initialDelayMs` | 两者 | 否 | 首次重连延迟（毫秒）；每次连续失败尝试翻倍（默认 500） | +| `reconnect.maxDelayMs` | 两者 | 否 | 退避上限（毫秒）；同时也是重置尝试预算所需的正常运行时长（默认 30000） | +| `reconnect.maxAttempts` | 两者 | 否 | 每次中断期间连续失败尝试次数上限，超出后彻底放弃（默认 10） | ## 工具命名 @@ -62,7 +66,9 @@ MCP 客户端桥接插件：连接外部 [Model Context Protocol](https://modelc - 工具执行：`client.callTool({ name: rawName, arguments }, { signal })`，支持超时 + 中止；公开名称绝不会发给服务器。 - 规范成功值是 `{ content: JsonValue[], structuredContent? }`；完整的 JSON MCP 块会保留给编程调用方。受支持且已声明的 `outputSchema` 会验证 `structuredContent`；不受支持的 schema 词汇会回退为不受约束的 `JsonValue`。 - Native／模型渲染保留现有文本投影：文本块以换行连接，图片、音频、资源和不受支持的块会变成占位符。 -- 断开／崩溃时：不自动重新连接。已注册工具会一直保留到对插件执行 dispose（资源释放）或成功重新同步，针对已关闭传输的调用可能失败；请通过 HMR 重新加载或重启 Host 来重新连接。 +- 断开／崩溃时：supervisor 以指数退避（`reconnect.initialDelayMs` 逐次翻倍，上限 `reconnect.maxDelayMs`）重启原始服务器配置，成功后重新执行发现——恢复的世代会替换前一个，因此工具既不会重复也不会泄漏。中断期间最后一个正常世代保持注册；针对它的调用在恢复前会失败。 +- 重连按中断预算控制：连续失败达到 `reconnect.maxAttempts` 次后，该服务器的工具会被注销，重连停止，直到 HMR 重载或重启 Host。连接存活超过 `maxDelayMs` 会重置预算，因此偶尔崩溃的服务器可以无限恢复，而崩溃循环的服务器——即使短暂连接成功——仍会耗尽上限而非永远重启。 +- 重连状态在日志中对用户可见：reconnecting（warn，含尝试次数和延迟）、recovered（info）、最终失败和 disabled-loss（error）。dispose（资源释放）会取消任何待执行的重连。设置 `reconnect.enabled: false` 时，连接丢失后工具保持注册但调用失败，直到重载——即手动恢复行为。 ## 消费的服务 @@ -76,7 +82,7 @@ MCP 客户端桥接插件：连接外部 [Model Context Protocol](https://modelc #### 模型看到的内容 -初始发现成功后，每个已声明的 MCP 工具都会显示为名为 `mcp____`（或其确定性规范化形式）的原生工具，并携带服务器提供的描述和输入 schema。成功的重新同步会替换整个世代；对插件执行 dispose 会移除该世代。 +初始发现成功后，每个已声明的 MCP 工具都会显示为名为 `mcp____`（或其确定性规范化形式）的原生工具，并携带服务器提供的描述和输入 schema。成功的重新同步——包括自动重连后的同步——会替换整个世代；对插件执行 dispose（资源释放）或重连预算耗尽会移除该世代。 #### Token 影响 @@ -84,7 +90,7 @@ MCP 客户端桥接插件：连接外部 [Model Context Protocol](https://modelc #### KV Cache 影响 -只要已发现工具集合及其 schema 不变，前缀就保持稳定。增加、移除、重命名或更改工具的重新同步会替换定义，并可能使从第一个变化的 schema token 起的复用失效。 +只要已发现工具集合及其 schema 不变，前缀就保持稳定。增加、移除、重命名或更改工具的重新同步会替换定义，并可能使从第一个变化的 schema token 起的复用失效；恢复了未变列表的重连会生成完全相同的定义，前缀保持稳定。 ### 工具调用历史与结果 @@ -104,6 +110,6 @@ MCP 客户端桥接插件：连接外部 [Model Context Protocol](https://modelc - **只桥接 MCP 的工具能力**：资源和提示词没有 harness 消费接口，暂缓实现。 - **启动超时继承自 MCP SDK**：DSH 尚未公开连接／发现超时。每次 initialize 请求或分页 `tools/list` 请求都使用 SDK 默认的 60 秒，因此在初始同步完成期间，无响应的 server 或 cursor chain 可能同时延迟激活与 teardown。 -- **崩溃恢复需要手动触发**：传输关闭后不会自动重新连接；已注册工具可能仍然可见，但会因传输已关闭而调用失败，直到 HMR 重载或重启 Host。 +- **重连在传输关闭时触发**：崩溃的 stdio 子进程会触发重连；Streamable HTTP 失败通过每次请求以及 SDK 传输自身的 SSE（Server-Sent Events）流恢复机制暴露，因此不可达的 HTTP 服务器会按调用重试，而非由 supervisor 重新 spawn。 - **Native 非文本渲染有损**：图片、音频与资源载荷在模型上下文中会变成占位符，即使执行局部的规范值保留了其 JSON 块。更丰富的 Native 多媒体投影暂缓实现。 - **不强制执行不受支持的 MCP 输出 schema**：已声明 schema 使用 harness 子集之外的词汇时，`structuredContent` 会回退到 `JsonValue`。 diff --git a/packages/mcp/mcp-client/package.json b/packages/mcp/mcp-client/package.json index 3f65271905..18f4cac7a8 100644 --- a/packages/mcp/mcp-client/package.json +++ b/packages/mcp/mcp-client/package.json @@ -28,6 +28,7 @@ "@deepseek-ai/dsh-invariants": "^0.0.1", "@deepseek-ai/dsh-llm": "^0.0.1", "@deepseek-ai/dsh-subprocess": "^0.0.1", + "@deepseek-ai/dsh-timeout": "^0.0.1", "@deepseek-ai/dsh-tools": "^0.0.1", "cordis": "^4.0.0-rc.7" }, @@ -40,6 +41,7 @@ "@deepseek-ai/dsh-invariants": "workspace:^", "@deepseek-ai/dsh-llm": "workspace:^", "@deepseek-ai/dsh-subprocess": "workspace:^", + "@deepseek-ai/dsh-timeout": "workspace:^", "@deepseek-ai/dsh-tools": "workspace:^", "@modelcontextprotocol/server-everything": "^2026.7.4", "@modelcontextprotocol/server-filesystem": "^2026.7.4", diff --git a/packages/mcp/mcp-client/src/connection.ts b/packages/mcp/mcp-client/src/connection.ts new file mode 100644 index 0000000000..47bf3c8003 --- /dev/null +++ b/packages/mcp/mcp-client/src/connection.ts @@ -0,0 +1,292 @@ +/** + * Connection supervisor: owns the MCP client/transport generations for one + * plugin instance, keeps the harness tool registry in sync with the live + * generation, and — when the connection drops — restarts the configured + * server with bounded exponential backoff. + * + * One outage shares one attempt budget (`maxAttempts` consecutive failed + * attempts, delays doubling from `initialDelayMs` up to `maxDelayMs`). A + * connection that stays up past the stability window closes the outage, so + * the next disconnect starts a fresh budget while a crash-looping server — + * even one whose connects briefly succeed — still exhausts the cap instead of + * restarting forever. Exhaustion unregisters the server's tools and stops; + * disposal (including HMR) is the only way back from that state. + * + * @module + */ + +import { Client } from '@modelcontextprotocol/sdk/client/index.js' +import { ToolListChangedNotificationSchema } from '@modelcontextprotocol/sdk/types.js' +import type { Context } from 'cordis' +import { MAX_TIMER_DELAY_MS } from '@deepseek-ai/dsh-timeout' +import { createTransport } from './transport.ts' +import { syncTools } from './tools.ts' +import type { ToolBridgeOptions, ToolDisposers } from './tools.ts' +import type { Config } from './index.ts' + +/** Automatic reconnect policy for one MCP server connection. */ +export interface ReconnectConfig { + /** Reconnect automatically after a lost connection (default true). */ + enabled?: boolean + /** First reconnect delay in milliseconds; doubles per consecutive failed attempt (default 500). */ + initialDelayMs?: number + /** Backoff ceiling in milliseconds; also the uptime after which the attempt budget resets (default 30000). */ + maxDelayMs?: number + /** Consecutive failed attempts per outage before giving up for good (default 10). */ + maxAttempts?: number +} + +/** Defaults shared by the Config schema and {@link resolveReconnectPolicy}. */ +export const RECONNECT_DEFAULTS: Required = Object.freeze({ + enabled: true, + initialDelayMs: 500, + maxDelayMs: 30_000, + maxAttempts: 10, +}) + +/** Fully resolved reconnect policy captured at plugin load. */ +export type ResolvedReconnectPolicy = Readonly> + +/** + * The one explicit resolve step from raw reconnect config to the policy the + * supervisor runs. Programmatic construction may bypass Schemastery + * normalization, so every default and bound is re-judged here — misconfiguration + * fails the plugin instance at load. + * + * @param config - Raw `reconnect` config; omission uses the defaults. + * @param path - Diagnostic prefix naming the config location in thrown messages. + * @returns The frozen resolved policy. + */ +export function resolveReconnectPolicy(config: ReconnectConfig | undefined, path: string): ResolvedReconnectPolicy { + if (config !== undefined) { + for (const key of Object.keys(config)) { + if (!Object.hasOwn(RECONNECT_DEFAULTS, key)) throw new Error(`${path}.${key} is not a reconnect option`) + } + } + const enabled = config?.enabled ?? RECONNECT_DEFAULTS.enabled + const initialDelayMs = config?.initialDelayMs ?? RECONNECT_DEFAULTS.initialDelayMs + const maxDelayMs = config?.maxDelayMs ?? RECONNECT_DEFAULTS.maxDelayMs + const maxAttempts = config?.maxAttempts ?? RECONNECT_DEFAULTS.maxAttempts + /* jscpd:ignore-start — domain-specific delay validation parallels llm retry-policy; not extractable */ + if (!Number.isFinite(initialDelayMs) || initialDelayMs <= 0 || initialDelayMs > MAX_TIMER_DELAY_MS) { + throw new Error(`${path}.initialDelayMs must be a positive finite number no greater than ${MAX_TIMER_DELAY_MS}`) + } + if (!Number.isFinite(maxDelayMs) || maxDelayMs <= 0 || maxDelayMs > MAX_TIMER_DELAY_MS) { + throw new Error(`${path}.maxDelayMs must be a positive finite number no greater than ${MAX_TIMER_DELAY_MS}`) + } + if (initialDelayMs > maxDelayMs) { + throw new Error(`${path}.initialDelayMs must be less than or equal to maxDelayMs`) + } + if (!Number.isInteger(maxAttempts) || maxAttempts < 1) { + throw new Error(`${path}.maxAttempts must be a positive integer`) + } + /* jscpd:ignore-end */ + return Object.freeze({ enabled, initialDelayMs, maxDelayMs, maxAttempts }) +} + +/** Result from the initial connection attempt, for startup-await semantics. */ +export interface ConnectionOutcome { + /** If the initial connection or tool sync failed, the error; otherwise absent. */ + error?: unknown +} + +/** Handle for one plugin instance's supervised connection. */ +export interface ConnectionHandle { + /** + * Settles when the first connection attempt completes (success or failure). + * The supervisor enters its reconnect loop regardless; the caller decides + * whether a failed startup is fatal via `failOnStartupError`. + */ + ready: Promise + /** + * Stop reconnection, close the live client, wait for the in-flight attempt + * and queued tool syncs to quiesce, then unregister every tool this server + * still owns. + */ + dispose(): Promise +} + +/** + * Start the supervised connection for one MCP server and keep it alive per + * the reconnect policy. + * + * @param ctx - Cordis context providing the `tools` registry and logger. + * @param config - Resolved plugin config selecting the transport and server identity. + * @param policy - Resolved reconnect policy from {@link resolveReconnectPolicy}. + * @returns Handle with a `ready` promise for startup-await and a `dispose` for teardown. + */ +export function startConnection(ctx: Context, config: Config, policy: ResolvedReconnectPolicy): ConnectionHandle { + const label = `mcp-client(${config.serverName})` + const opts: ToolBridgeOptions = { + registrationFailure: 'contain', + serverName: config.serverName, + toolCallTimeoutMs: config.toolCallTimeoutMs, + } + // The initial sync uses 'throw' when failOnStartupError is configured, so + // a registration conflict propagates to the startup-await path. Re-syncs + // and reconnect syncs always contain conflicts. + const startupOpts: ToolBridgeOptions = config.failOnStartupError + ? { ...opts, registrationFailure: 'throw' } + : opts + let isFirstSync = true + + let disposed = false + /** Current generation: the connecting or connected client; undefined during backoff waits and after final failure. */ + let client: Client | undefined + /** Live tool registrations owned by this server; only {@link enqueueSync} and dispose swap it. */ + let disposers: ToolDisposers = new Map() + let reconnectTimer: NodeJS.Timeout | undefined + /** Consecutive failed connection attempts within the current outage. */ + let failedAttempts = 0 + /** When the current generation finished connect + initial sync; undefined while down. */ + let connectedAt: number | undefined + /** The real error from the first connection attempt, for startup-await diagnostics. */ + let firstAttemptError: unknown + + /** A generation may act only while it is the current one on a live plugin. */ + const isCurrent = (generation: Client): boolean => !disposed && client === generation + + /** + * Serializes every syncTools call — initial syncs and notification re-syncs + * across all generations — so two syncs can never interleave their + * dispose-previous/register-next swap (which would double-dispose one + * generation and leak another). + */ + let syncChain: Promise = Promise.resolve() + function enqueueSync(generation: Client): Promise { + const syncOpts = isFirstSync ? startupOpts : opts + isFirstSync = false + const run = syncChain.then(async () => { + if (!isCurrent(generation)) return + disposers = await syncTools(generation, ctx, syncOpts, disposers) + }) + // The chain tail must survive a failed sync; the enqueuing caller owns reporting. + syncChain = run.catch(() => {}) + return run + } + + /** One disconnect decision per generation: the isCurrent guard makes racing close/error signals idempotent. */ + function generationDown(generation: Client): void { + if (!isCurrent(generation)) return + client = undefined + scheduleReconnect() + } + + function scheduleReconnect(): void { + if (!policy.enabled) { + const detail = connectedAt !== undefined + ? 'registered tools will fail until an HMR reload or Host restart' + : 'no tools were registered; reload the plugin or restart the Host to connect' + ctx.logger.error(`${label}: connection lost and reconnect is disabled — ${detail}`) + return + } + // A connection that stayed up past the stability window (= maxDelayMs, the + // longest backoff spacing) ended the previous outage: start a fresh budget. + if (connectedAt !== undefined && Date.now() - connectedAt >= policy.maxDelayMs) failedAttempts = 0 + connectedAt = undefined + failedAttempts += 1 + if (failedAttempts > policy.maxAttempts) { + // Enqueue the give-up disposal so it cannot race an in-flight sync's + // phase-2 swap (which checks isCurrent inside the queue). + syncChain = syncChain.then(() => { + for (const dispose of disposers.values()) dispose() + disposers = new Map() + }) + ctx.logger.error(`${label}: giving up after ${policy.maxAttempts} consecutive failed reconnect attempts — tools unregistered; reload the plugin or restart the Host to reconnect`) + return + } + const delayMs = Math.min(policy.maxDelayMs, policy.initialDelayMs * 2 ** (failedAttempts - 1)) + ctx.logger.warn(`${label}: connection lost; reconnecting in ${delayMs}ms (attempt ${failedAttempts}/${policy.maxAttempts})`) + reconnectTimer = setTimeout(() => { + reconnectTimer = undefined + settling = connectGeneration() + }, delayMs) + // An armed reconnect timer must never hold the process open on its own. + reconnectTimer.unref() + } + + /** + * One connection attempt: fresh transport + client (the MCP SDK binds a + * Protocol to one transport for life), connect, then queue the initial tool + * sync. Every failure funnels through {@link generationDown}; success arms + * the onclose-driven disconnect path. Never rejects. + */ + async function connectGeneration(): Promise { + const generation = new Client( + { name: 'dsh-mcp-client', version: '0.0.1' }, + { capabilities: {} }, + ) + client = generation + generation.onclose = () => { generationDown(generation) } + // Registered before connect so a list change during the initial sync is + // queued behind it rather than dropped. + generation.setNotificationHandler( + ToolListChangedNotificationSchema, + async () => { + if (!isCurrent(generation)) return + ctx.logger.info(`${label}: tool list changed, re-syncing`) + try { + await enqueueSync(generation) + } catch (error) { + // Fetch-phase failure: the previous generation is still registered + // and `disposers` still owns it — keep serving the last good list. + if (!disposed) ctx.logger.error(`${label}: tool re-sync failed: ${String(error)}`) + } + }, + ) + try { + await generation.connect(createTransport(config)) + await enqueueSync(generation) + } catch (error) { + if (firstAttemptError === undefined) firstAttemptError = error + // When the transport closed first, onclose already logged and scheduled. + if (isCurrent(generation)) ctx.logger.warn(`${label}: connection attempt failed: ${String(error)}`) + try { await generation.close() } catch { /* transport already gone */ } + generationDown(generation) + return + } + if (!isCurrent(generation)) return + connectedAt = Date.now() + if (failedAttempts > 0) ctx.logger.info(`${label}: reconnected and re-synced tools (attempt ${failedAttempts}/${policy.maxAttempts})`) + } + + /** The in-flight (or last settled) connection attempt; dispose awaits it for quiescence. */ + let settling = connectGeneration() + + // The ready promise settles when the first attempt finishes (regardless of + // success). If the first attempt fails and reconnect is enabled, the + // supervisor is already scheduling a retry — ready just reports the outcome. + const ready: Promise = settling.then(() => { + // After settling: if client is set the initial connect+sync succeeded. + // If not, the supervisor either scheduled a retry (error logged) or gave + // up (error logged). Either way the outcome is reported with the real error. + // Note: settling.then() is a microtask; stdio onclose is a macrotask — so + // a server that crashes AFTER a successful initial sync cannot flip client + // to undefined before this continuation runs. + if (client !== undefined) return {} + /* v8 ignore next -- defensive: firstAttemptError is always set when connect/sync fails */ + return { error: firstAttemptError ?? new Error(`${label}: initial connection failed`) } + }) + + return { + ready, + async dispose(): Promise { + disposed = true + if (reconnectTimer !== undefined) { + clearTimeout(reconnectTimer) + reconnectTimer = undefined + } + const current = client + client = undefined + if (current !== undefined) { + try { await current.close() } catch { /* transport already gone */ } + } + // Quiesce, don't just request it: the in-flight attempt enqueues its + // sync before settling, so awaiting both leaves `disposers` final. + await settling + await syncChain + for (const dispose of disposers.values()) dispose() + disposers = new Map() + }, + } +} diff --git a/packages/mcp/mcp-client/src/index.ts b/packages/mcp/mcp-client/src/index.ts index 0fe7dd9de3..6696b7ce15 100644 --- a/packages/mcp/mcp-client/src/index.ts +++ b/packages/mcp/mcp-client/src/index.ts @@ -15,14 +15,14 @@ import type { Context } from 'cordis' import z from 'schemastery' -import { Client } from '@modelcontextprotocol/sdk/client/index.js' -import { ToolListChangedNotificationSchema } from '@modelcontextprotocol/sdk/types.js' -import { createTransport } from './transport.ts' -import { syncTools } from './tools.ts' +import { MAX_TIMER_DELAY_MS } from '@deepseek-ai/dsh-timeout' +import { RECONNECT_DEFAULTS, resolveReconnectPolicy, startConnection } from './connection.ts' +import type { ReconnectConfig } from './connection.ts' // Side-effect type import: declaration-merges `ctx.tools` onto Context. import type {} from '@deepseek-ai/dsh-tools' export type { McpResult } from './tools.ts' +export type { ReconnectConfig, ResolvedReconnectPolicy } from './connection.ts' /** Cordis plugin name used by loader diagnostics. */ export const name = 'mcp-client' @@ -74,6 +74,8 @@ export interface StdioConfig { toolCallTimeoutMs: number /** Fail plugin activation when the initial connection or tool synchronization fails. */ failOnStartupError: boolean + /** Automatic reconnect policy after a lost connection; omission uses the defaults. */ + reconnect?: ReconnectConfig } /** Config for connecting to an MCP server over Streamable HTTP (SSE). */ @@ -94,11 +96,20 @@ export interface StreamableHttpConfig { toolCallTimeoutMs: number /** Fail plugin activation when the initial connection or tool synchronization fails. */ failOnStartupError: boolean + /** Automatic reconnect policy after a lost connection; omission uses the defaults. */ + reconnect?: ReconnectConfig } /** Configuration for one stdio or Streamable HTTP MCP server. */ export type Config = StdioConfig | StreamableHttpConfig +const Reconnect: z = z.object({ + enabled: z.boolean().default(RECONNECT_DEFAULTS.enabled), + initialDelayMs: z.number().min(1).max(MAX_TIMER_DELAY_MS).default(RECONNECT_DEFAULTS.initialDelayMs), + maxDelayMs: z.number().min(1).max(MAX_TIMER_DELAY_MS).default(RECONNECT_DEFAULTS.maxDelayMs), + maxAttempts: z.number().step(1).min(1).max(Number.MAX_SAFE_INTEGER).default(RECONNECT_DEFAULTS.maxAttempts), +}) + export const Config = z.union([ z.object({ transport: z.const('stdio'), @@ -109,6 +120,7 @@ export const Config = z.union([ cwd: z.string().default(''), toolCallTimeoutMs: z.number().default(DEFAULT_TOOL_CALL_TIMEOUT_MS), failOnStartupError: z.boolean().default(false), + reconnect: Reconnect, }), z.object({ transport: z.const('streamable-http'), @@ -117,6 +129,7 @@ export const Config = z.union([ headers: z.dict(String).default({}), toolCallTimeoutMs: z.number().default(DEFAULT_TOOL_CALL_TIMEOUT_MS), failOnStartupError: z.boolean().default(false), + reconnect: Reconnect, }), ]) as unknown as z @@ -131,7 +144,12 @@ export const Config = z.union([ * @returns startup readiness after connection and initial tool discovery settle. */ export async function apply(ctx: Context, config: Config): Promise { - // Reserve the namespace first: a duplicate `serverName` fails THIS instance + // Fail loud at load: reconnect misconfiguration (including programmatic + // construction that bypassed Schemastery) rejects THIS instance before any + // effect registers. + const reconnect = resolveReconnectPolicy(config.reconnect, `mcp-client(${config.serverName}): reconnect`) + + // Reserve the namespace next: a duplicate `serverName` fails THIS instance // at load with an actionable error and leaves the earlier instance intact. ctx.effect(() => { let names = activeServerNames.get(ctx.root) @@ -148,58 +166,22 @@ export async function apply(ctx: Context, config: Config): Promise { return () => void names.delete(config.serverName) }, 'mcp-client.serverName') - const transport = createTransport(config) - const client = new Client( - { name: 'dsh-mcp-client', version: '0.0.1' }, - { capabilities: {} }, - ) + // The supervisor owns the client/transport generations, the reconnect + // loop, and the live tool registrations; disposal stops reconnection, + // quiesces in-flight work, and unregisters the current generation. + const connection = startConnection(ctx, config, reconnect) - const opts = { - registrationFailure: 'contain' as const, - serverName: config.serverName, - toolCallTimeoutMs: config.toolCallTimeoutMs, - } - - // Connect and set up tools. `ready` always settles to an outcome so rollback - // can close a partially opened client even when strict startup later rejects. - // Its accessor returns the CURRENT disposer generation, so disposal always - // unregisters the live set, not the first one. - const ready = (async () => { - await client.connect(transport) - - let disposers = await syncTools(client, ctx, { - ...opts, - registrationFailure: config.failOnStartupError ? 'throw' : 'contain', - }, new Map()) - - client.setNotificationHandler( - ToolListChangedNotificationSchema, - async () => { - ctx.logger.info(`mcp-client(${config.serverName}): tool list changed, re-syncing`) - try { - disposers = await syncTools(client, ctx, opts, disposers) - } catch (error) { - // Fetch-phase failure: the previous generation is still registered - // and `disposers` still owns it — keep serving the last good list. - ctx.logger.error(`mcp-client(${config.serverName}): tool re-sync failed: ${String(error)}`) - } - }, - ) - - return { getDisposers: () => disposers } - })().catch((error: unknown) => { - ctx.logger.error(`mcp-client(${config.serverName}): startup failed: ${String(error)}`) - return { getDisposers: () => new Map void>(), error } - }) - - ctx.effect(() => async () => { - const outcome = await ready - for (const dispose of outcome.getDisposers().values()) dispose() - try { await client.close() } catch { /* transport already gone */ } + ctx.effect(() => { + return () => connection.dispose() }, 'mcp-client.connection') - const outcome = await ready - if ('error' in outcome && config.failOnStartupError) { + // Block plugin activation on the initial connection + tool discovery so + // Cordis consumers observe the tools immediately after the fiber activates. + // When failOnStartupError is true, a failed initial attempt rejects the + // fiber (Cordis rolls it back); otherwise the error is logged and the + // supervisor enters its reconnect loop. + const outcome = await connection.ready + if (outcome.error !== undefined && config.failOnStartupError) { throw new Error(`mcp-client(${config.serverName}): initial connection or tool synchronization failed`, { cause: outcome.error }) } } diff --git a/packages/mcp/mcp-client/tests/apply.spec.ts b/packages/mcp/mcp-client/tests/apply.spec.ts index e30a1ee716..f2a9982d59 100644 --- a/packages/mcp/mcp-client/tests/apply.spec.ts +++ b/packages/mcp/mcp-client/tests/apply.spec.ts @@ -124,6 +124,33 @@ describe('mcp-client plugin module exports', () => { } as never) expect(resolved.serverName).toBe('github-prod_1') }) + + it('Config schema materializes reconnect defaults and merges partial overrides', () => { + const omitted = ConfigSchema({ + transport: 'stdio', + serverName: 'srv', + command: 'echo', + } as never) + expect(omitted.reconnect).toEqual({ enabled: true, initialDelayMs: 500, maxDelayMs: 30_000, maxAttempts: 10 }) + + const partial = ConfigSchema({ + transport: 'stdio', + serverName: 'srv', + command: 'echo', + reconnect: { initialDelayMs: 100 }, + } as never) + expect(partial.reconnect).toEqual({ enabled: true, initialDelayMs: 100, maxDelayMs: 30_000, maxAttempts: 10 }) + }) + + it('Config schema rejects an invalid reconnect block', () => { + // schemastery unions wrap branch errors, so assert the throw only. + expect(() => ConfigSchema({ + transport: 'stdio', + serverName: 'srv', + command: 'echo', + reconnect: { maxAttempts: 0 }, + } as never)).toThrow() + }) }) describe('apply (plugin lifecycle)', () => { @@ -216,8 +243,8 @@ describe('apply (plugin lifecycle)', () => { expect(mockListTools).not.toHaveBeenCalled() expect(ctx.tools.get('mcp__srv__remote')).toBeUndefined() - // Disposal exercises the empty fallback accessor: nothing to unregister, - // close still attempted, no throw. + // Disposal cancels the scheduled reconnect attempt: nothing to + // unregister, close already attempted by the failed attempt, no throw. await ctx.fiber.dispose() await sleep(50) expect(mockClose).toHaveBeenCalled() diff --git a/packages/mcp/mcp-client/tests/fixture-server.ts b/packages/mcp/mcp-client/tests/fixture-server.ts index 8491b96634..974e2a26b0 100644 --- a/packages/mcp/mcp-client/tests/fixture-server.ts +++ b/packages/mcp/mcp-client/tests/fixture-server.ts @@ -51,6 +51,17 @@ server.registerTool('image', { ], })) +server.registerTool('crash', { + title: 'Crash Tool', + description: 'Replies, then exits the server process (crash-recovery test).', + inputSchema: {}, +}, async () => { + // Exit AFTER the response flushes so the caller observes a clean result + // followed by a transport close, like a real post-reply crash. + setTimeout(() => process.exit(7), 25) + return { content: [{ type: 'text', text: 'crashing' }] } +}) + // Dotted name: legal in MCP, illegal in the DeepSeek function-name contract. // Exercises the bridge's normalize-and-hash public-name path end to end. server.registerTool('admin.reset', { diff --git a/packages/mcp/mcp-client/tests/mcp-client.e2e.ts b/packages/mcp/mcp-client/tests/mcp-client.e2e.ts index e1f51d20e9..7f09b7afae 100644 --- a/packages/mcp/mcp-client/tests/mcp-client.e2e.ts +++ b/packages/mcp/mcp-client/tests/mcp-client.e2e.ts @@ -13,7 +13,7 @@ import { mkdtemp, rm, writeFile, readFile } from 'node:fs/promises' import { tmpdir } from 'node:os' import { join } from 'node:path' import { fileURLToPath } from 'node:url' -import { afterAll, beforeAll, describe, expect, it } from 'vitest' +import { afterAll, beforeAll, describe, expect, it, vi } from 'vitest' import { Context } from 'cordis' import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js' import { StreamableHTTPServerTransport } from '@modelcontextprotocol/sdk/server/streamableHttp.js' @@ -200,6 +200,87 @@ describe('fixture server — disposal', () => { }, 30_000) }) +describe('fixture server — crash recovery', () => { + function crashConfig(serverName: string, reconnect: NonNullable): Config { + return { + transport: 'stdio', + serverName, + command: process.execPath, + args: [fixtureServerPath], + env: {}, + cwd: packageDir, + toolCallTimeoutMs: 15_000, + failOnStartupError: false, + reconnect, + } + } + + it('auto-reconnects after a stdio crash and serves tool calls again', async () => { + const ctx = await mountRegistry() + await apply(ctx, crashConfig('crashy', { initialDelayMs: 50, maxDelayMs: 500, maxAttempts: 40 })) + + const before = await ctx.tools.execute({ + signal: testToolSignal, + callId: nextCallId(), name: 'mcp__crashy__add', arguments: { a: 2, b: 3 }, + }) + expect(textOf(before.content[0])).toBe('5') + + // The crash tool replies, then kills the real child process. + const crash = await ctx.tools.execute({ + signal: testToolSignal, + callId: nextCallId(), name: 'mcp__crashy__crash', arguments: {}, + }) + expect(crash.isError).toBe(false) + + // Recovery is proven by the world: a post-crash call round-trips through + // the respawned server process. + await vi.waitFor(async () => { + const after = await ctx.tools.execute({ + signal: testToolSignal, + callId: nextCallId(), name: 'mcp__crashy__add', arguments: { a: 20, b: 22 }, + }) + expect(after.isError).toBe(false) + expect(textOf(after.content[0])).toBe('42') + }, { timeout: 15_000, interval: 250 }) + + // The recovered generation replaced the dead one: no duplicates, no leak. + const addEntries = ctx.tools.schemas().map(s => s.name).filter(name => name === 'mcp__crashy__add') + expect(addEntries).toHaveLength(1) + + await ctx.fiber.dispose() + await sleep(200) + }, 30_000) + + it('plugin unload during an outage stops reconnection and unregisters tools', async () => { + const ctx = await mountRegistry() + const fiber = ctx.plugin( + { name: 'mcp-client', inject: ['tools'], apply }, + crashConfig('ephemeral', { initialDelayMs: 8_000, maxDelayMs: 8_000, maxAttempts: 5 }), + ) + // Cordis awaits async apply() as startup work; wait for it. + await vi.waitFor(() => { expect(ctx.tools.get('mcp__ephemeral__add')).toBeDefined() }, { timeout: 20_000 }) + + const crash = await ctx.tools.execute({ + signal: testToolSignal, + callId: nextCallId(), name: 'mcp__ephemeral__crash', arguments: {}, + }) + expect(crash.isError).toBe(false) + + // Give the transport close a moment to land the supervisor in its 8s + // backoff wait, then unload: disposal must not sit out the backoff. + await sleep(300) + const started = Date.now() + await fiber.dispose() + expect(Date.now() - started).toBeLessThan(4_000) + + expect(ctx.tools.get('mcp__ephemeral__add')).toBeUndefined() + await sleep(200) + expect(ctx.tools.get('mcp__ephemeral__add')).toBeUndefined() + + await ctx.fiber.dispose() + }, 30_000) +}) + // ---- @modelcontextprotocol/server-everything ---- describe('server-everything — official test server', () => { diff --git a/packages/mcp/mcp-client/tests/reconnect.spec.ts b/packages/mcp/mcp-client/tests/reconnect.spec.ts new file mode 100644 index 0000000000..f52dfdb65b --- /dev/null +++ b/packages/mcp/mcp-client/tests/reconnect.spec.ts @@ -0,0 +1,413 @@ +/** + * Tests for the mcp-client connection supervisor: crash-driven reconnection + * with bounded backoff, generation-safe tool re-registration, the failure + * cap, the stability-window budget reset, and disposal stopping reconnection. + * Isolated file so vi.mock of the MCP SDK doesn't pollute other test suites. + */ +import { describe, expect, it, vi, beforeEach } from 'vitest' +import { Context } from 'cordis' +import SystemPrompt from '@deepseek-ai/dsh-system-prompt' +import ToolRegistry from '@deepseek-ai/dsh-tools' +import { CallId } from '@deepseek-ai/dsh-llm' +import type { Config } from '@deepseek-ai/dsh-mcp-client' + +// ---- Mock MCP SDK ---- + +// vi.mock factories are hoisted above every import/const, so the mock fns and +// class must be created inside vi.hoisted to exist when the factories run. +const { mockConnect, mockClose, mockListTools, mockCallTool, mockSetNotificationHandler, MockClient, instances } = vi.hoisted(() => { + const mockConnect = vi.fn<() => Promise>() + const mockClose = vi.fn<() => Promise>() + const mockListTools = vi.fn<(_params?: Record) => Promise>() + const mockCallTool = vi.fn<( + _params?: Record, _compatibilitySchema?: unknown, _options?: unknown, + ) => Promise>() + const mockSetNotificationHandler = vi.fn() + const mockRequest = vi.fn(async ( + request: { method: string; params?: Record }, + _schema: unknown, + options?: unknown, + ): Promise => { + if (request.method === 'tools/list') return await mockListTools(request.params) + if (request.method === 'tools/call') return await mockCallTool(request.params, undefined, options) + throw new Error(`unexpected MCP request: ${request.method}`) + }) + class MockClient { + onclose: (() => void) | undefined + connect = mockConnect + close = mockClose + request = mockRequest + setNotificationHandler = mockSetNotificationHandler + constructor() { instances.push(this) } + } + const instances: MockClient[] = [] + return { mockConnect, mockClose, mockListTools, mockCallTool, mockSetNotificationHandler, MockClient, instances } +}) + +vi.mock('@modelcontextprotocol/sdk/client/index.js', () => ({ + Client: MockClient, +})) + +vi.mock('@modelcontextprotocol/sdk/client/stdio.js', () => ({ + StdioClientTransport: vi.fn(), +})) + +vi.mock('@modelcontextprotocol/sdk/client/streamableHttp.js', () => ({ + StreamableHTTPClientTransport: vi.fn(), +})) + +// vi.mock is hoisted above static imports, so the modules under test see the +// mocked SDK even through a static import. +import { apply } from '@deepseek-ai/dsh-mcp-client/src/index.ts' +import { RECONNECT_DEFAULTS, resolveReconnectPolicy } from '@deepseek-ai/dsh-mcp-client/src/connection.ts' + +// ---- Helpers ---- + +const testToolSignal = new AbortController().signal + +async function mountRegistry(): Promise { + const ctx = new Context() + await ctx.plugin(SystemPrompt) + await ctx.plugin(ToolRegistry) + return ctx +} + +function sleep(ms: number): Promise { + // Annotated binding (not withResolvers()): the tests lint layer runs + // no-invalid-void-type with default options, which rejects the explicit + // type argument in call position but accepts the inferred form. + const gate: PromiseWithResolvers = Promise.withResolvers() + setTimeout(gate.resolve, ms) + return gate.promise +} + +/** Capture the supervisor's logger lines by level on one context. */ +function captureLogs(ctx: Context): { warns: string[]; errors: string[]; infos: string[] } { + const warns: string[] = [] + const errors: string[] = [] + const infos: string[] = [] + ctx.logger.warn = ((message: unknown) => { warns.push(String(message)) }) as typeof ctx.logger.warn + ctx.logger.error = ((message: unknown) => { errors.push(String(message)) }) as typeof ctx.logger.error + ctx.logger.info = ((message: unknown) => { infos.push(String(message)) }) as typeof ctx.logger.info + return { warns, errors, infos } +} + +function stdioConfig(reconnect?: Config['reconnect']): Config { + return { + transport: 'stdio', + serverName: 'srv', + command: 'echo', + args: [], + env: {}, + cwd: '', + toolCallTimeoutMs: 60_000, + failOnStartupError: false, + ...reconnect === undefined ? {} : { reconnect }, + } +} + +/** The tool list the mock server advertises after a successful (re)connect. */ +function listing(...names: string[]): { tools: { name: string; inputSchema: { type: string } }[]; nextCursor: undefined } { + return { + tools: names.map(name => ({ name, inputSchema: { type: 'object' } })), + nextCursor: undefined, + } +} + +let callSeq = 0 +function nextCallId(): CallId { + return CallId(`reconnect-${++callSeq}`) +} + +// ---- Tests ---- + +describe('reconnect supervisor', () => { + let ctx: Context + + beforeEach(async () => { + vi.clearAllMocks() + instances.length = 0 + mockConnect.mockResolvedValue(undefined) + mockClose.mockResolvedValue(undefined) + mockListTools.mockResolvedValue(listing('remote')) + mockCallTool.mockResolvedValue({ content: [{ type: 'text', text: 'ok' }] }) + ctx = await mountRegistry() + }) + + it('reconnects after a transport close, re-syncs tools through the new generation, and serves calls', async () => { + const { warns, infos } = captureLogs(ctx) + await apply(ctx, stdioConfig({ initialDelayMs: 5, maxDelayMs: 40, maxAttempts: 5 })) + await vi.waitFor(() => { expect(ctx.tools.get('mcp__srv__remote')).toBeDefined() }) + expect(instances).toHaveLength(1) + + // The recovered server advertises a different list: the swap must neither + // duplicate nor leak the pre-crash generation. + mockListTools.mockResolvedValue(listing('revived')) + instances[0]!.onclose?.() + + await vi.waitFor(() => { expect(ctx.tools.get('mcp__srv__revived')).toBeDefined() }) + expect(ctx.tools.get('mcp__srv__remote')).toBeUndefined() + expect(instances).toHaveLength(2) + expect(mockConnect).toHaveBeenCalledTimes(2) + + // Post-recovery calls execute through the re-registered definition. + const result = await ctx.tools.execute({ + signal: testToolSignal, + callId: nextCallId(), name: 'mcp__srv__revived', arguments: {}, + }) + expect(result.isError).toBe(false) + + // User-visible state: reconnecting and recovered are distinct lines. + expect(warns.some(line => line.includes('reconnecting in 5ms (attempt 1/5)'))).toBe(true) + expect(infos.some(line => line.includes('reconnected and re-synced tools'))).toBe(true) + + // A late close signal from the replaced generation is ignored. + instances[0]!.onclose?.() + await sleep(30) + expect(instances).toHaveLength(2) + }) + + it('stops at the failure cap, unregisters the tools, and reports final failure', async () => { + const { warns, errors } = captureLogs(ctx) + await apply(ctx, stdioConfig({ initialDelayMs: 2, maxDelayMs: 8, maxAttempts: 2 })) + await vi.waitFor(() => { expect(ctx.tools.get('mcp__srv__remote')).toBeDefined() }) + + mockConnect.mockRejectedValue(new Error('server gone')) + // A failing close on the failed attempt's cleanup must not break the loop. + mockClose.mockRejectedValue(new Error('already closed')) + instances[0]!.onclose?.() + + await vi.waitFor(() => { + expect(errors.some(line => line.includes('giving up after 2 consecutive failed reconnect attempts'))).toBe(true) + }) + // Stale tools do not leak past final failure. + expect(ctx.tools.get('mcp__srv__remote')).toBeUndefined() + // Initial connect + exactly maxAttempts reconnect attempts. + expect(mockConnect).toHaveBeenCalledTimes(3) + expect(warns.some(line => line.includes('connection attempt failed: Error: server gone'))).toBe(true) + await sleep(30) + expect(mockConnect).toHaveBeenCalledTimes(3) + }) + + it('dispose during the backoff wait cancels the pending reconnect', async () => { + await apply(ctx, stdioConfig({ initialDelayMs: 60_000, maxDelayMs: 60_000, maxAttempts: 5 })) + await vi.waitFor(() => { expect(ctx.tools.get('mcp__srv__remote')).toBeDefined() }) + + instances[0]!.onclose?.() + // Now waiting out a 60s backoff; disposal must return promptly anyway. + await ctx.fiber.dispose() + await sleep(30) + expect(mockConnect).toHaveBeenCalledTimes(1) + expect(instances).toHaveLength(1) + }) + + it('a transport close after dispose schedules nothing', async () => { + const fiber = ctx.plugin({ name: 'mcp-client', inject: ['tools'], apply }, stdioConfig()) + await vi.waitFor(() => { expect(ctx.tools.get('mcp__srv__remote')).toBeDefined() }) + + await fiber.dispose() + expect(ctx.tools.get('mcp__srv__remote')).toBeUndefined() + + // The disposer's client.close() fires onclose in the real SDK. + instances[0]!.onclose?.() + await sleep(30) + expect(instances).toHaveLength(1) + expect(mockConnect).toHaveBeenCalledTimes(1) + }) + + it('reconnect disabled keeps the registered tools and reports manual recovery', async () => { + const { errors } = captureLogs(ctx) + await apply(ctx, stdioConfig({ enabled: false })) + await vi.waitFor(() => { expect(ctx.tools.get('mcp__srv__remote')).toBeDefined() }) + + instances[0]!.onclose?.() + await sleep(30) + expect(mockConnect).toHaveBeenCalledTimes(1) + // Pre-reconnect contract: the generation stays registered until disposal. + expect(ctx.tools.get('mcp__srv__remote')).toBeDefined() + expect(errors.some(line => line.includes('reconnect is disabled'))).toBe(true) + }) + it('reconnect disabled after a failed initial connect reports no registered tools', async () => { + const { errors } = captureLogs(ctx) + mockConnect.mockRejectedValue(new Error('refused')) + await apply(ctx, stdioConfig({ enabled: false })) + await sleep(30) + expect(ctx.tools.get('mcp__srv__remote')).toBeUndefined() + expect(errors.some(line => line.includes('no tools were registered'))).toBe(true) + }) + + it('an uptime past the stability window resets the attempt budget', async () => { + const { errors } = captureLogs(ctx) + await apply(ctx, stdioConfig({ initialDelayMs: 2, maxDelayMs: 30, maxAttempts: 1 })) + await vi.waitFor(() => { expect(ctx.tools.get('mcp__srv__remote')).toBeDefined() }) + + instances[0]!.onclose?.() + await vi.waitFor(() => { expect(instances).toHaveLength(2) }) + await vi.waitFor(() => { expect(ctx.tools.get('mcp__srv__remote')).toBeDefined() }) + + // Outlive the stability window (= maxDelayMs), then crash again: the + // budget restarts at attempt 1 instead of exceeding maxAttempts. + await sleep(40) + instances[1]!.onclose?.() + await vi.waitFor(() => { expect(instances).toHaveLength(3) }) + await vi.waitFor(() => { expect(ctx.tools.get('mcp__srv__remote')).toBeDefined() }) + expect(errors).toHaveLength(0) + }) + + it('a crash loop with briefly successful connects still exhausts the cap', async () => { + const { errors } = captureLogs(ctx) + await apply(ctx, stdioConfig({ initialDelayMs: 2, maxDelayMs: 10_000, maxAttempts: 1 })) + await vi.waitFor(() => { expect(ctx.tools.get('mcp__srv__remote')).toBeDefined() }) + + // Crash, recover (attempt 1 of 1), crash again well inside the stability + // window: the successful connect must not launder the budget. + instances[0]!.onclose?.() + await vi.waitFor(() => { expect(instances).toHaveLength(2) }) + await vi.waitFor(() => { expect(ctx.tools.get('mcp__srv__remote')).toBeDefined() }) + instances[1]!.onclose?.() + + await vi.waitFor(() => { + expect(errors.some(line => line.includes('giving up after 1 consecutive failed reconnect attempts'))).toBe(true) + }) + expect(instances).toHaveLength(2) + expect(ctx.tools.get('mcp__srv__remote')).toBeUndefined() + }) + + it('a connect rejection racing its own transport close schedules exactly one retry per attempt', async () => { + const { errors } = captureLogs(ctx) + await apply(ctx, stdioConfig({ initialDelayMs: 2, maxDelayMs: 8, maxAttempts: 3 })) + await vi.waitFor(() => { expect(ctx.tools.get('mcp__srv__remote')).toBeDefined() }) + + // Each reconnect attempt sees the stdio transport die (onclose) AND its + // connect() reject — the real SDK emits both for a spawn failure. + mockConnect.mockImplementation(async () => { + instances.at(-1)!.onclose?.() + throw new Error('spawn failed') + }) + instances[0]!.onclose?.() + + await vi.waitFor(() => { + expect(errors.some(line => line.includes('giving up after 3 consecutive failed reconnect attempts'))).toBe(true) + }) + // Initial generation + exactly one generation per budgeted attempt: a + // double-scheduled retry would create more. + expect(instances).toHaveLength(4) + expect(errors.filter(line => line.includes('giving up')).length).toBe(1) + }) + + it('a transport that closes during a resolving connect registers nothing from the dead generation', async () => { + await apply(ctx, stdioConfig({ initialDelayMs: 2, maxDelayMs: 8, maxAttempts: 2 })) + await vi.waitFor(() => { expect(ctx.tools.get('mcp__srv__remote')).toBeDefined() }) + expect(mockListTools).toHaveBeenCalledTimes(1) + + mockConnect.mockImplementation(async () => { + instances.at(-1)!.onclose?.() + }) + instances[0]!.onclose?.() + + await vi.waitFor(() => { expect(ctx.tools.get('mcp__srv__remote')).toBeUndefined() }) + // The dead generations never reached tool discovery. + expect(mockListTools).toHaveBeenCalledTimes(1) + }) + + it('dispose during an in-flight initial sync quiesces without leaking tools', async () => { + const fiber = ctx.plugin({ name: 'mcp-client', inject: ['tools'], apply }, stdioConfig({ initialDelayMs: 2, maxDelayMs: 8, maxAttempts: 5 })) + await vi.waitFor(() => { expect(ctx.tools.get('mcp__srv__remote')).toBeDefined() }) + + // Block the reconnect attempt's tool discovery until after dispose starts. + const gate: PromiseWithResolvers = Promise.withResolvers() + mockListTools.mockImplementation(() => gate.promise) + instances[0]!.onclose?.() + await vi.waitFor(() => { expect(mockListTools).toHaveBeenCalledTimes(2) }) + + const disposing = fiber.dispose() + await sleep(10) + gate.resolve(listing('late')) + await disposing + + // The late sync's swap ran, then disposal unregistered its result: no + // generation survives the plugin. + expect(ctx.tools.get('mcp__srv__remote')).toBeUndefined() + expect(ctx.tools.get('mcp__srv__late')).toBeUndefined() + }) + + it('a re-sync failing because dispose closed the transport stays silent', async () => { + const { errors } = captureLogs(ctx) + const fiber = ctx.plugin({ name: 'mcp-client', inject: ['tools'], apply }, stdioConfig()) + await vi.waitFor(() => { expect(ctx.tools.get('mcp__srv__remote')).toBeDefined() }) + + const gate: PromiseWithResolvers = Promise.withResolvers() + mockListTools.mockImplementation(() => gate.promise) + const handler = mockSetNotificationHandler.mock.calls[0]![1] as () => Promise + const resync = handler() + await vi.waitFor(() => { expect(mockListTools).toHaveBeenCalledTimes(2) }) + + const disposing = fiber.dispose() + await sleep(10) + gate.reject(new Error('Connection closed')) + await disposing + await resync + + expect(errors.some(line => line.includes('tool re-sync failed'))).toBe(false) + }) + + it('a stale notification handler from a replaced generation is ignored', async () => { + await apply(ctx, stdioConfig({ initialDelayMs: 2, maxDelayMs: 8, maxAttempts: 5 })) + await vi.waitFor(() => { expect(ctx.tools.get('mcp__srv__remote')).toBeDefined() }) + + instances[0]!.onclose?.() + await vi.waitFor(() => { expect(instances).toHaveLength(2) }) + await vi.waitFor(() => { expect(ctx.tools.get('mcp__srv__remote')).toBeDefined() }) + const listCalls = mockListTools.mock.calls.length + + const staleHandler = mockSetNotificationHandler.mock.calls[0]![1] as () => Promise + await staleHandler() + expect(mockListTools).toHaveBeenCalledTimes(listCalls) + }) +}) + +// ---- Policy resolution ---- + +describe('resolveReconnectPolicy', () => { + const path = 'mcp-client(srv): reconnect' + + it('resolves omission to the defaults, frozen', () => { + const policy = resolveReconnectPolicy(undefined, path) + expect(policy).toEqual(RECONNECT_DEFAULTS) + expect(Object.isFrozen(policy)).toBe(true) + }) + + it('keeps explicit values', () => { + expect(resolveReconnectPolicy( + { enabled: false, initialDelayMs: 1, maxDelayMs: 2, maxAttempts: 7 }, + path, + )).toEqual({ enabled: false, initialDelayMs: 1, maxDelayMs: 2, maxAttempts: 7 }) + }) + + it('rejects unknown keys', () => { + expect(() => resolveReconnectPolicy({ jitterRatio: 0.5 } as never, path)) + .toThrow(/reconnect\.jitterRatio is not a reconnect option/) + }) + + it('rejects out-of-range delays', () => { + expect(() => resolveReconnectPolicy({ initialDelayMs: 0 }, path)).toThrow(/initialDelayMs must be a positive finite number/) + expect(() => resolveReconnectPolicy({ initialDelayMs: Number.POSITIVE_INFINITY }, path)).toThrow(/initialDelayMs/) + expect(() => resolveReconnectPolicy({ maxDelayMs: -1 }, path)).toThrow(/maxDelayMs must be a positive finite number/) + }) + + it('rejects an initial delay above the ceiling', () => { + expect(() => resolveReconnectPolicy({ initialDelayMs: 100, maxDelayMs: 5 }, path)) + .toThrow(/initialDelayMs must be less than or equal to maxDelayMs/) + }) + + it('rejects non-positive-integer attempt caps', () => { + expect(() => resolveReconnectPolicy({ maxAttempts: 0 }, path)).toThrow(/maxAttempts must be a positive integer/) + expect(() => resolveReconnectPolicy({ maxAttempts: 1.5 }, path)).toThrow(/maxAttempts must be a positive integer/) + }) + + it('apply fails loud at load on a misconfigured reconnect', async () => { + const ctx = await mountRegistry() + await expect(apply(ctx, stdioConfig({ initialDelayMs: 100, maxDelayMs: 5 }))) + .rejects.toThrow(/initialDelayMs must be less than or equal to maxDelayMs/) + }) +}) diff --git a/packages/mcp/mcp-client/tsconfig.json b/packages/mcp/mcp-client/tsconfig.json index 461b250297..f42f4d4c2c 100644 --- a/packages/mcp/mcp-client/tsconfig.json +++ b/packages/mcp/mcp-client/tsconfig.json @@ -26,6 +26,9 @@ }, { "path": "../../support/invariants" + }, + { + "path": "../../util/timeout" } ] } diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 4912b264d7..abf6a6695b 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -4901,6 +4901,9 @@ importers: '@deepseek-ai/dsh-subprocess': specifier: workspace:^ version: link:../../subprocess/subprocess + '@deepseek-ai/dsh-timeout': + specifier: workspace:^ + version: link:../../util/timeout '@deepseek-ai/dsh-tools': specifier: workspace:^ version: link:../../core/tools