diff --git a/.agents/notes/implemented/bug-fix/2026-08-03-cli-signal-shutdown-escalation.i18n.yaml b/.agents/notes/implemented/bug-fix/2026-08-03-cli-signal-shutdown-escalation.i18n.yaml new file mode 100644 index 0000000000..3f2d61302c --- /dev/null +++ b/.agents/notes/implemented/bug-fix/2026-08-03-cli-signal-shutdown-escalation.i18n.yaml @@ -0,0 +1,6 @@ +# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each +# side as of the last confirmed-consistent state. Both languages carry equal authority; +# after editing either side, bring the other along and re-record with: +# pnpm run verify-translation-pairing --write .agents/notes/implemented/bug-fix/2026-08-03-cli-signal-shutdown-escalation.md +2026-08-03-cli-signal-shutdown-escalation.md: 62abe7a8e6887345232bd5c429a9abc61a424b5d +2026-08-03-cli-signal-shutdown-escalation.zh.md: b8ccf94dc14a9363d8e373ebd13c4a89cd03e23c diff --git a/.agents/notes/implemented/bug-fix/2026-08-03-cli-signal-shutdown-escalation.md b/.agents/notes/implemented/bug-fix/2026-08-03-cli-signal-shutdown-escalation.md new file mode 100644 index 0000000000..62abe7a8e6 --- /dev/null +++ b/.agents/notes/implemented/bug-fix/2026-08-03-cli-signal-shutdown-escalation.md @@ -0,0 +1,52 @@ +# Agent Note: Bounded, escalating signal shutdown for Web and headless + +Status: implemented + +English | [中文](2026-08-03-cli-signal-shutdown-escalation.zh.md) + +## Problem + +The default telemetry mount added SIGINT/SIGTERM handlers to `dsh web` and `dsh -p` so process exit could drain the Cordis tree instead of dropping queued telemetry. Each handler used a one-way boolean latch and exited only after `ctx.fiber.dispose()` settled. Headless normal completion also awaited that disposal without a bound. + +A user then reproduced `dsh -p` hanging immediately after the observation URL and ignoring repeated `Ctrl+C`; `DSH_TELEMETRY_DISABLED=1` removed the hang, while a standalone Node handler in the same Linux sandbox received SIGINT. This isolated the pending disposer to telemetry rather than terminal signal forwarding. OTel's `BatchLogRecordProcessor.shutdown()` awaits `exporter.forceFlush()` before the `exportTimeoutMillis`-bounded completion promise, and the OTLP exporter's `forceFlush()` waits directly on its in-flight HTTP Promise. A proxy/sandbox connection that never obtains a socket can therefore leave provider shutdown pending despite both configured SDK timeouts. + +The latch then turned that telemetry defect into an unkillable CLI: normal completion was already awaiting the single-shot root disposal; the first SIGINT joined the same pending disposal and set the signal latch; later SIGINTs returned at the latch, so the process had no remaining escape. A signal received before normal completion had the same unbounded wait. Web used the same latch shape. + +Telemetry's own timeouts cannot prove that the whole plugin tree settles. Any current or future disposer can wedge, and the process boundary must preserve both a graceful first attempt and a user-controlled way out. + +## Decision + +The fix has two ownership layers. The OTel backend adds `shutdownTimeoutMillis` (default and shipped value: three seconds) around the SDK provider's complete shutdown Promise. Crossing it rejects into the telemetry coordinator's existing contained-failure path, allowing the Cordis tree to finish disposal; pending records may be lost because OTel exposes no cancellation for the transport Promise. + +Web and headless share `createProcessShutdown`, one process-level controller around root disposal: + +- Normal shutdown calls coalesce onto one disposal and retain the first requested exit code; they never escalate one another. +- The first signal starts the same graceful disposal and a referenced five-second exit backstop. Disposal success or failure exits once; neither can cancel the process exit. +- A signal received while shutdown is pending forces immediate exit with that signal path's code. This includes the first `Ctrl+C` after headless normal completion has already entered disposal, and a second signal after a signal initiated the drain. +- The five-second bound is a process-safety invariant, not a deployment tunable. It matches the existing TUI root-disposal allowance and is long enough for the telemetry deployment's ordinary drain ceiling. + +Headless preserves exit 0 for a completed turn, exit 1 for another turn-end reason or API business error, 130 for SIGINT, and 143 for SIGTERM. Web preserves its existing SIGTERM exit 0 and SIGINT exit 130 behavior. + +This supersedes the [telemetry deployment Note's](../feature/2026-07-31-web-telemetry-default-mount.md) assumption that SDK exporter/processor timeouts bound complete provider shutdown, and its earlier decision to defer a process-level backstop. The backend owns its export loss/latency policy and closes the known SDK `forceFlush()` gap; the launcher owns the outer guarantee that no plugin can trap the process indefinitely. + +## Alternatives considered + +**Bound only the telemetry backend's `shutdown()`.** Insufficient because it protects the known OTel wait but cannot protect the launcher from another plugin's disposer. + +**Restore Node's default immediate signal exit.** Rejected because a healthy first signal should still flush telemetry and release other resources. Immediate exit is the explicit escalation path, not the default. + +**Add only the five-second timeout.** Rejected because a user pressing `Ctrl+C` again is asking to stop waiting now. Swallowing that intent for the rest of the grace period recreates the reported behavior at a shorter duration. + +## Consequences + +A healthy exit still disposes the complete Cordis tree. The known telemetry wait releases after at most three seconds; any other wedged exit lasts at most five seconds without further input, and a repeated signal ends it immediately. Forced or deadline-bounded exit can interrupt telemetry export or remaining cleanup, which is intentional only after the graceful contract has failed or the user has explicitly escalated. + +The controller is launcher infrastructure rather than a Cordis plugin: it makes no claim that disposal completed, and it does not weaken the lifecycle rule that ordinary disposers must reach quiescence. + +## Testing + +`apps/cli/tests/process-shutdown.spec.ts` pins resolved and rejected disposal, the five-second backstop, normal-call coalescing, a signal interrupting normal disposal, and second-signal escalation. + +`apps/cli/tests/headless-shutdown.e2e.ts` boots the real shipped Web/headless Loader tree in a PTY with a test-only plugin whose disposer announces entry and never settles. The test sends SIGINT after the observation URL, waits for proof that disposal started, sends SIGINT again, and requires exit 130. The source/artifact launch resolver keeps the same regression on both execution planes. This PTY case covers the user-visible process state; no model-output snapshot changes. + +`packages/telemetry/session-telemetry-otel/tests/otel.spec.ts` holds a real OTLP request open after timer export begins and pins that Cordis disposal returns at `shutdownTimeoutMillis`, despite the SDK's `forceFlush()` remaining pending. The collector is then released so the still-observed provider Promise settles cleanly. diff --git a/.agents/notes/implemented/bug-fix/2026-08-03-cli-signal-shutdown-escalation.zh.md b/.agents/notes/implemented/bug-fix/2026-08-03-cli-signal-shutdown-escalation.zh.md new file mode 100644 index 0000000000..b8ccf94dc1 --- /dev/null +++ b/.agents/notes/implemented/bug-fix/2026-08-03-cli-signal-shutdown-escalation.zh.md @@ -0,0 +1,52 @@ +# Agent Note:Web 与 headless 的有界信号关闭和重复信号强制退出 + +状态:已实现 + +[English](2026-08-03-cli-signal-shutdown-escalation.md) | 中文 + +## 问题 + +默认挂载遥测后,`dsh web` 与 `dsh -p` 新增了 SIGINT/SIGTERM 处理器,使进程退出时可以排空 Cordis 插件树,而不是丢弃排队中的遥测数据。每个处理器都使用单向布尔闩锁(latch),并且只有在 `ctx.fiber.dispose()` 结算后才退出。headless 正常完成时同样会无界等待整棵树执行 dispose(资源释放)。 + +随后有用户复现,`dsh -p` 在打印观察 URL 后立即卡死,重复按 `Ctrl+C` 也没有反应;设置 `DSH_TELEMETRY_DISABLED=1` 后不再卡死,而同一 Linux 沙箱中的独立 Node 信号处理器能够收到 SIGINT。这将待结算的 disposer 定位到遥测,而非终端信号转发。OTel 的 `BatchLogRecordProcessor.shutdown()` 会先等待 `exporter.forceFlush()`,再进入受 `exportTimeoutMillis` 限制的完成 promise;OTLP 导出器的 `forceFlush()` 则直接等待正在进行的 HTTP Promise。因此,代理/沙箱连接始终无法取得 socket 时,即使已经配置两项 SDK 超时,也会让提供方关闭一直待结算。 + +闩锁随后把这个遥测缺陷变成无法终止的 CLI(命令行界面):正常完成流程已经在等待单次根级 dispose;第一次 SIGINT 会加入同一个待结算的 dispose,并设置信号闩锁;后续 SIGINT 在闩锁处直接返回,因此进程再无退出途径。正常完成之前收到信号时,同样会陷入无界等待。Web 使用的闩锁结构与此相同。 + +遥测自身的超时无法证明整棵插件树都能结算。任何当前或未来的 disposer 都可能卡死;进程边界既要保留第一次优雅关闭的机会,也必须给用户留下强制退出的途径。 + +## 决策 + +修复分为两层归属。OTel 后端围绕 SDK 提供方的完整关闭 Promise 增加 `shutdownTimeoutMillis`(默认值和交付值均为 3 秒)。超过该截止时间时会 reject,并进入遥测协调器现有的失败隔离路径,使 Cordis 插件树能够完成 dispose;由于 OTel 未公开取消传输 Promise 的能力,待处理记录可能丢失。 + +Web 与 headless 共用 `createProcessShutdown`,它是围绕根级 dispose 建立的进程级控制器: + +- 多次正常关闭调用会汇合到同一次 dispose,并保留首次请求的退出码;这些调用不会相互触发强制退出。 +- 第一个信号会启动同一次优雅 dispose,并设置一个带引用的 5 秒退出兜底。dispose 无论成功或失败都会触发且仅触发一次退出;任何一种结果都无法取消进程退出。 +- 关闭待结算期间收到信号时,会立即按该信号路径的退出码强制退出。这既包括 headless 正常完成已经进入 dispose 后收到的第一次 `Ctrl+C`,也包括由信号启动排空后收到的第二个信号。 +- 5 秒上限是进程安全不变式,而不是部署调节项。它与现有 TUI 根级 dispose 的等待上限一致,也足以覆盖遥测部署的常规排空时限。 + +headless 对完成的轮次仍以 0 退出,对其他轮次结束原因或 API 业务错误仍以 1 退出,对 SIGINT 以 130 退出,对 SIGTERM 以 143 退出。Web 保留现有行为:SIGTERM 以 0 退出,SIGINT 以 130 退出。 + +这项决策取代了[遥测部署 Agent Note](../feature/2026-07-31-web-telemetry-default-mount.md) 中 SDK 导出器/处理器超时能够限制提供方完整关闭流程的假设,也取代了其中暂缓进程级退出兜底的决定。后端负责导出数据丢失与延迟策略,并封住已知的 SDK `forceFlush()` 缺口;启动器负责最外层保证,确保任何插件都无法无限期困住进程。 + +## 考虑过的替代方案 + +**只限制遥测后端的 `shutdown()`。** 仍不充分:它能保护已知的 OTel 等待,但无法保护启动器免受其他插件 disposer 的影响。 + +**恢复 Node 默认的信号即时退出。** 不予采纳:收到第一个信号时,健康流程仍应刷新遥测数据并释放其他资源。即时退出是显式的强制退出路径,而非默认行为。 + +**只增加 5 秒超时。** 不予采纳:用户再次按下 `Ctrl+C`,就是要求立即停止等待。若在剩余宽限期内继续吞掉这一意图,只是缩短了报告中故障的持续时间,并未解决问题。 + +## 后果 + +健康的退出流程仍会对整棵 Cordis 插件树执行 dispose。已知的遥测等待最多会在 3 秒后解除;其他退出流程卡死时,如无进一步输入,最多等待 5 秒,再次收到信号则立即结束进程。强制退出或受截止时间限制的退出可能中断遥测导出或尚未完成的清理工作;只有优雅关闭契约已经失败,或用户明确要求强制退出时,才会有意接受这一结果。 + +该控制器属于启动器基础设施,而不是 Cordis 插件:它不会声称 dispose 已经完成,也不会削弱普通 disposer 必须达到完全停稳状态的生命周期规则。 + +## 测试 + +`apps/cli/tests/process-shutdown.spec.ts` 固定了 dispose 成功与失败、5 秒退出兜底、正常调用汇合、信号中断正常 dispose,以及第二次信号强制退出的行为。 + +`apps/cli/tests/headless-shutdown.e2e.ts` 在 PTY 中启动真实交付的 Web/headless Loader 插件树,并挂载一个仅用于测试的插件;该插件的 disposer 会声明已经进入清理流程,但永不结算。测试在观察地址出现后发送 SIGINT,等待 dispose 已启动的证据,再次发送 SIGINT,并要求进程以 130 退出。源码/产物启动解析器使两个执行平面都覆盖同一项回归。该 PTY 用例覆盖用户可见的进程状态;模型输出快照没有变化。 + +`packages/telemetry/session-telemetry-otel/tests/otel.spec.ts` 在定时器导出开始后保持一条真实 OTLP 请求打开,并固定以下行为:即使 SDK 的 `forceFlush()` 仍待结算,Cordis dispose 也会在 `shutdownTimeoutMillis` 到期时返回。随后测试释放 collector,使仍受观察的提供方 Promise 干净结算。 diff --git a/.agents/notes/implemented/feature/2026-07-31-web-telemetry-default-mount.i18n.yaml b/.agents/notes/implemented/feature/2026-07-31-web-telemetry-default-mount.i18n.yaml index dd15a55a21..f8eab0293f 100644 --- a/.agents/notes/implemented/feature/2026-07-31-web-telemetry-default-mount.i18n.yaml +++ b/.agents/notes/implemented/feature/2026-07-31-web-telemetry-default-mount.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write .agents/notes/implemented/feature/2026-07-31-web-telemetry-default-mount.md -2026-07-31-web-telemetry-default-mount.md: e9ec7d0cda37db44e753c9aee572763b7e24ada6 -2026-07-31-web-telemetry-default-mount.zh.md: 68b411d0668772ce81d7f323c2d286714a223ca4 +2026-07-31-web-telemetry-default-mount.md: 9e7a28d52742535f86d22ba4892b2298575eae99 +2026-07-31-web-telemetry-default-mount.zh.md: 20f2262ad36a6b5c8d1325526a3e6d572af12172 diff --git a/.agents/notes/implemented/feature/2026-07-31-web-telemetry-default-mount.md b/.agents/notes/implemented/feature/2026-07-31-web-telemetry-default-mount.md index e9ec7d0cda..9e7a28d527 100644 --- a/.agents/notes/implemented/feature/2026-07-31-web-telemetry-default-mount.md +++ b/.agents/notes/implemented/feature/2026-07-31-web-telemetry-default-mount.md @@ -10,7 +10,7 @@ The telemetry seam and OTel backend ([revival Note](2026-07-23-session-telemetry ## Decision -The shared `dsh` core (`apps/cli/config/base.cordis.yml`) mounts the `telemetry-otel` row by default with a baked-in production endpoint, so every surface — TUI, web, and headless — reports; this is the **internal-testing deployment stance** — reporting is on when an endpoint exists, and users opt out through the environment. Each surface's exit path drains the queue: web/headless dispose on SIGINT/SIGTERM (headless gained those handlers in this change), and the TUI's normal exit runs `disposeRootAndExit` (root dispose, 5s bounded — above the ~1s drain ceiling configured here) while its `/resume` handoff disposes the root before `execve`. +The shared `dsh` core (`apps/cli/config/base.cordis.yml`) mounts the `telemetry-otel` row by default with a baked-in production endpoint, so every surface — TUI, web, and headless — reports; this is the **internal-testing deployment stance** — reporting is on when an endpoint exists, and users opt out through the environment. Each surface's exit path drains the queue: web/headless use the [bounded, escalating process-shutdown controller](../bug-fix/2026-08-03-cli-signal-shutdown-escalation.md) on SIGINT/SIGTERM, and the TUI's normal exit runs `disposeRootAndExit` (root dispose, 5s bounded — above the backend's 3s shutdown deadline) while its `/resume` handoff disposes the root before `execve`. | Ruling | Value | Rationale | |---|---|---| @@ -18,7 +18,7 @@ The shared `dsh` core (`apps/cli/config/base.cordis.yml`) mounts the `telemetry- | Endpoint | `DSH_TELEMETRY_OTLP_URL`, default `https://harness-telemetry.deepseeksvc.com/v1/logs` | Internal collector; the env override serves local/dev runs | | Opt-out switch | any non-empty `DSH_TELEMETRY_DISABLED` (including `0`/`false`) disables | A privacy switch prefers off-by-mistake over on-by-mistake; a row can only be disabled at AppCLIEntry's patch layer (config has no disable semantic, and the switch must precede the load-time `exporter.url` validation) | | Cadence | `processor.scheduledDelayMillis: 10000` (10s/batch) | Streaming while the session runs, never exit-time-only; a crash loses at most the last unexported interval | -| Exit-drain bound | `exporter.timeoutMillis: 1000` + `maxExportBatchSize: 2048` (== maxQueueSize) + `exportTimeoutMillis: 1500` | Dispose must release within ~1s against an unreachable collector: timeoutMillis doubles as the per-attempt socket timeout and the retry deadline (1s effectively disables the SDK's 5-try backoff), and aligning batch size with the queue cap makes the drain a single batch; SDK defaults can stall 40s+ | +| Exit-drain bound | `exporter.timeoutMillis: 1000` + `maxExportBatchSize: 2048` (== maxQueueSize) + `exportTimeoutMillis: 1500` + `shutdownTimeoutMillis: 3000` | Ordinary unreachable-collector failure releases in ~1s: timeoutMillis is the per-attempt socket timeout and retry deadline, while one queue-sized batch avoids sequential drain multiplication. The DSH-owned 3s outer bound covers the SDK's preceding unbounded `forceFlush()` wait when the transport Promise never obtains a socket. | | Compression | `compression: gzip` | Event bodies carry full content; cross-datacenter bandwidth | | CI isolation | top-level `env: DSH_TELEMETRY_DISABLED: '1'` in all 8 GitHub workflows | Every CI channel that boots the web composition (e2e/snapshot/built smokes) must not stream test sessions to the production endpoint | @@ -30,7 +30,7 @@ The keyless integration test `apps/cli/tests/telemetry-web.e2e.ts` pins the depl **A config field instead of an env patch for the switch.** Infeasible: cordis rows have no config-level disable semantic, and `exporter.url` validation fails loud at plugin construction, so the switch must take effect before the Loader — AppCLIEntry's patch layer is the only seat. -**A `Promise.race` timeout backstop around exit.** Deferred: the parameter set already bounds the worst-case drain to ~1.5-3s (typically <100ms), measured SIGINT-to-exit 110ms-1.1s; the unbounded drip-feed-response risk stays under observation, and on real evidence the race lands inside the backend's `shutdown()` (never the coordinator — that would decide loss semantics for every backend). +**A `Promise.race` timeout backstop around exit.** Originally deferred because the SDK parameters appeared to bound the backend's drain to ~1.5-3s (typically <100ms), with measured SIGINT-to-exit of 110ms-1.1s. A Linux sandbox reproduction later proved that `BatchLogRecordProcessor.shutdown()` can wait forever in `exporter.forceFlush()` before reaching its `exportTimeoutMillis`-bounded completion Promise. The [CLI shutdown fix](../bug-fix/2026-08-03-cli-signal-shutdown-escalation.md) therefore adds both a three-second backend bound for that specific gap and a five-second process-level bound plus repeated-signal escape for the whole plugin tree. ## Consequences diff --git a/.agents/notes/implemented/feature/2026-07-31-web-telemetry-default-mount.zh.md b/.agents/notes/implemented/feature/2026-07-31-web-telemetry-default-mount.zh.md index 68b411d066..20f2262ad3 100644 --- a/.agents/notes/implemented/feature/2026-07-31-web-telemetry-default-mount.zh.md +++ b/.agents/notes/implemented/feature/2026-07-31-web-telemetry-default-mount.zh.md @@ -10,7 +10,7 @@ Status: implemented ## Decision -`dsh` 共享核心(`apps/cli/config/base.cordis.yml`)默认挂载 `telemetry-otel` 行,内置生产 endpoint,因此所有 surface——TUI、web、headless——都上报;这是**内部测试期的部署立场**——有 endpoint 就报,用户可经环境变量退出。各 surface 的退出路径都会排空队列:web/headless 在 SIGINT/SIGTERM 上 dispose(headless 的信号处理是本次补上的),TUI 的正常退出走 `disposeRootAndExit`(根 dispose,5s 兜底——高于此处配置的 ~1s drain 上界),其 `/resume` 移交也在 `execve` 前 dispose 根。 +`dsh` 共享核心(`apps/cli/config/base.cordis.yml`)默认挂载 `telemetry-otel` 行,内置生产 endpoint,因此所有 surface——TUI、web、headless——都上报;这是**内部测试期的部署立场**——有 endpoint 就报,用户可经环境变量退出。各 surface 的退出路径都会排空队列:web/headless 在 SIGINT/SIGTERM 时使用[有界、可升级的进程关闭控制器](../bug-fix/2026-08-03-cli-signal-shutdown-escalation.md),TUI 的正常退出走 `disposeRootAndExit`(根 dispose,5s 兜底——高于后端的 3s 关闭截止时间),其 `/resume` 移交也在 `execve` 前 dispose 根。 | 决策项 | 取值 | 理由 | |---|---|---| @@ -18,7 +18,7 @@ Status: implemented | endpoint | `DSH_TELEMETRY_OTLP_URL`,缺省 `https://harness-telemetry.deepseeksvc.com/v1/logs` | 内部 collector;env 覆盖供本地/联调 | | 退出开关 | `DSH_TELEMETRY_DISABLED` 非空(含 `0`/`false`)即关 | 隐私向开关取「宁关勿误开」;行级 disable 只能在 AppCLIEntry 的 patch 层做(config 无 disable 语义,且必须先于 `exporter.url` 的加载期校验生效) | | 上报节奏 | `processor.scheduledDelayMillis: 10000`(10s/批) | 流式回流,非退出才报;崩溃至多丢最后一个未导出间隔 | -| 退出 drain 上界 | `exporter.timeoutMillis: 1000` + `maxExportBatchSize: 2048(== maxQueueSize)` + `exportTimeoutMillis: 1500` | collector 不可达时 dispose 必须 ~1s 内放行:timeoutMillis 同时是单次 socket 超时与重试 deadline(1s 等效关掉 SDK 5 次 backoff),批大小对齐队列上限使 drain 恒为单批;默认参数下最坏可卡 40s+ | +| 退出 drain 上界 | `exporter.timeoutMillis: 1000` + `maxExportBatchSize: 2048(== maxQueueSize)` + `exportTimeoutMillis: 1500` + `shutdownTimeoutMillis: 3000` | collector 不可达的常规故障会在约 1s 内放行:timeoutMillis 是单次 socket 超时与重试 deadline,使用与队列等大的单批可避免依次排空导致耗时倍增。由 DSH 管理的 3s 外层上限覆盖 SDK 先执行的无界 `forceFlush()` 等待,即传输 Promise 始终无法取得 socket 的情况。 | | 压缩 | `compression: gzip` | 事件 body 含全文,跨机房带宽 | | CI 隔离 | 全部 8 个 GitHub workflow 顶层 `env: DSH_TELEMETRY_DISABLED: '1'` | CI 启动 web 组合的所有通道(e2e/snapshot/built smoke)不得向生产 endpoint 泄测试会话 | @@ -30,7 +30,7 @@ Status: implemented **开关做成 config 字段而非 env patch。** 不可行:cordis 行没有 config 层的 disable 语义,且 `exporter.url` 校验在插件构造期 fail-loud,开关必须在 Loader 之前生效——AppCLIEntry patch 层是唯一落点。 -**退出时 `Promise.race` 兜底超时。** 暂缓:参数组合已把最坏 drain 压到 ~1.5-3s(典型 <100ms),实测 SIGINT→退出 110ms-1.1s;drip-feed 慢滴响应的无界等待风险留观,出现实证再在 backend `shutdown()` 内加 race(不放 coordinator——那会替所有 backend 决定丢失语义)。 +**退出时 `Promise.race` 兜底超时。** 最初暂缓,是因为 SDK 参数看似已经将后端排空耗时限制在约 1.5-3s(通常 <100ms),实测 SIGINT 到退出耗时 110ms-1.1s。后来在 Linux 沙箱中复现并证明,`BatchLogRecordProcessor.shutdown()` 可能在 `exporter.forceFlush()` 中永久等待,无法进入受 `exportTimeoutMillis` 限制的完成 Promise。因此,[CLI 关闭修复](../bug-fix/2026-08-03-cli-signal-shutdown-escalation.md) 既为这一特定缺口增加 3 秒后端上限,也为整棵插件树增加 5 秒进程级上限和重复信号退出途径。 ## Consequences diff --git a/apps/cli/README.i18n.yaml b/apps/cli/README.i18n.yaml index 96a4588f2c..5acc6298e4 100644 --- a/apps/cli/README.i18n.yaml +++ b/apps/cli/README.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write apps/cli/README.md -README.md: 76d9ed65398322cb9244a31661ee59b60c23f793 -README.zh.md: 16a7a4ec52b830e45c32a61a103d87be5941ab3b +README.md: a6ecc4ea804dba2e61a0d91056148d457e30d2ca +README.zh.md: dbe8948d55db92e775712e4a71a1b92f6f7fa351 diff --git a/apps/cli/README.md b/apps/cli/README.md index 76d9ed6539..a6ecc4ea80 100644 --- a/apps/cli/README.md +++ b/apps/cli/README.md @@ -23,6 +23,8 @@ The TUI surface: The Web and headless surfaces boot `base.cordis.yml` plus `web.cordis.yml`, then apply `$DSH_HOME/config.yaml`; an explicit `--config ` replaces that personal overlay. Both surfaces otherwise share the same composition: both tell the coding agent its resolved model and session working directory, treat the invoking directory as the default project and Workspace root, create named Workspaces beneath that root unless `--workspace-root ` overrides it, load applicable `AGENTS.md`/`CLAUDE.md` instructions into each agent-loop request prefix with a 65,536-byte render budget, opt into first-message model titles, use the same bounded transient model-request retry policy as the TUI, and mount a disposable in-memory SQLite content-index service. Web additionally names the DeepSeek Harness Web GUI as the interaction surface, this checkout as its own source location, and the process's canonical local URL and mode in both the prompt and managed `$DSH_WEB_URL`/`$DSH_WEB_MODE`; references such as “this page” therefore identify the GUI without claiming access to implicit DOM, route, or screenshot state. In production mode the host reads rebuilt frontend dist and client bundles on the next request, so refreshing the existing URL updates that GUI without replacing its process. `dsh web --dev` mounts the client-plugin HMR receiver, but no-refresh updates additionally require `pnpm run dev:web` in the same checkout to watch and rebuild plugin bundles; shell and ordinary package changes still require a rebuild and page refresh. Bare `apps/web` Vite serving fails before listening because it cannot inject `window.__DSH_BOOT__`. The index service is ACTIVE at boot, while its `node:sqlite` module and database handle open only on the first content search. This keeps Node 22 startup output free of SQLite's experimental warning before search is used; the first actual search may still emit the runtime warning. Each service instance owns its database, so parallel invocations neither share unsupported SQLite state nor leave derived index files behind, and the first search lazily reconciles live and persisted logs. Headless differs only in listening on an OS-assigned port (parallel `dsh -p` runs never collide; the stderr-printed URL opens the live session in a browser). Both need the frontend dist and client bundles built (`pnpm run build && pnpm run build:web`). +Web and headless process shutdown gives the plugin tree up to five seconds to dispose. The first `SIGINT`/`SIGTERM` starts that graceful drain; a second signal forces immediate exit. If headless normal completion is already stuck in disposal, the first `Ctrl+C` is the escalation and exits immediately instead of being swallowed. + The shared composition defaults new TUI, Web, and headless sessions to the `workspace-write` permission preset (`workspace-write` file mode plus `ask` approval policy). Sandbox-enforced bash and filesystem mutations may write only under the session workspace and platform temporary roots; reads, network access, and process visibility are not confined. The browser answers one-shot approval requests and exposes the Access picker; the TUI exposes `/permission`, but has no approval-request answerer, so an automatic wider retry there fails closed until the user deliberately changes the session preset. `DSH_PERMISSION_MODE` changes the process fallback, while a stored General-settings Permission value applies to later sessions without changing an open one. All three surfaces consume `$DSH_HOME/config.yaml`; the TUI and Web apply valid edits live, while one-shot headless runs read it at startup. The shipped trees include an empty `repository-plugins` row, so a standalone user can add prepared GitHub Plugins without an SDK project or install command: diff --git a/apps/cli/README.zh.md b/apps/cli/README.zh.md index 16a7a4ec52..dbe8948d55 100644 --- a/apps/cli/README.zh.md +++ b/apps/cli/README.zh.md @@ -23,6 +23,8 @@ TUI 界面: Web 和无头界面启动 `base.cordis.yml` 与 `web.cordis.yml`,随后应用 `$DSH_HOME/config.yaml`;显式的 `--config ` 会替代该个人覆盖。除此之外,两者共享同一套组合:两者都会告知编码 agent 所用模型和会话工作目录,将调用目录视为默认项目和 Workspace 根目录,除非通过 `--workspace-root ` 覆盖,否则会在该根目录下创建具名 Workspace;它们会把适用的 `AGENTS.md`/`CLAUDE.md` 指令加载到每个 agent-loop 请求前缀中,渲染预算为 65,536 字节,选用首条消息模型标题,采用与 TUI 相同的有界暂时性模型请求重试策略,并挂载一个可丢弃的内存 SQLite 内容索引服务。Web 还会明确说明交互界面是 DeepSeek Harness Web GUI、当前 checkout 是自身源码位置,并在提示词及受管的 `$DSH_WEB_URL`/`$DSH_WEB_MODE` 中提供该进程的规范本地 URL 和模式;因此,「这个页面」等表述会指向该 GUI,但 agent 不会声称可以访问未显式提供的 DOM、路由或截图状态。在生产模式下,宿主会在下次请求时读取重新构建的前端 dist 和客户端 bundle,因此刷新现有 URL 即可更新该 GUI,无须替换其进程。`dsh web --dev` 会挂载客户端插件的 HMR(热模块替换)接收端,但要实现无刷新更新,还需在同一 checkout 中运行 `pnpm run dev:web`,以监视并重新构建插件 bundle;shell 和普通包(package)的更改仍需重新构建并刷新页面。直接使用裸 `apps/web` Vite 服务会在开始监听前失败,因为它无法注入 `window.__DSH_BOOT__`。索引服务在启动时处于 ACTIVE 状态,但其 `node:sqlite` 模块与数据库句柄分别要到首次内容搜索才会导入和打开。这样可使 Node 22 在尚未使用搜索时的启动输出不出现 SQLite 实验性警告;首次实际搜索仍可能发出运行时警告。每个服务实例独占自己的数据库,因此并行调用既不会共享不受支持的 SQLite 状态,也不会留下派生索引文件,首次搜索还会惰性对账实时日志与持久化日志。无头界面唯一的差异是监听操作系统分配的端口(并行 `dsh -p` 运行绝不冲突;stderr 打印的 URL 会在浏览器中打开实时会话)。两者都需要先构建前端 dist 和客户端 bundle(`pnpm run build && pnpm run build:web`)。 +Web 与无头进程关闭时,最多给插件树 5 秒执行 dispose。第一次 `SIGINT`/`SIGTERM` 会启动这次优雅排空;第二次信号会立即强制退出。如果无头运行正常完成时已经卡在 dispose 中,第一次 `Ctrl+C` 就是强制退出请求,会立即退出,不再被吞掉。 + 共享组合把新建 TUI、Web 和无头会话的权限默认设为 `workspace-write` preset(`workspace-write` 文件模式加 `ask` 审批策略)。由沙箱强制约束的 bash 与文件系统修改只能写入会话工作区和平台临时根目录;读取、网络访问和进程可见性不受该策略约束。浏览器可以应答一次性审批请求,并提供 Access 选择器;TUI 提供 `/permission`,但没有审批请求应答者,因此自动请求更宽权限的重试会以拒绝方式关闭,直到用户主动更改会话 preset。`DSH_PERMISSION_MODE` 会更改进程回退值,而「通用」设置中已存储的「权限」值只适用于之后的会话,不会更改已打开的会话。 三个界面都会使用 `$DSH_HOME/config.yaml`;TUI 和 Web 实时应用有效编辑,而一次性无头运行只在启动时读取。已交付的配置树包含一个空的 `repository-plugins` 配置项,因此独立用户无需 SDK 项目或安装命令,只需配置即可添加已准备的 GitHub 插件: diff --git a/apps/cli/config/base.cordis.yml b/apps/cli/config/base.cordis.yml index b7860e2eaa..63a50daf27 100644 --- a/apps/cli/config/base.cordis.yml +++ b/apps/cli/config/base.cordis.yml @@ -113,18 +113,20 @@ # process out (the launchers patch the row disabled; config cannot disable # a row). Exports carry the harness home's anonymous user id ($DSH_HOME/.userid, # random UUID; delete the file to reset the identity) as the Resource's -# user.id. The exporter/processor values bound the shutdown drain to ~1s -# against an unreachable collector: exporter.timeoutMillis is both the -# per-attempt socket timeout and the retry deadline (1s effectively -# disables the SDK's 5-try backoff), maxExportBatchSize == maxQueueSize -# (both explicit) makes the drain a single batch, and exportTimeoutMillis -# is the processor's own cap on that one export cycle — the second bound -# when the exporter's clock alone does not fire. Every surface's exit path +# user.id. The exporter/processor values normally bound the shutdown drain +# to ~1s against an unreachable collector: exporter.timeoutMillis is both +# the per-attempt socket timeout and the retry deadline (1s effectively +# disables the SDK's 5-try backoff), while maxExportBatchSize == maxQueueSize +# (both explicit) makes the drain a single batch. The SDK awaits +# exporter.forceFlush() outside exportTimeoutMillis, so the backend's 3s +# shutdownTimeoutMillis is the load-bearing outer bound when a transport +# promise never settles. Every surface's exit path # drains it: web/headless dispose on SIGINT/SIGTERM, and the TUI's normal # exit and /resume handoff both dispose the root. - id: telemetry-otel name: '@deepseek-ai/dsh-session-telemetry-otel' config: + shutdownTimeoutMillis: 3000 exporter: url: !!js process.env.DSH_TELEMETRY_OTLP_URL ?? 'https://harness-telemetry.deepseeksvc.com/v1/logs' compression: gzip diff --git a/apps/cli/src/headless.ts b/apps/cli/src/headless.ts index e41bc03c6c..28794e73c4 100644 --- a/apps/cli/src/headless.ts +++ b/apps/cli/src/headless.ts @@ -14,6 +14,7 @@ import type { MuxFrame } from '@deepseek-ai/dsh-host-apiproxy/api' import type { RpcRequest, RpcResponse } from '@deepseek-ai/dsh-host-apiproxy/api/rpc' import type { SessionId } from '@deepseek-ai/dsh-session' import { AppCLIEntry } from './app-cli-entry.ts' +import { createProcessShutdown } from './process-shutdown.ts' /** Outcome of one headless turn: aggregated final text plus the turn-end reason kind. */ interface TurnOutcome { @@ -21,12 +22,12 @@ interface TurnOutcome { reason: string } -/** Unwrap an RpcResponse or fail loud: business errors print and exit 1 (dispose first). */ -async function unwrap(response: RpcResponse, dispose: () => Promise): Promise { +/** Unwrap an RpcResponse or fail loud: business errors print and exit 1 (shutdown first). */ +async function unwrap(response: RpcResponse, shutdown: () => Promise): Promise { if (response.result.ok) return response.result.value const { code, message } = response.result.error process.stderr.write(`dsh: ${code}: ${message}\n`) - await dispose() + await shutdown() process.exit(1) } @@ -82,23 +83,16 @@ export async function runHeadless(task: string): Promise { port: 0, }) const { ctx, port } = await entry.run() - const dispose = async (): Promise => { await ctx.fiber.dispose() } - // Signal exits must still dispose the tree: the composition mounts - // exit-drained plugins (telemetry's queued tail and shutdown marker would - // otherwise be lost), and Node's default signal exit skips disposal. - let signalled = false - const disposeAndExit = (code: number): void => { - if (signalled) return - signalled = true - void dispose().finally(() => { process.exit(code) }) - } - process.on('SIGTERM', () => { disposeAndExit(143) }) - process.on('SIGINT', () => { disposeAndExit(130) }) + // Normal completion and signals share one bounded drain. A signal received + // during that drain escalates immediately instead of becoming a no-op. + const shutdown = createProcessShutdown(async () => { await ctx.fiber.dispose() }) + process.on('SIGTERM', () => { shutdown.interrupt(143) }) + process.on('SIGINT', () => { shutdown.interrupt(130) }) // The headless session is web-observable while it runs (same composition). process.stderr.write(`dsh: observing at http://127.0.0.1:${String(port)}\n`) const api = new InProcessApiClient(toFetchHandler(ctx.apiProxy)) - const created = await unwrap(await api.sessions.create({}), dispose) + const created = await unwrap(await api.sessions.create({}), () => shutdown.shutdown(1)) // Open the stream before prompting so no frame is lost — kept in this order // even though in-process delivery has no race, so the code survives a move @@ -111,11 +105,10 @@ export async function runHeadless(task: string): Promise { sessionId: created.sessionId, mode: 'queue', content: [{ type: 'text', text: task }], - }), dispose) + }), () => shutdown.shutdown(1)) const outcome = await done process.stdout.write(outcome.text + '\n') abort.abort() - await dispose() - process.exit(outcome.reason === 'completed' ? 0 : 1) + await shutdown.shutdown(outcome.reason === 'completed' ? 0 : 1) } diff --git a/apps/cli/src/process-shutdown.ts b/apps/cli/src/process-shutdown.ts new file mode 100644 index 0000000000..1f1d785800 --- /dev/null +++ b/apps/cli/src/process-shutdown.ts @@ -0,0 +1,57 @@ +/** Bounded, escalating process shutdown for the long-lived CLI surfaces. */ + +/** Maximum grace allowed for the application tree to dispose before process exit. */ +export const PROCESS_SHUTDOWN_TIMEOUT_MS = 5_000 + +/** Process-exit controller shared by normal completion and Unix signal handlers. */ +export interface ProcessShutdown { + /** Start or join graceful disposal before exiting with `code`. */ + shutdown(code: number): Promise + /** Start graceful disposal, or force exit when a shutdown is already running. */ + interrupt(code: number): void +} + +/** + * Create one process-exit controller around an application disposer. + * @param dispose - Whole-application teardown that resolves at quiescence. + * @param exit - Process exit boundary, replaceable by tests. + * @param timeoutMs - Grace before forced exit, replaceable by tests. + * @returns A controller whose normal calls coalesce and whose repeated signal call escalates. + */ +export function createProcessShutdown( + dispose: () => Promise, + exit: (code: number) => void = (code) => { process.exit(code) }, + timeoutMs = PROCESS_SHUTDOWN_TIMEOUT_MS, +): ProcessShutdown { + let pending: Promise | undefined + let timeout: ReturnType | undefined + let exited = false + + const exitOnce = (code: number): void => { + if (exited) return + exited = true + if (timeout !== undefined) clearTimeout(timeout) + exit(code) + } + + const shutdown = (code: number): Promise => { + if (pending !== undefined) return pending + timeout = setTimeout(() => { exitOnce(code) }, timeoutMs) + pending = Promise.resolve().then(dispose).then( + () => { exitOnce(code) }, + () => { exitOnce(code) }, + ) + return pending + } + + return { + shutdown, + interrupt(code) { + if (pending !== undefined) { + exitOnce(code) + return + } + void shutdown(code) + }, + } +} diff --git a/apps/cli/src/web.ts b/apps/cli/src/web.ts index 4fbfba4d8d..8f7ab4823a 100644 --- a/apps/cli/src/web.ts +++ b/apps/cli/src/web.ts @@ -13,6 +13,7 @@ import type {} from '@deepseek-ai/dsh-host-webserver' import type {} from '@deepseek-ai/dsh-system-prompt' import type {} from '@deepseek-ai/dsh-tool-bash' import { AppCLIEntry } from './app-cli-entry.ts' +import { createProcessShutdown } from './process-shutdown.ts' // The shared core every `dsh` surface mounts, plus this surface's overlay over it. const BASE_CONFIG = fileURLToPath(new URL('../config/base.cordis.yml', import.meta.url)) @@ -118,17 +119,12 @@ export async function runWeb( const { ctx, port: boundPort } = await entry.run() const resolvedLocalWebUrl = localWebUrl(ctx) - let exiting = false - const shutdown = (code: number): void => { - if (exiting) return - exiting = true - void Promise.resolve(ctx.fiber.dispose()).finally(() => { process.exit(code) }) - } + const shutdown = createProcessShutdown(async () => { await ctx.fiber.dispose() }) // Install shutdown handling before publishing readiness: supervisors may // send a signal as soon as they observe the URL line. - process.on('SIGTERM', () => { shutdown(0) }) - process.on('SIGINT', () => { shutdown(130) }) + process.on('SIGTERM', () => { shutdown.interrupt(0) }) + process.on('SIGINT', () => { shutdown.interrupt(130) }) // The entry's boot-time snapshot, not a fresh sample: the printed LAN URL // must name an address the /api trust fence was configured with. diff --git a/apps/cli/tests/fixtures/never-dispose.mjs b/apps/cli/tests/fixtures/never-dispose.mjs new file mode 100644 index 0000000000..702e1511e4 --- /dev/null +++ b/apps/cli/tests/fixtures/never-dispose.mjs @@ -0,0 +1,14 @@ +/** Test-only Cordis plugin whose disposer announces entry and never settles. */ + +/** + * Register a disposer that keeps process shutdown pending until it is forced. + * @param {import('cordis').Context} ctx - loader-mounted test plugin context. + */ +export function apply(ctx) { + const keepAlive = setInterval(() => {}, 60_000) + ctx.effect(() => async () => { + clearInterval(keepAlive) + process.stderr.write('dsh-test: never-dispose started\n') + await new Promise(() => {}) + }) +} diff --git a/apps/cli/tests/headless-shutdown.e2e.ts b/apps/cli/tests/headless-shutdown.e2e.ts new file mode 100644 index 0000000000..bdb483f1bc --- /dev/null +++ b/apps/cli/tests/headless-shutdown.e2e.ts @@ -0,0 +1,43 @@ +import { mkdir, writeFile } from 'node:fs/promises' +import { join } from 'node:path' +import { fileURLToPath, pathToFileURL } from 'node:url' +import { describe, expect, it } from 'vitest' +import { LOADER_SMOKE_TEST_TIMEOUT_MS } from '@deepseek-ai/dsh-loader-smoke' +import { runTuiPtySmoke } from './pty-harness.ts' + +const dshBinScript = fileURLToPath(new URL('../src/bin.ts', import.meta.url)) +const tsconfigPath = fileURLToPath(new URL('../../../tsconfig.json', import.meta.url)) +const neverDisposePlugin = pathToFileURL( + fileURLToPath(new URL('./fixtures/never-dispose.mjs', import.meta.url)), +).href + +describe.skipIf(process.platform === 'win32')('headless process shutdown (real Loader tree in a PTY)', () => { + it('lets a second Ctrl+C force exit while the first signal is draining', async () => { + const output = await runTuiPtySmoke({ + label: 'dsh headless repeated Ctrl+C', + tempDirPrefix: 'dsh-headless-shutdown-', + binScript: dshBinScript, + tsconfigPath, + configArgs: ['-p', 'never complete'], + env: { DEEPSEEK_API_KEY: 'keyless-shutdown-no-call', DSH_TELEMETRY_DISABLED: '1' }, + expectedExitCode: 130, + timeoutMs: 15_000, + prepare: async (cwd) => { + const home = join(cwd, '.dsh') + await mkdir(home, { recursive: true }) + await writeFile(join(home, 'config.yaml'), [ + '- insert:', + ' - id: never-dispose', + ` name: '${neverDisposePlugin}'`, + '', + ].join('\n')) + }, + actions: [ + { waitFor: 'dsh: observing at ', send: '\u0003' }, + { waitFor: 'dsh-test: never-dispose started', send: '\u0003' }, + ], + }) + expect(output).toContain('dsh: observing at ') + expect(output).toContain('dsh-test: never-dispose started') + }, LOADER_SMOKE_TEST_TIMEOUT_MS) +}) diff --git a/apps/cli/tests/process-shutdown.spec.ts b/apps/cli/tests/process-shutdown.spec.ts new file mode 100644 index 0000000000..5281d59b07 --- /dev/null +++ b/apps/cli/tests/process-shutdown.spec.ts @@ -0,0 +1,102 @@ +import { afterEach, describe, expect, it, vi } from 'vitest' +import { + createProcessShutdown, + PROCESS_SHUTDOWN_TIMEOUT_MS, +} from '../src/process-shutdown.ts' + +function deferred(): { promise: Promise; resolve: () => void; reject: (error: Error) => void } { + let resolve!: () => void + let reject!: (error: Error) => void + const promise = new Promise((accept, fail) => { + resolve = accept + reject = fail + }) + return { promise, resolve, reject } +} + +afterEach(() => { vi.useRealTimers() }) + +describe('process shutdown', () => { + it('exits once after graceful disposal resolves or rejects', async () => { + const resolvedExit = vi.fn() + const resolved = createProcessShutdown(() => Promise.resolve(), resolvedExit) + await resolved.shutdown(0) + expect(resolvedExit).toHaveBeenCalledOnce() + expect(resolvedExit).toHaveBeenCalledWith(0) + + const rejectedExit = vi.fn() + const rejected = createProcessShutdown(() => Promise.reject(new Error('dispose failed')), rejectedExit) + await rejected.shutdown(1) + expect(rejectedExit).toHaveBeenCalledOnce() + expect(rejectedExit).toHaveBeenCalledWith(1) + }) + + it('forces exit when graceful disposal reaches its bound', async () => { + vi.useFakeTimers() + const disposal = deferred() + const exit = vi.fn() + const shutdown = createProcessShutdown(() => disposal.promise, exit) + const pending = shutdown.shutdown(0) + + await vi.advanceTimersByTimeAsync(PROCESS_SHUTDOWN_TIMEOUT_MS - 1) + expect(exit).not.toHaveBeenCalled() + await vi.advanceTimersByTimeAsync(1) + expect(exit).toHaveBeenCalledOnce() + expect(exit).toHaveBeenCalledWith(0) + + disposal.resolve() + await pending + expect(exit).toHaveBeenCalledOnce() + }) + + it('lets Ctrl+C force a normal shutdown already stuck in disposal', async () => { + const disposal = deferred() + const exit = vi.fn() + const shutdown = createProcessShutdown(() => disposal.promise, exit) + const pending = shutdown.shutdown(0) + + shutdown.interrupt(130) + expect(exit).toHaveBeenCalledOnce() + expect(exit).toHaveBeenCalledWith(130) + + disposal.resolve() + await pending + expect(exit).toHaveBeenCalledOnce() + }) + + it('drains on the first signal and forces on the second signal', async () => { + const disposal = deferred() + const dispose = vi.fn(() => disposal.promise) + const exit = vi.fn() + const shutdown = createProcessShutdown(dispose, exit) + + shutdown.interrupt(143) + await Promise.resolve() + expect(dispose).toHaveBeenCalledOnce() + expect(exit).not.toHaveBeenCalled() + + shutdown.interrupt(130) + expect(exit).toHaveBeenCalledOnce() + expect(exit).toHaveBeenCalledWith(130) + + disposal.resolve() + await shutdown.shutdown(0) + expect(exit).toHaveBeenCalledOnce() + }) + + it('coalesces normal shutdown calls without treating them as escalation', async () => { + const disposal = deferred() + const exit = vi.fn() + const shutdown = createProcessShutdown(() => disposal.promise, exit) + + const first = shutdown.shutdown(0) + const second = shutdown.shutdown(1) + expect(second).toBe(first) + expect(exit).not.toHaveBeenCalled() + + disposal.resolve() + await first + expect(exit).toHaveBeenCalledOnce() + expect(exit).toHaveBeenCalledWith(0) + }) +}) diff --git a/docs/config-catalog.md b/docs/config-catalog.md index 617cfb82be..0d912e128d 100644 --- a/docs/config-catalog.md +++ b/docs/config-catalog.md @@ -1219,10 +1219,9 @@ Requires: `sessions` ```ts config-catalog /** - * Plugin configuration: two verbatim SDK option shapes plus nothing else. - * `exporter.url` is the one field this package validates itself — required, - * no default, must parse as an `http(s)` URL — because a missing endpoint - * must fail at plugin load, not at first export. + * Plugin configuration: two verbatim SDK option shapes plus one DSH-owned + * shutdown bound. The package validates its endpoint and shutdown deadline + * because both must fail at plugin load rather than at first export or exit. */ export interface Config { /** @@ -1240,6 +1239,8 @@ export interface Config { * which this plugin fills); the SDK owns and documents these knobs. */ processor?: Omit + /** Maximum time spent awaiting the SDK provider's complete shutdown path. */ + shutdownTimeoutMillis?: number } ``` diff --git a/packages/telemetry/session-telemetry-otel/README.i18n.yaml b/packages/telemetry/session-telemetry-otel/README.i18n.yaml index db61f28a67..fcd57049fc 100644 --- a/packages/telemetry/session-telemetry-otel/README.i18n.yaml +++ b/packages/telemetry/session-telemetry-otel/README.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write packages/telemetry/session-telemetry-otel/README.md -README.md: ad4a97868c28dc3873c839490aa506271459e249 -README.zh.md: f1ad73ddf66aacc30a9024a9290df2c44686efe9 +README.md: 3abd97187cafee132823c02a0b0d103a86bda7db +README.zh.md: 223e6a663933da81032a1fbbb4211555c4bcc159 diff --git a/packages/telemetry/session-telemetry-otel/README.md b/packages/telemetry/session-telemetry-otel/README.md index ad4a97868c..3abd97187c 100644 --- a/packages/telemetry/session-telemetry-otel/README.md +++ b/packages/telemetry/session-telemetry-otel/README.md @@ -10,6 +10,7 @@ The OpenTelemetry backend for [the telemetry seam](../session-telemetry/) — th - id: telemetry-otel name: '@deepseek-ai/dsh-session-telemetry-otel' config: + shutdownTimeoutMillis: 3000 # optional; defaults to 3000 exporter: # passed verbatim to the SDK's OTLP/HTTP log exporter url: https://collector.example.com/v1/logs headers: @@ -17,7 +18,7 @@ The OpenTelemetry backend for [the telemetry seam](../session-telemetry/) — th processor: {} # optional; passed verbatim to BatchLogRecordProcessor ``` -`exporter.url` is the one field this package validates itself — required, no default, must parse as `http(s)` — so a missing endpoint fails at plugin load (as does a non-positive-integer `processor.maxExportBatchSize`, which the SDK accepts but then hangs on at shutdown). Everything else is the SDK's option shape, owned and documented by the SDK, and both blocks pass through whole: every `OTLPExporterNodeConfigBase` field (`headers`, `timeoutMillis`, `compression`, `keepAlive`, …) reaches the exporter, and batching, export cadence (`scheduledDelayMillis`), retry, queue bounds, and loss policy under sustained failure are the SDK's documented behavior, tuned through the `processor` passthrough. The backend deliberately implements no `flush()`: the batch processor is the only flusher in the process, which is what makes `shutdown()`'s drain complete. Removing this block from `cordis.yml` is the opt-out: no residual state, no `enabled` flag. +`exporter.url` is required, has no default, and must parse as `http(s)`; `shutdownTimeoutMillis` is a positive finite DSH-owned outer deadline and defaults to 3000 ms; a non-positive-integer `processor.maxExportBatchSize` also fails at plugin load because the SDK accepts it but then hangs on shutdown. Both SDK blocks pass through whole: every `OTLPExporterNodeConfigBase` field (`headers`, `timeoutMillis`, `compression`, `keepAlive`, …) reaches the exporter, and batching, export cadence (`scheduledDelayMillis`), retry, queue bounds, and loss policy under sustained failure are SDK behavior tuned through `processor`. The backend implements no `flush()`: the batch processor owns ordinary flushing. During shutdown, however, OTel awaits `exporter.forceFlush()` before the processor's `exportTimeoutMillis`-bounded completion promise; if that transport promise never settles, this package abandons the wait at `shutdownTimeoutMillis`, logs the contained shutdown failure through the coordinator, and lets application teardown continue. The deadline cannot cancel the SDK transport, so records still pending then may be lost at process exit. Removing this block from `cordis.yml` is the opt-out: no residual state, no `enabled` flag. ## What leaves the machine diff --git a/packages/telemetry/session-telemetry-otel/README.zh.md b/packages/telemetry/session-telemetry-otel/README.zh.md index f1ad73ddf6..223e6a6639 100644 --- a/packages/telemetry/session-telemetry-otel/README.zh.md +++ b/packages/telemetry/session-telemetry-otel/README.zh.md @@ -10,6 +10,7 @@ - id: telemetry-otel name: '@deepseek-ai/dsh-session-telemetry-otel' config: + shutdownTimeoutMillis: 3000 # optional; defaults to 3000 exporter: # passed verbatim to the SDK's OTLP/HTTP log exporter url: https://collector.example.com/v1/logs headers: @@ -17,7 +18,7 @@ processor: {} # optional; passed verbatim to BatchLogRecordProcessor ``` -`exporter.url` 是本包(package)唯一自行校验的字段:必填、无默认值、必须能解析为 `http(s)`,因此缺失端点会在插件加载时失败(`processor.maxExportBatchSize` 不是正整数时同样如此:SDK 会接受该值,随后却在关闭时因它挂起)。其余全部是 SDK 自己的选项形态,由 SDK 拥有并在 SDK 文档中说明,两个配置块都整体透传(passthrough):`OTLPExporterNodeConfigBase` 的每个字段(`headers`、`timeoutMillis`、`compression`、`keepAlive` 等)都会到达导出器;批处理、导出节奏(`scheduledDelayMillis`)、重试、队列上限,以及持续失败下的丢失策略,都是 SDK 的文档化行为,经 `processor` 透传调优。该后端刻意不实现 `flush()`:批处理器是进程内唯一执行 flush 的组件,`shutdown()` 的排空正因如此才是完整的。从 `cordis.yml` 中删除该配置块即为退出方式:无残留状态,也没有 `enabled` 开关。 +`exporter.url` 是必填项、没有默认值,并且必须能解析为 `http(s)`;`shutdownTimeoutMillis` 是由 DSH 管理的有限正数外层截止时间,默认值为 3000 ms;`processor.maxExportBatchSize` 不是正整数时也会在插件加载时失败,因为 SDK 会接受该值,随后却在关闭时挂起。两个 SDK 配置块都整体透传(passthrough):`OTLPExporterNodeConfigBase` 的每个字段(`headers`、`timeoutMillis`、`compression`、`keepAlive` 等)都会到达导出器;批处理、导出节奏(`scheduledDelayMillis`)、重试、队列上限,以及持续失败下的丢失策略,都是通过 `processor` 调节的 SDK 行为。该后端不实现 `flush()`:常规 flush 由批处理器负责。但在关闭期间,OTel 会先等待 `exporter.forceFlush()`,再进入受 `exportTimeoutMillis` 限制的处理器完成 promise;如果该传输 promise 始终不结算,本包(package)会在 `shutdownTimeoutMillis` 到期时放弃等待,沿协调器现有的失败隔离路径记录关闭失败,并让应用继续拆卸。该截止时间无法取消 SDK 传输,因此届时仍待处理的记录可能在进程退出时丢失。从 `cordis.yml` 中删除该配置块即为退出方式:无残留状态,也没有 `enabled` 开关。 ## 哪些数据会离开本机 diff --git a/packages/telemetry/session-telemetry-otel/src/index.ts b/packages/telemetry/session-telemetry-otel/src/index.ts index 55caa1037b..062bac1e27 100644 --- a/packages/telemetry/session-telemetry-otel/src/index.ts +++ b/packages/telemetry/session-telemetry-otel/src/index.ts @@ -6,8 +6,9 @@ * record handed over by the seam onto `logger.emit()`. Per the seam's * boundary axiom, everything downstream of that call (batching, retry, * queueing, loss policy) is the SDK's documented behavior, configured - * verbatim through the `exporter`/`processor` passthroughs; this package - * adds no knobs of its own on top of them. + * verbatim through the `exporter`/`processor` passthroughs. The one + * backend-owned policy is an outer shutdown deadline: the SDK's export + * timeout does not bound its preceding `forceFlush()` wait. * * @module @deepseek-ai/dsh-session-telemetry-otel */ @@ -33,10 +34,9 @@ import { resourceFromAttributes } from '@opentelemetry/resources' const { version } = createRequire(import.meta.url)('../package.json') as { version: string } /** - * Plugin configuration: two verbatim SDK option shapes plus nothing else. - * `exporter.url` is the one field this package validates itself — required, - * no default, must parse as an `http(s)` URL — because a missing endpoint - * must fail at plugin load, not at first export. + * Plugin configuration: two verbatim SDK option shapes plus one DSH-owned + * shutdown bound. The package validates its endpoint and shutdown deadline + * because both must fail at plugin load rather than at first export or exit. */ export interface Config { /** @@ -54,21 +54,31 @@ export interface Config { * which this plugin fills); the SDK owns and documents these knobs. */ processor?: Omit + /** Maximum time spent awaiting the SDK provider's complete shutdown path. */ + shutdownTimeoutMillis?: number } /** * Schemastery validator for {@link Config}; cordis runs it before the plugin - * starts. Shape-level only — the load-bearing `exporter.url` check lives in - * the constructor so its error message names the field. Both slots are opaque - * passthroughs: the SDK owns their shapes and validates its own options; + * starts. Shape-level only — load-bearing value checks live in the constructor + * so their errors name the fields. Both SDK slots are opaque passthroughs: + * the SDK owns their shapes and validates its own options; * re-declaring them field-by-field here would violate the boundary axiom * (and silently drop every field not re-declared). */ export const Config: z = z.object({ exporter: z.any(), processor: z.any(), + shutdownTimeoutMillis: z.number(), }) +/** Default outer allowance for the SDK's complete shutdown sequence. */ +export const DEFAULT_SHUTDOWN_TIMEOUT_MILLIS = 3_000 + +// Node clamps larger timer delays to one millisecond. This is a runtime +// protocol limit, not a deployment default. +const MAX_TIMER_DELAY_MILLIS = 2_147_483_647 + /** Severity mapping from the seam's three-level vocabulary to OTel severity numbers. */ const SEVERITY: Record = { info: { severityNumber: SeverityNumber.INFO, severityText: 'INFO' }, @@ -90,6 +100,7 @@ export class TelemetryOtel extends Telemetry { private readonly provider: LoggerProvider private readonly ledger: Logger private readonly ops: Logger + private readonly shutdownTimeoutMillis: number constructor(ctx: Context, config: Config) { super(ctx) @@ -115,6 +126,11 @@ export class TelemetryOtel extends Telemetry { if (batchSize !== undefined && (!Number.isInteger(batchSize) || batchSize < 1)) { throw new Error(`session-telemetry-otel: processor.maxExportBatchSize must be a positive integer, got ${String(batchSize)}`) } + const shutdownTimeoutMillis = config.shutdownTimeoutMillis ?? DEFAULT_SHUTDOWN_TIMEOUT_MILLIS + if (!Number.isFinite(shutdownTimeoutMillis) || shutdownTimeoutMillis <= 0 || shutdownTimeoutMillis > MAX_TIMER_DELAY_MILLIS) { + throw new Error(`session-telemetry-otel: shutdownTimeoutMillis must be a positive finite number no greater than ${MAX_TIMER_DELAY_MILLIS}, got ${String(shutdownTimeoutMillis)}`) + } + this.shutdownTimeoutMillis = shutdownTimeoutMillis this.provider = new LoggerProvider({ resource: resourceFromAttributes({ 'service.name': APP_IDENTITY.product, @@ -170,16 +186,26 @@ export class TelemetryOtel extends Telemetry { // the revival Agent Note. /** - * Delegate disposal to the SDK's shutdown contract: drain the queue and - * quiesce. With no concurrent `forceFlush()` in the process (see above), - * shutdown's internal drain is complete — everything emitted before this - * call, including the coordinator's dispose-time `shutdown` markers, is - * exported before the exporter closes. Awaited (and error-contained) by - * the coordinator's disposer. - * @returns resolves when the SDK pipeline has quiesced. + * Ask the SDK to drain and quiesce, but reject after the backend-owned + * deadline. OTel's processor export timeout wraps `exportCompleted` only; + * shutdown awaits `exporter.forceFlush()` first, which can remain pending + * when the transport never obtains a socket. The provider promise remains + * observed after the deadline so a later rejection cannot become unhandled. + * @returns resolves when the SDK pipeline quiesces, or rejects at the configured deadline. */ - shutdown(): Promise { - return this.provider.shutdown() + async shutdown(): Promise { + const providerShutdown = this.provider.shutdown() + let timer: ReturnType | undefined + const deadline = new Promise((_resolve, reject) => { + timer = setTimeout(() => { + reject(new Error(`session-telemetry-otel: provider shutdown exceeded ${this.shutdownTimeoutMillis}ms`)) + }, this.shutdownTimeoutMillis) + }) + try { + await Promise.race([providerShutdown, deadline]) + } finally { + if (timer !== undefined) clearTimeout(timer) + } } } diff --git a/packages/telemetry/session-telemetry-otel/tests/otel.spec.ts b/packages/telemetry/session-telemetry-otel/tests/otel.spec.ts index 6be7f81977..ae3dda5145 100644 --- a/packages/telemetry/session-telemetry-otel/tests/otel.spec.ts +++ b/packages/telemetry/session-telemetry-otel/tests/otel.spec.ts @@ -180,6 +180,38 @@ describe('TelemetryOtel wire', () => { expect(ops[0]!.record.attributes).toContainEqual({ key: 'telemetry.op', value: { stringValue: 'shutdown' } }) }) + it('bounds the SDK forceFlush wait when an in-flight transport never settles', async () => { + const gate = Promise.withResolvers() + const arrived = Promise.withResolvers() + const { url, captures } = await mockCollector(async (index) => { + if (index === 0) { + arrived.resolve(true) + await gate.promise + } + }) + const ctx = new Context() + await ctx.plugin(SessionStore) + const fiber = await ctx.plugin(TelemetryOtel, { + exporter: { url, timeoutMillis: 60_000 }, + processor: { scheduledDelayMillis: 10, exportTimeoutMillis: 60_000 }, + shutdownTimeoutMillis: 50, + }) + const session = ctx.sessions.create(SessionId('bounded-shutdown'), { meta: {} }) + session.append('turn/start', { turn: 1, trigger: { kind: 'message', source: { kind: 'user' } } }) + await arrived.promise + + const started = performance.now() + await fiber.dispose() + expect(performance.now() - started).toBeLessThan(1_000) + expect(captures).toHaveLength(0) + + // The outer deadline cannot cancel the SDK transport. Let it finish so + // the real provider promise remains clean after the test has proved the + // Cordis disposer no longer waits for it. + gate.resolve(true) + await expect.poll(() => captures.length).toBeGreaterThanOrEqual(2) + }) + it('passes exporter options beyond url and headers through to the SDK exporter', async () => { const { url, captures } = await mockCollector() const ctx = new Context() @@ -227,6 +259,8 @@ describe('TelemetryOtel config fails loud', () => { // splices empty batches forever — dispose would hang, so reject at load. [{ exporter: { url: 'http://c/v1/logs' }, processor: { maxExportBatchSize: 0 } }, /maxExportBatchSize/], [{ exporter: { url: 'http://c/v1/logs' }, processor: { maxExportBatchSize: 0.5 } }, /maxExportBatchSize/], + [{ exporter: { url: 'http://c/v1/logs' }, shutdownTimeoutMillis: 0 }, /shutdownTimeoutMillis/], + [{ exporter: { url: 'http://c/v1/logs' }, shutdownTimeoutMillis: Number.POSITIVE_INFINITY }, /shutdownTimeoutMillis/], ])('rejects %j at plugin load', async (config, message) => { const ctx = new Context() await ctx.plugin(SessionStore)