mirror of
https://github.com/deepseek-ai/deepseek-harness
synced 2026-08-15 21:04:50 +00:00
fix(subprocess): clean managed processes on host exit
This commit is contained in:
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write packages/subprocess/subprocess-local/README.md
|
||||
README.md: 15fc001b9fcf2eadd7b37415fd92386565b649c3
|
||||
README.zh.md: 48b662302183f07d514f975089d4b49bd17c8e69
|
||||
README.md: 40c01caa3daf00d490e935bd828e15ce8fa7fb68
|
||||
README.zh.md: edafa0e030cd2af3308bcdea3a0e190de4448125
|
||||
|
||||
@@ -12,7 +12,8 @@ Local Service provider for the [`@deepseek-ai/dsh-subprocess`](../subprocess/REA
|
||||
- **Offset-based reads** — collect-mode readers return deltas in whole-stream byte coordinates; the service never holds a cursor, so consumer-owned cursors (the bash background read path) and full-stream re-reads coexist, before and after settlement.
|
||||
- **Executable lookup** — `resolveExecutable` checks absolute files or searches the scrubbed effective PATH with platform-aware executable extensions; relative paths containing separators are rejected at the seam, and relative PATH entries resolve from the host process cwd.
|
||||
- **Terminal-process ownership** — `spawnTerminal` allocates `node-pty`, bridges UTF-8 terminal text, inspects and signals the current foreground process group, and exposes one awaited termination operation that sweeps descendants before and after terminating the top-level shell. Each foreground inspection retains exact identities from the rooted tree; Linux also enumerates the POSIX session after its leader exits. A previously observed macOS descendant and any same-session Linux member therefore remain fenced after reparenting, while pid/start identity prevents cleanup from following PID reuse. The higher PTY backend owns prompt readiness, buffers, and model-facing operations.
|
||||
- **Terminate-and-join disposal** — the service retains live handles only so its own disposal can escalate every running tree and await its exit; settled and spawn-failed handles leave the live set on settlement.
|
||||
- **Terminate-and-join disposal** — the service retains live handles so its own disposal can escalate every running tree and await its exit; quiescent and spawn-failed handles leave the live set after whole-tree or terminal-session cleanup finishes.
|
||||
- **Synchronous host-exit finalization** — while the service effect is active, a Node `exit` listener force-terminates every ordinary tree and observable terminal session still in the same live sets. The local-only operations send POSIX SIGKILL to the managed group, run Windows `taskkill /T /F`, and synchronously signal captured/current terminal identities around the PTY root kill; they create no promise or timer, preserve the host's exit code and diagnostic, contain each target's failure, and do not claim quiescence. Normal disposal keeps the awaited graceful path above. See the [host-exit cleanup decision](../../../.agents/notes/implemented/bug-fix/2026-08-11-synchronous-subprocess-exit-cleanup.md).
|
||||
|
||||
## Model Experience
|
||||
|
||||
@@ -27,6 +28,7 @@ No direct invalidation; the named consumers own any request-prefix changes.
|
||||
- **Windows tree support is best-effort** — termination routes through `taskkill /PID <pid> /T /F` with all outcomes contained (absent tree, races, missing binary), and liveness falls back to the direct-child boundary.
|
||||
- **Terminal process inspection is Linux/macOS only** — the terminal primitive fails when its inspector has no supported platform implementation; Linux exact probes cover x64 and arm64, while macOS uses `ps` snapshots.
|
||||
- **A daemonized terminal descendant can still escape the observable boundary** — on macOS, a child that reparents before any foreground-inspection snapshot is no longer discoverable from the `node-pty` root; on Linux, a child that calls `setsid` leaves both the tree and owned terminal session. The local provider does not add a continuous process-table monitor.
|
||||
- **In-process cleanup requires a JavaScript-observable exit** — direct `process.exit()`, default uncaught exceptions, and default unhandled rejections emit Node's synchronous `exit` event. `SIGKILL`, fatal OOM, `process.abort()`, native crashes, power loss, and any failure that cannot run JavaScript require an external supervisor, container init, or equivalent OS owner.
|
||||
- **The credential scrub is a name heuristic** — `*KEY*`/`*PASSWORD*`/`*SECRET*`/`*TOKEN*` only; differently-named secrets (e.g. `*PASSPHRASE*`) pass through, and a whitelist for over-scrubbed vars is noted future work.
|
||||
- **Completed spill files are not deleted** — bounded full-output recovery files (and the private per-process spill dir) accumulate under the OS tmpdir until something external cleans them; oversize incomplete spills are discarded and deletion is attempted immediately, but a cleanup failure can leave a bounded file behind.
|
||||
|
||||
|
||||
@@ -12,7 +12,8 @@
|
||||
- **基于偏移量的读取**:收集模式的读取器按完整流的字节坐标返回增量;服务自身从不持有游标,因此消费方自有的游标(bash 的后台读取路径)与完整流重读可以共存,结算前后皆然。
|
||||
- **可执行文件查找**:`resolveExecutable` 检查绝对文件,或根据平台可执行文件扩展名在清理后的有效 PATH 中搜索;含分隔符的相对路径在该能力入口被拒绝,相对 PATH 条目从宿主进程 cwd 解析。
|
||||
- **终端进程所有权**:`spawnTerminal` 分配 `node-pty`,桥接 UTF-8 终端文本,检查当前前台进程组并向其发送信号,还会公开一项须等待的终止操作,在终止顶层 shell 前后清理后代进程。每次前台检查都会保留根进程树中的精确身份;Linux 还会在 POSIX 会话 leader 退出后枚举该会话。因此,之前观察到的 macOS 后代以及同会话 Linux 成员在重新设定父进程后仍受围栏保护,pid/start 身份则防止清理跟随 PID 复用。上层 PTY 后端负责提示符就绪、缓冲区与面向模型的操作。
|
||||
- **先终止再等待退出的 dispose(资源释放)**:服务保留存活句柄,只为让自身的 dispose 能对每个仍在运行的进程树执行升级并等待其退出;已结算与 spawn 失败的句柄在结算时即离开存活集合。
|
||||
- **先终止再等待退出的 dispose(资源释放)**:服务保留存活句柄,使自身的 dispose 能对每个仍在运行的进程树执行升级并等待其退出;完全停稳与 spawn失败的句柄会在整棵进程树或 terminal session清理完成后离开存活集合。
|
||||
- **同步宿主退出最终清理**:服务 effect仍有效时,Node `exit` listener会强制终止同一组存活集合中仍存在的每棵普通进程树和可观察 terminal session。这些仅供本地实现使用的操作会向受管 POSIX进程组发送 SIGKILL、在 Windows运行 `taskkill /T /F`,并在终止 PTY root前后同步向已捕获及当前可观察的 terminal身份发送信号;它们不会创建 Promise或 timer,不改变宿主退出码与诊断,会分别包含每个目标的失败,也不会声称已经完全停稳。正常 dispose仍使用上面的须等待温和路径。参见[宿主退出清理决策](../../../.agents/notes/implemented/bug-fix/2026-08-11-synchronous-subprocess-exit-cleanup.md)。
|
||||
|
||||
## 模型体验
|
||||
|
||||
@@ -27,6 +28,7 @@
|
||||
- **Windows 进程树支持仅为尽力而为**:终止经由 `taskkill /PID <pid> /T /F` 完成,所有结果都被就地吸收,不向外抛出(进程树已不存在、竞态、二进制缺失),存活探测则回退到直接子进程边界。
|
||||
- **终端进程检查仅支持 Linux/macOS**:检查器没有受支持的平台实现时,终端原语会失败;Linux 精确探针覆盖 x64 与 arm64,macOS 则使用 `ps` 快照。
|
||||
- **守护化的终端后代仍可能逃出可观察边界**:在 macOS 上,子进程如果在任何前台检查快照之前重新设定父进程,将无法再从 `node-pty` 根进程发现;在 Linux 上,调用 `setsid` 的子进程会同时离开进程树与自有终端会话。本地提供方不会新增持续进程表监视器。
|
||||
- **进程内清理要求退出阶段仍能执行 JavaScript**:直接 `process.exit()`、默认未捕获异常和默认未处理 rejection会发出 Node同步 `exit`事件。`SIGKILL`、fatal OOM、`process.abort()`、native crash、断电,以及任何无法运行 JavaScript的故障,都需要外部 supervisor、容器 init或等价的 OS所有者负责。
|
||||
- **凭据清除依赖名称启发式规则**:只匹配 `*KEY*`/`*PASSWORD*`/`*SECRET*`/`*TOKEN*`;名称不同的 secret(例如 `*PASSPHRASE*`)会继续传递,对误删变量引入白名单属于已记录的后续工作。
|
||||
- **不会删除已完成的 spill 文件**:有界的完整输出恢复文件(以及每个进程的私有 spill 目录)会在 OS tmpdir 下累积,直到外部机制进行清理;超大的不完整 spill 会被丢弃并立即尝试删除,但清理失败可能留下一个有界文件。
|
||||
|
||||
|
||||
@@ -46,6 +46,7 @@
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/dsh-invariants": "workspace:^",
|
||||
"@deepseek-ai/dsh-loader-smoke": "workspace:^",
|
||||
"@deepseek-ai/dsh-subprocess": "workspace:^",
|
||||
"@deepseek-ai/dsh-timeout": "workspace:^",
|
||||
"@deepseek-ai/cordis": "workspace:^"
|
||||
|
||||
@@ -1,7 +1,8 @@
|
||||
/**
|
||||
* Local Service provider for the subprocess capability seam. Each spawn is a detached
|
||||
* process tree with the spec's per-stream stdio dispositions; disposal
|
||||
* terminates and joins live trees. It has no config: every disposition and
|
||||
* process tree with the spec's per-stream stdio dispositions. Normal disposal
|
||||
* terminates and joins live trees; Node's synchronous exit phase force-stops
|
||||
* any trees the service still owns. It has no config: every disposition and
|
||||
* limit arrives on the spec, so the deployment-varying choices stay with the
|
||||
* caller's config (the bash executor's, the LSP host's, …).
|
||||
* @module @deepseek-ai/dsh-subprocess-local
|
||||
@@ -21,7 +22,7 @@ import type {
|
||||
SubprocessTerminalSpawnSpec,
|
||||
} from '@deepseek-ai/dsh-subprocess'
|
||||
import { childEnv, spawnSubprocess } from './spawn.ts'
|
||||
import type { SpawnInternals } from './spawn.ts'
|
||||
import type { LocalSubprocessHandle, SpawnInternals } from './spawn.ts'
|
||||
import { createProcessInspector } from './process-inspector.ts'
|
||||
import type { ProcessInspector } from './process-inspector.ts'
|
||||
import { LocalTerminalHandle } from './terminal.ts'
|
||||
@@ -30,13 +31,14 @@ import { LocalTerminalHandle } from './terminal.ts'
|
||||
* Local subprocess service: detached process trees, Node-shaped stdio
|
||||
* dispositions (raw pipes, inherit, bounded tail-keep collection with spill
|
||||
* files), credential-scrubbed environment, and tree-scoped signalling with
|
||||
* SIGTERM→grace→SIGKILL escalation.
|
||||
* SIGTERM→grace→SIGKILL escalation, plus synchronous final termination during
|
||||
* JavaScript-observable host exit.
|
||||
*/
|
||||
export class LocalSubprocessService extends SubprocessService {
|
||||
/** Live handles retained only so disposal can terminate and join them. */
|
||||
private live = new Set<SubprocessHandle>()
|
||||
/** Live terminal sessions retained through whole-session quiescence. */
|
||||
private terminals = new Set<SubprocessTerminalHandle>()
|
||||
/** Live handles retained for normal disposal and synchronous host-exit finalization. */
|
||||
private live = new Set<LocalSubprocessHandle>()
|
||||
/** Live terminals retained through normal quiescence or host-exit finalization. */
|
||||
private terminals = new Set<LocalTerminalHandle>()
|
||||
/** Test hook: spill and platform knobs forwarded to spawnSubprocess. */
|
||||
internals: SpawnInternals = {}
|
||||
/** Test hook for platform process inspection; production resolves lazily on terminal spawn. */
|
||||
@@ -44,30 +46,61 @@ export class LocalSubprocessService extends SubprocessService {
|
||||
|
||||
constructor(ctx: Context) {
|
||||
super(ctx)
|
||||
ctx.effect(() => async () => {
|
||||
// Terminate (escalating), then await WHOLE-TREE exit — not just the
|
||||
// direct child's settlement — so even a TERM-trapping descendant cannot
|
||||
// outlive the fiber.
|
||||
const pending: Promise<unknown>[] = []
|
||||
for (const handle of this.live) {
|
||||
handle.terminate()
|
||||
// Spawn-failure rejections already settled and left the live set.
|
||||
pending.push(handle.done.catch(() => {}).then(() => handle.waitForExit()))
|
||||
ctx.effect(() => {
|
||||
const onHostExit = (): void => { this.terminateForHostExit() }
|
||||
process.on('exit', onHostExit)
|
||||
return async () => {
|
||||
try {
|
||||
await this.disposeManagedProcesses()
|
||||
} finally {
|
||||
process.off('exit', onHostExit)
|
||||
}
|
||||
}
|
||||
for (const terminal of this.terminals) {
|
||||
pending.push(terminal.terminate())
|
||||
}
|
||||
this.live.clear()
|
||||
this.terminals.clear()
|
||||
const outcomes = await Promise.allSettled(pending)
|
||||
const failures = outcomes.flatMap<unknown>(outcome => outcome.status === 'rejected'
|
||||
? [outcome.reason as unknown]
|
||||
: [])
|
||||
if (failures.length === 1) throw failures[0]
|
||||
if (failures.length > 1) throw new AggregateError(failures, 'local subprocess teardown failed')
|
||||
}, 'local subprocess teardown')
|
||||
}
|
||||
|
||||
private terminateForHostExit(): void {
|
||||
for (const handle of this.live) {
|
||||
try {
|
||||
handle.terminateForHostExit()
|
||||
} catch (_ordinaryTreeTerminationFailed) {
|
||||
// Host exit cannot await or report one target; continue with the rest.
|
||||
}
|
||||
}
|
||||
for (const terminal of this.terminals) {
|
||||
try {
|
||||
terminal.terminateForHostExit()
|
||||
} catch (_terminalTerminationFailed) {
|
||||
// One terminal must not prevent final termination of another target.
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private async disposeManagedProcesses(): Promise<void> {
|
||||
// Terminate (escalating), then await WHOLE-TREE exit — not just the
|
||||
// direct child's settlement — so even a TERM-trapping descendant cannot
|
||||
// outlive the fiber. Keep both sets authoritative while these waits are
|
||||
// pending so a shorter process-level exit bound can still force-kill them.
|
||||
const pending: Promise<unknown>[] = []
|
||||
for (const handle of this.live) {
|
||||
handle.terminate()
|
||||
// Spawn-failure rejections already settled and left the live set.
|
||||
pending.push(handle.done.catch(() => {}).then(() => handle.waitForExit()))
|
||||
}
|
||||
for (const terminal of this.terminals) {
|
||||
pending.push(terminal.terminate())
|
||||
}
|
||||
const outcomes = await Promise.allSettled(pending)
|
||||
const failures = outcomes.flatMap<unknown>(outcome => outcome.status === 'rejected'
|
||||
? [outcome.reason as unknown]
|
||||
: [])
|
||||
if (failures.length > 0) this.terminateForHostExit()
|
||||
this.live.clear()
|
||||
this.terminals.clear()
|
||||
if (failures.length === 1) throw failures[0]
|
||||
if (failures.length > 1) throw new AggregateError(failures, 'local subprocess teardown failed')
|
||||
}
|
||||
|
||||
async resolveExecutable(
|
||||
command: string,
|
||||
env?: Readonly<Record<string, string>>,
|
||||
|
||||
@@ -58,6 +58,15 @@ export interface SpawnInternals {
|
||||
linuxProcessGroupHasLiveMembers?: (processGroupId: number) => boolean | undefined
|
||||
}
|
||||
|
||||
/**
|
||||
* Local-only extension used by the owning service during Node's synchronous
|
||||
* host-exit phase. It is intentionally absent from the public subprocess seam.
|
||||
*/
|
||||
export interface LocalSubprocessHandle extends SubprocessHandle {
|
||||
/** Force-terminate the current tree synchronously without starting timers or waits. */
|
||||
terminateForHostExit(): void
|
||||
}
|
||||
|
||||
/**
|
||||
* Liveness-poll cadence for tree-exit waits. The timer stays ref'd: an
|
||||
* awaited teardown must keep the event loop alive until the tree really
|
||||
@@ -313,7 +322,7 @@ function signalTree(
|
||||
* @returns live subprocess handle.
|
||||
* @throws when `graceMs` cannot be represented by one Node timer.
|
||||
*/
|
||||
export function spawnSubprocess(spec: SubprocessSpawnSpec, internals: SpawnInternals = {}): SubprocessHandle {
|
||||
export function spawnSubprocess(spec: SubprocessSpawnSpec, internals: SpawnInternals = {}): LocalSubprocessHandle {
|
||||
if (!Number.isFinite(spec.graceMs) || spec.graceMs <= 0 || spec.graceMs > MAX_TIMER_DELAY_MS) {
|
||||
throw new Error(`subprocess graceMs must be a positive finite number no greater than ${MAX_TIMER_DELAY_MS}`)
|
||||
}
|
||||
@@ -442,6 +451,10 @@ export function spawnSubprocess(spec: SubprocessSpawnSpec, internals: SpawnInter
|
||||
graceTimer = setTimeout(() => { kill('SIGKILL') }, spec.graceMs)
|
||||
}
|
||||
|
||||
const terminateForHostExit = (): void => {
|
||||
kill('SIGKILL')
|
||||
}
|
||||
|
||||
// The caller owns timeout classification; this layer only reacts to abort.
|
||||
const onAbort = (): void => { terminate() }
|
||||
spec.signal?.addEventListener('abort', onAbort, { once: true })
|
||||
@@ -523,6 +536,7 @@ export function spawnSubprocess(spec: SubprocessSpawnSpec, internals: SpawnInter
|
||||
},
|
||||
done,
|
||||
terminate,
|
||||
terminateForHostExit,
|
||||
waitForExit,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -110,6 +110,33 @@ export class LocalTerminalHandle implements SubprocessTerminalHandle {
|
||||
return cleanup
|
||||
}
|
||||
|
||||
/**
|
||||
* Force-terminate the observable session synchronously during Node's exit
|
||||
* event. This does not claim quiescence and does not replace terminate().
|
||||
*/
|
||||
terminateForHostExit(): void {
|
||||
this.forceStopDescendants()
|
||||
this.forceStopShell()
|
||||
this.forceStopDescendants()
|
||||
}
|
||||
|
||||
private forceStopShell(): void {
|
||||
if (this.exited) return
|
||||
if (this.rootIdentity !== undefined) {
|
||||
try {
|
||||
this.inspector.signalProcess(this.rootIdentity, 'SIGKILL')
|
||||
} catch (_rootExitedDuringHostExit) {
|
||||
// Exact identity signalling contains both exit races and PID reuse.
|
||||
}
|
||||
return
|
||||
}
|
||||
try {
|
||||
this.terminal.kill('SIGKILL')
|
||||
} catch (_unidentifiedShellExitedDuringHostExit) {
|
||||
// Without a captured identity, node-pty is the only root kill primitive.
|
||||
}
|
||||
}
|
||||
|
||||
private survivors(members: ProcessIdentity[]): ProcessIdentity[] {
|
||||
return members.filter(member => this.inspector.isAlive(member))
|
||||
}
|
||||
@@ -152,6 +179,16 @@ export class LocalTerminalHandle implements SubprocessTerminalHandle {
|
||||
}
|
||||
}
|
||||
|
||||
private forceStopDescendants(): void {
|
||||
let members = this.trackedDescendants
|
||||
try {
|
||||
members = this.descendants()
|
||||
} catch (_processTableUnavailableDuringHostExit) {
|
||||
// Preserve already-captured identities when a final process-table scan fails.
|
||||
}
|
||||
this.signalMembers(members, 'SIGKILL')
|
||||
}
|
||||
|
||||
private unionMembers(...groups: ProcessIdentity[][]): ProcessIdentity[] {
|
||||
const members: ProcessIdentity[] = []
|
||||
const seen = new Set<string>()
|
||||
|
||||
16
packages/subprocess/subprocess-local/tests/fixtures/managed-tree.ts
vendored
Normal file
16
packages/subprocess/subprocess-local/tests/fixtures/managed-tree.ts
vendored
Normal file
@@ -0,0 +1,16 @@
|
||||
import { spawn } from 'node:child_process'
|
||||
import { writeFile } from 'node:fs/promises'
|
||||
|
||||
const [statePath] = process.argv.slice(2)
|
||||
if (statePath === undefined) throw new Error('usage: managed-tree.ts <state-path>')
|
||||
|
||||
process.on('SIGTERM', () => {})
|
||||
process.on('SIGHUP', () => {})
|
||||
const descendant = spawn(process.execPath, [
|
||||
'-e',
|
||||
'process.on("SIGTERM",()=>{});process.on("SIGHUP",()=>{});setInterval(()=>{},60_000)',
|
||||
], { stdio: 'ignore' })
|
||||
if (descendant.pid === undefined) throw new Error('managed descendant did not publish a pid')
|
||||
|
||||
await writeFile(statePath, JSON.stringify({ root: process.pid, descendant: descendant.pid }))
|
||||
setInterval(() => {}, 60_000)
|
||||
79
packages/subprocess/subprocess-local/tests/fixtures/process-exit-host.ts
vendored
Normal file
79
packages/subprocess/subprocess-local/tests/fixtures/process-exit-host.ts
vendored
Normal file
@@ -0,0 +1,79 @@
|
||||
import { access, readFile, writeFile } from 'node:fs/promises'
|
||||
import { join } from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import { Context } from '@deepseek-ai/cordis'
|
||||
import LocalSubprocessService from '@deepseek-ai/dsh-subprocess-local'
|
||||
|
||||
const [kind, trigger, root] = process.argv.slice(2)
|
||||
if ((kind !== 'ordinary' && kind !== 'terminal')
|
||||
|| (trigger !== 'direct' && trigger !== 'uncaught-exception'
|
||||
&& trigger !== 'unhandled-rejection' && trigger !== 'dispose')
|
||||
|| root === undefined) {
|
||||
throw new Error('usage: process-exit-host.ts <ordinary|terminal> <direct|uncaught-exception|unhandled-rejection|dispose> <root>')
|
||||
}
|
||||
|
||||
const treeState = join(root, 'tree.json')
|
||||
const ready = join(root, 'ready')
|
||||
const proceed = join(root, 'proceed')
|
||||
const managedTree = fileURLToPath(new URL('./managed-tree.ts', import.meta.url))
|
||||
|
||||
async function waitForFile(path: string): Promise<void> {
|
||||
for (;;) {
|
||||
try {
|
||||
await access(path)
|
||||
return
|
||||
} catch (_notReady) {
|
||||
await new Promise(resolve => setTimeout(resolve, 10))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const listenersBefore = process.listenerCount('exit')
|
||||
const ctx = new Context()
|
||||
const fiber = await ctx.plugin(LocalSubprocessService)
|
||||
const listenersAfterLoad = process.listenerCount('exit')
|
||||
if (kind === 'ordinary') {
|
||||
ctx.subprocess.spawn({
|
||||
argv: [process.execPath, managedTree, treeState],
|
||||
cwd: process.cwd(),
|
||||
stdio: {
|
||||
stdin: 'ignore',
|
||||
stdout: { maxBytes: 1024 },
|
||||
stderr: { maxBytes: 1024 },
|
||||
},
|
||||
graceMs: trigger === 'dispose' ? 100 : 30_000,
|
||||
})
|
||||
} else {
|
||||
await ctx.subprocess.spawnTerminal({
|
||||
argv: [process.execPath, managedTree, treeState],
|
||||
cwd: process.cwd(),
|
||||
rows: 24,
|
||||
cols: 80,
|
||||
graceMs: 30_000,
|
||||
})
|
||||
}
|
||||
|
||||
await waitForFile(treeState)
|
||||
const published = JSON.parse(await readFile(treeState, 'utf8')) as { root?: unknown; descendant?: unknown }
|
||||
if (!Number.isSafeInteger(published.root) || !Number.isSafeInteger(published.descendant)) {
|
||||
throw new Error('managed tree published invalid process ids')
|
||||
}
|
||||
await writeFile(ready, 'ready')
|
||||
await waitForFile(proceed)
|
||||
|
||||
if (trigger === 'dispose') {
|
||||
await fiber.dispose()
|
||||
await writeFile(join(root, 'dispose.json'), JSON.stringify({
|
||||
listenersBefore,
|
||||
listenersAfterLoad,
|
||||
listenersAfterDispose: process.listenerCount('exit'),
|
||||
}))
|
||||
} else if (trigger === 'direct') {
|
||||
process.exit(23)
|
||||
} else if (trigger === 'uncaught-exception') {
|
||||
setImmediate(() => { throw new Error('host-exit-uncaught-exception') })
|
||||
await new Promise(() => {})
|
||||
} else {
|
||||
void Promise.reject(new Error('host-exit-unhandled-rejection'))
|
||||
await new Promise(() => {})
|
||||
}
|
||||
@@ -21,6 +21,77 @@ function spec(command: string, overrides: Partial<SubprocessSpawnSpec> = {}): Su
|
||||
}
|
||||
|
||||
describe('LocalSubprocessService', () => {
|
||||
it('keeps the host-exit finalizer active until normal disposal reaches quiescence', async () => {
|
||||
const before = new Set(process.listeners('exit'))
|
||||
const ctx = new Context()
|
||||
const fiber = await ctx.plugin(LocalSubprocessService)
|
||||
const listener = process.listeners('exit').find(candidate => !before.has(candidate))
|
||||
expect(listener).toBeTypeOf('function')
|
||||
|
||||
let finishExit!: () => void
|
||||
const exited = new Promise<void>((resolve) => { finishExit = resolve })
|
||||
const terminate = vi.fn()
|
||||
const terminateForHostExit = vi.fn()
|
||||
const live = (ctx.subprocess as unknown as {
|
||||
live: Set<{
|
||||
done: Promise<{ exitCode: number; signal: null }>
|
||||
terminate(): void
|
||||
terminateForHostExit(): void
|
||||
waitForExit(): Promise<boolean>
|
||||
}>
|
||||
}).live
|
||||
live.add({
|
||||
done: Promise.resolve({ exitCode: 0, signal: null }),
|
||||
terminate,
|
||||
terminateForHostExit,
|
||||
waitForExit: async () => { await exited; return true },
|
||||
})
|
||||
|
||||
let disposed = false
|
||||
const disposing = fiber.dispose().then(() => { disposed = true })
|
||||
await new Promise(resolve => setImmediate(resolve))
|
||||
expect(disposed).toBe(false)
|
||||
expect(live.size).toBe(1)
|
||||
listener?.(0)
|
||||
expect(terminate).toHaveBeenCalledOnce()
|
||||
expect(terminateForHostExit).toHaveBeenCalledOnce()
|
||||
|
||||
finishExit()
|
||||
await disposing
|
||||
expect(live.size).toBe(0)
|
||||
expect(process.listeners('exit')).not.toContain(listener)
|
||||
})
|
||||
|
||||
it('contains each host-exit termination failure and continues with the other targets', async () => {
|
||||
const before = new Set(process.listeners('exit'))
|
||||
const ctx = new Context()
|
||||
const fiber = await ctx.plugin(LocalSubprocessService)
|
||||
const listener = process.listeners('exit').find(candidate => !before.has(candidate))
|
||||
expect(listener).toBeTypeOf('function')
|
||||
const ordinaryFailure = vi.fn(() => { throw new Error('ordinary failed') })
|
||||
const ordinarySuccess = vi.fn()
|
||||
const terminalFailure = vi.fn(() => { throw new Error('terminal failed') })
|
||||
const terminalSuccess = vi.fn()
|
||||
const service = ctx.subprocess as unknown as {
|
||||
live: Set<{ terminateForHostExit(): void }>
|
||||
terminals: Set<{ terminateForHostExit(): void }>
|
||||
}
|
||||
service.live.add({ terminateForHostExit: ordinaryFailure })
|
||||
service.live.add({ terminateForHostExit: ordinarySuccess })
|
||||
service.terminals.add({ terminateForHostExit: terminalFailure })
|
||||
service.terminals.add({ terminateForHostExit: terminalSuccess })
|
||||
|
||||
expect(() => { listener?.(0) }).not.toThrow()
|
||||
expect(ordinaryFailure).toHaveBeenCalledOnce()
|
||||
expect(ordinarySuccess).toHaveBeenCalledOnce()
|
||||
expect(terminalFailure).toHaveBeenCalledOnce()
|
||||
expect(terminalSuccess).toHaveBeenCalledOnce()
|
||||
|
||||
service.live.clear()
|
||||
service.terminals.clear()
|
||||
await fiber.dispose()
|
||||
})
|
||||
|
||||
it('resolves absolute and PATH executables and honors lookup cancellation', async () => {
|
||||
const ctx = new Context()
|
||||
const fiber = await ctx.plugin(LocalSubprocessService)
|
||||
@@ -177,6 +248,30 @@ describe('LocalSubprocessService', () => {
|
||||
expect(disposalErrors).toEqual([failure])
|
||||
})
|
||||
|
||||
it('force-terminates remaining targets before releasing a failed disposal', async () => {
|
||||
const before = new Set(process.listeners('exit'))
|
||||
const ctx = new Context()
|
||||
const fiber = await ctx.plugin(LocalSubprocessService)
|
||||
const listener = process.listeners('exit').find(candidate => !before.has(candidate))
|
||||
expect(listener).toBeTypeOf('function')
|
||||
const failure = new Error('cleanup failed')
|
||||
const terminateForHostExit = vi.fn(() => {
|
||||
expect(process.listeners('exit')).toContain(listener)
|
||||
})
|
||||
const terminal = {
|
||||
terminate: vi.fn(async () => { throw failure }),
|
||||
terminateForHostExit,
|
||||
}
|
||||
const terminals = (ctx.subprocess as unknown as { terminals: Set<typeof terminal> }).terminals
|
||||
terminals.add(terminal)
|
||||
|
||||
await fiber.dispose()
|
||||
|
||||
expect(terminateForHostExit).toHaveBeenCalledOnce()
|
||||
expect(terminals.size).toBe(0)
|
||||
expect(process.listeners('exit')).not.toContain(listener)
|
||||
})
|
||||
|
||||
it('releases a terminal after top-level exit reaches quiescence', async () => {
|
||||
let exitListener: ((event: { exitCode: number; signal?: number }) => void) | undefined
|
||||
const inspector = {
|
||||
|
||||
169
packages/subprocess/subprocess-local/tests/process-exit.spec.ts
Normal file
169
packages/subprocess/subprocess-local/tests/process-exit.spec.ts
Normal file
@@ -0,0 +1,169 @@
|
||||
import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import { execa } from 'execa'
|
||||
import { describe, expect, it, vi } from 'vitest'
|
||||
import { resolveExampleLaunch } from '@deepseek-ai/dsh-loader-smoke'
|
||||
import { createProcessInspector } from '../src/process-inspector.ts'
|
||||
import type { ProcessIdentity, ProcessInspector } from '../src/process-inspector.ts'
|
||||
import { taskkillProcessTree } from '../src/spawn.ts'
|
||||
|
||||
type ExitTrigger = 'direct' | 'uncaught-exception' | 'unhandled-rejection' | 'dispose'
|
||||
type ManagedKind = 'ordinary' | 'terminal'
|
||||
interface TreeState { root: number; descendant: number }
|
||||
|
||||
const repoRoot = fileURLToPath(new URL('../../../../', import.meta.url))
|
||||
const hostScript = fileURLToPath(new URL('./fixtures/process-exit-host.ts', import.meta.url))
|
||||
const scenarioTimeoutMs = 30_000
|
||||
|
||||
function processExists(pid: number): boolean {
|
||||
try {
|
||||
process.kill(pid, 0)
|
||||
return true
|
||||
} catch (error: unknown) {
|
||||
if ((error as NodeJS.ErrnoException).code === 'ESRCH') return false
|
||||
throw error
|
||||
}
|
||||
}
|
||||
|
||||
async function readTree(path: string): Promise<TreeState> {
|
||||
return vi.waitFor(async () => {
|
||||
const text = await readFile(path, 'utf8')
|
||||
const state = JSON.parse(text) as Partial<TreeState>
|
||||
if (!Number.isSafeInteger(state.root) || !Number.isSafeInteger(state.descendant)
|
||||
|| (state.root ?? 0) <= 0 || (state.descendant ?? 0) <= 0 || state.root === state.descendant) {
|
||||
throw new Error(`invalid managed-tree state: ${text}`)
|
||||
}
|
||||
return state as TreeState
|
||||
}, { interval: 10, timeout: scenarioTimeoutMs })
|
||||
}
|
||||
|
||||
async function captureIdentities(inspector: ProcessInspector, state: TreeState): Promise<ProcessIdentity[]> {
|
||||
return vi.waitFor(() => {
|
||||
const expected = new Set([state.root, state.descendant])
|
||||
const identities = inspector.processTree(state.root).filter(identity => expected.has(identity.pid))
|
||||
if (identities.length !== expected.size) throw new Error('managed tree is not fully observable yet')
|
||||
return identities
|
||||
}, { interval: 10, timeout: scenarioTimeoutMs })
|
||||
}
|
||||
|
||||
async function waitForGone(state: TreeState): Promise<void> {
|
||||
await Promise.all([state.root, state.descendant].map(pid => vi.waitFor(() => {
|
||||
if (processExists(pid)) throw new Error(`managed pid ${pid} is still alive`)
|
||||
}, { interval: 25, timeout: 10_000 })))
|
||||
}
|
||||
|
||||
function cleanupTree(state: TreeState | undefined, identities: ProcessIdentity[]): void {
|
||||
if (state === undefined) return
|
||||
if (process.platform === 'win32') {
|
||||
taskkillProcessTree(state.root)
|
||||
for (const pid of [state.descendant, state.root]) {
|
||||
try {
|
||||
process.kill(pid, 'SIGKILL')
|
||||
} catch (_alreadyGone) {
|
||||
// The exact recorded process already exited.
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
const inspector = createProcessInspector()
|
||||
for (const identity of identities) {
|
||||
try {
|
||||
inspector.signalProcess(identity, 'SIGKILL')
|
||||
} catch (_alreadyGone) {
|
||||
// Exact start identity prevents PID-reuse cleanup from reaching another process.
|
||||
}
|
||||
}
|
||||
if (identities.length === 0) {
|
||||
for (const pid of [state.descendant, state.root]) {
|
||||
try {
|
||||
process.kill(pid, 'SIGKILL')
|
||||
} catch (_alreadyGone) {
|
||||
// The scenario failed before process identities became observable.
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function runScenario(kind: ManagedKind, trigger: ExitTrigger) {
|
||||
const root = await mkdtemp(join(tmpdir(), `dsh-subprocess-host-exit-${kind}-${trigger}-`))
|
||||
const launch = resolveExampleLaunch({
|
||||
srcBin: hostScript,
|
||||
mode: 'src',
|
||||
tsconfigPath: join(repoRoot, 'tsconfig.json'),
|
||||
configArgs: [kind, trigger, root],
|
||||
})
|
||||
const child = execa(launch.command, launch.args, {
|
||||
cwd: repoRoot,
|
||||
env: launch.env,
|
||||
stdin: 'ignore',
|
||||
reject: false,
|
||||
timeout: scenarioTimeoutMs,
|
||||
})
|
||||
let state: TreeState | undefined
|
||||
let identities: ProcessIdentity[] = []
|
||||
let settled = false
|
||||
try {
|
||||
state = await readTree(join(root, 'tree.json'))
|
||||
await vi.waitFor(() => readFile(join(root, 'ready'), 'utf8'), {
|
||||
interval: 10,
|
||||
timeout: scenarioTimeoutMs,
|
||||
})
|
||||
if (process.platform !== 'win32') identities = await captureIdentities(createProcessInspector(), state)
|
||||
await writeFile(join(root, 'proceed'), 'proceed')
|
||||
const outcome = await child
|
||||
settled = true
|
||||
await waitForGone(state)
|
||||
const disposeCounts = trigger === 'dispose'
|
||||
? JSON.parse(await readFile(join(root, 'dispose.json'), 'utf8')) as {
|
||||
listenersBefore: number
|
||||
listenersAfterLoad: number
|
||||
listenersAfterDispose: number
|
||||
}
|
||||
: undefined
|
||||
return { outcome, disposeCounts }
|
||||
} finally {
|
||||
if (!settled) {
|
||||
child.kill('SIGKILL')
|
||||
await child.catch(() => {})
|
||||
}
|
||||
cleanupTree(state, identities)
|
||||
if (state !== undefined) await waitForGone(state).catch(() => {})
|
||||
await rm(root, { recursive: true, force: true })
|
||||
}
|
||||
}
|
||||
|
||||
describe('synchronous cleanup on host exit', () => {
|
||||
it.each([
|
||||
{ trigger: 'direct' as const, expectedCode: 23, diagnostic: undefined },
|
||||
{ trigger: 'uncaught-exception' as const, expectedCode: 1, diagnostic: 'host-exit-uncaught-exception' },
|
||||
{ trigger: 'unhandled-rejection' as const, expectedCode: 1, diagnostic: 'host-exit-unhandled-rejection' },
|
||||
])('removes an ordinary managed tree after $trigger', { timeout: 45_000 }, async ({
|
||||
trigger,
|
||||
expectedCode,
|
||||
diagnostic,
|
||||
}) => {
|
||||
const { outcome } = await runScenario('ordinary', trigger)
|
||||
expect(outcome.exitCode).toBe(expectedCode)
|
||||
expect(outcome.signal).toBeUndefined()
|
||||
if (diagnostic !== undefined) expect(outcome.stderr).toContain(diagnostic)
|
||||
})
|
||||
|
||||
it.skipIf(process.platform === 'win32')(
|
||||
'removes a terminal root and descendant after direct exit',
|
||||
{ timeout: 45_000 },
|
||||
async () => {
|
||||
const { outcome } = await runScenario('terminal', 'direct')
|
||||
expect(outcome.exitCode).toBe(23)
|
||||
expect(outcome.signal).toBeUndefined()
|
||||
},
|
||||
)
|
||||
|
||||
it('preserves normal terminate-and-join disposal and removes the exit listener', { timeout: 45_000 }, async () => {
|
||||
const { outcome, disposeCounts } = await runScenario('ordinary', 'dispose')
|
||||
expect(outcome.exitCode).toBe(0)
|
||||
expect(disposeCounts?.listenersAfterLoad).toBe((disposeCounts?.listenersBefore ?? 0) + 1)
|
||||
expect(disposeCounts?.listenersAfterDispose).toBe(disposeCounts?.listenersBefore)
|
||||
})
|
||||
})
|
||||
@@ -582,6 +582,25 @@ describe('stdio dispositions', () => {
|
||||
})
|
||||
|
||||
describe('windows tree semantics (injected platform)', () => {
|
||||
it('host-exit termination routes through taskkill immediately', async () => {
|
||||
const killed: number[] = []
|
||||
const running = spawnSubprocess(spec('sleep 60', { graceMs: 60_000 }), {
|
||||
spillDir,
|
||||
platform: 'win32',
|
||||
taskkill: (pid) => {
|
||||
killed.push(pid)
|
||||
try {
|
||||
process.kill(pid, 'SIGKILL')
|
||||
} catch {
|
||||
// Already gone — matches taskkill's tolerated not-found status.
|
||||
}
|
||||
},
|
||||
})
|
||||
running.terminateForHostExit()
|
||||
await running.done
|
||||
expect(killed).toEqual([running.pid])
|
||||
})
|
||||
|
||||
it('terminate routes through taskkill by root pid', async () => {
|
||||
const killed: number[] = []
|
||||
const running = spawnSubprocess(spec('sleep 60', { graceMs: 100 }), {
|
||||
@@ -631,6 +650,23 @@ describe('waitForExit', () => {
|
||||
})
|
||||
})
|
||||
|
||||
describe('synchronous host-exit termination', () => {
|
||||
it('force-kills the current process tree without waiting for the normal grace', async () => {
|
||||
const running = spawnSubprocess(spec('trap "" TERM; sleep 60', { graceMs: 60_000 }))
|
||||
running.terminateForHostExit()
|
||||
await expect(running.done).resolves.toMatchObject({ exitCode: null, signal: 'SIGKILL' })
|
||||
await expect(running.waitForExit()).resolves.toBe(true)
|
||||
|
||||
const kill = vi.spyOn(process, 'kill')
|
||||
try {
|
||||
running.terminateForHostExit()
|
||||
expect(kill).not.toHaveBeenCalled()
|
||||
} finally {
|
||||
kill.mockRestore()
|
||||
}
|
||||
})
|
||||
})
|
||||
|
||||
describe('tree-survivor escalation (terminate and bounded waits reach helpers the leader left behind)', () => {
|
||||
it('terminate() SIGKILLs a TERM-trapping descendant after the direct child settles', async () => {
|
||||
// The leader spawns a TERM-trapping helper with all stdio detached from
|
||||
|
||||
@@ -74,6 +74,7 @@ class FakeInspector implements ProcessInspector {
|
||||
}
|
||||
signalProcess(identity: ProcessIdentity, signal: 'SIGTERM' | 'SIGKILL') {
|
||||
if (this.throwProcess) throw new Error('process raced')
|
||||
if (!this.isAlive(identity)) return
|
||||
this.processes.push([identity.pid, signal])
|
||||
if (this.removeOnSignal) this.alive.delete(identity.pid)
|
||||
}
|
||||
@@ -82,6 +83,85 @@ class FakeInspector implements ProcessInspector {
|
||||
afterEach(() => { vi.useRealTimers() })
|
||||
|
||||
describe('LocalTerminalHandle', () => {
|
||||
it('force-kills descendants around the shell during synchronous host exit', () => {
|
||||
const pty = new FakePty()
|
||||
const inspector = new FakeInspector()
|
||||
const first = { pid: 124, started: 'first' }
|
||||
const late = { pid: 125, started: 'late' }
|
||||
inspector.members = [first]
|
||||
inspector.alive.add(pty.pid)
|
||||
inspector.alive.add(first.pid)
|
||||
const signalProcess = inspector.signalProcess.bind(inspector)
|
||||
inspector.signalProcess = (identity, signal) => {
|
||||
signalProcess(identity, signal)
|
||||
if (identity.pid === pty.pid) {
|
||||
inspector.members = [first, late]
|
||||
inspector.alive.add(late.pid)
|
||||
}
|
||||
}
|
||||
const handle = new LocalTerminalHandle(pty.asPty(), inspector, 10)
|
||||
|
||||
handle.terminateForHostExit()
|
||||
expect(inspector.processes).toEqual([
|
||||
[first.pid, 'SIGKILL'],
|
||||
[pty.pid, 'SIGKILL'],
|
||||
[late.pid, 'SIGKILL'],
|
||||
])
|
||||
expect(pty.kills).toEqual([])
|
||||
|
||||
pty.emitExit()
|
||||
handle.terminateForHostExit()
|
||||
expect(pty.kills).toEqual([])
|
||||
})
|
||||
|
||||
it('uses captured identities and contains shell races when final inspection fails', async () => {
|
||||
const pty = new FakePty()
|
||||
const inspector = new FakeInspector()
|
||||
const captured = { pid: 124, started: 'captured' }
|
||||
inspector.members = [captured]
|
||||
inspector.alive.add(pty.pid)
|
||||
inspector.alive.add(captured.pid)
|
||||
const handle = new LocalTerminalHandle(pty.asPty(), inspector, 10)
|
||||
await handle.inspectForeground()
|
||||
inspector.processTree = () => { throw new Error('process table unavailable') }
|
||||
inspector.throwProcess = true
|
||||
|
||||
expect(() => { handle.terminateForHostExit() }).not.toThrow()
|
||||
expect(inspector.processes).toEqual([])
|
||||
expect(pty.kills).toEqual([])
|
||||
})
|
||||
|
||||
it('uses node-pty only when the shell start identity was unavailable', () => {
|
||||
const pty = new FakePty()
|
||||
const inspector = new FakeInspector()
|
||||
inspector.root = undefined
|
||||
const handle = new LocalTerminalHandle(pty.asPty(), inspector, 10)
|
||||
|
||||
handle.terminateForHostExit()
|
||||
expect(pty.kills).toEqual(['SIGKILL'])
|
||||
|
||||
const racingPty = new FakePty()
|
||||
const racingInspector = new FakeInspector()
|
||||
racingInspector.root = undefined
|
||||
racingPty.throwKill = true
|
||||
const racingHandle = new LocalTerminalHandle(racingPty.asPty(), racingInspector, 10)
|
||||
expect(() => { racingHandle.terminateForHostExit() }).not.toThrow()
|
||||
})
|
||||
|
||||
it('does not signal a recycled terminal root before its delayed exit callback', () => {
|
||||
const pty = new FakePty()
|
||||
const inspector = new FakeInspector()
|
||||
inspector.alive.add(pty.pid)
|
||||
const handle = new LocalTerminalHandle(pty.asPty(), inspector, 10)
|
||||
inspector.root = { pid: pty.pid, started: 'recycled' }
|
||||
inspector.isAlive = identity => identity.started === 'recycled'
|
||||
|
||||
handle.terminateForHostExit()
|
||||
|
||||
expect(inspector.processes).toEqual([])
|
||||
expect(pty.kills).toEqual([])
|
||||
})
|
||||
|
||||
it('bridges terminal bytes, foreground control, and signalled exit facts', async () => {
|
||||
const pty = new FakePty()
|
||||
const inspector = new FakeInspector()
|
||||
|
||||
Reference in New Issue
Block a user