mirror of
https://github.com/deepseek-ai/deepseek-harness
synced 2026-08-15 21:04:50 +00:00
Merge branch 'master' into xtr/identified-immutable-messages
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
2026-07-06-parallel-pre-push-gates.md: 0c3311b259a2fcf00deb4eed491c301a0c330186
|
||||
2026-07-06-parallel-pre-push-gates.zh.md: 6949237d2e025034162f66950033d3ad6ecf11ea
|
||||
# pnpm run verify-translation-pairing --write .agents/notes/implemented/process/2026-07-06-parallel-pre-push-gates.md
|
||||
2026-07-06-parallel-pre-push-gates.md: d86642b7feb82908ec792db0c6a3da403cfc79fd
|
||||
2026-07-06-parallel-pre-push-gates.zh.md: 0425cf1a01b56604a07be366dd46d3920c5fb487
|
||||
|
||||
@@ -12,12 +12,18 @@ Aggregate jobs such as documentation synchronization hide long sequential chains
|
||||
|
||||
## Decision
|
||||
|
||||
[scripts/run-gates.ts](../../../../scripts/run-gates.ts) owns the bounded scheduler used by CI, `doc-sync`, and the opt-in `check:all` command. It expands named modes into leaf gates, respects artifact dependencies, buffers attributable output, and accepts `DSH_GATE_CONCURRENCY` when a caller needs a different worker bound.
|
||||
[scripts/run-gates.ts](../../../../scripts/run-gates.ts) owns the bounded scheduler used by CI, `doc-sync`, and the opt-in `check:all` command. It expands named modes into leaf gates, rejects empty or ambiguous dependency graphs before starting a child, respects artifact dependencies, buffers attributable output, reports exit and signal outcomes independently, and accepts `DSH_GATE_CONCURRENCY` when a caller needs a different worker bound.
|
||||
|
||||
The Node 24 consumer job is one seven-gate mode rather than a shell-owned process pool. Its default worker count equals its gate count while dependencies control readiness: `publint` precedes built-package invariant validation, and snapshot replay, NodeNext type checks, built-bin smokes, and lint wait for that validation. Lint waits because the invariant verifier temporarily stages package views that ESLint must not traverse; source compatibility checks can overlap the validation chain.
|
||||
|
||||
[scripts/publint-all.ts](../../../../scripts/publint-all.ts) discovers packages from `packages/<group>/<pkg>` and runs `publint` with a worker pool sized from `availableParallelism()`. `DSH_PUBLINT_CONCURRENCY` can cap or raise the worker count for local machines and CI runners with different resource profiles. Results are buffered per package and printed in deterministic package order, so parallel execution does not scramble each package's log block.
|
||||
|
||||
The per-gate package scripts remain the vocabulary for ad hoc local runs. `hygiene` stays an aggregate `&&` chain, while `doc-sync` owns its member list in the scheduler ([doc-sync through the gate scheduler](../../archived/process/2026-07-21-doc-sync-through-gate-scheduler.md)).
|
||||
|
||||
## Verification
|
||||
|
||||
[scripts/run-gates.spec.ts](../../../../scripts/run-gates.spec.ts) rejects invalid graphs before the executor runs, pins the consumer inventory and dependency edges, and exercises signal termination through a real child process. [scripts/publint-all.spec.ts](../../../../scripts/publint-all.spec.ts) rejects a missing public export before downstream artifact consumers run.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Keep aggregate jobs serial** — simpler execution but makes wall clock equal the sum of independent checks and repeats command-wrapper startup.
|
||||
@@ -28,6 +34,8 @@ The per-gate package scripts remain the vocabulary for ad hoc local runs. `hygie
|
||||
|
||||
## Consequences
|
||||
|
||||
Scheduler-backed commands take the slowest dependency chain instead of the sum of independent gates and report the gate that dominates. The cost is a custom scheduler with an explicit mode inventory.
|
||||
Scheduler-backed commands take the slowest dependency chain instead of the sum of independent gates and report the gate that dominates. Invalid graphs fail before partial execution. The cost is a custom scheduler with an explicit mode inventory.
|
||||
|
||||
The consumer validation chain delays restored-artifact consumers and lint until the shared artifact view is known-good and transient staging is gone; those downstream gates can still overlap one another.
|
||||
|
||||
`publint-all.ts` is asynchronous and buffers command output instead of inheriting stdio live. The payoff is package-level parallelism with stable output order and one environment variable for resource tuning.
|
||||
|
||||
@@ -12,12 +12,18 @@ Status: implemented
|
||||
|
||||
## 决策
|
||||
|
||||
[scripts/run-gates.ts](../../../../scripts/run-gates.ts) 拥有 CI、`doc-sync` 和选择启用的 `check:all` 命令所使用的有界调度器。它将具名模式展开为叶子门禁,遵守产物依赖,缓冲可归因的输出,并在调用方需要不同 worker 上限时接受 `DSH_GATE_CONCURRENCY`。
|
||||
[scripts/run-gates.ts](../../../../scripts/run-gates.ts) 拥有 CI、`doc-sync` 和选择启用的 `check:all` 命令所使用的有界调度器。它将具名模式展开为叶子门禁,在启动子进程前拒绝空的或有歧义的依赖图,遵守产物依赖,缓冲可归因的输出,分别报告退出结果与信号结果,并在调用方需要不同 worker 上限时接受 `DSH_GATE_CONCURRENCY`。
|
||||
|
||||
Node 24 消费方 job 采用单个包含七道门禁的模式,而非由 shell 管理的进程池。其默认 worker 数等于门禁数,但门禁是否就绪由依赖关系控制:`publint` 先于已构建包不变式验证运行,快照回放、NodeNext 类型检查、built-bin 冒烟测试和 lint 则等待该验证完成。lint 之所以等待,是因为不变式验证器会临时暂存包视图,而 ESLint 不得遍历这些视图;源码兼容性检查可以与这条验证链重叠运行。
|
||||
|
||||
[scripts/publint-all.ts](../../../../scripts/publint-all.ts) 从 `packages/<group>/<pkg>` 发现包,并以根据 `availableParallelism()` 确定大小的 worker 池运行 `publint`。`DSH_PUBLINT_CONCURRENCY` 可以针对资源配置不同的本地机器和 CI runner 限制或提高 worker 数量。结果按包缓冲,并按确定性的包顺序打印,因此并行执行不会打乱各包的日志块。
|
||||
|
||||
各门禁的包脚本仍是临时本地运行所用的词汇。`hygiene` 继续作为聚合 `&&` 链,而 `doc-sync` 在调度器中拥有其成员列表([通过门禁调度器运行 doc-sync](../../archived/process/2026-07-21-doc-sync-through-gate-scheduler.md))。
|
||||
|
||||
## 验证
|
||||
|
||||
[scripts/run-gates.spec.ts](../../../../scripts/run-gates.spec.ts) 在执行器运行前拒绝无效图,锁定消费方清单和依赖边,并通过真实子进程验证信号终止。[scripts/publint-all.spec.ts](../../../../scripts/publint-all.spec.ts) 在下游产物消费方运行前拒绝缺失的公开导出。
|
||||
|
||||
## 曾考虑的替代方案
|
||||
|
||||
- **保持聚合 job 串行**:执行更简单,但墙钟时间等于各独立检查之和,并重复启动命令包装器。
|
||||
@@ -28,6 +34,8 @@ Status: implemented
|
||||
|
||||
## 后果
|
||||
|
||||
由调度器支持的命令耗时取最慢依赖链,而非各独立门禁之和,并会报告主导耗时的门禁。代价是维护一个具有显式模式清单的定制调度器。
|
||||
由调度器支持的命令耗时取最慢依赖链,而非各独立门禁之和,并会报告主导耗时的门禁。无效图会直接失败,不会先执行其中一部分。代价是维护一个具有显式模式清单的定制调度器。
|
||||
|
||||
这条验证链会让已恢复产物的下游消费方和 lint 延后启动,直至共享产物视图经确认有效且临时暂存已清除;这些下游门禁仍可彼此重叠运行。
|
||||
|
||||
`publint-all.ts` 采用异步执行并缓冲命令输出,而不是实时继承 stdio。换来的是具有稳定输出顺序的包级并行,以及用于资源调节的单一环境变量。
|
||||
|
||||
39
.github/workflows/ci.yml
vendored
39
.github/workflows/ci.yml
vendored
@@ -240,44 +240,7 @@ jobs:
|
||||
exit "$sandbox_status"
|
||||
|
||||
- name: Run compatibility, snapshot, and artifact gates
|
||||
run: |
|
||||
pnpm run check:ci:lint &
|
||||
lint_pid=$!
|
||||
pnpm run check:node-compat &
|
||||
compat_pid=$!
|
||||
DSH_EXAMPLE_MODE=lib pnpm run test:snapshot &
|
||||
snapshot_pid=$!
|
||||
pnpm run publint &
|
||||
publint_pid=$!
|
||||
pnpm run verify-node-next-types &
|
||||
node_next_pid=$!
|
||||
pnpm run verify-built-package-invariants &
|
||||
built_invariants_pid=$!
|
||||
DSH_EXAMPLE_MODE=lib pnpm exec vitest run --config vitest.e2e.config.ts \
|
||||
examples/headless-agent/tests/keyless-smoke.e2e.ts \
|
||||
examples/tui-agent/tests/tui-keyless-smoke.e2e.ts \
|
||||
packages/examples/cli-demo/tests/built-bin.e2e.ts \
|
||||
packages/examples/acp-demo/tests/built-bin.e2e.ts \
|
||||
packages/ui/jsonrpc/tests/built-scope-carrier.e2e.ts \
|
||||
packages/workflow/workflow-workerthread/tests/built-worker.e2e.ts \
|
||||
packages/code-runtime/code-runtime-worker/tests/built-lib.e2e.ts &
|
||||
built_bin_pid=$!
|
||||
|
||||
final_status=0
|
||||
capture_status() {
|
||||
local child_status=0
|
||||
wait "$1" || child_status=$?
|
||||
if (( final_status == 0 && child_status != 0 )); then
|
||||
final_status=$child_status
|
||||
fi
|
||||
}
|
||||
for child_pid in \
|
||||
"$lint_pid" "$compat_pid" "$snapshot_pid" \
|
||||
"$publint_pid" "$node_next_pid" "$built_invariants_pid" "$built_bin_pid"
|
||||
do
|
||||
capture_status "$child_pid"
|
||||
done
|
||||
exit "$final_status"
|
||||
run: pnpm run check:ci:consumers
|
||||
|
||||
|
||||
node-compat:
|
||||
|
||||
@@ -38,6 +38,7 @@
|
||||
"check:ci:coverage": "tsx scripts/run-gates.ts ci-coverage",
|
||||
"check:ci:snapshot": "tsx scripts/run-gates.ts ci-snapshot",
|
||||
"check:ci:artifacts": "tsx scripts/run-gates.ts ci-artifacts",
|
||||
"check:ci:consumers": "tsx scripts/run-gates.ts ci-consumers",
|
||||
"check:ci:windows-blocking": "tsx scripts/run-gates.ts ci-windows-blocking",
|
||||
"check:ci:windows-complete": "tsx scripts/run-gates.ts ci-windows-complete",
|
||||
"check:ci:windows-observational": "tsx scripts/run-gates.ts ci-windows-observational",
|
||||
|
||||
@@ -58,4 +58,10 @@ describe('publint package runner', () => {
|
||||
expect(result.status).toBe(1)
|
||||
expect(result.stdout).toContain('unpublished.js')
|
||||
})
|
||||
|
||||
it('rejects a public export whose built file is missing', () => {
|
||||
const result = run(fixture('./lib/missing.js'))
|
||||
expect(result.status).toBe(1)
|
||||
expect(result.stdout).toContain('missing.js')
|
||||
})
|
||||
})
|
||||
|
||||
137
scripts/run-gates.spec.ts
Normal file
137
scripts/run-gates.spec.ts
Normal file
@@ -0,0 +1,137 @@
|
||||
import { describe, expect, it, vi } from 'vitest'
|
||||
import {
|
||||
defaultConcurrency,
|
||||
formatGateResultReason,
|
||||
gatesForMode,
|
||||
runGate,
|
||||
runGates,
|
||||
type Gate,
|
||||
type GateResult,
|
||||
} from './run-gates.ts'
|
||||
|
||||
function gate(id: string, options: Partial<Gate> = {}): Gate {
|
||||
return {
|
||||
id,
|
||||
label: id,
|
||||
displayCommand: `run ${id}`,
|
||||
command: process.execPath,
|
||||
args: ['-e', ''],
|
||||
...options,
|
||||
}
|
||||
}
|
||||
|
||||
function resultFor(subject: Gate, status: GateResult['status'] = 'passed'): GateResult {
|
||||
return {
|
||||
gate: subject,
|
||||
status,
|
||||
durationMs: 10,
|
||||
output: [],
|
||||
exitCode: status === 'passed' ? 0 : 1,
|
||||
signalCode: null,
|
||||
}
|
||||
}
|
||||
|
||||
function withPnpmEntrypoint<T>(action: () => T): T {
|
||||
const previous = process.env.npm_execpath
|
||||
process.env.npm_execpath = '/private/pnpm.cjs'
|
||||
try {
|
||||
return action()
|
||||
} finally {
|
||||
if (previous === undefined) Reflect.deleteProperty(process.env, 'npm_execpath')
|
||||
else process.env.npm_execpath = previous
|
||||
}
|
||||
}
|
||||
|
||||
describe('gate graph validation', () => {
|
||||
it.each([
|
||||
'ci-primary',
|
||||
'ci-static',
|
||||
'ci-lint',
|
||||
'ci-coverage',
|
||||
'ci-snapshot',
|
||||
'ci-artifacts',
|
||||
'ci-consumers',
|
||||
'ci-windows-blocking',
|
||||
'ci-windows-complete',
|
||||
'ci-windows-observational',
|
||||
'node-compat',
|
||||
'check-all',
|
||||
'doc-sync',
|
||||
] as const)('constructs and executes preflight for a valid non-empty %s graph', async (mode) => {
|
||||
const subject = withPnpmEntrypoint(() => gatesForMode(mode))
|
||||
const execute = vi.fn(async (item: Gate) => resultFor(item))
|
||||
|
||||
await expect(runGates(subject, subject.length, execute)).resolves.toHaveLength(subject.length)
|
||||
})
|
||||
|
||||
it.each([
|
||||
['empty', [], /gate graph has no gates/],
|
||||
['duplicate ids', [gate('same'), gate('same')], /duplicate gate id "same"/],
|
||||
['unknown dependencies', [gate('subject', { needs: ['missing'] })], /depends on unknown gate "missing"/],
|
||||
['cycles', [gate('first', { needs: ['second'] }), gate('second', { needs: ['first'] })], /dependency cycle: first -> second -> first/],
|
||||
] as const)('rejects %s before starting a child', async (_label, invalid, message) => {
|
||||
const execute = vi.fn(async (subject: Gate) => resultFor(subject))
|
||||
|
||||
await expect(runGates([...invalid], 1, execute)).rejects.toThrow(message)
|
||||
expect(execute).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('rejects an invalid worker count before starting a child', async () => {
|
||||
const execute = vi.fn(async (subject: Gate) => resultFor(subject))
|
||||
|
||||
await expect(runGates([gate('subject')], 0, execute)).rejects.toThrow('max concurrency must be a positive integer')
|
||||
expect(execute).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('skips dependents after their prerequisite fails', async () => {
|
||||
const dependent = gate('dependent', { needs: ['root'] })
|
||||
const root = gate('root')
|
||||
const execute = vi.fn(async (subject: Gate) => resultFor(subject, 'failed'))
|
||||
|
||||
const results = await runGates([dependent, root], 1, execute)
|
||||
|
||||
expect(execute).toHaveBeenCalledOnce()
|
||||
expect(execute).toHaveBeenCalledWith(root)
|
||||
expect(results[0]).toMatchObject({ gate: dependent, status: 'skipped', error: 'dependency failed or skipped: root' })
|
||||
})
|
||||
})
|
||||
|
||||
describe('Node 24 consumer graph', () => {
|
||||
it('owns the seven-command pool and orders restored-artifact consumers', () => {
|
||||
const subject = withPnpmEntrypoint(() => gatesForMode('ci-consumers'))
|
||||
|
||||
expect(defaultConcurrency('ci-consumers', subject.length, 4)).toEqual({
|
||||
workers: 7,
|
||||
source: 'ci-consumers gate count',
|
||||
})
|
||||
expect(subject.map(item => item.id)).toEqual([
|
||||
'lint-and-duplication',
|
||||
'node-compat',
|
||||
'snapshot',
|
||||
'publint',
|
||||
'node-next-types',
|
||||
'built-package-invariants',
|
||||
'built-bin-smoke',
|
||||
])
|
||||
expect(subject.find(item => item.id === 'publint')?.needs).toBeUndefined()
|
||||
expect(subject.find(item => item.id === 'built-package-invariants')?.needs).toEqual(['publint'])
|
||||
expect(subject.find(item => item.id === 'lint-and-duplication')?.needs).toEqual(['built-package-invariants'])
|
||||
for (const id of ['snapshot', 'node-next-types', 'built-bin-smoke']) {
|
||||
expect(subject.find(item => item.id === id)?.needs).toEqual(['built-package-invariants'])
|
||||
}
|
||||
expect(subject.find(item => item.id === 'snapshot')?.env).toEqual({ DSH_EXAMPLE_MODE: 'lib' })
|
||||
})
|
||||
})
|
||||
|
||||
describe('gate process outcomes', () => {
|
||||
it.skipIf(process.platform === 'win32')('reports signal termination independently from exit status', async () => {
|
||||
const result = await runGate(gate('terminated', {
|
||||
args: ['-e', "process.kill(process.pid, 'SIGTERM')"],
|
||||
}))
|
||||
|
||||
expect(result.status).toBe('failed')
|
||||
expect(result.exitCode).toBeNull()
|
||||
expect(result.signalCode).toBe('SIGTERM')
|
||||
expect(formatGateResultReason(result)).toBe('signal SIGTERM')
|
||||
})
|
||||
})
|
||||
@@ -1,31 +1,36 @@
|
||||
/**
|
||||
* Run local and CI quality gates with bounded in-process scheduling.
|
||||
*
|
||||
* The gate vocabulary stays in package.json; this runner only decides which
|
||||
* independent commands can overlap and which commands wait for built artifacts.
|
||||
* Package scripts own public aggregate names; this runner owns their validated
|
||||
* dependency graphs, scheduler environment, and process diagnostics.
|
||||
* @see ../.agents/notes/implemented/process/2026-07-06-parallel-pre-push-gates.md
|
||||
*/
|
||||
import { spawn } from 'node:child_process'
|
||||
import { availableParallelism } from 'node:os'
|
||||
import { resolve } from 'node:path'
|
||||
import { performance } from 'node:perf_hooks'
|
||||
|
||||
type Mode =
|
||||
/** A named aggregate exposed by the gate runner. */
|
||||
export type Mode =
|
||||
| 'ci-primary'
|
||||
| 'ci-static'
|
||||
| 'ci-lint'
|
||||
| 'ci-coverage'
|
||||
| 'ci-snapshot'
|
||||
| 'ci-artifacts'
|
||||
| 'ci-consumers'
|
||||
| 'ci-windows-blocking'
|
||||
| 'ci-windows-complete'
|
||||
| 'ci-windows-observational'
|
||||
| 'node-compat'
|
||||
| 'pre-push'
|
||||
| 'check-all'
|
||||
| 'doc-sync'
|
||||
type GateStatus = 'pending' | 'running' | 'passed' | 'failed' | 'skipped'
|
||||
|
||||
interface Gate {
|
||||
type GateResultStatus = 'passed' | 'failed' | 'skipped'
|
||||
type GateState = 'pending' | 'running' | GateResultStatus
|
||||
|
||||
/** A command and its dependency metadata inside one aggregate. */
|
||||
export interface Gate {
|
||||
id: string
|
||||
label: string
|
||||
displayCommand: string
|
||||
@@ -33,19 +38,17 @@ interface Gate {
|
||||
args: string[]
|
||||
needs?: string[]
|
||||
env?: Record<string, string | undefined>
|
||||
input?: string
|
||||
verify?: (result: GateResult) => Promise<void>
|
||||
allowFailure?: boolean
|
||||
}
|
||||
|
||||
interface GateResult {
|
||||
/** The observed outcome of one gate process. */
|
||||
export interface GateResult {
|
||||
gate: Gate
|
||||
status: GateStatus
|
||||
status: GateResultStatus
|
||||
durationMs: number
|
||||
stdout: string
|
||||
stderr: string
|
||||
output: GateOutputChunk[]
|
||||
exitCode: number | null
|
||||
signalCode: NodeJS.Signals | null
|
||||
error?: string
|
||||
}
|
||||
|
||||
@@ -64,25 +67,31 @@ interface ConcurrencyDefault {
|
||||
source: string
|
||||
}
|
||||
|
||||
type GateExecutor = (gate: Gate) => Promise<GateResult>
|
||||
type ResultObserver = (result: GateResult) => void
|
||||
|
||||
const root = resolve(import.meta.dirname, '..')
|
||||
const mode = parseMode(process.argv[2])
|
||||
const gates = gatesForMode(mode)
|
||||
const concurrencyDefault = defaultConcurrency(mode, gates.length)
|
||||
const concurrencyOverride = process.env.DSH_GATE_CONCURRENCY
|
||||
const maxConcurrency = concurrencyFromEnv('DSH_GATE_CONCURRENCY', concurrencyDefault.workers)
|
||||
const verbose = process.env.DSH_GATE_VERBOSE === '1'
|
||||
const startedAt = performance.now()
|
||||
if (import.meta.main) {
|
||||
process.exitCode = await main(process.argv.slice(2))
|
||||
}
|
||||
|
||||
const concurrencySource = concurrencyOverride === undefined || concurrencyOverride === ''
|
||||
? concurrencyDefault.source
|
||||
: '$DSH_GATE_CONCURRENCY'
|
||||
console.log(`run-gates: ${mode} running ${gates.length} gate(s) with ${maxConcurrency} worker(s) from ${concurrencySource}.`)
|
||||
async function main(args: string[]): Promise<number> {
|
||||
const mode = parseMode(args[0])
|
||||
const gates = gatesForMode(mode)
|
||||
const concurrencyDefault = defaultConcurrency(mode, gates.length)
|
||||
const concurrencyOverride = process.env.DSH_GATE_CONCURRENCY
|
||||
const maxConcurrency = concurrencyFromEnv('DSH_GATE_CONCURRENCY', concurrencyDefault.workers)
|
||||
const concurrencySource = concurrencyOverride === undefined || concurrencyOverride === ''
|
||||
? concurrencyDefault.source
|
||||
: '$DSH_GATE_CONCURRENCY'
|
||||
const startedAt = performance.now()
|
||||
console.log(`run-gates: ${mode} running ${gates.length} gate(s) with ${maxConcurrency} worker(s) from ${concurrencySource}.`)
|
||||
|
||||
const results = await runGates(gates, maxConcurrency)
|
||||
printSummary(results, performance.now() - startedAt)
|
||||
|
||||
if (results.some(result => result.gate.allowFailure !== true && (result.status === 'failed' || result.status === 'skipped'))) {
|
||||
process.exit(1)
|
||||
const results = await runGates(gates, maxConcurrency, runGate, printResult)
|
||||
printSummary(results, performance.now() - startedAt)
|
||||
return results.some(result => result.gate.allowFailure !== true && (result.status === 'failed' || result.status === 'skipped'))
|
||||
? 1
|
||||
: 0
|
||||
}
|
||||
|
||||
function parseMode(raw: string | undefined): Mode {
|
||||
@@ -93,26 +102,37 @@ function parseMode(raw: string | undefined): Mode {
|
||||
case 'ci-coverage':
|
||||
case 'ci-snapshot':
|
||||
case 'ci-artifacts':
|
||||
case 'ci-consumers':
|
||||
case 'ci-windows-blocking':
|
||||
case 'ci-windows-complete':
|
||||
case 'ci-windows-observational':
|
||||
case 'node-compat':
|
||||
case 'pre-push':
|
||||
case 'check-all':
|
||||
case 'doc-sync':
|
||||
return raw
|
||||
default:
|
||||
throw new Error(
|
||||
`run-gates: expected mode ci-primary | ci-static | ci-lint | ci-coverage | ci-snapshot | ci-artifacts | ci-windows-blocking | ci-windows-complete | ci-windows-observational | node-compat | pre-push | check-all | doc-sync, got ${JSON.stringify(raw)}.`,
|
||||
`run-gates: expected mode ci-primary | ci-static | ci-lint | ci-coverage | ci-snapshot | ci-artifacts | ci-consumers | ci-windows-blocking | ci-windows-complete | ci-windows-observational | node-compat | check-all | doc-sync, got ${JSON.stringify(raw)}.`,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
function defaultConcurrency(selectedMode: Mode, total: number): ConcurrencyDefault {
|
||||
const available = availableParallelism()
|
||||
/**
|
||||
* Resolve the default worker count for one aggregate.
|
||||
* @param selectedMode - aggregate whose resource posture applies.
|
||||
* @param total - number of gates in the aggregate.
|
||||
* @param available - host CPU availability for ordinary modes.
|
||||
* @returns the default worker count and its diagnostic source.
|
||||
*/
|
||||
export function defaultConcurrency(
|
||||
selectedMode: Mode,
|
||||
total: number,
|
||||
available = availableParallelism(),
|
||||
): ConcurrencyDefault {
|
||||
if (selectedMode === 'ci-consumers') return { workers: total, source: 'ci-consumers gate count' }
|
||||
// Local modes cap workers: several doc gates each build a full ts.Program,
|
||||
// so an uncapped default on a large host trades wall clock for memory blowups.
|
||||
const localCap = selectedMode === 'pre-push' || selectedMode === 'check-all' || selectedMode === 'doc-sync'
|
||||
const localCap = selectedMode === 'check-all' || selectedMode === 'doc-sync'
|
||||
const modeLimit = localCap ? Math.min(4, available) : available
|
||||
return {
|
||||
workers: Math.min(total, modeLimit),
|
||||
@@ -165,7 +185,12 @@ function nodeOptions(...options: string[]): string {
|
||||
return [process.env.NODE_OPTIONS, ...options].filter(option => option !== undefined && option !== '').join(' ')
|
||||
}
|
||||
|
||||
function gatesForMode(selected: Mode): Gate[] {
|
||||
/**
|
||||
* Construct the complete gate list for a named aggregate.
|
||||
* @param selected - aggregate mode to construct.
|
||||
* @returns the aggregate's gate graph.
|
||||
*/
|
||||
export function gatesForMode(selected: Mode): Gate[] {
|
||||
switch (selected) {
|
||||
case 'ci-primary':
|
||||
return ciPrimaryGates()
|
||||
@@ -182,6 +207,8 @@ function gatesForMode(selected: Mode): Gate[] {
|
||||
return [pnpmScript('build', 'build'), snapshotGate()]
|
||||
case 'ci-artifacts':
|
||||
return ciArtifactGates()
|
||||
case 'ci-consumers':
|
||||
return ciConsumerGates()
|
||||
case 'ci-windows-blocking':
|
||||
return ciWindowsBlockingGates()
|
||||
case 'ci-windows-complete':
|
||||
@@ -190,7 +217,6 @@ function gatesForMode(selected: Mode): Gate[] {
|
||||
return ciWindowsObservationalGates()
|
||||
case 'node-compat':
|
||||
return nodeCompatGates()
|
||||
case 'pre-push': return []
|
||||
case 'check-all':
|
||||
return [
|
||||
pnpmScript('runtime-closure', 'verify-runtime-closure', { label: 'runtime closure' }),
|
||||
@@ -294,6 +320,26 @@ function ciArtifactGates(): Gate[] {
|
||||
]
|
||||
}
|
||||
|
||||
function ciConsumerGates(): Gate[] {
|
||||
const publicArtifacts = ['publint']
|
||||
const restoredBuild = ['built-package-invariants']
|
||||
return [
|
||||
pnpmScript('lint-and-duplication', 'check:ci:lint', {
|
||||
label: 'lint and duplication',
|
||||
needs: restoredBuild,
|
||||
}),
|
||||
pnpmScript('node-compat', 'check:node-compat', { label: 'Node compatibility' }),
|
||||
snapshotGate(restoredBuild),
|
||||
pnpmScript('publint', 'publint'),
|
||||
pnpmScript('node-next-types', 'verify-node-next-types', {
|
||||
label: 'node-next types',
|
||||
needs: restoredBuild,
|
||||
}),
|
||||
builtPackageInvariantsGate(publicArtifacts),
|
||||
builtBinSmokeGate(restoredBuild),
|
||||
]
|
||||
}
|
||||
|
||||
function ciWindowsBlockingGates(): Gate[] {
|
||||
return [
|
||||
pnpmScript('windows-build', 'build', { label: 'build' }),
|
||||
@@ -381,11 +427,11 @@ function coverageGate(): Gate {
|
||||
|
||||
// Example and package snapshots boot their bins in `lib` mode (built artifacts under plain Node,
|
||||
// plugins via real exports); repository-script snapshots execute their real source entry path.
|
||||
// CI and check-all already build before either class runs, so the suite waits on `build`.
|
||||
function snapshotGate(): Gate {
|
||||
// Build-owning modes wait on `build`; a restored-artifact mode passes its validation dependency.
|
||||
function snapshotGate(needs: string[] = ['build']): Gate {
|
||||
return pnpmScript('snapshot', 'test:snapshot', {
|
||||
env: { DSH_EXAMPLE_MODE: 'lib' },
|
||||
needs: ['build'],
|
||||
needs,
|
||||
})
|
||||
}
|
||||
|
||||
@@ -468,7 +514,7 @@ function docSyncLeafGates(options: {
|
||||
]
|
||||
}
|
||||
|
||||
function builtBinSmokeGate(): Gate {
|
||||
function builtBinSmokeGate(needs: string[] = ['build']): Gate {
|
||||
return pnpmExec('built-bin-smoke', [
|
||||
'vitest',
|
||||
'run',
|
||||
@@ -486,44 +532,125 @@ function builtBinSmokeGate(): Gate {
|
||||
'packages/code-runtime/code-runtime-worker/tests/built-lib.e2e.ts',
|
||||
], {
|
||||
label: 'built-bin smoke',
|
||||
needs: ['build'],
|
||||
needs,
|
||||
env: { DSH_EXAMPLE_MODE: 'lib' },
|
||||
})
|
||||
}
|
||||
|
||||
async function runGates(allGates: Gate[], maxActive: number): Promise<GateResult[]> {
|
||||
const states = new Map<string, GateStatus>(allGates.map(gate => [gate.id, 'pending']))
|
||||
/**
|
||||
* Reject a gate list whose graph cannot be executed unambiguously.
|
||||
* @param gates - complete aggregate to validate.
|
||||
*/
|
||||
function validateGateGraph(gates: readonly Gate[]): void {
|
||||
if (gates.length === 0) throw new Error('run-gates: gate graph has no gates.')
|
||||
|
||||
const ids = new Set<string>()
|
||||
for (const gate of gates) {
|
||||
if (ids.has(gate.id)) throw new Error(`run-gates: duplicate gate id ${JSON.stringify(gate.id)}.`)
|
||||
ids.add(gate.id)
|
||||
}
|
||||
for (const gate of gates) {
|
||||
for (const dependency of gate.needs ?? []) {
|
||||
if (!ids.has(dependency)) {
|
||||
throw new Error(`run-gates: gate ${JSON.stringify(gate.id)} depends on unknown gate ${JSON.stringify(dependency)}.`)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const cycle = findDependencyCycle(gates)
|
||||
if (cycle !== undefined) throw new Error(`run-gates: dependency cycle: ${cycle.join(' -> ')}.`)
|
||||
}
|
||||
|
||||
function findDependencyCycle(gates: readonly Gate[]): string[] | undefined {
|
||||
const byId = new Map(gates.map(gate => [gate.id, gate]))
|
||||
const complete = new Set<string>()
|
||||
const active = new Map<string, number>()
|
||||
const path: string[] = []
|
||||
|
||||
const visit = (id: string): string[] | undefined => {
|
||||
if (complete.has(id)) return undefined
|
||||
const cycleStart = active.get(id)
|
||||
if (cycleStart !== undefined) return [...path.slice(cycleStart), id]
|
||||
const gate = byId.get(id)
|
||||
if (gate === undefined) return undefined
|
||||
|
||||
active.set(id, path.length)
|
||||
path.push(id)
|
||||
for (const dependency of gate.needs ?? []) {
|
||||
const cycle = visit(dependency)
|
||||
if (cycle !== undefined) return cycle
|
||||
}
|
||||
path.pop()
|
||||
active.delete(id)
|
||||
complete.add(id)
|
||||
return undefined
|
||||
}
|
||||
|
||||
for (const gate of gates) {
|
||||
const cycle = visit(gate.id)
|
||||
if (cycle !== undefined) return cycle
|
||||
}
|
||||
return undefined
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate and run one aggregate before the injected executor can start a child.
|
||||
* @param gates - complete aggregate to execute.
|
||||
* @param maxActive - maximum concurrent child count.
|
||||
* @param execute - child-process executor.
|
||||
* @param observe - result observer invoked when each gate settles.
|
||||
* @returns results in aggregate order.
|
||||
*/
|
||||
export async function runGates(
|
||||
gates: Gate[],
|
||||
maxActive: number,
|
||||
execute: GateExecutor,
|
||||
observe: ResultObserver = () => {},
|
||||
): Promise<GateResult[]> {
|
||||
validateGateGraph(gates)
|
||||
if (!Number.isSafeInteger(maxActive) || maxActive < 1) {
|
||||
throw new Error(`run-gates: max concurrency must be a positive integer, got ${JSON.stringify(maxActive)}.`)
|
||||
}
|
||||
const states = new Map<string, GateState>(gates.map(gate => [gate.id, 'pending']))
|
||||
const results = new Map<string, GateResult>()
|
||||
const running: RunningGate[] = []
|
||||
|
||||
for (;;) {
|
||||
let madeProgress = false
|
||||
while (running.length < maxActive) {
|
||||
const ready = allGates.find(gate => states.get(gate.id) === 'pending' && dependenciesPassed(gate, states))
|
||||
const ready = gates.find(gate => states.get(gate.id) === 'pending' && dependenciesPassed(gate, states))
|
||||
if (ready === undefined) break
|
||||
states.set(ready.id, 'running')
|
||||
running.push({ gate: ready, promise: runGate(ready) })
|
||||
running.push({ gate: ready, promise: execute(ready) })
|
||||
console.log(`run-gates: start ${ready.label}`)
|
||||
madeProgress = true
|
||||
}
|
||||
|
||||
if (running.length === 0) {
|
||||
const pending = allGates.filter(gate => states.get(gate.id) === 'pending')
|
||||
for (const gate of pending) {
|
||||
const failedDeps = (gate.needs ?? []).filter(id => states.get(id) !== 'passed')
|
||||
let pending = gates.filter(gate => states.get(gate.id) === 'pending')
|
||||
while (pending.length > 0) {
|
||||
const gate = pending.find(item => (item.needs ?? []).some((id) => {
|
||||
const state = states.get(id)
|
||||
return state === 'failed' || state === 'skipped'
|
||||
}))
|
||||
if (gate === undefined) throw new Error('run-gates: validated graph stalled without a failed dependency.')
|
||||
const failedDeps = (gate.needs ?? []).filter((id) => {
|
||||
const state = states.get(id)
|
||||
return state === 'failed' || state === 'skipped'
|
||||
})
|
||||
const result: GateResult = {
|
||||
gate,
|
||||
status: 'skipped',
|
||||
durationMs: 0,
|
||||
stdout: '',
|
||||
stderr: '',
|
||||
output: [],
|
||||
exitCode: null,
|
||||
signalCode: null,
|
||||
error: `dependency failed or skipped: ${failedDeps.join(', ')}`,
|
||||
}
|
||||
states.set(gate.id, 'skipped')
|
||||
results.set(gate.id, result)
|
||||
printResult(result)
|
||||
observe(result)
|
||||
pending = pending.filter(item => item !== gate)
|
||||
}
|
||||
break
|
||||
}
|
||||
@@ -533,29 +660,35 @@ async function runGates(allGates: Gate[], maxActive: number): Promise<GateResult
|
||||
running.splice(running.indexOf(settled.item), 1)
|
||||
states.set(settled.item.gate.id, settled.result.status)
|
||||
results.set(settled.item.gate.id, settled.result)
|
||||
printResult(settled.result)
|
||||
observe(settled.result)
|
||||
}
|
||||
}
|
||||
|
||||
return allGates.map((gate) => {
|
||||
return gates.map((gate) => {
|
||||
const result = results.get(gate.id)
|
||||
if (result === undefined) throw new Error(`run-gates: missing result for ${gate.id}.`)
|
||||
return result
|
||||
})
|
||||
}
|
||||
|
||||
function dependenciesPassed(gate: Gate, states: Map<string, GateStatus>): boolean {
|
||||
function dependenciesPassed(gate: Gate, states: Map<string, GateState>): boolean {
|
||||
return (gate.needs ?? []).every(id => states.get(id) === 'passed')
|
||||
}
|
||||
|
||||
async function runGate(gate: Gate): Promise<GateResult> {
|
||||
/**
|
||||
* Execute one gate through the real shell-free child-process boundary.
|
||||
* @param gate - command and scheduler environment to execute.
|
||||
* @returns the complete process outcome.
|
||||
*/
|
||||
export async function runGate(gate: Gate): Promise<GateResult> {
|
||||
const started = performance.now()
|
||||
let stdout = ''
|
||||
let stderr = ''
|
||||
const output: GateOutputChunk[] = []
|
||||
let spawnError: string | undefined
|
||||
|
||||
const exitCode = await new Promise<number | null>((resolveExit) => {
|
||||
const outcome = await new Promise<{
|
||||
exitCode: number | null
|
||||
signalCode: NodeJS.Signals | null
|
||||
}>((resolveExit) => {
|
||||
const child = spawn(gate.command, gate.args, {
|
||||
cwd: root,
|
||||
env: { ...process.env, ...gate.env },
|
||||
@@ -564,47 +697,50 @@ async function runGate(gate: Gate): Promise<GateResult> {
|
||||
child.stdout.setEncoding('utf8')
|
||||
child.stderr.setEncoding('utf8')
|
||||
child.stdout.on('data', (chunk: string) => {
|
||||
stdout += chunk
|
||||
output.push({ stream: 'stdout', text: chunk })
|
||||
})
|
||||
child.stderr.on('data', (chunk: string) => {
|
||||
stderr += chunk
|
||||
output.push({ stream: 'stderr', text: chunk })
|
||||
})
|
||||
child.on('error', (error) => {
|
||||
spawnError = `failed to start command: ${error.message}`
|
||||
resolveExit(null)
|
||||
resolveExit({ exitCode: null, signalCode: null })
|
||||
})
|
||||
child.on('close', resolveExit)
|
||||
if (gate.input !== undefined) child.stdin.end(gate.input)
|
||||
else child.stdin.end()
|
||||
child.on('close', (exitCode, signalCode) => {
|
||||
resolveExit({ exitCode, signalCode })
|
||||
})
|
||||
child.stdin.end()
|
||||
})
|
||||
const { exitCode, signalCode } = outcome
|
||||
|
||||
let status: GateStatus = exitCode === 0 && spawnError === undefined ? 'passed' : 'failed'
|
||||
let error = spawnError
|
||||
if (status === 'passed' && gate.verify !== undefined) {
|
||||
try {
|
||||
await gate.verify({ gate, status, durationMs: performance.now() - started, stdout, stderr, output, exitCode })
|
||||
} catch (verifyError: unknown) {
|
||||
status = 'failed'
|
||||
error = verifyError instanceof Error ? verifyError.message : String(verifyError)
|
||||
}
|
||||
}
|
||||
|
||||
const status: GateResultStatus = exitCode === 0 && signalCode === null && spawnError === undefined ? 'passed' : 'failed'
|
||||
const result: GateResult = {
|
||||
gate,
|
||||
status,
|
||||
durationMs: performance.now() - started,
|
||||
stdout,
|
||||
stderr,
|
||||
output,
|
||||
exitCode,
|
||||
signalCode,
|
||||
}
|
||||
if (error !== undefined) result.error = error
|
||||
if (spawnError !== undefined) result.error = spawnError
|
||||
return result
|
||||
}
|
||||
|
||||
/**
|
||||
* Format every independently observed failure fact for the aggregate summary.
|
||||
* @param result - unsuccessful gate result.
|
||||
* @returns error, exit, and signal facts without allowing one to hide another.
|
||||
*/
|
||||
export function formatGateResultReason(result: GateResult): string {
|
||||
const facts: string[] = []
|
||||
if (result.error !== undefined) facts.push(result.error)
|
||||
if (result.exitCode !== null) facts.push(`exit ${result.exitCode}`)
|
||||
if (result.signalCode !== null) facts.push(`signal ${result.signalCode}`)
|
||||
return facts.length === 0 ? 'no exit code or signal' : facts.join(', ')
|
||||
}
|
||||
|
||||
function printResult(result: GateResult): void {
|
||||
const verbose = process.env.DSH_GATE_VERBOSE === '1'
|
||||
const seconds = (result.durationMs / 1000).toFixed(2)
|
||||
if (result.status === 'passed' && !verbose) {
|
||||
console.log(`run-gates: PASS ${result.gate.label} (${seconds}s)`)
|
||||
@@ -614,9 +750,11 @@ function printResult(result: GateResult): void {
|
||||
const heading = `${result.status.toUpperCase()} ${result.gate.label} (${seconds}s)`
|
||||
const writeHeading = result.status === 'passed' ? console.log : console.error
|
||||
writeHeading(`\n== ${heading} ==`)
|
||||
if (result.status !== 'passed') console.error(`command: ${result.gate.displayCommand}`)
|
||||
if (result.status !== 'passed') {
|
||||
console.error(`command: ${result.gate.displayCommand}`)
|
||||
console.error(`outcome: ${formatGateResultReason(result)}`)
|
||||
}
|
||||
printOutput(result.output)
|
||||
if (result.error !== undefined) console.error(result.error)
|
||||
}
|
||||
|
||||
function printSummary(results: GateResult[], durationMs: number): void {
|
||||
@@ -632,7 +770,7 @@ function printSummary(results: GateResult[], durationMs: number): void {
|
||||
console.error('run-gates: unsuccessful gates:')
|
||||
for (const result of unsuccessful) {
|
||||
const duration = (result.durationMs / 1000).toFixed(2)
|
||||
const reason = result.error ?? (result.exitCode === null ? 'no exit code' : `exit ${result.exitCode}`)
|
||||
const reason = formatGateResultReason(result)
|
||||
const disposition = result.gate.allowFailure === true ? 'NON-BLOCKING ' : ''
|
||||
console.error(` - ${disposition}${result.status.toUpperCase()} ${result.gate.label} (${duration}s, ${reason})`)
|
||||
console.error(` ${result.gate.displayCommand}`)
|
||||
|
||||
Reference in New Issue
Block a user