mirror of
https://github.com/deepseek-ai/deepseek-harness
synced 2026-08-15 21:04:50 +00:00
Merge origin/master into worktree/ci-under-minute
This commit is contained in:
@@ -20,7 +20,7 @@ The build gate makes the hook self-contained from a clean worktree. `publint`, `
|
||||
|
||||
[scripts/publint-all.ts](../../../../scripts/publint-all.ts) discovers the package list from `packages/<group>/<pkg>` and calls publint's supported API against an in-memory view of each manifest's declared publication files plus npm's mandatory metadata files. That keeps unpublished workspace files invisible to publint without a package-manager subprocess per package. A worker pool sized from `availableParallelism()` bounds parallel file loading and linting; `DSH_PUBLINT_CONCURRENCY` can cap or raise it, and results print in deterministic package order.
|
||||
|
||||
The aggregate package scripts remain the source of truth for ad hoc local runs. The scheduler is a parallel execution plan over their member gates, not a replacement vocabulary.
|
||||
The per-gate package scripts remain the vocabulary for ad hoc local runs. `hygiene` stays an aggregate `&&` chain the scheduler mirrors, while `doc-sync` has since moved its member list into the scheduler itself ([doc-sync through the gate scheduler](2026-07-21-doc-sync-through-gate-scheduler.md)).
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
2026-07-21-doc-sync-through-gate-scheduler.md: b79df2dd7d3515cb0434ac672f7f87c3271d900b
|
||||
2026-07-21-doc-sync-through-gate-scheduler.zh.md: 9395244e3c7700166ad87c49219073210c66bc7e
|
||||
@@ -0,0 +1,25 @@
|
||||
# Agent Note: doc-sync through the gate scheduler
|
||||
|
||||
Status: implemented
|
||||
|
||||
English | [中文](2026-07-21-doc-sync-through-gate-scheduler.zh.md)
|
||||
|
||||
## Problem
|
||||
|
||||
`pnpm run doc-sync` was a `&&` chain of 24 `pnpm run` subcommands. Each link paid a full pnpm wrapper start (workspace resolution, script lookup, tsx boot) before its script ran; measured on a development host, the 24 script bodies together finish in about 34 seconds while the chained form takes around 3 minutes, and the wrapper stall reproduces on local disk, so every developer and CI lane pays it, not just network-filesystem checkouts. The chain also ran serially even though the member gates are read-only and independent, and it silently drifted from [scripts/run-gates.ts](../../../../scripts/run-gates.ts): `verify-cordis-api` joined the chain when the runtime API catalog landed but was never added to `docSyncLeafGates`, so CI and pre-push never enforced that catalog's freshness.
|
||||
|
||||
## Decision
|
||||
|
||||
`doc-sync` in `package.json` now delegates to the existing bounded scheduler — `tsx scripts/run-gates.ts doc-sync` — the same way `check:pre-push` and the `check:ci:*` scripts already do ([parallel pre-push gates](2026-07-06-parallel-pre-push-gates.md), [parallel GitHub CI gates](2026-07-06-parallel-github-ci-gates.md)). The new `doc-sync` mode expands to exactly `docSyncLeafGates()`, making the leaf list in `run-gates.ts` the single source of truth for the member set; the chain that could drift from it is gone. Like `pre-push`, the mode caps default concurrency at four workers because several doc gates each build a full `ts.Program`; `DSH_GATE_CONCURRENCY` still overrides.
|
||||
|
||||
The drift this consolidation surfaced is fixed in the same change: `docSyncLeafGates` gains the missing `verify-cordis-api` leaf, so CI and pre-push now gate the generated runtime API catalog alongside the other generated docs.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **Keep the `&&` chain and only fix the missing leaf** — repairs today's drift but keeps two member lists that will drift again, and keeps the 24 serial pnpm wrapper starts.
|
||||
- **A dedicated `scripts/doc-sync.ts` importing each verify module in one process** — saves even the per-gate tsx boot, but requires refactoring all 24 scripts from run-at-import to callable entry points and loses the scheduler's per-gate timing, isolation, and failure grouping; the wrapper start the scheduler already avoids is the dominant cost.
|
||||
- **Shell loop over `tsx scripts/*.ts`** — avoids pnpm wrapper starts cheaply but adds a second execution vocabulary next to the scheduler CI already uses, with none of its scheduling or reporting.
|
||||
|
||||
## Consequences
|
||||
|
||||
One `pnpm run doc-sync` now costs one pnpm wrapper start plus the slowest dependency chain of member gates instead of 24 wrapper starts plus the sum of all members. Adding a doc gate is one edit in `docSyncLeafGates` (plus the package script itself for ad hoc runs); `package.json` keeps the per-gate `verify-*` scripts as the vocabulary for running one gate by hand. The scheduler prints per-gate timing, so a slow doc-sync points at the gate that dominates. `pnpm run doc-sync` output is now interleaved scheduler output rather than sequential per-command output; anything parsing that output must key on the `run-gates:` summary lines.
|
||||
@@ -0,0 +1,25 @@
|
||||
# Agent Note: doc-sync 走门禁调度器
|
||||
|
||||
Status: implemented
|
||||
|
||||
[English](2026-07-21-doc-sync-through-gate-scheduler.md) | 中文
|
||||
|
||||
## 问题
|
||||
|
||||
`pnpm run doc-sync` 原本是把 24 个 `pnpm run` 子命令用 `&&` 串起来的链。每一环都要先付一次完整的 pnpm 包装层启动(workspace 解析、脚本查找、tsx 启动)才轮到脚本本体;在开发机上实测,24 个脚本本体合计约 34 秒即可跑完,而链式形态耗时约 3 分钟,且包装层的停顿在本地磁盘上同样复现,因此每位开发者和每条 CI 车道都在付这笔开销,并非只有网络文件系统上的检出受影响。这条链还是串行执行的,尽管各成员门禁只读且相互独立;它也在悄悄偏离 [scripts/run-gates.ts](../../../../scripts/run-gates.ts):运行时 API 目录落地时 `verify-cordis-api` 加入了链,却从未加进 `docSyncLeafGates`,导致 CI 和 pre-push 从未把关该目录的新鲜度。
|
||||
|
||||
## 决策
|
||||
|
||||
`package.json` 中的 `doc-sync` 现在委托给既有的有界调度器——`tsx scripts/run-gates.ts doc-sync`——与 `check:pre-push` 和各 `check:ci:*` 脚本的做法一致([并行 pre-push 门禁](2026-07-06-parallel-pre-push-gates.md)、[并行 GitHub CI 门禁](2026-07-06-parallel-github-ci-gates.md))。新增的 `doc-sync` 模式恰好展开为 `docSyncLeafGates()`,使 `run-gates.ts` 里的叶子列表成为成员集合的唯一真源;那条可能与之漂移的链不复存在。与 `pre-push` 一样,该模式把默认并发上限设为四个 worker,因为多个文档门禁各自要构建完整的 `ts.Program`;`DSH_GATE_CONCURRENCY` 仍可覆盖。
|
||||
|
||||
这次整合暴露出的漂移在同一变更中修复:`docSyncLeafGates` 补上缺失的 `verify-cordis-api` 叶子,CI 和 pre-push 从此与其他生成文档一起把关生成的运行时 API 目录。
|
||||
|
||||
## 考虑过的替代方案
|
||||
|
||||
- **保留 `&&` 链,只补缺失的叶子**——能修好今天的漂移,但保留了两份还会再漂移的成员列表,也保留了 24 次串行的 pnpm 包装层启动。
|
||||
- **专门的 `scripts/doc-sync.ts` 在单进程内 import 各校验模块**——连每个门禁的 tsx 启动也能省掉,但需要把全部 24 个脚本从 import 即执行改造成可调用入口,还会失去调度器的按门禁计时、隔离和失败分组;而调度器已经避免的包装层启动才是开销的大头。
|
||||
- **用 shell 循环跑 `tsx scripts/*.ts`**——以低成本避开 pnpm 包装层启动,却在 CI 已经使用的调度器旁边增加了第二套执行词汇,且没有它的任何调度与报告能力。
|
||||
|
||||
## 结果
|
||||
|
||||
一次 `pnpm run doc-sync` 的成本从 24 次包装层启动加全部成员之和,变为一次包装层启动加成员门禁中最慢的依赖链。新增文档门禁只需在 `docSyncLeafGates` 改一处(外加 package script 本身以便手工单独运行);`package.json` 保留各 `verify-*` 脚本作为手工运行单个门禁的词汇。调度器输出按门禁计时,doc-sync 变慢时能直接指向占大头的门禁。`pnpm run doc-sync` 的输出从逐命令顺序输出变为调度器的交错输出;解析该输出的工具必须以 `run-gates:` 摘要行为准。
|
||||
@@ -57,7 +57,7 @@ pnpm run lint
|
||||
pnpm run duplication # cross-file TypeScript clone detection
|
||||
pnpm run build # tsc emits lib/types, tsdown bundles runtime
|
||||
pnpm run hygiene # knip + publint + workspace constraints + NodeNext consumer check
|
||||
pnpm run doc-sync # all documentation gates; see the doc-sync script in package.json
|
||||
pnpm run doc-sync # all documentation gates; see the doc-sync leaf list in scripts/run-gates.ts
|
||||
pnpm run website:build # VitePress build (doubles as the site's dead-link check)
|
||||
pnpm run demo:headless "task" # one-shot agent (needs DEEPSEEK_API_KEY)
|
||||
pnpm run demo:tui # full-screen TUI coding agent (needs DEEPSEEK_API_KEY)
|
||||
|
||||
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
development.md: 3327d094a31ad9af62a23c9562cdfa03218961a5
|
||||
development.zh.md: cb1cb86f9f3a34c455844467a20dfdfd08e15098
|
||||
development.md: f0db7fbcb4a9df98e83d6c1edd5610e5cc4dd517
|
||||
development.zh.md: 62e16479a49d5548e1fbd773dabca5bd741a24fe
|
||||
|
||||
@@ -90,7 +90,7 @@ pnpm run verify-md-wrap # fail on hard-wrapped prose paragraphs in docs/README
|
||||
pnpm run verify-mermaid # fail if a ```mermaid diagram has invalid Mermaid syntax
|
||||
pnpm run verify-type-equiv # fail if a ```ts type-equiv doc block drifts from its source type
|
||||
pnpm run verify-doc-budgets # fail if a budgeted standing doc exceeds its word ceiling
|
||||
pnpm run doc-sync # all Markdown/doc gates; see the doc-sync script in package.json for the full list
|
||||
pnpm run doc-sync # all Markdown/doc gates, scheduled concurrently; the doc-sync leaf list in scripts/run-gates.ts is the full list
|
||||
pnpm run gen-module-graph # regenerate docs/module-graph.md from package peerDeps
|
||||
pnpm run verify-module-graph # fail if docs/module-graph.md is stale
|
||||
pnpm run build # emit lib/types intermediates, then bundle lib/index.* runtime files
|
||||
|
||||
@@ -90,7 +90,7 @@ pnpm run verify-md-wrap # fail on hard-wrapped prose paragraphs in docs/README
|
||||
pnpm run verify-mermaid # fail if a ```mermaid diagram has invalid Mermaid syntax
|
||||
pnpm run verify-type-equiv # fail if a ```ts type-equiv doc block drifts from its source type
|
||||
pnpm run verify-doc-budgets # fail if a budgeted standing doc exceeds its word ceiling
|
||||
pnpm run doc-sync # all Markdown/doc gates; see the doc-sync script in package.json for the full list
|
||||
pnpm run doc-sync # all Markdown/doc gates, scheduled concurrently; the doc-sync leaf list in scripts/run-gates.ts is the full list
|
||||
pnpm run gen-module-graph # regenerate docs/module-graph.md from package peerDeps
|
||||
pnpm run verify-module-graph # fail if docs/module-graph.md is stale
|
||||
pnpm run build # emit lib/types intermediates, then bundle lib/index.* runtime files
|
||||
|
||||
@@ -78,7 +78,7 @@
|
||||
"verify-scoped-events": "tsx scripts/gen-scoped-events.ts --check",
|
||||
"verify-module-graph": "tsx scripts/gen-module-graph.ts --check",
|
||||
"constraints": "tsx scripts/check-workspace-constraints.ts",
|
||||
"doc-sync": "pnpm run doc-typecheck && pnpm run verify-cordis-catalog && pnpm run verify-cordis-api && pnpm run verify-export-jsdoc && pnpm run verify-tool-catalog && pnpm run verify-config-catalog && pnpm run verify-persistence-catalog && pnpm run verify-doc-graphs && pnpm run verify-scoped-events && pnpm run verify-md-wrap && pnpm run verify-md-links && pnpm run verify-doc-refs && pnpm run verify-package-paths && pnpm run verify-package-readme-model-experience && pnpm run verify-mermaid && pnpm run verify-agent-note-classification && pnpm run verify-agent-note-format && pnpm run verify-type-equiv && pnpm run verify-translation-prompt && pnpm run verify-translation-pairing && pnpm run verify-doc-budgets && pnpm run verify-package-readme-limitations && pnpm run docs:check",
|
||||
"doc-sync": "tsx scripts/run-gates.ts doc-sync",
|
||||
"hygiene": "pnpm run knip && pnpm run publint && pnpm run constraints && pnpm run verify-package-invariants && pnpm run verify-built-package-invariants && pnpm run verify-cordis-config && pnpm run verify-node-next-types && pnpm run verify-runtime-closure",
|
||||
"demo:headless": "node --expose-internals --import tsx packages/examples/cli-demo/src/bin.ts --config examples/headless-agent/cordis.yml",
|
||||
"demo:tui": "node --expose-internals --import tsx packages/examples/tui-demo/src/bin.ts examples/tui-agent/cordis.yml",
|
||||
|
||||
@@ -20,6 +20,7 @@ type Mode =
|
||||
| 'ci-artifacts'
|
||||
| 'node-compat'
|
||||
| 'pre-push'
|
||||
| 'doc-sync'
|
||||
type GateStatus = 'pending' | 'running' | 'passed' | 'failed' | 'skipped'
|
||||
|
||||
interface Gate {
|
||||
@@ -89,21 +90,25 @@ function parseMode(raw: string | undefined): Mode {
|
||||
case 'ci-artifacts':
|
||||
case 'node-compat':
|
||||
case 'pre-push':
|
||||
case 'doc-sync':
|
||||
return raw
|
||||
default:
|
||||
throw new Error(
|
||||
`run-gates: expected mode ci-primary | ci-static | ci-lint | ci-coverage | ci-snapshot | ci-artifacts | node-compat | pre-push, got ${JSON.stringify(raw)}.`,
|
||||
`run-gates: expected mode ci-primary | ci-static | ci-lint | ci-coverage | ci-snapshot | ci-artifacts | node-compat | pre-push | doc-sync, got ${JSON.stringify(raw)}.`,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
function defaultConcurrency(selectedMode: Mode, total: number): ConcurrencyDefault {
|
||||
const available = availableParallelism()
|
||||
const modeLimit = selectedMode === 'pre-push' ? Math.min(4, available) : available
|
||||
// Local modes cap workers: several doc gates each build a full ts.Program,
|
||||
// so an uncapped default on a large host trades wall clock for memory blowups.
|
||||
const localCap = selectedMode === 'pre-push' || selectedMode === 'doc-sync'
|
||||
const modeLimit = localCap ? Math.min(4, available) : available
|
||||
return {
|
||||
workers: Math.min(total, modeLimit),
|
||||
source: selectedMode === 'pre-push'
|
||||
? `${available} available CPU(s), pre-push cap 4`
|
||||
source: localCap
|
||||
? `${available} available CPU(s), ${selectedMode} cap 4`
|
||||
: `${available} available CPU(s)`,
|
||||
}
|
||||
}
|
||||
@@ -199,6 +204,8 @@ function gatesForMode(selected: Mode): Gate[] {
|
||||
}),
|
||||
pnpmScript('module-graph', 'verify-module-graph', { label: 'module graph' }),
|
||||
]
|
||||
case 'doc-sync':
|
||||
return docSyncLeafGates()
|
||||
}
|
||||
}
|
||||
|
||||
@@ -363,6 +370,7 @@ function docSyncLeafGates(options: {
|
||||
return [
|
||||
pnpmScript('doc-typecheck', 'doc-typecheck', docTypecheckOptions),
|
||||
pnpmScript('cordis-catalog', 'verify-cordis-catalog', { label: 'cordis catalog' }),
|
||||
pnpmScript('cordis-api', 'verify-cordis-api', { label: 'cordis api' }),
|
||||
pnpmScript('export-jsdoc', 'verify-export-jsdoc', { label: 'export jsdoc' }),
|
||||
pnpmScript('tool-catalog', 'verify-tool-catalog', { label: 'tool catalog' }),
|
||||
pnpmScript('config-catalog', 'verify-config-catalog', { label: 'config catalog' }),
|
||||
|
||||
Reference in New Issue
Block a user