mirror of
https://github.com/deepseek-ai/deepseek-harness
synced 2026-08-15 21:04:50 +00:00
Merge pull request #676 from deepseek-harness/nih-imp-gates
refactor(scripts): consolidate gate scripts on mdast fences, parseArgs, and globSync
This commit is contained in:
Tianyi Cui
GitHub
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
2026-07-26-consolidate-gate-scripts-on-existing-deps.md: 0bf32e01ea407e2718f8ec39ca962587a37df9cc
|
||||
2026-07-26-consolidate-gate-scripts-on-existing-deps.zh.md: ba9f157a61ffdc415acf9b2a61857026fc2c8bf1
|
||||
2026-07-26-consolidate-gate-scripts-on-existing-deps.md: 6370c8f92eff7296327e941e698ec4f733100bb2
|
||||
2026-07-26-consolidate-gate-scripts-on-existing-deps.zh.md: 3587c92e0e9655d8b38d24d184c1c68c44b131d4
|
||||
@@ -0,0 +1,33 @@
|
||||
# Agent Note: Consolidate gate scripts on already-present deps and builtins
|
||||
|
||||
Status: implemented
|
||||
|
||||
English | [中文](2026-07-26-consolidate-gate-scripts-on-existing-deps.zh.md)
|
||||
|
||||
## Problem
|
||||
|
||||
The `scripts/` gates mostly used the right tools (`node:fs` `globSync` in 15+ gates, mdast/micromark in the markdown gates), but a handful of stragglers hand-rolled what a sibling gate already did with an existing dependency or builtin:
|
||||
|
||||
- **Duplicated fence scanners.** `scripts/md-fences.ts` (~55 lines, consumed by `doc-typecheck.ts`) and `extractEquivBlocks` in `scripts/verify-type-equiv.ts` (~39 lines) were two copies of the same regex line-scanner for fenced code blocks, while `scripts/verify-mermaid.ts` already extracted fences by visiting mdast `code` nodes — and `markdownProseLines` in `scripts/markdown.ts` itself parsed to mdast but then hand-tracked fence state with a second regex. The regex scanners only recognized backtick fences at column 0, so they silently disagreed with the mdast-based gates on tilde and indented fences.
|
||||
- **Hand-rolled argv parsing.** `parseOptions` in `scripts/publint-all.ts` and its near-identical copy in `scripts/verify-built-package-invariants.mjs` (~26 lines) stepped argv indexes manually, while sibling scripts (`verify-runtime-closure.ts`, `build-exe-for-python-sdk.ts`, `packages/sdk/scripts/src/args.ts`) already used the `node:util` `parseArgs` builtin.
|
||||
- **Hand-rolled directory walks.** Five sites re-derived nested `readdirSync` walks that `globSync` covers: `verify-runtime-closure.ts` (packages + vendor manifests), `dev-web.ts` `discoverPluginDirs`, `verify-package-paths.ts` `realPackageNames`, `verify-client-domain-graph.ts` `listSources`, and `publint-all.ts` `addPath` (~55–65 lines total). `scripts/package-invariants.ts` shows the one-line `globSync` template.
|
||||
|
||||
No new dependency was needed anywhere; every replacement is an existing devDep or a Node builtin.
|
||||
|
||||
## Decision
|
||||
|
||||
- A shared mdast fence helper, `markdownFences` in `scripts/markdown.ts`, visits `code` nodes for the language, full info string, body, and 1-based opening-fence line; `doc-typecheck.ts` and `verify-type-equiv.ts` extract fences through it. `md-fences.ts` and the duplicated `extractEquivBlocks` scanner are deleted, and `markdownProseLines` derives fenced lines from the parsed `code` nodes' positions instead of a second regex.
|
||||
- Both CLIs parse argv via `parseArgs`; unknown options and missing values still fail loud, with `parseArgs`'s own error text instead of the bespoke usage strings.
|
||||
- The five straggler walks use `globSync`. The walks in `check-workspace-constraints.ts` and `clean.ts` stay: they need dirent-level detail to diagnose malformed trees, which glob-by-pattern cannot report.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **A new glob/walking dependency (`tinyglobby`, `fdir`).** Rejected: the builtin already won repo-wide; these were stragglers, not a gap.
|
||||
- **`p-map` for `publint-all.ts`'s ~19-line ordered worker pool.** Deliberately left out: one new devDep for one small deletion is at the edge of the [dependency policy](../process/2026-07-26-dependencies-over-hand-rolling.md) bar, and the pool's requirements (bounded workers, deterministic order, env override) are documented in the [parallel-gates note](../process/2026-07-06-parallel-pre-push-gates.md). Fold it in only if `p-map` earns a second consumer.
|
||||
- **Leaving the fence scanners.** Rejected: two drifting copies of a parser beside a third correct implementation is exactly the duplication the shared `markdown.ts` helper exists to prevent, and the column-0-backtick-only limitation was a latent inconsistency between sibling gates.
|
||||
|
||||
## Consequences
|
||||
|
||||
- One fence parser: every markdown gate now classifies fences through mdast, so tilde, indented, and 4-backtick container fences behave identically everywhere. The docs tree contained no fence shape the regex scanners mishandled, so gate results are unchanged on the tree that landed the swap: `pnpm run doc-sync` and each rewritten gate ran before and after with byte-identical output (`doc-typecheck` block/opt-out counts, `verify-type-equiv` match counts, `publint`, `verify-built-package-invariants`, `verify-runtime-closure`, `verify-package-paths`, `verify-client-domain-graph`, and both package-README prose gates).
|
||||
- `verify-type-equiv` still rejects an unterminated type-equivalence fence: mdast silently closes an unterminated block at end-of-file (its comparisons could then pass), so the shared helper reports whether a closing delimiter exists and the gate errors on an unclosed block, preserving the removed scanner's rejection. The `doc-typecheck` scanner never had that error path.
|
||||
- `parseArgs` keeps the last value of a duplicated option instead of erroring — a dev-tool edge case the tests don't pin, accepted in exchange for deleting the two bespoke parsers. (Strict mode still rejects a `--`-prefixed token where a value is expected, matching the replaced parsers.)
|
||||
@@ -0,0 +1,33 @@
|
||||
# Agent Note: 把门禁脚本统一到已有依赖与内置模块上
|
||||
|
||||
Status: implemented
|
||||
|
||||
[English](2026-07-26-consolidate-gate-scripts-on-existing-deps.md) | 中文
|
||||
|
||||
## 问题
|
||||
|
||||
`scripts/` 下的门禁大多本已在用正确的工具（15 个以上的门禁使用 `node:fs` 的 `globSync`，markdown 门禁使用 mdast/micromark），但少数几个掉队的脚本曾手写同类门禁早已用既有依赖或内置模块完成的事情：
|
||||
|
||||
- **重复的围栏扫描器。**`scripts/md-fences.ts`（约 55 行，由 `doc-typecheck.ts` 消费）和 `scripts/verify-type-equiv.ts` 中的 `extractEquivBlocks`（约 39 行）曾是同一个围栏代码块正则行扫描器的两份拷贝，而 `scripts/verify-mermaid.ts` 早已通过访问 mdast `code` 节点来提取代码围栏；`scripts/markdown.ts` 自己的 `markdownProseLines` 也曾先解析成 mdast，再用第二个正则手工跟踪围栏状态。这两个正则扫描器只识别第 0 列的反引号围栏，因此在波浪线围栏和缩进围栏上与基于 mdast 的门禁悄悄不一致。
|
||||
- **手写的 argv 解析。**`scripts/publint-all.ts` 中的 `parseOptions` 和 `scripts/verify-built-package-invariants.mjs` 中与之几乎相同的拷贝（约 26 行）曾手工推进 argv 下标，而同类脚本（`verify-runtime-closure.ts`、`build-exe-for-python-sdk.ts`、`packages/sdk/scripts/src/args.ts`）早已在使用 `node:util` 的内置 `parseArgs`。
|
||||
- **手写的目录遍历。**五处代码曾各自重写 `globSync` 已覆盖的嵌套 `readdirSync` 遍历：`verify-runtime-closure.ts` 对 packages 与 vendor manifest（元数据清单）的扫描、`dev-web.ts` 的 `discoverPluginDirs`、`verify-package-paths.ts` 的 `realPackageNames`、`verify-client-domain-graph.ts` 的 `listSources`，以及 `publint-all.ts` 的 `addPath`（合计约 55–65 行）。`scripts/package-invariants.ts` 展示了一行式的 `globSync` 模板。
|
||||
|
||||
所有替换都不需要引入新依赖；每一处替换用的都是既有的 devDependency 或 Node 内置模块。
|
||||
|
||||
## 决策
|
||||
|
||||
- `scripts/markdown.ts` 中的共享 mdast 围栏辅助函数 `markdownFences` 访问 `code` 节点，读取语言、完整 info string、块体以及以 1 起始的开围栏行号；`doc-typecheck.ts` 和 `verify-type-equiv.ts` 通过它提取代码围栏。`md-fences.ts` 和重复的 `extractEquivBlocks` 扫描器已删除，`markdownProseLines` 也改为从解析出的 `code` 节点位置推导围栏内的行，而不再用第二个正则。
|
||||
- 两个 CLI 都改用 `parseArgs` 解析 argv；未知选项和缺失取值仍然大声失败，只是错误文案换成了 `parseArgs` 自带的文本，而非原先手写的用法字符串。
|
||||
- 那五处掉队的目录遍历改用 `globSync`。`check-workspace-constraints.ts` 和 `clean.ts` 中的遍历保留：它们需要 dirent 级别的细节来诊断结构异常的目录树，按模式匹配的 glob 报告不了这些信息。
|
||||
|
||||
## 曾考虑的替代方案
|
||||
|
||||
- **新的 glob/目录遍历依赖（`tinyglobby`、`fdir`）。**不予采纳：内置模块已在全仓库范围内胜出；这几处只是掉队者，不是能力缺口。
|
||||
- **用 `p-map` 替换 `publint-all.ts` 中约 19 行的有序 worker 池。**刻意未纳入：为一次小删除引入一个新 devDependency，正处在[依赖策略](../process/2026-07-26-dependencies-over-hand-rolling.md)门槛的边缘，而且该池的需求（worker 数量有界、确定性顺序、环境变量覆盖）已记录在[并行 pre-push 门禁决策记录](../process/2026-07-06-parallel-pre-push-gates.md)中。仅当 `p-map` 赢得第二个消费方时再顺带纳入。
|
||||
- **保留这两个围栏扫描器。**不予采纳：在第三个正确实现旁边放着两份逐渐漂移的解析器拷贝，正是共享的 `markdown.ts` 辅助函数要防止的那种重复；「只认第 0 列反引号」的限制也是同类门禁之间的潜在不一致。
|
||||
|
||||
## 后果
|
||||
|
||||
- 只剩一个围栏解析器：所有 markdown 门禁现在都经由 mdast 归类代码围栏，因此波浪线围栏、缩进围栏和四反引号容器围栏在各处的行为完全一致。文档树中不存在正则扫描器处理有误的围栏形态，所以在落地这次替换的代码树上门禁结果不变：`pnpm run doc-sync` 及每个被改写的门禁在改动前后各跑一遍，输出逐字节相同（`doc-typecheck` 的块数/opt-out 计数、`verify-type-equiv` 的匹配计数、`publint`、`verify-built-package-invariants`、`verify-runtime-closure`、`verify-package-paths`、`verify-client-domain-graph`，以及两个包 README 散文门禁）。
|
||||
- `verify-type-equiv` 仍然拒绝未闭合的类型等价围栏：mdast 会在文件末尾静默闭合未闭合的代码块（其比较随后可能通过），因此共享辅助函数会报告闭合定界符是否存在，门禁在块未闭合时报错，保留了被删扫描器的这条拒绝路径。`doc-typecheck` 的扫描器本来就没有这条错误路径。
|
||||
- `parseArgs` 对重复出现的选项保留最后一个值而不报错——一个测试未固定的开发工具边缘用例，作为删除两份手写解析器的交换被接受。（严格模式下，需要取值处遇到以 `--` 开头的 token 仍会拒绝，与被替换的解析器行为一致。）
|
||||
@@ -1,38 +0,0 @@
|
||||
# Agent Note: Consolidate gate scripts on already-present deps and builtins
|
||||
|
||||
Status: proposed
|
||||
|
||||
English | [中文](2026-07-26-consolidate-gate-scripts-on-existing-deps.zh.md)
|
||||
|
||||
## Problem
|
||||
|
||||
The `scripts/` gates mostly use the right tools (`node:fs` `globSync` in 15+ gates, mdast/micromark in the markdown gates), but a handful of stragglers hand-roll what a sibling gate already does with an existing dependency or builtin:
|
||||
|
||||
- **Duplicated fence scanners.** `scripts/md-fences.ts` (~55 lines, consumed by `doc-typecheck.ts`) and `extractEquivBlocks` in `scripts/verify-type-equiv.ts` (~39 lines) are two copies of the same regex line-scanner for fenced code blocks, while `scripts/verify-mermaid.ts` already extracts fences by visiting mdast `code` nodes via the shared `scripts/markdown.ts` helpers — and `markdownProseLines` in `markdown.ts` itself parses to mdast but then hand-tracks fence state with a second regex. The regex scanners only recognize backtick fences at column 0, so they silently disagree with the mdast-based gates on tilde and indented fences.
|
||||
- **Hand-rolled argv parsing.** `parseOptions` in `scripts/publint-all.ts` and its near-identical copy in `scripts/verify-built-package-invariants.mjs` (~26 lines) step argv indexes manually, while sibling scripts (`verify-runtime-closure.ts`, `build-exe-for-python-sdk.ts`, `packages/sdk/scripts/src/args.ts`) already use the `node:util` `parseArgs` builtin.
|
||||
- **Hand-rolled directory walks.** Five sites re-derive nested `readdirSync` walks that `globSync` covers: `verify-runtime-closure.ts` (packages + vendor manifests), `dev-web.ts` `discoverPluginDirs`, `verify-package-paths.ts` `realPackageNames`, `verify-client-domain-graph.ts` `listSources`, and `publint-all.ts` `addPath` (~55–65 lines total). `scripts/package-invariants.ts` shows the one-line `globSync` template.
|
||||
|
||||
No new dependency is needed anywhere; every replacement is an existing devDep or a Node builtin.
|
||||
|
||||
## Proposal
|
||||
|
||||
- Extract a shared ~10–15-line mdast fence helper (visiting `code` nodes for `lang`, `meta`, `value`, `position.start.line`) into `scripts/markdown.ts`; rewrite `doc-typecheck.ts` and `verify-type-equiv.ts` onto it; delete `md-fences.ts` and the duplicated scanner; drop the redundant fence regex in `markdownProseLines`.
|
||||
- Replace both `parseOptions` copies with `parseArgs`.
|
||||
- Replace the five straggler walks with `globSync`. Keep the walks in `check-workspace-constraints.ts` and `clean.ts`: they need dirent-level detail to diagnose malformed trees, which glob-by-pattern cannot report.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
- **A new glob/walking dependency (`tinyglobby`, `fdir`).** Rejected: the builtin already won repo-wide; these are stragglers, not a gap.
|
||||
- **`p-map` for `publint-all.ts`'s ~19-line ordered worker pool.** Deliberately left out: one new devDep for one small deletion is at the edge of the [dependency policy](../../implemented/process/2026-07-26-dependencies-over-hand-rolling.md) bar, and the pool's requirements (bounded workers, deterministic order, env override) are documented in the [parallel-gates note](../../implemented/process/2026-07-06-parallel-pre-push-gates.md). Fold it in only if `p-map` earns a second consumer.
|
||||
- **Leaving the fence scanners.** Rejected: two drifting copies of a parser beside a third correct implementation is exactly the duplication the shared `markdown.ts` helper exists to prevent, and the column-0-backtick-only limitation is a latent inconsistency between sibling gates.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- `md-fences.ts` is gone; `doc-typecheck` and `verify-type-equiv` extract fences through `scripts/markdown.ts`; `pnpm run doc-sync` passes with unchanged results on the current tree (any delta traces to a fence shape the regex scanners mishandled).
|
||||
- Both CLIs parse via `parseArgs`; unknown options still fail loud.
|
||||
- The five walk sites use `globSync`; the gates they feed pass unchanged.
|
||||
|
||||
## Risks
|
||||
|
||||
- Behavioral deltas on pathological markdown: mdast honors tilde/indented fences the regex scanners ignored, so `doc-typecheck`'s opt-out ratio could shift if any stray fence shape exists in the docs tree; verify by running `doc-sync` before/after.
|
||||
- `parseArgs` keeps the last value of a duplicated option instead of erroring — a dev-tool edge case the tests don't pin. (Strict mode still rejects a `--`-prefixed token where a value is expected, matching the current parsers.)
|
||||
@@ -1,38 +0,0 @@
|
||||
# Agent Note: 把门禁脚本统一到已有依赖与内置模块上
|
||||
|
||||
Status: proposed
|
||||
|
||||
[English](2026-07-26-consolidate-gate-scripts-on-existing-deps.md) | 中文
|
||||
|
||||
## 问题
|
||||
|
||||
`scripts/` 下的门禁大多已经在用正确的工具（15 个以上的门禁使用 `node:fs` 的 `globSync`，markdown 门禁使用 mdast/micromark），但少数几个掉队的脚本仍在手写同类门禁早已用既有依赖或内置模块完成的事情：
|
||||
|
||||
- **重复的围栏扫描器。**`scripts/md-fences.ts`（约 55 行，由 `doc-typecheck.ts` 消费）和 `scripts/verify-type-equiv.ts` 中的 `extractEquivBlocks`（约 39 行）是同一个围栏代码块正则行扫描器的两份拷贝，而 `scripts/verify-mermaid.ts` 已经通过共享的 `scripts/markdown.ts` 辅助函数访问 mdast `code` 节点来提取代码围栏；`markdown.ts` 自己的 `markdownProseLines` 也是先解析成 mdast，再用第二个正则手工跟踪围栏状态。这两个正则扫描器只识别第 0 列的反引号围栏，因此在波浪线围栏和缩进围栏上与基于 mdast 的门禁悄悄不一致。
|
||||
- **手写的 argv 解析。**`scripts/publint-all.ts` 中的 `parseOptions` 和 `scripts/verify-built-package-invariants.mjs` 中与之几乎相同的拷贝（约 26 行）手工推进 argv 下标，而同类脚本（`verify-runtime-closure.ts`、`build-exe-for-python-sdk.ts`、`packages/sdk/scripts/src/args.ts`）已经在使用 `node:util` 的内置 `parseArgs`。
|
||||
- **手写的目录遍历。**五处代码各自重写了 `globSync` 已覆盖的嵌套 `readdirSync` 遍历：`verify-runtime-closure.ts` 对 packages 与 vendor manifest（元数据清单）的扫描、`dev-web.ts` 的 `discoverPluginDirs`、`verify-package-paths.ts` 的 `realPackageNames`、`verify-client-domain-graph.ts` 的 `listSources`，以及 `publint-all.ts` 的 `addPath`（合计约 55–65 行）。`scripts/package-invariants.ts` 展示了一行式的 `globSync` 模板。
|
||||
|
||||
所有替换都不需要引入新依赖；每一处替换用的都是既有的 devDependency 或 Node 内置模块。
|
||||
|
||||
## 提案
|
||||
|
||||
- 在 `scripts/markdown.ts` 中提取一个约 10–15 行的共享 mdast 围栏辅助函数（访问 `code` 节点，读取 `lang`、`meta`、`value`、`position.start.line`）；把 `doc-typecheck.ts` 和 `verify-type-equiv.ts` 改写到它上面；删除 `md-fences.ts` 和重复的扫描器；去掉 `markdownProseLines` 中冗余的围栏正则。
|
||||
- 用 `parseArgs` 替换两份 `parseOptions` 拷贝。
|
||||
- 用 `globSync` 替换那五处掉队的目录遍历。保留 `check-workspace-constraints.ts` 和 `clean.ts` 中的遍历：它们需要 dirent 级别的细节来诊断结构异常的目录树，按模式匹配的 glob 报告不了这些信息。
|
||||
|
||||
## 曾考虑的替代方案
|
||||
|
||||
- **新的 glob/目录遍历依赖（`tinyglobby`、`fdir`）。**不予采纳：内置模块已在全仓库范围内胜出；这几处只是掉队者，不是能力缺口。
|
||||
- **用 `p-map` 替换 `publint-all.ts` 中约 19 行的有序 worker 池。**刻意未纳入：为一次小删除引入一个新 devDependency，正处在[依赖策略](../../implemented/process/2026-07-26-dependencies-over-hand-rolling.md)门槛的边缘，而且该池的需求（worker 数量有界、确定性顺序、环境变量覆盖）已记录在[并行 pre-push 门禁决策记录](../../implemented/process/2026-07-06-parallel-pre-push-gates.md)中。仅当 `p-map` 赢得第二个消费方时再顺带纳入。
|
||||
- **保留这两个围栏扫描器。**不予采纳：在第三个正确实现旁边放着两份逐渐漂移的解析器拷贝，正是共享的 `markdown.ts` 辅助函数要防止的那种重复；「只认第 0 列反引号」的限制也是同类门禁之间的潜在不一致。
|
||||
|
||||
## 验收标准
|
||||
|
||||
- `md-fences.ts` 已删除；`doc-typecheck` 与 `verify-type-equiv` 通过 `scripts/markdown.ts` 提取代码围栏；`pnpm run doc-sync` 在当前代码树上通过且结果不变（如有差异，必须能追溯到正则扫描器处理有误的某种围栏形态）。
|
||||
- 两个 CLI 都改用 `parseArgs` 解析；未知选项仍然大声失败。
|
||||
- 五处遍历代码改用 `globSync`；它们供给的门禁保持原样通过。
|
||||
|
||||
## 风险
|
||||
|
||||
- 病态 markdown 上的行为差异：mdast 会承认正则扫描器忽略的波浪线围栏和缩进围栏，因此如果文档树中存在任何零散的此类围栏形态，`doc-typecheck` 的 opt-out 比例可能变化；应在改动前后分别运行 `doc-sync` 加以验证。
|
||||
- `parseArgs` 对重复出现的选项保留最后一个值而不报错——一个测试未固定的开发工具边缘用例。（严格模式下，需要取值处遇到以 `--` 开头的 token 仍会拒绝，与现有解析器行为一致。）
|
||||
@@ -17,8 +17,8 @@
|
||||
* `watch` through API-level inline config (tsdown workspace mode fills inline
|
||||
* keys under each package's file config, and no package config defines it).
|
||||
*/
|
||||
import { readdirSync, readFileSync } from 'node:fs'
|
||||
import { join } from 'node:path'
|
||||
import { globSync, readFileSync } from 'node:fs'
|
||||
import { dirname, join, sep } from 'node:path'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import { build } from 'tsdown'
|
||||
|
||||
@@ -33,20 +33,9 @@ const repoRoot = fileURLToPath(new URL('..', import.meta.url))
|
||||
*/
|
||||
function discoverPluginDirs(): string[] {
|
||||
const dirs: string[] = []
|
||||
for (const group of readdirSync(join(repoRoot, 'packages'), { withFileTypes: true })) {
|
||||
if (!group.isDirectory()) continue
|
||||
for (const pkg of readdirSync(join(repoRoot, 'packages', group.name), { withFileTypes: true })) {
|
||||
if (!pkg.isDirectory()) continue
|
||||
let manifest: { dshClient?: { platform?: unknown } }
|
||||
try {
|
||||
manifest = JSON.parse(
|
||||
readFileSync(join(repoRoot, 'packages', group.name, pkg.name, 'package.json'), 'utf8'),
|
||||
) as { dshClient?: { platform?: unknown } }
|
||||
} catch {
|
||||
continue // no package.json (support dirs, scratch): not a workspace package
|
||||
}
|
||||
if (manifest.dshClient?.platform === 'web') dirs.push(`packages/${group.name}/${pkg.name}`)
|
||||
}
|
||||
for (const manifestPath of globSync('packages/*/*/package.json', { cwd: repoRoot }).sort()) {
|
||||
const manifest = JSON.parse(readFileSync(join(repoRoot, manifestPath), 'utf8')) as { dshClient?: { platform?: unknown } }
|
||||
if (manifest.dshClient?.platform === 'web') dirs.push(dirname(manifestPath).split(sep).join('/'))
|
||||
}
|
||||
return dirs
|
||||
}
|
||||
|
||||
@@ -10,7 +10,7 @@ import { globSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node
|
||||
import { join, relative, resolve } from 'node:path'
|
||||
import ts from 'typescript'
|
||||
import { builtDeclarationPath } from './doc-typecheck-paths.ts'
|
||||
import { extractFences } from './md-fences.ts'
|
||||
import { markdownFences } from './markdown.ts'
|
||||
import { partitionPairedMarkdownDerivatives } from './paired-markdown-derivatives.ts'
|
||||
import { isArchivedAgentNotePath } from './repo-files.ts'
|
||||
|
||||
@@ -46,8 +46,10 @@ const KIND_BY_INFO: Record<string, BlockKind> = {
|
||||
/** Extract every recognized TypeScript fence from one Markdown file. */
|
||||
function extractBlocks(absPath: string): Block[] {
|
||||
const file = relative(root, absPath)
|
||||
return extractFences(absPath, info => KIND_BY_INFO[info] ?? null)
|
||||
.map(f => ({ file, line: f.line, kind: f.kind, code: f.code }))
|
||||
return markdownFences(readFileSync(absPath, 'utf8')).flatMap((fence) => {
|
||||
const kind = KIND_BY_INFO[fence.info]
|
||||
return kind === undefined ? [] : [{ file, line: fence.line, kind, code: fence.code }]
|
||||
})
|
||||
}
|
||||
|
||||
const configHost: ts.ParseConfigFileHost = {
|
||||
|
||||
@@ -21,6 +21,24 @@ export interface MarkdownHeadingLine extends MarkdownProseLine {
|
||||
text: string
|
||||
}
|
||||
|
||||
/** One code block from a parsed Markdown source. */
|
||||
export interface MarkdownFence {
|
||||
/** 1-based source line of the opening fence. */
|
||||
line: number
|
||||
/** Info-string language (its first word), null on a bare or indented block. */
|
||||
lang: string | null
|
||||
/** Full info string (e.g. `ts ignore-check`), '' on a bare or indented block. */
|
||||
info: string
|
||||
/** Block body without the fence delimiters. */
|
||||
code: string
|
||||
/**
|
||||
* Whether a closing fence delimiter terminates the block — mdast silently
|
||||
* closes an unterminated fence at end of file. False on indented
|
||||
* (non-fenced) blocks, whose end line is code.
|
||||
*/
|
||||
closed: boolean
|
||||
}
|
||||
|
||||
/** Parse GitHub-flavored Markdown with the repository's standard extensions. */
|
||||
export function parseMarkdown(source: string): Nodes {
|
||||
return fromMarkdown(source, { extensions: [gfm()], mdastExtensions: [gfmFromMarkdown()] })
|
||||
@@ -38,6 +56,26 @@ export function visitMarkdown(node: Nodes, visitor: (node: Nodes) => boolean | v
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract every parsed code block with its info string, in document order.
|
||||
* @param source - Markdown source to scan.
|
||||
* @returns each block's opening line, language, info string, and body.
|
||||
*/
|
||||
export function markdownFences(source: string): MarkdownFence[] {
|
||||
const lines = source.split('\n')
|
||||
const fences: MarkdownFence[] = []
|
||||
visitMarkdown(parseMarkdown(source), (node) => {
|
||||
if (node.type !== 'code' || node.position === undefined) return
|
||||
const lang = node.lang ?? null
|
||||
const meta = node.meta ?? ''
|
||||
const info = lang === null ? '' : meta === '' ? lang : `${lang} ${meta}`
|
||||
const endLine = lines[node.position.end.line - 1] ?? ''
|
||||
const closed = /^ {0,3}(`{3,}|~{3,})\s*$/.test(endLine)
|
||||
fences.push({ line: node.position.start.line, lang, info, code: node.value, closed })
|
||||
})
|
||||
return fences
|
||||
}
|
||||
|
||||
/** Text a reader sees from one Markdown node; raw HTML itself contributes none. */
|
||||
function renderedText(node: Nodes): string {
|
||||
if (node.type === 'text' || node.type === 'inlineCode') return node.value
|
||||
@@ -115,27 +153,22 @@ function hasRenderedTextOutsideComments(raw: string, ranges: readonly ColumnRang
|
||||
}
|
||||
|
||||
/**
|
||||
* Return source lines outside backtick or tilde fences and HTML comments.
|
||||
* Return source lines outside code blocks and HTML comments.
|
||||
* @param source - Markdown source whose prose should be retained verbatim.
|
||||
* @returns unfenced lines with their original 1-based locations.
|
||||
*/
|
||||
export function markdownProseLines(source: string): MarkdownProseLine[] {
|
||||
let fence: { marker: '`' | '~'; length: number } | undefined
|
||||
const kept: MarkdownProseLine[] = []
|
||||
const rawLines = source.split('\n')
|
||||
const comments = htmlCommentRanges(source, rawLines)
|
||||
const fenced = new Set<number>()
|
||||
visitMarkdown(parseMarkdown(source), (node) => {
|
||||
if (node.type !== 'code' || node.position === undefined) return
|
||||
for (let line = node.position.start.line; line <= node.position.end.line; line += 1) fenced.add(line)
|
||||
})
|
||||
const kept: MarkdownProseLine[] = []
|
||||
rawLines.forEach((raw, i) => {
|
||||
const token = /^ {0,3}(`{3,}|~{3,})/.exec(raw)?.[1]
|
||||
if (token !== undefined) {
|
||||
const marker = token[0] as '`' | '~'
|
||||
if (fence === undefined) {
|
||||
fence = { marker, length: token.length }
|
||||
} else if (marker === fence.marker && token.length >= fence.length) {
|
||||
fence = undefined
|
||||
}
|
||||
return
|
||||
}
|
||||
if (fence === undefined && hasRenderedTextOutsideComments(raw, comments.get(i + 1))) {
|
||||
if (fenced.has(i + 1)) return
|
||||
if (hasRenderedTextOutsideComments(raw, comments.get(i + 1))) {
|
||||
kept.push({ index: i + 1, raw })
|
||||
}
|
||||
})
|
||||
|
||||
@@ -1,55 +0,0 @@
|
||||
/**
|
||||
* Shared fenced-code-block extractor for the Markdown doc gates
|
||||
* (currently `doc-typecheck.ts`; future Markdown gates can share it). One scanner, per-gate
|
||||
* classification: each gate maps a fence info string (` ```ts `,
|
||||
* ` ```yaml ignore-check `, …) to its own kind tag and receives every
|
||||
* classified block with its 1-based opening-fence line.
|
||||
*/
|
||||
|
||||
import { readFileSync } from 'node:fs'
|
||||
|
||||
/** One extracted fenced block, classified by the caller's `classify`. */
|
||||
export interface Fence<K> {
|
||||
/** 1-based line of the opening fence. */
|
||||
line: number
|
||||
kind: K
|
||||
code: string
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract every fenced block of `absPath` whose info string `classify` maps
|
||||
* to a kind. Blocks classified `null` are skipped (their bodies are still
|
||||
* consumed, so an unrelated fence can never leak into a tracked one).
|
||||
*
|
||||
* @param absPath — absolute path of the Markdown file.
|
||||
* @param classify — info string (trimmed, e.g. `ts ignore-check`) → kind, or
|
||||
* null for fences this gate does not track.
|
||||
* @returns the classified blocks in document order.
|
||||
*/
|
||||
export function extractFences<K>(absPath: string, classify: (info: string) => K | null): Fence<K>[] {
|
||||
const lines = readFileSync(absPath, 'utf8').split('\n')
|
||||
const blocks: Fence<K>[] = []
|
||||
let open: { line: number; kind: K; body: string[] } | null = null
|
||||
let skipping = false
|
||||
|
||||
lines.forEach((raw, i) => {
|
||||
const fence = /^```(\s*)(\S.*)?$/.exec(raw)
|
||||
if (!fence) {
|
||||
if (open) open.body.push(raw)
|
||||
return
|
||||
}
|
||||
if (open) {
|
||||
blocks.push({ line: open.line, kind: open.kind, code: open.body.join('\n') })
|
||||
open = null
|
||||
return
|
||||
}
|
||||
if (skipping) {
|
||||
skipping = false
|
||||
return
|
||||
}
|
||||
const kind = classify((fence[2] ?? '').trim())
|
||||
if (kind !== null) open = { line: i + 1, kind, body: [] }
|
||||
else skipping = true
|
||||
})
|
||||
return blocks
|
||||
}
|
||||
@@ -2,19 +2,23 @@
|
||||
|
||||
import {
|
||||
globSync,
|
||||
readFileSync,
|
||||
readdirSync,
|
||||
readFileSync,
|
||||
statSync,
|
||||
} from 'node:fs'
|
||||
import { availableParallelism } from 'node:os'
|
||||
import { dirname, relative, resolve, sep } from 'node:path'
|
||||
import { parseArgs } from 'node:util'
|
||||
import { publint, type Message, type PackFile } from 'publint'
|
||||
import { formatMessage } from 'publint/utils'
|
||||
|
||||
const CONCURRENCY_ENV = 'DSH_PUBLINT_CONCURRENCY'
|
||||
const repositoryRoot = resolve(import.meta.dirname, '..')
|
||||
const options = parseOptions(process.argv.slice(2))
|
||||
const packagesRoot = resolve(options.get('--packages-root') ?? repositoryRoot)
|
||||
const { values: options } = parseArgs({
|
||||
args: process.argv.slice(2),
|
||||
options: { 'packages-root': { type: 'string' } },
|
||||
})
|
||||
const packagesRoot = resolve(options['packages-root'] ?? repositoryRoot)
|
||||
|
||||
interface PackageTarget {
|
||||
path: string
|
||||
@@ -88,7 +92,12 @@ function publicationFiles(target: PackageTarget): PackFile[] {
|
||||
function addPath(path: string, paths: Set<string>): void {
|
||||
const stat = statSync(path)
|
||||
if (stat.isDirectory()) {
|
||||
for (const entry of readdirSync(path)) addPath(resolve(path, entry), paths)
|
||||
// readdirSync, not globSync: `**/*` skips dot-prefixed segments, but npm
|
||||
// pack publishes dotfiles inside included directories, and this view must
|
||||
// match what npm publishes.
|
||||
for (const entry of readdirSync(path, { recursive: true, withFileTypes: true })) {
|
||||
if (entry.isFile()) paths.add(resolve(entry.parentPath, entry.name))
|
||||
}
|
||||
} else if (stat.isFile()) {
|
||||
paths.add(path)
|
||||
}
|
||||
@@ -144,20 +153,6 @@ function printResult(result: PublintResult): void {
|
||||
if (result.status === 'passed' && result.messages.length === 0) console.log('All good!')
|
||||
}
|
||||
|
||||
function parseOptions(args: string[]): Map<string, string> {
|
||||
const parsed = new Map<string, string>()
|
||||
for (let index = 0; index < args.length; index += 2) {
|
||||
const name = args[index]
|
||||
const value = args[index + 1]
|
||||
if (name !== '--packages-root' || value === undefined || value.startsWith('--')) {
|
||||
throw new Error(`publint-all: expected [--packages-root PATH], got ${JSON.stringify(args)}.`)
|
||||
}
|
||||
if (parsed.has(name)) throw new Error(`publint-all: duplicate option ${name}.`)
|
||||
parsed.set(name, value)
|
||||
}
|
||||
return parsed
|
||||
}
|
||||
|
||||
const packages = workspacePackages()
|
||||
const concurrency = publintConcurrency(packages.length)
|
||||
console.log(`publint-all: linting ${packages.length} package(s) with ${concurrency} worker(s).`)
|
||||
|
||||
@@ -13,11 +13,15 @@ import {
|
||||
} from 'node:fs'
|
||||
import { dirname, resolve } from 'node:path'
|
||||
import { pathToFileURL } from 'node:url'
|
||||
import { parseArgs } from 'node:util'
|
||||
|
||||
const repositoryRoot = resolve(import.meta.dirname, '..')
|
||||
const options = parseOptions(process.argv.slice(2))
|
||||
const packagesRoot = resolve(options.get('--packages-root') ?? repositoryRoot)
|
||||
const loaderUrl = options.get('--loader-url')
|
||||
const { values: options } = parseArgs({
|
||||
args: process.argv.slice(2),
|
||||
options: { 'packages-root': { type: 'string' }, 'loader-url': { type: 'string' } },
|
||||
})
|
||||
const packagesRoot = resolve(options['packages-root'] ?? repositoryRoot)
|
||||
const loaderUrl = options['loader-url']
|
||||
?? pathToFileURL(resolve(repositoryRoot, 'vendor/loader/lib/index.js')).href
|
||||
const failures = []
|
||||
const manifests = globSync('packages/*/*/package.json', { cwd: packagesRoot }).sort()
|
||||
@@ -77,21 +81,6 @@ if (failures.length > 0) {
|
||||
|
||||
console.log(`verify-built-package-invariants: ${manifests.length} compiled companion(s) passed plain-Node Loader checks.`)
|
||||
|
||||
function parseOptions(args) {
|
||||
const allowed = new Set(['--packages-root', '--loader-url'])
|
||||
const parsed = new Map()
|
||||
for (let index = 0; index < args.length; index += 2) {
|
||||
const name = args[index]
|
||||
const value = args[index + 1]
|
||||
if (!allowed.has(name) || value === undefined || value.startsWith('--')) {
|
||||
throw new Error(`verify-built-package-invariants: expected [--packages-root PATH] [--loader-url URL], got ${JSON.stringify(args)}.`)
|
||||
}
|
||||
if (parsed.has(name)) throw new Error(`verify-built-package-invariants: duplicate option ${name}.`)
|
||||
parsed.set(name, value)
|
||||
}
|
||||
return parsed
|
||||
}
|
||||
|
||||
function copyDeclaredLibFiles(packageDir, stagedPackageDir, files) {
|
||||
for (const pattern of files) {
|
||||
if (!pattern.startsWith('lib/')) continue
|
||||
|
||||
@@ -14,8 +14,8 @@
|
||||
* pnpm exec tsx scripts/verify-client-domain-graph.ts
|
||||
*/
|
||||
|
||||
import { readdirSync, readFileSync, statSync } from 'node:fs'
|
||||
import { join, resolve } from 'node:path'
|
||||
import { globSync, readdirSync, readFileSync, statSync } from 'node:fs'
|
||||
import { join, resolve, sep } from 'node:path'
|
||||
|
||||
const root = resolve(import.meta.dirname, '..')
|
||||
const CLIENT_DIR = join(root, 'packages/client')
|
||||
@@ -28,15 +28,11 @@ const ASSEMBLY_FILES = new Set(['apply.ts', 'index.ts', 'index.tsx'])
|
||||
interface Violation { file: string; imported: string; reason: string }
|
||||
|
||||
/** Recursively list .ts/.tsx files under dir (relative paths). */
|
||||
function listSources(dir: string, prefix = ''): string[] {
|
||||
const out: string[] = []
|
||||
for (const name of readdirSync(dir)) {
|
||||
const full = join(dir, name)
|
||||
const rel = prefix ? `${prefix}/${name}` : name
|
||||
if (statSync(full).isDirectory()) out.push(...listSources(full, rel))
|
||||
else if (/\.tsx?$/.test(name) && !/\.legacy\./.test(name)) out.push(rel)
|
||||
}
|
||||
return out
|
||||
function listSources(dir: string): string[] {
|
||||
return globSync('**/*.{ts,tsx}', { cwd: dir })
|
||||
.map(rel => rel.split(sep).join('/'))
|
||||
.filter(rel => !/\.legacy\./.test(rel.slice(rel.lastIndexOf('/') + 1)))
|
||||
.sort()
|
||||
}
|
||||
|
||||
/** First path segment of a client-relative file, or '' for top-level files. */
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
* outside the check.
|
||||
*/
|
||||
|
||||
import { existsSync, readdirSync } from 'node:fs'
|
||||
import { existsSync, globSync } from 'node:fs'
|
||||
import { resolve } from 'node:path'
|
||||
import {
|
||||
findReferenceViolations,
|
||||
@@ -41,12 +41,8 @@ const isExcluded = (p: string): boolean =>
|
||||
*/
|
||||
function realPackageNames(): Set<string> {
|
||||
const names = new Set<string>()
|
||||
const pkgRoot = resolve(root, 'packages')
|
||||
for (const group of readdirSync(pkgRoot, { withFileTypes: true })) {
|
||||
if (!group.isDirectory()) continue
|
||||
for (const pkg of readdirSync(resolve(pkgRoot, group.name), { withFileTypes: true })) {
|
||||
if (pkg.isDirectory()) names.add(pkg.name)
|
||||
}
|
||||
for (const pkg of globSync('packages/*/*', { cwd: root, withFileTypes: true })) {
|
||||
if (pkg.isDirectory()) names.add(pkg.name)
|
||||
}
|
||||
return names
|
||||
}
|
||||
|
||||
@@ -3,8 +3,9 @@
|
||||
* peer in its dependency graph. With auto peer installation disabled, a missing
|
||||
* root peer can otherwise fail only when Cordis loads the packaged plugin.
|
||||
*/
|
||||
import { readFile, readdir } from 'node:fs/promises'
|
||||
import { join, resolve } from 'node:path'
|
||||
import { globSync } from 'node:fs'
|
||||
import { readFile } from 'node:fs/promises'
|
||||
import { resolve } from 'node:path'
|
||||
import { parseArgs } from 'node:util'
|
||||
|
||||
interface PackageManifest {
|
||||
@@ -72,15 +73,9 @@ if (failures.length > 0) {
|
||||
console.log(`verify-runtime-closure: ${queue.length} workspace packages form a closed runtime dependency graph.`)
|
||||
|
||||
async function loadWorkspacePackages(): Promise<Map<string, WorkspacePackage>> {
|
||||
const paths: string[] = []
|
||||
for (const group of await childDirectories(join(root, 'packages'))) {
|
||||
for (const packageDir of await childDirectories(join(root, 'packages', group))) {
|
||||
paths.push(join(root, 'packages', group, packageDir, 'package.json'))
|
||||
}
|
||||
}
|
||||
for (const packageDir of await childDirectories(join(root, 'vendor'))) {
|
||||
paths.push(join(root, 'vendor', packageDir, 'package.json'))
|
||||
}
|
||||
const paths = globSync(['packages/*/*/package.json', 'vendor/*/package.json'], { cwd: root })
|
||||
.sort()
|
||||
.map(relative => resolve(root, relative))
|
||||
const result = new Map<string, WorkspacePackage>()
|
||||
for (const path of paths) {
|
||||
const manifest = await loadManifest(path)
|
||||
@@ -89,11 +84,6 @@ async function loadWorkspacePackages(): Promise<Map<string, WorkspacePackage>> {
|
||||
return result
|
||||
}
|
||||
|
||||
async function childDirectories(path: string): Promise<string[]> {
|
||||
const entries = await readdir(path, { withFileTypes: true })
|
||||
return entries.filter(entry => entry.isDirectory()).map(entry => entry.name).sort()
|
||||
}
|
||||
|
||||
async function loadManifest(path: string): Promise<PackageManifest> {
|
||||
return JSON.parse(await readFile(path, 'utf8')) as PackageManifest
|
||||
}
|
||||
|
||||
@@ -11,6 +11,7 @@
|
||||
import { globSync, readFileSync, existsSync } from 'node:fs'
|
||||
import { resolve, sep } from 'node:path'
|
||||
import ts from 'typescript'
|
||||
import { markdownFences } from './markdown.ts'
|
||||
import { partitionPairedMarkdownDerivatives } from './paired-markdown-derivatives.ts'
|
||||
import { isArchivedAgentNotePath } from './repo-files.ts'
|
||||
|
||||
@@ -81,42 +82,27 @@ function blockSymbol(code: string): string | null {
|
||||
|
||||
/** Extract every source-equivalence block from one Markdown file. */
|
||||
function extractEquivBlocks(docRel: string): EquivBlock[] {
|
||||
const text = readFileSync(resolve(root, docRel), 'utf8')
|
||||
const lines = text.split('\n')
|
||||
const blocks: EquivBlock[] = []
|
||||
let open: { line: number; body: string[]; projection?: 'public-api' } | null = null
|
||||
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
const raw = lines[i] ?? ''
|
||||
const fence = /^```(\s*)(\S.*)?$/.exec(raw)
|
||||
if (!fence) {
|
||||
if (open) open.body.push(raw)
|
||||
continue
|
||||
for (const fence of markdownFences(readFileSync(resolve(root, docRel), 'utf8'))) {
|
||||
if (fence.info === 'ts type-equiv public-api') {
|
||||
throw new Error(`verify-type-equiv: ${docRel}:${fence.line} — use the concise \`ts public-api\` fence`)
|
||||
}
|
||||
if (open) {
|
||||
const code = open.body.join('\n')
|
||||
const symbol = blockSymbol(code)
|
||||
if (!symbol) {
|
||||
throw new Error(`verify-type-equiv: ${docRel}:${open.line} — type-equiv block has no parseable interface/type/class declaration`)
|
||||
}
|
||||
blocks.push({
|
||||
doc: docRel,
|
||||
line: open.line,
|
||||
symbol,
|
||||
code,
|
||||
...(open.projection === undefined ? {} : { projection: open.projection }),
|
||||
})
|
||||
open = null
|
||||
continue
|
||||
if (fence.info !== 'ts type-equiv' && fence.info !== 'ts public-api') continue
|
||||
if (!fence.closed) {
|
||||
throw new Error(`verify-type-equiv: ${docRel}:${fence.line} — unterminated type-equivalence fence (missing closing \`\`\`)`)
|
||||
}
|
||||
const info = (fence[2] ?? '').trim()
|
||||
if (info === 'ts type-equiv public-api') {
|
||||
throw new Error(`verify-type-equiv: ${docRel}:${i + 1} — use the concise \`ts public-api\` fence`)
|
||||
const symbol = blockSymbol(fence.code)
|
||||
if (symbol === null) {
|
||||
throw new Error(`verify-type-equiv: ${docRel}:${fence.line} — type-equiv block has no parseable interface/type/class declaration`)
|
||||
}
|
||||
if (info === 'ts type-equiv') open = { line: i + 1, body: [] }
|
||||
if (info === 'ts public-api') open = { line: i + 1, body: [], projection: 'public-api' }
|
||||
blocks.push({
|
||||
doc: docRel,
|
||||
line: fence.line,
|
||||
symbol,
|
||||
code: fence.code,
|
||||
...(fence.info === 'ts public-api' ? { projection: 'public-api' as const } : {}),
|
||||
})
|
||||
}
|
||||
if (open) throw new Error(`verify-type-equiv: ${docRel}:${open.line} — unterminated type-equiv block`)
|
||||
return blocks
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user