diff --git a/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.i18n.yaml b/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.i18n.yaml new file mode 100644 index 0000000000..303a021a38 --- /dev/null +++ b/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.i18n.yaml @@ -0,0 +1,6 @@ +# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each +# side as of the last confirmed-consistent state. Both languages carry equal authority; +# after editing either side, bring the other along and re-record with: +# pnpm run verify-translation-pairing --write +2026-07-22-evidence-based-larger-hosted-runners.md: 992a622f0d7a2662cc65353d6c4cd1350e1f3f5f +2026-07-22-evidence-based-larger-hosted-runners.zh.md: 622911af8400fbe159038b69b440c0e97fb1f2cc diff --git a/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.md b/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.md new file mode 100644 index 0000000000..992a622f0d --- /dev/null +++ b/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.md @@ -0,0 +1,44 @@ +# Agent Note: Evidence-based larger hosted runners + +Status: implemented + +English | [中文](2026-07-22-evidence-based-larger-hosted-runners.zh.md) + +## Problem + +The sharded primary CI fits its latency targets on standard GitHub-hosted runners, but the margin depends on cold setup and install variance. Larger runners may add useful headroom, yet their per-minute price rises much faster than these short lanes can use extra cores. Selecting a size from machine specifications or a synthetic benchmark would spend more without proving that repository CI becomes faster. + +## Decision + +The organization keeps twelve x64 larger-runner pools in the repo-restricted `dsh-larger-ci` group: Ubuntu 24.04 and Windows 2025 at 4, 8, 16, 32, 64, and 96 cores. Public IPs are disabled. Each pool has an autoscaling ceiling of 256, while the repository bounds actual demand through its workflow matrices; an idle ceiling does not allocate machines. + +The `CI` workflow exposes `suite=larger-runner-benchmark` only through manual dispatch. Its six Linux legs run the critical typecheck lane, and its six Windows legs run the critical production-site lane. Every leg reports the observed CPU and memory, has a 15-minute timeout, and uses the same setup and caching policy as the production lane it represents. Push and pull-request events skip this benchmark. + +The [twelve-size benchmark](https://github.com/deepseek-harness/deepseek-harness/actions/runs/29895295659) used a workflow-only commit on top of the standard-runner [baseline](https://github.com/deepseek-harness/deepseek-harness/actions/runs/29850033610), so the code, lockfile, and critical commands were identical: + +| Critical job | Standard | 4 cores | 8 cores | 16 cores | 32 cores | 64 cores | 96 cores | +|---|---:|---:|---:|---:|---:|---:|---:| +| Linux typecheck | 56 s | 38 s | 35 s | 40 s | 35 s | 44 s | 40 s | +| Windows production site | 160 s | 117 s | 103 s | 113 s | 75 s | 105 s | 108 s | + +The repository therefore uses the 4-core pools for the primary Node matrix and all Windows jobs. Linux 4-core finished within three seconds of the fastest measured size at the lowest larger-runner rate. Windows 4-core stayed below two minutes; the isolated 32-core result was faster, but adjacent larger sizes regressed and the production-site command itself varied only from 28 to 36 seconds, so setup and install noise—not scalable compute—created most of the spread. Node compatibility, Python, and the final aggregator remain on standard runners because their baseline jobs already finish well below one minute. + +The workflow also exposes `suite=optimized-larger-runners` through manual dispatch. That path runs the production matrices against the branch ref itself, providing an exact-head timing check when a pull request cannot form a merge commit. + +## Alternatives considered + +**Keep every job on standard runners.** This meets the threshold but gives the critical lanes no cold-run margin and leaves the larger-runner suggestion untested. + +**Select 8 or 32 cores from the fastest individual result.** The small differences were not monotonic, while billing grows sharply with size. Treating one noisy minimum as scaling evidence would make recurring CI substantially more expensive. + +**Move every job to a larger runner.** Compatibility, Python, and aggregation were already short; paying the larger-runner premium there cannot improve the critical path enough to justify the dependency or cost. + +**Use a synthetic CPU benchmark.** A microbenchmark would not include checkout, action startup, package installation, cache restore, or the repository command mix that dominates these jobs. + +## Consequences + +The benchmark incurred $2.936 across dedicated larger-runner SKUs, as recorded by organization billing immediately after the run. The existing zero-dollar Actions budget did not block those jobs, so the repo-only runner group, manual benchmark trigger, matrix width, and timeout are the observed cost controls; the budget is not treated as an execution guard. + +Production CI now depends on the organization-owned runner names in this note and in `.github/workflows/ci.yml`. Missing or renamed pools leave jobs queued instead of silently falling back to standard capacity. The 256 autoscaling ceiling permits future concurrency but does not override the bounded matrices in the current workflow. + +The selected pools buy latency headroom at a recurring per-job premium. The manual benchmark retains all sizes so a future image, dependency graph, or workload change can be measured before changing the production labels again. diff --git a/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.zh.md b/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.zh.md new file mode 100644 index 0000000000..622911af84 --- /dev/null +++ b/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.zh.md @@ -0,0 +1,44 @@ +# Agent Note: 基于实证选用 GitHub 托管大型运行器 + +Status: implemented + +[English](2026-07-22-evidence-based-larger-hosted-runners.md) | 中文 + +## 问题 + +分片后的主 CI 在标准 GitHub 托管运行器上能够达到延迟目标，但余量会受冷启动设置和安装耗时波动影响。大型运行器或许能提供更多有效余量，但随着核心数增加，其每分钟单价增长得远比这些短通道能够利用额外核心的速度快。仅根据机器规格或合成基准测试选择配置，只会增加支出，无法证明仓库 CI 确实变快。 + +## 决策 + +组织在仅限本仓库使用的 `dsh-larger-ci` 运行器组中保留 12 个 x64 大型运行器池：Ubuntu 24.04 和 Windows 2025 各设 4、8、16、32、64、96 核规格。公网 IP 已禁用。每个池的自动扩缩容上限为 256，仓库则通过工作流矩阵限制实际需求；空闲状态下的上限不会分配机器。 + +`CI` 工作流仅在手动触发时提供 `suite=larger-runner-benchmark`。其中 6 个 Linux 矩阵分支运行关键类型检查通道，6 个 Windows 矩阵分支运行生产网站关键通道。每个矩阵分支都会报告实测 CPU 和内存，超时限制为 15 分钟，并采用与其所代表的生产通道相同的设置和缓存策略。push 和拉取请求事件会跳过这项基准测试。 + +这项[涵盖 12 种规格的基准测试](https://github.com/deepseek-harness/deepseek-harness/actions/runs/29895295659)以标准运行器[基线](https://github.com/deepseek-harness/deepseek-harness/actions/runs/29850033610)为基础，只叠加了一个仅修改工作流的提交，因此代码、锁文件和关键命令完全相同： + +| 关键作业 | 标准 | 4 核 | 8 核 | 16 核 | 32 核 | 64 核 | 96 核 | +|---|---:|---:|---:|---:|---:|---:|---:| +| Linux 类型检查 | 56 秒 | 38 秒 | 35 秒 | 40 秒 | 35 秒 | 44 秒 | 40 秒 | +| Windows 生产网站 | 160 秒 | 117 秒 | 103 秒 | 113 秒 | 75 秒 | 105 秒 | 108 秒 | + +因此，仓库让主 Node 矩阵和所有 Windows 作业使用 4 核池。Linux 4 核规格比实测最快规格至多慢 3 秒，同时采用大型运行器中的最低费率。Windows 4 核规格保持在 2 分钟以内；单次测得的 32 核结果虽然更快，但相邻的大型运行器规格反而耗时回升，而生产网站命令本身的耗时仅在 28 至 36 秒之间波动，因此大部分差异来自设置和安装噪声，而非可随核心数扩展的计算负载。Node 兼容性、Python 和最终聚合作业仍使用标准运行器，因为它们的基线耗时本就远低于 1 分钟。 + +工作流还在手动触发时提供 `suite=optimized-larger-runners`。这一路径直接以分支引用本身运行生产矩阵；当拉取请求无法生成合并提交时，它仍能针对该分支实际的头部提交提供精确计时检查。 + +## 曾考虑的替代方案 + +**所有作业均保留在标准运行器上。** 此方案能达到时长目标，但关键通道面对冷启动时没有余量，也无法验证大型运行器方案。 + +**根据单次最快结果选择 8 核或 32 核。** 各规格间的微小差异并不呈单调变化，计费却随规格急剧增长。把一次带有噪声的最低值当作扩展性证据，会大幅提高 CI 的经常性成本。 + +**将所有作业迁移到大型运行器。** 兼容性、Python 和聚合作业原本就很短；为这些作业支付大型运行器溢价，无法充分缩短关键路径，因而不足以证明这种依赖或成本合理。 + +**使用合成 CPU 基准测试。** 微基准测试不会包含代码检出、操作启动、包安装、缓存恢复或主导这些作业耗时的仓库命令组合。 + +## 后果 + +这次基准测试使用专用大型运行器 SKU，共产生 $2.936 费用，组织计费系统在运行结束后立即记录了这笔费用。现有的零美元 Actions 预算并未阻止这些作业，因此经实测有效的成本控制机制是仅限本仓库的运行器组、手动基准测试触发、矩阵宽度和超时限制；该预算不被视为执行防护措施。 + +生产 CI 目前依赖本 Agent Note 和 `.github/workflows/ci.yml` 中由组织持有的运行器名称。池缺失或改名会让作业一直排队，不会静默回退到标准容量。256 的自动扩缩容上限允许未来提高并发量，但不会突破当前工作流对矩阵规模的限制。 + +选定的运行器池以每次作业都需支付的溢价换取延迟余量。手动基准测试保留了所有规格，以便未来映像、依赖图或工作负载发生变化时，先完成测量，再次调整生产标签。 diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 574685cd45..060078cc1f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -14,6 +14,7 @@ on: options: - serial-reference - larger-runner-benchmark + - optimized-larger-runners concurrency: group: ${{ github.workflow }}-${{ github.ref }} @@ -27,8 +28,8 @@ env: jobs: node-24: - if: github.event_name != 'workflow_dispatch' - runs-on: ubuntu-latest + if: github.event_name != 'workflow_dispatch' || inputs.suite == 'optimized-larger-runners' + runs-on: dsh-ubuntu-24-04-4core name: node 24 / ${{ matrix.lane }} env: DSH_GATE_CONCURRENCY: ${{ matrix.gate_concurrency }} @@ -316,7 +317,7 @@ jobs: run: ${{ matrix.command }} node-compat: - if: github.event_name != 'workflow_dispatch' + if: github.event_name != 'workflow_dispatch' || inputs.suite == 'optimized-larger-runners' runs-on: ubuntu-latest name: node ${{ matrix.node }} env: @@ -359,7 +360,7 @@ jobs: run: pnpm run check:node-compat python-sdk: - if: github.event_name != 'workflow_dispatch' + if: github.event_name != 'workflow_dispatch' || inputs.suite == 'optimized-larger-runners' runs-on: ubuntu-latest name: python 3.10 / keyless SDK steps: @@ -381,8 +382,8 @@ jobs: # The broader observational gate matrix below exposes the remaining # portability work without blocking mainline merges. windows-build: - if: github.event_name != 'workflow_dispatch' - runs-on: windows-2025 + if: github.event_name != 'workflow_dispatch' || inputs.suite == 'optimized-larger-runners' + runs-on: dsh-windows-2025-4core name: windows / build steps: - uses: actions/checkout@v6 @@ -400,8 +401,8 @@ jobs: run: pnpm run build windows-site: - if: github.event_name != 'workflow_dispatch' - runs-on: windows-2025 + if: github.event_name != 'workflow_dispatch' || inputs.suite == 'optimized-larger-runners' + runs-on: dsh-windows-2025-4core name: windows / production site steps: - uses: actions/checkout@v6 @@ -424,9 +425,9 @@ jobs: # would change the environment being measured. This job intentionally stays # out of all-checks-passed.needs. windows-gates: - if: github.event_name != 'workflow_dispatch' + if: github.event_name != 'workflow_dispatch' || inputs.suite == 'optimized-larger-runners' continue-on-error: true - runs-on: windows-2025 + runs-on: dsh-windows-2025-4core name: windows node 24 / ${{ matrix.lane }} env: DSH_GATE_CONCURRENCY: ${{ matrix.gate_concurrency }} @@ -725,7 +726,7 @@ jobs: name: all checks passed runs-on: ubuntu-latest needs: [node-24, node-compat, python-sdk, windows-build, windows-site] - if: always() && github.event_name != 'workflow_dispatch' + if: always() && (github.event_name != 'workflow_dispatch' || inputs.suite == 'optimized-larger-runners') steps: - name: Fail if any needed job did not succeed if: contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled') || contains(needs.*.result, 'skipped')