diff --git a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.i18n.yaml b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.i18n.yaml index efb5fdd1cc..26f7f23f85 100644 --- a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.i18n.yaml +++ b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write .agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md -2026-07-26-ci-failover-runbook.md: 0bce83e0f9c842fa3dd73ae9c0a3eefc0975cdae -2026-07-26-ci-failover-runbook.zh.md: 4bc6c67bab754ad3f0127557b0d5e04f7934c8a2 +2026-07-26-ci-failover-runbook.md: ab1a727caa045d2074a9c577416f96f45efcd0aa +2026-07-26-ci-failover-runbook.zh.md: 5dfaca0c1c0f443307bea28bb6544385ebb68bb7 diff --git a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md index 0bce83e0f9..ab1a727caa 100644 --- a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md +++ b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md @@ -6,11 +6,11 @@ English | [中文](2026-07-26-ci-failover-runbook.zh.md) ## Problem -The three required Linux jobs in [CI](../../../../.github/workflows/ci.yml) (`node 24 / static`, `node 24 / coverage`, `node 24 / snapshots and artifacts`) run on the hosted enterprise 32-core pools. When those pools degrade — jobs queue indefinitely, the enterprise labels vanish, or GitHub-side capacity fails — every open pull request becomes unmergeable, and the ordinary recovery of merging a fix is itself deadlocked behind the very required checks that cannot run. An outage therefore needs a switch a repository admin can throw without merging anything. +The three required Linux worker jobs in [CI](../../../../.github/workflows/ci.yml) (`node 24 / static`, `node 24 / coverage`, `node 24 / snapshots and artifacts`) and the required verdict job that aggregates them (`all checks passed`) run on the hosted enterprise 32-core pools. When those pools degrade — jobs queue indefinitely, the enterprise labels vanish, or GitHub-side capacity fails — every open pull request becomes unmergeable, and the ordinary recovery of merging a fix is itself deadlocked behind the very required checks that cannot run. An outage therefore needs a switch a repository admin can throw without merging anything. ## Decision -Each of the three required Linux jobs resolves its runner pool through the `DSH_CI_FAILOVER` repository variable. Unset (normal), they run on the hosted enterprise pools. Set to `selfhosted` by a repository admin, all three retarget onto the in-house self-hosted `vm-backup` pool, coverage and snapshot concurrency drop to shared-VM bounds, and the hosted-path pnpm cache restores are skipped. The switch is admin-only repository state, not a merge, so it works while every check is red. The in-house pool's readiness is continuously re-proven by the `serial / linux (self-hosted standby)` lane, which runs the complete unsharded aggregate on every master push. +Each of the three required Linux worker jobs — and the `all checks passed` verdict job, which would otherwise stay queued on the failed pool even after every worker passed — resolves its runner pool through the `DSH_CI_FAILOVER` repository variable. Unset (normal), they run on the hosted enterprise pools. Set to `selfhosted` by a repository admin, all four retarget onto the in-house self-hosted `vm-backup` pool, coverage and snapshot concurrency drop to shared-VM bounds, and the hosted-path pnpm cache restores are skipped. The switch is admin-only repository state, not a merge, so it works while every check is red. The in-house pool's readiness is continuously re-proven by the `serial / linux (self-hosted standby)` lane, which runs the complete unsharded aggregate on every master push. ### What the in-house pool is @@ -20,7 +20,7 @@ Each of the three required Linux jobs resolves its runner pool through the `DSH_ 1. Repository **Settings → Secrets and variables → Actions → Variables → New repository variable**: name `DSH_CI_FAILOVER`, value `selfhosted`. 2. Retrigger the required jobs so they re-resolve their pool. Jobs already **queued** for the hosted labels do not retarget and cannot be re-run in place, so for the documented indefinite-queue outage, cancel the stuck run and re-run all jobs, or push a new commit; "Re-run failed jobs" only helps once a job has actually failed rather than queued. -3. That is the entire switch. Under failover the workflow also, automatically: halves `DSH_COVERAGE_MAX_WORKERS` to 12 and `DSH_SNAPSHOT_MAX_CONCURRENCY` to 16 (shared-VM contention bounds), and skips the hosted-path pnpm cache restores (the VM's persistent store serves warm installs). +3. That is the entire switch. Under failover the workflow also, automatically: drops `DSH_COVERAGE_MAX_WORKERS` to 8 and `DSH_SNAPSHOT_MAX_CONCURRENCY` to 12 (sized for six always-on instances: worst case 6 × 8 = 48 coverage workers on the 64-core VM) (shared-VM contention bounds), and skips the hosted-path pnpm cache restores (the VM's persistent store serves warm installs). ### Capacity during failover diff --git a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md index 4bc6c67bab..5dfaca0c1c 100644 --- a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md +++ b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md @@ -6,11 +6,11 @@ Status: implemented ## 问题 -[CI](../../../../.github/workflows/ci.yml) 中三个必需的 Linux 作业(`node 24 / static`、`node 24 / coverage`、`node 24 / snapshots and artifacts`)运行在托管的企业级 32 核池上。当这些托管池发生故障——作业无限排队、企业标签消失或 GitHub 侧容量故障——所有开启的拉取请求都无法合并,而"合并一个修复"这一常规恢复手段本身正被那些无法运行的必需检查死锁。因此故障需要一个仓库管理员无需合并任何代码即可触发的开关。 +[CI](../../../../.github/workflows/ci.yml) 中三个必需的 Linux 工作作业(`node 24 / static`、`node 24 / coverage`、`node 24 / snapshots and artifacts`)以及聚合它们的必需判定作业(`all checks passed`)运行在托管的企业级 32 核池上。当这些托管池发生故障——作业无限排队、企业标签消失或 GitHub 侧容量故障——所有开启的拉取请求都无法合并,而"合并一个修复"这一常规恢复手段本身正被那些无法运行的必需检查死锁。因此故障需要一个仓库管理员无需合并任何代码即可触发的开关。 ## 决策 -三个必需的 Linux 作业各自通过仓库变量 `DSH_CI_FAILOVER` 解析运行器池。变量不存在(正常)时它们运行在托管企业池上;由仓库管理员设为 `selfhosted` 时,三者全部切换到公司自有的自托管 `vm-backup` 池,coverage 与 snapshot 的并发降到共享虚拟机上限,并跳过托管路径的 pnpm 缓存恢复。这个开关是仅限管理员的仓库状态而非一次合并,因此在所有检查都是红色时仍然有效。自有池的就绪状态由 `serial / linux (self-hosted standby)` 通道持续验证——每次 master 推送都在其上运行完整的未分片聚合流程。 +三个必需的 Linux 工作作业——以及 `all checks passed` 判定作业(若不随切换,即使全部工作作业通过,它仍会滞留在故障池的队列中)——各自通过仓库变量 `DSH_CI_FAILOVER` 解析运行器池。变量不存在(正常)时它们运行在托管企业池上;由仓库管理员设为 `selfhosted` 时,四者全部切换到公司自有的自托管 `vm-backup` 池,coverage 与 snapshot 的并发降到共享虚拟机上限,并跳过托管路径的 pnpm 缓存恢复。这个开关是仅限管理员的仓库状态而非一次合并,因此在所有检查都是红色时仍然有效。自有池的就绪状态由 `serial / linux (self-hosted standby)` 通道持续验证——每次 master 推送都在其上运行完整的未分片聚合流程。 ### 自有池是什么 @@ -20,7 +20,7 @@ Status: implemented 1. 仓库 **Settings → Secrets and variables → Actions → Variables → New repository variable**:名称 `DSH_CI_FAILOVER`,值 `selfhosted`。 2. 重新触发必需作业,使其重新解析运行器池。已经为托管标签**排队**的作业不会重定向,也无法原地 re-run,因此对于本手册所述的无限排队故障,应取消卡住的运行并 re-run all jobs,或推送一个新提交;“Re-run failed jobs”只有在作业真正失败(而非仍在排队)时才有用。 -3. 切换到此完成。故障切换状态下工作流还会自动:把 `DSH_COVERAGE_MAX_WORKERS` 降为 12、`DSH_SNAPSHOT_MAX_CONCURRENCY` 降为 16(共享虚拟机的争抢上限),并跳过托管路径的 pnpm 缓存恢复(虚拟机的持久 store 直接提供热安装)。 +3. 切换到此完成。故障切换状态下工作流还会自动:把 `DSH_COVERAGE_MAX_WORKERS` 降为 8、`DSH_SNAPSHOT_MAX_CONCURRENCY` 降为 12(按 6 个常驻实例定容:最坏 6 × 8 = 48 个覆盖率工作进程对 64 核)(共享虚拟机的争抢上限),并跳过托管路径的 pnpm 缓存恢复(虚拟机的持久 store 直接提供热安装)。 ### 切换期间的容量 diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index df3d386e39..8306b7e034 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -44,6 +44,7 @@ jobs: if: github.event_name == 'pull_request' runs-on: >- ${{ vars.DSH_CI_FAILOVER == 'selfhosted' + && github.event.pull_request.user.login != 'dependabot[bot]' && fromJSON('["self-hosted", "linux", "x64", "vm-backup"]') || 'dsh-enterprise-ubuntu-latest-32core-test' }} name: node 24 / static @@ -60,7 +61,7 @@ jobs: # compression and upload on the paid latency-critical path. Skipped # under failover — see the coverage lane's identical rationale. - uses: actions/cache/restore@v4 - if: vars.DSH_CI_FAILOVER != 'selfhosted' + if: vars.DSH_CI_FAILOVER != 'selfhosted' || github.event.pull_request.user.login == 'dependabot[bot]' with: path: /home/runner/.local/share/pnpm/store/v11 key: ${{ runner.os }}-node-${{ env.PRIMARY_NODE_VERSION }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }} @@ -98,6 +99,7 @@ jobs: if: github.event_name == 'pull_request' runs-on: >- ${{ vars.DSH_CI_FAILOVER == 'selfhosted' + && github.event.pull_request.user.login != 'dependabot[bot]' && fromJSON('["self-hosted", "linux", "x64", "vm-backup"]') || 'dsh-enterprise-ubuntu-24-04-32core-test' }} name: node 24 / coverage @@ -107,7 +109,7 @@ jobs: # across six always-on runner instances, and the timing-sensitive # process suites have documented aggregate-contention failures. # 8 × 6 instances = 48 workers worst case on 64 cores. - DSH_COVERAGE_MAX_WORKERS: ${{ vars.DSH_CI_FAILOVER == 'selfhosted' && '8' || '24' }} + DSH_COVERAGE_MAX_WORKERS: ${{ vars.DSH_CI_FAILOVER == 'selfhosted' && github.event.pull_request.user.login != 'dependabot[bot]' && '8' || '24' }} DSH_GATE_CONCURRENCY: '8' steps: - uses: actions/checkout@v6 @@ -118,7 +120,7 @@ jobs: # serves warm installs directly, and this hosted-path restore would # spend ~52 s pulling ~180 MB into a path pnpm never reads there. - uses: actions/cache/restore@v4 - if: vars.DSH_CI_FAILOVER != 'selfhosted' + if: vars.DSH_CI_FAILOVER != 'selfhosted' || github.event.pull_request.user.login == 'dependabot[bot]' with: path: /home/runner/.local/share/pnpm/store/v11 key: ${{ runner.os }}-node-${{ env.PRIMARY_NODE_VERSION }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }} @@ -151,6 +153,7 @@ jobs: if: github.event_name == 'pull_request' runs-on: >- ${{ vars.DSH_CI_FAILOVER == 'selfhosted' + && github.event.pull_request.user.login != 'dependabot[bot]' && fromJSON('["self-hosted", "linux", "x64", "vm-backup"]') || 'dsh-enterprise-ubuntu-latest-32core-test' }} name: node 24 / snapshots and artifacts @@ -161,7 +164,7 @@ jobs: DSH_NODE_COMPAT_SKIP_TYPECHECK: '1' DSH_PUBLINT_CONCURRENCY: '8' # Failover halves snapshot concurrency for the shared 64-core VM. - DSH_SNAPSHOT_MAX_CONCURRENCY: ${{ vars.DSH_CI_FAILOVER == 'selfhosted' && '12' || '32' }} + DSH_SNAPSHOT_MAX_CONCURRENCY: ${{ vars.DSH_CI_FAILOVER == 'selfhosted' && github.event.pull_request.user.login != 'dependabot[bot]' && '12' || '32' }} steps: - uses: actions/checkout@v6 with: @@ -177,7 +180,7 @@ jobs: # Skipped under failover — see the coverage lane's identical rationale. - uses: actions/cache/restore@v4 - if: vars.DSH_CI_FAILOVER != 'selfhosted' + if: vars.DSH_CI_FAILOVER != 'selfhosted' || github.event.pull_request.user.login == 'dependabot[bot]' with: path: /home/runner/.local/share/pnpm/store/v11 key: ${{ runner.os }}-node-${{ env.PRIMARY_NODE_VERSION }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }} @@ -773,6 +776,7 @@ jobs: # queued forever on the failed pool. runs-on: >- ${{ vars.DSH_CI_FAILOVER == 'selfhosted' + && github.event.pull_request.user.login != 'dependabot[bot]' && fromJSON('["self-hosted", "linux", "x64", "vm-backup"]') || 'dsh-enterprise-ubuntu-latest-32core-test' }} needs: [node-24, node-24-coverage, node-24-consumers, node-compat, python-sdk, windows]