ci: dependabot stays hosted under failover; runbook matches shipped bounds

- All four failover selectors (three workers + the verdict job) and the paired env/cache expressions now exclude dependabot[bot]: under failover, dependency-supplied code keeps queueing for the hosted pool instead of executing on the persistent VM. A delayed Dependabot PR during an outage is an acceptable cost; dependency code on the privileged host is not. - Runbook (both languages): records the shipped failover bounds (coverage 8, snapshots 12, sized for six instances) and documents that the verdict job follows the selector too — operators previously had no explanation for a verdict queued after all workers passed. - Local static gate green: 32 passed, 0 failed (translation pairing 519 pairs consistent).
2026-08-15 21:04:50 +00:00 · 2026-07-27 15:36:11 +08:00
parent aedf7fbf34
commit caabf8f671
4 changed files with 17 additions and 13 deletions
--- a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.i18n.yaml
+++ b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.i18n.yaml
@@ -2,5 +2,5 @@
 # side as of the last confirmed-consistent state. Both languages carry equal authority;
 # after editing either side, bring the other along and re-record with:
 #   pnpm run verify-translation-pairing --write .agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md
-2026-07-26-ci-failover-runbook.md: 0bce83e0f9c842fa3dd73ae9c0a3eefc0975cdae
-2026-07-26-ci-failover-runbook.zh.md: 4bc6c67bab754ad3f0127557b0d5e04f7934c8a2
+2026-07-26-ci-failover-runbook.md: ab1a727caa045d2074a9c577416f96f45efcd0aa
+2026-07-26-ci-failover-runbook.zh.md: 5dfaca0c1c0f443307bea28bb6544385ebb68bb7
--- a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md
+++ b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md
@@ -6,11 +6,11 @@ English | [中文](2026-07-26-ci-failover-runbook.zh.md)

 ## Problem

-The three required Linux jobs in [CI](../../../../.github/workflows/ci.yml) (`node 24 / static`, `node 24 / coverage`, `node 24 / snapshots and artifacts`) run on the hosted enterprise 32-core pools. When those pools degrade — jobs queue indefinitely, the enterprise labels vanish, or GitHub-side capacity fails — every open pull request becomes unmergeable, and the ordinary recovery of merging a fix is itself deadlocked behind the very required checks that cannot run. An outage therefore needs a switch a repository admin can throw without merging anything.
+The three required Linux worker jobs in [CI](../../../../.github/workflows/ci.yml) (`node 24 / static`, `node 24 / coverage`, `node 24 / snapshots and artifacts`) and the required verdict job that aggregates them (`all checks passed`) run on the hosted enterprise 32-core pools. When those pools degrade — jobs queue indefinitely, the enterprise labels vanish, or GitHub-side capacity fails — every open pull request becomes unmergeable, and the ordinary recovery of merging a fix is itself deadlocked behind the very required checks that cannot run. An outage therefore needs a switch a repository admin can throw without merging anything.

 ## Decision

-Each of the three required Linux jobs resolves its runner pool through the `DSH_CI_FAILOVER` repository variable. Unset (normal), they run on the hosted enterprise pools. Set to `selfhosted` by a repository admin, all three retarget onto the in-house self-hosted `vm-backup` pool, coverage and snapshot concurrency drop to shared-VM bounds, and the hosted-path pnpm cache restores are skipped. The switch is admin-only repository state, not a merge, so it works while every check is red. The in-house pool's readiness is continuously re-proven by the `serial / linux (self-hosted standby)` lane, which runs the complete unsharded aggregate on every master push.
+Each of the three required Linux worker jobs — and the `all checks passed` verdict job, which would otherwise stay queued on the failed pool even after every worker passed — resolves its runner pool through the `DSH_CI_FAILOVER` repository variable. Unset (normal), they run on the hosted enterprise pools. Set to `selfhosted` by a repository admin, all four retarget onto the in-house self-hosted `vm-backup` pool, coverage and snapshot concurrency drop to shared-VM bounds, and the hosted-path pnpm cache restores are skipped. The switch is admin-only repository state, not a merge, so it works while every check is red. The in-house pool's readiness is continuously re-proven by the `serial / linux (self-hosted standby)` lane, which runs the complete unsharded aggregate on every master push.

 ### What the in-house pool is

@@ -20,7 +20,7 @@ Each of the three required Linux jobs resolves its runner pool through the `DSH_

 1. Repository **Settings → Secrets and variables → Actions → Variables → New repository variable**: name `DSH_CI_FAILOVER`, value `selfhosted`.
 2. Retrigger the required jobs so they re-resolve their pool. Jobs already **queued** for the hosted labels do not retarget and cannot be re-run in place, so for the documented indefinite-queue outage, cancel the stuck run and re-run all jobs, or push a new commit; "Re-run failed jobs" only helps once a job has actually failed rather than queued.
-3. That is the entire switch. Under failover the workflow also, automatically: halves `DSH_COVERAGE_MAX_WORKERS` to 12 and `DSH_SNAPSHOT_MAX_CONCURRENCY` to 16 (shared-VM contention bounds), and skips the hosted-path pnpm cache restores (the VM's persistent store serves warm installs).
+3. That is the entire switch. Under failover the workflow also, automatically: drops `DSH_COVERAGE_MAX_WORKERS` to 8 and `DSH_SNAPSHOT_MAX_CONCURRENCY` to 12 (sized for six always-on instances: worst case 6 × 8 = 48 coverage workers on the 64-core VM) (shared-VM contention bounds), and skips the hosted-path pnpm cache restores (the VM's persistent store serves warm installs).

 ### Capacity during failover

--- a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md
+++ b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md
@@ -6,11 +6,11 @@ Status: implemented

 ## 问题

-[CI](../../../../.github/workflows/ci.yml) 中三个必需的 Linux 作业（`node 24 / static`、`node 24 / coverage`、`node 24 / snapshots and artifacts`）运行在托管的企业级 32 核池上。当这些托管池发生故障——作业无限排队、企业标签消失或 GitHub 侧容量故障——所有开启的拉取请求都无法合并，而"合并一个修复"这一常规恢复手段本身正被那些无法运行的必需检查死锁。因此故障需要一个仓库管理员无需合并任何代码即可触发的开关。
+[CI](../../../../.github/workflows/ci.yml) 中三个必需的 Linux 工作作业（`node 24 / static`、`node 24 / coverage`、`node 24 / snapshots and artifacts`）以及聚合它们的必需判定作业（`all checks passed`）运行在托管的企业级 32 核池上。当这些托管池发生故障——作业无限排队、企业标签消失或 GitHub 侧容量故障——所有开启的拉取请求都无法合并，而"合并一个修复"这一常规恢复手段本身正被那些无法运行的必需检查死锁。因此故障需要一个仓库管理员无需合并任何代码即可触发的开关。

 ## 决策

-三个必需的 Linux 作业各自通过仓库变量 `DSH_CI_FAILOVER` 解析运行器池。变量不存在（正常）时它们运行在托管企业池上；由仓库管理员设为 `selfhosted` 时，三者全部切换到公司自有的自托管 `vm-backup` 池，coverage 与 snapshot 的并发降到共享虚拟机上限，并跳过托管路径的 pnpm 缓存恢复。这个开关是仅限管理员的仓库状态而非一次合并，因此在所有检查都是红色时仍然有效。自有池的就绪状态由 `serial / linux (self-hosted standby)` 通道持续验证——每次 master 推送都在其上运行完整的未分片聚合流程。
+三个必需的 Linux 工作作业——以及 `all checks passed` 判定作业（若不随切换，即使全部工作作业通过，它仍会滞留在故障池的队列中）——各自通过仓库变量 `DSH_CI_FAILOVER` 解析运行器池。变量不存在（正常）时它们运行在托管企业池上；由仓库管理员设为 `selfhosted` 时，四者全部切换到公司自有的自托管 `vm-backup` 池，coverage 与 snapshot 的并发降到共享虚拟机上限，并跳过托管路径的 pnpm 缓存恢复。这个开关是仅限管理员的仓库状态而非一次合并，因此在所有检查都是红色时仍然有效。自有池的就绪状态由 `serial / linux (self-hosted standby)` 通道持续验证——每次 master 推送都在其上运行完整的未分片聚合流程。

 ### 自有池是什么

@@ -20,7 +20,7 @@ Status: implemented

 1. 仓库 **Settings → Secrets and variables → Actions → Variables → New repository variable**：名称 `DSH_CI_FAILOVER`，值 `selfhosted`。
 2. 重新触发必需作业，使其重新解析运行器池。已经为托管标签**排队**的作业不会重定向，也无法原地 re-run，因此对于本手册所述的无限排队故障，应取消卡住的运行并 re-run all jobs，或推送一个新提交；“Re-run failed jobs”只有在作业真正失败（而非仍在排队）时才有用。
-3. 切换到此完成。故障切换状态下工作流还会自动：把 `DSH_COVERAGE_MAX_WORKERS` 降为 12、`DSH_SNAPSHOT_MAX_CONCURRENCY` 降为 16（共享虚拟机的争抢上限），并跳过托管路径的 pnpm 缓存恢复（虚拟机的持久 store 直接提供热安装）。
+3. 切换到此完成。故障切换状态下工作流还会自动：把 `DSH_COVERAGE_MAX_WORKERS` 降为 8、`DSH_SNAPSHOT_MAX_CONCURRENCY` 降为 12（按 6 个常驻实例定容：最坏 6 × 8 = 48 个覆盖率工作进程对 64 核）（共享虚拟机的争抢上限），并跳过托管路径的 pnpm 缓存恢复（虚拟机的持久 store 直接提供热安装）。

 ### 切换期间的容量

--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -44,6 +44,7 @@ jobs:
    if: github.event_name == 'pull_request'
    runs-on: >-
      ${{ vars.DSH_CI_FAILOVER == 'selfhosted'
+          && github.event.pull_request.user.login != 'dependabot[bot]'
          && fromJSON('["self-hosted", "linux", "x64", "vm-backup"]')
          || 'dsh-enterprise-ubuntu-latest-32core-test' }}
    name: node 24 / static
@@ -60,7 +61,7 @@ jobs:
      # compression and upload on the paid latency-critical path. Skipped
      # under failover — see the coverage lane's identical rationale.
      - uses: actions/cache/restore@v4
-        if: vars.DSH_CI_FAILOVER != 'selfhosted'
+        if: vars.DSH_CI_FAILOVER != 'selfhosted' || github.event.pull_request.user.login == 'dependabot[bot]'
        with:
          path: /home/runner/.local/share/pnpm/store/v11
          key: ${{ runner.os }}-node-${{ env.PRIMARY_NODE_VERSION }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }}
@@ -98,6 +99,7 @@ jobs:
    if: github.event_name == 'pull_request'
    runs-on: >-
      ${{ vars.DSH_CI_FAILOVER == 'selfhosted'
+          && github.event.pull_request.user.login != 'dependabot[bot]'
          && fromJSON('["self-hosted", "linux", "x64", "vm-backup"]')
          || 'dsh-enterprise-ubuntu-24-04-32core-test' }}
    name: node 24 / coverage
@@ -107,7 +109,7 @@ jobs:
      # across six always-on runner instances, and the timing-sensitive
      # process suites have documented aggregate-contention failures.
      # 8 × 6 instances = 48 workers worst case on 64 cores.
-      DSH_COVERAGE_MAX_WORKERS: ${{ vars.DSH_CI_FAILOVER == 'selfhosted' && '8' || '24' }}
+      DSH_COVERAGE_MAX_WORKERS: ${{ vars.DSH_CI_FAILOVER == 'selfhosted' && github.event.pull_request.user.login != 'dependabot[bot]' && '8' || '24' }}
      DSH_GATE_CONCURRENCY: '8'
    steps:
      - uses: actions/checkout@v6
@@ -118,7 +120,7 @@ jobs:
      # serves warm installs directly, and this hosted-path restore would
      # spend ~52 s pulling ~180 MB into a path pnpm never reads there.
      - uses: actions/cache/restore@v4
-        if: vars.DSH_CI_FAILOVER != 'selfhosted'
+        if: vars.DSH_CI_FAILOVER != 'selfhosted' || github.event.pull_request.user.login == 'dependabot[bot]'
        with:
          path: /home/runner/.local/share/pnpm/store/v11
          key: ${{ runner.os }}-node-${{ env.PRIMARY_NODE_VERSION }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }}
@@ -151,6 +153,7 @@ jobs:
    if: github.event_name == 'pull_request'
    runs-on: >-
      ${{ vars.DSH_CI_FAILOVER == 'selfhosted'
+          && github.event.pull_request.user.login != 'dependabot[bot]'
          && fromJSON('["self-hosted", "linux", "x64", "vm-backup"]')
          || 'dsh-enterprise-ubuntu-latest-32core-test' }}
    name: node 24 / snapshots and artifacts
@@ -161,7 +164,7 @@ jobs:
      DSH_NODE_COMPAT_SKIP_TYPECHECK: '1'
      DSH_PUBLINT_CONCURRENCY: '8'
      # Failover halves snapshot concurrency for the shared 64-core VM.
-      DSH_SNAPSHOT_MAX_CONCURRENCY: ${{ vars.DSH_CI_FAILOVER == 'selfhosted' && '12' || '32' }}
+      DSH_SNAPSHOT_MAX_CONCURRENCY: ${{ vars.DSH_CI_FAILOVER == 'selfhosted' && github.event.pull_request.user.login != 'dependabot[bot]' && '12' || '32' }}
    steps:
      - uses: actions/checkout@v6
        with:
@@ -177,7 +180,7 @@ jobs:

      # Skipped under failover — see the coverage lane's identical rationale.
      - uses: actions/cache/restore@v4
-        if: vars.DSH_CI_FAILOVER != 'selfhosted'
+        if: vars.DSH_CI_FAILOVER != 'selfhosted' || github.event.pull_request.user.login == 'dependabot[bot]'
        with:
          path: /home/runner/.local/share/pnpm/store/v11
          key: ${{ runner.os }}-node-${{ env.PRIMARY_NODE_VERSION }}-pnpm-${{ hashFiles('pnpm-lock.yaml') }}
@@ -773,6 +776,7 @@ jobs:
    # queued forever on the failed pool.
    runs-on: >-
      ${{ vars.DSH_CI_FAILOVER == 'selfhosted'
+          && github.event.pull_request.user.login != 'dependabot[bot]'
          && fromJSON('["self-hosted", "linux", "x64", "vm-backup"]')
          || 'dsh-enterprise-ubuntu-latest-32core-test' }}
    needs: [node-24, node-24-coverage, node-24-consumers, node-compat, python-sdk, windows]