From 68e280ce4ff86629ea0443a012d7c7080289ce4d Mon Sep 17 00:00:00 2001 From: Chinesezjc Date: Sun, 26 Jul 2026 05:22:28 +0800 Subject: [PATCH] docs(ci): make the failover runbook a conforming dated Agent Note The failover runbook landed as .agents/notes/implemented/process/ci-failover-runbook.md, which fails three doc-sync gates: the classification/format gates require a yyyy-mm-dd-topic.md filename and the implemented Agent Note skeleton (Problem/Decision/Alternatives/Consequences), and the bilingual pairing gate requires cross-note link targets to match between the two language sides. Rename to 2026-07-26-ci-failover-runbook.md/.zh.md, reshape both sides into the implemented skeleton (the runbook steps live in bespoke sections under Decision), point the sibling topology note and the ci.yml comment at the dated filename, and make both sides link the canonical .md per the bilingual convention. Re-recorded the i18n pairing records. --- ...ence-based-larger-hosted-runners.i18n.yaml | 4 +- ...22-evidence-based-larger-hosted-runners.md | 2 +- ...evidence-based-larger-hosted-runners.zh.md | 2 +- ... 2026-07-26-ci-failover-runbook.i18n.yaml} | 4 +- .../process/2026-07-26-ci-failover-runbook.md | 49 +++++++++++++++++++ .../2026-07-26-ci-failover-runbook.zh.md | 49 +++++++++++++++++++ .../process/ci-failover-runbook.md | 33 ------------- .../process/ci-failover-runbook.zh.md | 33 ------------- .github/workflows/ci.yml | 2 +- 9 files changed, 105 insertions(+), 73 deletions(-) rename .agents/notes/implemented/process/{ci-failover-runbook.i18n.yaml => 2026-07-26-ci-failover-runbook.i18n.yaml} (65%) create mode 100644 .agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md create mode 100644 .agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md delete mode 100644 .agents/notes/implemented/process/ci-failover-runbook.md delete mode 100644 .agents/notes/implemented/process/ci-failover-runbook.zh.md diff --git a/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.i18n.yaml b/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.i18n.yaml index b2d20fb999..84a10e5ab9 100644 --- a/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.i18n.yaml +++ b/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write -2026-07-22-evidence-based-larger-hosted-runners.md: 6e989a908b1faa363d04746e4efaa1a77358be9d -2026-07-22-evidence-based-larger-hosted-runners.zh.md: 02c2ab405ec10dd381b051581d72662ec342e21e +2026-07-22-evidence-based-larger-hosted-runners.md: 21e602b2b5850176df981dcf448f4f827b756719 +2026-07-22-evidence-based-larger-hosted-runners.zh.md: ba49ff18ac304f4078d4c8ebfd00bb1a85ada0b3 diff --git a/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.md b/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.md index 6e989a908b..21e602b2b5 100644 --- a/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.md +++ b/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.md @@ -52,7 +52,7 @@ The process-bound coverage project contains exactly five suite files. Thirty-two Complete serial Linux, macOS, and Windows references run only when `master` moves. Pull requests use the enterprise required path plus standard-hosted compatibility jobs, while other larger-runner sizes run only by manual dispatch. -An additional serial Linux reference runs on the in-house self-hosted pool (`vm-backup` label: a 64-core VM with four always-on systemd-managed runner instances plus four registered spares) on every `master` push. It is a hot-standby drill, not a required check: each run re-proves that the persistent VM can execute the complete unsharded aggregate. The actual switch is pre-wired: the three required Linux jobs resolve their pool through the admin-only `DSH_CI_FAILOVER` repository variable, so an outage response is setting one variable and re-running — no merge, which would be deadlocked behind the failing checks themselves ([runbook](ci-failover-runbook.md)). Because the standby lane is push-triggered, it always executes the base branch's workflow definition — no pull-request-editable path can route code to these runners, and the repository additionally keeps forking disabled. +An additional serial Linux reference runs on the in-house self-hosted pool (`vm-backup` label: a 64-core VM with four always-on systemd-managed runner instances plus four registered spares) on every `master` push. It is a hot-standby drill, not a required check: each run re-proves that the persistent VM can execute the complete unsharded aggregate. The actual switch is pre-wired: the three required Linux jobs resolve their pool through the admin-only `DSH_CI_FAILOVER` repository variable, so an outage response is setting one variable and re-running — no merge, which would be deadlocked behind the failing checks themselves ([runbook](2026-07-26-ci-failover-runbook.md)). Because the standby lane is push-triggered, it always executes the base branch's workflow definition — no pull-request-editable path can route code to these runners, and the repository additionally keeps forking disabled. ## Alternatives considered diff --git a/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.zh.md b/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.zh.md index 02c2ab405e..ba49ff18ac 100644 --- a/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.zh.md +++ b/.agents/notes/implemented/process/2026-07-22-evidence-based-larger-hosted-runners.zh.md @@ -52,7 +52,7 @@ Windows 仓库工作在超过 16 核后收益很小，但 32 核池可以让完 只有在 `master` 移动时，才运行完整的 Linux、macOS 和 Windows 串行参考。拉取请求使用企业级运行器必需路径和标准托管兼容性作业，其他大型运行器规格仅通过手动触发运行。 -另有一条串行 Linux 参考在每次 `master` 推送时运行于公司自有的自托管池（`vm-backup` 标签：一台 64 核虚拟机，运行 4 个常驻的 systemd 管理运行器实例，另有 4 个已注册备用位）。它是热备演练而非必需检查：每次运行都重新证明这台持久化虚拟机能够执行完整的未分片聚合流程。实际切换机制已预先布线：三个必需 Linux 作业通过仅限管理员的仓库变量 `DSH_CI_FAILOVER` 解析运行器池，因此故障响应就是设置一个变量并重跑——无需合并（合并本身会被正在失败的检查死锁）（[切换手册](ci-failover-runbook.zh.md)）。该热备通道由 push 触发，执行的始终是基线分支自身的工作流定义——不存在任何可由拉取请求编辑的路径能把代码路由到这些运行器上；此外仓库继续保持禁用 fork。 +另有一条串行 Linux 参考在每次 `master` 推送时运行于公司自有的自托管池（`vm-backup` 标签：一台 64 核虚拟机，运行 4 个常驻的 systemd 管理运行器实例，另有 4 个已注册备用位）。它是热备演练而非必需检查：每次运行都重新证明这台持久化虚拟机能够执行完整的未分片聚合流程。实际切换机制已预先布线：三个必需 Linux 作业通过仅限管理员的仓库变量 `DSH_CI_FAILOVER` 解析运行器池，因此故障响应就是设置一个变量并重跑——无需合并（合并本身会被正在失败的检查死锁）（[切换手册](2026-07-26-ci-failover-runbook.md)）。该热备通道由 push 触发，执行的始终是基线分支自身的工作流定义——不存在任何可由拉取请求编辑的路径能把代码路由到这些运行器上；此外仓库继续保持禁用 fork。 ## 曾考虑的替代方案 diff --git a/.agents/notes/implemented/process/ci-failover-runbook.i18n.yaml b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.i18n.yaml similarity index 65% rename from .agents/notes/implemented/process/ci-failover-runbook.i18n.yaml rename to .agents/notes/implemented/process/2026-07-26-ci-failover-runbook.i18n.yaml index 294ed38ddf..7a65ff5479 100644 --- a/.agents/notes/implemented/process/ci-failover-runbook.i18n.yaml +++ b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write -ci-failover-runbook.md: d22c93fbedd3216e71bc24101dfa06dc606521c2 -ci-failover-runbook.zh.md: d7d26287191165cd3cb2666de4b1c7d6217ba71d +2026-07-26-ci-failover-runbook.md: 9100cf226467d06835478b13c41904bc50270b78 +2026-07-26-ci-failover-runbook.zh.md: 4ec80ae36411335a378f7979b9bca704c17732d0 diff --git a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md new file mode 100644 index 0000000000..9100cf2264 --- /dev/null +++ b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md @@ -0,0 +1,49 @@ +# Agent Note: CI failover runbook — hosted pools → in-house pool + +Status: implemented + +English | [中文](2026-07-26-ci-failover-runbook.zh.md) + +## Problem + +The three required Linux jobs in [CI](../../../../.github/workflows/ci.yml) (`node 24 / static`, `node 24 / coverage`, `node 24 / snapshots and artifacts`) run on the hosted enterprise 32-core pools. When those pools degrade — jobs queue indefinitely, the enterprise labels vanish, or GitHub-side capacity fails — every open pull request becomes unmergeable, and the ordinary recovery of merging a fix is itself deadlocked behind the very required checks that cannot run. An outage therefore needs a switch a repository admin can throw without merging anything. + +## Decision + +Each of the three required Linux jobs resolves its runner pool through the `DSH_CI_FAILOVER` repository variable. Unset (normal), they run on the hosted enterprise pools. Set to `selfhosted` by a repository admin, all three retarget onto the in-house self-hosted `vm-backup` pool, coverage and snapshot concurrency drop to shared-VM bounds, and the hosted-path pnpm cache restores are skipped. The switch is admin-only repository state, not a merge, so it works while every check is red. The in-house pool's readiness is continuously re-proven by the `serial / linux (self-hosted standby)` lane, which runs the complete unsharded aggregate on every master push. + +### What the in-house pool is + +`vm-backup`: one 64-core VM, four always-on systemd-managed runner instances, four registered spares. Check the latest `serial / linux (self-hosted standby)` run before switching: a green standby is verified-yesterday capacity. + +### Switch (repo admin, ~1 minute, no merge) + +1. Repository **Settings → Secrets and variables → Actions → Variables → New repository variable**: name `DSH_CI_FAILOVER`, value `selfhosted`. +2. Re-run the failed/queued required jobs (Re-run failed jobs on affected PRs, or let new pushes pick it up). +3. That is the entire switch. Under failover the workflow also, automatically: halves `DSH_COVERAGE_MAX_WORKERS` to 12 and `DSH_SNAPSHOT_MAX_CONCURRENCY` to 16 (shared-VM contention bounds), and skips the hosted-path pnpm cache restores (the VM's persistent store serves warm installs). + +### Capacity during failover + +Four always-on instances absorb normal PR traffic. If queues build, bring the four registered spares online on the VM (no token needed — they are already registered): + +```bash +for i in 7 8 9 10; do cd /data_local/actions-runner-$i && sudo ./svc.sh install ubuntu && sudo ./svc.sh start; done +``` + +### Switch back + +Delete the `DSH_CI_FAILOVER` variable (or set it to anything other than `selfhosted`). New runs resolve back to the hosted enterprise pools. Stop the spare instances if they were started. + +### Trust boundary + +The variable is repository-admin-only state: a pull request can neither set it nor read a different value into effect, and the expressions live in the base branch's workflow definition. This failover path therefore adds no PR-editable route to the self-hosted pool. Runner-side enforcement — an org-level runner group restricting these runners to the master-ref workflow — is tracked separately and composes with this mechanism. + +## Alternatives considered + +**Merge a workflow change to switch pools.** Rejected because the outage that motivates the switch is exactly the state in which no PR can merge: the required checks are the ones failing. A repository variable is admin-controlled state that takes effect on re-run without a merge. + +**Keep the self-hosted pool always in the required path.** Rejected because it trades hosted-pool availability for the in-house VM's, moving a single point of failure rather than adding a fallback. The variable keeps the hosted pools primary and the self-hosted pool a proven, one-action standby. + +## Consequences + +Recovering from a hosted-pool outage is a single admin variable plus a re-run, with no merge on the critical path. The cost is a second runner topology to keep working: the standby lane exercises it on every master push so the failover target never goes stale, and the concurrency and cache-restore branches in `ci.yml` carry a `selfhosted` leg that must stay in step with the hosted leg. diff --git a/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md new file mode 100644 index 0000000000..4ec80ae364 --- /dev/null +++ b/.agents/notes/implemented/process/2026-07-26-ci-failover-runbook.zh.md @@ -0,0 +1,49 @@ +# Agent Note: CI 故障切换手册 — 托管池 → 自有池 + +Status: implemented + +[English](2026-07-26-ci-failover-runbook.md) | 中文 + +## 问题 + +[CI](../../../../.github/workflows/ci.yml) 中三个必需的 Linux 作业（`node 24 / static`、`node 24 / coverage`、`node 24 / snapshots and artifacts`）运行在托管的企业级 32 核池上。当这些托管池发生故障——作业无限排队、企业标签消失或 GitHub 侧容量故障——所有开启的拉取请求都无法合并，而"合并一个修复"这一常规恢复手段本身正被那些无法运行的必需检查死锁。因此故障需要一个仓库管理员无需合并任何代码即可触发的开关。 + +## 决策 + +三个必需的 Linux 作业各自通过仓库变量 `DSH_CI_FAILOVER` 解析运行器池。变量不存在（正常）时它们运行在托管企业池上；由仓库管理员设为 `selfhosted` 时，三者全部切换到公司自有的自托管 `vm-backup` 池，coverage 与 snapshot 的并发降到共享虚拟机上限，并跳过托管路径的 pnpm 缓存恢复。这个开关是仅限管理员的仓库状态而非一次合并，因此在所有检查都是红色时仍然有效。自有池的就绪状态由 `serial / linux (self-hosted standby)` 通道持续验证——每次 master 推送都在其上运行完整的未分片聚合流程。 + +### 自有池是什么 + +`vm-backup`：一台 64 核虚拟机，4 个常驻 systemd 管理的运行器实例，另有 4 个已注册备用位。切换前先看 `serial / linux (self-hosted standby)` 最近一次运行：绿色 = 这套环境昨天刚被全量验证过。 + +### 切换步骤（仓库管理员，约 1 分钟，无需合并） + +1. 仓库 **Settings → Secrets and variables → Actions → Variables → New repository variable**：名称 `DSH_CI_FAILOVER`，值 `selfhosted`。 +2. 对受影响 PR 的失败/排队作业点 Re-run failed jobs（或等新推送自然触发）。 +3. 切换到此完成。故障切换状态下工作流还会自动：把 `DSH_COVERAGE_MAX_WORKERS` 降为 12、`DSH_SNAPSHOT_MAX_CONCURRENCY` 降为 16（共享虚拟机的争抢上限），并跳过托管路径的 pnpm 缓存恢复（虚拟机的持久 store 直接提供热安装）。 + +### 切换期间的容量 + +4 个常驻实例可承接正常 PR 流量。若出现排队，在虚拟机上把 4 个已注册的备用位拉起（无需 token——它们已注册）： + +```bash +for i in 7 8 9 10; do cd /data_local/actions-runner-$i && sudo ./svc.sh install ubuntu && sudo ./svc.sh start; done +``` + +### 切回 + +删除 `DSH_CI_FAILOVER` 变量（或改为 `selfhosted` 以外的任何值），新的运行即解析回托管企业池。若启动过备用实例，将其停止。 + +### 信任边界 + +该变量是仅限仓库管理员的状态：拉取请求既不能设置它，也不能让不同的值生效，且表达式存在于基线分支的工作流定义中。因此这条故障切换路径没有增加任何可由 PR 编辑的自托管池访问途径。运行器侧的强制约束——通过组织级 runner group 把这批运行器限定到 master 引用的工作流——另行跟踪，与本机制互补。 + +## 曾考虑的替代方案 + +**通过合并一次工作流改动来切换池。** 否决，因为触发切换的故障状态恰恰是任何 PR 都无法合并的状态：必需检查正是失败的那些。仓库变量是管理员控制的状态，重跑即生效，无需合并。 + +**让自托管池长期处于必需路径中。** 否决，因为这是拿托管池的可用性去换自有虚拟机的可用性，只是搬移了单点故障而非增加回退。该变量让托管池保持主路径，自托管池作为一个经过验证、一步即可启用的热备。 + +## 后果 + +从托管池故障中恢复只需一个管理员变量加一次重跑，关键路径上没有合并。代价是要维护第二套运行器拓扑：热备通道在每次 master 推送时都运行它，使故障切换目标永不失效；而 `ci.yml` 中的并发与缓存恢复分支带有一条 `selfhosted` 支路，必须与托管支路保持同步。 diff --git a/.agents/notes/implemented/process/ci-failover-runbook.md b/.agents/notes/implemented/process/ci-failover-runbook.md deleted file mode 100644 index d22c93fbed..0000000000 --- a/.agents/notes/implemented/process/ci-failover-runbook.md +++ /dev/null @@ -1,33 +0,0 @@ -# Agent Note: CI failover runbook — hosted pools → in-house pool - -Status: implemented - -English | [中文](ci-failover-runbook.zh.md) - -## What this is - -The three required Linux jobs in [CI](../../../../.github/workflows/ci.yml) (`node 24 / static`, `node 24 / coverage`, `node 24 / snapshots and artifacts`) resolve their runner pool through the `DSH_CI_FAILOVER` repository variable. Normally the variable is unset and they run on the hosted enterprise 32-core pools. When the hosted pools are degraded (jobs queue indefinitely, the enterprise labels vanish, or GitHub-side capacity fails), a repository admin can retarget all three onto the in-house self-hosted pool without merging anything — merging would itself be blocked by the very checks that are failing. - -The in-house pool (`vm-backup`: one 64-core VM, four always-on systemd-managed runner instances, four registered spares) is continuously re-proven by the `serial / linux (self-hosted standby)` lane, which runs the complete unsharded aggregate on every master push. Check its latest run before switching: green standby = verified-yesterday capacity. - -## Switch (repo admin, ~1 minute, no merge) - -1. Repository **Settings → Secrets and variables → Actions → Variables → New repository variable**: name `DSH_CI_FAILOVER`, value `selfhosted`. -2. Re-run the failed/queued required jobs (Re-run failed jobs on affected PRs, or let new pushes pick it up). -3. That is the entire switch. Under failover the workflow also, automatically: halves `DSH_COVERAGE_MAX_WORKERS` to 12 and `DSH_SNAPSHOT_MAX_CONCURRENCY` to 16 (shared-VM contention bounds), and skips the hosted-path pnpm cache restores (the VM's persistent store serves warm installs). - -## Capacity during failover - -Four always-on instances absorb normal PR traffic. If queues build, bring the four registered spares online on the VM (no token needed — they are already registered): - -```bash -for i in 7 8 9 10; do cd /data_local/actions-runner-$i && sudo ./svc.sh install ubuntu && sudo ./svc.sh start; done -``` - -## Switch back - -Delete the `DSH_CI_FAILOVER` variable (or set it to anything other than `selfhosted`). New runs resolve back to the hosted enterprise pools. Stop the spare instances if they were started. - -## Trust boundary - -The variable is repository-admin-only state: a pull request can neither set it nor read a different value into effect, and the expressions live in the base branch's workflow definition. This failover path therefore adds no PR-editable route to the self-hosted pool. (Runner-side enforcement — an org-level runner group restricting these runners to the master-ref workflow — is tracked separately and composes with this mechanism.) diff --git a/.agents/notes/implemented/process/ci-failover-runbook.zh.md b/.agents/notes/implemented/process/ci-failover-runbook.zh.md deleted file mode 100644 index d7d2628719..0000000000 --- a/.agents/notes/implemented/process/ci-failover-runbook.zh.md +++ /dev/null @@ -1,33 +0,0 @@ -# Agent Note: CI 故障切换手册 — 托管池 → 自有池 - -Status: implemented - -[English](ci-failover-runbook.md) | 中文 - -## 这是什么 - -[CI](../../../../.github/workflows/ci.yml) 中三个必需的 Linux 作业（`node 24 / static`、`node 24 / coverage`、`node 24 / snapshots and artifacts`）通过仓库变量 `DSH_CI_FAILOVER` 解析运行器池。正常情况下该变量不存在，作业运行在托管的企业级 32 核池上。当托管池发生故障（作业无限排队、企业标签消失或 GitHub 侧容量故障）时，仓库管理员无需合并任何代码即可把三个作业整体切换到公司自有的自托管池——此时合并本身正被这些失败的检查阻塞，任何"先合 PR 再切换"的方案都是死锁。 - -自有池（`vm-backup`：一台 64 核虚拟机，4 个常驻 systemd 管理的运行器实例，另有 4 个已注册备用位）由 `serial / linux (self-hosted standby)` 通道持续验证——每次 master 推送都在其上运行完整的未分片聚合流程。切换前先看该通道最近一次运行：绿色 = 这套环境昨天刚被全量验证过。 - -## 切换步骤（仓库管理员，约 1 分钟，无需合并） - -1. 仓库 **Settings → Secrets and variables → Actions → Variables → New repository variable**：名称 `DSH_CI_FAILOVER`，值 `selfhosted`。 -2. 对受影响 PR 的失败/排队作业点 Re-run failed jobs（或等新推送自然触发）。 -3. 切换到此完成。故障切换状态下工作流还会自动：把 `DSH_COVERAGE_MAX_WORKERS` 降为 12、`DSH_SNAPSHOT_MAX_CONCURRENCY` 降为 16（共享虚拟机的争抢上限），并跳过托管路径的 pnpm 缓存恢复（虚拟机的持久 store 直接提供热安装）。 - -## 切换期间的容量 - -4 个常驻实例可承接正常 PR 流量。若出现排队，在虚拟机上把 4 个已注册的备用位拉起（无需 token——它们已注册）： - -```bash -for i in 7 8 9 10; do cd /data_local/actions-runner-$i && sudo ./svc.sh install ubuntu && sudo ./svc.sh start; done -``` - -## 切回 - -删除 `DSH_CI_FAILOVER` 变量（或改为 `selfhosted` 以外的任何值），新的运行即解析回托管企业池。若启动过备用实例，将其停止。 - -## 信任边界 - -该变量是仅限仓库管理员的状态：拉取请求既不能设置它，也不能让不同的值生效，且表达式存在于基线分支的工作流定义中。因此这条故障切换路径没有增加任何可由 PR 编辑的自托管池访问途径。（运行器侧的强制约束——通过组织级 runner group 把这批运行器限定到 master 引用的工作流——另行跟踪，与本机制互补。） diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 88722804f9..0be7655193 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -37,7 +37,7 @@ jobs: # 'selfhosted' (repo Settings → Actions → Variables; admin-only, not # PR-editable, no merge required) retargets all three onto the in-house # vm-backup pool and re-running the failed jobs is the entire switch — - # see .agents/notes/implemented/process/ci-failover-runbook.md. The + # see .agents/notes/implemented/process/2026-07-26-ci-failover-runbook.md. The # in-house pool's readiness is re-proven on every master push by the # serial-linux-selfhosted standby lane below. node-24: