Replace the single DSH_CI_FAILOVER variable with two independent
switches so an outage on one platform no longer retargets the other:
- DSH_CI_FAILOVER_LINUX: the three required Linux workers (node-24,
node-24-coverage, node-24-consumers) and the all-checks-passed verdict,
which resolves its pool to vm-backup and keeps its concurrency and
cache-restore branches.
- DSH_CI_FAILOVER_WINDOWS: the non-blocking windows-native job, which
resolves to the dsh-win-ci pool.
all-checks-passed rides the Linux switch because it aggregates the
required Linux workers and runs on the vm-backup pool. The Dependabot
exclusion is preserved on both switches. The failover runbook (EN/ZH)
and its translation pairing, plus the docs that referenced the old
variable, are updated in the same change.
cancel-in-progress is evaluated against the newly triggered run, so
exempting push means one master merge does not cancel the drill still
running from the previous one — not that a drill always finishes. A
benchmark dispatched on master shares the group and does cancel a
mid-flight drill. Record that bound in the runbook and drop the
overstated wording from the workflow comment and the spec name.
Also repair two gates that fail on master and block every pull request:
the telemetry note referenced an SDK proposal deleted in 408721954a,
and the ui-settings-general README pair carried stale recorded hashes
after both sides were updated together in aa1ec02bc6.
The two self-hosted standby drills each run their complete unsharded
aggregate with one gate worker, which takes longer than the interval
between master merges, so unconditional cancel-in-progress supersedes a
drill before it reaches a verdict and the lane yields no readiness
evidence for the failover runbook to point a responder at.
Exempt push and nothing else. This has to be decided at workflow level:
cancellation applies to the whole superseded run, so a job-level
concurrency group cannot exempt its job. The negated form is
load-bearing — naming pull_request alone would also stop cancelling
workflow_dispatch, and each runner benchmark fans out to twelve larger
runners for up to fifteen minutes in this same group on master, so a
re-dispatch would queue ahead of a drill instead of replacing a stale
measurement. It does not promise every push run finishes: a newer
pending run still displaces an older one, only that the lanes
periodically reach a verdict.
A master push carries only wine-apt-cache and the two drills; every other
job is pull-request-gated, workflow_dispatch-gated, or if: false. The
spec pins that set and classifies by exact condition, since a negated
event test mentions the event it excludes.
Cover the hosted Windows-pool outage in scope, add the dsh-win-ci pool description (32 scheduled-task runner instances, 96-core machine, Git Bash on PATH prerequisite), and update the switch to retarget all five jobs (three Linux workers, native Windows, all-checks-passed).
Restore the Wine-emulated windows job as the required pull-request
Windows signal. Add DSH_CI_FAILOVER switch to windows-native so it
retargets onto the self-hosted [self-hosted, dsh-win-ci, windows]
pool under failover. Enable serial-windows as a master-only
self-hosted standby mirroring serial-linux-selfhosted.
The windows-native job remains absent from all-checks-passed.needs —
Wine owns the required verdict, native Windows reports independently.
Restore wine-apt-cache, wine-windows-gates.sh, check:windows-wine
script, and the original Agent Note topology.
- Rewrite ci-workflow.spec.ts to assert new single-job native Windows topology
- Remove check:windows-wine from root AGENTS.md command list
- Archive 2026-08-08-native-windows-pull-request-ci (dual-lane topology
superseded) with Archived: metadata and inbound-link updates
- Update cross-references in serial-cross-platform-ci-reference,
portable-required-pull-request-ci, and pnpm-action-setup notes
- Add Windows pool coverage to the failover runbook (EN + zh)
- Remove dead wine-windows-gates.sh
- Remove unnecessary Playwright Chromium install from serial-windows
- Fix .github/AGENTS.md windows-* label wording
- Fix PR description before-state accuracy
- The clone exclusion is now '.runner*' (catches .runner_migrated /
.credentials_migrated, which GitHub writes on migrated runners and
which equally trip the already-configured refusal — hit live during
the 2026-07-27 org-group migration).
- The problem statement now states the verdict runs on ubuntu-latest,
scopes the switch to enterprise Linux-pool outages, and names the
residual dependency: node-compat/python-sdk/windows stay standard-
hosted, so a broader hosted-capacity failure still blocks the
verdict, with Windows the only leg lacking any in-house substitute.
Static gate green locally: 32 passed, 0 failed.
config.sh only registers; the runner stays offline until svc.sh
install/start. Both language sides updated so emergency capacity
actually comes online.
- serial-linux-selfhosted checks out fetch-depth 0: depth 2 misses
github.event.before on multi-commit or force pushes, failing the
archive verifier on a valid tree. Full fetch is cheap against the
VM's local mirror.
- Runbook (both languages): every remaining admin phrasing (problem
statement, switch heading, alternatives, consequences) now says
writer; and the 'composes with this mechanism' claim about a
master-ref-pinned runner group is replaced with the truth observed
live on 2026-07-27 — master-ref pinning blocks PR failover, and the
shipped posture is repository-scoped all-workflow group access.
Static gate green locally: 32 passed, 0 failed.
- Sweep every remaining 'admin-only' claim (workflow comments, runbook
lines 13/40, topology note, all zh pairs): the variable is
writer-manageable, and the boundary against untrusted code is
repository membership (private, forking disabled, Dependabot
excluded) — stated identically at every site instead of only in the
'who can flip' paragraph.
- Serial cross-platform reference note (both languages): master now
runs four references — the three hosted OS legs plus the self-hosted
standby drill, linked to the failover runbook.
Static gate green locally: 32 passed, 0 failed.
- serial-linux-selfhosted now fetches depth 2 and passes
DSH_ARCHIVE_BASE_REF=github.event.before, running the same
frozen-archive comparison as serial-linux instead of diffing the
new manifest against itself.
- Runbook (both languages): documents the deliberate dependabot
exception (queued-on-hosted during failover is expected, not a
failed switch); corrects the emergency-capacity bootstrap to
exclude .runner/.credentials when cloning a runner directory; and
replaces the 'admin-only' variable claim with the accurate
trust-model statement — repository variables are writer-manageable,
which in this private fork-disabled repo with an all-workflows
runner group is routing among members, not an escalation.
Static gate green locally: 32 passed, 0 failed.
- All four failover selectors (three workers + the verdict job) and the
paired env/cache expressions now exclude dependabot[bot]: under
failover, dependency-supplied code keeps queueing for the hosted pool
instead of executing on the persistent VM. A delayed Dependabot PR
during an outage is an acceptable cost; dependency code on the
privileged host is not.
- Runbook (both languages): records the shipped failover bounds
(coverage 8, snapshots 12, sized for six instances) and documents
that the verdict job follows the selector too — operators previously
had no explanation for a verdict queued after all workers passed.
- Local static gate green: 32 passed, 0 failed (translation pairing
519 pairs consistent).
The spare tier is retired. Steady-state pool load is one serial standby
job per master push, so six always-on instances already are the
failover capacity; pre-registered offline runners are a silently
expiring guarantee (GitHub garbage-collects them after 30 days
offline). Incident-time extra capacity is a one-minute org-token
registration, now documented in the runbook.
Review round on the pivoted design:
- node-24 (static) kept an unconditional hosted pnpm cache restore while the
coverage and consumers lanes skip it under failover. On the self-hosted VM
that restore downloads ~180 MB into /home/runner, a path pnpm never reads
there, adding latency and contention during an outage. Gate it with the same
`vars.DSH_CI_FAILOVER != 'selfhosted'` condition so all three lanes match.
- Runbook switch step 2 said "Re-run failed jobs", but the documented
indefinite-queue outage leaves jobs queued (not failed), which cannot be
re-run in place and do not retarget on variable change. Correct both language
sides to cancel the run and re-run all jobs, or push a new commit.
- The standby-lane comment still described the switch as a one-line runs-on
change; it is now setting the admin-only DSH_CI_FAILOVER variable.
The failover runbook landed as .agents/notes/implemented/process/ci-failover-runbook.md,
which fails three doc-sync gates: the classification/format gates require a
yyyy-mm-dd-topic.md filename and the implemented Agent Note skeleton
(Problem/Decision/Alternatives/Consequences), and the bilingual pairing gate
requires cross-note link targets to match between the two language sides.
Rename to 2026-07-26-ci-failover-runbook.md/.zh.md, reshape both sides into the
implemented skeleton (the runbook steps live in bespoke sections under Decision),
point the sibling topology note and the ci.yml comment at the dated filename, and
make both sides link the canonical .md per the bilingual convention. Re-recorded
the i18n pairing records.
