- serial-linux-selfhosted checks out fetch-depth 0: depth 2 misses
github.event.before on multi-commit or force pushes, failing the
archive verifier on a valid tree. Full fetch is cheap against the
VM's local mirror.
- Runbook (both languages): every remaining admin phrasing (problem
statement, switch heading, alternatives, consequences) now says
writer; and the 'composes with this mechanism' claim about a
master-ref-pinned runner group is replaced with the truth observed
live on 2026-07-27 — master-ref pinning blocks PR failover, and the
shipped posture is repository-scoped all-workflow group access.
Static gate green locally: 32 passed, 0 failed.
The standby lane itself is push-only, but under failover pull_request
jobs do reach these runners with the PR merge ref's workflow. The
workflow comment and the larger-runner note (both languages) now state
that plainly and name the actual boundary — repository membership
(private, forking disabled, Dependabot excluded) — matching the
runbook. Static gate green locally: 32 passed, 0 failed.
- Sweep every remaining 'admin-only' claim (workflow comments, runbook
lines 13/40, topology note, all zh pairs): the variable is
writer-manageable, and the boundary against untrusted code is
repository membership (private, forking disabled, Dependabot
excluded) — stated identically at every site instead of only in the
'who can flip' paragraph.
- Serial cross-platform reference note (both languages): master now
runs four references — the three hosted OS legs plus the self-hosted
standby drill, linked to the failover runbook.
Static gate green locally: 32 passed, 0 failed.
- serial-linux-selfhosted now fetches depth 2 and passes
DSH_ARCHIVE_BASE_REF=github.event.before, running the same
frozen-archive comparison as serial-linux instead of diffing the
new manifest against itself.
- Runbook (both languages): documents the deliberate dependabot
exception (queued-on-hosted during failover is expected, not a
failed switch); corrects the emergency-capacity bootstrap to
exclude .runner/.credentials when cloning a runner directory; and
replaces the 'admin-only' variable claim with the accurate
trust-model statement — repository variables are writer-manageable,
which in this private fork-disabled repo with an all-workflows
runner group is routing among members, not an escalation.
Static gate green locally: 32 passed, 0 failed.
- All four failover selectors (three workers + the verdict job) and the
paired env/cache expressions now exclude dependabot[bot]: under
failover, dependency-supplied code keeps queueing for the hosted pool
instead of executing on the persistent VM. A delayed Dependabot PR
during an outage is an acceptable cost; dependency code on the
privileged host is not.
- Runbook (both languages): records the shipped failover bounds
(coverage 8, snapshots 12, sized for six instances) and documents
that the verdict job follows the selector too — operators previously
had no explanation for a verdict queued after all workers passed.
- Local static gate green: 32 passed, 0 failed (translation pairing
519 pairs consistent).
- all-checks-passed now resolves its pool through the same
DSH_CI_FAILOVER expression as the worker jobs it aggregates.
Pinned to the hosted pool it would leave the branch-protection
verdict queued on the failed pool after every failover job passed —
observed live during the 2026-07-27 outage as a required check
looping against dead capacity.
- Coverage worker bound under failover drops 12 → 8 and snapshot
concurrency 16 → 12: the pool now runs six always-on instances (the
spare tier was retired), so worst case is 6 × 8 = 48 coverage
workers on the shared 64-core VM.
Review round on the pivoted design:
- node-24 (static) kept an unconditional hosted pnpm cache restore while the
coverage and consumers lanes skip it under failover. On the self-hosted VM
that restore downloads ~180 MB into /home/runner, a path pnpm never reads
there, adding latency and contention during an outage. Gate it with the same
`vars.DSH_CI_FAILOVER != 'selfhosted'` condition so all three lanes match.
- Runbook switch step 2 said "Re-run failed jobs", but the documented
indefinite-queue outage leaves jobs queued (not failed), which cannot be
re-run in place and do not retarget on variable change. Correct both language
sides to cancel the run and re-run all jobs, or push a new commit.
- The standby-lane comment still described the switch as a one-line runs-on
change; it is now setting the admin-only DSH_CI_FAILOVER variable.
The failover runbook landed as .agents/notes/implemented/process/ci-failover-runbook.md,
which fails three doc-sync gates: the classification/format gates require a
yyyy-mm-dd-topic.md filename and the implemented Agent Note skeleton
(Problem/Decision/Alternatives/Consequences), and the bilingual pairing gate
requires cross-note link targets to match between the two language sides.
Rename to 2026-07-26-ci-failover-runbook.md/.zh.md, reshape both sides into the
implemented skeleton (the runbook steps live in bespoke sections under Decision),
point the sibling topology note and the ci.yml comment at the dated filename, and
make both sides link the canonical .md per the bilingual convention. Re-recorded
the i18n pairing records.
The three required Linux jobs now resolve their pool through the
DSH_CI_FAILOVER repository variable. Unset, everything runs exactly as
today on the hosted enterprise pools. Setting it to 'selfhosted'
(repo-admin-only, not PR-editable, no merge required — a merge would be
deadlocked behind the failing checks themselves) retargets all three
onto the vm-backup pool, halves the coverage worker bound and snapshot
concurrency for the shared VM, and skips the hosted-path cache restores.
Adds a bilingual failover runbook (switch, capacity via the four
registered spare instances, switch-back, trust boundary) and links it
from the topology note. The push-triggered standby lane remains the
continuous proof that the failover target works.
Direction change after review discussion. Moving a REQUIRED check onto
a single in-house VM traded merge-path availability for modest savings
and accumulated trust/contention caveats (six review rounds' worth).
Revert every coverage-lane change: coverage stays on the enterprise
Ubuntu 24.04 32-core pool exactly as on master.
Instead, add serial-linux-selfhosted: on every master push the in-house
pool (vm-backup) runs the complete unsharded primary aggregate as a
hot-standby drill. It blocks nothing, yet continuously proves the
environment end to end, so any hosted-pool outage can be answered with
a one-line runs-on retarget onto continuously verified capacity.
Push-triggered lanes execute the base branch's own workflow definition,
so no PR-editable path selects these runners — the entire fork-trust
discussion is structurally moot for this lane. Topology notes (en/zh +
pairing records) describe the standby lane and the switch play.
The hosted 32-core runner is exclusive to one job, but the vm-backup
pool shares one 64-core VM across four runner instances; concurrent
PRs could stack 4×24 = 96 Vitest workers and re-trigger the documented
aggregate-contention failures in the timing-sensitive process suites.
Bound the self-hosted leg at 12 workers per job (48 host-wide fully
loaded) and keep 24 on the hosted leg, selected by the same expression
as the pool.
Sweep all remaining sources that still described coverage as an
enterprise 32-core job: the ci.yml jobs preamble, the three-job
paragraph of the larger-hosted-runners note, and the required-pool
sentence of the portable-recovery note — English and Chinese sides of
both notes, with their i18n pairing records re-recorded.
Keep the cache restore for the ephemeral hosted (untrusted-PR) leg where
it is a genuine speedup, gated by the same expression as the runs-on
pool selector; the self-hosted leg skips it and installs from the
persistent local store.
- Route untrusted PRs (forks + Dependabot, same author test as e2e.yml)
back to the hosted enterprise pool via a runs-on expression: Dependabot
PRs are same-repo, so the previous head.repo guard admitted
dependency-supplied code onto the persistent self-hosted VM. A single
job with pool selection keeps all-checks-passed free of skips.
- Drop the pnpm-store cache restore from this lane: on self-hosted the
hosted-path cache actually HIT (Linux key) and spent ~52 s pulling
181 MB into a path pnpm never reads; the persistent local store
already serves warm installs in seconds.
- Update the larger-hosted-runners Agent Note (en/zh + i18n pairing
record) so the decision record describes the shipped topology:
coverage on the in-house vm-backup pool for trusted PRs, hosted
Ubuntu 24.04 32-core retained for untrusted PRs.
- Restrict node-24-coverage to same-repo PRs so fork-originated code can
never reach the self-hosted runner (defense in depth; the repo is
private with forking disabled today).
- Revert the pnpm cache path to the literal /home/runner/... save-side
path: actions/cache hashes the path into the cache version, so the ~
variant could never match the cache saved by the master lane. On
self-hosted the persistent local pnpm store covers warm installs.
- Drop the incorrect 'does not gate merges' claim: node-24-coverage is
needed by all-checks-passed. Pool capacity notes moved into comments.
Coverage does not gate merges, so move it off the metered
dsh-enterprise-ubuntu-24-04-32core-test pool onto the in-house
self-hosted pool (vm-backup label, 64-core).
Also switch the pnpm store cache path to ~ so it resolves under
both /home/runner (hosted) and self-hosted home directories.
Verified on the self-hosted pool: the full coverage job (including
prepare-ci-bubblewrap and the exhaustive suite) completed green in
~5 min.
tsconfig.vitest.json is deleted; its job (a paths map applying to all
test files) is inherent in tsconfig.base.json having no include —
vite-tsconfig-paths treats that as match-all. All four vitest configs
now pin the same facade: the unit config gains a shared pathsPlugin()
helper, the web lane drops its handwritten webserver alias (apps/web
tests are covered by match-all), and the snapshot lane leaves the root
solution (which no longer carries paths) and stops resolving client
imports through package exports. tsconfig.base.json documents the
facade role and bans include/files; the CI eslint cache key hashes the
four graph tsconfigs; the eslint tests-block comment states resolution
via the solution to tsconfig.host.json.
Per missions/tsconfig-single-graph-migration.md §3.
Skipped matrix jobs never expand matrix display values, so let GitHub use stable job identifiers. Bind optimized jobs to pull requests and the three serial references to master pushes.
