mirror of
https://github.com/deepseek-ai/deepseek-harness
synced 2026-08-15 21:04:50 +00:00
Merge remote-tracking branch 'origin/master' into worktree/llm-reasoning-effort
# Conflicts: # docs/architecture.i18n.yaml # docs/config-catalog.md
This commit is contained in:
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
2026-07-24-web-gui-browser-e2e-lane.md: f7f5d6382e6b7d4a7f2949cc43e0be1778ab9516
|
||||
2026-07-24-web-gui-browser-e2e-lane.zh.md: f3ded2085eb383dbbfb847e2166533328a6e32a8
|
||||
@@ -0,0 +1,90 @@
|
||||
# Agent Note: Keyless browser e2e lane for the web GUI
|
||||
|
||||
Status: implemented
|
||||
|
||||
English | [中文](2026-07-24-web-gui-browser-e2e-lane.zh.md)
|
||||
|
||||
## Problem
|
||||
|
||||
The web GUI ships as a real assembled chain — chromium page → client plugin bundles → HTTP unary RPC + two SSE streams → `toFetchHandler`/apiproxy → the host agent loop, tools, and JSONL persistence — and no test exercised that chain keylessly and deterministically. The [GUI testing system](../process/2026-07-20-gui-testing-system.md) covers tier 1 (wire isomorphism in node), tier 2 (object-layer state machines), and tier-3 smokes, but the keyless smoke drives `FixtureApiClient` — no host, no wire, no agent loop — while the full-chain smoke needs `DEEPSEEK_API_KEY` and a live model, so it is nondeterministic and self-skips in keyless CI. The snapshot philosophy of [docs/testing.md](../../../../docs/testing.md) — record once with a key, replay forever keyless, refresh on format churn — already covers the ACP, headless `stream-json`, and TUI transcript surfaces; the web surface was the one assembled product shape without it. The gap is exactly where the two confirmed GUI P0s hid: the wire carriage chain the fixture client short-circuits.
|
||||
|
||||
## Decision
|
||||
|
||||
`pnpm run test:web` carries a keyless, deterministic browser e2e lane under `apps/web/tests/`: recorded session-log fixtures replayed through `@deepseek-ai/dsh-llm-replay` against the real in-process web composition, asserting a normalized conversation aria golden plus in-process world state. No new package; the product deltas are two additive `dsh-llm-replay` surfaces (`paceMs`, `ReplayHandle`).
|
||||
|
||||
### Scaffold: `apps/web/tests/scaffold.ts`
|
||||
|
||||
A plain shared-fixture module (the [testing-policy sanctioned shape](../../../../docs/testing.md)), not a package: the gate-worthy logic — replay derivation, session parsing, log scrubbing, persistence — lives in the gated packages `dsh-llm-replay`, `dsh-acp-snapshot`, and `dsh-session-persistence-jsonl`; what remains is boot wiring and browser glue, and chromium-driving code cannot hold per-file 100% coverage on the browserless coverage runners.
|
||||
|
||||
`launchWebScaffold()` boots the real web composition from the shipped `apps/cli/cordis.yml` through the vendored Loader's include mechanism — the same tree and mechanism `AppCLIEntry` drives for `dsh web`. Divergences ride include patches over that tree, the ACP `cordis.snapshot.yml` pattern expressed in-process: temp `persistenceRoot`, `workspace-context` disabled (recorded fixtures must not embed this repo's AGENTS.md), `session-title-llm` disabled (its fire-and-forget title call would race the loop for the session's replay cursor), the webserver row pinned to port 0 with the built dist, and in keyless modes `llm-deepseek` disabled. A patch id that stops matching a row fails the boot sweep loudly instead of drifting. The boot runs `chdir`'d to the temp workspace so the api-gateway's `process.cwd()` session default, tool cwds, and fixtures agree; the `dsh web` bin's own glue (argv, profile json, AppCLIEntry) stays held by the keyless CLI smokes in `smoke-real.e2e.ts`. Setup rollback and ordinary close both dispose the Cordis tree before removing the two owned temp roots, attempt every cleanup independently, and report cleanup failures without masking the setup failure.
|
||||
|
||||
Keyless model displacement is the disabled adapter row plus `installLlmReplay` filling the open seam on the settled root ctx in providers-catalog mode — never catch-all: with the adapter row disabled no adapter exists, so catch-all would leave `resolveModelInfo` unroutable and `compact-basic`'s post-step pressure check would warn every step instead of being provably inert (the published 128k `contextWindow` keeps it inert for small fixtures). The direct install rather than an inserted replay plugin row is deliberate: it returns the `ReplayHandle` the teardown consumption check needs. A scenario with no fixture leaves the seam empty, so a stray stream fails loud with NO_ADAPTER.
|
||||
|
||||
`seedSession()` seeds cold sessions through the real persistence API — a throwaway `Context` mounting `SessionStore` + `SessionPersistenceJsonl` against the host's root, `create()` + `append()`, one `utimes` backdate for deterministic sidebar order (the `semantic-checkpoint.snapshot.ts` precedent) — never raw file writes, so the seeder knows nothing of bucket hashing, filename encoding, or compression, and the host's zstd default needs no boot knob. Seeds are validated at seed time (parseable, ending in `turn/end` — an open final turn would be mutated by resume's crash repair).
|
||||
|
||||
### Determinism rules
|
||||
|
||||
The barrier stack for replay-mode browser assertions is, in order: (1) host-side `await agent.whenIdle()` under a timeout, keyed off the in-process `turn/end` — the idle flip follows the persistence flush, so one await covers turn completion and durability; (2) browser settled poll (streaming detached, final text visible). Record-mode log harvest runs after `whenIdle()` and before scaffold disposal while the live session remains available. An in-process `turn/end` listener alone is a wrong barrier (it fires before the SSE frame reaches the browser and before the fsync); file polling is banned (slow on NFS, superseded by `whenIdle`); `networkidle` is banned outright (never resolves while an SSE stream is open).
|
||||
|
||||
No single-shot transient-DOM assertions: every hop from replay yield to React commit can coalesce chunks, so sampling `[data-streaming]` is a race by construction. Streaming incrementality is asserted from the persisted `assistant/chunk` events (model-visible ⟺ logged makes the log the authoritative proof). `dsh-llm-replay`'s opt-in `paceMs` (default absent = burst) is a realism knob so the browser observes genuinely incremental SSE; correctness never leans on it, and abort during a pace wait cancels promptly.
|
||||
|
||||
Every scenario fails on any pageerror and on the client's connection-loss/gap-repair console warnings: the reconnect machine plus history resync would otherwise self-heal a dead SSE path and the suite would certify a broken wire. Scaffold `close()` calls the `ReplayHandle.assertConsumed()` teardown check (every recorded script bound, every cursor drained), converting silent underruns and shifted bindings into crisp diagnostics. No vitest retry on the lane; one chromium per file, fresh context per scenario, one host per scenario; viewport pinned; interaction selectors anchor on roles, `data-*` attributes, and visible text, while the frame and conversation-region captures use the existing CSS-module local-name anchors.
|
||||
|
||||
### Expected outputs
|
||||
|
||||
One committed golden per scenario: a normalized `ariaSnapshot()` of the conversation region (`ui.expected.md`) — uuid/cwd/workspace-basename/duration tokens normalized, captured poll-until-equal at the settled milestone — plus a few role/text anchor assertions that stay green under a semantics-preserving component rewrite while the golden churns reviewably. The aria tree is the mechanization of the client rule "assert what the user would see, never class names". World-state assertions ride root-context session events inline (which tool call produced which durable result, whether `turn/end` completed) instead of a second committed log golden: the persisted-log surface is pinned by the ACP/headless/TUI suites through the same loop and persistence, and re-pinning it here would double refresh cost against the tier discipline. `refresh` is the sole golden writer — a missing golden in replay mode fails with the healing command rather than self-bootstrapping.
|
||||
|
||||
The typecheck plane split is structural: the three files that boot the host spine (`scaffold`, `replay-round-trip.e2e`, and `seeded-history.e2e`) are excluded from the client-registered `apps/web` project. Those files and their shared `support.ts` are included file-by-file in `tsconfig.host.json` — one program cannot hold both sides of the cordis `Context` merges.
|
||||
|
||||
### Modes and fixtures
|
||||
|
||||
`DSH_SNAPSHOT` selects replay (default, keyless), record (with key), or refresh (keyless) as inline spec branches — the TUI shape, not a suite factory: at two scenarios the acp-snapshot factory machinery has no owner, and the genuinely shared parts are already exported (`scrubRequestHeaders`, `parseSessionLog`, `installLlmReplay`). Each spec splits into drive steps (type, send, `whenTurnSettled` — run in all modes, never waiting on model-content selectors, so record cannot hang on a live model answering differently) and assertion steps (replay/refresh only). Record = drive live through the real composer + harvest the in-memory `session.header`/`session.events` (the TUI `rawSessionLog` shape — no file decompression) + `scrubRequestHeaders` + `{{sessionId}}`/`{{cwd}}`/`{{rpcId}}` tokenization; a follow-up keyless refresh regenerates `ui.expected.md`. Both scenarios' fixtures were recorded against this assembly through this flow. A drift guard ties each spec's drive prompt to the fixture's recorded `user/message`. A fixture-inventory guard holds each scenario directory closed (exact file set, every JSONL a scrub fixed-point with no run-local `rpcId`). Web fixtures scrub headers everywhere and pin no header class, following the TUI precedent over the strict [pinned-header](2026-07-06-pin-request-header-content-in-one-scenario.md) reading — see Deferred.
|
||||
|
||||
### Scenarios
|
||||
|
||||
1. **`replay-round-trip`** — new session, prompt through the real composer, replay streams reasoning + a `bash` tool call that really executes in the temp workspace + final text (paced 15ms). Asserts settled markdown, the aria golden, and inline world state (the bash call's durable result is exactly `WEB_E2E_OK\n`, completed `turn/end`, >10 chunk events).
|
||||
2. **`seeded-history`** — a recorded session seeded cold; the sidebar lists it (group row → session row, collapsed by default), opening renders tool cards and text purely from the log through the implicit cold-resume attach inside `session.history` — zero model calls in replay, so no binding constraints; record mode drives the same turn live (real `read` tool against seeded workspace files) to produce the seed.
|
||||
|
||||
### CI stance
|
||||
|
||||
The lane ships gate-exempt inside `pnpm run test:web`, exactly as that config's header records. Adding chromium to CI would reverse the "no browser infrastructure in CI" premise in the [GUI testing note](../process/2026-07-20-gui-testing-system.md) and therefore requires its own Agent Note cross-linked from there, staged as a non-required job first with measured promotion criteria (consecutive green runs, wall time, zero-retry flake budget, runner browser-cache strategy). `TODO(ci-browser)` marks the seam. Scenarios are POSIX-oriented (the lane is not in the Windows matrix).
|
||||
|
||||
## Prior art
|
||||
|
||||
Surveyed AI-chat/agent web UIs and mocking layers (LibreChat, vercel/ai-chatbot + AI SDK, lobe-chat, open-webui, OpenHands, Chainlit, continue, cline, langfuse, gradio/streamlit; Playwright HAR/route, MSW, Polly/nock, WireMock, aimock). The dominant proven architecture for apps that own their backend is an in-process fake/replay model behind the real backend seam with everything downstream real (LibreChat's `LIBRECHAT_TEST_RUN_HOOK` fake model; ai-chatbot's `MockLanguageModelV3` + `simulateReadableStream`; continue's scripted mock provider classes) — which is what `dsh-llm-replay` already is. Browser-level SSE interception cannot exercise incremental rendering (`route.fulfill` delivers the whole body at once; playwright#33564) and leaves the server SSE stack untested, so projects use it only for edge cases. Chunk pacing as a fixture parameter recurs everywhere (LibreChat 10ms default with slow profiles; ai-chatbot 500ms); real models in CI rot (open-webui's suite grew 120-second timeouts, was disabled, then deleted); sessions are seeded at the persistence layer with controlled timestamps (LibreChat inserts backdated Mongo documents; langfuse seeds its DB). No surveyed project replays a recorded agent-event log through the real backend for UI tests — the closest are provider-level recorded fixtures (aimock) and frontend-level socket history emission (OpenHands MSW) — so the session-log-as-fixture design goes one step beyond prior art along the axis this repo's model-visible ⟺ logged invariant makes natural.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
**Browser-network SSE interception (`page.route`).** Rejected: `route.fulfill` cannot stream, so incremental token rendering is unexercisable and the server-side SSE/backpressure/close path — where both confirmed P0s hid — goes untested.
|
||||
|
||||
**Mock HTTP provider at `DEEPSEEK_BASE_URL`.** Rejected as the lane's mechanism (kept for the one existing workspace-probe smoke): fixtures become hand-authored OpenAI SSE byte scripts, a second fixture format that drifts from the session-log format the rest of the repo records and replays; the adapter's real HTTP path is with-key e2e's job.
|
||||
|
||||
**Growing the `?fixture` client.** Rejected: tier separation — `FixtureApiClient` exists to test the client shell without a server; everything below the client API seam stays untested by construction.
|
||||
|
||||
**Placeholder `DEEPSEEK_API_KEY` + replay interception instead of disabling the adapter row.** Rejected despite zero composition change and two in-tree precedents: it satisfies `llm-deepseek`'s fail-loud key check with a lie and leaves a dead adapter mounted-but-intercepted; the disabled row (the ACP overlay's move) is honest keylessness and fails loud at the earliest resolvable point.
|
||||
|
||||
**A `packages/support/web-snapshot` package with a `defineWebSnapshotSuite` factory.** Rejected: chromium-driving source cannot honestly hold per-file 100% coverage on browserless coverage runners, and at two scenarios a factory generalizes from one consumer while the genuinely shared logic is already exported from gated packages. Re-entry trigger: a second web-shaped consumer or ≥6 scenarios with demonstrably drifting inline branches; the package boundary would then be drawn browser-free.
|
||||
|
||||
**A committed normalized-session-log golden as a second expected surface.** Rejected: the log surface is pinned by the ACP/headless/TUI suites through the same loop and persistence; here it would double refresh cost and re-test lower tiers. Inline world-state assertions on root-context events keep the world-verification duty.
|
||||
|
||||
**Spawning the `dsh web` bin with a `DSH_SNAPSHOT` replay branch.** Rejected: it needs a test-only replay branch plus environment plumbing in the shipped CLI. The in-process scaffold already loads the same `apps/cli/cordis.yml`; only argv, profile JSON, and `AppCLIEntry` glue remain outside it, and the keyless CLI smokes cover those paths.
|
||||
|
||||
**Changing the wire protocol for testability.** Rejected: the contract already has a first-class keyless isomorphic seam (`InProcessApiClient(toFetchHandler(api))`), the per-event unbatched SSE is exactly what makes replay observable in a browser, and testing a wire we no longer ship would invert the tier's purpose.
|
||||
|
||||
**Real-model browser tests as the keyless lane.** Rejected: nondeterministic by construction; the surveyed cautionary case (open-webui) grew unbounded timeouts and was deleted. The with-key W5 smoke stays as the live-model complement.
|
||||
|
||||
**A client `data-dsh-busy` settled signal.** Deferred: the multi-condition settled polls proved sufficient at two scenarios and the host-side `whenIdle` barrier does the heavy lifting. Re-entry trigger: the first settled-poll flake, or a scenario needing a state the DOM does not expose.
|
||||
|
||||
## Testing
|
||||
|
||||
The lane itself: `pnpm run test:web` runs both scenarios keylessly alongside the existing smoke pair; `DSH_SNAPSHOT=record pnpm exec vitest run --config vitest.web.config.ts apps/web/tests/<spec>` re-records a scenario's fixture against the live model; `DSH_SNAPSHOT=refresh` rewrites both aria goldens keylessly. `paceMs` validation, pacing floor, abort-during-pace, and both `assertConsumed` failure shapes are pinned in `packages/support/llm-replay/tests/llm-replay.spec.ts`.
|
||||
|
||||
## Deferred
|
||||
|
||||
- **Web header-class pin**: web fixtures tokenize `{{system}}`/`{{tools}}` everywhere and no scenario pins the web composition's prompt/tool schemas (`TODO(web-header-pin)` — the scaffold `recordFixture` JSDoc marks it). Following the TUI scrub-everywhere precedent; revisit when the web assembly's header diverges from the repl composition it mirrors.
|
||||
- **CI browser provisioning**: reversal of the no-browser-in-CI ruling, staged criteria above (`TODO(ci-browser)`).
|
||||
- **Follow-up-prompt-after-resume scenario**: the history/live stitch path over the real wire; add as its own scenario when that code changes or regresses.
|
||||
|
||||
## Consequences
|
||||
|
||||
The web surface gains its record-once/replay-forever tier: the real chromium → SSE → apiproxy → loop → tools → persistence chain runs keylessly in ~10-30s, deterministic across repeat runs, with fixtures owned and re-recordable by the lane itself. Costs accepted: every intentional conversation-UI change ends with a keyless `DSH_SNAPSHOT=refresh` (golden churn is reviewed diff, anchors keep semantic green); the aria format is Playwright-owned — the one committed snapshot format the repo does not control — so playwright version bumps must be deliberate bump-and-refresh commits (the dependency floats `^1.49.0` in `apps/web/package.json`; pin exactly if churn bites); replay's first-call-order binding constrains scenarios to one prompting session each, with the consumption assertion as the tripwire; `compact-basic` shares the session's replay cursor and stays inert only under the published 128k catalog window; and the lane guards regressions only where it runs (locally, `test:web`) until the CI reversal is separately decided.
|
||||
@@ -0,0 +1,90 @@
|
||||
# Agent Note: Web GUI 的无密钥浏览器 e2e 车道
|
||||
|
||||
Status: implemented
|
||||
|
||||
[English](2026-07-24-web-gui-browser-e2e-lane.md) | 中文
|
||||
|
||||
## 问题
|
||||
|
||||
Web GUI 以一条真实组装链交付——chromium 页面 → client 插件 bundle → HTTP 单次 RPC + 两条 SSE(Server-Sent Events)流 → `toFetchHandler`/apiproxy → host 端的 agent loop(智能体循环)、工具与 JSONL 持久化——却没有任何测试无密钥且确定性地检验这条链。[GUI 测试体系](../process/2026-07-20-gui-testing-system.md)覆盖第 1 层(Node 中的协议同构)、第 2 层(对象层状态机)与第 3 层冒烟测试,但无密钥冒烟驱动的是 `FixtureApiClient`——没有 host、没有 wire、没有 agent loop——而全链路冒烟需要 `DEEPSEEK_API_KEY` 和真实模型,因此不确定、在无密钥 CI 中自行跳过。[docs/testing.md](../../../../docs/testing.md) 的快照哲学——带密钥录制一次、永久无密钥回放、格式变动时刷新——已覆盖 ACP(Agent Client Protocol)、headless `stream-json` 与 TUI 三个文本记录(transcript)表面;web 表面是唯一没有这层保障的组装形态。而缺口恰恰是两起已实证 GUI P0 藏身之处:fixture(测试前置数据)客户端短路掉的 wire 承载链。
|
||||
|
||||
## 决策
|
||||
|
||||
`pnpm run test:web` 携带 `apps/web/tests/` 下的无密钥、确定性浏览器 e2e 车道:录制的会话日志 fixture 经 `@deepseek-ai/dsh-llm-replay` 对真实进程内 web 组合回放,断言规范化后的会话区 aria 预期输出加进程内世界状态。不新增包(package);产品侧增量只有 `dsh-llm-replay` 的两处增量接口(`paceMs`、`ReplayHandle`)。
|
||||
|
||||
### Scaffold:`apps/web/tests/scaffold.ts`
|
||||
|
||||
一个普通的共享 fixture 模块([测试政策认可的形态](../../../../docs/testing.md)),不是包:值得门禁把守的逻辑——回放推导、会话解析、日志脱敏、持久化——都在已受门禁的包 `dsh-llm-replay`、`dsh-acp-snapshot`、`dsh-session-persistence-jsonl` 中;剩下的只是启动接线和浏览器胶水,而驱动 chromium 的源码在无浏览器的覆盖率 runner 上无法诚实保持逐文件 100% 覆盖率。
|
||||
|
||||
`launchWebScaffold()` 通过 vendored Loader 的 include 机制,从交付的 `apps/cli/cordis.yml` 启动真实 web 组合——与 `AppCLIEntry` 为 `dsh web` 驱动的是同一棵树、同一套机制。差异全部经 include patch 覆盖在这棵树上,即 ACP `cordis.snapshot.yml` 模式的进程内表达:临时 `persistenceRoot`;禁用 `workspace-context`(录制的 fixture 不得嵌入本仓库的 AGENTS.md);禁用 `session-title-llm`(其发后不管的标题调用会与循环争抢会话的回放游标);webserver 行钉到端口 0 加已构建 dist;无密钥模式下禁用 `llm-deepseek`。patch 的 id 一旦不再匹配任何行,boot 扫描会大声失败而不是漂移。boot 在临时工作区 `chdir` 下运行,使 api-gateway 的 `process.cwd()` 会话默认值、工具 cwd 与 fixture 一致;`dsh web` bin 自身的胶水(argv、profile json、AppCLIEntry)仍由 `smoke-real.e2e.ts` 中的无密钥 CLI 冒烟把守。初始化回滚和正常关闭都会先对 Cordis 树执行 dispose(资源释放),再删除 scaffold 持有的两个临时根目录;每项清理都会独立尝试,并会报告清理失败而不掩盖初始化失败。
|
||||
|
||||
无密钥的模型替换 = 禁用适配器行的 patch 加 `installLlmReplay` 在停稳的根 ctx 上以提供方目录(providers-catalog)模式填充开放的 seam——绝不用 catch-all:适配器行被禁用后不存在任何适配器,catch-all 会让 `resolveModelInfo` 无路由可走,`compact-basic` 的步后压力检查将步步告警,而不是被可证明地闲置(发布的 128k `contextWindow` 使该路径对小 fixture 保持闲置)。选择直接安装而非插入回放插件行是刻意的:直接安装返回收尾消费检查所需的 `ReplayHandle`。没有 fixture 的场景让 seam 保持空置,任何离群的流式调用都会以 NO_ADAPTER 大声失败。
|
||||
|
||||
`seedSession()` 通过真实持久化 API 播种冷会话——一次性 `Context` 挂载 `SessionStore` + `SessionPersistenceJsonl` 指向 host 的根目录,`create()` + `append()`,一次 `utimes` 回拨保证侧栏顺序确定(`semantic-checkpoint.snapshot.ts` 先例)——绝不裸写文件,因此播种器对桶哈希、文件名编码、压缩一无所知,host 的 zstd 默认值也无需任何启动开关。种子在播种时即校验(可解析、以 `turn/end` 结尾——未闭合的最终轮次会被恢复(resume)的崩溃修复改写)。
|
||||
|
||||
### 确定性规则
|
||||
|
||||
回放模式下浏览器断言的屏障栈,按序:(1)host 侧 `await agent.whenIdle()` 加超时,以进程内 `turn/end` 为锚——空闲翻转发生在持久化落盘之后,一次等待同时覆盖轮次完成与持久性;(2)浏览器安定轮询(流式输出节点已卸载、最终文本可见)。录制模式下,日志采收在 `whenIdle()` 之后、scaffold 释放之前进行,此时运行中的会话仍然可用。单独监听进程内 `turn/end` 是错误屏障(它先于 SSE 帧到达浏览器、先于 fsync 触发);文件轮询被禁止(NFS 上慢,且被 `whenIdle` 取代);`networkidle` 被彻底禁止(SSE 流保持打开时它永不解析)。
|
||||
|
||||
不做单次瞬态 DOM 断言:从回放产出到 React 提交的每一跳都可能合并分片,采样 `[data-streaming]` 天然就是竞态。流式输出的增量性由持久化的 `assistant/chunk` 事件断言(模型可见 ⟺ 已记录,使日志成为权威证据)。`dsh-llm-replay` 的可选 `paceMs`(默认缺省 = 突发)只是让浏览器观察到真正增量 SSE 的真实感旋钮;正确性绝不依赖它,且节奏等待期间中止会即时取消。
|
||||
|
||||
每个场景都会因任何 pageerror 或客户端的连接丢失/间隙修复控制台警告而失败:否则重连机制加历史重同步会把一条死掉的 SSE 通路自愈掉,套件反而认证了坏 wire。Scaffold 的 `close()` 调用 `ReplayHandle.assertConsumed()` 收尾检查(每个已录脚本都被绑定、每个游标都耗尽),把静默的少放与错绑变成清晰诊断。车道不设 vitest 重试;每文件一个 chromium、每场景一个新 context、每场景一个 host;视口固定;交互选择器锚定 role、`data-*` 属性和可见文本,而 frame 与会话区采集则使用既有的 CSS 模块局部类名锚点。
|
||||
|
||||
### 预期输出
|
||||
|
||||
每场景一份提交的预期输出:会话区规范化 `ariaSnapshot()`(`ui.expected.md`)——uuid/cwd/工作区目录名/时长归一为稳定 token,在安定里程碑处轮询至两次相等再采集——外加几条 role/文本锚断言,让保语义的组件重写在预期输出可评审地变动时仍保持绿色锚点。aria 树是 client 规则「断言用户所见,绝不断言类名」的机械化。世界状态断言内联在根上下文的会话事件上(哪次工具调用产生了哪项已持久化的工具结果、`turn/end` 是否完成)而不是第二份提交的日志预期输出:持久化日志表面已由 ACP/headless/TUI 套件经同一循环和持久化钉住,在此重复钉住会违背分层纪律、翻倍刷新成本。`refresh` 是预期输出的唯一写入者——回放模式下预期输出缺失会连同修复命令一起报错,而不是静默自举。
|
||||
|
||||
类型检查平面切分是结构性的:启动 host 主干的三个文件(`scaffold`、`replay-round-trip.e2e` 和 `seeded-history.e2e`)被排除出注册在 client 侧的 `apps/web` 工程。这三个文件及其共享的 `support.ts` 逐文件纳入 `tsconfig.host.json`——一个程序不能同时持有 cordis `Context` 合并的两侧。
|
||||
|
||||
### 模式与 fixture
|
||||
|
||||
`DSH_SNAPSHOT` 以内联 spec 分支选择 replay(默认,无密钥)、record(带密钥)或 refresh(无密钥)——TUI 的形态,不是套件工厂:两个场景撑不起 acp-snapshot 工厂机制,且真正共享的部分已被导出(`scrubRequestHeaders`、`parseSessionLog`、`installLlmReplay`)。每个 spec 切分为驱动步骤(输入、发送、`whenTurnSettled`——所有模式都执行,绝不等待模型内容选择器,因此 record 不会因真实模型答法不同而挂起)与断言步骤(仅 replay/refresh)。Record = 经真实输入框实时驱动 + 采收内存中的 `session.header`/`session.events`(TUI 的 `rawSessionLog` 形态——无需文件解压)+ `scrubRequestHeaders` + `{{sessionId}}`/`{{cwd}}`/`{{rpcId}}` token 化;随后一次无密钥 refresh 重新生成 `ui.expected.md`。两个场景的 fixture 都经此流程对本组装录制。一条漂移防线把每个 spec 的驱动提示词与 fixture 录制的 `user/message` 绑定。fixture 清单防线保持每个场景目录封闭(精确文件集合,每个 JSONL 都是脱敏不动点,不含当次运行的 `rpcId`)。Web fixture 全部脱敏请求头且不钉任何头类别,沿用 TUI 先例而非[钉住请求头](2026-07-06-pin-request-header-content-in-one-scenario.md)的严格读法——见「暂缓」。
|
||||
|
||||
### 场景
|
||||
|
||||
1. **`replay-round-trip`**——新会话,经真实输入框发送提示词,回放流式输出推理(reasoning)+ 一次在临时工作区真实执行的 `bash` 工具调用 + 最终文本(15ms 节奏)。断言安定后的 markdown、aria 预期输出与内联世界状态(这次 bash 调用的已持久化工具结果严格等于 `WEB_E2E_OK\n`、完成的 `turn/end`、>10 个分片事件)。
|
||||
2. **`seeded-history`**——冷播种一份已录会话;侧栏列出它(分组行 → 会话行,默认折叠),打开后纯凭日志经 `session.history` 内的隐式冷恢复挂载渲染工具卡片与文本——replay 下零模型调用,因此没有任何绑定约束;record 模式实时驱动同一轮(真实 `read` 工具读取播种的工作区文件)来产出种子。
|
||||
|
||||
### CI 立场
|
||||
|
||||
车道随 `pnpm run test:web` 交付、豁免门禁,与该配置头部注释所记一致。往 CI 加 chromium 会推翻 [GUI 测试笔记](../process/2026-07-20-gui-testing-system.md)中「CI 无浏览器基础设施」的前提,因此需要自己的 Agent Note 并从那里交叉链接,分阶段推进:先作为非必需任务,再以量化标准晋升(连续绿色运行次数、耗时、零重试的抖动预算、runner 浏览器缓存策略)。`TODO(ci-browser)` 标记该接缝。场景目前面向 POSIX(车道不在 Windows 矩阵中)。
|
||||
|
||||
## 业界先例
|
||||
|
||||
调研了 AI 聊天/agent web UI 与 mock 层(LibreChat、vercel/ai-chatbot + AI SDK、lobe-chat、open-webui、OpenHands、Chainlit、continue、cline、langfuse、gradio/streamlit;Playwright HAR/route、MSW、Polly/nock、WireMock、aimock)。自有后端的应用的主流成熟架构是:真实后端 seam 后放一个进程内伪造/回放模型,下游全部真实(LibreChat 的 `LIBRECHAT_TEST_RUN_HOOK` 伪模型;ai-chatbot 的 `MockLanguageModelV3` + `simulateReadableStream`;continue 的脚本化 mock 提供方类)——这正是 `dsh-llm-replay` 已然所是。浏览器层 SSE 拦截无法检验增量渲染(`route.fulfill` 一次性交付整个响应体;playwright#33564),且服务端 SSE 栈完全失测,因此各项目只把它用于边缘用例。分片节奏作为 fixture 参数反复出现(LibreChat 默认 10ms 附慢速档;ai-chatbot 500ms);CI 里的真实模型会腐烂(open-webui 的套件长出 120 秒超时,先被禁用后被删除);会话在持久化层以受控时间戳播种(LibreChat 直插回拨时间的 Mongo 文档;langfuse 播种其数据库)。没有任何被调研项目为 UI 测试把录制的 agent 事件日志经真实后端回放——最接近的是提供方层录制 fixture(aimock)与前端层 socket 历史发射(OpenHands MSW)——因此会话日志即 fixture 的设计沿着本仓库「模型可见 ⟺ 已记录」不变式所指的方向比业界先例多走了一步。
|
||||
|
||||
## 曾考虑的替代方案
|
||||
|
||||
**浏览器网络层 SSE 拦截(`page.route`)。** 已否决:`route.fulfill` 无法流式输出,增量 token 渲染无从检验,且服务端 SSE/背压/关闭路径——两起已实证 P0 的藏身处——完全失测。
|
||||
|
||||
**`DEEPSEEK_BASE_URL` 处的 mock HTTP 提供方。** 作为本车道机制已否决(仅保留给既有的工作区探针冒烟):fixture 会变成手写的 OpenAI SSE 字节脚本,一种与仓库其余部分录制回放的会话日志格式渐行渐远的第二 fixture 格式;适配器的真实 HTTP 路径归带密钥 e2e 管。
|
||||
|
||||
**扩展 `?fixture` 客户端。** 已否决:分层纪律——`FixtureApiClient` 的存在意义就是脱离服务器测试客户端 shell;client API seam 以下按构造即失测。
|
||||
|
||||
**用占位 `DEEPSEEK_API_KEY` + 回放拦截替代禁用适配器行。** 尽管零组合改动且树内有两处先例仍被否决:它用谎言满足 `llm-deepseek` 的快速失败密钥检查,还留下一个挂载却被拦截的死适配器;禁用行(ACP overlay 的同款做法)是诚实的无密钥,并在最早可解析点快速失败。
|
||||
|
||||
**`packages/support/web-snapshot` 包 + `defineWebSnapshotSuite` 工厂。** 已否决:驱动 chromium 的源码在无浏览器的覆盖率 runner 上无法诚实保持逐文件 100%,且两个场景就上工厂是从单一消费方过度泛化,真正共享的逻辑已从受门禁的包中导出。重启条件:出现第二个 web 形态消费方,或 ≥6 个场景的内联分支被证实各自漂移;届时包边界将画在无浏览器一侧。
|
||||
|
||||
**第二份提交的规范化会话日志预期输出。** 已否决:日志表面已由 ACP/headless/TUI 套件经同一循环与持久化钉住;在此只会翻倍刷新成本并重复测试下层。内联在根上下文事件上的世界状态断言保住了验证世界的义务。
|
||||
|
||||
**以 `DSH_SNAPSHOT` 回放分支拉起 `dsh web` bin。** 已否决:它需要在交付的 CLI 中增加测试专用回放分支和环境变量管道。进程内 scaffold 已加载同一份 `apps/cli/cordis.yml`;只剩 argv、profile JSON 和 `AppCLIEntry` 胶水不在其覆盖范围内,而这些路径已由无密钥 CLI 冒烟覆盖。
|
||||
|
||||
**为可测试性改 wire 协议。** 已否决:契约已有第一等的无密钥同构 seam(`InProcessApiClient(toFetchHandler(api))`),逐事件不合批的 SSE 恰是回放在浏览器中可观测的原因,测试一条不再交付的 wire 会颠倒该层的存在意义。
|
||||
|
||||
**以真实模型浏览器测试充当无密钥车道。** 已否决:按构造即不确定;被调研的前车之鉴(open-webui)长出无界超时后被删除。带密钥的 W5 冒烟仍是真实模型侧的补充。
|
||||
|
||||
**客户端 `data-dsh-busy` 安定信号。** 暂缓:两个场景下多条件安定轮询已经够用,host 侧 `whenIdle` 屏障承担了重活。重启条件:第一次安定轮询抖动,或某场景需要等待 DOM 不暴露的状态。
|
||||
|
||||
## Testing
|
||||
|
||||
车道自身:`pnpm run test:web` 与既有冒烟对一起无密钥运行两个场景;`DSH_SNAPSHOT=record pnpm exec vitest run --config vitest.web.config.ts apps/web/tests/<spec>` 对真实模型重录某场景的 fixture;`DSH_SNAPSHOT=refresh` 无密钥重写两份 aria 预期输出。`paceMs` 校验、节奏下限、节奏中中止、`assertConsumed` 的两种失败形态钉在 `packages/support/llm-replay/tests/llm-replay.spec.ts`。
|
||||
|
||||
## 暂缓
|
||||
|
||||
- **Web 头类别钉住**:web fixture 处处 token 化 `{{system}}`/`{{tools}}`,没有场景钉住 web 组合的提示词/工具 schema(`TODO(web-header-pin)`——scaffold 的 `recordFixture` JSDoc 有标记)。沿用 TUI 处处脱敏先例;当 web 组装的请求头与其镜像的 repl 组合进一步分叉时重审。
|
||||
- **CI 浏览器供给**:推翻 CI 无浏览器裁定,分阶段标准见上(`TODO(ci-browser)`)。
|
||||
- **恢复后追问场景**:真实 wire 上的历史/实时缝合路径;当该代码变更或回归时作为独立场景补充。
|
||||
|
||||
## 后果
|
||||
|
||||
Web 表面获得了录制一次/永久回放的层级:真实 chromium → SSE → apiproxy → 循环 → 工具 → 持久化的链路以约 10-30 秒无密钥运行,重复运行结果确定,fixture 由车道自身持有并可重录。接受的成本:每次有意的会话 UI 变更都以一次无密钥 `DSH_SNAPSHOT=refresh` 收尾(预期输出变动是受评审的 diff,锚断言保住语义绿色);aria 格式归 Playwright 所有——仓库唯一不受自己控制的提交快照格式——因此 playwright 版本升级必须是刻意的升级加刷新提交(依赖在 `apps/web/package.json` 中浮动为 `^1.49.0`;若变动伤人则改为精确锁定);回放的首次调用顺序绑定把每个场景限制为至多一个发起提示的会话,消费断言是绊线;`compact-basic` 与会话共享回放游标,仅在发布的 128k 目录窗口下保持闲置;在 CI 反转被单独决策之前,车道只在其运行之处(本地,`test:web`)把守回归。
|
||||
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
2026-07-25-scriptable-llm-wire-fault-server.md: 0795f71f0eab1a107740aaa8cba6fa04b1fbd306
|
||||
2026-07-25-scriptable-llm-wire-fault-server.zh.md: a0e3c98d729adcc74e6d98933ab2beb538f71cb3
|
||||
@@ -0,0 +1,41 @@
|
||||
# Agent Note: Scriptable LLM wire fault server
|
||||
|
||||
Status: implemented
|
||||
|
||||
English | [中文](2026-07-25-scriptable-llm-wire-fault-server.zh.md)
|
||||
|
||||
## Problem
|
||||
|
||||
Adapter unit tests use local HTTP servers to classify individual provider failures, while retry tests use an in-process scripted `LlmAdapter` to prove closed-step recovery. Neither boundary provides a reusable server for running the shipping HTTP adapter, agent loop, and retry policy together, and neither lets a developer point an existing app at deterministic transport faults by changing only its base URL and API key.
|
||||
|
||||
Connection refusal, a reset before the first event, clean EOF without `[DONE]`, a valid content-less completion, and a reset after partial output have different adapter and recovery outcomes. Treating them as one generic mock failure hides whether the provider boundary preserved the distinction and whether failed chunks remained outside committed model history.
|
||||
|
||||
## Decision
|
||||
|
||||
`@deepseek-ai/dsh-llm-mock-server` is a support package with an importable Node HTTP server and a standalone CLI. It accepts OpenAI-compatible root and `/v1` chat-completions paths, validates an optional bearer token, captures requests, and consumes one explicit behavior per accepted request. Script exhaustion fails loud; repetition requires `repeatLast`.
|
||||
|
||||
Request behaviors cover socket reset, post-header disconnect, partial disconnect, stall, valid empty completion, clean truncated streams, malformed payloads, representative HTTP failures, complete text/reasoning/tool-call responses, slow streaming, and max-token completion. A true `connection_refused` is a CLI listener-lifecycle phase because a bound request handler cannot refuse its own TCP connection.
|
||||
|
||||
The `random` script entry performs a new weighted selection for every request. The server exposes and logs its unsigned 32-bit seed, accepts caller-supplied relative weights, and ships a success-heavy stress profile that mixes transport, protocol, provider, timeout, and semantic-empty outcomes. The profile is configurable test pressure rather than an estimate of production incident frequency; `connection_refused` remains outside the request-level pool.
|
||||
|
||||
The server reports wire facts only and does not classify retryability. Real-composition tests route it through `dsh-llm-deepseek`, `dsh-agent-loop`, and `dsh-llm-retry`: connection refusal, hard disconnect, partial reset, idle timeout, and a valid content-less completion recover under the existing default policy; clean partial EOF remains `STREAM_CLOSED` and is not retried by default. The package does not change those policies.
|
||||
|
||||
## Verification
|
||||
|
||||
Package tests exercise every request behavior, split UTF-8 request decoding, HTTP validation without script consumption, script exhaustion/repetition, stalled-connection teardown, CLI parsing and delay bounds, IPv6 base URLs, random seed reproducibility, weight validation, single-result telemetry, lifecycle cleanup, and the invariant companion under the per-file coverage gate. The retry integration suite proves exact request counts, numbered retry steps, request-body identity, failed partial-chunk isolation, semantic-empty recovery, clean-EOF classification, timeout recovery, true refused-connection recovery after delayed listener startup, and bounded exhaustion through the real HTTP/SSE adapter.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
**Implement the server in Python** — rejected because Node's standard HTTP and socket APIs expose every required fault, while TypeScript keeps the server, CLI parser, tests, package build, lint, and coverage inside the repository's existing toolchain. A second runtime would add environment and subprocess dependencies without increasing wire isolation.
|
||||
|
||||
**Keep separate inline mock servers in adapter tests** — rejected because those fixtures cannot be launched by an existing app and would duplicate behavior sequencing, randomization, telemetry, and connection cleanup across suites. A support package gives tests a shared implementation without promoting it to product API.
|
||||
|
||||
**Use only an in-process `LlmAdapter` mock** — rejected because it bypasses fetch, HTTP status/header parsing, SSE framing, socket termination, and the adapter idle watchdog: the exact boundaries this test infrastructure exists to exercise.
|
||||
|
||||
**Change retry defaults with the server** — rejected because the server reveals existing semantics rather than deciding policy. Extending recovery to `STREAM_CLOSED` requires a separate decision with its own cost, latency, and duplicate-generation trade-offs.
|
||||
|
||||
## Consequences
|
||||
|
||||
Developers can reproduce fault sequences by changing only provider URL/key configuration, and automated tests can keep socket-level failures deterministic through explicit scripts and seeds. The same wire fixture now exposes gaps between hard resets, clean truncation, and recovered empty completions without splicing attempts or modifying model history.
|
||||
|
||||
The server adds a support package, executable build entry, and behavior vocabulary that must remain compatible with both direct tests and CLI examples. Arrival-ordered scripts are intentionally shared across clients, random defaults are stress weights rather than operational truth, and exact connection refusal requires coordinating the client attempt with the pre-listen interval.
|
||||
@@ -0,0 +1,41 @@
|
||||
# Agent Note: 可脚本控制的 LLM(大语言模型)协议层故障服务器
|
||||
|
||||
Status: implemented
|
||||
|
||||
[English](2026-07-25-scriptable-llm-wire-fault-server.md) | 中文
|
||||
|
||||
## 问题
|
||||
|
||||
适配器单元测试使用本地 HTTP 服务器对各类提供方故障逐一分类,重试测试则使用进程内的脚本化 `LlmAdapter` 证明已关闭步骤的恢复能力。这两个边界都无法提供可复用的服务器,以便同时运行交付版本的 HTTP 适配器、agent loop(智能体循环)和重试策略;开发者也无法仅修改现有应用的 base URL 与 API key,就让应用连接到确定性的传输故障。
|
||||
|
||||
连接遭拒、首个事件前连接被重置、未收到 `[DONE]` 即正常 EOF、合法但无内容的完成,以及输出部分内容后连接被重置,会产生不同的适配器与恢复结果。把它们统一视为普通 mock 故障,会掩盖提供方边界是否保留了这些区别,以及失败请求的分片是否确实没有进入已提交的模型历史。
|
||||
|
||||
## 决策
|
||||
|
||||
`@deepseek-ai/dsh-llm-mock-server` 是一个支持包(package),提供可导入的 Node HTTP 服务器和独立 CLI(命令行界面)。它接受兼容 OpenAI 的根路径和 `/v1` chat-completions 路径,校验可选的 bearer token,捕获请求,并对每个已接受请求消耗一个显式行为。脚本耗尽时快速失败;只有设置 `repeatLast` 才会重复最后一个行为。
|
||||
|
||||
请求行为覆盖 socket 重置、发送 header 后断开、发送部分内容后断开、停滞、合法空完成、正常关闭但被截断的流、畸形 payload、典型 HTTP 故障、完整的文本/推理/工具调用响应、慢速流式输出以及达到 token 上限的完成。真正的 `connection_refused` 由 CLI 的监听器生命周期阶段实现,因为已经绑定端口的请求处理器无法拒绝自身的 TCP 连接。
|
||||
|
||||
脚本项 `random` 会为每个请求重新执行一次加权选择。服务器公开并记录其无符号 32 位 seed,允许调用方提供相对权重,并内置一套偏重成功结果的压力测试配置,将传输、协议、提供方、超时和语义空结果混合在一起。该配置用于提供可调的测试压力,并非对生产事故发生频率的估算;`connection_refused` 仍不进入请求级随机池。
|
||||
|
||||
服务器只报告协议层事实,不判断是否可重试。真实组合测试让请求依次经过 `dsh-llm-deepseek`、`dsh-agent-loop` 和 `dsh-llm-retry`:在现有默认策略下,连接遭拒、硬断开、部分输出后重置、空闲超时以及合法的无内容完成均可恢复;正常关闭的部分输出 EOF 仍归类为 `STREAM_CLOSED`,默认不重试。该包不会改变这些策略。
|
||||
|
||||
## 验证
|
||||
|
||||
包测试覆盖所有请求行为、跨分片 UTF-8 请求解码、不消耗脚本的 HTTP 校验、脚本耗尽与重复、停滞连接清理、CLI 解析及延迟边界、IPv6 base URL、随机 seed 可复现性、权重校验、单结果遥测、生命周期清理,以及逐文件覆盖率门禁下的配套不变式插件。重试集成套件通过真实 HTTP/SSE(Server-Sent Events)适配器,验证准确的请求次数、带编号的重试步骤、请求体完全一致、失败的部分分片不会泄漏、语义空结果恢复、正常 EOF 分类、超时恢复、监听器延迟启动后从真实连接遭拒中恢复,以及有界重试耗尽。
|
||||
|
||||
## 曾考虑的替代方案
|
||||
|
||||
**使用 Python 实现服务器**:不予采纳。Node 的标准 HTTP 与 socket API 足以暴露所有所需故障,而 TypeScript 可以让服务器、CLI 解析器、测试、包构建、lint 和覆盖率全部留在仓库现有工具链中。引入第二套运行时会增加环境与子进程依赖,却不能增强协议隔离。
|
||||
|
||||
**在适配器测试中继续使用各自独立的内联 mock 服务器**:不予采纳。这些 fixture(测试前置数据)无法作为独立服务器启动并供现有应用连接,还会让不同测试套件重复实现行为编排、随机化、遥测和连接清理。支持包让测试共享同一套实现,又不会将其提升为产品 API。
|
||||
|
||||
**仅使用进程内的 `LlmAdapter` mock**:不予采纳。它会绕过 fetch、HTTP 状态与 header 解析、SSE 分帧、socket 终止以及适配器的空闲看门狗,而这正是这套测试基础设施要覆盖的边界。
|
||||
|
||||
**随服务器一起修改默认重试策略**:不予采纳。服务器用于揭示既有语义,而非决定策略。是否将恢复能力扩展到 `STREAM_CLOSED`,需要单独决策,并权衡成本、延迟和重复生成风险。
|
||||
|
||||
## 后果
|
||||
|
||||
开发者只需修改提供方 URL/key 配置即可复现故障序列;自动化测试则可通过显式脚本和 seed,让 socket 层故障保持确定性。同一套协议 fixture 现在可以暴露硬重置、正常截断与恢复后的空完成之间的差异,而不会拼接多次尝试的内容或修改模型历史。
|
||||
|
||||
服务器新增了一个支持包、可执行构建入口和行为词汇,三者必须同时兼容直接测试与 CLI 示例。按请求到达顺序执行的脚本有意由所有客户端共享;随机模式的默认值代表压力测试权重,而非实际运行规律;精确模拟连接遭拒时,需要让客户端尝试与监听开始前的时间区间协调一致。
|
||||
Reference in New Issue
Block a user