mirror of
https://github.com/deepseek-ai/deepseek-harness
synced 2026-08-15 21:04:50 +00:00
Merge branch 'master' into fix-update-builderror
This commit is contained in:
imccyu
GitHub
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
2026-07-20-gui-testing-system.md: e42dafcdf37e48475e7d420eaad9600e6c20891c
|
||||
2026-07-20-gui-testing-system.zh.md: e4ef6246e59e0ad6c0a3070a38c546f964e347fa
|
||||
2026-07-20-gui-testing-system.md: 546f65f065c0c2266773acc3c28b2833a094ba9b
|
||||
2026-07-20-gui-testing-system.zh.md: 6601ae0a1c2bd1671af6f02961fbda81d30ab971
|
||||
|
||||
@@ -20,7 +20,7 @@ Cut along the architecture's natural test seams into three tiers, bottom-up:
|
||||
|---|---|---|---|
|
||||
| 1 Protocol isomorphism | `AbstractApiClient` + `toFetchHandler` (bidirectional data / rpcId / zod types / SSE streams / batching / timeouts) | **The full chain at the isomorphic point**: `InProcessApiClient(toFetchHandler(脚本化 impl))` skips the network but genuinely runs the wire serialization — zero browser, pure node env | `packages/host/apiproxy/tests/client-handler.spec.ts` |
|
||||
| 2 Object-layer orchestration | `Session`/`SessionManager`/`ConnectionController` (state machines and timing: stitching / dedup / paging / optimistic draft clearing / pendingBuffers / reconnect / backoff) | **The "event sequence in → snapshot out" golden path**: programmable fakes + deferreds controlling timing + fake timers controlling backoff | `packages/client/{runtime,connection}/tests/` |
|
||||
| 3 Assembled presentation | Built artifacts × the real client loader and plugin composition | App-owned semantic snapshots boot all eight built client plugins under jsdom for deterministic cross-plugin state changes; bare Playwright smoke separately proves the real browser/carrier boundary, with real-host cases self-skipping without a key | `apps/web/tests/*.snapshot.ts`, `apps/web/tests/smoke-{fixture,real}.e2e.ts` |
|
||||
| 3 Assembled presentation | Built artifacts × the real client loader and plugin composition | App-owned semantic snapshots boot all eight built client plugins under jsdom for deterministic cross-plugin state changes; bare Playwright smoke separately proves the real browser/carrier boundary, with real-host cases self-skipping without a key; the keyless browser e2e lane disables the shipped model-adapter row and replays recorded session fixtures through `dsh-llm-replay` in the real in-process web assembly against conversation aria goldens ([web e2e lane](../testing/2026-07-24-web-gui-browser-e2e-lane.md)) | `apps/web/tests/*.snapshot.ts`, `apps/web/tests/smoke-{fixture,real}.e2e.ts`, `apps/web/tests/{replay-round-trip,seeded-history}.e2e.ts` |
|
||||
|
||||
Inter-tier discipline: **each tier tests its own layer, upper tiers never re-test lower ones** — an app semantic snapshot pins only user-visible projection across the assembled plugin boundary, while Playwright smoke proves browser and carrier liveness; wire semantics belong to tier 1 and data semantics to tier 2. Pure-function layers (lineage/partial/notifier/fold-adapter) are tested directly with zero fakes in the same package's tests/ alongside tier 2.
|
||||
|
||||
@@ -33,15 +33,15 @@ Inter-tier discipline: **each tier tests its own layer, upper tiers never re-tes
|
||||
|---|---|---|---|
|
||||
| Baseline | `pnpm run test:gui` | Tier 1+2 vitest (`packages/client packages/host`), seconds-fast, no browser, no server | Casually, after touching any GUI source |
|
||||
| Semantic snapshot | `DSH_EXAMPLE_MODE=lib pnpm run test:snapshot` | Keyless assembled-application semantics plus the repo's transport-specific expected outputs | After a human-visible GUI change; before delivery |
|
||||
| Browser end-to-end | `pnpm run test:web` | Rebuilds the front-end dist first, then runs the tier-3 two-level smoke (fixture level + real-host level self-skip) | After touching the build surface/boot/carriage; before delivery |
|
||||
| Browser end-to-end | `pnpm run test:web` | Rebuilds the front-end dist first, then runs the tier-3 browser set: the two-level smoke (fixture level + real-host level self-skip) plus the keyless replayed e2e scenarios (`DSH_SNAPSHOT=record`/`refresh` re-record fixtures / rewrite goldens) | After touching the build surface/boot/carriage; before delivery |
|
||||
| Gate | `pnpm run test:coverage` | The repo-wide gate (host and client GUI packages included, except annotated browser-grade exclusions) | The PR window |
|
||||
|
||||
**Division of labor between the browser scripts and vitest**: Playwright owns browser/carrier black-box regression and long sequential user journeys; ordinary vitest owns data-layer semantics such as reference stability, timing, and wire shapes; snapshot vitest owns stable app-level semantic output through the built composition. These lanes complement each other rather than duplicating assertions.
|
||||
|
||||
## Anti-regression discipline
|
||||
|
||||
- **Every bug fix pins an assertion**: a browser-visible bug is pinned into the regression section of its owning verify script (one pin = one report line); a data-layer bug is pinned into the matching spec (precedent: the res-close misjudgment pinned in the webserver bridge suite — pure Node, reproduces in seconds, no longer needs the 12s browser sentinel as the only defense).
|
||||
- **All-green on fixture is not done, the real host must pass too**: what the fixture short-circuits is exactly the wire carriage chain (node:http bridge close semantics, real network timing); both empirically confirmed bugs hid there. Changes touching connection/bridge/handler/SSE must run `verify-session-real`.
|
||||
- **Every bug fix pins an assertion**: a browser-visible bug is pinned into its owning browser spec (smoke or e2e scenario); a data-layer bug is pinned into the matching spec (precedent: the res-close misjudgment pinned in the webserver bridge suite — pure Node, reproduces in seconds, no longer needs the 12s browser sentinel as the only defense).
|
||||
- **All-green on fixture is not done, the real wire must pass too**: what the fixture short-circuits is exactly the wire carriage chain (node:http bridge close semantics, real network timing); both empirically confirmed bugs hid there. Changes touching connection/bridge/handler/SSE must run the browser lane (`pnpm run test:web`) — its keyless e2e scenarios drive the real HTTP/SSE carriage, and the with-key real-host smoke remains the live-model complement.
|
||||
- The code-on-disk-is-the-answer reconciliation workflow: when a behavior change lands and turns existing cases red, reconcile on the spot (fix the test or fix the code, with the RFC/contract as arbiter); no red left hanging.
|
||||
|
||||
## Consequences
|
||||
|
||||
@@ -20,7 +20,7 @@ GUI 栈需要考虑多种应用形态，同应用形态内的不同运行环境
|
||||
|---|---|---|---|
|
||||
| 1 协议同构层 | `AbstractApiClient` + `toFetchHandler`（双向数据/rpcId/ZOD类型/SSE 流/合批/超时） | **同构点全链**：`InProcessApiClient(toFetchHandler(脚本化 impl))` 不过网络但真跑 wire 序列化——零浏览器、纯 node env | `packages/host/apiproxy/tests/client-handler.spec.ts` |
|
||||
| 2 对象层编排 | `Session`/`SessionManager`/`ConnectionController`（状态机与时序：缝合/去重/翻页/乐观清稿/pendingBuffers/重连/退避） | **「事件序列进→快照出」黄金路径**：可编程假体 + deferred 控时序 + fake timers 控退避 | `packages/client/{runtime,connection}/tests/` |
|
||||
| 3 组装呈现层 | 构建产物 × 真实 client loader 与插件组合 | 归应用所有的语义快照会在 jsdom 下启动全部 8 个已构建的 client 插件，以固定确定性的跨插件状态变化；独立使用 Playwright 裸库的冒烟测试负责验证真实浏览器/承载层边界，真 host 用例在无密钥时自行跳过 | `apps/web/tests/*.snapshot.ts`、`apps/web/tests/smoke-{fixture,real}.e2e.ts` |
|
||||
| 3 组装呈现层 | 构建产物 × 真实 client loader 与插件组合 | 归应用所有的语义快照会在 jsdom 下启动全部 8 个已构建的 client 插件，以固定确定性的跨插件状态变化；独立使用 Playwright 裸库的冒烟测试负责验证真实浏览器/承载层边界，真 host 用例在无密钥时自行跳过；无密钥浏览器 e2e 车道会禁用交付配置中的模型适配器行，并通过 `dsh-llm-replay` 在真实进程内 web 组装中回放录制的会话 fixture，与会话区 aria 期望输出比对（[web e2e 车道](../testing/2026-07-24-web-gui-browser-e2e-lane.md)） | `apps/web/tests/*.snapshot.ts`、`apps/web/tests/smoke-{fixture,real}.e2e.ts`、`apps/web/tests/{replay-round-trip,seeded-history}.e2e.ts` |
|
||||
|
||||
层间纪律：**下层各测各的，上层不重测下层**：应用语义快照只固定组装后插件边界上的用户可见投影，Playwright 冒烟测试负责验证浏览器与承载层是否存活；wire 语义归 1 层，数据语义归 2 层。纯函数层（lineage/partial/notifier/fold-adapter）随 2 层同包 tests/ 零假体直测。
|
||||
|
||||
@@ -33,15 +33,15 @@ GUI 栈需要考虑多种应用形态，同应用形态内的不同运行环境
|
||||
|---|---|---|---|
|
||||
| 基础 | `pnpm run test:gui` | 1+2 层 vitest（`packages/client packages/host`），秒级、无浏览器无 server | 改 GUI 任意源码后随手跑 |
|
||||
| 语义快照 | `DSH_EXAMPLE_MODE=lib pnpm run test:snapshot` | 无需密钥的组装应用语义，以及仓库按传输形态划分的预期输出 | 用户可见的 GUI 变更后；交付前 |
|
||||
| 浏览器端到端 | `pnpm run test:web` | 先重建前端 dist，再跑 3 层双级 smoke（fixture 级 + 真 host 级 self-skip） | 改构建面/boot/承载后；交付前 |
|
||||
| 浏览器端到端 | `pnpm run test:web` | 先重建前端 dist，再跑 3 层浏览器全集：双级 smoke（fixture 级 + 真 host 级 self-skip）加上无密钥回放 e2e 场景（`DSH_SNAPSHOT=record`/`refresh` 重录 fixture / 重写期望输出） | 改构建面/boot/承载后；交付前 |
|
||||
| 门禁 | `pnpm run test:coverage` | 全仓 gate（host 与 client GUI 包均纳入，仅排除带注释的浏览器级例外） | PR 窗口 |
|
||||
|
||||
**浏览器脚本与 vitest 的分工**：Playwright 负责浏览器/承载层黑盒回归和较长的连续用户操作流程；普通 vitest 负责引用稳定性、时序和 wire 结构等数据层语义；快照 vitest 通过构建后的组合负责稳定的应用层语义输出。这些车道彼此互补，而不重复断言。
|
||||
|
||||
## 防回归纪律
|
||||
|
||||
- **修一个 bug 钉一条断言**：浏览器可见的 bug 钉进所属 verify 脚本的回归节（一钉一行 report）；数据层 bug 钉进对应 spec（先例：res-close 误判钉在 webserver 桥 suite——纯 Node 秒级复现，不再需要 12s 浏览器哨兵作唯一防线）。
|
||||
- **fixture 全绿不算完，真 host 也要过**：fixture 短路的恰是 wire 承载链（node:http 桥 close 语义、真网络时序），两次实证 bug 都藏在那里。改动触及连接/桥/handler/SSE 的，`verify-session-real` 必跑。
|
||||
- **修一个 bug 钉一条断言**：浏览器可见的 bug 钉进所属浏览器 spec（smoke 或 e2e 场景）；数据层 bug 钉进对应 spec（先例：res-close 误判钉在 webserver 桥 suite——纯 Node 秒级复现，不再需要 12s 浏览器哨兵作唯一防线）。
|
||||
- **fixture 全绿不算完，真 wire 也要过**：fixture 短路的恰是 wire 承载链（node:http 桥 close 语义、真网络时序），两次实证 bug 都藏在那里。改动触及连接/桥/handler/SSE 的，浏览器车道（`pnpm run test:web`）必跑——其无密钥 e2e 场景驱动真实 HTTP/SSE 承载，带密钥的真 host smoke 仍是真模型侧的补充。
|
||||
- 落盘代码即答案的对表工作流：行为改动落盘打红既有用例时，当场对表校准（改测试还是改代码以 RFC/契约为裁），不留悬红。
|
||||
|
||||
## Consequences
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
2026-07-24-web-gui-browser-e2e-lane.md: fb870c4bb2c85d8be7ac11f9f29f05bf24f446a4
|
||||
2026-07-24-web-gui-browser-e2e-lane.zh.md: c43e526d27fd4d8cf4f774e8a03480930682041d
|
||||
@@ -0,0 +1,90 @@
|
||||
# Agent Note: Keyless browser e2e lane for the web GUI
|
||||
|
||||
Status: implemented
|
||||
|
||||
English | [中文](2026-07-24-web-gui-browser-e2e-lane.zh.md)
|
||||
|
||||
## Problem
|
||||
|
||||
The web GUI ships as a real assembled chain — chromium page → client plugin bundles → HTTP unary RPC + two SSE streams → `toFetchHandler`/apiproxy → the host agent loop, tools, and JSONL persistence — and no test exercised that chain keylessly and deterministically. The [GUI testing system](../process/2026-07-20-gui-testing-system.md) covers tier 1 (wire isomorphism in node), tier 2 (object-layer state machines), and tier-3 smokes, but the keyless smoke drives `FixtureApiClient` — no host, no wire, no agent loop — while the full-chain smoke needs `DEEPSEEK_API_KEY` and a live model, so it is nondeterministic and self-skips in keyless CI. The snapshot philosophy of [docs/testing.md](../../../../docs/testing.md) — record once with a key, replay forever keyless, refresh on format churn — already covers the ACP, headless `stream-json`, and TUI transcript surfaces; the web surface was the one assembled product shape without it. The gap is exactly where the two confirmed GUI P0s hid: the wire carriage chain the fixture client short-circuits.
|
||||
|
||||
## Decision
|
||||
|
||||
`pnpm run test:web` carries a keyless, deterministic browser e2e lane under `apps/web/tests/`: recorded session-log fixtures replayed through `@deepseek-ai/dsh-llm-replay` against the real in-process web composition, asserting a normalized conversation aria golden plus in-process world state. No new package; the product deltas are two additive `dsh-llm-replay` surfaces (`paceMs`, `ReplayHandle`).
|
||||
|
||||
### Scaffold: `apps/web/tests/scaffold.ts`
|
||||
|
||||
A plain shared-fixture module (the [testing-policy sanctioned shape](../../../../docs/testing.md)), not a package: the gate-worthy logic — replay derivation, session parsing, log scrubbing, persistence — lives in the gated packages `dsh-llm-replay`, `dsh-acp-snapshot`, and `dsh-session-persistence-jsonl`; what remains is boot wiring and browser glue, and chromium-driving code cannot hold per-file 100% coverage on the browserless coverage runners.
|
||||
|
||||
`launchWebScaffold()` boots the real web composition from the shipped `apps/cli/cordis.yml` through the vendored Loader's include mechanism — the same tree and mechanism `AppCLIEntry` drives for `dsh web`. Divergences ride include patches over that tree, the ACP `cordis.snapshot.yml` pattern expressed in-process: temp `persistenceRoot`, `workspace-context` disabled (recorded fixtures must not embed this repo's AGENTS.md), `session-title-llm` disabled (its fire-and-forget title call would race the loop for the session's replay cursor), the webserver row pinned to port 0 with the built dist, and in keyless modes `llm-deepseek` disabled. A patch id that stops matching a row fails the boot sweep loudly instead of drifting. The boot runs `chdir`'d to the temp workspace so the api-gateway's `process.cwd()` session default, tool cwds, and fixtures agree; the `dsh web` bin's own glue (argv, profile json, AppCLIEntry) stays held by the keyless CLI smokes in `smoke-real.e2e.ts`. Setup rollback and ordinary close both dispose the Cordis tree before removing the two owned temp roots, attempt every cleanup independently, and report cleanup failures without masking the setup failure.
|
||||
|
||||
Keyless model displacement is the disabled adapter row plus `installLlmReplay` filling the open seam on the settled root ctx in providers-catalog mode — never catch-all: with the adapter row disabled no adapter exists, so catch-all would leave `resolveModelContext` unroutable and `compact-basic`'s post-step pressure check would warn every step instead of being provably inert (the published 128k `contextWindow` keeps it inert for small fixtures). The direct install rather than an inserted replay plugin row is deliberate: it returns the `ReplayHandle` the teardown consumption check needs. A scenario with no fixture leaves the seam empty, so a stray stream fails loud with NO_ADAPTER.
|
||||
|
||||
`seedSession()` seeds cold sessions through the real persistence API — a throwaway `Context` mounting `SessionStore` + `SessionPersistenceJsonl` against the host's root, `create()` + `append()`, one `utimes` backdate for deterministic sidebar order (the `semantic-checkpoint.snapshot.ts` precedent) — never raw file writes, so the seeder knows nothing of bucket hashing, filename encoding, or compression, and the host's zstd default needs no boot knob. Seeds are validated at seed time (parseable, ending in `turn/end` — an open final turn would be mutated by resume's crash repair).
|
||||
|
||||
### Determinism rules
|
||||
|
||||
The barrier stack for replay-mode browser assertions is, in order: (1) host-side `await agent.whenIdle()` under a timeout, keyed off the in-process `turn/end` — the idle flip follows the persistence flush, so one await covers turn completion and durability; (2) browser settled poll (streaming detached, final text visible). Record-mode log harvest runs after `whenIdle()` and before scaffold disposal while the live session remains available. An in-process `turn/end` listener alone is a wrong barrier (it fires before the SSE frame reaches the browser and before the fsync); file polling is banned (slow on NFS, superseded by `whenIdle`); `networkidle` is banned outright (never resolves while an SSE stream is open).
|
||||
|
||||
No single-shot transient-DOM assertions: every hop from replay yield to React commit can coalesce chunks, so sampling `[data-streaming]` is a race by construction. Streaming incrementality is asserted from the persisted `assistant/chunk` events (model-visible ⟺ logged makes the log the authoritative proof). `dsh-llm-replay`'s opt-in `paceMs` (default absent = burst) is a realism knob so the browser observes genuinely incremental SSE; correctness never leans on it, and abort during a pace wait cancels promptly.
|
||||
|
||||
Every scenario fails on any pageerror and on the client's connection-loss/gap-repair console warnings: the reconnect machine plus history resync would otherwise self-heal a dead SSE path and the suite would certify a broken wire. Scaffold `close()` calls the `ReplayHandle.assertConsumed()` teardown check (every recorded script bound, every cursor drained), converting silent underruns and shifted bindings into crisp diagnostics. No vitest retry on the lane; one chromium per file, fresh context per scenario, one host per scenario; viewport pinned; interaction selectors anchor on roles, `data-*` attributes, and visible text, while the frame and conversation-region captures use the existing CSS-module local-name anchors.
|
||||
|
||||
### Expected outputs
|
||||
|
||||
One committed golden per scenario: a normalized `ariaSnapshot()` of the conversation region (`ui.expected.md`) — uuid/cwd/workspace-basename/duration tokens normalized, captured poll-until-equal at the settled milestone — plus a few role/text anchor assertions that stay green under a semantics-preserving component rewrite while the golden churns reviewably. The aria tree is the mechanization of the client rule "assert what the user would see, never class names". World-state assertions ride root-context session events inline (which tool call produced which durable result, whether `turn/end` completed) instead of a second committed log golden: the persisted-log surface is pinned by the ACP/headless/TUI suites through the same loop and persistence, and re-pinning it here would double refresh cost against the tier discipline. `refresh` is the sole golden writer — a missing golden in replay mode fails with the healing command rather than self-bootstrapping.
|
||||
|
||||
The typecheck plane split is structural: the three files that boot the host spine (`scaffold`, `replay-round-trip.e2e`, and `seeded-history.e2e`) are excluded from the client-registered `apps/web` project. Those files and their shared `support.ts` are included file-by-file in `tsconfig.host.json` — one program cannot hold both sides of the cordis `Context` merges.
|
||||
|
||||
### Modes and fixtures
|
||||
|
||||
`DSH_SNAPSHOT` selects replay (default, keyless), record (with key), or refresh (keyless) as inline spec branches — the TUI shape, not a suite factory: at two scenarios the acp-snapshot factory machinery has no owner, and the genuinely shared parts are already exported (`scrubRequestHeaders`, `parseSessionLog`, `installLlmReplay`). Each spec splits into drive steps (type, send, `whenTurnSettled` — run in all modes, never waiting on model-content selectors, so record cannot hang on a live model answering differently) and assertion steps (replay/refresh only). Record = drive live through the real composer + harvest the in-memory `session.header`/`session.events` (the TUI `rawSessionLog` shape — no file decompression) + `scrubRequestHeaders` + `{{sessionId}}`/`{{cwd}}`/`{{rpcId}}` tokenization; a follow-up keyless refresh regenerates `ui.expected.md`. Both scenarios' fixtures were recorded against this assembly through this flow. A drift guard ties each spec's drive prompt to the fixture's recorded `user/message`. A fixture-inventory guard holds each scenario directory closed (exact file set, every JSONL a scrub fixed-point with no run-local `rpcId`). Web fixtures scrub headers everywhere and pin no header class, following the TUI precedent over the strict [pinned-header](2026-07-06-pin-request-header-content-in-one-scenario.md) reading — see Deferred.
|
||||
|
||||
### Scenarios
|
||||
|
||||
1. **`replay-round-trip`** — new session, prompt through the real composer, replay streams reasoning + a `bash` tool call that really executes in the temp workspace + final text (paced 15ms). Asserts settled markdown, the aria golden, and inline world state (the bash call's durable result is exactly `WEB_E2E_OK\n`, completed `turn/end`, >10 chunk events).
|
||||
2. **`seeded-history`** — a recorded session seeded cold; the sidebar lists it (group row → session row, collapsed by default), opening renders tool cards and text purely from the log through the implicit cold-resume attach inside `session.history` — zero model calls in replay, so no binding constraints; record mode drives the same turn live (real `read` tool against seeded workspace files) to produce the seed.
|
||||
|
||||
### CI stance
|
||||
|
||||
The lane ships gate-exempt inside `pnpm run test:web`, exactly as that config's header records. Adding chromium to CI would reverse the "no browser infrastructure in CI" premise in the [GUI testing note](../process/2026-07-20-gui-testing-system.md) and therefore requires its own Agent Note cross-linked from there, staged as a non-required job first with measured promotion criteria (consecutive green runs, wall time, zero-retry flake budget, runner browser-cache strategy). `TODO(ci-browser)` marks the seam. Scenarios are POSIX-oriented (the lane is not in the Windows matrix).
|
||||
|
||||
## Prior art
|
||||
|
||||
Surveyed AI-chat/agent web UIs and mocking layers (LibreChat, vercel/ai-chatbot + AI SDK, lobe-chat, open-webui, OpenHands, Chainlit, continue, cline, langfuse, gradio/streamlit; Playwright HAR/route, MSW, Polly/nock, WireMock, aimock). The dominant proven architecture for apps that own their backend is an in-process fake/replay model behind the real backend seam with everything downstream real (LibreChat's `LIBRECHAT_TEST_RUN_HOOK` fake model; ai-chatbot's `MockLanguageModelV3` + `simulateReadableStream`; continue's scripted mock provider classes) — which is what `dsh-llm-replay` already is. Browser-level SSE interception cannot exercise incremental rendering (`route.fulfill` delivers the whole body at once; playwright#33564) and leaves the server SSE stack untested, so projects use it only for edge cases. Chunk pacing as a fixture parameter recurs everywhere (LibreChat 10ms default with slow profiles; ai-chatbot 500ms); real models in CI rot (open-webui's suite grew 120-second timeouts, was disabled, then deleted); sessions are seeded at the persistence layer with controlled timestamps (LibreChat inserts backdated Mongo documents; langfuse seeds its DB). No surveyed project replays a recorded agent-event log through the real backend for UI tests — the closest are provider-level recorded fixtures (aimock) and frontend-level socket history emission (OpenHands MSW) — so the session-log-as-fixture design goes one step beyond prior art along the axis this repo's model-visible ⟺ logged invariant makes natural.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
**Browser-network SSE interception (`page.route`).** Rejected: `route.fulfill` cannot stream, so incremental token rendering is unexercisable and the server-side SSE/backpressure/close path — where both confirmed P0s hid — goes untested.
|
||||
|
||||
**Mock HTTP provider at `DEEPSEEK_BASE_URL`.** Rejected as the lane's mechanism (kept for the one existing workspace-probe smoke): fixtures become hand-authored OpenAI SSE byte scripts, a second fixture format that drifts from the session-log format the rest of the repo records and replays; the adapter's real HTTP path is with-key e2e's job.
|
||||
|
||||
**Growing the `?fixture` client.** Rejected: tier separation — `FixtureApiClient` exists to test the client shell without a server; everything below the client API seam stays untested by construction.
|
||||
|
||||
**Placeholder `DEEPSEEK_API_KEY` + replay interception instead of disabling the adapter row.** Rejected despite zero composition change and two in-tree precedents: it satisfies `llm-deepseek`'s fail-loud key check with a lie and leaves a dead adapter mounted-but-intercepted; the disabled row (the ACP overlay's move) is honest keylessness and fails loud at the earliest resolvable point.
|
||||
|
||||
**A `packages/support/web-snapshot` package with a `defineWebSnapshotSuite` factory.** Rejected: chromium-driving source cannot honestly hold per-file 100% coverage on browserless coverage runners, and at two scenarios a factory generalizes from one consumer while the genuinely shared logic is already exported from gated packages. Re-entry trigger: a second web-shaped consumer or ≥6 scenarios with demonstrably drifting inline branches; the package boundary would then be drawn browser-free.
|
||||
|
||||
**A committed normalized-session-log golden as a second expected surface.** Rejected: the log surface is pinned by the ACP/headless/TUI suites through the same loop and persistence; here it would double refresh cost and re-test lower tiers. Inline world-state assertions on root-context events keep the world-verification duty.
|
||||
|
||||
**Spawning the `dsh web` bin with a `DSH_SNAPSHOT` replay branch.** Rejected: it needs a test-only replay branch plus environment plumbing in the shipped CLI. The in-process scaffold already loads the same `apps/cli/cordis.yml`; only argv, profile JSON, and `AppCLIEntry` glue remain outside it, and the keyless CLI smokes cover those paths.
|
||||
|
||||
**Changing the wire protocol for testability.** Rejected: the contract already has a first-class keyless isomorphic seam (`InProcessApiClient(toFetchHandler(api))`), the per-event unbatched SSE is exactly what makes replay observable in a browser, and testing a wire we no longer ship would invert the tier's purpose.
|
||||
|
||||
**Real-model browser tests as the keyless lane.** Rejected: nondeterministic by construction; the surveyed cautionary case (open-webui) grew unbounded timeouts and was deleted. The with-key W5 smoke stays as the live-model complement.
|
||||
|
||||
**A client `data-dsh-busy` settled signal.** Deferred: the multi-condition settled polls proved sufficient at two scenarios and the host-side `whenIdle` barrier does the heavy lifting. Re-entry trigger: the first settled-poll flake, or a scenario needing a state the DOM does not expose.
|
||||
|
||||
## Testing
|
||||
|
||||
The lane itself: `pnpm run test:web` runs both scenarios keylessly alongside the existing smoke pair; `DSH_SNAPSHOT=record pnpm exec vitest run --config vitest.web.config.ts apps/web/tests/<spec>` re-records a scenario's fixture against the live model; `DSH_SNAPSHOT=refresh` rewrites both aria goldens keylessly. `paceMs` validation, pacing floor, abort-during-pace, and both `assertConsumed` failure shapes are pinned in `packages/support/llm-replay/tests/llm-replay.spec.ts`.
|
||||
|
||||
## Deferred
|
||||
|
||||
- **Web header-class pin**: web fixtures tokenize `{{system}}`/`{{tools}}` everywhere and no scenario pins the web composition's prompt/tool schemas (`TODO(web-header-pin)` — the scaffold `recordFixture` JSDoc marks it). Following the TUI scrub-everywhere precedent; revisit when the web assembly's header diverges from the repl composition it mirrors.
|
||||
- **CI browser provisioning**: reversal of the no-browser-in-CI ruling, staged criteria above (`TODO(ci-browser)`).
|
||||
- **Follow-up-prompt-after-resume scenario**: the history/live stitch path over the real wire; add as its own scenario when that code changes or regresses.
|
||||
|
||||
## Consequences
|
||||
|
||||
The web surface gains its record-once/replay-forever tier: the real chromium → SSE → apiproxy → loop → tools → persistence chain runs keylessly in ~10-30s, deterministic across repeat runs, with fixtures owned and re-recordable by the lane itself. Costs accepted: every intentional conversation-UI change ends with a keyless `DSH_SNAPSHOT=refresh` (golden churn is reviewed diff, anchors keep semantic green); the aria format is Playwright-owned — the one committed snapshot format the repo does not control — so playwright version bumps must be deliberate bump-and-refresh commits (the dependency floats `^1.49.0` in `apps/web/package.json`; pin exactly if churn bites); replay's first-call-order binding constrains scenarios to one prompting session each, with the consumption assertion as the tripwire; `compact-basic` shares the session's replay cursor and stays inert only under the published 128k catalog window; and the lane guards regressions only where it runs (locally, `test:web`) until the CI reversal is separately decided.
|
||||
@@ -0,0 +1,90 @@
|
||||
# Agent Note: Web GUI 的无密钥浏览器 e2e 车道
|
||||
|
||||
Status: implemented
|
||||
|
||||
[English](2026-07-24-web-gui-browser-e2e-lane.md) | 中文
|
||||
|
||||
## 问题
|
||||
|
||||
Web GUI 以一条真实组装链交付——chromium 页面 → client 插件 bundle → HTTP 单次 RPC + 两条 SSE（Server-Sent Events）流 → `toFetchHandler`/apiproxy → host 端的 agent loop（智能体循环）、工具与 JSONL 持久化——却没有任何测试无密钥且确定性地检验这条链。[GUI 测试体系](../process/2026-07-20-gui-testing-system.md)覆盖第 1 层（Node 中的协议同构）、第 2 层（对象层状态机）与第 3 层冒烟测试，但无密钥冒烟驱动的是 `FixtureApiClient`——没有 host、没有 wire、没有 agent loop——而全链路冒烟需要 `DEEPSEEK_API_KEY` 和真实模型，因此不确定、在无密钥 CI 中自行跳过。[docs/testing.md](../../../../docs/testing.md) 的快照哲学——带密钥录制一次、永久无密钥回放、格式变动时刷新——已覆盖 ACP（Agent Client Protocol）、headless `stream-json` 与 TUI 三个文本记录（transcript）表面；web 表面是唯一没有这层保障的组装形态。而缺口恰恰是两起已实证 GUI P0 藏身之处：fixture（测试前置数据）客户端短路掉的 wire 承载链。
|
||||
|
||||
## 决策
|
||||
|
||||
`pnpm run test:web` 携带 `apps/web/tests/` 下的无密钥、确定性浏览器 e2e 车道：录制的会话日志 fixture 经 `@deepseek-ai/dsh-llm-replay` 对真实进程内 web 组合回放，断言规范化后的会话区 aria 预期输出加进程内世界状态。不新增包（package）；产品侧增量只有 `dsh-llm-replay` 的两处增量接口（`paceMs`、`ReplayHandle`）。
|
||||
|
||||
### Scaffold：`apps/web/tests/scaffold.ts`
|
||||
|
||||
一个普通的共享 fixture 模块（[测试政策认可的形态](../../../../docs/testing.md)），不是包：值得门禁把守的逻辑——回放推导、会话解析、日志脱敏、持久化——都在已受门禁的包 `dsh-llm-replay`、`dsh-acp-snapshot`、`dsh-session-persistence-jsonl` 中；剩下的只是启动接线和浏览器胶水，而驱动 chromium 的源码在无浏览器的覆盖率 runner 上无法诚实保持逐文件 100% 覆盖率。
|
||||
|
||||
`launchWebScaffold()` 通过 vendored Loader 的 include 机制，从交付的 `apps/cli/cordis.yml` 启动真实 web 组合——与 `AppCLIEntry` 为 `dsh web` 驱动的是同一棵树、同一套机制。差异全部经 include patch 覆盖在这棵树上，即 ACP `cordis.snapshot.yml` 模式的进程内表达：临时 `persistenceRoot`；禁用 `workspace-context`（录制的 fixture 不得嵌入本仓库的 AGENTS.md）；禁用 `session-title-llm`（其发后不管的标题调用会与循环争抢会话的回放游标）；webserver 行钉到端口 0 加已构建 dist；无密钥模式下禁用 `llm-deepseek`。patch 的 id 一旦不再匹配任何行，boot 扫描会大声失败而不是漂移。boot 在临时工作区 `chdir` 下运行，使 api-gateway 的 `process.cwd()` 会话默认值、工具 cwd 与 fixture 一致；`dsh web` bin 自身的胶水（argv、profile json、AppCLIEntry）仍由 `smoke-real.e2e.ts` 中的无密钥 CLI 冒烟把守。初始化回滚和正常关闭都会先对 Cordis 树执行 dispose（资源释放），再删除 scaffold 持有的两个临时根目录；每项清理都会独立尝试，并会报告清理失败而不掩盖初始化失败。
|
||||
|
||||
无密钥的模型替换 = 禁用适配器行的 patch 加 `installLlmReplay` 在停稳的根 ctx 上以提供方目录（providers-catalog）模式填充开放的 seam——绝不用 catch-all：适配器行被禁用后不存在任何适配器，catch-all 会让 `resolveModelContext` 无路由可走，`compact-basic` 的步后压力检查将步步告警，而不是被可证明地闲置（发布的 128k `contextWindow` 使该路径对小 fixture 保持闲置）。选择直接安装而非插入回放插件行是刻意的：直接安装返回收尾消费检查所需的 `ReplayHandle`。没有 fixture 的场景让 seam 保持空置，任何离群的流式调用都会以 NO_ADAPTER 大声失败。
|
||||
|
||||
`seedSession()` 通过真实持久化 API 播种冷会话——一次性 `Context` 挂载 `SessionStore` + `SessionPersistenceJsonl` 指向 host 的根目录，`create()` + `append()`，一次 `utimes` 回拨保证侧栏顺序确定（`semantic-checkpoint.snapshot.ts` 先例）——绝不裸写文件，因此播种器对桶哈希、文件名编码、压缩一无所知，host 的 zstd 默认值也无需任何启动开关。种子在播种时即校验（可解析、以 `turn/end` 结尾——未闭合的最终轮次会被恢复（resume）的崩溃修复改写）。
|
||||
|
||||
### 确定性规则
|
||||
|
||||
回放模式下浏览器断言的屏障栈，按序：（1）host 侧 `await agent.whenIdle()` 加超时，以进程内 `turn/end` 为锚——空闲翻转发生在持久化落盘之后，一次等待同时覆盖轮次完成与持久性；（2）浏览器安定轮询（流式输出节点已卸载、最终文本可见）。录制模式下，日志采收在 `whenIdle()` 之后、scaffold 释放之前进行，此时运行中的会话仍然可用。单独监听进程内 `turn/end` 是错误屏障（它先于 SSE 帧到达浏览器、先于 fsync 触发）；文件轮询被禁止（NFS 上慢，且被 `whenIdle` 取代）；`networkidle` 被彻底禁止（SSE 流保持打开时它永不解析）。
|
||||
|
||||
不做单次瞬态 DOM 断言：从回放产出到 React 提交的每一跳都可能合并分片，采样 `[data-streaming]` 天然就是竞态。流式输出的增量性由持久化的 `assistant/chunk` 事件断言（模型可见 ⟺ 已记录，使日志成为权威证据）。`dsh-llm-replay` 的可选 `paceMs`（默认缺省 = 突发）只是让浏览器观察到真正增量 SSE 的真实感旋钮；正确性绝不依赖它，且节奏等待期间中止会即时取消。
|
||||
|
||||
每个场景都会因任何 pageerror 或客户端的连接丢失/间隙修复控制台警告而失败：否则重连机制加历史重同步会把一条死掉的 SSE 通路自愈掉，套件反而认证了坏 wire。Scaffold 的 `close()` 调用 `ReplayHandle.assertConsumed()` 收尾检查（每个已录脚本都被绑定、每个游标都耗尽），把静默的少放与错绑变成清晰诊断。车道不设 vitest 重试；每文件一个 chromium、每场景一个新 context、每场景一个 host；视口固定；交互选择器锚定 role、`data-*` 属性和可见文本，而 frame 与会话区采集则使用既有的 CSS 模块局部类名锚点。
|
||||
|
||||
### 预期输出
|
||||
|
||||
每场景一份提交的预期输出：会话区规范化 `ariaSnapshot()`（`ui.expected.md`）——uuid/cwd/工作区目录名/时长归一为稳定 token，在安定里程碑处轮询至两次相等再采集——外加几条 role/文本锚断言，让保语义的组件重写在预期输出可评审地变动时仍保持绿色锚点。aria 树是 client 规则「断言用户所见，绝不断言类名」的机械化。世界状态断言内联在根上下文的会话事件上（哪次工具调用产生了哪项已持久化的工具结果、`turn/end` 是否完成）而不是第二份提交的日志预期输出：持久化日志表面已由 ACP/headless/TUI 套件经同一循环和持久化钉住，在此重复钉住会违背分层纪律、翻倍刷新成本。`refresh` 是预期输出的唯一写入者——回放模式下预期输出缺失会连同修复命令一起报错，而不是静默自举。
|
||||
|
||||
类型检查平面切分是结构性的：启动 host 主干的三个文件（`scaffold`、`replay-round-trip.e2e` 和 `seeded-history.e2e`）被排除出注册在 client 侧的 `apps/web` 工程。这三个文件及其共享的 `support.ts` 逐文件纳入 `tsconfig.host.json`——一个程序不能同时持有 cordis `Context` 合并的两侧。
|
||||
|
||||
### 模式与 fixture
|
||||
|
||||
`DSH_SNAPSHOT` 以内联 spec 分支选择 replay（默认，无密钥）、record（带密钥）或 refresh（无密钥）——TUI 的形态，不是套件工厂：两个场景撑不起 acp-snapshot 工厂机制，且真正共享的部分已被导出（`scrubRequestHeaders`、`parseSessionLog`、`installLlmReplay`）。每个 spec 切分为驱动步骤（输入、发送、`whenTurnSettled`——所有模式都执行，绝不等待模型内容选择器，因此 record 不会因真实模型答法不同而挂起）与断言步骤（仅 replay/refresh）。Record = 经真实输入框实时驱动 + 采收内存中的 `session.header`/`session.events`（TUI 的 `rawSessionLog` 形态——无需文件解压）+ `scrubRequestHeaders` + `{{sessionId}}`/`{{cwd}}`/`{{rpcId}}` token 化；随后一次无密钥 refresh 重新生成 `ui.expected.md`。两个场景的 fixture 都经此流程对本组装录制。一条漂移防线把每个 spec 的驱动提示词与 fixture 录制的 `user/message` 绑定。fixture 清单防线保持每个场景目录封闭（精确文件集合，每个 JSONL 都是脱敏不动点，不含当次运行的 `rpcId`）。Web fixture 全部脱敏请求头且不钉任何头类别，沿用 TUI 先例而非[钉住请求头](2026-07-06-pin-request-header-content-in-one-scenario.md)的严格读法——见「暂缓」。
|
||||
|
||||
### 场景
|
||||
|
||||
1. **`replay-round-trip`**——新会话，经真实输入框发送提示词，回放流式输出推理（reasoning）+ 一次在临时工作区真实执行的 `bash` 工具调用 + 最终文本（15ms 节奏）。断言安定后的 markdown、aria 预期输出与内联世界状态（这次 bash 调用的已持久化工具结果严格等于 `WEB_E2E_OK\n`、完成的 `turn/end`、>10 个分片事件）。
|
||||
2. **`seeded-history`**——冷播种一份已录会话；侧栏列出它（分组行 → 会话行，默认折叠），打开后纯凭日志经 `session.history` 内的隐式冷恢复挂载渲染工具卡片与文本——replay 下零模型调用，因此没有任何绑定约束；record 模式实时驱动同一轮（真实 `read` 工具读取播种的工作区文件）来产出种子。
|
||||
|
||||
### CI 立场
|
||||
|
||||
车道随 `pnpm run test:web` 交付、豁免门禁，与该配置头部注释所记一致。往 CI 加 chromium 会推翻 [GUI 测试笔记](../process/2026-07-20-gui-testing-system.md)中「CI 无浏览器基础设施」的前提，因此需要自己的 Agent Note 并从那里交叉链接，分阶段推进：先作为非必需任务，再以量化标准晋升（连续绿色运行次数、耗时、零重试的抖动预算、runner 浏览器缓存策略）。`TODO(ci-browser)` 标记该接缝。场景目前面向 POSIX（车道不在 Windows 矩阵中）。
|
||||
|
||||
## 业界先例
|
||||
|
||||
调研了 AI 聊天/agent web UI 与 mock 层（LibreChat、vercel/ai-chatbot + AI SDK、lobe-chat、open-webui、OpenHands、Chainlit、continue、cline、langfuse、gradio/streamlit；Playwright HAR/route、MSW、Polly/nock、WireMock、aimock）。自有后端的应用的主流成熟架构是：真实后端 seam 后放一个进程内伪造/回放模型，下游全部真实（LibreChat 的 `LIBRECHAT_TEST_RUN_HOOK` 伪模型；ai-chatbot 的 `MockLanguageModelV3` + `simulateReadableStream`；continue 的脚本化 mock 提供方类）——这正是 `dsh-llm-replay` 已然所是。浏览器层 SSE 拦截无法检验增量渲染（`route.fulfill` 一次性交付整个响应体；playwright#33564），且服务端 SSE 栈完全失测，因此各项目只把它用于边缘用例。分片节奏作为 fixture 参数反复出现（LibreChat 默认 10ms 附慢速档；ai-chatbot 500ms）；CI 里的真实模型会腐烂（open-webui 的套件长出 120 秒超时，先被禁用后被删除）；会话在持久化层以受控时间戳播种（LibreChat 直插回拨时间的 Mongo 文档；langfuse 播种其数据库）。没有任何被调研项目为 UI 测试把录制的 agent 事件日志经真实后端回放——最接近的是提供方层录制 fixture（aimock）与前端层 socket 历史发射（OpenHands MSW）——因此会话日志即 fixture 的设计沿着本仓库「模型可见 ⟺ 已记录」不变式所指的方向比业界先例多走了一步。
|
||||
|
||||
## 曾考虑的替代方案
|
||||
|
||||
**浏览器网络层 SSE 拦截（`page.route`）。** 已否决：`route.fulfill` 无法流式输出，增量 token 渲染无从检验，且服务端 SSE/背压/关闭路径——两起已实证 P0 的藏身处——完全失测。
|
||||
|
||||
**`DEEPSEEK_BASE_URL` 处的 mock HTTP 提供方。** 作为本车道机制已否决（仅保留给既有的工作区探针冒烟）：fixture 会变成手写的 OpenAI SSE 字节脚本，一种与仓库其余部分录制回放的会话日志格式渐行渐远的第二 fixture 格式；适配器的真实 HTTP 路径归带密钥 e2e 管。
|
||||
|
||||
**扩展 `?fixture` 客户端。** 已否决：分层纪律——`FixtureApiClient` 的存在意义就是脱离服务器测试客户端 shell；client API seam 以下按构造即失测。
|
||||
|
||||
**用占位 `DEEPSEEK_API_KEY` + 回放拦截替代禁用适配器行。** 尽管零组合改动且树内有两处先例仍被否决：它用谎言满足 `llm-deepseek` 的快速失败密钥检查，还留下一个挂载却被拦截的死适配器；禁用行（ACP overlay 的同款做法）是诚实的无密钥，并在最早可解析点快速失败。
|
||||
|
||||
**`packages/support/web-snapshot` 包 + `defineWebSnapshotSuite` 工厂。** 已否决：驱动 chromium 的源码在无浏览器的覆盖率 runner 上无法诚实保持逐文件 100%，且两个场景就上工厂是从单一消费方过度泛化，真正共享的逻辑已从受门禁的包中导出。重启条件：出现第二个 web 形态消费方，或 ≥6 个场景的内联分支被证实各自漂移；届时包边界将画在无浏览器一侧。
|
||||
|
||||
**第二份提交的规范化会话日志预期输出。** 已否决：日志表面已由 ACP/headless/TUI 套件经同一循环与持久化钉住；在此只会翻倍刷新成本并重复测试下层。内联在根上下文事件上的世界状态断言保住了验证世界的义务。
|
||||
|
||||
**以 `DSH_SNAPSHOT` 回放分支拉起 `dsh web` bin。** 已否决：它需要在交付的 CLI 中增加测试专用回放分支和环境变量管道。进程内 scaffold 已加载同一份 `apps/cli/cordis.yml`；只剩 argv、profile JSON 和 `AppCLIEntry` 胶水不在其覆盖范围内，而这些路径已由无密钥 CLI 冒烟覆盖。
|
||||
|
||||
**为可测试性改 wire 协议。** 已否决：契约已有第一等的无密钥同构 seam（`InProcessApiClient(toFetchHandler(api))`），逐事件不合批的 SSE 恰是回放在浏览器中可观测的原因，测试一条不再交付的 wire 会颠倒该层的存在意义。
|
||||
|
||||
**以真实模型浏览器测试充当无密钥车道。** 已否决：按构造即不确定；被调研的前车之鉴（open-webui）长出无界超时后被删除。带密钥的 W5 冒烟仍是真实模型侧的补充。
|
||||
|
||||
**客户端 `data-dsh-busy` 安定信号。** 暂缓：两个场景下多条件安定轮询已经够用，host 侧 `whenIdle` 屏障承担了重活。重启条件：第一次安定轮询抖动，或某场景需要等待 DOM 不暴露的状态。
|
||||
|
||||
## Testing
|
||||
|
||||
车道自身：`pnpm run test:web` 与既有冒烟对一起无密钥运行两个场景；`DSH_SNAPSHOT=record pnpm exec vitest run --config vitest.web.config.ts apps/web/tests/<spec>` 对真实模型重录某场景的 fixture；`DSH_SNAPSHOT=refresh` 无密钥重写两份 aria 预期输出。`paceMs` 校验、节奏下限、节奏中中止、`assertConsumed` 的两种失败形态钉在 `packages/support/llm-replay/tests/llm-replay.spec.ts`。
|
||||
|
||||
## 暂缓
|
||||
|
||||
- **Web 头类别钉住**：web fixture 处处 token 化 `{{system}}`/`{{tools}}`，没有场景钉住 web 组合的提示词/工具 schema（`TODO(web-header-pin)`——scaffold 的 `recordFixture` JSDoc 有标记）。沿用 TUI 处处脱敏先例；当 web 组装的请求头与其镜像的 repl 组合进一步分叉时重审。
|
||||
- **CI 浏览器供给**：推翻 CI 无浏览器裁定，分阶段标准见上（`TODO(ci-browser)`）。
|
||||
- **恢复后追问场景**：真实 wire 上的历史/实时缝合路径；当该代码变更或回归时作为独立场景补充。
|
||||
|
||||
## 后果
|
||||
|
||||
Web 表面获得了录制一次/永久回放的层级：真实 chromium → SSE → apiproxy → 循环 → 工具 → 持久化的链路以约 10-30 秒无密钥运行，重复运行结果确定，fixture 由车道自身持有并可重录。接受的成本：每次有意的会话 UI 变更都以一次无密钥 `DSH_SNAPSHOT=refresh` 收尾（预期输出变动是受评审的 diff，锚断言保住语义绿色）；aria 格式归 Playwright 所有——仓库唯一不受自己控制的提交快照格式——因此 playwright 版本升级必须是刻意的升级加刷新提交（依赖在 `apps/web/package.json` 中浮动为 `^1.49.0`；若变动伤人则改为精确锁定）；回放的首次调用顺序绑定把每个场景限制为至多一个发起提示的会话，消费断言是绊线；`compact-basic` 与会话共享回放游标，仅在发布的 128k 目录窗口下保持闲置；在 CI 反转被单独决策之前，车道只在其运行之处（本地，`test:web`）把守回归。
|
||||
129
apps/web/tests/replay-round-trip.e2e.ts
Normal file
129
apps/web/tests/replay-round-trip.e2e.ts
Normal file
@@ -0,0 +1,129 @@
|
||||
// Web e2e scenario: fresh round trip. A real chromium types a prompt into the
|
||||
// real composer; the wire, apiproxy, agent loop, and the REAL bash tool (echo
|
||||
// in the temp workspace) all run; the model seam is dsh-llm-replay (keyless)
|
||||
// or the live adapter (record). Drive steps run in every mode and wait only
|
||||
// on generic completion (whenTurnSettled — never model-content selectors, so
|
||||
// record cannot hang on a live model answering differently); assertion steps
|
||||
// run in replay/refresh only. Settled states only — streaming incrementality
|
||||
// is asserted from the persisted assistant/chunk events, not transient DOM.
|
||||
// Record: DSH_SNAPSHOT=record rewrites session.jsonl, then a keyless
|
||||
// DSH_SNAPSHOT=refresh regenerates ui.expected.md.
|
||||
import { readFile } from 'node:fs/promises'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import type { Browser, Page } from 'playwright'
|
||||
import { chromium } from 'playwright'
|
||||
import { afterAll, beforeAll, describe, expect, it, onTestFailed } from 'vitest'
|
||||
import type { SessionEvent } from '@deepseek-ai/dsh-session'
|
||||
import {
|
||||
assertFixtureInventory, captureStableAria, compareOrRefreshGolden, fixtureUserPrompts,
|
||||
launchWebScaffold, recordFixture, watchConsole, webSnapshotMode, type WebScaffold,
|
||||
} from './scaffold.ts'
|
||||
import { saveFailureShot } from './support.ts'
|
||||
|
||||
const SNAPSHOT_DIR = fileURLToPath(new URL('./snapshots/fresh-round-trip', import.meta.url))
|
||||
const FIXTURE = fileURLToPath(new URL('./snapshots/fresh-round-trip/session.jsonl', import.meta.url))
|
||||
const UI_EXPECTED = fileURLToPath(new URL('./snapshots/fresh-round-trip/ui.expected.md', import.meta.url))
|
||||
const MODE = webSnapshotMode()
|
||||
|
||||
// The scenario's one drive prompt. Record sends it; replay asserts the
|
||||
// committed fixture recorded exactly it, so drive script and fixture cannot
|
||||
// drift apart.
|
||||
const PROMPT = 'Use the bash tool to run exactly: echo WEB_E2E_OK. Then reply with the single word DONE and stop.'
|
||||
|
||||
describe('web e2e: fresh round trip through the real assembly', () => {
|
||||
let scaffold: WebScaffold
|
||||
let browser: Browser
|
||||
let page: Page
|
||||
let tripwire: ReturnType<typeof watchConsole>
|
||||
const sessionEvents: SessionEvent[] = []
|
||||
|
||||
beforeAll(async () => {
|
||||
scaffold = await launchWebScaffold({
|
||||
...(MODE === 'record' ? {} : { replayFixture: FIXTURE, paceMs: 15 }),
|
||||
})
|
||||
scaffold.ctx.on('session/event', (_session, event: SessionEvent) => { sessionEvents.push(event) })
|
||||
browser = await chromium.launch()
|
||||
page = await browser.newPage({ viewport: { width: 1680, height: 1000 } })
|
||||
tripwire = watchConsole(page)
|
||||
await page.goto(scaffold.baseUrl, { waitUntil: 'load' })
|
||||
await page.waitForSelector('[class*="frame"]', { timeout: 30_000 })
|
||||
}, 120_000)
|
||||
|
||||
afterAll(async () => {
|
||||
await browser?.close()
|
||||
await scaffold?.close()
|
||||
})
|
||||
|
||||
it('drives the recorded prompt to a settled turn (all modes)', async () => {
|
||||
onTestFailed(() => saveFailureShot(page, 'web-e2e-round-trip'))
|
||||
if (MODE !== 'record') {
|
||||
// Drift guard: the committed fixture must carry exactly the drive prompt.
|
||||
expect(fixtureUserPrompts(await readFile(FIXTURE, 'utf8'))).toEqual([PROMPT])
|
||||
}
|
||||
const input = page.locator('textarea').first()
|
||||
await input.waitFor({ timeout: 10_000 })
|
||||
// Arm the host-side settled barrier BEFORE the send click.
|
||||
const settled = scaffold.whenTurnSettled()
|
||||
await input.fill(PROMPT)
|
||||
await input.press('Enter')
|
||||
const sessionId = await settled
|
||||
if (MODE === 'record') {
|
||||
await recordFixture(scaffold, sessionId, FIXTURE)
|
||||
}
|
||||
}, 200_000)
|
||||
|
||||
it.skipIf(MODE === 'record')('rendered the settled turn: markdown, tool row, composer restore', async () => {
|
||||
onTestFailed(() => saveFailureShot(page, 'web-e2e-round-trip-settled'))
|
||||
// Browser settled-poll after host completion (host strictly precedes render).
|
||||
await page.locator('[data-streaming="true"]').waitFor({ state: 'detached', timeout: 15_000 }).catch(() => {
|
||||
// Chunks may coalesce into one commit; a never-mounted streaming node is
|
||||
// legal — the chunk-event assertions below carry incrementality.
|
||||
})
|
||||
await expect.poll(() => page.getByText('DONE', { exact: true }).count(), { timeout: 15_000 }).toBeGreaterThanOrEqual(1)
|
||||
// World state, not self-report: the real bash executor returned the exact
|
||||
// command output, and the turn closed cleanly.
|
||||
const bashCall = sessionEvents.find(event => event.type === 'tool/call' && event.data.name === 'bash')
|
||||
if (bashCall?.type !== 'tool/call') throw new Error('the replayed turn did not call the bash tool')
|
||||
const bashResult = sessionEvents.find(event =>
|
||||
event.type === 'tool/result' && event.data.callId === bashCall.data.callId)
|
||||
if (bashResult?.type !== 'tool/result') throw new Error('the bash tool call produced no durable result')
|
||||
expect(bashResult.data.isError).toBe(false)
|
||||
expect(bashResult.data.content.filter(block => block.type === 'text').map(block => block.text).join(''))
|
||||
.toBe('WEB_E2E_OK\n')
|
||||
const turnEnds = sessionEvents.filter(e => e.type === 'turn/end')
|
||||
expect(turnEnds.length).toBe(1)
|
||||
expect((turnEnds[0] as SessionEvent & { data: { reason: { kind: string } } }).data.reason.kind).toBe('completed')
|
||||
// The persisted chunk events are the authoritative incrementality proof.
|
||||
expect(sessionEvents.filter(e => e.type === 'assistant/chunk').length).toBeGreaterThan(10)
|
||||
}, 60_000)
|
||||
|
||||
it.skipIf(MODE === 'record')('matches the conversation aria golden with stable anchors', async () => {
|
||||
onTestFailed(() => saveFailureShot(page, 'web-e2e-round-trip-aria'))
|
||||
// Anchor assertions survive a semantics-preserving component rewrite even
|
||||
// while the whole-region golden churns.
|
||||
await expect(page.getByRole('textbox').first().isVisible()).resolves.toBe(true)
|
||||
expect(await page.getByText('WEB_E2E_OK', { exact: false }).count()).toBeGreaterThanOrEqual(1)
|
||||
const snapshot = await captureStableAria(page, '[class*="centerCol"]', scaffold.workspaceCwd)
|
||||
await compareOrRefreshGolden(UI_EXPECTED, snapshot, MODE)
|
||||
})
|
||||
|
||||
it.skipIf(MODE === 'record')('expands and collapses the reasoning fold from its click target', async () => {
|
||||
onTestFailed(() => saveFailureShot(page, 'web-e2e-round-trip-think'))
|
||||
// Interaction over the REAL wire-delivered transcript (the fixture-client
|
||||
// tier pins the same gesture against FixtureApiClient; this one runs on
|
||||
// mux-frame-fed state). Runs after the golden capture so the committed
|
||||
// aria surface stays the untouched settled state.
|
||||
const think = page.getByRole('button', { name: /^Think/ }).first()
|
||||
expect(await think.getAttribute('aria-expanded')).toBe('false')
|
||||
await think.click()
|
||||
await expect.poll(() => think.getAttribute('aria-expanded'), { timeout: 5_000 }).toBe('true')
|
||||
await think.click()
|
||||
await expect.poll(() => think.getAttribute('aria-expanded'), { timeout: 5_000 }).toBe('false')
|
||||
})
|
||||
|
||||
it.skipIf(MODE === 'record')('stayed clean: no pageerrors, no reconnect self-healing, no server errors', async () => {
|
||||
expect(tripwire.pageErrors).toEqual([])
|
||||
expect(tripwire.warnings).toEqual([])
|
||||
await assertFixtureInventory(SNAPSHOT_DIR, ['session.jsonl', 'ui.expected.md'])
|
||||
})
|
||||
})
|
||||
440
apps/web/tests/scaffold.ts
Normal file
440
apps/web/tests/scaffold.ts
Normal file
@@ -0,0 +1,440 @@
|
||||
// Shared scaffold for the keyless browser e2e lane (Agent Note:
|
||||
// .agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md).
|
||||
// Boots the REAL web composition — the shipped apps/cli/cordis.yml through
|
||||
// the vendored Loader (the same include boot AppCLIEntry drives), patched the
|
||||
// snapshot way — so a real chromium exercises the real HTTP/SSE wire, the
|
||||
// api-gateway, agent loop, tools, and persistence. Modes ride $DSH_SNAPSHOT:
|
||||
// replay (default, keyless: llm-deepseek row disabled, dsh-llm-replay row
|
||||
// inserted in providers mode), record (real adapter + key, harvests fixtures
|
||||
// from live session memory), refresh (keyless replay that rewrites goldens).
|
||||
//
|
||||
// Composition divergences from `dsh web`, all deliberate, all via include
|
||||
// patches over the SAME tree (never a second yml): temp persistenceRoot;
|
||||
// workspace-context disabled (recorded fixtures must not embed this repo's
|
||||
// AGENTS.md); session-title-llm disabled (its fire-and-forget title call
|
||||
// would race the loop for the session's replay cursor); webserver pinned to
|
||||
// port 0 with the built dist; keyless modes disable llm-deepseek and fill
|
||||
// the open llm seam post-boot with installLlmReplay on the settled root ctx
|
||||
// (the plugin-row path discards the ReplayHandle; the direct install keeps
|
||||
// assertConsumed for the teardown fixture-consumption check).
|
||||
import { existsSync, readFileSync } from 'node:fs'
|
||||
import { mkdtemp, readFile, readdir, rm, utimes, writeFile } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join, resolve } from 'node:path'
|
||||
import { pathToFileURL } from 'node:url'
|
||||
import type { Page } from 'playwright'
|
||||
import { expect } from 'vitest'
|
||||
import { Context } from 'cordis'
|
||||
import Loader from '@cordisjs/plugin-loader'
|
||||
import Include, { type PatchOptions } from '@cordisjs/plugin-include'
|
||||
import { scrubRequestHeaders } from '@deepseek-ai/dsh-acp-snapshot'
|
||||
import { assertEntriesLoaded } from '@deepseek-ai/dsh-app-boot'
|
||||
import type { ReplayHandle } from '@deepseek-ai/dsh-llm-replay'
|
||||
import { installLlmReplay, parseSessionLog } from '@deepseek-ai/dsh-llm-replay'
|
||||
import SessionStore, { SESSION_FORMAT_VERSION, SessionId } from '@deepseek-ai/dsh-session'
|
||||
import type { Session, SessionEvent, SessionHeader } from '@deepseek-ai/dsh-session'
|
||||
import SessionPersistenceJsonl from '@deepseek-ai/dsh-session-persistence-jsonl'
|
||||
// Empty type imports carry the httpServer/agents/sessionPersistence Context merges.
|
||||
import type {} from '@deepseek-ai/dsh-host-webserver'
|
||||
import type {} from '@deepseek-ai/dsh-agent'
|
||||
import { DIST_INDEX, REPO_ROOT, requireDist } from './support.ts'
|
||||
|
||||
/** Snapshot mode for the lane, from $DSH_SNAPSHOT (same vocabulary as the ACP/TUI suites). */
|
||||
export type WebSnapshotMode = 'replay' | 'record' | 'refresh'
|
||||
|
||||
/**
|
||||
* Resolve and validate the lane's snapshot mode.
|
||||
* @returns the active mode; unset/empty selects replay.
|
||||
*/
|
||||
export function webSnapshotMode(): WebSnapshotMode {
|
||||
const value = process.env.DSH_SNAPSHOT
|
||||
if (value === undefined || value === '' || value === 'replay') return 'replay'
|
||||
if (value === 'record' || value === 'refresh') return value
|
||||
throw new Error(`DSH_SNAPSHOT must be replay, record, or refresh; got ${JSON.stringify(value)}`)
|
||||
}
|
||||
|
||||
/** The shipped composition under test: apps/cli's config tree. */
|
||||
const CONFIG_PATH = join(REPO_ROOT, 'apps/cli/cordis.yml')
|
||||
|
||||
// Replay publishes the provider catalog the gateway routes to (providers
|
||||
// mode, never catch-all: with llm-deepseek disabled no adapter exists, so a
|
||||
// catch-all would leave resolveModelContext unroutable and compact-basic's
|
||||
// post-step pressure check would warn every step). The published
|
||||
// contextWindow keeps that pressure path provably inert for small fixtures.
|
||||
const REPLAY_PROVIDERS = [{ id: 'deepseek', name: 'DeepSeek', models: [{ id: 'deepseek-v4-flash', contextWindow: 128_000 }] }]
|
||||
|
||||
/** Repo-root .env → process.env for record mode (never overrides set vars); the smoke-real convention. */
|
||||
function loadRootEnv(): void {
|
||||
const envPath = join(REPO_ROOT, '.env')
|
||||
if (!existsSync(envPath)) return
|
||||
for (const line of readFileSync(envPath, 'utf8').split('\n')) {
|
||||
const m = /^([A-Za-z_][A-Za-z0-9_]*)=(.*)$/.exec(line.trim())
|
||||
if (m !== null && process.env[m[1]!] === undefined) process.env[m[1]!] = m[2]
|
||||
}
|
||||
}
|
||||
|
||||
/** A booted web scaffold: real composition, mode-selected model backend, temp world. */
|
||||
export interface WebScaffold {
|
||||
/** The active snapshot mode this scaffold booted under. */
|
||||
mode: WebSnapshotMode
|
||||
/** Browser-facing origin (http://127.0.0.1:<bound port>). */
|
||||
baseUrl: string
|
||||
/** Settled root context (the in-process barrier seam; headless event subscription is its sanctioned use). */
|
||||
ctx: Context
|
||||
/** Temp project directory sessions run in (bash/fs tool cwd). */
|
||||
workspaceCwd: string
|
||||
/** Temp persistence root (seeded sessions land here through the real API). */
|
||||
persistenceRoot: string
|
||||
/** Await a settled turn end: in-process turn/end, then the agent's idle flip (which follows the persistence flush). */
|
||||
whenTurnSettled(timeoutMs?: number): Promise<SessionId>
|
||||
/** Tear everything down; asserts the replay fixture was fully consumed first (replay/refresh). */
|
||||
close(): Promise<void>
|
||||
}
|
||||
|
||||
/** Options for {@link launchWebScaffold}. */
|
||||
export interface LaunchOptions {
|
||||
/**
|
||||
* Replay fixture (session.jsonl) served by the inserted dsh-llm-replay row
|
||||
* in replay/refresh modes; ignored in record mode (the real adapter
|
||||
* answers). Omit for scenarios issuing no model calls — a stray stream then
|
||||
* fails loud with NO_ADAPTER (llm-deepseek is disabled and no replay row
|
||||
* mounts).
|
||||
*/
|
||||
replayFixture?: string
|
||||
/** Per-chunk replay pacing (ms) so the browser observes genuinely incremental SSE; replay/refresh only. */
|
||||
paceMs?: number
|
||||
}
|
||||
|
||||
/** Dispose the booted tree and remove both owned temp roots, reporting every independent cleanup failure. */
|
||||
async function cleanupScaffoldWorld(ctx: Context, workspaceCwd: string, persistenceRoot: string): Promise<unknown[]> {
|
||||
const failures: unknown[] = []
|
||||
await Promise.resolve(ctx.fiber.dispose()).catch((error: unknown) => failures.push(error))
|
||||
await rm(workspaceCwd, { recursive: true, force: true }).catch((error: unknown) => failures.push(error))
|
||||
await rm(persistenceRoot, { recursive: true, force: true }).catch((error: unknown) => failures.push(error))
|
||||
return failures
|
||||
}
|
||||
|
||||
/**
|
||||
* Boot the real web composition under the current snapshot mode.
|
||||
* @param options - replay fixture selection and pacing.
|
||||
* @returns the running scaffold.
|
||||
*/
|
||||
export async function launchWebScaffold(options: LaunchOptions = {}): Promise<WebScaffold> {
|
||||
requireDist()
|
||||
const mode = webSnapshotMode()
|
||||
if (mode === 'record') {
|
||||
loadRootEnv()
|
||||
if (process.env.DEEPSEEK_API_KEY === undefined || process.env.DEEPSEEK_API_KEY.length === 0) {
|
||||
throw new Error('web e2e record mode needs DEEPSEEK_API_KEY (env or repo-root .env)')
|
||||
}
|
||||
}
|
||||
const workspaceCwd = await mkdtemp(join(tmpdir(), 'dsh-web-e2e-ws-'))
|
||||
let persistenceRoot: string
|
||||
try {
|
||||
persistenceRoot = await mkdtemp(join(tmpdir(), 'dsh-web-e2e-sessions-'))
|
||||
} catch (error) {
|
||||
const failures: unknown[] = [error]
|
||||
await rm(workspaceCwd, { recursive: true, force: true }).catch((cleanupError: unknown) => failures.push(cleanupError))
|
||||
if (failures.length > 1) throw new AggregateError(failures, 'web scaffold temp-root setup failed')
|
||||
throw error
|
||||
}
|
||||
|
||||
// The include patch set — the same mechanism AppCLIEntry and the ACP
|
||||
// snapshot overlay use, applied over the SAME shipped tree (a patch id that
|
||||
// stops matching a row fails the boot sweep loudly instead of drifting).
|
||||
const patches: PatchOptions[] = [
|
||||
{ id: 'session-persistence-jsonl', config: { root: persistenceRoot } },
|
||||
// storage-json's './.storages' yml default is cwd-relative and resolves
|
||||
// per write; the scaffold restores the original cwd after boot, so the
|
||||
// row gets an absolute temp root (removed with the workspace at close).
|
||||
{ id: 'storage-json', config: { root: join(workspaceCwd, '.dsh-storages') } },
|
||||
// fs/bash cwd default to process.cwd(); the gateway injects the same
|
||||
// value into session.cwd — chdir below anchors all three to the temp
|
||||
// workspace, keeping the composition untouched.
|
||||
{ id: 'workspace-context', disabled: true },
|
||||
{ id: 'session-title-llm', disabled: true },
|
||||
{ id: 'webserver', config: { host: '127.0.0.1', port: 0, distIndex: DIST_INDEX } },
|
||||
...mode === 'record' ? [] : [{ id: 'llm-deepseek', disabled: true }],
|
||||
]
|
||||
|
||||
// Sessions inherit the gateway's process.cwd() default; run the boot from
|
||||
// the temp workspace so tool cwd, session cwd, and fixtures agree.
|
||||
const originalCwd = process.cwd()
|
||||
const ctx = new Context()
|
||||
let port = 0
|
||||
let replayHandle: ReplayHandle | undefined
|
||||
try {
|
||||
process.chdir(workspaceCwd)
|
||||
ctx.baseUrl = pathToFileURL(join(resolve(CONFIG_PATH), '..')).href + '/'
|
||||
await ctx.plugin(Loader)
|
||||
ctx.loader.builtins.include = Include
|
||||
await ctx.loader.create({
|
||||
name: 'cordis:include',
|
||||
config: { path: pathToFileURL(resolve(CONFIG_PATH)).href, patches },
|
||||
})
|
||||
await ctx.loader.await()
|
||||
assertEntriesLoaded(ctx, 'web e2e scaffold')
|
||||
const boundPort = ctx.get('httpServer')?.port
|
||||
if (boundPort === undefined) {
|
||||
throw new Error('web e2e scaffold: httpServer service missing after settled boot')
|
||||
}
|
||||
port = boundPort
|
||||
|
||||
// Fill the open llm seam on the settled root ctx (llm-deepseek is disabled
|
||||
// in keyless modes; a scenario with no fixture leaves the seam empty so a
|
||||
// stray stream fails loud with NO_ADAPTER). The direct install, unlike the
|
||||
// plugin row, returns the ReplayHandle for the teardown consumption check.
|
||||
if (mode !== 'record' && options.replayFixture !== undefined) {
|
||||
replayHandle = installLlmReplay(ctx, {
|
||||
file: options.replayFixture,
|
||||
providers: REPLAY_PROVIDERS,
|
||||
...(options.paceMs === undefined ? {} : { paceMs: options.paceMs }),
|
||||
})
|
||||
}
|
||||
} catch (error) {
|
||||
if (process.cwd() !== originalCwd) process.chdir(originalCwd)
|
||||
const cleanupFailures = await cleanupScaffoldWorld(ctx, workspaceCwd, persistenceRoot)
|
||||
if (cleanupFailures.length > 0) {
|
||||
throw new AggregateError([error, ...cleanupFailures], 'web scaffold setup failed and cleanup was incomplete')
|
||||
}
|
||||
throw error
|
||||
} finally {
|
||||
if (process.cwd() !== originalCwd) process.chdir(originalCwd)
|
||||
}
|
||||
|
||||
return {
|
||||
mode,
|
||||
baseUrl: `http://127.0.0.1:${port}`,
|
||||
ctx,
|
||||
workspaceCwd,
|
||||
persistenceRoot,
|
||||
// Barrier stack: the in-process turn/end identifies the session, then
|
||||
// agent.whenIdle() covers the persistence flush (the idle flip follows
|
||||
// the flush), and the caller's browser settled-poll comes last because
|
||||
// host completion strictly precedes render.
|
||||
whenTurnSettled(timeoutMs = mode === 'record' ? 180_000 : 30_000): Promise<SessionId> {
|
||||
return new Promise<SessionId>((resolveSettled, reject) => {
|
||||
const timer = setTimeout(() => {
|
||||
off()
|
||||
reject(new Error(`no turn/end within ${timeoutMs}ms`))
|
||||
}, timeoutMs)
|
||||
const off = ctx.on('session/event', (session: { id: SessionId }, event: SessionEvent) => {
|
||||
if (event.type !== 'turn/end') return
|
||||
clearTimeout(timer)
|
||||
off()
|
||||
const agent = ctx.agents.get(session.id)
|
||||
if (agent === undefined) {
|
||||
reject(new Error(`turn/end for ${session.id} but no live agent`))
|
||||
return
|
||||
}
|
||||
agent.whenIdle().then(() => { resolveSettled(session.id) }, reject)
|
||||
})
|
||||
})
|
||||
},
|
||||
async close(): Promise<void> {
|
||||
const failures: unknown[] = []
|
||||
// Fixture-consumption check first, while the run's binding state is
|
||||
// still authoritative — a scenario that drove fewer model calls than
|
||||
// recorded fails here instead of drifting green.
|
||||
try {
|
||||
replayHandle?.assertConsumed()
|
||||
} catch (error) {
|
||||
failures.push(error)
|
||||
}
|
||||
failures.push(...await cleanupScaffoldWorld(ctx, workspaceCwd, persistenceRoot))
|
||||
if (failures.length > 0) throw new AggregateError(failures, 'web scaffold teardown failed')
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Serialize a live session back to raw session-JSONL (header + events) — the
|
||||
* in-memory record-mode harvest, so the on-disk zstd default never matters.
|
||||
* Mirrors the TUI suite's rawSessionLog.
|
||||
*/
|
||||
function rawSessionLog(session: Session): string {
|
||||
return [
|
||||
JSON.stringify({ type: 'session', ...session.header }),
|
||||
...session.events.map(event => JSON.stringify(event)),
|
||||
'',
|
||||
].join('\n')
|
||||
}
|
||||
|
||||
/**
|
||||
* Record-mode fixture write-back: harvest the live session, scrub request
|
||||
* headers to {{system}}/{{tools}} (TODO(web-header-pin): the web lane pins no
|
||||
* header class — a deliberate deviation logged in the Agent Note's deferred
|
||||
* work), tokenize the run-local session id, cwd, and browser RPC id
|
||||
* ({{sessionId}}/{{cwd}}/{{rpcId}}, the committed fixture convention —
|
||||
* re-records then diff only on real content), and write the fixture.
|
||||
* @param scaffold - the record-mode scaffold.
|
||||
* @param sessionId - the driven session.
|
||||
* @param fixturePath - the committed session.jsonl / seed.jsonl target.
|
||||
*/
|
||||
export async function recordFixture(scaffold: WebScaffold, sessionId: SessionId, fixturePath: string): Promise<void> {
|
||||
const agent = scaffold.ctx.agents.get(sessionId)
|
||||
if (agent === undefined) throw new Error(`record harvest: no live agent for ${sessionId}`)
|
||||
const tokenized = scrubRequestHeaders(rawSessionLog(agent.session))
|
||||
.split(sessionId).join('{{sessionId}}')
|
||||
.split(scaffold.workspaceCwd).join('{{cwd}}')
|
||||
.replace(/"rpcId":"[^"]+"/g, '"rpcId":"{{rpcId}}"')
|
||||
await writeFile(fixturePath, tokenized)
|
||||
}
|
||||
|
||||
/**
|
||||
* The user prompts recorded in a fixture, in order — the single source tying
|
||||
* spec drive steps to recorded reality so script and fixture cannot drift.
|
||||
* @param fixtureText - raw session.jsonl contents.
|
||||
* @returns the recorded user prompt texts.
|
||||
*/
|
||||
export function fixtureUserPrompts(fixtureText: string): string[] {
|
||||
return parseSessionLog(fixtureText).flatMap((event) => {
|
||||
if (event.type !== 'user/message' || event.data.source.kind !== 'user') return []
|
||||
const text = event.data.content.filter(block => block.type === 'text').map(block => block.text).join('')
|
||||
return text.length > 0 ? [text] : []
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* Seed a recorded session fixture into the scaffold's persistence root
|
||||
* through the REAL backend API (throwaway Context + SessionStore + JSONL
|
||||
* plugin — the semantic-checkpoint precedent), never raw file writes: no
|
||||
* knowledge of bucket hashing, filename encoding, or compression, and
|
||||
* malformed shapes fail loud at seed time. The fixture's tokenized identity
|
||||
* ({{sessionId}}/{{cwd}}) is realized for this world before parsing.
|
||||
* @param scaffold - the target scaffold.
|
||||
* @param fixtureText - raw recorded session.jsonl contents.
|
||||
* @param id - the seeded session id (stable for deterministic goldens).
|
||||
* @returns the seeded id.
|
||||
*/
|
||||
export async function seedSession(scaffold: WebScaffold, fixtureText: string, id: string): Promise<SessionId> {
|
||||
const realized = fixtureText
|
||||
.split('{{sessionId}}').join(id)
|
||||
.split('{{cwd}}').join(scaffold.workspaceCwd)
|
||||
const fixtureCwd = (JSON.parse(realized.split('\n', 1)[0]!) as { cwd?: string }).cwd
|
||||
const rewritten = fixtureCwd === undefined
|
||||
? realized
|
||||
: realized.split(fixtureCwd).join(scaffold.workspaceCwd)
|
||||
const events = parseSessionLog(rewritten)
|
||||
if (events.length === 0) throw new Error('seed fixture has no events')
|
||||
const last = events[events.length - 1]!
|
||||
// An open final turn would be mutated by resume's crash repair on first
|
||||
// open; a committed seed must be a closed recording.
|
||||
if (last.type !== 'turn/end') throw new Error(`seed fixture must end in turn/end, got ${last.type}`)
|
||||
const meta: SessionHeader = {
|
||||
version: SESSION_FORMAT_VERSION,
|
||||
id: SessionId(id),
|
||||
createdAt: Date.now() - 60_000,
|
||||
cwd: scaffold.workspaceCwd,
|
||||
delegationDepth: 0,
|
||||
}
|
||||
const seeder = new Context()
|
||||
try {
|
||||
await seeder.plugin(SessionStore)
|
||||
// Same root as the booted tree with the plugin's own default compression,
|
||||
// so the host's directory-scan list() sees one consistent encoding.
|
||||
await seeder.plugin(SessionPersistenceJsonl, { root: scaffold.persistenceRoot })
|
||||
await seeder.sessionPersistence.create(meta)
|
||||
await seeder.sessionPersistence.append(meta.id, events)
|
||||
// Deterministic sidebar order: cold summaries take updatedAt from mtime.
|
||||
const located = seeder.sessionPersistence.locate(meta)
|
||||
if (located !== undefined) {
|
||||
const backdated = new Date(meta.createdAt)
|
||||
await utimes(located.path, backdated, backdated)
|
||||
}
|
||||
} finally {
|
||||
await seeder.fiber.dispose()
|
||||
}
|
||||
return meta.id
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalize an aria snapshot: uuid, cwd, workspace-basename, and duration
|
||||
* volatility collapse to stable tokens.
|
||||
*/
|
||||
function normalizeAria(snapshot: string, workspaceCwd: string): string {
|
||||
// The header breadcrumb renders the workspace's basename, not the full
|
||||
// path, so both spellings must collapse to the token.
|
||||
const base = workspaceCwd.split('/').pop()!
|
||||
return snapshot
|
||||
.split(workspaceCwd).join('{{cwd}}')
|
||||
.split(base).join('{{workspace}}')
|
||||
.replace(/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/gi, '{{uuid}}')
|
||||
.replace(/\b\d+(?:\.\d+)?(?:ms|s|秒)\b/g, '{{duration}}')
|
||||
}
|
||||
|
||||
/**
|
||||
* Capture the region's aria snapshot at a settled milestone: poll until two
|
||||
* consecutive normalized captures are equal — a single-shot capture races the
|
||||
* last React commits.
|
||||
* @param page - the page under test.
|
||||
* @param selector - the region locator selector.
|
||||
* @param workspaceCwd - normalization input.
|
||||
* @returns the stable normalized snapshot.
|
||||
*/
|
||||
export async function captureStableAria(page: Page, selector: string, workspaceCwd: string): Promise<string> {
|
||||
const region = page.locator(selector).first()
|
||||
let previous = normalizeAria(await region.ariaSnapshot(), workspaceCwd)
|
||||
await expect.poll(async () => {
|
||||
const current = normalizeAria(await region.ariaSnapshot(), workspaceCwd)
|
||||
const stable = current === previous
|
||||
previous = current
|
||||
return stable
|
||||
}, { timeout: 5_000, message: 'aria snapshot did not stabilize' }).toBe(true)
|
||||
return previous
|
||||
}
|
||||
|
||||
/**
|
||||
* Compare a normalized golden, or rewrite it under refresh. Refresh is the
|
||||
* ONLY writer: a missing golden in replay mode fails with the healing command
|
||||
* instead of silently self-bootstrapping.
|
||||
* @param goldenPath - the committed ui.expected.md path.
|
||||
* @param actual - the stable normalized snapshot.
|
||||
* @param mode - the active snapshot mode.
|
||||
*/
|
||||
export async function compareOrRefreshGolden(goldenPath: string, actual: string, mode: WebSnapshotMode): Promise<void> {
|
||||
const payload = `${actual}\n`
|
||||
if (mode === 'refresh') {
|
||||
await writeFile(goldenPath, payload)
|
||||
return
|
||||
}
|
||||
if (!existsSync(goldenPath)) {
|
||||
throw new Error(`missing golden ${goldenPath} — run DSH_SNAPSHOT=refresh pnpm run test:web to generate it`)
|
||||
}
|
||||
expect(payload).toBe(await readFile(goldenPath, 'utf8'))
|
||||
}
|
||||
|
||||
/**
|
||||
* Fixture-inventory guard (the TUI afterAll shape): the scenario directory
|
||||
* holds exactly the expected files and every committed JSONL is a scrub
|
||||
* fixed-point without a run-local browser RPC id.
|
||||
* @param dir - the scenario snapshot directory.
|
||||
* @param expected - the exact expected file inventory.
|
||||
*/
|
||||
export async function assertFixtureInventory(dir: string, expected: string[]): Promise<void> {
|
||||
const entries = (await readdir(dir)).sort()
|
||||
expect(entries).toEqual([...expected].sort())
|
||||
for (const entry of entries.filter(name => name.endsWith('.jsonl'))) {
|
||||
const content = await readFile(join(dir, entry), 'utf8')
|
||||
expect(scrubRequestHeaders(content), `${dir}/${entry} carries request-header bulk`).toBe(content)
|
||||
expect(content, `${dir}/${entry} carries a run-local rpcId`)
|
||||
.not.toMatch(/"rpcId":"(?!\{\{rpcId\}\})[^"]+"/)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Console tripwires: reconnect/gap-repair self-healing or a pageerror must
|
||||
* fail the scenario, not mask a dead wire behind eventual consistency.
|
||||
* @param page - the page under test.
|
||||
* @returns live warning/pageerror collectors to assert empty at scenario end.
|
||||
*/
|
||||
export function watchConsole(page: Page): { warnings: string[]; pageErrors: string[] } {
|
||||
const warnings: string[] = []
|
||||
const pageErrors: string[] = []
|
||||
page.on('console', (message) => {
|
||||
const text = message.text()
|
||||
if (/connection lost|gap repair|discontinuous/i.test(text)) warnings.push(text)
|
||||
})
|
||||
page.on('pageerror', (error) => { pageErrors.push(String(error)) })
|
||||
return { warnings, pageErrors }
|
||||
}
|
||||
124
apps/web/tests/seeded-history.e2e.ts
Normal file
124
apps/web/tests/seeded-history.e2e.ts
Normal file
@@ -0,0 +1,124 @@
|
||||
// Web e2e scenario: seeded history. A recorded session seeded cold through
|
||||
// the REAL persistence API renders purely from the log — the surface nothing
|
||||
// else covers: sidebar cold listing, the implicit resume/attach inside the
|
||||
// history RPC, history-page tool views, and the client fold of historical
|
||||
// events — with ZERO model calls in replay (no replay fixture; a stray stream
|
||||
// fails loud on the open llm seam). The seed is a recorded fixture under the
|
||||
// same record discipline as every other: DSH_SNAPSHOT=record drives the turn
|
||||
// live through the composer (real read tool against seeded workspace files)
|
||||
// and harvests seed.jsonl; replay/refresh seed it cold and only render.
|
||||
import { readFile, writeFile, mkdir } from 'node:fs/promises'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import type { Browser, Page } from 'playwright'
|
||||
import { chromium } from 'playwright'
|
||||
import { afterAll, beforeAll, describe, expect, it, onTestFailed } from 'vitest'
|
||||
import { join } from 'node:path'
|
||||
import {
|
||||
assertFixtureInventory, captureStableAria, compareOrRefreshGolden, fixtureUserPrompts,
|
||||
launchWebScaffold, recordFixture, seedSession, watchConsole, webSnapshotMode, type WebScaffold,
|
||||
} from './scaffold.ts'
|
||||
import { saveFailureShot } from './support.ts'
|
||||
|
||||
const SNAPSHOT_DIR = fileURLToPath(new URL('./snapshots/seeded-history', import.meta.url))
|
||||
const SEED = fileURLToPath(new URL('./snapshots/seeded-history/seed.jsonl', import.meta.url))
|
||||
const UI_EXPECTED = fileURLToPath(new URL('./snapshots/seeded-history/ui.expected.md', import.meta.url))
|
||||
const MODE = webSnapshotMode()
|
||||
const SEED_ID = 'seeded-history-web-e2e'
|
||||
|
||||
const PROMPT = 'Use the read tool twice in one assistant message: read a.txt and b.txt. Then reply with the single word DONE and stop.'
|
||||
|
||||
describe('web e2e: seeded history renders through cold resume', () => {
|
||||
let scaffold: WebScaffold
|
||||
let browser: Browser
|
||||
let page: Page
|
||||
let tripwire: ReturnType<typeof watchConsole>
|
||||
|
||||
beforeAll(async () => {
|
||||
scaffold = await launchWebScaffold({})
|
||||
// The workspace-aware flow runs sessions in <workspaceRoot>/workspace
|
||||
// (the composer's default draft name); the read-tool targets must live in
|
||||
// that session cwd. Pre-creating the directory is safe: create-by-name
|
||||
// adopts an existing directory.
|
||||
const sessionCwd = join(scaffold.workspaceCwd, 'workspace')
|
||||
await mkdir(sessionCwd, { recursive: true })
|
||||
await writeFile(join(sessionCwd, 'a.txt'), 'alpha\n')
|
||||
await writeFile(join(sessionCwd, 'b.txt'), 'beta\n')
|
||||
if (MODE !== 'record') {
|
||||
const raw = await readFile(SEED, 'utf8')
|
||||
expect(fixtureUserPrompts(raw), 'seed fixture must carry exactly the drive prompt').toEqual([PROMPT])
|
||||
await seedSession(scaffold, raw, SEED_ID)
|
||||
}
|
||||
browser = await chromium.launch()
|
||||
page = await browser.newPage({ viewport: { width: 1680, height: 1000 } })
|
||||
tripwire = watchConsole(page)
|
||||
await page.goto(scaffold.baseUrl, { waitUntil: 'load' })
|
||||
await page.waitForSelector('[class*="frame"]', { timeout: 30_000 })
|
||||
}, 120_000)
|
||||
|
||||
afterAll(async () => {
|
||||
await browser?.close()
|
||||
await scaffold?.close()
|
||||
})
|
||||
|
||||
it.skipIf(MODE !== 'record')('records the seed turn live through the composer', async () => {
|
||||
onTestFailed(() => saveFailureShot(page, 'web-e2e-seeded-record'))
|
||||
const input = page.locator('textarea').first()
|
||||
await input.waitFor({ timeout: 10_000 })
|
||||
const settled = scaffold.whenTurnSettled()
|
||||
await input.fill(PROMPT)
|
||||
await input.press('Enter')
|
||||
const sessionId = await settled
|
||||
await recordFixture(scaffold, sessionId, SEED)
|
||||
}, 200_000)
|
||||
|
||||
it.skipIf(MODE === 'record')('lists the seeded session cold and renders its history from the log', async () => {
|
||||
onTestFailed(() => saveFailureShot(page, 'web-e2e-seeded-history'))
|
||||
// The sidebar tree collapses workspace groups by default: click the group
|
||||
// row (treeitem 0) to expand, then the revealed session row.
|
||||
const groupRow = page.locator('[role="treeitem"]').first()
|
||||
await groupRow.waitFor({ timeout: 15_000 })
|
||||
await groupRow.click()
|
||||
const sessionRow = page.locator('[role="treeitem"]').nth(1)
|
||||
await sessionRow.waitFor({ timeout: 10_000 })
|
||||
await sessionRow.click()
|
||||
// Settled barrier for history: the recorded final assistant text renders.
|
||||
await expect.poll(() => page.getByText('DONE', { exact: true }).count(), { timeout: 15_000 }).toBe(1)
|
||||
// Tool cards render from logged tool/call + tool/result alone (views are
|
||||
// host-recomputed per page; the generic card is the documented default).
|
||||
const toolRows = page.locator('[data-variant], [data-sample]')
|
||||
await expect.poll(() => toolRows.count(), { timeout: 10_000 }).toBeGreaterThanOrEqual(2)
|
||||
expect(await page.getByText('a.txt', { exact: false }).count()).toBeGreaterThan(0)
|
||||
}, 60_000)
|
||||
|
||||
it.skipIf(MODE === 'record')('matches the historical conversation aria golden', async () => {
|
||||
onTestFailed(() => saveFailureShot(page, 'web-e2e-seeded-aria'))
|
||||
const snapshot = (await captureStableAria(page, '[class*="centerCol"]', scaffold.workspaceCwd))
|
||||
.split(SEED_ID).join('{{seededId}}')
|
||||
await compareOrRefreshGolden(UI_EXPECTED, snapshot, MODE)
|
||||
})
|
||||
|
||||
it.skipIf(MODE === 'record')('expands and collapses a tool row rebuilt from the cold log', async () => {
|
||||
onTestFailed(() => saveFailureShot(page, 'web-e2e-seeded-toolrow'))
|
||||
// Interaction over cold-resumed history: read rows are expand-in-place
|
||||
// rows (rowExpands routes the click to toggleExpand, not openDetails), so
|
||||
// the gesture under test is the inline fold over log-rebuilt content.
|
||||
// Runs after the golden capture; still zero model calls.
|
||||
const row = page.locator('[data-variant] [data-clickable][role="button"]').first()
|
||||
await row.waitFor({ timeout: 10_000 })
|
||||
expect(await row.getAttribute('aria-expanded')).toBe('false')
|
||||
await row.click()
|
||||
await expect.poll(() => row.getAttribute('aria-expanded'), { timeout: 5_000 }).toBe('true')
|
||||
// The expanded body renders the recorded tool result (a.txt's contents).
|
||||
await expect.poll(() => page.getByText('alpha', { exact: false }).count(), { timeout: 5_000 }).toBeGreaterThan(0)
|
||||
await row.click()
|
||||
await expect.poll(() => row.getAttribute('aria-expanded'), { timeout: 5_000 }).toBe('false')
|
||||
})
|
||||
|
||||
it.skipIf(MODE === 'record')('issued zero model calls and stayed clean', async () => {
|
||||
// No replay fixture was installed and the llm seam is open — any stray
|
||||
// stream would have failed the turn loudly. Cleanliness pins the wire.
|
||||
expect(tripwire.pageErrors).toEqual([])
|
||||
expect(tripwire.warnings).toEqual([])
|
||||
await assertFixtureInventory(SNAPSHOT_DIR, ['seed.jsonl', 'ui.expected.md'])
|
||||
})
|
||||
})
|
||||
95
apps/web/tests/snapshots/fresh-round-trip/session.jsonl
Normal file
95
apps/web/tests/snapshots/fresh-round-trip/session.jsonl
Normal file
@@ -0,0 +1,95 @@
|
||||
{"type":"session","version":0,"id":"{{sessionId}}","createdAt":1784973850091,"cwd":"{{cwd}}/workspace"}
|
||||
{"type":"turn/start","seq":0,"time":1784973850102,"data":{"turn":1,"trigger":{"kind":"message","source":{"kind":"user","rpcId":"{{rpcId}}"}}}}
|
||||
{"type":"user/message","seq":1,"time":1784973850103,"data":{"content":[{"type":"text","text":"Use the bash tool to run exactly: echo WEB_E2E_OK. Then reply with the single word DONE and stop."}],"source":{"kind":"user","rpcId":"{{rpcId}}"}},"surfaceOp":"append"}
|
||||
{"type":"session/title","seq":2,"time":1784973850105,"data":{"title":"Use the bash tool to","messageSeqs":[1],"source":{"kind":"fallback"}}}
|
||||
{"type":"step/start","seq":3,"time":1784973850164,"data":{"turn":1,"step":1}}
|
||||
{"type":"request/header","seq":4,"time":1784973850165,"data":{"header":{"config":{"provider":"deepseek","model":"deepseek-v4-flash"},"system":"{{system}}","tools":"{{tools}}","messagePrefix":["{{messagePrefix}}"]},"reason":"initial"}}
|
||||
{"type":"assistant/chunk","seq":5,"time":1784973850888,"data":{"turn":1,"step":1,"chunk":{"type":"block-start","index":0,"blockType":"reasoning"}}}
|
||||
{"type":"assistant/chunk","seq":6,"time":1784973850889,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":"The"}}}
|
||||
{"type":"assistant/chunk","seq":7,"time":1784973851088,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" user"}}}
|
||||
{"type":"assistant/chunk","seq":8,"time":1784973851089,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" wants"}}}
|
||||
{"type":"assistant/chunk","seq":9,"time":1784973851089,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" me"}}}
|
||||
{"type":"assistant/chunk","seq":10,"time":1784973851089,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" to"}}}
|
||||
{"type":"assistant/chunk","seq":11,"time":1784973851089,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" run"}}}
|
||||
{"type":"assistant/chunk","seq":12,"time":1784973851107,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" a"}}}
|
||||
{"type":"assistant/chunk","seq":13,"time":1784973851108,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" simple"}}}
|
||||
{"type":"assistant/chunk","seq":14,"time":1784973851108,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" bash"}}}
|
||||
{"type":"assistant/chunk","seq":15,"time":1784973851108,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" command"}}}
|
||||
{"type":"assistant/chunk","seq":16,"time":1784973851108,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" and"}}}
|
||||
{"type":"assistant/chunk","seq":17,"time":1784973851108,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" reply"}}}
|
||||
{"type":"assistant/chunk","seq":18,"time":1784973851135,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" with"}}}
|
||||
{"type":"assistant/chunk","seq":19,"time":1784973851135,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" \""}}}
|
||||
{"type":"assistant/chunk","seq":20,"time":1784973851136,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":"D"}}}
|
||||
{"type":"assistant/chunk","seq":21,"time":1784973851136,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":"ONE"}}}
|
||||
{"type":"assistant/chunk","seq":22,"time":1784973851136,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":"\"."}}}
|
||||
{"type":"assistant/chunk","seq":23,"time":1784973851217,"data":{"turn":1,"step":1,"chunk":{"type":"block-start","index":1,"blockType":"tool-call"}}}
|
||||
{"type":"assistant/chunk","seq":24,"time":1784973851217,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_BYXlxjFaalMg95YVqEeF2495","name":"bash","argumentsDelta":""}}}
|
||||
{"type":"assistant/chunk","seq":25,"time":1784973851244,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_BYXlxjFaalMg95YVqEeF2495","name":"bash","argumentsDelta":"{"}}}
|
||||
{"type":"assistant/chunk","seq":26,"time":1784973851244,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_BYXlxjFaalMg95YVqEeF2495","name":"bash","argumentsDelta":"\""}}}
|
||||
{"type":"assistant/chunk","seq":27,"time":1784973851244,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_BYXlxjFaalMg95YVqEeF2495","name":"bash","argumentsDelta":"command"}}}
|
||||
{"type":"assistant/chunk","seq":28,"time":1784973851244,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_BYXlxjFaalMg95YVqEeF2495","name":"bash","argumentsDelta":"\""}}}
|
||||
{"type":"assistant/chunk","seq":29,"time":1784973851245,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_BYXlxjFaalMg95YVqEeF2495","name":"bash","argumentsDelta":": "}}}
|
||||
{"type":"assistant/chunk","seq":30,"time":1784973851271,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_BYXlxjFaalMg95YVqEeF2495","name":"bash","argumentsDelta":"\""}}}
|
||||
{"type":"assistant/chunk","seq":31,"time":1784973851271,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_BYXlxjFaalMg95YVqEeF2495","name":"bash","argumentsDelta":"echo"}}}
|
||||
{"type":"assistant/chunk","seq":32,"time":1784973851271,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_BYXlxjFaalMg95YVqEeF2495","name":"bash","argumentsDelta":" WEB"}}}
|
||||
{"type":"assistant/chunk","seq":33,"time":1784973851272,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_BYXlxjFaalMg95YVqEeF2495","name":"bash","argumentsDelta":"_E"}}}
|
||||
{"type":"assistant/chunk","seq":34,"time":1784973851299,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_BYXlxjFaalMg95YVqEeF2495","name":"bash","argumentsDelta":"2"}}}
|
||||
{"type":"assistant/chunk","seq":35,"time":1784973851299,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_BYXlxjFaalMg95YVqEeF2495","name":"bash","argumentsDelta":"E"}}}
|
||||
{"type":"assistant/chunk","seq":36,"time":1784973851299,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_BYXlxjFaalMg95YVqEeF2495","name":"bash","argumentsDelta":"_OK"}}}
|
||||
{"type":"assistant/chunk","seq":37,"time":1784973851300,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_BYXlxjFaalMg95YVqEeF2495","name":"bash","argumentsDelta":"\""}}}
|
||||
{"type":"assistant/chunk","seq":38,"time":1784973851326,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_BYXlxjFaalMg95YVqEeF2495","name":"bash","argumentsDelta":", "}}}
|
||||
{"type":"assistant/chunk","seq":39,"time":1784973851326,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_BYXlxjFaalMg95YVqEeF2495","name":"bash","argumentsDelta":"\""}}}
|
||||
{"type":"assistant/chunk","seq":40,"time":1784973851352,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_BYXlxjFaalMg95YVqEeF2495","name":"bash","argumentsDelta":"description"}}}
|
||||
{"type":"assistant/chunk","seq":41,"time":1784973851353,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_BYXlxjFaalMg95YVqEeF2495","name":"bash","argumentsDelta":"\""}}}
|
||||
{"type":"assistant/chunk","seq":42,"time":1784973851353,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_BYXlxjFaalMg95YVqEeF2495","name":"bash","argumentsDelta":": "}}}
|
||||
{"type":"assistant/chunk","seq":43,"time":1784973851353,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_BYXlxjFaalMg95YVqEeF2495","name":"bash","argumentsDelta":"\""}}}
|
||||
{"type":"assistant/chunk","seq":44,"time":1784973851379,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_BYXlxjFaalMg95YVqEeF2495","name":"bash","argumentsDelta":"E"}}}
|
||||
{"type":"assistant/chunk","seq":45,"time":1784973851406,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_BYXlxjFaalMg95YVqEeF2495","name":"bash","argumentsDelta":"cho"}}}
|
||||
{"type":"assistant/chunk","seq":46,"time":1784973851406,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_BYXlxjFaalMg95YVqEeF2495","name":"bash","argumentsDelta":" the"}}}
|
||||
{"type":"assistant/chunk","seq":47,"time":1784973851435,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_BYXlxjFaalMg95YVqEeF2495","name":"bash","argumentsDelta":" test"}}}
|
||||
{"type":"assistant/chunk","seq":48,"time":1784973851435,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_BYXlxjFaalMg95YVqEeF2495","name":"bash","argumentsDelta":" string"}}}
|
||||
{"type":"assistant/chunk","seq":49,"time":1784973851435,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_BYXlxjFaalMg95YVqEeF2495","name":"bash","argumentsDelta":"\""}}}
|
||||
{"type":"assistant/chunk","seq":50,"time":1784973851461,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_BYXlxjFaalMg95YVqEeF2495","name":"bash","argumentsDelta":"}"}}}
|
||||
{"type":"assistant/chunk","seq":51,"time":1784973851493,"data":{"turn":1,"step":1,"chunk":{"type":"block-end","index":0,"block":{"type":"reasoning","text":"The user wants me to run a simple bash command and reply with \"DONE\"."}}}}
|
||||
{"type":"assistant/chunk","seq":52,"time":1784973851493,"data":{"turn":1,"step":1,"chunk":{"type":"block-end","index":1,"block":{"type":"tool-call","id":"call_00_BYXlxjFaalMg95YVqEeF2495","name":"bash","arguments":"{\"command\": \"echo WEB_E2E_OK\", \"description\": \"Echo the test string\"}"}}}}
|
||||
{"type":"assistant/chunk","seq":53,"time":1784973851493,"data":{"turn":1,"step":1,"chunk":{"type":"usage","usage":{"inputTokens":122,"outputTokens":85,"cacheReadTokens":7680,"reasoningTokens":17}}}}
|
||||
{"type":"assistant/chunk","seq":54,"time":1784973851493,"data":{"turn":1,"step":1,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}}
|
||||
{"type":"assistant/message","seq":55,"time":1784973851498,"data":{"turn":1,"step":1,"content":[{"type":"reasoning","text":"The user wants me to run a simple bash command and reply with \"DONE\"."},{"type":"tool-call","id":"call_00_BYXlxjFaalMg95YVqEeF2495","name":"bash","arguments":"{\"command\": \"echo WEB_E2E_OK\", \"description\": \"Echo the test string\"}"}],"provenance":{"provider":"deepseek","model":"deepseek-v4-flash"},"usage":{"inputTokens":122,"outputTokens":85,"cacheReadTokens":7680,"reasoningTokens":17}},"sourceEventSeqs":[5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54],"surfaceOp":"append"}
|
||||
{"type":"tool/call","seq":56,"time":1784973851499,"data":{"turn":1,"step":1,"callId":"call_00_BYXlxjFaalMg95YVqEeF2495","name":"bash","arguments":"{\"command\": \"echo WEB_E2E_OK\", \"description\": \"Echo the test string\"}"}}
|
||||
{"type":"tool/result","seq":57,"time":1784973851515,"data":{"turn":1,"step":1,"callId":"call_00_BYXlxjFaalMg95YVqEeF2495","content":[{"type":"text","text":"WEB_E2E_OK\n"}],"isError":false},"sourceEventSeqs":[56],"surfaceOp":"append"}
|
||||
{"type":"step/end","seq":58,"time":1784973851517,"data":{"turn":1,"step":1}}
|
||||
{"type":"step/start","seq":59,"time":1784973851518,"data":{"turn":1,"step":2}}
|
||||
{"type":"assistant/chunk","seq":60,"time":1784973852194,"data":{"turn":1,"step":2,"chunk":{"type":"block-start","index":0,"blockType":"reasoning"}}}
|
||||
{"type":"assistant/chunk","seq":61,"time":1784973852195,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"The"}}}
|
||||
{"type":"assistant/chunk","seq":62,"time":1784973852309,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" command"}}}
|
||||
{"type":"assistant/chunk","seq":63,"time":1784973852338,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" executed"}}}
|
||||
{"type":"assistant/chunk","seq":64,"time":1784973852339,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" successfully"}}}
|
||||
{"type":"assistant/chunk","seq":65,"time":1784973852339,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" and"}}}
|
||||
{"type":"assistant/chunk","seq":66,"time":1784973852339,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" output"}}}
|
||||
{"type":"assistant/chunk","seq":67,"time":1784973852370,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" \""}}}
|
||||
{"type":"assistant/chunk","seq":68,"time":1784973852370,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"WEB"}}}
|
||||
{"type":"assistant/chunk","seq":69,"time":1784973852371,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"_E"}}}
|
||||
{"type":"assistant/chunk","seq":70,"time":1784973852371,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"2"}}}
|
||||
{"type":"assistant/chunk","seq":71,"time":1784973852371,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"E"}}}
|
||||
{"type":"assistant/chunk","seq":72,"time":1784973852371,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"_OK"}}}
|
||||
{"type":"assistant/chunk","seq":73,"time":1784973852398,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"\"."}}}
|
||||
{"type":"assistant/chunk","seq":74,"time":1784973852398,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" I"}}}
|
||||
{"type":"assistant/chunk","seq":75,"time":1784973852398,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" just"}}}
|
||||
{"type":"assistant/chunk","seq":76,"time":1784973852398,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" need"}}}
|
||||
{"type":"assistant/chunk","seq":77,"time":1784973852428,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" to"}}}
|
||||
{"type":"assistant/chunk","seq":78,"time":1784973852429,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" reply"}}}
|
||||
{"type":"assistant/chunk","seq":79,"time":1784973852429,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" with"}}}
|
||||
{"type":"assistant/chunk","seq":80,"time":1784973852429,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" \""}}}
|
||||
{"type":"assistant/chunk","seq":81,"time":1784973852429,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"D"}}}
|
||||
{"type":"assistant/chunk","seq":82,"time":1784973852429,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"ONE"}}}
|
||||
{"type":"assistant/chunk","seq":83,"time":1784973852459,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"\"."}}}
|
||||
{"type":"assistant/chunk","seq":84,"time":1784973852459,"data":{"turn":1,"step":2,"chunk":{"type":"block-start","index":1,"blockType":"text"}}}
|
||||
{"type":"assistant/chunk","seq":85,"time":1784973852459,"data":{"turn":1,"step":2,"chunk":{"type":"text-delta","index":1,"text":"D"}}}
|
||||
{"type":"assistant/chunk","seq":86,"time":1784973852459,"data":{"turn":1,"step":2,"chunk":{"type":"text-delta","index":1,"text":"ONE"}}}
|
||||
{"type":"assistant/chunk","seq":87,"time":1784973852459,"data":{"turn":1,"step":2,"chunk":{"type":"block-end","index":0,"block":{"type":"reasoning","text":"The command executed successfully and output \"WEB_E2E_OK\". I just need to reply with \"DONE\"."}}}}
|
||||
{"type":"assistant/chunk","seq":88,"time":1784973852460,"data":{"turn":1,"step":2,"chunk":{"type":"block-end","index":1,"block":{"type":"text","text":"DONE"}}}}
|
||||
{"type":"assistant/chunk","seq":89,"time":1784973852460,"data":{"turn":1,"step":2,"chunk":{"type":"usage","usage":{"inputTokens":97,"outputTokens":26,"cacheReadTokens":7808,"reasoningTokens":23}}}}
|
||||
{"type":"assistant/chunk","seq":90,"time":1784973852460,"data":{"turn":1,"step":2,"chunk":{"type":"finish","reason":{"kind":"stop"}}}}
|
||||
{"type":"assistant/message","seq":91,"time":1784973852461,"data":{"turn":1,"step":2,"content":[{"type":"reasoning","text":"The command executed successfully and output \"WEB_E2E_OK\". I just need to reply with \"DONE\"."},{"type":"text","text":"DONE"}],"provenance":{"provider":"deepseek","model":"deepseek-v4-flash"},"usage":{"inputTokens":97,"outputTokens":26,"cacheReadTokens":7808,"reasoningTokens":23}},"sourceEventSeqs":[60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90],"surfaceOp":"append"}
|
||||
{"type":"step/end","seq":92,"time":1784973852461,"data":{"turn":1,"step":2}}
|
||||
{"type":"turn/end","seq":93,"time":1784973852462,"data":{"turn":1,"reason":{"kind":"completed"}}}
|
||||
31
apps/web/tests/snapshots/fresh-round-trip/ui.expected.md
Normal file
31
apps/web/tests/snapshots/fresh-round-trip/ui.expected.md
Normal file
@@ -0,0 +1,31 @@
|
||||
- banner:
|
||||
- navigation "Session hierarchy":
|
||||
- button "Use the bash tool to" [disabled]
|
||||
- text: · 1 turns
|
||||
- tablist:
|
||||
- tab "Chat" [selected]
|
||||
- tab "Trajectory"
|
||||
- tab "Waterfall"
|
||||
- text: "Use the bash tool to run exactly: echo WEB_E2E_OK. Then reply with the single word DONE and stop."
|
||||
- button "Think The user wants me to run a simple bash command and reply with \"DONE\".":
|
||||
- img
|
||||
- text: Think The user wants me to run a simple bash command and reply with "DONE".
|
||||
- text: Echo the test string
|
||||
- button "Think The command executed successfully and output \"WEB_E2E_OK\". I just need to reply with \"DONE\".":
|
||||
- img
|
||||
- text: Think The command executed successfully and output "WEB_E2E_OK". I just need to reply with "DONE".
|
||||
- paragraph: DONE
|
||||
- text: cache hit 99% · 15,818 tokens · 1 turns · 2 steps
|
||||
- textbox "Message the agent"
|
||||
- button "Add attachment":
|
||||
- img
|
||||
- combobox "Plan mode":
|
||||
- option "Plan" [selected]
|
||||
- option "Agent"
|
||||
- combobox "Access mode":
|
||||
- option "Read-only" [selected]
|
||||
- option "Read-write"
|
||||
- combobox "Model":
|
||||
- option "DeepSeek-V4-Pro High" [selected]
|
||||
- option "DeepSeek-V4-Pro"
|
||||
- button "Send message" [disabled]
|
||||
112
apps/web/tests/snapshots/seeded-history/seed.jsonl
Normal file
112
apps/web/tests/snapshots/seeded-history/seed.jsonl
Normal file
@@ -0,0 +1,112 @@
|
||||
{"type":"session","version":0,"id":"{{sessionId}}","createdAt":1784974100747,"cwd":"{{cwd}}/workspace"}
|
||||
{"type":"turn/start","seq":0,"time":1784974100758,"data":{"turn":1,"trigger":{"kind":"message","source":{"kind":"user","rpcId":"{{rpcId}}"}}}}
|
||||
{"type":"user/message","seq":1,"time":1784974100759,"data":{"content":[{"type":"text","text":"Use the read tool twice in one assistant message: read a.txt and b.txt. Then reply with the single word DONE and stop."}],"source":{"kind":"user","rpcId":"{{rpcId}}"}},"surfaceOp":"append"}
|
||||
{"type":"session/title","seq":2,"time":1784974100761,"data":{"title":"Use the read tool twice","messageSeqs":[1],"source":{"kind":"fallback"}}}
|
||||
{"type":"step/start","seq":3,"time":1784974100827,"data":{"turn":1,"step":1}}
|
||||
{"type":"request/header","seq":4,"time":1784974100828,"data":{"header":{"config":{"provider":"deepseek","model":"deepseek-v4-flash"},"system":"{{system}}","tools":"{{tools}}","messagePrefix":["{{messagePrefix}}"]},"reason":"initial"}}
|
||||
{"type":"assistant/chunk","seq":5,"time":1784974101296,"data":{"turn":1,"step":1,"chunk":{"type":"block-start","index":0,"blockType":"reasoning"}}}
|
||||
{"type":"assistant/chunk","seq":6,"time":1784974101297,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":"The"}}}
|
||||
{"type":"assistant/chunk","seq":7,"time":1784974101422,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" user"}}}
|
||||
{"type":"assistant/chunk","seq":8,"time":1784974101452,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" wants"}}}
|
||||
{"type":"assistant/chunk","seq":9,"time":1784974101452,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" me"}}}
|
||||
{"type":"assistant/chunk","seq":10,"time":1784974101453,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" to"}}}
|
||||
{"type":"assistant/chunk","seq":11,"time":1784974101453,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" read"}}}
|
||||
{"type":"assistant/chunk","seq":12,"time":1784974101453,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" a"}}}
|
||||
{"type":"assistant/chunk","seq":13,"time":1784974101483,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":".txt"}}}
|
||||
{"type":"assistant/chunk","seq":14,"time":1784974101484,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" and"}}}
|
||||
{"type":"assistant/chunk","seq":15,"time":1784974101484,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" b"}}}
|
||||
{"type":"assistant/chunk","seq":16,"time":1784974101484,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":".txt"}}}
|
||||
{"type":"assistant/chunk","seq":17,"time":1784974101484,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":","}}}
|
||||
{"type":"assistant/chunk","seq":18,"time":1784974101484,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" then"}}}
|
||||
{"type":"assistant/chunk","seq":19,"time":1784974101514,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" reply"}}}
|
||||
{"type":"assistant/chunk","seq":20,"time":1784974101515,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" with"}}}
|
||||
{"type":"assistant/chunk","seq":21,"time":1784974101515,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" \""}}}
|
||||
{"type":"assistant/chunk","seq":22,"time":1784974101515,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":"D"}}}
|
||||
{"type":"assistant/chunk","seq":23,"time":1784974101545,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":"ONE"}}}
|
||||
{"type":"assistant/chunk","seq":24,"time":1784974101545,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":"\"."}}}
|
||||
{"type":"assistant/chunk","seq":25,"time":1784974101545,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" Let"}}}
|
||||
{"type":"assistant/chunk","seq":26,"time":1784974101545,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" me"}}}
|
||||
{"type":"assistant/chunk","seq":27,"time":1784974101545,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" do"}}}
|
||||
{"type":"assistant/chunk","seq":28,"time":1784974101546,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" both"}}}
|
||||
{"type":"assistant/chunk","seq":29,"time":1784974101576,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" reads"}}}
|
||||
{"type":"assistant/chunk","seq":30,"time":1784974101577,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" in"}}}
|
||||
{"type":"assistant/chunk","seq":31,"time":1784974101577,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":" parallel"}}}
|
||||
{"type":"assistant/chunk","seq":32,"time":1784974101577,"data":{"turn":1,"step":1,"chunk":{"type":"reasoning-delta","index":0,"text":"."}}}
|
||||
{"type":"assistant/chunk","seq":33,"time":1784974101666,"data":{"turn":1,"step":1,"chunk":{"type":"block-start","index":1,"blockType":"tool-call"}}}
|
||||
{"type":"assistant/chunk","seq":34,"time":1784974101667,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_OsndvlcKnCcUmae7QXal8633","name":"read","argumentsDelta":""}}}
|
||||
{"type":"assistant/chunk","seq":35,"time":1784974101697,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_OsndvlcKnCcUmae7QXal8633","name":"read","argumentsDelta":"{"}}}
|
||||
{"type":"assistant/chunk","seq":36,"time":1784974101697,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_OsndvlcKnCcUmae7QXal8633","name":"read","argumentsDelta":"\""}}}
|
||||
{"type":"assistant/chunk","seq":37,"time":1784974101697,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_OsndvlcKnCcUmae7QXal8633","name":"read","argumentsDelta":"file"}}}
|
||||
{"type":"assistant/chunk","seq":38,"time":1784974101697,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_OsndvlcKnCcUmae7QXal8633","name":"read","argumentsDelta":"_path"}}}
|
||||
{"type":"assistant/chunk","seq":39,"time":1784974101697,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_OsndvlcKnCcUmae7QXal8633","name":"read","argumentsDelta":"\""}}}
|
||||
{"type":"assistant/chunk","seq":40,"time":1784974101697,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_OsndvlcKnCcUmae7QXal8633","name":"read","argumentsDelta":": "}}}
|
||||
{"type":"assistant/chunk","seq":41,"time":1784974101726,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_OsndvlcKnCcUmae7QXal8633","name":"read","argumentsDelta":"\""}}}
|
||||
{"type":"assistant/chunk","seq":42,"time":1784974101727,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_OsndvlcKnCcUmae7QXal8633","name":"read","argumentsDelta":"a"}}}
|
||||
{"type":"assistant/chunk","seq":43,"time":1784974101727,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_OsndvlcKnCcUmae7QXal8633","name":"read","argumentsDelta":".txt"}}}
|
||||
{"type":"assistant/chunk","seq":44,"time":1784974101756,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_OsndvlcKnCcUmae7QXal8633","name":"read","argumentsDelta":"\""}}}
|
||||
{"type":"assistant/chunk","seq":45,"time":1784974101757,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":1,"id":"call_00_OsndvlcKnCcUmae7QXal8633","name":"read","argumentsDelta":"}"}}}
|
||||
{"type":"assistant/chunk","seq":46,"time":1784974101821,"data":{"turn":1,"step":1,"chunk":{"type":"block-start","index":2,"blockType":"tool-call"}}}
|
||||
{"type":"assistant/chunk","seq":47,"time":1784974101822,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":2,"id":"call_01_Hw6AQjhf9gjxnOtppcGx0725","name":"read","argumentsDelta":""}}}
|
||||
{"type":"assistant/chunk","seq":48,"time":1784974101849,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":2,"id":"call_01_Hw6AQjhf9gjxnOtppcGx0725","name":"read","argumentsDelta":"{"}}}
|
||||
{"type":"assistant/chunk","seq":49,"time":1784974101849,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":2,"id":"call_01_Hw6AQjhf9gjxnOtppcGx0725","name":"read","argumentsDelta":"\""}}}
|
||||
{"type":"assistant/chunk","seq":50,"time":1784974101849,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":2,"id":"call_01_Hw6AQjhf9gjxnOtppcGx0725","name":"read","argumentsDelta":"file"}}}
|
||||
{"type":"assistant/chunk","seq":51,"time":1784974101849,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":2,"id":"call_01_Hw6AQjhf9gjxnOtppcGx0725","name":"read","argumentsDelta":"_path"}}}
|
||||
{"type":"assistant/chunk","seq":52,"time":1784974101850,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":2,"id":"call_01_Hw6AQjhf9gjxnOtppcGx0725","name":"read","argumentsDelta":"\""}}}
|
||||
{"type":"assistant/chunk","seq":53,"time":1784974101881,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":2,"id":"call_01_Hw6AQjhf9gjxnOtppcGx0725","name":"read","argumentsDelta":": "}}}
|
||||
{"type":"assistant/chunk","seq":54,"time":1784974101881,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":2,"id":"call_01_Hw6AQjhf9gjxnOtppcGx0725","name":"read","argumentsDelta":"\""}}}
|
||||
{"type":"assistant/chunk","seq":55,"time":1784974101882,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":2,"id":"call_01_Hw6AQjhf9gjxnOtppcGx0725","name":"read","argumentsDelta":"b"}}}
|
||||
{"type":"assistant/chunk","seq":56,"time":1784974101882,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":2,"id":"call_01_Hw6AQjhf9gjxnOtppcGx0725","name":"read","argumentsDelta":".txt"}}}
|
||||
{"type":"assistant/chunk","seq":57,"time":1784974101908,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":2,"id":"call_01_Hw6AQjhf9gjxnOtppcGx0725","name":"read","argumentsDelta":"\""}}}
|
||||
{"type":"assistant/chunk","seq":58,"time":1784974101909,"data":{"turn":1,"step":1,"chunk":{"type":"tool-call-delta","index":2,"id":"call_01_Hw6AQjhf9gjxnOtppcGx0725","name":"read","argumentsDelta":"}"}}}
|
||||
{"type":"assistant/chunk","seq":59,"time":1784974101974,"data":{"turn":1,"step":1,"chunk":{"type":"block-end","index":0,"block":{"type":"reasoning","text":"The user wants me to read a.txt and b.txt, then reply with \"DONE\". Let me do both reads in parallel."}}}}
|
||||
{"type":"assistant/chunk","seq":60,"time":1784974101974,"data":{"turn":1,"step":1,"chunk":{"type":"block-end","index":1,"block":{"type":"tool-call","id":"call_00_OsndvlcKnCcUmae7QXal8633","name":"read","arguments":"{\"file_path\": \"a.txt\"}"}}}}
|
||||
{"type":"assistant/chunk","seq":61,"time":1784974101974,"data":{"turn":1,"step":1,"chunk":{"type":"block-end","index":2,"block":{"type":"tool-call","id":"call_01_Hw6AQjhf9gjxnOtppcGx0725","name":"read","arguments":"{\"file_path\": \"b.txt\"}"}}}}
|
||||
{"type":"assistant/chunk","seq":62,"time":1784974101974,"data":{"turn":1,"step":1,"chunk":{"type":"usage","usage":{"inputTokens":124,"outputTokens":103,"cacheReadTokens":7680,"reasoningTokens":27}}}}
|
||||
{"type":"assistant/chunk","seq":63,"time":1784974101974,"data":{"turn":1,"step":1,"chunk":{"type":"finish","reason":{"kind":"tool-calls"}}}}
|
||||
{"type":"assistant/message","seq":64,"time":1784974101978,"data":{"turn":1,"step":1,"content":[{"type":"reasoning","text":"The user wants me to read a.txt and b.txt, then reply with \"DONE\". Let me do both reads in parallel."},{"type":"tool-call","id":"call_00_OsndvlcKnCcUmae7QXal8633","name":"read","arguments":"{\"file_path\": \"a.txt\"}"},{"type":"tool-call","id":"call_01_Hw6AQjhf9gjxnOtppcGx0725","name":"read","arguments":"{\"file_path\": \"b.txt\"}"}],"provenance":{"provider":"deepseek","model":"deepseek-v4-flash"},"usage":{"inputTokens":124,"outputTokens":103,"cacheReadTokens":7680,"reasoningTokens":27}},"sourceEventSeqs":[5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63],"surfaceOp":"append"}
|
||||
{"type":"tool/call","seq":65,"time":1784974101979,"data":{"turn":1,"step":1,"callId":"call_00_OsndvlcKnCcUmae7QXal8633","name":"read","arguments":"{\"file_path\": \"a.txt\"}"}}
|
||||
{"type":"tool/call","seq":66,"time":1784974101981,"data":{"turn":1,"step":1,"callId":"call_01_Hw6AQjhf9gjxnOtppcGx0725","name":"read","arguments":"{\"file_path\": \"b.txt\"}"}}
|
||||
{"type":"tool/result","seq":67,"time":1784974101985,"data":{"turn":1,"step":1,"callId":"call_00_OsndvlcKnCcUmae7QXal8633","content":[{"type":"text","text":"<path>{{cwd}}/workspace/a.txt</path>\n<type>file</type>\n<content>\n1: alpha\n\n(End of file - total 1 lines)\n</content>"}],"isError":false},"sourceEventSeqs":[65],"surfaceOp":"append"}
|
||||
{"type":"tool/result","seq":68,"time":1784974101986,"data":{"turn":1,"step":1,"callId":"call_01_Hw6AQjhf9gjxnOtppcGx0725","content":[{"type":"text","text":"<path>{{cwd}}/workspace/b.txt</path>\n<type>file</type>\n<content>\n1: beta\n\n(End of file - total 1 lines)\n</content>"}],"isError":false},"sourceEventSeqs":[66],"surfaceOp":"append"}
|
||||
{"type":"step/end","seq":69,"time":1784974101988,"data":{"turn":1,"step":1}}
|
||||
{"type":"step/start","seq":70,"time":1784974101988,"data":{"turn":1,"step":2}}
|
||||
{"type":"assistant/chunk","seq":71,"time":1784974102397,"data":{"turn":1,"step":2,"chunk":{"type":"block-start","index":0,"blockType":"reasoning"}}}
|
||||
{"type":"assistant/chunk","seq":72,"time":1784974102397,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"Both"}}}
|
||||
{"type":"assistant/chunk","seq":73,"time":1784974102505,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" files"}}}
|
||||
{"type":"assistant/chunk","seq":74,"time":1784974102534,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" have"}}}
|
||||
{"type":"assistant/chunk","seq":75,"time":1784974102535,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" been"}}}
|
||||
{"type":"assistant/chunk","seq":76,"time":1784974102535,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" read"}}}
|
||||
{"type":"assistant/chunk","seq":77,"time":1784974102535,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"."}}}
|
||||
{"type":"assistant/chunk","seq":78,"time":1784974102565,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" a"}}}
|
||||
{"type":"assistant/chunk","seq":79,"time":1784974102595,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":".txt"}}}
|
||||
{"type":"assistant/chunk","seq":80,"time":1784974102596,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" contains"}}}
|
||||
{"type":"assistant/chunk","seq":81,"time":1784974102596,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" \""}}}
|
||||
{"type":"assistant/chunk","seq":82,"time":1784974102596,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"alpha"}}}
|
||||
{"type":"assistant/chunk","seq":83,"time":1784974102596,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"\""}}}
|
||||
{"type":"assistant/chunk","seq":84,"time":1784974102625,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" and"}}}
|
||||
{"type":"assistant/chunk","seq":85,"time":1784974102625,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" b"}}}
|
||||
{"type":"assistant/chunk","seq":86,"time":1784974102625,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":".txt"}}}
|
||||
{"type":"assistant/chunk","seq":87,"time":1784974102625,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" contains"}}}
|
||||
{"type":"assistant/chunk","seq":88,"time":1784974102625,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" \""}}}
|
||||
{"type":"assistant/chunk","seq":89,"time":1784974102626,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"beta"}}}
|
||||
{"type":"assistant/chunk","seq":90,"time":1784974102656,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"\"."}}}
|
||||
{"type":"assistant/chunk","seq":91,"time":1784974102656,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" I"}}}
|
||||
{"type":"assistant/chunk","seq":92,"time":1784974102656,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"'ll"}}}
|
||||
{"type":"assistant/chunk","seq":93,"time":1784974102656,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" now"}}}
|
||||
{"type":"assistant/chunk","seq":94,"time":1784974102689,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" reply"}}}
|
||||
{"type":"assistant/chunk","seq":95,"time":1784974102690,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" with"}}}
|
||||
{"type":"assistant/chunk","seq":96,"time":1784974102690,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" D"}}}
|
||||
{"type":"assistant/chunk","seq":97,"time":1784974102716,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"ONE"}}}
|
||||
{"type":"assistant/chunk","seq":98,"time":1784974102717,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" as"}}}
|
||||
{"type":"assistant/chunk","seq":99,"time":1784974102748,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":" instructed"}}}
|
||||
{"type":"assistant/chunk","seq":100,"time":1784974102749,"data":{"turn":1,"step":2,"chunk":{"type":"reasoning-delta","index":0,"text":"."}}}
|
||||
{"type":"assistant/chunk","seq":101,"time":1784974102749,"data":{"turn":1,"step":2,"chunk":{"type":"block-start","index":1,"blockType":"text"}}}
|
||||
{"type":"assistant/chunk","seq":102,"time":1784974102749,"data":{"turn":1,"step":2,"chunk":{"type":"text-delta","index":1,"text":"D"}}}
|
||||
{"type":"assistant/chunk","seq":103,"time":1784974102749,"data":{"turn":1,"step":2,"chunk":{"type":"text-delta","index":1,"text":"ONE"}}}
|
||||
{"type":"assistant/chunk","seq":104,"time":1784974102749,"data":{"turn":1,"step":2,"chunk":{"type":"block-end","index":0,"block":{"type":"reasoning","text":"Both files have been read. a.txt contains \"alpha\" and b.txt contains \"beta\". I'll now reply with DONE as instructed."}}}}
|
||||
{"type":"assistant/chunk","seq":105,"time":1784974102749,"data":{"turn":1,"step":2,"chunk":{"type":"block-end","index":1,"block":{"type":"text","text":"DONE"}}}}
|
||||
{"type":"assistant/chunk","seq":106,"time":1784974102750,"data":{"turn":1,"step":2,"chunk":{"type":"usage","usage":{"inputTokens":215,"outputTokens":32,"cacheReadTokens":7808,"reasoningTokens":29}}}}
|
||||
{"type":"assistant/chunk","seq":107,"time":1784974102750,"data":{"turn":1,"step":2,"chunk":{"type":"finish","reason":{"kind":"stop"}}}}
|
||||
{"type":"assistant/message","seq":108,"time":1784974102750,"data":{"turn":1,"step":2,"content":[{"type":"reasoning","text":"Both files have been read. a.txt contains \"alpha\" and b.txt contains \"beta\". I'll now reply with DONE as instructed."},{"type":"text","text":"DONE"}],"provenance":{"provider":"deepseek","model":"deepseek-v4-flash"},"usage":{"inputTokens":215,"outputTokens":32,"cacheReadTokens":7808,"reasoningTokens":29}},"sourceEventSeqs":[71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107],"surfaceOp":"append"}
|
||||
{"type":"step/end","seq":109,"time":1784974102751,"data":{"turn":1,"step":2}}
|
||||
{"type":"turn/end","seq":110,"time":1784974102751,"data":{"turn":1,"reason":{"kind":"completed"}}}
|
||||
36
apps/web/tests/snapshots/seeded-history/ui.expected.md
Normal file
36
apps/web/tests/snapshots/seeded-history/ui.expected.md
Normal file
@@ -0,0 +1,36 @@
|
||||
- banner:
|
||||
- navigation "Session hierarchy":
|
||||
- button "Use the read tool twice" [disabled]
|
||||
- text: · 1 turns
|
||||
- tablist:
|
||||
- tab "Chat" [selected]
|
||||
- tab "Trajectory"
|
||||
- tab "Waterfall"
|
||||
- text: "Use the read tool twice in one assistant message: read a.txt and b.txt. Then reply with the single word DONE and stop."
|
||||
- button "Think The user wants me to read a.txt and b.txt, then reply with \"DONE\". Let me do both reads in parallel.":
|
||||
- img
|
||||
- text: Think The user wants me to read a.txt and b.txt, then reply with "DONE". Let me do both reads in parallel.
|
||||
- button:
|
||||
- img
|
||||
- text: Read a.txt
|
||||
- button:
|
||||
- img
|
||||
- text: Read b.txt
|
||||
- button "Think Both files have been read. a.txt contains \"alpha\" and b.txt contains \"beta\". I'll now reply with DONE as instructed.":
|
||||
- img
|
||||
- text: Think Both files have been read. a.txt contains "alpha" and b.txt contains "beta". I'll now reply with DONE as instructed.
|
||||
- paragraph: DONE
|
||||
- text: cache hit 98% · 15,962 tokens · 1 turns · 2 steps
|
||||
- textbox "Message the agent"
|
||||
- button "Add attachment":
|
||||
- img
|
||||
- combobox "Plan mode":
|
||||
- option "Plan" [selected]
|
||||
- option "Agent"
|
||||
- combobox "Access mode":
|
||||
- option "Read-only" [selected]
|
||||
- option "Read-write"
|
||||
- combobox "Model":
|
||||
- option "DeepSeek-V4-Pro High" [selected]
|
||||
- option "DeepSeek-V4-Pro"
|
||||
- button "Send message" [disabled]
|
||||
@@ -17,6 +17,15 @@
|
||||
"src",
|
||||
"tests"
|
||||
],
|
||||
// The web e2e lane (scaffold + replay specs) boots the host spine and reads
|
||||
// its Context merges — host-plane programs, checked in tsconfig.host.json;
|
||||
// this client-registered project must not also hold them (one program
|
||||
// cannot see both sides of the cordis Context merges).
|
||||
"exclude": [
|
||||
"tests/scaffold.ts",
|
||||
"tests/replay-round-trip.e2e.ts",
|
||||
"tests/seeded-history.e2e.ts"
|
||||
],
|
||||
"references": [
|
||||
{
|
||||
"path": "../../packages/client/web"
|
||||
|
||||
@@ -661,6 +661,8 @@ export interface Config {
|
||||
childFiles?: string[]
|
||||
/** Optional replay-only provider catalog; absent or empty selects catch-all waterfall replay. */
|
||||
providers?: ReplayProviderConfig[]
|
||||
/** Optional per-chunk pacing delay in ms (see {@link ReplayConfig.paceMs}); absent keeps burst yield. */
|
||||
paceMs?: number
|
||||
}
|
||||
|
||||
/** One provider route exposed by the replay adapter. */
|
||||
@@ -686,7 +688,7 @@ export interface ReplayModelConfig {
|
||||
}
|
||||
```
|
||||
|
||||
Source: [`packages/support/llm-replay/src/index.ts:392`](../packages/support/llm-replay/src/index.ts)
|
||||
Source: [`packages/support/llm-replay/src/index.ts:459`](../packages/support/llm-replay/src/index.ts)
|
||||
|
||||
## `@deepseek-ai/dsh-llm-retry`
|
||||
|
||||
|
||||
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
testing.md: ac5324c6ad2cc0718d9ad4f2d699120abb033374
|
||||
testing.zh.md: 54e7a91b69480893a871d73b95696b10ffd69926
|
||||
testing.md: 678d2e218590f70e6424a60286e46db87cf278cc
|
||||
testing.zh.md: 776f09bfe534f8460efda59623dc8f139cd43fe7
|
||||
|
||||
@@ -10,6 +10,7 @@ How this repo tests, tier by tier, and the rules that keep a green suite meaning
|
||||
- **Coverage gate** (`pnpm run test:coverage`): the gating run, per-file 100% on `packages/*/*/src`. An uncovered line is often dead code the gate is correctly flagging for deletion, not a missing test to bolt on. Line coverage is necessary, never sufficient — it proves lines ran, not that the feature works as shipped.
|
||||
- **Real-API e2e** (`pnpm run test:e2e`): with-key tests against live provider APIs — the DeepSeek model plus provider-specific smokes that gate on their own keys (`EXA_API_KEY`, `PERPLEXITY_API_KEY`, …); each suite self-skips without its key so keyless CI stays green ([real-API e2e Agent Note](../.agents/notes/implemented/testing/2026-06-19-real-api-e2e-ci.md)).
|
||||
- **Snapshot** (`pnpm run test:snapshot`): keyless expected outputs cover external behavior — transport contracts and presentation, while persisted logs pin assembled backend behavior. ACP boots the real automation-server example, replays a recorded session, and diffs normalized JSON-RPC plus the re-persisted log ([ACP snapshot Agent Note](../.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md)); headless pins `stream-json` through its real one-shot process. TUI journeys replay primary/child JSONL through the real loop and tools, then project ANSI into semantic terminal-state outputs; package snapshots retain transient states and a real PTY covers the process boundary ([TUI snapshot Agent Note](../.agents/notes/implemented/testing/2026-07-18-tui-terminal-state-snapshots.md)). Use `pnpm run test:snapshot:record` when a model transcript changes and `pnpm run test:snapshot:refresh` when replay input remains valid; review every JSONL and expected-output diff. One ACP scenario (`text-turn`) pins full system-prompt/tool-schema content; other fixtures tokenize it so an edit churns one line ([pinned-header Agent Note](../.agents/notes/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)).
|
||||
- **Web browser snapshot** (gate-exempt `pnpm run test:web`): real chromium over the in-process web composition replays recorded fixtures against conversation aria goldens (`apps/web/tests/snapshots/`); record/refresh semantics and the deferred CI browser decision: [web e2e lane Agent Note](../.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md).
|
||||
|
||||
## The with-key policy: inference is cheap here
|
||||
|
||||
@@ -43,4 +44,4 @@ An e2e assertion re-runs the command or re-reads the file externally; a keyword
|
||||
|
||||
## When a snapshot test is required
|
||||
|
||||
Every non-trivial model-, protocol-, or human-visible change adds or updates a keyless scenario in the same PR through a runnable example's owning snapshot suite. Package tests, e2e assertions, mock/test-only compositions, and PR rationale do not replace the assembled transcript; extend the harness when needed. ACP automation scenarios use `examples/<name>/tests/snapshots/`, a scenario table over the [`dsh-acp-snapshot`](../packages/support/acp-snapshot/README.md) suite factory (`examples/acp-agent` is primary); `examples/headless-agent` owns the `stream-json` snapshot and replay fixtures. Completed interactive-terminal journeys use JSONL-driven scenarios under `examples/tui-agent/tests/snapshots/`; transient presentation uses the package-local semantic matrix, with a PTY case when input, Loader selection, or terminal teardown changes. New capability seams, lifecycle shapes, or transcript surfaces name every coverage tier at plan time and verify the harness can express it before implementation.
|
||||
Every non-trivial model-, protocol-, or human-visible change adds or updates a keyless scenario in the same PR through a runnable example's owning snapshot suite. Package tests, e2e assertions, mock/test-only compositions, and PR rationale do not replace the assembled transcript; extend the harness when needed. ACP automation scenarios use `examples/<name>/tests/snapshots/`, a scenario table over the [`dsh-acp-snapshot`](../packages/support/acp-snapshot/README.md) suite factory (`examples/acp-agent` is primary); `examples/headless-agent` owns the `stream-json` snapshot and replay fixtures. Completed interactive-terminal journeys use JSONL-driven scenarios under `examples/tui-agent/tests/snapshots/`; transient presentation uses the package-local semantic matrix, with a PTY case when input, Loader selection, or terminal teardown changes. Browser-rendered web GUI journeys use `apps/web/tests/snapshots/`. New capability seams, lifecycle shapes, or transcript surfaces name every coverage tier at plan time and verify the harness can express it before implementation.
|
||||
|
||||
@@ -10,6 +10,7 @@
|
||||
- **覆盖率门禁**（`pnpm run test:coverage`）：门禁级运行，对 `packages/*/*/src` 按文件 100% 覆盖。未覆盖的行往往是门禁正确标记出的死代码（应删除），而非需要补写的测试。行覆盖率是必要条件，但永远不是充分条件：它证明行被执行过，不证明功能按交付预期工作。
|
||||
- **真实 API e2e**（`pnpm run test:e2e`）：带密钥测试调用真实提供方 API，包括 DeepSeek 模型以及各提供方特有的冒烟测试；这些测试各自由自己的密钥控制（`EXA_API_KEY`、`PERPLEXITY_API_KEY` 等），缺少密钥时套件会自动跳过，使 keyless CI 保持绿色（[真实 API e2e Agent Note](../.agents/notes/implemented/testing/2026-06-19-real-api-e2e-ci.md)）。
|
||||
- **快照**（`pnpm run test:snapshot`）：无密钥预期输出覆盖对外行为（传输契约与呈现），持久化日志则固定组装后的后端行为。ACP 启动真实的自动化服务器示例、回放录制会话，并对归一化 JSON-RPC 与重新持久化的日志执行 diff（[ACP 快照 Agent Note](../.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md)）；headless 通过真实单次运行进程固定 `stream-json`。TUI 旅程通过真实循环与工具回放主会话与子会话 JSONL，再将 ANSI 投影为语义化终端状态输出；包级快照保留瞬态状态，真实 PTY 覆盖进程边界（[TUI 快照 Agent Note](../.agents/notes/implemented/testing/2026-07-18-tui-terminal-state-snapshots.md)）。当模型 transcript（文本记录）发生变化时使用 `pnpm run test:snapshot:record`，回放输入仍然有效时使用 `pnpm run test:snapshot:refresh`；请审查每一处 JSONL 与预期输出差异。一个 ACP 场景（`text-turn`）固定完整的系统提示词与工具 schema 内容；其他 fixture（测试前置数据）将其 token 化，因此修改只会扰动一行（[pinned-header Agent Note](../.agents/notes/implemented/testing/2026-07-06-pin-request-header-content-in-one-scenario.md)）。
|
||||
- **Web 浏览器快照**（豁免门禁的 `pnpm run test:web`）：真实 chromium 在进程内 web 组装之上回放已录制 fixture，与会话区 aria 预期输出比对（`apps/web/tests/snapshots/`）；`DSH_SNAPSHOT=record`/`refresh` 的语义与暂缓的 CI 浏览器决策见 [web e2e 车道 Agent Note](../.agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md)。
|
||||
|
||||
## 带密钥策略：推理在这里很便宜
|
||||
|
||||
|
||||
@@ -65,7 +65,7 @@ The GUI test structure (three tiers, lane map) is settled in the [GUI testing sy
|
||||
Run the narrowest rung that covers what you touched; escalate only when the change surface demands it.
|
||||
|
||||
1. **Every GUI code change** — `pnpm run test:gui` (seconds; no browser, no server): the client suites plus the host-side GUI packages. This is the inner loop; run it as freely as a typecheck.
|
||||
2. **Changes to the build surface, boot wiring, or static serving** (`apps/web`, vite config, `dsh-host-webserver`) — additionally `pnpm run test:web`: rebuilds the frontend dist, then runs the browser smoke pair (the real-host case self-skips without `DEEPSEEK_API_KEY`).
|
||||
2. **Changes to the build surface, boot wiring, static serving, or the wire carriage** (`apps/web`, vite config, `dsh-host-webserver`, connection/handler/SSE) — additionally `pnpm run test:web`: rebuilds the frontend dist, then runs the browser smoke pair (the real-host case self-skips without `DEEPSEEK_API_KEY`) plus the keyless replayed e2e scenarios (`DSH_SNAPSHOT=refresh` rewrites their aria goldens after an intentional conversation-UI change; `DSH_SNAPSHOT=record` re-records fixtures with a key).
|
||||
3. **Before a PR** — `pnpm run check:pre-push` (the repo-wide gate ladder). Between PR windows this rung is not expected on every commit.
|
||||
|
||||
If `test:gui` is red on code you did not touch, neither silently fix nor ignore it: note it in your handoff so it lands in the next PR window's sweep.
|
||||
|
||||
@@ -53,7 +53,7 @@ Every scenario compares `stdout.expected.jsonl` with cwd-rooted separators canon
|
||||
|
||||
The example also ships a `cordis.snapshot.yml` replay overlay next to its `cordis.yml` (the bin swaps them under `DSH_SNAPSHOT=replay` — [single-source replay config Agent Note](../../../.agents/notes/implemented/testing/2026-07-04-single-source-acp-replay-config.md)); replay fixtures are served by [`dsh-llm-replay`](../llm-replay/README.md), which this package points at via the `DSH_SNAPSHOT_*` env vars it sets on the child. `pnpm run test:snapshot:record` calls the live LLM and rewrites the recorded scenarios' model fixtures; `pnpm run test:snapshot:refresh` stays keyless, runs the replay overlay, and rewrites stdout, comparable session-log expected outputs, and each pin's prompt and tool-schema sidecars from the committed model scripts. Fixture roles, record/replay/refresh semantics, and scenario-table fields are documented on `Scenario` and in the [snapshot Agent Note](../../../.agents/notes/implemented/testing/2026-06-19-acp-snapshot-tests.md).
|
||||
|
||||
Constraints: `suite.ts` imports vitest, so the package entry is importable only inside a vitest run (the launcher, harness, and normalizers have no such dependency but ship from the same entry). ACP-specific by design — the launcher speaks the SDK's `ClientSideConnection`. Input scripts cover initialization, fresh-session creation, text prompting, cancellation, expected RPC failures, and durable turn-boundary waits. Permission round-trips are a FIFO queue of option-kind selections (`allow_once`, `reject_once`, …) mapped to the agent-issued `optionId`; an absent or exhausted queue answers `cancelled`, and an unoffered kind rejects the run.
|
||||
Constraints: `suite.ts` imports vitest, so the package entry is importable only inside a vitest run (the launcher, harness, and normalizers have no such dependency but ship from the same entry). The launcher and suite factory are ACP-specific by design — the launcher speaks the SDK's `ClientSideConnection` — while the normalizers are transport-neutral session-log/text helpers also consumed by the TUI snapshot suite and the web browser e2e lane. Input scripts cover initialization, fresh-session creation, text prompting, cancellation, expected RPC failures, and durable turn-boundary waits. Permission round-trips are a FIFO queue of option-kind selections (`allow_once`, `reject_once`, …) mapped to the agent-issued `optionId`; an absent or exhausted queue answers `cancelled`, and an unoffered kind rejects the run.
|
||||
|
||||
## Model Experience
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
A replay LLM plugin for keyless snapshot tests. It yields model streams reconstructed from a recorded **session JSONL** fixture, so a test can boot the real agent against a fixed model transcript with no API key. With `providers` configured it registers a replay-only adapter whose catalog is available to scenarios that exercise model discovery; without `providers` it installs the catch-all `llm/stream` waterfall used by tests that do not need discovery.
|
||||
|
||||
Its consumers are the ACP snapshot harness in `examples/acp-agent` and the `stream-json` snapshot in `examples/headless-agent`; each loads this plugin in place of a real LLM adapter. Keeping derivation and replay here places that logic under the per-file 100% coverage gate on `packages/*/src`.
|
||||
Its consumers are the ACP, headless `stream-json`, and TUI snapshot suites plus the web browser e2e lane. Loader-driven suites mount this plugin in place of a real LLM adapter; the web lane installs it directly to retain the teardown consumption handle. Keeping derivation and replay here places that logic under the per-file 100% coverage gate on `packages/*/src`.
|
||||
|
||||
## How the fixture works
|
||||
|
||||
@@ -24,6 +24,7 @@ Replay keys every call by its calling session id (`GenerateOptions.sessionId`, s
|
||||
| `overrideFile` | string | `$DSH_SNAPSHOT_OVERRIDE` | Optional path to a `ReplayEntry[]` sidecar that replaces the PRIMARY session's derived script. |
|
||||
| `childFiles` | string[] | `$DSH_SNAPSHOT_CHILD_FILES` (path-delimited) | Recorded subagent child-session logs for a nested scenario; empty for a single-session scenario. |
|
||||
| `providers` | `ReplayProviderConfig[]` | — | Optional replay-only provider and model catalog. Each model may publish `contextWindow`; configured routes dispatch through the replay adapter and never perform provider I/O. |
|
||||
| `paceMs` | number | — (burst) | Optional per-chunk delay in ms so downstream transports (e.g. the web SSE mux observed by a real browser) see genuinely incremental delivery. A realism knob only — tests must not depend on it for correctness. Non-negative integer; abort during a pace wait cancels the stream promptly. |
|
||||
|
||||
```yaml
|
||||
- id: llm-replay
|
||||
@@ -43,11 +44,11 @@ Replay keys every call by its calling session id (`GenerateOptions.sessionId`, s
|
||||
|
||||
## Exports
|
||||
|
||||
- `installLlmReplay(ctx, config)` — install the configured replay adapter or catch-all `llm/stream` listener; returns the disposer (HMR safety). Use this in tests to drive replay without the Loader or env vars.
|
||||
- `installLlmReplay(ctx, config)` — install the configured replay adapter or catch-all `llm/stream` listener; returns a `ReplayHandle` (`dispose()` for HMR safety plus `assertConsumed()`, the teardown check that every recorded script bound to a live session and every bound cursor drained — turning a scenario that silently drove fewer model calls than recorded into a crisp diagnostic). Use this in tests to drive replay without the Loader or env vars.
|
||||
- `loadSessionScripts(config)` — resolve the ordered `SessionScript[]` (primary + children) for a scenario, ready to bind to live sessions in first-call order.
|
||||
- `loadReplayScript(config)` — resolve the `ReplayEntry[]` for the PRIMARY session only (sidecar override if present, else derived from the JSONL; fail-loud if the fixture is missing).
|
||||
- `deriveReplayScript(events)` / `parseSessionLog(text)` / `parseSessionHeader(text)` — the pure helpers that turn a recorded session log into a script and read its header `id`/`createdAt`. A derived group must end in a `finish` chunk; a group without one is the fingerprint of a thrown `stream()` and must instead be expressed via an override sidecar.
|
||||
- Types `ReplayEntry` / `SessionScript` / `ReplayConfig` / `ReplayProviderConfig` / `ReplayModelConfig` / `Config`.
|
||||
- Types `ReplayEntry` / `SessionScript` / `ReplayConfig` / `ReplayProviderConfig` / `ReplayModelConfig` / `ReplayHandle` / `Config`.
|
||||
|
||||
## Plugin export shape
|
||||
|
||||
|
||||
@@ -78,6 +78,32 @@ export interface ReplayConfig {
|
||||
* by tests that do not need discovery.
|
||||
*/
|
||||
providers?: ReplayProviderConfig[]
|
||||
/**
|
||||
* Optional per-chunk pacing delay in milliseconds: each replayed chunk waits
|
||||
* this long before yielding, so a downstream transport (e.g. the web SSE
|
||||
* mux observed by a browser) sees genuinely incremental delivery. A realism
|
||||
* knob only — correctness must never depend on it. Absent or `0` keeps
|
||||
* today's synchronous burst yield. Must be a non-negative finite integer;
|
||||
* aborting mid-wait cancels the stream like any other abort.
|
||||
*/
|
||||
paceMs?: number
|
||||
}
|
||||
|
||||
/**
|
||||
* Handle returned by {@link installLlmReplay}: removal plus the end-of-run
|
||||
* consumption check that turns silent fixture underruns (a scenario that
|
||||
* issued fewer calls than recorded, or never bound a recorded child script)
|
||||
* into a crisp diagnostic at teardown.
|
||||
*/
|
||||
export interface ReplayHandle {
|
||||
/** Remove the registered adapter or waterfall listener (HMR safety). Freestanding closure — safe to destructure. */
|
||||
dispose(this: void): void
|
||||
/**
|
||||
* Throw unless every recorded script was bound to a live session and every
|
||||
* bound cursor consumed its full entry list. Call at scenario teardown.
|
||||
* Freestanding closure — safe to destructure.
|
||||
*/
|
||||
assertConsumed(this: void): void
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -281,12 +307,32 @@ class ReplayAdapter extends LlmAdapter {
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Wait `paceMs` between chunk yields, aborting the wait (and the stream) the
|
||||
* moment the signal fires — a paced replay must cancel as promptly as a burst
|
||||
* one.
|
||||
*/
|
||||
function paceDelay(paceMs: number, signal: AbortSignal | undefined): Promise<void> {
|
||||
return new Promise<void>((resolve, reject) => {
|
||||
const timer = setTimeout(() => {
|
||||
signal?.removeEventListener('abort', onAbort)
|
||||
resolve()
|
||||
}, paceMs)
|
||||
const onAbort = (): void => {
|
||||
clearTimeout(timer)
|
||||
reject(new Error('aborted'))
|
||||
}
|
||||
signal?.addEventListener('abort', onAbort, { once: true })
|
||||
})
|
||||
}
|
||||
|
||||
/** Yield a recorded stream back, honoring abort like a real adapter. */
|
||||
async function* replayEntry(entry: ReplayEntry, signal: AbortSignal | undefined): AsyncIterable<StreamChunk> {
|
||||
async function* replayEntry(entry: ReplayEntry, signal: AbortSignal | undefined, paceMs: number): AsyncIterable<StreamChunk> {
|
||||
switch (entry.kind) {
|
||||
case 'chunks':
|
||||
for (const chunk of entry.chunks) {
|
||||
if (signal?.aborted) throw new Error('aborted')
|
||||
if (paceMs > 0) await paceDelay(paceMs, signal)
|
||||
yield chunk
|
||||
}
|
||||
return
|
||||
@@ -297,6 +343,7 @@ async function* replayEntry(entry: ReplayEntry, signal: AbortSignal | undefined)
|
||||
// mid-stream STREAM_CLOSED after partial chunks).
|
||||
for (const chunk of entry.chunks) {
|
||||
if (signal?.aborted) throw new Error('aborted')
|
||||
if (paceMs > 0) await paceDelay(paceMs, signal)
|
||||
yield chunk
|
||||
}
|
||||
throw new LlmError(entry.message, entry.code)
|
||||
@@ -324,14 +371,17 @@ async function* replayEntry(entry: ReplayEntry, signal: AbortSignal | undefined)
|
||||
* next ordered recorded script, then advances its own cursor synchronously at
|
||||
* invocation time; calls without `sessionId` share one anonymous session. A
|
||||
* non-empty provider catalog registers a routed replay adapter; otherwise a
|
||||
* catch-all waterfall intercepts requests. Returns the effect disposer for
|
||||
* HMR-safe removal.
|
||||
* catch-all waterfall intercepts requests.
|
||||
*
|
||||
* @param ctx - the context whose LLM service receives the replay route or waterfall.
|
||||
* @param config - the resolved fixture paths (env-var defaulting is `apply`'s job).
|
||||
* @returns the disposer that removes the registered adapter or listener.
|
||||
* @returns the {@link ReplayHandle} carrying the disposer and the teardown consumption check.
|
||||
*/
|
||||
export function installLlmReplay(ctx: Context, config: ReplayConfig): () => void {
|
||||
export function installLlmReplay(ctx: Context, config: ReplayConfig): ReplayHandle {
|
||||
const paceMs = config.paceMs ?? 0
|
||||
if (!Number.isInteger(paceMs) || paceMs < 0) {
|
||||
throw new Error(`llm-replay: paceMs must be a non-negative integer, got ${String(config.paceMs)}`)
|
||||
}
|
||||
const scripts = loadSessionScripts(config)
|
||||
// Live-session → its bound script + cursor. A new live session id claims the
|
||||
// next not-yet-bound script (scripts are in bind order); `nextScript` is the
|
||||
@@ -375,14 +425,31 @@ export function installLlmReplay(ctx: Context, config: ReplayConfig): () => void
|
||||
+ `but its script has only ${boundState.entries.length}; re-record the scenario`,
|
||||
)
|
||||
}
|
||||
yield* replayEntry(entry, options.signal)
|
||||
yield* replayEntry(entry, options.signal, paceMs)
|
||||
})()
|
||||
}
|
||||
const providers = config.providers ?? []
|
||||
if (providers.length > 0) {
|
||||
return ctx.llm.registerAdapter(providers.map(provider => provider.id), new ReplayAdapter(providers, replay))
|
||||
const dispose = providers.length > 0
|
||||
? ctx.llm.registerAdapter(providers.map(provider => provider.id), new ReplayAdapter(providers, replay))
|
||||
: ctx.on('llm/stream', (options: GenerateOptions, _next) => replay(options))
|
||||
return {
|
||||
dispose,
|
||||
assertConsumed(): void {
|
||||
const problems: string[] = []
|
||||
if (nextScript < scripts.length) {
|
||||
problems.push(`${scripts.length - nextScript} recorded script(s) never bound to a live session`)
|
||||
}
|
||||
for (const [key, state] of bound) {
|
||||
if (state.cursor < state.entries.length) {
|
||||
const who = key === ANON ? 'the anonymous session' : `session ${key}`
|
||||
problems.push(`${who} consumed ${state.cursor}/${state.entries.length} recorded call(s)`)
|
||||
}
|
||||
}
|
||||
if (problems.length > 0) {
|
||||
throw new Error(`llm-replay: fixture not fully consumed — ${problems.join('; ')}; the scenario drove fewer model calls than recorded`)
|
||||
}
|
||||
},
|
||||
}
|
||||
return ctx.on('llm/stream', (options: GenerateOptions, _next) => replay(options))
|
||||
}
|
||||
|
||||
export const name = 'llm-replay'
|
||||
@@ -402,6 +469,8 @@ export interface Config {
|
||||
childFiles?: string[]
|
||||
/** Optional replay-only provider catalog; absent or empty selects catch-all waterfall replay. */
|
||||
providers?: ReplayProviderConfig[]
|
||||
/** Optional per-chunk pacing delay in ms (see {@link ReplayConfig.paceMs}); absent keeps burst yield. */
|
||||
paceMs?: number
|
||||
}
|
||||
|
||||
export function apply(ctx: Context, config: Config = {}): void {
|
||||
@@ -418,5 +487,6 @@ export function apply(ctx: Context, config: Config = {}): void {
|
||||
...overrideFile !== undefined && overrideFile.length > 0 ? { overrideFile } : {},
|
||||
...childFiles.length > 0 ? { childFiles } : {},
|
||||
...config.providers !== undefined ? { providers: config.providers } : {},
|
||||
...config.paceMs !== undefined ? { paceMs: config.paceMs } : {},
|
||||
})
|
||||
}
|
||||
|
||||
@@ -234,7 +234,7 @@ describe('installLlmReplay (through the real LlmService)', () => {
|
||||
writeLog(TEXT_CHUNKS)
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
const dispose = installLlmReplay(ctx, {
|
||||
const { dispose } = installLlmReplay(ctx, {
|
||||
file,
|
||||
providers: [
|
||||
{
|
||||
@@ -431,6 +431,92 @@ describe('installLlmReplay (through the real LlmService)', () => {
|
||||
await iterator.next()
|
||||
await expect(iterator.next()).rejects.toThrow('aborted')
|
||||
})
|
||||
|
||||
it('rejects a paceMs that is not a non-negative integer', async () => {
|
||||
writeLog(TEXT_CHUNKS)
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
expect(() => installLlmReplay(ctx, { file, paceMs: -1 })).toThrow(/paceMs/)
|
||||
expect(() => installLlmReplay(ctx, { file, paceMs: 1.5 })).toThrow(/paceMs/)
|
||||
})
|
||||
|
||||
it('paces chunk yields when paceMs is set (each chunk waits at least the pace)', async () => {
|
||||
writeLog(TEXT_CHUNKS)
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
installLlmReplay(ctx, { file, paceMs: 10 })
|
||||
const started = performance.now()
|
||||
const chunks = await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))
|
||||
expect(chunks).toEqual(TEXT_CHUNKS)
|
||||
// N chunks × 10ms; allow generous scheduling slack, assert the floor only.
|
||||
expect(performance.now() - started).toBeGreaterThanOrEqual(TEXT_CHUNKS.length * 10 - 5)
|
||||
})
|
||||
|
||||
it('aborting DURING a pace wait cancels the stream promptly', async () => {
|
||||
writeLog(TEXT_CHUNKS)
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
installLlmReplay(ctx, { file, paceMs: 60_000 })
|
||||
const controller = new AbortController()
|
||||
const pending = drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [], signal: controller.signal }))
|
||||
// Let the generator park inside the pace timer, then abort — the reject
|
||||
// must come from the abort listener, not the (distant) timer.
|
||||
await new Promise(r => setImmediate(r))
|
||||
controller.abort()
|
||||
await expect(pending).rejects.toThrow('aborted')
|
||||
})
|
||||
|
||||
it('assertConsumed passes only after every recorded call replayed', async () => {
|
||||
writeLog(TEXT_CHUNKS, TEXT_CHUNKS)
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
const handle = installLlmReplay(ctx, { file })
|
||||
await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))
|
||||
// One of two recorded calls consumed — the underrun must name the gap.
|
||||
expect(() => { handle.assertConsumed() }).toThrow(/consumed 1\/2 recorded call/)
|
||||
await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))
|
||||
expect(() => { handle.assertConsumed() }).not.toThrow()
|
||||
})
|
||||
|
||||
it('paces a throw-entry prefix too (the recorded partial streams at the same cadence)', async () => {
|
||||
writeFileSync(file, sessionJsonl([]), 'utf8')
|
||||
const overrideFile = join(dir, 'replay.override.json')
|
||||
const partial: StreamChunk[] = [{ type: 'block-start', index: 0, blockType: 'text' }]
|
||||
writeFileSync(overrideFile, JSON.stringify([
|
||||
{ kind: 'throw', chunks: partial, message: 'boom', code: 'STREAM_CLOSED' },
|
||||
]), 'utf8')
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
installLlmReplay(ctx, { file, overrideFile, paceMs: 10 })
|
||||
const started = performance.now()
|
||||
await expect(drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).rejects.toThrow('boom')
|
||||
expect(performance.now() - started).toBeGreaterThanOrEqual(5)
|
||||
})
|
||||
|
||||
it('assertConsumed names an underrunning identified session by its id', async () => {
|
||||
writeLog(TEXT_CHUNKS, TEXT_CHUNKS)
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
const handle = installLlmReplay(ctx, { file })
|
||||
const sessionId = 'live-underrun' as NonNullable<GenerateOptions['sessionId']>
|
||||
await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [], sessionId }))
|
||||
expect(() => { handle.assertConsumed() }).toThrow(/session live-underrun consumed 1\/2/)
|
||||
})
|
||||
|
||||
it('assertConsumed reports recorded scripts no live session ever bound', async () => {
|
||||
writeLog(TEXT_CHUNKS)
|
||||
const childFile = join(dir, 'session.1.jsonl')
|
||||
writeFileSync(childFile, sessionJsonl(
|
||||
TEXT_CHUNKS.map((chunk, i) => chunkEvent(i + 1, 1, 1, chunk)),
|
||||
{ id: 'child', createdAt: 10 },
|
||||
), 'utf8')
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
const handle = installLlmReplay(ctx, { file, childFiles: [childFile] })
|
||||
await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [], sessionId: 'live-parent' as NonNullable<GenerateOptions['sessionId']> }))
|
||||
// The child script never bound: the scenario drove fewer sessions than recorded.
|
||||
expect(() => { handle.assertConsumed() }).toThrow(/1 recorded script\(s\) never bound/)
|
||||
})
|
||||
})
|
||||
|
||||
describe('parseSessionHeader', () => {
|
||||
@@ -631,7 +717,7 @@ describe('apply (the plugin entry)', () => {
|
||||
writeFileSync(file, sessionJsonl(TEXT_CHUNKS.map((c, i) => chunkEvent(i + 1, 1, 1, c))), 'utf8')
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(LlmService)
|
||||
apply(ctx, { file, providers: [{ id: 'm', models: [{ id: 'm' }] }] })
|
||||
apply(ctx, { file, providers: [{ id: 'm', models: [{ id: 'm' }] }], paceMs: 1 })
|
||||
expect(ctx.llm.listProviders()).toEqual([{ id: 'm', name: 'm' }])
|
||||
expect(await drain(ctx.llm.stream({ provider: 'm', model: 'm', messages: [] }))).toEqual(TEXT_CHUNKS)
|
||||
})
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
"docs/architecture.md": 1800,
|
||||
"docs/cordis-primer.md": 600,
|
||||
"docs/defensive-patterns.md": 550,
|
||||
"docs/testing.md": 1020,
|
||||
"docs/testing.md": 1100,
|
||||
"examples/AGENTS.md": 310,
|
||||
"packages/AGENTS.md": 660,
|
||||
"packages/README.md": 790
|
||||
|
||||
@@ -8,6 +8,10 @@
|
||||
"rewriteRelativeImportExtensions": false
|
||||
},
|
||||
"include": [
|
||||
"apps/web/tests/scaffold.ts",
|
||||
"apps/web/tests/support.ts",
|
||||
"apps/web/tests/replay-round-trip.e2e.ts",
|
||||
"apps/web/tests/seeded-history.e2e.ts",
|
||||
"apps/cli/tests/**/*.ts",
|
||||
"examples/*/src/**/*.ts",
|
||||
"examples/*/start.ts",
|
||||
|
||||
@@ -1,10 +1,14 @@
|
||||
import tsconfigPaths from 'vite-tsconfig-paths'
|
||||
import { defineConfig } from 'vitest/config'
|
||||
|
||||
// Web smoke lane (GUI, gate-exempt — not part of the CI sequence yet): real
|
||||
// host entry points plus built-client interaction snapshots, outside the
|
||||
// unit/e2e includes. Real-model cases self-skip without DEEPSEEK_API_KEY;
|
||||
// fixture branches stay keyless and deterministic.
|
||||
// Web browser lane (GUI, gate-exempt — not part of the CI sequence yet): real
|
||||
// host entry points, built-client interaction snapshots, and the replayed
|
||||
// keyless e2e scenarios, outside the unit/e2e includes. Real-model cases
|
||||
// self-skip without DEEPSEEK_API_KEY; fixture branches and replay stay
|
||||
// keyless and deterministic.
|
||||
// TODO(ci-browser): running this lane in CI requires chromium provisioning
|
||||
// and reverses the no-browser-in-CI ruling — staged criteria in
|
||||
// .agents/notes/implemented/testing/2026-07-24-web-gui-browser-e2e-lane.md.
|
||||
try {
|
||||
// Node >= 21.7 native; throws when the file does not exist.
|
||||
process.loadEnvFile(new URL('.env', import.meta.url).pathname)
|
||||
|
||||
Reference in New Issue
Block a user