DeepSeek/deepseek-harness
1
0
Fork 0
mirror of https://github.com/deepseek-ai/deepseek-harness synced 2026-08-15 21:04:50 +00:00
Code Issues Packages Projects Releases Wiki Activity

test(feedback): fix the assembled e2e record path and teardown restore

Browse Source 
The feedback-command e2e now drives the recorded prompt through a separate
all-modes test that arms whenTurnSettled before sending and writes the
fixture back via recordFixture in record mode; the acknowledgement golden
test runs only in replay/refresh. The scaffold restores the pinned DSH_HOME
on the persistence-root setup failure path, and the telemetry subsystems
page links the README's sharing-disclosure anchor.
This commit is contained in:
Chinesezjc
2026-08-10 14:53:05 +08:00
parent 3f9d0436eb
commit 6a6148a08c
7 changed files with 76 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files

28
apps/web/tests/feedback-command.e2e.ts
View File

@@ -16,7 +16,7 @@ import { chromium } from 'playwright'
import { afterAll, beforeAll, describe, expect, it, onTestFailed } from 'vitest'
import {
assertFixtureInventory, captureStableAria, compareOrRefreshGolden, fixtureUserPrompts,
launchWebScaffold, watchConsole, webSnapshotMode, type WebScaffold,
launchWebScaffold, recordFixture, watchConsole, webSnapshotMode, type WebScaffold,
} from './scaffold.ts'
import { connectFreshWorkspace, newEnglishPage, saveFailureShot } from './support.ts'
@@ -56,20 +56,32 @@ describe('web e2e: /feedback command acknowledgement', () => {
await scaffold?.close()
})
it('records feedback and renders the acknowledgement with session id and sharing status', async () => {
onTestFailed(() => saveFailureShot(page, 'web-e2e-feedback-command'))
it('drives the recorded prompt to a settled turn (all modes)', async () => {
onTestFailed(() => saveFailureShot(page, 'web-e2e-feedback-drive'))
if (MODE !== 'record') {
// Drift guard: the committed fixture must carry exactly the drive prompt.
expect(fixtureUserPrompts(await readFile(FIXTURE, 'utf8'))).toEqual([PROMPT])
}
// First send the recorded prompt so the transcript is active — a command
// row does not render while a fresh session is still blank.
const input = page.locator('textarea').first()
await input.waitFor({ timeout: 10_000 })
// Arm the turn-boundary waiter BEFORE sending, so a burst replay cannot
// miss the turn/end that settles the recorded turn.
const settled = scaffold.whenTurnSettled()
await input.fill(PROMPT)
await input.press('Enter')
await scaffold.whenTurnSettled()
await page.getByText('LIGHTHOUSE', { exact: true }).waitFor({ timeout: 15_000 })
const sessionId = await settled
if (MODE === 'record') {
await recordFixture(scaffold, sessionId, FIXTURE)
}
}, 60_000)
it.skipIf(MODE === 'record')('records feedback and renders the acknowledgement with session id and sharing status', async () => {
onTestFailed(() => saveFailureShot(page, 'web-e2e-feedback-command'))
// The drive test settled the recorded turn: the transcript is active (a
// command row does not render while a fresh session is still blank) and
// the replayed reply is on screen.
await page.getByText('LIGHTHOUSE', { exact: true }).waitFor({ timeout: 15_000 })
const input = page.locator('textarea').first()
await input.fill('/feedback the diff view is unreadable')
await input.press('Enter')
// The command plane settles without a model turn: the ack row names the

2
apps/web/tests/snapshots/feedback-command/ack.expected.md
View File

@@ -1,6 +1,8 @@
- banner:
- navigation "Session hierarchy":
- button "Reply with the single word" [disabled]
- img
- text: 标准模式
- tablist:
- tab "Chat" [selected]
- tab "Trajectory"

4
docs/module-graph.i18n.yaml
View File

@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write docs/module-graph.md
module-graph.md: 2218d79e28e835ab96abce96eaf92bbae25e2182
module-graph.zh.md: 276b70d69c2898d74ac6897e398b02a8944fd503
module-graph.md: 8dea030a68f5dde3ce072a8f9ae7156ad162967b
module-graph.zh.md: a611ca8300f17b19c5d4f01032dc767dd04743c1

4
docs/subsystems/telemetry.i18n.yaml
View File

@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write docs/subsystems/telemetry.md
telemetry.md: 5ea5c67210ce1387cbd886935e914baf7f904fbb
telemetry.zh.md: bd8fc8acc4c8522d8b1e4bc543431c0abf224411
telemetry.md: 1b34f25361049611483ac9a10b2f90d7dac64439
telemetry.zh.md: 9d20831d74792944f5b17e43e6ed14f02dc00275

32
docs/subsystems/telemetry.md
View File

@@ -2,7 +2,7 @@
English | [中文](telemetry.zh.md)
Outbound session reporting is one [capability seam](../capability-seams.md): its Service Definition ([dsh-session-telemetry](../../packages/session/session-telemetry), `ctx.telemetry`) declares the minimal backend contract, and its capture coordinator owns the capture points, fixed chunk projection, `telemetry/record` redaction waterfall, and handoff cursor; the Service provider a deployment loads ([dsh-session-telemetry-otel](../../packages/session/session-telemetry-otel)) uses the OpenTelemetry JS SDK's log pipeline with its configuration unchanged. This optional capability is not part of the agent loop, and nothing here reaches a model request. The harness stops after it calls `emit()`; the reporting SDK owns batching, retry, queueing, and loss policy. The [revival Agent Note](../../.agents/notes/implemented/feature/2026-07-23-session-telemetry-otel-revival.md) records that rule and the rejected alternatives. The [Service Definition README](../../packages/session/session-telemetry/README.md) defines the capture-point, cursor, and projection contracts.
Outbound session reporting is split as a [capability seam](../capability-seams.md): the Service Definition and capture coordinator ([dsh-session-telemetry](../../packages/session/session-telemetry), `ctx.telemetry`) own the capture points, fixed chunk projection, `telemetry/record` redaction waterfall, handoff cursor, and minimal backend contract; the Service provider a deployment loads ([dsh-session-telemetry-otel](../../packages/session/session-telemetry-otel)) is the OpenTelemetry JS SDK's log pipeline configured verbatim. It is one optional capability, not part of the agent-loop spine, and nothing here reaches a model request. The boundary axiom — the harness's aspect ends at `emit()`; batching, retry, queueing, and loss policy belong to the reporting SDK — and the rejected alternatives are pinned in the [revival Agent Note](../../.agents/notes/implemented/feature/2026-07-23-session-telemetry-otel-revival.md); the capture points, cursor, and projection contracts live in the [Service Definition README](../../packages/session/session-telemetry/README.md).
Source: [`packages/session/session-telemetry/src/index.ts`](../../packages/session/session-telemetry/src/index.ts)
@@ -56,12 +56,28 @@ interface TelemetryRecord {
Only the first `assistant/chunk` of each `(turn, step)` ships — the stream-started signal; the rest drop at capture, so `seq` gaps are routine on the wire and never a loss signal. Every other [session event](session.md) type, including plugin-merged ones the seam never heard of, passes through whole. Delivery is best-effort: the cursor marks handed-off, not delivered, records can be lost (crash, reload window) and duplicated (cursor-less re-adoption, SDK retries), so receivers dedupe ledger records on `(session.id, event.seq)`; ops records deliberately omit that identity — they are signals to alert on, not entries to sum, and tolerate duplicates instead.
## The sharing disclosure
The seam's acknowledgement contract (owned by the [Service Definition README's sharing-disclosure section](../../packages/session/session-telemetry/README.md#the-sharing-disclosure)): every backend discloses its deployment-selected sharing policy through the required abstract `sharing` member on `ctx.telemetry`, and consumers render "not configured" only when no telemetry service is mounted. The disclosure states the current policy, never delivery or retention — handoff is the non-blocking enqueue, and batching, retry, and loss policy stay the reporting SDK's.
```ts type-equiv
/**
* Deployment-selected session-sharing policy disclosed by a mounted
* {@link Telemetry} backend to human-facing acknowledgement surfaces (the
* `/feedback` command's confirmation text). The seam owns the vocabulary so
* any backend can disclose a policy without depending on the OTel package;
* the values mirror the OTel backend's serialized `TelemetryMode` choices.
*/
type TelemetrySharingStatus = 'full' | 'feedback-only' | 'disabled'
```
## The backend contract
```ts type-equiv
/**
* The minimum backend contract the coordinator requires. {@link Telemetry} is
* its service-registered form; tests compose the coordinator with a bare
* The backend contract the coordinator hands records to — the minimum any
* reporting SDK satisfies with zero bending. {@link Telemetry} is its
* service-registered form; tests compose the coordinator with a bare
* implementation of this interface.
*/
interface TelemetryBackend {
@@ -76,8 +92,8 @@ interface TelemetryBackend {
*/
emit(record: TelemetryRecord): void
/**
* Optional hint that a turn ended. A backend may forward it to its SDK's
* flush so records are exported after each turn. Called
* Optional hint that a natural boundary (turn end) passed — a backend may
* forward it to its SDK's flush so records land at turn boundaries. Called
* fire-and-forget; implementations must not block and must not throw
* meaningfully (the coordinator contains exceptions). Most backends should
* leave this unimplemented and let their SDK's own batching cadence govern
@@ -104,7 +120,7 @@ interface TelemetryBackend {
}
```
`Telemetry` (`ctx.telemetry`, [signatures](#ctxtelemetry--telemetry-abstract-seam)) is the loadable form of this contract: each context accepts one implementation and throws on a duplicate. A backend constructs `TelemetryCoordinator` in its constructor to install capture.
`Telemetry` (`ctx.telemetry`, [signatures](#ctxtelemetry--telemetry-abstract-seam)) is the contract's loadable form — one implementation per context, duplicate load throws — and a backend composes the seam's `TelemetryCoordinator` in its constructor to install the capture side.
## The redact waterfall: `telemetry/record`
@@ -122,7 +138,7 @@ Generated from source by `scripts/gen-cordis-catalog.ts` (verified fresh by `pnp
### `ctx.telemetry` — `Telemetry` (abstract seam)
Loadable form of the backend contract: one implementation per context — the cordis `Service` registration under the `telemetry` key throws on a duplicate, cordis' standard behavior. A backend composes a TelemetryCoordinator in its constructor to install the capture side.
The backend contract in its loadable form: one implementation per context — the cordis `Service` registration under the `telemetry` key throws on a duplicate, cordis' standard behavior. A backend composes a TelemetryCoordinator in its constructor to install the capture side.
```ts cordis-catalog
/**
@@ -141,7 +157,7 @@ flush?(): void
abstract shutdown(): Promise<void>
```
Source: [`packages/session/session-telemetry/src/index.ts:139`](../../packages/session/session-telemetry/src/index.ts)
Source: [`packages/session/session-telemetry/src/index.ts:149`](../../packages/session/session-telemetry/src/index.ts)
<a id="telemetry-events"></a>

32
docs/subsystems/telemetry.zh.md
View File

@@ -2,7 +2,7 @@
[English](telemetry.md) | 中文
对外会话上报是一项[能力 seam](../capability-seams.md)：其 Service Definition（[dsh-session-telemetry](../../packages/session/session-telemetry)，`ctx.telemetry`）声明最小后端约定，其捕获协调器负责捕获点、固定分片投影、`telemetry/record` 脱敏 waterfall（瀑布式事件）和 handoff 游标；部署方加载的 Service provider（[dsh-session-telemetry-otel](../../packages/session/session-telemetry-otel)）按原配置使用 OpenTelemetry JS SDK 日志流水线。这项能力可选，不属于 agent loop（智能体循环），这里也没有任何内容会进入模型请求。Harness 调用 `emit()` 后停止处理；上报 SDK 负责批处理、重试、排队和丢失策略。[复活 Agent Note](../../.agents/notes/implemented/feature/2026-07-23-session-telemetry-otel-revival.md)记录了这条规则和被否决的替代方案。[Service Definition README](../../packages/session/session-telemetry/README.md) 定义捕获点、游标和投影约定。
对外的会话上报拆分为一项[能力 seam](../capability-seams.md)：Service Definition 与捕获协调器（[dsh-session-telemetry](../../packages/session/session-telemetry)，`ctx.telemetry`）拥有捕获点、固定分片投影、`telemetry/record` 脱敏 waterfall（瀑布式事件）、handoff 游标与最小后端约定；部署方加载的 Service provider（[dsh-session-telemetry-otel](../../packages/session/session-telemetry-otel)）则是原样配置的 OpenTelemetry JS SDK 日志流水线。它是一项可选能力，不属于 agent loop（智能体循环）主干，这里也没有任何内容会进入模型请求。边界公理（harness 的职责止于 `emit()`；批处理、重试、排队与丢失策略都属于上报 SDK）连同被否决的替代方案，均已在[复活 Agent Note（agent 决策记录）](../../.agents/notes/implemented/feature/2026-07-23-session-telemetry-otel-revival.md)中定案；捕获点、游标与投影的约定见 [Service Definition README](../../packages/session/session-telemetry/README.md)。
源码：[`packages/session/session-telemetry/src/index.ts`](../../packages/session/session-telemetry/src/index.ts)
@@ -56,12 +56,28 @@ interface TelemetryRecord {
每个 `(turn, step)` 只发出第一条 `assistant/chunk`，即「流已开始」的信号；其余分片在捕获时丢弃，因此导出流中的 `seq` 缺口是常态，绝不是丢失信号。其他所有[会话事件](session.md)类型都会完整透传，包括该 seam 从未听说过、由插件合并进来的事件类型。投递是尽力而为的：游标标记的是「已交接」而非「已送达」，记录可能丢失（崩溃、重载窗口）也可能重复（无游标的重新接管、SDK 重试），因此接收端对 ledger 记录基于 `(session.id, event.seq)` 去重；ops 记录刻意省略这类标识——它们是用于告警的信号，而非用于累加的条目，重复被容忍而非被去重。
## 共享披露
该 seam 的确认契约（归属 [Service Definition README 的共享披露段](../../packages/session/session-telemetry/README.md#the-sharing-disclosure)）：每个后端都通过 `ctx.telemetry` 上必需的抽象 `sharing` 成员披露其部署级共享策略，消费方只有在未挂载任何遥测服务时才渲染「未配置」。披露只陈述当前策略，绝不承诺投递或留存——交接是非阻塞入队，批处理、重试与丢失策略仍归上报 SDK。
```ts type-equiv
/**
* Deployment-selected session-sharing policy disclosed by a mounted
* {@link Telemetry} backend to human-facing acknowledgement surfaces (the
* `/feedback` command's confirmation text). The seam owns the vocabulary so
* any backend can disclose a policy without depending on the OTel package;
* the values mirror the OTel backend's serialized `TelemetryMode` choices.
*/
type TelemetrySharingStatus = 'full' | 'feedback-only' | 'disabled'
```
## 后端约定
```ts type-equiv
/**
* The minimum backend contract the coordinator requires. {@link Telemetry} is
* its service-registered form; tests compose the coordinator with a bare
* The backend contract the coordinator hands records to — the minimum any
* reporting SDK satisfies with zero bending. {@link Telemetry} is its
* service-registered form; tests compose the coordinator with a bare
* implementation of this interface.
*/
interface TelemetryBackend {
@@ -76,8 +92,8 @@ interface TelemetryBackend {
*/
emit(record: TelemetryRecord): void
/**
* Optional hint that a turn ended. A backend may forward it to its SDK's
* flush so records are exported after each turn. Called
* Optional hint that a natural boundary (turn end) passed — a backend may
* forward it to its SDK's flush so records land at turn boundaries. Called
* fire-and-forget; implementations must not block and must not throw
* meaningfully (the coordinator contains exceptions). Most backends should
* leave this unimplemented and let their SDK's own batching cadence govern
@@ -104,7 +120,7 @@ interface TelemetryBackend {
}
```
`Telemetry`（`ctx.telemetry`，[签名](#ctxtelemetry--telemetry-abstract-seam)）是该约定的可加载类型：每个上下文只允许一个实现，重复加载会抛出异常。后端在构造函数中创建 `TelemetryCoordinator`，以安装捕获处理。
`Telemetry`（`ctx.telemetry`，[签名](#ctxtelemetry--telemetry-abstract-seam)）是该约定的可加载形态：每个上下文只允许一个实现，重复加载会抛出异常；后端在其构造函数中组合 seam 的 `TelemetryCoordinator`，以此装配捕获侧。
## 脱敏 waterfall：`telemetry/record`
@@ -122,7 +138,7 @@ Generated from source by `scripts/gen-cordis-catalog.ts` (verified fresh by `pnp
### `ctx.telemetry` — `Telemetry` (abstract seam)
Loadable form of the backend contract: one implementation per context — the cordis `Service` registration under the `telemetry` key throws on a duplicate, cordis' standard behavior. A backend composes a TelemetryCoordinator in its constructor to install the capture side.
The backend contract in its loadable form: one implementation per context — the cordis `Service` registration under the `telemetry` key throws on a duplicate, cordis' standard behavior. A backend composes a TelemetryCoordinator in its constructor to install the capture side.
```ts cordis-catalog
/**
@@ -141,7 +157,7 @@ flush?(): void
abstract shutdown(): Promise<void>
```
Source: [`packages/session/session-telemetry/src/index.ts:139`](../../packages/session/session-telemetry/src/index.ts)
Source: [`packages/session/session-telemetry/src/index.ts:149`](../../packages/session/session-telemetry/src/index.ts)
<a id="telemetry-events"></a>

4
packages/feedback/command-feedback/README.i18n.yaml
View File

@@ -2,5 +2,5 @@
# side as of the last confirmed-consistent state. Both languages carry equal authority;
# after editing either side, bring the other along and re-record with:
# pnpm run verify-translation-pairing --write packages/feedback/command-feedback/README.md
README.md: 52b8fb6a423fca69f76397deec36ecd22a6a6023
README.zh.md: ca74d53f2531a46c2c16aa1423cee52e89c8256f
README.md: 6db5bc4b18815778628d5caa74b75c8b46d3f6a8
README.zh.md: d9ad1aba9bdbc04110430aa3e3f6604313b8ee20