fix: harden model experience documentation gate

This commit is contained in:
Tianyi Cui
2026-07-13 20:13:46 +08:00
parent 7195d9eda2
commit 3fc2a4ca82
7 changed files with 35 additions and 21 deletions

View File

@@ -15,7 +15,7 @@ Each fact has one owning tier; other tiers link to it. Restated rules drift, whi
| [rfc/](rfc/README.md) | Decision records: the why and the what-was-given-up; `implemented/` RFCs describe shipped reality in present tense | Migration plans, test checklists, and spec-speak ("should…") once the decision has shipped |
| [postmortem/](postmortem/README.md) | Incident stories — the only tier where war-story narrative belongs | — |
| [cookbook/](cookbook/adding-a-package.md) | Step-by-step how-tos with numbered verify steps | Design rationale (→ the RFC each guide links) |
| Package README | The per-package contract: config, semantics, limitations, extension points, and [Model Experience](#package-model-experience) | JSDoc restatement, generated-catalog restatement (event/tool tables), other packages' concerns |
| Package README | The per-package contract: config, semantics, limitations, extension points, and [Model Experience](cookbook/adding-a-package.md#4-write-the-package-readme) | JSDoc restatement, generated-catalog restatement (event/tool tables), other packages' concerns |
| [development.md](development.md) | First-stop contributor onboarding: local setup, daily workflow, and CI shape at summary level; a bilingual pair under the [i18n contract](i18n/README.md) | Runtime/version rationale (→ RFCs), gate-by-gate enumerations that drift from `package.json` scripts |
| Generated catalogs: [cordis events](cordis-catalog/events.md), [cordis services](cordis-catalog/services.md), [tool-catalog](tool-catalog.md), [config-catalog](config-catalog.md), [persistence-catalog](persistence-catalog.md), [module-graph.md](module-graph.md) | Exhaustive enumerations regenerated from source, freshness-gated | Hand edits of any kind |
| Skills (`.agents/skills/`) | Workflows: how to carry out a recurring task against the contracts | The contracts themselves (→ docs) |
@@ -28,20 +28,10 @@ Placement: bugs → postmortems; rationale → RFCs; procedures → cookbooks; t
- **A decision worth re-litigating gets an RFC in the same PR.** The test: would a maintainer six months out ask "why was it done this way?" and find no answer in the code? If yes, write one ([when to write one](rfc/README.md)); mechanical or self-evident changes need none.
- **One physical line per paragraph** (`verify-md-wrap`): the editor soft-wraps; hard breaks make a one-word edit re-diff the whole paragraph. Prose only — code blocks, tables, and list structure stay; code comments stay under the linter's column limit.
- **Fenced `ts` blocks must compile** (`doc-typecheck`); a pasted type definition is fenced ` ```ts type-equiv ` and registered in the manifest so it cannot drift ([mechanics](development.md#documenting-types-verbatim-ts-type-equiv)).
- **Every new event's JSDoc carries an `@mode` tag** (emit | waterfall | parallel | serial); the catalog generator hard-errors without it. Write the JSDoc to stand alone — it becomes the catalog entry ([catalog RFC](rfc/implemented/process/2026-06-20-generated-cordis-catalog.md)).
- **The [core-data-structures catalog](core-data-structures/core.md) updates in the same change** that reshapes a documented type. `verify-type-equiv` catches drifted pastes, not never-documented new types ([what counts as core](core-data-structures/core.md#what-counts-as-core)).
- **Bilingual pairs update together**: editing either side obligates the counterpart and a re-record in the same change ([i18n contract](i18n/README.md)).
- Your audience is professional programmers. Prefer concise and straight-forward English over metaphor. Do not overuse words like "gate", "vocabulary", "surface", "seams".
## Package Model Experience
Every package README ends with `## Model Experience` immediately before `## Known Limitations and Deferred Work`; [allowlisted no-limitations packages](../scripts/verify-readme-limitations.ts) end after Model Experience. Packages with direct, multi-surface, conditional, capped, or lifetime effects use this table:
| Context surface | What the model sees | Token effect |
|---|---|---|
Rows state what reaches which model and classify token cost or lifetime; prompt text and tool schemas stay separate when visibility differs. Packages explicitly classified in [`SENTENCE_MODEL_EXPERIENCE`](../scripts/verify-package-readme-model-experience.ts) instead carry one sentence beginning `None, as ` or `Indirectly, through ` and ending with a period. The verifier gates the sentence allowlist, table shape, and section order; review owns factual accuracy ([rationale](rfc/implemented/process/2026-07-12-package-model-experience-contract.md)).
## Wordcount Budgets
Every PR has a lesson it wants to append, and without pressure nothing leaves. [scripts/doc-budgets.manifest.json](../scripts/doc-budgets.manifest.json) stores the allowed word-count ceiling for each budgeted standing doc; `pnpm run verify-doc-budgets` fails when a doc exceeds its ceiling or a budgeted file is missing.

View File

@@ -49,16 +49,16 @@ Keep package-specific service API, config, events, extension points, and design
| Context surface | What the model sees | Token effect |
|---|---|---|
| Request surface and condition | Exact context visible to the model | Fixed, conditional, retained, replaced, capped, or zero-direct token effect |
| Request surface and condition | Verbatim stable text, or the exact data-dependent shape visible to the model | Fixed, conditional, retained, replaced, capped, or zero-direct token effect |
## Known Limitations and Deferred Work
- **Consumer-visible gap** — exact boundary or deliberately deferred work.
```
Fill [Model Experience](../AGENTS.md#package-model-experience) from the implementation. Direct, multi-surface, conditional, capped, or lifetime effects use the table; name each request contribution and token-growth condition. Do not infer prompt visibility from tool-schema visibility because independently registered guidance can remain after a scoped tool restriction.
Fill Model Experience from the implementation. Direct, multi-surface, conditional, capped, or lifetime effects use the table; name each request contribution and token-growth condition. Quote stable model-visible source literals verbatim in inline code, using named placeholders such as `<mode>` only for interpolated values. Summarize only data-dependent payloads, provider-owned text, or schemas too large to reproduce, and identify their exact shape and renderer. Do not infer prompt visibility from tool-schema visibility because independently registered guidance can remain after a scoped tool restriction.
An audited package with no context effect or one simple consumer-owned path joins [`SENTENCE_MODEL_EXPERIENCE`](../../scripts/verify-package-readme-model-experience.ts) and replaces the table with one line beginning `None, as ` or `Indirectly, through `. Every package outside that allowlist must keep the exact table. A package with genuinely no limitations joins the separate allowlist in [`verify-readme-limitations.ts`](../../scripts/verify-readme-limitations.ts) and ends after Model Experience.
An audited package with no context effect or one simple consumer-owned path joins [`SENTENCE_MODEL_EXPERIENCE`](../../scripts/verify-package-readme-model-experience.ts) and replaces the table with one line beginning `None, as ` or `Indirectly, through `. Every package outside that allowlist must keep the exact table. A package with genuinely no limitations joins the separate allowlist in [`verify-readme-limitations.ts`](../../scripts/verify-readme-limitations.ts) and ends after Model Experience. The [Model Experience RFC](../rfc/implemented/process/2026-07-12-package-model-experience-contract.md) records the rationale.
## 5. Verify

View File

@@ -8,7 +8,7 @@ A package README can explain APIs and runtime mechanics without answering the qu
## Decision
Every workspace package README ends with the canonical [Model Experience section](../../../AGENTS.md#package-model-experience), immediately before `## Known Limitations and Deferred Work`; a package on the no-limitations allowlist ends with Model Experience itself. Packages with direct, multi-surface, conditional, capped, or lifetime effects use the three-column table. Each row identifies a concrete request surface, says what the relevant model literally receives and when, and classifies the token effect. The default subject is the conversation model; a package that invokes an auxiliary model, such as a summarizer or search provider, names that request separately. Agent-scoped visibility is stated where it changes which agent receives a contribution. Prompt text and tool schemas are described separately whenever configuration or scoping can hide one without the other.
Every workspace package README ends with the canonical [Model Experience section](../../../cookbook/adding-a-package.md#4-write-the-package-readme), immediately before `## Known Limitations and Deferred Work`; a package on the no-limitations allowlist ends with Model Experience itself. Packages with direct, multi-surface, conditional, capped, or lifetime effects use the three-column table. Each row identifies a concrete request surface, says what the relevant model literally receives and when, and classifies the token effect. Stable source literals are quoted verbatim, with named placeholders only for interpolated values; summaries are reserved for data-dependent payloads, provider-owned text, or schemas too large to reproduce. The default subject is the conversation model; a package that invokes an auxiliary model, such as a summarizer or search provider, names that request separately. Agent-scoped visibility is stated where it changes which agent receives a contribution. Prompt text and tool schemas are described separately whenever configuration or scoping can hide one without the other.
Every package participates. A package with no model-context effect, or one simple effect rendered entirely by another package, can join the verifier's audited sentence allowlist. It then uses exactly one sentence beginning `None, as ` or `Indirectly, through ` instead of stretching a negative fact across a three-column table. Implementations that shape results, caps, history, lifetimes, or more than one request surface keep the table even when they add zero direct prompt tokens.

View File

@@ -14,5 +14,5 @@ Naming notes:
- `src/types.ts` contains only types — no runtime code.
- Tests live at package level under `tests/`, not `src/__tests__/`.
- A package's README and JSDoc are part of the change: altered behavior (config keys, defaults, error codes, wire fields) updates them in the same commit. `doc-sync` gates what it can; prose accuracy stays on the author ([the documentation standard](../docs/AGENTS.md)).
- Package READMEs document model/token effects in [Model Experience](../docs/AGENTS.md#package-model-experience).
- Package READMEs document model/token effects using the [canonical Model Experience format](../docs/cookbook/adding-a-package.md#4-write-the-package-readme).
- Package READMEs carry `## Known Limitations and Deferred Work` or a justified [allowlist entry](../scripts/verify-readme-limitations.ts) ([rationale](../docs/rfc/implemented/process/2026-07-10-readme-known-limitations-gate.md)).

View File

@@ -37,4 +37,4 @@ The inter-package dependency graph is generated: [docs/module-graph.md](../docs/
The rule it must obey: **extension plugins depend on interfaces, never on the concrete loop.** `dsh-agent-loop` is swappable — UI/hook/tool plugins keep working against the `dsh-agent` vocabulary if the loop is replaced. The sanctioned exception is a **composition/bundle** package like `dsh-agent-core`, whose whole job is to assemble the concrete spine: it depends on `dsh-agent-loop` (and the other concrete spine plugins) on purpose. The rule constrains plugins that EXTEND the system, not the bundle that COMPOSES it. A swappable capability splits into interface / implementation / consumer packages (the bash trio is the template — see [capability seams](../docs/rfc/implemented/architecture/2026-06-13-capability-seams.md)).
Each package has its own `README.md` with purpose, service API, events, extension points, deliberate non-goals, and the standard [Model Experience table](../docs/AGENTS.md#package-model-experience).
Each package has its own `README.md` with purpose, service API, events, extension points, deliberate non-goals, and the standard [Model Experience section](../docs/cookbook/adding-a-package.md#4-write-the-package-readme).

View File

@@ -36,8 +36,9 @@ The keyless consumer-integration proofs are `tests/bwrap.e2e.ts`, `tests/landloc
| Context surface | What the model sees | Token effect |
|---|---|---|
| Bash tool schema, indirectly | By advertising a confining `sandboxMode`, this backend makes `dsh-tool-bash` expose `sandbox_permissions` and `justification`. The backend adds no prompt prose, and the session's effective mode remains unstated. | Small fixed schema increment on requests where `bash` is visible; mode switches add no context tokens. |
| Bash tool result, indirectly | The model sees ordinary bounded command output plus denial markers, the mode used, and sandbox-unavailable failures shaped by `dsh-tool-bash`; runner details stay internal. | Zero additional tokens on an unremarkable allowed run beyond ordinary output. Denial or failure adds a small conditional marker or error retained until compaction. |
| Bash tool schema, indirectly | By advertising a confining `sandboxMode`, this backend makes `dsh-tool-bash` expose `sandbox_permissions` with enum `workspace-write` \| `danger-full-access` and `justification`. The backend adds no prompt prose, and the session's effective mode remains unstated. | Small fixed schema increment on requests where `bash` is visible; mode switches add no context tokens. |
| Bash tool result, indirectly | After ordinary bounded output, a denied call appends exactly `[sandbox: file access denied under <mode> mode]`. When escalation is available it next appends `[sandbox: escalation available — retry this exact command once with sandbox_permissions (the narrowest wider mode that suffices) + justification; the approval prompt asks the user]`. A settled background runner failure instead appends `[sandbox: the sandbox runner itself failed under <mode> mode — the command did not run; this is a sandbox problem, not a command failure]`. | Zero additional tokens on an unremarkable allowed run beyond ordinary output. Denial or failure adds the quoted conditional marker, retained until compaction. |
| Bash tool error, indirectly | If no runner can enforce a confined mode, the foreground call fails with code `SANDBOX_UNAVAILABLE` and the exact message `sandbox mode "<mode>" is requested but no sandbox backend is usable on this host; refusing to run the command unconfined. Install bubblewrap or run a Landlock-enforcing kernel (Linux), ensure sandbox-exec is usable (macOS) — Windows has no confinement backend yet — or switch the consumer to danger-full-access.` An execution-time runner failure appends ` Runner failure: <first stderr line>`. | Conditional error text is visible for that call and retained in history until compaction. |
## Known Limitations and Deferred Work

View File

@@ -78,6 +78,29 @@ function proseLines(text: string): Line[] {
return kept
}
/** Split a canonical table row without treating escaped pipes as delimiters. */
function tableCells(raw: string): string[] | undefined {
const last = raw.length - 1
if (!raw.startsWith('|') || !raw.endsWith('|') || isEscaped(raw, last)) return undefined
const cells: string[] = []
let start = 1
for (let index = 1; index < last; index += 1) {
if (raw[index] !== '|' || isEscaped(raw, index)) continue
cells.push(raw.slice(start, index).trim())
start = index + 1
}
cells.push(raw.slice(start, last).trim())
return cells
}
/** Whether the character at `index` follows an odd-length backslash run. */
function isEscaped(text: string, index: number): boolean {
let backslashes = 0
for (let cursor = index - 1; cursor >= 0 && text[cursor] === '\\'; cursor -= 1) backslashes += 1
return backslashes % 2 === 1
}
const failures: Failure[] = []
const packageJsons = globSync('packages/*/*/package.json', { cwd: root }).sort()
const scannedPackages = new Set(packageJsons.map(path => path.slice(0, -'/package.json'.length)))
@@ -171,8 +194,8 @@ for (const packageJson of packageJsons) {
continue
}
for (const row of rows) {
const cells = row.raw.split('|').slice(1, -1).map(cell => cell.trim())
if (cells.length !== 3 || cells.some(cell => cell.length === 0)) {
const cells = tableCells(row.raw)
if (cells === undefined || cells.length !== 3 || cells.some(cell => cell.length === 0)) {
failures.push({ path: readme, message: `line ${row.index}: invalid three-column Model Experience row: ${row.raw}` })
}
}