mirror of
https://github.com/deepseek-ai/deepseek-harness
synced 2026-08-15 21:04:50 +00:00
fix(compact): size the compaction e2e to actually cross threshold; sync stale docs (CBR-007)
The compaction e2e never exercised compaction: its window/fixture combo (contextWindow 8000, thresholdRatio 0.5 → threshold 4000; four small files) peaked at ~1389 estimated tokens, so compactIfNeeded declined every pre-step and compact/start never landed. Shrink the window (contextWindow 2400 → threshold 1200; retainTokens 500 + summarizationMaxTokens 300 = 800 < 1200, convergence holds) and grow the fixture to six files so a couple of bash steps reliably cross the threshold. Verified compaction fires and the suite passes across repeated real-API runs. Sync docs left stale by the landed compaction work: list compaction.e2e.ts and keyless-smoke.e2e.ts in the coding-agent README (and fix the wrong "Both self-skip" count), add compaction to the examples with-key inventory, and replace the hypothetical compaction/marker / "future plugin" naming in the session README, session types JSDoc, and the core-data-structures catalog with the real compact/start, compact/summary, compact/end events.
This commit is contained in:
@@ -48,5 +48,6 @@ This example is a thin leaf `cordis.yml`: it picks the swappable backends and lo
|
||||
- `tests/full-loop.e2e.ts` — the canary: real model runs `echo e2e-ok` through the real bash tool; asserts `tool/call`/`tool/result` session events and the final answer.
|
||||
- `tests/coding-task.e2e.ts` — the swebench-style smoke: a temp dir holds `add.js` (with `a - b` where `a + b` belongs) and a failing `add.test.js`; the agent must fix the bug and verify. The test re-runs `node add.test.js` ITSELF and inspects the files — agent claims are not trusted.
|
||||
- `tests/resume.e2e.ts` — durable continuity across processes: run 1 tells the real model a secret code and persists the turn to a temp JSONL root, then the whole context is disposed; run 2 is a fresh context over the same root that RESUMES the session id and asks the model to recall the code. The recall can only come from the rehydrated log.
|
||||
- `tests/compaction.e2e.ts` — the compaction smoke: a real multi-step bash task runs with a deliberately tiny context window so the auto-compaction listener fires MID-SESSION. Verifies the WORLD — a `compact/start…end` pair landed in the real log, the surface shrank (a replace node shadowed older nodes), and the agent still produced a correct final answer after compaction.
|
||||
|
||||
Both self-skip without `DEEPSEEK_API_KEY`.
|
||||
All four self-skip without `DEEPSEEK_API_KEY`. The keyless boot smoke is `tests/keyless-smoke.e2e.ts` (boots the full real tree with a dummy key and no prompt, so no model call), which runs in the default e2e gate.
|
||||
|
||||
@@ -33,21 +33,23 @@ afterEach(async () => {
|
||||
describe.skipIf(!process.env.DEEPSEEK_API_KEY)('compaction: a long session compacts mid-flight and keeps running', () => {
|
||||
it('summarizes older history into a checkpoint without breaking the task', async () => {
|
||||
workdir = await mkdtemp(join(tmpdir(), 'dsh-compaction-'))
|
||||
// A few files for the model to read, so multiple bash steps accumulate
|
||||
// surface nodes (tool calls + results) and grow the history.
|
||||
for (let i = 1; i <= 4; i++) {
|
||||
// A handful of files for the model to read, so multiple bash steps
|
||||
// accumulate surface nodes (tool calls + results) and grow the history past
|
||||
// the (deliberately tiny) window.
|
||||
for (let i = 1; i <= 6; i++) {
|
||||
await writeFile(join(workdir, `file${i}.txt`), `This is file number ${i}. `.repeat(40))
|
||||
}
|
||||
|
||||
// Tiny window so a handful of steps crosses the threshold. The convergence
|
||||
// invariant requires summarizationMaxTokens + retainTokens <= window *
|
||||
// ratio = floor(8000 * 0.5) = 4000; 1500 + 2000 = 3500 <= 4000.
|
||||
// Tiny window so a couple of steps crosses the threshold. The convergence
|
||||
// invariant requires summarizationMaxTokens + retainTokens to be strictly
|
||||
// BELOW the threshold = floor(contextWindow * thresholdRatio) =
|
||||
// floor(2400 * 0.5) = 1200; 300 + 500 = 800 < 1200.
|
||||
ctx = await codingHarness(workdir, {
|
||||
compact: {
|
||||
contextWindow: 8000,
|
||||
contextWindow: 2400,
|
||||
thresholdRatio: 0.5,
|
||||
retainTokens: 2000,
|
||||
summarizationMaxTokens: 1500,
|
||||
retainTokens: 500,
|
||||
summarizationMaxTokens: 300,
|
||||
},
|
||||
})
|
||||
const agent = ctx.agentLoop.create(AgentId('e2e-compaction'), {
|
||||
@@ -57,9 +59,9 @@ describe.skipIf(!process.env.DEEPSEEK_API_KEY)('compaction: a long session compa
|
||||
|
||||
agent.send([{
|
||||
type: 'text',
|
||||
text: 'Read file1.txt, file2.txt, file3.txt, and file4.txt one at a time using cat '
|
||||
+ '(a separate bash command for each). After reading all four, tell me how many '
|
||||
+ 'files you read and the number mentioned in file1.txt.',
|
||||
text: 'Read file1.txt, file2.txt, file3.txt, file4.txt, file5.txt, and file6.txt one at a '
|
||||
+ 'time using cat (a separate bash command for each). After reading all six, tell me how '
|
||||
+ 'many files you read and the number mentioned in file1.txt.',
|
||||
}])
|
||||
await waitForIdle(ctx, agent)
|
||||
|
||||
@@ -87,9 +89,9 @@ describe.skipIf(!process.env.DEEPSEEK_API_KEY)('compaction: a long session compa
|
||||
expect(summaryData.shadowedSeqs.length).toBeGreaterThan(0)
|
||||
|
||||
// The conversation survived compaction: the agent produced a final answer
|
||||
// that reflects the work (it read four files).
|
||||
// that reflects the work (it read six files).
|
||||
const answer = finalText(events).toLowerCase()
|
||||
expect(answer.length).toBeGreaterThan(0)
|
||||
expect(answer).toMatch(/\b(4|four)\b/)
|
||||
expect(answer).toMatch(/\b(6|six)\b/)
|
||||
}, 240_000)
|
||||
})
|
||||
|
||||
Reference in New Issue
Block a user