mirror of
https://github.com/deepseek-ai/deepseek-harness
synced 2026-08-15 21:04:50 +00:00
fix(session-query): harden SQLite search reconciliation
This commit is contained in:
@@ -613,7 +613,7 @@ export interface Config {
|
||||
export type JournalMode = 'wal' | 'delete' | 'truncate' | 'persist'
|
||||
```
|
||||
|
||||
Source: [`packages/session-query/session-query-sqlite/src/index.ts:58`](../packages/session-query/session-query-sqlite/src/index.ts)
|
||||
Source: [`packages/session-query/session-query-sqlite/src/index.ts:67`](../packages/session-query/session-query-sqlite/src/index.ts)
|
||||
|
||||
## `@deepseek-ai/dsh-skill`
|
||||
|
||||
|
||||
@@ -195,11 +195,12 @@ abstract create(meta: SessionHeader): Promise<void>
|
||||
abstract append(id: SessionId, events: readonly SessionEvent[]): Promise<void>
|
||||
abstract load(id: SessionId): Promise<{ meta: SessionHeader; events: SessionEvent[] }>
|
||||
abstract list(): Promise<SessionHeader[]>
|
||||
abstract listSnapshots(): Promise<SessionPersistenceSnapshot[]>
|
||||
```
|
||||
|
||||
Types: [SessionEvent](../core-data-structures/core.md)
|
||||
|
||||
Source: [`packages/session-persistence/session-persistence/src/index.ts:102`](../../packages/session-persistence/session-persistence/src/index.ts)
|
||||
Source: [`packages/session-persistence/session-persistence/src/index.ts:112`](../../packages/session-persistence/session-persistence/src/index.ts)
|
||||
|
||||
## `ctx.sessionQuery` — `SessionQueryService`
|
||||
|
||||
@@ -207,12 +208,13 @@ Live-preferred logical-corpus and exact-event read service.
|
||||
|
||||
```ts cordis-catalog
|
||||
listSessions(): Promise<SessionRecord[]>
|
||||
async filterSessions(filters: readonly SessionResultFilter[]): Promise<SessionRecord[]>
|
||||
async listEvents(sessionId: SessionId): Promise<SessionEventRecord[]>
|
||||
async filterEvents( sessionId: SessionId, filters: readonly SessionEventResultFilter[], ): Promise<SessionEventSearchDocument[]>
|
||||
async readEvent(request: SessionEventReadRequest): Promise<SessionEventWindow>
|
||||
```
|
||||
|
||||
Source: [`packages/session-query/session-query/src/index.ts:83`](../../packages/session-query/session-query/src/index.ts)
|
||||
Source: [`packages/session-query/session-query/src/index.ts:96`](../../packages/session-query/session-query/src/index.ts)
|
||||
|
||||
## `ctx.sessions` — `SessionStore`
|
||||
|
||||
@@ -244,7 +246,7 @@ abstract searchSessions( request: SessionSearchRequest, exec?: SessionSearchExec
|
||||
abstract searchEvents( request: SessionEventSearchRequest, exec?: SessionSearchExecContext, ): Promise<SessionSearchPage<SessionEventSearchHit>>
|
||||
```
|
||||
|
||||
Source: [`packages/session-query/session-query/src/index.ts:54`](../../packages/session-query/session-query/src/index.ts)
|
||||
Source: [`packages/session-query/session-query/src/index.ts:67`](../../packages/session-query/session-query/src/index.ts)
|
||||
|
||||
## `ctx.skills` — `SkillService`
|
||||
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
The **durability seam** for the event log. [session.md](session.md) describes the in-memory `Session` — the append-only `SessionEvent` log that is the source of truth. This page describes how that log is made durable: the abstract `SessionPersistence` service, its backends, the flush checkpoint, crash recovery, and the metadata header that travels alongside the log. The event vocabulary the log carries is enumerated, member by member, in the generated [persistence log event catalog](../persistence-catalog.md).
|
||||
|
||||
The seam is a textbook [capability seam](../rfc/implemented/architecture/2026-06-13-capability-seams.md): one abstract service ([dsh-session-persistence](../../packages/session-persistence/session-persistence), `ctx.sessionPersistence`) defining create/append/load/list over the existing `SessionEvent` — **no parallel persisted type** — and two interchangeable backends that pass the same `runPersistenceContract` suite. See the [session-persistence RFC](../rfc/implemented/architecture/2026-06-14-session-persistence.md).
|
||||
The seam is a textbook [capability seam](../rfc/implemented/architecture/2026-06-13-capability-seams.md): one abstract service ([dsh-session-persistence](../../packages/session-persistence/session-persistence), `ctx.sessionPersistence`) defining create/append/load plus lightweight list/snapshot observation over the existing `SessionEvent` — **no parallel persisted event type** — and two interchangeable backends that pass the same `runPersistenceContract` suite. See the [session-persistence RFC](../rfc/implemented/architecture/2026-06-14-session-persistence.md).
|
||||
|
||||
## The flush checkpoint
|
||||
|
||||
@@ -78,9 +78,24 @@ interface CreateSessionOptions {
|
||||
|
||||
Replay/fork is therefore `ctx.sessions.create(id, { seed: seedEvents })`; resuming a *persisted* session into a live agent is `ctx.agents.resume({ resumeSessionId })`.
|
||||
|
||||
## Lightweight source revisions
|
||||
|
||||
Consumers of derived state compare a cheap opaque revision before loading a full event log. The persistence backend owns its representation and changes it transactionally with append or mutating load repair; callers compare it only for equality.
|
||||
|
||||
```ts type-equiv
|
||||
export type SessionPersistenceRevision = Branded<'SessionPersistenceRevision'>
|
||||
```
|
||||
|
||||
```ts type-equiv
|
||||
export interface SessionPersistenceSnapshot {
|
||||
header: SessionHeader
|
||||
revision: SessionPersistenceRevision
|
||||
}
|
||||
```
|
||||
|
||||
## The backends
|
||||
|
||||
Both implement the same abstract `SessionPersistence` (create/append/load/list over `SessionEvent`) and pass `runPersistenceContract`, proving the seam is genuinely backend-agnostic:
|
||||
Both implement the same abstract `SessionPersistence` (create/append/load/list/listSnapshots over `SessionEvent`) and pass `runPersistenceContract`, proving the seam is genuinely backend-agnostic:
|
||||
|
||||
- **[dsh-session-persistence-jsonl](../../packages/session-persistence/session-persistence-jsonl)** — an append-only JSONL log per session with crash-safe atomic writes, the interrupted-turn crash recovery above, and a read/replay path.
|
||||
- **[dsh-session-persistence-sqlite](../../packages/session-persistence/session-persistence-sqlite)** — `node:sqlite`, one row per `SessionEvent`. The row shape `(session_id, seq, type, time, data, source_event_seqs, surface_op)` maps 1:1 onto the event, including optional surface metadata, so there is no parallel persisted schema to keep in sync.
|
||||
|
||||
@@ -58,19 +58,23 @@ export interface SessionEventSearchDocument extends SessionEventRecord {
|
||||
}
|
||||
```
|
||||
|
||||
`ctx.sessionQuery.filterEvents(sessionId, filters)` returns these documents in ascending seq order. Messages, reasoning, tool calls/results, blocked prompts, todos, and failure/status detail contribute semantic text; structural events and stream chunks do not.
|
||||
`ctx.sessionQuery.filterSessions(filters)` applies `SessionResultFilter` to the complete logical corpus; `ctx.sessionQuery.filterEvents(sessionId, filters)` returns matching documents in ascending seq order. Messages, reasoning, tool calls/results, blocked prompts, todos, and failure/status detail contribute semantic text; structural events and stream chunks do not.
|
||||
|
||||
## Full-text search pages
|
||||
|
||||
The independent `ctx.sessionSearch` seam has two scopes. `searchSessions()` groups the corpus by strongest matching event; `searchEvents()` searches one session. Requests bind an opaque cursor to the normalized query, metadata filters, and limit. The event text scan is intentionally absent from provider metadata filters.
|
||||
|
||||
```ts type-equiv
|
||||
export type SessionSearchCursor = Branded<'SessionSearchCursor'>
|
||||
```
|
||||
|
||||
```ts type-equiv
|
||||
export interface SessionSearchRequest {
|
||||
query: string
|
||||
sessionFilters?: readonly SessionResultFilter[]
|
||||
eventFilters?: readonly SessionEventMetadataFilter[]
|
||||
limit?: number
|
||||
cursor?: string
|
||||
cursor?: SessionSearchCursor
|
||||
}
|
||||
```
|
||||
|
||||
@@ -80,14 +84,14 @@ export interface SessionEventSearchRequest {
|
||||
query: string
|
||||
filters?: readonly SessionEventMetadataFilter[]
|
||||
limit?: number
|
||||
cursor?: string
|
||||
cursor?: SessionSearchCursor
|
||||
}
|
||||
```
|
||||
|
||||
```ts type-equiv
|
||||
export interface SessionSearchPage<T> {
|
||||
items: readonly T[]
|
||||
nextCursor?: string
|
||||
nextCursor?: SessionSearchCursor
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
@@ -151,6 +151,7 @@ flowchart TD
|
||||
pkg_web_search_deepseek --> pkg_web
|
||||
pkg_web_search_exa --> pkg_web
|
||||
pkg_web_search_perplexity --> pkg_web
|
||||
pkg_session_persistence --> pkg_brand
|
||||
pkg_session_persistence --> pkg_session
|
||||
pkg_llm_replay --> pkg_llm
|
||||
pkg_llm_replay --> pkg_session
|
||||
@@ -168,6 +169,7 @@ flowchart TD
|
||||
pkg_session_persistence_jsonl --> pkg_session_persistence
|
||||
pkg_session_persistence_sqlite --> pkg_session
|
||||
pkg_session_persistence_sqlite --> pkg_session_persistence
|
||||
pkg_session_query --> pkg_brand
|
||||
pkg_session_query --> pkg_llm
|
||||
pkg_session_query --> pkg_session
|
||||
pkg_session_query --> pkg_session_persistence
|
||||
@@ -361,7 +363,7 @@ flowchart TD
|
||||
| [`web-search-deepseek`](../packages/web/web-search-deepseek) | `web` | [`web`](../packages/web/web) |
|
||||
| [`web-search-exa`](../packages/web/web-search-exa) | `web` | [`web`](../packages/web/web) |
|
||||
| [`web-search-perplexity`](../packages/web/web-search-perplexity) | `web` | [`web`](../packages/web/web) |
|
||||
| [`session-persistence`](../packages/session-persistence/session-persistence) | `session-persistence` | [`session`](../packages/core/session) |
|
||||
| [`session-persistence`](../packages/session-persistence/session-persistence) | `session-persistence` | [`brand`](../packages/util/brand), [`session`](../packages/core/session) |
|
||||
| [`llm-replay`](../packages/support/llm-replay) | `support` | [`llm`](../packages/llm/llm), [`session`](../packages/core/session) |
|
||||
| [`sandbox-local`](../packages/sandbox/sandbox-local) | `sandbox` | [`llm`](../packages/llm/llm), [`sandbox`](../packages/sandbox/sandbox) |
|
||||
| [`bash-local`](../packages/bash/bash-local) | `bash` | [`bash`](../packages/bash/bash), [`timeout`](../packages/util/timeout) |
|
||||
@@ -369,7 +371,7 @@ flowchart TD
|
||||
| [`hook-protocol`](../packages/hooks/hook-protocol) | `hooks` | [`bash`](../packages/bash/bash), [`session`](../packages/core/session) |
|
||||
| [`session-persistence-jsonl`](../packages/session-persistence/session-persistence-jsonl) | `session-persistence` | [`session`](../packages/core/session), [`session-persistence`](../packages/session-persistence/session-persistence) |
|
||||
| [`session-persistence-sqlite`](../packages/session-persistence/session-persistence-sqlite) | `session-persistence` | [`session`](../packages/core/session), [`session-persistence`](../packages/session-persistence/session-persistence) |
|
||||
| [`session-query`](../packages/session-query/session-query) | `session-query` | [`llm`](../packages/llm/llm), [`session`](../packages/core/session), [`session-persistence`](../packages/session-persistence/session-persistence) |
|
||||
| [`session-query`](../packages/session-query/session-query) | `session-query` | [`brand`](../packages/util/brand), [`llm`](../packages/llm/llm), [`session`](../packages/core/session), [`session-persistence`](../packages/session-persistence/session-persistence) |
|
||||
| [`invariants`](../packages/support/invariants) | `support` | [`agent`](../packages/core/agent), [`llm`](../packages/llm/llm), [`scope`](../packages/core/scope), [`session`](../packages/core/session) |
|
||||
| [`user-approval`](../packages/ui/user-approval) | `ui` | [`agent`](../packages/core/agent), [`brand`](../packages/util/brand), [`llm`](../packages/llm/llm), [`scope`](../packages/core/scope), [`session`](../packages/core/session), [`system-prompt`](../packages/core/system-prompt) |
|
||||
| [`user-interaction`](../packages/ui/user-interaction) | `ui` | [`agent`](../packages/core/agent), [`llm`](../packages/llm/llm) |
|
||||
|
||||
@@ -10,7 +10,7 @@ Full-text search is related but materially larger. Putting provider coordination
|
||||
|
||||
## Decision
|
||||
|
||||
`@deepseek-ai/dsh-session-query` owns `ctx.sessionQuery`, a small trusted exact-read service over one logical corpus. It exposes `listSessions()`, `listEvents(sessionId)`, provider-independent `filterEvents(sessionId, filters)`, and bounded `readEvent(request)`. The package also declares the separate abstract `ctx.sessionSearch` contract and shared semantic extraction used by the [SQLite search decision](2026-07-10-sqlite-session-query-provider.md), but `ctx.sessionQuery` does not coordinate providers or synchronize a derived index.
|
||||
`@deepseek-ai/dsh-session-query` owns `ctx.sessionQuery`, a small trusted exact-read service over one logical corpus. It exposes `listSessions()`, provider-independent `filterSessions(filters)`, `listEvents(sessionId)`, `filterEvents(sessionId, filters)`, and bounded `readEvent(request)`. The package also declares the separate abstract `ctx.sessionSearch` contract and shared semantic extraction used by the [SQLite search decision](2026-07-10-sqlite-session-query-provider.md), but `ctx.sessionQuery` does not coordinate providers or synchronize a derived index.
|
||||
|
||||
The service observes the optional `ctx.sessionPersistence` binding dynamically but retains no persisted cache or invalidation listener. Each cross-corpus list asks the active backend for authoritative metadata, then overlays a fresh live-store list. Matching ids become one `SessionRecord`: the live header wins and `live`/`persisted` independently report source availability. Immutable header disagreement is `SESSION_QUERY_SOURCE_CONFLICT`.
|
||||
|
||||
|
||||
@@ -10,19 +10,19 @@ Splitting those concerns across a provider coordinator and a database implementa
|
||||
|
||||
## Decision
|
||||
|
||||
`@deepseek-ai/dsh-session-query` declares an independent abstract `ctx.sessionSearch` service without changing the exact-read `ctx.sessionQuery` key. `searchSessions(request, exec?)` returns cursor-paginated `SessionSearchHit`s grouped by each session's strongest matching event; `searchEvents(request, exec?)` returns `SessionEventSearchHit`s within one logical session. Both requests require `query`, accept `limit` and an opaque `cursor`, and support an optional abort signal. Session search accepts `sessionFilters` plus event metadata filters; event search accepts event metadata filters. Results expose bounded plain-text snippets but no provider identifier or numeric relevance score.
|
||||
`@deepseek-ai/dsh-session-query` declares an independent abstract `ctx.sessionSearch` service without changing the exact-read `ctx.sessionQuery` key. `searchSessions(request, exec?)` returns cursor-paginated `SessionSearchHit`s grouped by each session's strongest matching event; `searchEvents(request, exec?)` returns `SessionEventSearchHit`s within one logical session. Both requests require `query`, accept `limit` and an owned branded `SessionSearchCursor`, and support an optional abort signal. Session search accepts `sessionFilters` plus event metadata filters; event search accepts event metadata filters. Results expose bounded plain-text snippets but no provider identifier or numeric relevance score.
|
||||
|
||||
`@deepseek-ai/dsh-session-query-sqlite` is the sole concrete owner of `ctx.sessionSearch`. It depends on live `ctx.sessions`, observes optional `ctx.sessionPersistence` dynamically, and owns a dedicated derived SQLite database. There is no search-provider registry, coordinator, persistence event, or agent-loop integration.
|
||||
|
||||
The interface package also owns shared first-party semantic extraction and provider-independent filtering. `SessionResultFilter` covers id, nullable cwd, created-at range, nullable parent, and availability. `SessionEventResultFilter` covers seq/time ranges, event type, surface, and literal semantic text. Arrays are ANDed and list values are ORed. The text clause escapes caller input into a Unicode case-insensitive regular expression whose whitespace runs match one or more whitespace characters; it is available through `ctx.sessionQuery.filterEvents()` and is not delegated to an FTS provider.
|
||||
The interface package also owns shared first-party semantic extraction and provider-independent filtering. `SessionResultFilter` covers id, nullable cwd, created-at range, nullable parent, and availability; `ctx.sessionQuery.filterSessions()` applies it without an FTS provider. `SessionEventResultFilter` covers seq/time ranges, event type, surface, and literal semantic text. Arrays are ANDed and list values are ORed. The text clause escapes caller input into a Unicode case-insensitive regular expression whose whitespace runs match one or more whitespace characters; it is available through `ctx.sessionQuery.filterEvents()` and is not delegated to an FTS provider.
|
||||
|
||||
## Search semantics
|
||||
|
||||
Each semantic event is one FTS document carrying session metadata, event metadata, surface classification, and extracted text. All `current`, `shadowed`, and `log-only` documents participate unless a surface filter narrows them. Metadata filters compile to parameterized SQL before ranking. Session results partition matching documents by session and retain the strongest one.
|
||||
|
||||
Ordering is deterministic: BM25 ascending, event time descending, session id ascending for the cross-session scope, and seq descending. Provider scores remain private. Snippets normalize whitespace and are bounded by Unicode code points. Opaque cursors bind to the service instance, scope, canonical normalized request, offset, and relevant generation. Any corpus change invalidates cross-session cursors; a within-session cursor changes only when its target source/generation changes, so unrelated sessions do not invalidate it. Reopening creates a new service instance and invalidates old cursors.
|
||||
Ordering is deterministic and comparable across the persistent and TEMP FTS tables: actual FTS5 highlighted-match span count descending, indexed document code-point length ascending, event time descending, session id ascending for the cross-session scope, and seq descending. Snippets use those actual highlight positions, strip the reserved markers, normalize whitespace, and bound by Unicode code points. Opaque cursors bind to the service instance, scope, canonical normalized request, offset, and relevant generation. Any corpus change invalidates cross-session cursors; a within-session cursor changes only when its target source/generation changes, so unrelated sessions do not invalidate it. Reopening creates a new service instance and invalidates old cursors.
|
||||
|
||||
Queries are trimmed, whitespace-normalized, and quoted as one literal FTS5 phrase. Embedded quotes are doubled before binding, so MATCH operators such as `OR`, `NEAR`, quotes, parentheses, and `*` remain data rather than executable query syntax. Phrase matching follows tokenizer tokens rather than arbitrary substrings.
|
||||
Queries are trimmed, whitespace-normalized, and quoted as one literal FTS5 phrase. Embedded quotes are doubled before binding, so MATCH operators such as `OR`, `NEAR`, quotes, parentheses, and `*` remain data rather than executable query syntax. NUL is rejected before SQLite execution. Reserved highlight noncharacters and NUL in documents are normalized before indexing, making inserted presentation markers collision-free. Phrase matching follows tokenizer tokens rather than arbitrary substrings.
|
||||
|
||||
## Tokenizer choice
|
||||
|
||||
@@ -32,11 +32,11 @@ Both persistent and live FTS5 tables use `unicode61`. The implementation experim
|
||||
|
||||
The shared extractor includes message text, reasoning, nested tool-call/result content, tool names and arguments, blocked-prompt reasons, todo status/content, and error or terminal status detail. Structural boundaries, stream chunks, request headers, successful completion markers, and unknown declaration-merged event/content variants produce no document. Surface classification reuses `foldSurface()` so search agrees with model-history derivation.
|
||||
|
||||
One serialized operation observes complete persisted and live sources, computes stable fingerprints, reconciles rows in one transaction, and executes the query. Unchanged persisted sessions retain their rows and generation. New, changed, and deleted persisted sessions update on the next search. A source or extraction failure cannot mark a row current, and a transaction failure rolls back so a later search retries.
|
||||
One serialized operation reads the provider-neutral `SessionPersistence` snapshot listing, compares each opaque revision with the revision stored beside the indexed session, loads only new or changed logs, reconciles rows in one transaction, and executes the query. Observation repeats when listing changes during a load; this incorporates a mutating load repair's refreshed revision before commit. Repeated queries and unchanged reopen load no full persisted logs. New, changed, and deleted sessions update on the next stable search. A source or extraction failure cannot mark a row current, and a transaction failure rolls back so a later search retries.
|
||||
|
||||
Persisted documents survive restarts. Live sessions use connection-local TEMP tables, shadow the persisted base for the same id, and reveal that base on detach. Closing the database drops live rows. Unmounting persistence hides durable rows without treating absence as authoritative deletion; remounting observes and reconciles the backend again. Conflicting immutable live and durable headers fail rather than combining sources.
|
||||
|
||||
The derived schema has its own application id and monotonic schema version. A recognized incompatible version resets only this derived database. A database with a foreign application id or unrecognized user tables is refused, which prevents an accidentally configured canonical session database from being reset.
|
||||
The derived schema has its own application id and monotonic schema version. A recognized incompatible version resets only this derived database. A database with a foreign application id or unrecognized user tables is refused before journal-mode mutation, which prevents an accidentally configured canonical session database from being changed. One service in one process exclusively owns a derived-index path; cross-process writers are unsupported because generations and live TEMP shadow state are connection-owned.
|
||||
|
||||
Cancellation rejects queued operations and caller waits around asynchronous source observation without committing an aborted observation. Node's synchronous `DatabaseSync` MATCH call cannot be interrupted once it is executing on the JavaScript thread, so the service checks the signal at serialized boundaries but does not promise mid-statement preemption.
|
||||
|
||||
@@ -46,11 +46,11 @@ Cancellation rejects queued operations and caller waits around asynchronous sour
|
||||
- **Add a phase-one provider registry and coordinator** — rejected because one implementation provides no evidence for registration semantics and would split one reconciliation lifecycle across two owners.
|
||||
- **Persist live overrides immediately** — rejected because live events are not canonical until the existing checkpoint commits.
|
||||
- **Use the FTS5 trigram tokenizer** — rejected because it omits useful queries shorter than three characters and measured about 2.1× the index size of `unicode61`; literal substring filtering remains available through the scan path.
|
||||
- **Return BM25 scores** — rejected because provider-specific numeric scales are unstable across corpus changes.
|
||||
- **Use FTS5 BM25 independently in each table** — rejected because scores from differently populated persistent and TEMP corpora are not comparable; actual matched spans and document length have one shared scale.
|
||||
|
||||
## Consequences
|
||||
|
||||
Search has a small provider-neutral API while its only backend owns every derived-index state transition. The separate database adds configuration and a reconciliation read before queries, but index corruption, reset, and tokenizer changes cannot endanger canonical logs. Persistent fingerprints avoid rewriting unchanged sessions; TEMP live overlays preserve current-session truth without making uncheckpointed events durable.
|
||||
Search has a small provider-neutral API while its only backend owns every derived-index state transition. The separate database adds configuration and a lightweight snapshot read before queries, but index corruption, reset, and tokenizer changes cannot endanger canonical logs. Durable revisions avoid full-log reads and rewrites for unchanged sessions; TEMP live overlays preserve current-session truth without making uncheckpointed events durable.
|
||||
|
||||
The chosen tokenizer supports short tokens with a smaller index but does not promise substring recall. Literal phrases make query syntax safe and predictable at the cost of excluding boolean/full MATCH expressions. Cancellation is effective while queued or awaiting sources, but synchronous SQLite execution remains a non-preemptible section.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user