mirror of
https://github.com/deepseek-ai/deepseek-harness
synced 2026-08-15 21:04:50 +00:00
Add web multimodal image attachments
This commit is contained in:
Yichen Jiang
@@ -0,0 +1,6 @@
|
||||
# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
2026-07-22-web-multimodal-image-input-and-durable-attachments.md: 14ad4e5ceb28fd7df8b639db403be8accd8c2028
|
||||
2026-07-22-web-multimodal-image-input-and-durable-attachments.zh.md: 5752247b080ee1be0b0bfbbf7a9fb7486ff6c41b
|
||||
@@ -0,0 +1,210 @@
|
||||
# Agent Note: Web multimodal image input and durable attachments
|
||||
|
||||
Status: proposed
|
||||
|
||||
English | [中文](2026-07-22-web-multimodal-image-input-and-durable-attachments.zh.md)
|
||||
|
||||
## Problem
|
||||
|
||||
The Web composer accepts only text: `InputBar` receives a string draft, `ConversationService.send()` creates text content, and the host forwards that content to the agent. Users cannot paste an image, inspect it before sending, submit an image-only prompt, or recover sent images from history.
|
||||
|
||||
This is not only a composer gap. Core needs a durable image content block, providers need explicit modality handling, and the session log must reconstruct everything visible to a model. [The previous image-block removal](../../implemented/simplification/2026-07-04-drop-image-content-block.md) rejected a partial design that could silently lose or flatten images. A browser object URL, local path, provider URL, or base64 payload cannot be canonical session content.
|
||||
|
||||
The [Web client architecture](../../implemented/architecture/2026-07-19-gui-web-client-architecture.md) keeps components pure and per-session composer state in `ctx.conversation`; the [GUI layering and RPC protocol](../../implemented/architecture/2026-07-19-gui-layering-and-rpc-protocol.md) makes durable events the source of truth for both live rendering and history replay. Image intake, persistence, provider conversion, and rendering therefore need one explicit lifecycle.
|
||||
|
||||
Peer products converge on an attachment rail above the editor, but their storage choices differ. Codex-style paths such as `/var/folders/.../codex-clipboard-*.png` are reasonable intake staging locations, not durable message identities: the operating system may delete them, another host cannot read them, and a resumed session cannot rely on them.
|
||||
|
||||
## Proposal
|
||||
|
||||
Add pasted or dropped raster images to the Web composer as the first consumer of a durable attachment capability. Unsent files remain temporary client-owned draft state. The host validates and durably commits every accepted user image before appending its message event. A provider adapter that produces structured image output must durably commit the output before appending its assistant block. Canonical user and assistant content contains only role-neutral `ImageBlock` references.
|
||||
|
||||
Version one supports PNG, JPEG, WebP, and GIF paste and drag-and-drop, image-only or mixed prompts, historical user and assistant image rendering, and original-image preview on double-click. File picking, generic files, PDF, audio, video, image copying, and a custom context menu are separate follow-ups.
|
||||
|
||||
### Product behavior
|
||||
|
||||
- Pasting or dropping one or more supported images adds ordered thumbnails above the textarea without inserting placeholder text. Dragging files over the composer highlights the drop target.
|
||||
- The rail is shared by the empty-state and resident composers, is hidden when empty, and scrolls horizontally instead of widening the composer.
|
||||
- Each approximately 72-by-72-pixel thumbnail has a remove action and opens its original draft image on double-click.
|
||||
- A prompt may contain text and images or images only. Pure text paste remains native browser behavior; the paste handler prevents the default only when it accepts an image file. File drops on the composer always prevent browser navigation, accept supported images, and report unsupported files locally.
|
||||
- A failed send restores the complete text and image draft. Removal, successful send, and session-scope disposal revoke obsolete object URLs.
|
||||
- Historical user and assistant images use one `MessageImage` control. Inline images preserve intrinsic aspect ratio, do not upscale, and stay within a 240-by-240-pixel box.
|
||||
- Double-clicking a message image opens the stored original in a viewport-bounded modal. Escape, the close control, and backdrop activation close it and restore focus.
|
||||
- Version one does not override the browser context menu and provides no explicit image-copy action.
|
||||
|
||||
### Storage lifecycle and ownership
|
||||
|
||||
The persistence boundary is message acceptance, not paste:
|
||||
|
||||
| State | Allowed representation | Durability and ordering |
|
||||
| --- | --- | --- |
|
||||
| Unsent user draft | Browser `File` plus object URL; a native client may use an OS temporary file such as `/var/...` | Temporary and client-owned. It may disappear on reload or process exit and never appears in a session event. |
|
||||
| Accepted user image | Immutable object below `DSH_HOME` plus `ImageAttachmentRef` | The host commits every image before `agent.send()` or `agent.steer()` can append the owning user event. |
|
||||
| Structured model image output | Immutable object below `DSH_HOME` plus `ImageAttachmentRef` | The provider adapter commits the bytes before it emits a completed image block or assistant message event. Temporary URLs, paths, and base64 are forbidden in the event. |
|
||||
|
||||
The framework-owned chat store keeps the per-session draft text and ordered attachment identifiers. `ConversationService` owns the corresponding browser-only `File` and object-URL registry:
|
||||
|
||||
```ts
|
||||
export {}
|
||||
|
||||
interface ChatStoreState {
|
||||
selection: object | null
|
||||
draft: string
|
||||
imageIds: string[]
|
||||
view: string | null
|
||||
}
|
||||
|
||||
interface ComposerAttachment {
|
||||
id: string
|
||||
file: File
|
||||
previewUrl: string
|
||||
}
|
||||
```
|
||||
|
||||
This split uses the slots framework's store seat and bound actions as the single subscription path for UI state while keeping non-serializable browser objects out of persisted JSON. Draft text and ordered image identifiers continue to use `localStorage`; after a reload, `ConversationRoot` prunes identifiers whose runtime objects no longer exist. Unsent images therefore do not survive reload because browser `File` and object URLs are not durable. A native client may stage input in an OS temporary directory, but it must treat that path exactly like the browser object URL: delete it when no longer needed and copy the bytes into the durable store before message acceptance.
|
||||
|
||||
The local attachment backend resolves an explicit `dshHome`, then `$DSH_HOME`, then `~/.dsh`. It stores content-addressed objects below `$DSH_HOME/attachments/v1/objects/<prefix>/<sha256>` with owner-only directory and file permissions. A temporary file is written, synchronized, and atomically published before the service returns a reference. The content digest is encoded in the opaque `sha256:<digest>` identifier, and every read verifies the digest, media type, byte length, width, and height.
|
||||
|
||||
The store performs no automatic deletion in version one. Sent user images and model-generated images remain reachable for history, resume, and fork. Reference-aware garbage collection needs a separate design because an age-only rule can delete data still referenced by a durable session.
|
||||
|
||||
### Durable content and prompt wire
|
||||
|
||||
The attachment seam exposes immutable image write and verified read operations. The canonical metadata is deliberately narrower than a generic file record:
|
||||
|
||||
```ts
|
||||
import type { Branded } from '@deepseek-ai/dsh-brand'
|
||||
|
||||
type AttachmentId = Branded<'AttachmentId'>
|
||||
|
||||
interface ImageAttachmentRef {
|
||||
attachmentId: AttachmentId
|
||||
mediaType: 'image/png' | 'image/jpeg' | 'image/webp' | 'image/gif'
|
||||
bytes: number
|
||||
width: number
|
||||
height: number
|
||||
name?: string
|
||||
}
|
||||
|
||||
interface ImageBlock {
|
||||
type: 'image'
|
||||
attachment: ImageAttachmentRef
|
||||
}
|
||||
```
|
||||
|
||||
`ImageBlock` joins the merge-extensible core `ContentBlockMap` and is valid in either user or assistant content. It never carries base64, an object URL, a filesystem path, or a provider-owned locator. This keeps the session event plus immutable object store sufficient to reconstruct the exact model-visible image.
|
||||
|
||||
The browser cannot mint a durable reference, so `session.prompt` accepts a narrow intake union rather than canonical `ContentBlock[]`:
|
||||
|
||||
```ts
|
||||
export {}
|
||||
|
||||
type PromptInputPart =
|
||||
| { type: 'text'; text: string }
|
||||
| {
|
||||
type: 'image'
|
||||
mediaType: 'image/png' | 'image/jpeg' | 'image/webp' | 'image/gif'
|
||||
data: string
|
||||
name?: string
|
||||
}
|
||||
```
|
||||
|
||||
Base64 crosses JSON-RPC once and is discarded after persistence. The host validates canonical base64, image count, aggregate bytes, individual bytes, magic-byte MIME, intrinsic dimensions, and decoded-pixel count. Only after every image succeeds does it call the agent with normalized text and durable image blocks. A failure appends no user event and exposes no attachment path or raw bytes.
|
||||
|
||||
`session.attachment` is a read-only, session-scoped endpoint. The host serves bytes only when a durable event in that session references the requested attachment identifier. The client caches the resulting object URL by session and attachment identifier for its service lifetime and revokes it on disposal.
|
||||
|
||||
### Model capabilities and provider behavior
|
||||
|
||||
Model catalog entries gain optional merge-extensible input and output modality declarations. A missing declaration means unknown; a present list without `image` is an explicit negative capability.
|
||||
|
||||
The host is the authoritative preflight boundary. If the selected model explicitly excludes image input, it rejects the prompt before writing any attachment or event, and the client restores the draft. Unknown capability proceeds to the adapter guard so uncatalogued model identifiers remain usable. Immediate intake-time rejection in the UI may be added after model selection is exposed consistently across every Web entry path.
|
||||
|
||||
The Pi-AI adapter is the first visual-input route: it resolves each durable reference through `ctx.attachments` and emits native image content only for models that declare image input. The hand-written DeepSeek adapter throws typed `UNSUPPORTED_CONTENT` for an image anywhere in the request, including nested tool results. No adapter may flatten or skip an image.
|
||||
|
||||
Core supports structured assistant image blocks, but no current production provider route is certified for image output. Any future output-capable adapter must retrieve provider bytes under bounded size and time policy, validate them through the same attachment service, persist them, and only then publish the atomic `ImageBlock`. A URL in assistant Markdown remains text and is never downloaded automatically.
|
||||
|
||||
Token estimation accounts for image dimensions without counting base64 or attachment locators as text. Provider-reported usage remains authoritative. ACP renders an explicit image marker until that protocol surface gains native image support rather than silently omitting the block.
|
||||
|
||||
### History rendering and original preview
|
||||
|
||||
History folding preserves `ImageBlock` in both user and assistant messages. User images align to the trailing edge above their text; assistant images align to the leading narration flow. `MessageImage` derives a stable inline box from recorded dimensions, resolves bytes through the session-authorized loader, uses `object-fit: contain`, and turns a missing or corrupt object into a retryable error control.
|
||||
|
||||
Composer thumbnails and each `MessageImage` own ephemeral original-preview state and invoke the same pure `ImageLightbox`. The modal uses the already resolved original object URL, constrains only display size, focuses its close control, and restores the previous focus target when closed.
|
||||
|
||||
### Limits and trust boundaries
|
||||
|
||||
Version one accepts PNG, JPEG, WebP, and GIF only. SVG and remote URLs are excluded. Default limits are 5 MiB per image, 10 images and 20 MiB aggregate image bytes per message, and 40 million intrinsic pixels per image. These deployment-varying limits are validated backend configuration and are projected to the client for fast-path guidance; host validation remains authoritative.
|
||||
|
||||
Malformed base64, unsupported or mismatched media, truncated headers, excess bytes, excess image count, excess pixels, missing objects, and integrity mismatches return stable structured failures. Original filenames are reduced to a display basename, control characters are removed, and no local path is logged or returned to the browser.
|
||||
|
||||
### Package and surface changes
|
||||
|
||||
| Surface | Responsibility |
|
||||
| --- | --- |
|
||||
| `packages/attachment/attachment` | Opaque attachment identifier, image reference, limits, failures, and `ctx.attachments` service. |
|
||||
| `packages/attachment/attachment-local` | Private content-addressed storage, image-header validation, integrity verification, and configuration. |
|
||||
| `packages/llm/llm` and `packages/llm/token-meter` | Role-neutral `ImageBlock`, modality metadata, and image cost estimation. |
|
||||
| `packages/llm/llm-pi-ai` | Resolve durable supported image input into native provider content. |
|
||||
| `packages/llm/llm-deepseek` | Reject image content explicitly. |
|
||||
| `packages/host/apiproxy` and `packages/host/runtime` | Narrow upload wire, persist-before-event ordering, session-authorized reads, limits, and model preflight. |
|
||||
| `packages/client/connection` and `packages/client/runtime` | Wire types, fixture images, prompt uploads, attachment reads, and durable-reference folding. |
|
||||
| `packages/client/ui-conversation` | Per-session draft images, attachment rail, user and assistant image controls, and original preview. |
|
||||
| `packages/ui/acp` | Explicit fallback rendering for image blocks. |
|
||||
|
||||
The attachment packages form the interface/implementation side of one capability seam. Composer behavior stays in the conversation object layer, provider conversion stays in adapters, and no change is required in `agent-loop`.
|
||||
|
||||
### Delivery
|
||||
|
||||
1. Land the attachment seam, role-neutral image block, image-aware token estimation, Pi-AI input conversion, DeepSeek rejection, and durable host ordering.
|
||||
2. Land the Web upload/read protocol, in-memory draft images, paste/drop rail, user and assistant history rendering, double-click preview, and assembled keyless Web coverage.
|
||||
3. Add immediate intake-time capability feedback when active model selection is consistently available to the composer.
|
||||
4. Propose file picking, generic files/PDF, audio/video, durable draft staging, output-provider certification, and reference-aware garbage collection independently.
|
||||
|
||||
No compatibility shim is required for the pre-release prompt wire; all call sites and fixtures change with the introducing slice.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
### Keep every intake image in `/var` or another temporary directory
|
||||
|
||||
Temporary storage is appropriate before send, including for a native client that receives clipboard files through the operating system. It is not appropriate after acceptance: cleanup is outside the harness's control, paths are host-specific, and resume or fork can outlive the file. The proposal permits temporary staging but copies accepted bytes into `DSH_HOME` before the event.
|
||||
|
||||
### Persist immediately on paste or drop
|
||||
|
||||
Immediate persistence makes drafts reload-resistant but creates durable objects before a session or message owns them, which requires quota, orphan lifetime, and cleanup policy. Version one keeps the unsent draft temporary and makes send acceptance the durability boundary.
|
||||
|
||||
### Inline base64 in messages and session logs
|
||||
|
||||
This duplicates binary data across RPC, events, history pages, forks, compaction, and browser storage, and invites token accounting to treat encoding text as model text. One immutable object plus small references keeps the durable representation bounded.
|
||||
|
||||
### Use browser object URLs, local paths, or provider URLs as canonical content
|
||||
|
||||
Object URLs expire with the document, local paths are not portable, and provider URLs may expire, track viewers, or expose credentials. They remain temporary transport or preview details only.
|
||||
|
||||
### Use one generic `AttachmentBlock` for images, files, audio, and video
|
||||
|
||||
Composer presentation can use a generic attachment rail, but provider semantics are modality-specific. Images are native multimodal input; PDFs may be provider files or extracted text; video may be native, sampled, or unsupported. A specific `ImageBlock` forces every consumer to handle or reject the modality explicitly.
|
||||
|
||||
### Rely on UI capability checks or silently filter images
|
||||
|
||||
UI state can be stale and does not protect direct SDK, ACP, replay, or uncatalogued model paths. Silent filtering changes user intent. Provider enforcement remains mandatory, while UI checks are optional earlier feedback.
|
||||
|
||||
## Acceptance criteria
|
||||
|
||||
- Pasting or dropping one or more supported images shows ordered removable thumbnails above both composer variants without changing textarea text; drag-over highlights the target, unsupported drops cannot navigate away, and image-only send works.
|
||||
- Unsent browser images exist only as `File` and object URLs, survive session switches in memory, do not enter `localStorage`, and are revoked after removal, accepted send, or service disposal.
|
||||
- Every accepted user image is committed below resolved `DSH_HOME` before its `user/message` event. The event contains only `ImageBlock` references and never base64 or temporary paths.
|
||||
- Structured assistant images can be represented only by a durable `ImageBlock`; a future output adapter must persist bytes before emitting the assistant event, while Markdown image URLs remain text.
|
||||
- Cold history renders user and assistant image references through the same bounded control. Double-click opens the original; Escape, backdrop, and close control dismiss it without a custom context menu.
|
||||
- Session attachment reads fail unless the same session log references the identifier. Missing or corrupt objects fail explicitly and never return unverified bytes.
|
||||
- Pi-AI emits native input images for a compatible route. DeepSeek and every non-implementing consumer return an explicit unsupported-content failure rather than dropping the block.
|
||||
- An explicitly text-only active model rejects image send before attachment persistence or session event append; unknown metadata still reaches adapter enforcement and a failed send restores the draft.
|
||||
- Keyless unit, host integration, client integration, and assembled Chromium coverage exercise persistence ordering, absence of base64 in logs, authorization, paste and drop, image-only send, historical user and assistant images, original preview, and object-URL cleanup.
|
||||
- The current production adapter set declares text-only output; output-provider certification, file picking, non-image files, video, persistent drafts, and garbage collection remain outside version one.
|
||||
|
||||
## Risks
|
||||
|
||||
- Durable storage grows without garbage collection. Version one chooses replay safety over premature deletion.
|
||||
- A missing or corrupt object makes exact model reconstruction fail. Failing loud preserves integrity but may prevent that session from continuing until repaired.
|
||||
- JSON-RPC base64 adds upload memory and roughly one-third encoding overhead. Version-one limits bound it; larger media needs streaming or a binary transport.
|
||||
- Unsent images do not survive reload. Durable drafts need quota and orphan cleanup rather than reusing message storage implicitly.
|
||||
- Original preview decodes more pixels than the inline control displays. Pixel limits, one clicked preview, and object-URL disposal bound but do not eliminate transient browser memory.
|
||||
- Capability metadata may be missing or stale. Host preflight improves feedback, while adapter enforcement remains authoritative.
|
||||
- A future output provider may require authenticated retrieval before an assistant image can complete, adding latency and a new failure point. Persist-before-event ordering favors replay integrity.
|
||||
@@ -0,0 +1,210 @@
|
||||
# Agent Note: Web 多模态图片输入与持久附件
|
||||
|
||||
Status: proposed
|
||||
|
||||
[English](2026-07-22-web-multimodal-image-input-and-durable-attachments.md) | 中文
|
||||
|
||||
## 问题
|
||||
|
||||
Web 输入区目前仅接受文本：`InputBar` 接收字符串草稿，`ConversationService.send()` 创建文本内容，宿主再把该内容转发给 agent（智能体）。用户无法粘贴图片、在发送前查看图片、提交仅含图片的提示词，也无法从历史记录中恢复已发送图片。
|
||||
|
||||
这不只是输入区功能缺失。核心层需要持久图片内容块，提供方需要明确处理模态，会话日志则必须重建模型可见的全部内容。[此前移除图片块的决策](../../implemented/simplification/2026-07-04-drop-image-content-block.md)否决了可能静默丢失图片或将其展平的不完整设计。浏览器对象 URL、本地路径、提供方 URL 或 base64 数据都不能成为规范会话内容。
|
||||
|
||||
[Web 客户端架构](../../implemented/architecture/2026-07-19-gui-web-client-architecture.md)要求组件保持纯粹，并将每个会话的输入区状态放在 `ctx.conversation` 中；[GUI 分层与 RPC 协议](../../implemented/architecture/2026-07-19-gui-layering-and-rpc-protocol.md)则要求持久事件成为实时渲染与历史回放的共同真源。因此，图片接收、持久化、提供方转换和渲染需要遵循同一个明确的生命周期。
|
||||
|
||||
同类产品普遍在编辑器上方设置附件栏，但存储方案各不相同。诸如 `/var/folders/.../codex-clipboard-*.png` 的 Codex 式路径适合作为接收输入时的暂存位置，却不能作为持久消息身份：操作系统可能删除文件，另一台宿主无法读取文件，恢复后的会话也不能依赖文件仍然存在。
|
||||
|
||||
## 提案
|
||||
|
||||
把粘贴或拖放的光栅图片作为持久附件能力的首个消费方，接入 Web 输入区。未发送文件仍是由客户端持有的临时草稿状态。宿主在追加相应消息事件前，校验并持久提交每张已接受的用户图片。生成结构化图片输出的提供方适配器在追加相应助手块前，也必须持久提交输出。规范用户内容与助手内容只包含角色无关的 `ImageBlock` 引用。
|
||||
|
||||
第一版支持粘贴和拖放 PNG、JPEG、WebP 与 GIF，支持仅图片或混合提示词，支持渲染历史用户图片与助手图片，并支持双击预览原图。文件选择、通用文件、PDF、音频、视频、图片复制和自定义上下文菜单分别作为后续工作。
|
||||
|
||||
### 产品行为
|
||||
|
||||
- 粘贴或拖放一张或多张受支持的图片后，文本框上方会按顺序显示缩略图，但不会插入占位文本。文件拖入输入区时会高亮放置目标。
|
||||
- 空状态输入区与常驻输入区共用附件栏；附件栏为空时隐藏，通过横向滚动避免撑宽输入区。
|
||||
- 每个缩略图约为 72 × 72 像素，带有移除操作；双击时打开草稿原图。
|
||||
- 提示词可同时包含文本与图片，也可仅包含图片。粘贴纯文本时保持浏览器原生行为；粘贴处理器只有接受图片文件后才阻止默认行为。无论文件是否受支持，在输入区放置文件时都会阻止浏览器导航；系统会接受受支持的图片，并在本地提示哪些文件不受支持。
|
||||
- 发送失败时恢复完整的文本与图片草稿。移除、发送成功和会话作用域释放都会撤销过期的对象 URL。
|
||||
- 历史用户图片与助手图片共用一个 `MessageImage` 控件。行内图片保持固有宽高比、不放大，并限制在 240 × 240 像素的边界框内。
|
||||
- 双击消息图片会在不超出视口的模态框中打开存储的原图。按 Escape、激活关闭控件或激活背景区域都会关闭模态框并恢复焦点。
|
||||
- 第一版不覆盖浏览器上下文菜单，也不提供明确的图片复制操作。
|
||||
|
||||
### 存储生命周期与归属
|
||||
|
||||
持久化边界是消息被接受，而不是图片被粘贴：
|
||||
|
||||
| 状态 | 允许的表示 | 持久性与顺序 |
|
||||
| --- | --- | --- |
|
||||
| 未发送的用户草稿 | 浏览器 `File` 加对象 URL；原生客户端可以使用 `/var/...` 等操作系统临时文件 | 临时且由客户端持有。它可能在重载或进程退出后消失，绝不出现在会话事件中。 |
|
||||
| 已接受的用户图片 | `DSH_HOME` 下的不可变对象加 `ImageAttachmentRef` | 在 `agent.send()` 或 `agent.steer()` 能够追加所属用户事件前，宿主提交每张图片。 |
|
||||
| 结构化模型图片输出 | `DSH_HOME` 下的不可变对象加 `ImageAttachmentRef` | 提供方适配器在发出已完成的图片块或助手消息事件前提交字节。事件中禁止出现临时 URL、路径和 base64。 |
|
||||
|
||||
框架持有的 chat store 保存每个会话的草稿文本和有序附件标识符，`ConversationService` 则持有相应的浏览器专用 `File` 与对象 URL 注册表：
|
||||
|
||||
```ts
|
||||
export {}
|
||||
|
||||
interface ChatStoreState {
|
||||
selection: object | null
|
||||
draft: string
|
||||
imageIds: string[]
|
||||
view: string | null
|
||||
}
|
||||
|
||||
interface ComposerAttachment {
|
||||
id: string
|
||||
file: File
|
||||
previewUrl: string
|
||||
}
|
||||
```
|
||||
|
||||
这一拆分让 UI 状态通过 slots 框架的 store 席位和绑定 actions 使用唯一的订阅路径，同时避免把不可序列化的浏览器对象写进持久 JSON。草稿文本和有序图片标识符继续使用 `localStorage`；重载后，`ConversationRoot` 会清理缺少对应运行时对象的标识符。未发送图片因此无法跨重载保留，因为浏览器 `File` 与对象 URL 不具备持久性。原生客户端可以在操作系统临时目录中暂存输入，但必须像对待浏览器对象 URL 一样对待该路径：不再需要时删除，并在消息被接受前把字节复制进持久存储。
|
||||
|
||||
本地附件后端依次解析显式 `dshHome`、`$DSH_HOME` 和 `~/.dsh`。它把内容寻址对象存储在 `$DSH_HOME/attachments/v1/objects/<prefix>/<sha256>` 下，并为目录和文件设置仅所有者可访问的权限。服务先写入并同步临时文件，再以原子方式发布，之后才返回引用。内容摘要编码在不透明的 `sha256:<digest>` 标识符中，每次读取都会校验摘要、媒体类型、字节长度、宽度和高度。
|
||||
|
||||
第一版不对存储执行自动删除。已发送的用户图片和模型生成图片会一直保留，以供历史记录、恢复和 fork 使用。按引用感知的垃圾回收需要单独设计，因为仅按时间清理可能删除仍被持久会话引用的数据。
|
||||
|
||||
### 持久内容与提示词协议
|
||||
|
||||
附件服务边界公开不可变图片写入和经过校验的读取操作。规范元数据刻意比通用文件记录更窄：
|
||||
|
||||
```ts
|
||||
import type { Branded } from '@deepseek-ai/dsh-brand'
|
||||
|
||||
type AttachmentId = Branded<'AttachmentId'>
|
||||
|
||||
interface ImageAttachmentRef {
|
||||
attachmentId: AttachmentId
|
||||
mediaType: 'image/png' | 'image/jpeg' | 'image/webp' | 'image/gif'
|
||||
bytes: number
|
||||
width: number
|
||||
height: number
|
||||
name?: string
|
||||
}
|
||||
|
||||
interface ImageBlock {
|
||||
type: 'image'
|
||||
attachment: ImageAttachmentRef
|
||||
}
|
||||
```
|
||||
|
||||
`ImageBlock` 加入可合并扩展的核心 `ContentBlockMap`，在用户内容和助手内容中都有效。它绝不携带 base64、对象 URL、文件系统路径或提供方持有的定位符。因此，会话事件与不可变对象存储足以共同重建模型可见的确切图片。
|
||||
|
||||
浏览器无法生成持久引用，因此 `session.prompt` 接受范围狭窄的接收联合类型，而不是规范 `ContentBlock[]`：
|
||||
|
||||
```ts
|
||||
export {}
|
||||
|
||||
type PromptInputPart =
|
||||
| { type: 'text'; text: string }
|
||||
| {
|
||||
type: 'image'
|
||||
mediaType: 'image/png' | 'image/jpeg' | 'image/webp' | 'image/gif'
|
||||
data: string
|
||||
name?: string
|
||||
}
|
||||
```
|
||||
|
||||
Base64 只跨越一次 JSON-RPC，并在持久化后丢弃。宿主会校验规范 base64、图片数量、总字节数、单张图片字节数、与魔数字节一致的 MIME、固有尺寸和解码像素数。只有每张图片都成功后，宿主才会用规范化文本和持久图片块调用 agent。失败时不追加用户事件，也不公开任何附件路径或原始字节。
|
||||
|
||||
`session.attachment` 是只读且限定于会话作用域的端点。只有该会话中的持久事件引用了所请求的附件标识符，宿主才提供字节。客户端在服务生命周期内，以会话和附件标识符为键缓存生成的对象 URL，并在释放时撤销它。
|
||||
|
||||
### 模型能力与提供方行为
|
||||
|
||||
模型目录项增加可选且可合并扩展的输入与输出模态声明。缺少声明表示未知；声明存在但不含 `image`，则明确表示不支持图片。
|
||||
|
||||
宿主是权威的前置检查边界。如果所选模型明确排除图片输入，宿主会在写入任何附件或事件前拒绝提示词，客户端则恢复草稿。能力未知时继续进入适配器强制检查，使未收录的模型标识符仍然可用。在所有 Web 入口路径都能一致公开模型选择后，可以在 UI 中增加粘贴时立即拒绝的反馈。
|
||||
|
||||
Pi-AI 适配器是首条视觉输入路径：它通过 `ctx.attachments` 解析每个持久引用，并且仅为声明支持图片输入的模型生成提供方原生图片内容。手写 DeepSeek 适配器遇到请求中任何位置的图片时都会抛出类型化的 `UNSUPPORTED_CONTENT` 错误，包括嵌套工具结果中的图片。任何适配器都不得将图片展平或跳过。
|
||||
|
||||
核心层支持结构化助手图片块，但当前没有任何生产提供方路径通过图片输出认证。未来任何支持输出的适配器都必须在有界的大小和时间策略下获取提供方字节，通过同一个附件服务校验并持久化字节，之后才能以原子方式发布 `ImageBlock`。助手 Markdown 中的 URL 仍是文本，绝不自动下载。
|
||||
|
||||
token 估算根据图片尺寸计量，但不把 base64 或附件定位符作为文本计数。提供方返回的用量仍是权威值。在 ACP（Agent Client Protocol）接口原生支持图片前，ACP 会渲染明确的图片标记，而不是静默省略该块。
|
||||
|
||||
### 历史渲染与原图预览
|
||||
|
||||
历史记录折叠会在用户消息和助手消息中保留 `ImageBlock`。用户图片在文本上方靠尾端对齐；助手图片在叙述流中靠前端对齐。`MessageImage` 根据记录的尺寸派生稳定的行内边界框，通过会话授权加载器解析字节，使用 `object-fit: contain`，并将对象缺失或损坏转换为可重试的错误控件。
|
||||
|
||||
输入区缩略图和每个 `MessageImage` 各自持有临时原图预览状态，并调用同一个纯 `ImageLightbox`。模态框使用已经解析的原始对象 URL，只限制显示尺寸；它会聚焦关闭控件，并在关闭时把焦点恢复到先前的目标。
|
||||
|
||||
### 限制与信任边界
|
||||
|
||||
第一版仅接受 PNG、JPEG、WebP 和 GIF。不接受 SVG 和远程 URL。默认限制为每张图片 5 MiB、每条消息 10 张图片和 20 MiB 图片总字节数，以及每张图片 4,000 万个固有像素。这些随部署变化的限制属于经过校验的后端配置，并会投影给客户端以提供快速路径指引；宿主校验仍是权威结果。
|
||||
|
||||
格式错误的 base64、不支持或不匹配的媒体、截断的文件头、超出字节限制、超出图片数量、超出像素限制、对象缺失和完整性不匹配都会返回稳定的结构化错误。原始文件名只保留用于显示的末段，控制字符会被移除，并且任何本地路径都不会写入日志或返回浏览器。
|
||||
|
||||
### 包与接口变更
|
||||
|
||||
| 接口 | 职责 |
|
||||
| --- | --- |
|
||||
| `packages/attachment/attachment` | 不透明附件标识符、图片引用、限制、错误和 `ctx.attachments` 服务。 |
|
||||
| `packages/attachment/attachment-local` | 私有内容寻址存储、图片头校验、完整性校验和配置。 |
|
||||
| `packages/llm/llm` 和 `packages/llm/token-meter` | 角色无关的 `ImageBlock`、模态元数据和图片成本估算。 |
|
||||
| `packages/llm/llm-pi-ai` | 将持久且受支持的图片输入解析为提供方原生内容。 |
|
||||
| `packages/llm/llm-deepseek` | 明确拒绝图片内容。 |
|
||||
| `packages/host/apiproxy` 和 `packages/host/runtime` | 范围狭窄的上传协议、先持久化再追加事件的顺序、会话授权读取、限制和模型前置检查。 |
|
||||
| `packages/client/connection` 和 `packages/client/runtime` | 协议类型、fixture（测试前置数据）图片、提示词上传、附件读取和持久引用折叠。 |
|
||||
| `packages/client/ui-conversation` | 每个会话的草稿图片、附件栏、用户与助手图片控件和原图预览。 |
|
||||
| `packages/ui/acp` | 图片块的明确兜底渲染。 |
|
||||
|
||||
附件包（package）构成一个能力服务边界的接口与实现侧。输入区行为留在会话对象层，提供方转换留在适配器中，无需修改 `agent-loop`。
|
||||
|
||||
### 交付
|
||||
|
||||
1. 交付附件服务边界、角色无关的图片块、图片感知的 token 估算、Pi-AI 输入转换、DeepSeek 拒绝和宿主侧的持久化顺序。
|
||||
2. 交付 Web 上传与读取协议、内存草稿图片、支持粘贴与拖放的附件栏、用户与助手历史图片渲染、双击预览，以及组装后无需密钥的 Web 覆盖。
|
||||
3. 在输入区能够一致获取当前模型选择后，增加接收图片时立即提供的能力反馈。
|
||||
4. 分别为文件选择、通用文件与 PDF、音频与视频、持久草稿暂存、输出提供方认证和按引用感知的垃圾回收提出方案。
|
||||
|
||||
预发布提示词协议不需要兼容包装层；引入相应切片时会同时修改所有调用点和 fixture。
|
||||
|
||||
## 曾考虑的替代方案
|
||||
|
||||
### 将每张粘贴图片保留在 `/var` 或其他临时目录中
|
||||
|
||||
临时存储适合在发送前使用，也适合通过操作系统接收剪贴板文件的原生客户端。但它不适合在消息被接受后继续使用：清理不在 harness 的控制范围内，路径因宿主而异，恢复或 fork 后的会话也可能比文件存在得更久。提案允许临时暂存，但会在追加事件前将已接受的字节复制进 `DSH_HOME`。
|
||||
|
||||
### 粘贴或拖放后立即持久化
|
||||
|
||||
立即持久化可以让草稿在重载后继续存在，但会在会话或消息持有对象前就创建持久对象，因此必须定义配额、遗留对象生命周期和清理策略。第一版保持未发送草稿为临时状态，并把发送被接受作为持久性边界。
|
||||
|
||||
### 在消息与会话日志中内联 base64
|
||||
|
||||
这种方式会在 RPC、事件、历史分页、fork、压缩（compaction）和浏览器存储中复制二进制数据，还会诱使 token 计量把编码文本当成模型文本。单一不可变对象配合小型引用，可以让持久表示保持有界。
|
||||
|
||||
### 使用浏览器对象 URL、本地路径或提供方 URL 作为规范内容
|
||||
|
||||
对象 URL 会随文档失效，本地路径不可移植，提供方 URL 则可能过期、跟踪查看者或暴露凭据。它们只能作为临时传输或预览细节存在。
|
||||
|
||||
### 用一个通用 `AttachmentBlock` 表示图片、文件、音频和视频
|
||||
|
||||
输入区展示可以使用通用附件栏，但提供方语义取决于具体模态。图片是原生多模态输入；PDF 可能是提供方文件或提取后的文本；视频可能由模型原生支持、抽帧处理或不受支持。特定的 `ImageBlock` 会迫使每个消费方明确处理或拒绝该模态。
|
||||
|
||||
### 依赖 UI 能力检查或静默过滤图片
|
||||
|
||||
UI 状态可能陈旧，也无法保护直接 SDK、ACP、回放或未收录模型的路径。静默过滤会改变用户意图。提供方强制检查仍是必需项，UI 检查则是可选的提前反馈。
|
||||
|
||||
## 验收标准
|
||||
|
||||
- 在两种输入区中粘贴或拖放一张或多张受支持的图片时，文本框上方会按顺序显示可移除缩略图，且不更改文本框内容；拖入时会高亮目标，放置不支持的文件也不会触发页面跳转，仅图片的发送可正常工作。
|
||||
- 未发送的浏览器图片仅以 `File` 与对象 URL 的形式存在，可以在内存中跨会话切换保留，不进入 `localStorage`，并在移除、发送被接受或服务释放后撤销。
|
||||
- 每张已接受的用户图片都会提交到解析所得 `DSH_HOME` 下，之后才会追加相应的 `user/message` 事件。事件只包含 `ImageBlock` 引用，绝不包含 base64 或临时路径。
|
||||
- 结构化助手图片只能由持久 `ImageBlock` 表示；未来的输出适配器必须在发出助手事件前持久化字节，而 Markdown 图片 URL 仍是文本。
|
||||
- 冷启动历史记录通过同一个有界控件渲染用户与助手图片引用。双击打开原图；Escape、激活背景区域和激活关闭控件可以关闭预览，且不提供自定义上下文菜单。
|
||||
- 除非同一个会话日志引用了该标识符，否则会话附件读取会失败。对象缺失或损坏会明确失败，绝不返回未经校验的字节。
|
||||
- Pi-AI 会为兼容路径生成提供方原生输入图片。DeepSeek 与所有未实现该能力的消费方返回明确的不支持内容错误，而不是丢弃该块。
|
||||
- 明确仅支持文本的当前模型会在持久化附件或追加会话事件前拒绝图片发送；未知元数据仍会到达适配器强制检查，发送失败则恢复草稿。
|
||||
- 无需密钥的单元测试、宿主集成测试、客户端集成测试和组装应用的 Chromium 覆盖会验证持久化顺序、日志中不含 base64、授权、粘贴与拖放、仅图片发送、历史用户与助手图片、原图预览和对象 URL 清理。
|
||||
- 当前生产适配器集合声明仅支持文本输出；输出提供方认证、文件选择、非图片文件、视频、持久草稿和垃圾回收不在第一版范围内。
|
||||
|
||||
## 风险
|
||||
|
||||
- 持久存储会在没有垃圾回收时持续增长。第一版选择回放安全，而不是过早删除。
|
||||
- 对象缺失或损坏会让模型请求无法精确重建。明确失败可以保持完整性，但在修复前可能阻止该会话继续运行。
|
||||
- JSON-RPC base64 会增加上传内存，并带来约三分之一的编码开销。第一版的限制可以约束开销；更大的媒体需要流式传输或二进制传输协议。
|
||||
- 未发送图片无法跨重载保留。持久草稿需要配额和遗留对象清理，而不是隐式复用消息存储。
|
||||
- 原图预览解码的像素多于行内控件显示的像素。像素限制、一次只打开一个预览和对象 URL 释放可以约束但无法消除浏览器瞬时内存占用。
|
||||
- 能力元数据可能缺失或陈旧。宿主前置检查可以改善反馈，适配器强制检查仍是权威结果。
|
||||
- 未来输出提供方可能需要经过身份认证的下载，助手图片才能完成，这会增加延迟与新的故障点。先持久化再追加事件的顺序优先保障回放完整性。
|
||||
@@ -3,8 +3,8 @@
|
||||
// chromium. First describe: manifest injection + fail-loud half. Second
|
||||
// describe: the settled success pass — five REAL tsdown bundles (the
|
||||
// infrastructure four + layout) load through the DI chain in ?fixture mode
|
||||
// and the three-column frame appears in one flip. The full conversation
|
||||
// round lands in smoke-real under the W5 real-host standard.
|
||||
// and the three-column frame appears in one flip. The full eight-plugin pass
|
||||
// also exercises durable history images without a model key.
|
||||
import { existsSync } from 'node:fs'
|
||||
import { fileURLToPath } from 'node:url'
|
||||
import type { Browser, Page } from 'playwright'
|
||||
@@ -17,21 +17,24 @@ import { DIST_INDEX, probeFreePort, requireDist, saveFailureShot } from './suppo
|
||||
const bundlePath = (dir: string): string =>
|
||||
fileURLToPath(new URL(`../../../packages/client/${dir}/lib/client.js`, import.meta.url))
|
||||
|
||||
/** id ↔ bundle table for the success pass (immediately four + layout). */
|
||||
/** id ↔ bundle table for the success pass (the production Web plugin chain). */
|
||||
const REAL_PLUGINS: { id: string; dir: string; inject: string[]; immediately?: boolean }[] = [
|
||||
{ id: '@deepseek-ai/dsh-client-connection', dir: 'connection', inject: [], immediately: true },
|
||||
{ id: '@deepseek-ai/dsh-client-runtime', dir: 'runtime', inject: ['@deepseek-ai/dsh-client-connection'], immediately: true },
|
||||
{ id: '@deepseek-ai/dsh-client-ui-theme', dir: 'ui-theme', inject: [], immediately: true },
|
||||
{ id: '@deepseek-ai/dsh-client-i18n', dir: 'i18n', inject: [], immediately: true },
|
||||
{ id: '@deepseek-ai/dsh-client-ui-layout', dir: 'ui-layout', inject: ['@deepseek-ai/dsh-client-runtime'] },
|
||||
{ id: '@deepseek-ai/dsh-client-ui-sidebar', dir: 'ui-sidebar', inject: ['@deepseek-ai/dsh-client-ui-layout'] },
|
||||
{ id: '@deepseek-ai/dsh-client-ui-conversation', dir: 'ui-conversation', inject: ['@deepseek-ai/dsh-client-ui-layout'] },
|
||||
{ id: '@deepseek-ai/dsh-client-ui-trajectory', dir: 'ui-trajectory', inject: ['@deepseek-ai/dsh-client-ui-conversation'] },
|
||||
]
|
||||
|
||||
/** Manifest served by the fake registry: one live bundle row, one missing row. */
|
||||
const ROWS: WebPluginBootEntry[] = [
|
||||
{ id: '@deepseek-ai/dsh-client-ui-layout', url: '/plugins/@deepseek-ai/dsh-client-ui-layout/client.js', inject: [] },
|
||||
{ id: '@deepseek-ai/dsh-client-ui-theme', url: '/plugins/@deepseek-ai/dsh-client-ui-theme/client.js', inject: [] },
|
||||
{ id: '@probe/absent', url: '/plugins/@probe/absent/client.js', inject: [] },
|
||||
]
|
||||
const LAYOUT_BUNDLE = bundlePath('ui-layout')
|
||||
const LIVE_BUNDLE = bundlePath('ui-theme')
|
||||
|
||||
describe('web boot chain (keyless, real carrier)', () => {
|
||||
let server: Awaited<ReturnType<typeof startWebServer>>
|
||||
@@ -50,7 +53,7 @@ describe('web boot chain (keyless, real carrier)', () => {
|
||||
apiHandler,
|
||||
webPlugins: {
|
||||
snapshot: () => ROWS,
|
||||
clientPath: id => (id === ROWS[0]!.id ? LAYOUT_BUNDLE : undefined),
|
||||
clientPath: id => (id === ROWS[0]!.id ? LIVE_BUNDLE : undefined),
|
||||
},
|
||||
}, (err) => { pageErrors.push(`server: ${String(err)}`) })
|
||||
browser = await chromium.launch()
|
||||
@@ -91,7 +94,7 @@ describe('web boot chain (keyless, real carrier)', () => {
|
||||
})
|
||||
})
|
||||
|
||||
describe('web boot chain success pass (keyless, five real bundles, ?fixture)', () => {
|
||||
describe('web boot chain success pass (keyless, production plugin chain, ?fixture)', () => {
|
||||
const missing = REAL_PLUGINS.filter(p => !existsSync(bundlePath(p.dir)))
|
||||
let server: Awaited<ReturnType<typeof startWebServer>>
|
||||
let browser: Browser
|
||||
@@ -143,6 +146,88 @@ describe('web boot chain success pass (keyless, five real bundles, ?fixture)', (
|
||||
expect(owners).toContain('@deepseek-ai/dsh-client-ui-layout')
|
||||
})
|
||||
|
||||
it('renders historical user and assistant images and opens the original on double-click', async () => {
|
||||
onTestFailed(() => saveFailureShot(page, 'smoke-fixture-images'))
|
||||
await page.getByRole('treeitem', { name: /fixture 3 sessions/ }).click()
|
||||
await page.locator('[role="treeitem"][aria-selected]').first().click()
|
||||
await page.waitForSelector('text=历史用户图片', { timeout: 15_000 })
|
||||
await page.waitForSelector('text=结构化模型图片', { timeout: 15_000 })
|
||||
const images = page.getByTitle('双击查看原图')
|
||||
await expect.poll(() => images.count()).toBeGreaterThanOrEqual(2)
|
||||
const first = images.first()
|
||||
const box = await first.boundingBox()
|
||||
expect(box?.width).toBeLessThanOrEqual(240)
|
||||
expect(box?.height).toBeLessThanOrEqual(240)
|
||||
await first.dblclick()
|
||||
const preview = page.getByRole('dialog', { name: '原图预览' })
|
||||
await preview.waitFor({ state: 'visible' })
|
||||
await page.keyboard.press('Escape')
|
||||
await preview.waitFor({ state: 'detached' })
|
||||
})
|
||||
|
||||
it('pastes and drops images into the composer, then sends them as durable history', async () => {
|
||||
onTestFailed(() => saveFailureShot(page, 'smoke-fixture-image-paste'))
|
||||
await page.getByRole('button', { name: '停止' }).click()
|
||||
const textarea = page.locator('textarea')
|
||||
await textarea.waitFor({ state: 'visible' })
|
||||
await expect.poll(() => textarea.isEnabled()).toBe(true)
|
||||
await textarea.evaluate((element) => {
|
||||
const binary = atob('iVBORw0KGgoAAAANSUhEUgAAAKAAAABaCAYAAAA/xl1SAAAAvklEQVR42u3SMQ0AAAjAMIyhELM4AAe8PD1qYFlk9cCXEAEDYkAwIAYEA2JAMCAGBANiQDAgBgQDYkAwIAYEA2JAMCAGBANiQDAgBgQDYkAwIAYEA2JAMCAGxIBCYEAMCAbEgGBADAgGxIBgQAwIBsSAYEAMCAbEgGBADAgGxIBgQAwIBsSAYEAMCAbEgGBADAgGxIAYEAyIAcGAGBAMiAHBgBgQDIgBwYAYEAyIAcGAGBAMiAHBgBgQDIgB4bYWLb6pnOb1xAAAAABJRU5ErkJggg==')
|
||||
const bytes = Uint8Array.from(binary, character => character.charCodeAt(0))
|
||||
const transfer = new DataTransfer()
|
||||
transfer.items.add(new File([bytes], 'clipboard.png', { type: 'image/png' }))
|
||||
element.dispatchEvent(new ClipboardEvent('paste', {
|
||||
bubbles: true,
|
||||
cancelable: true,
|
||||
clipboardData: transfer,
|
||||
}))
|
||||
})
|
||||
|
||||
const rail = page.getByLabel('待发送图片')
|
||||
await rail.waitFor({ state: 'visible' })
|
||||
const draftImage = rail.getByTitle('双击查看原图')
|
||||
await draftImage.dblclick()
|
||||
const preview = page.getByRole('dialog', { name: '原图预览' })
|
||||
await preview.waitFor({ state: 'visible' })
|
||||
await page.keyboard.press('Escape')
|
||||
await preview.waitFor({ state: 'detached' })
|
||||
|
||||
await page.getByRole('button', { name: '发送' }).click()
|
||||
await rail.waitFor({ state: 'detached' })
|
||||
await expect.poll(() => page.getByTitle('双击查看原图').count()).toBeGreaterThanOrEqual(3)
|
||||
|
||||
await page.getByRole('button', { name: '停止' }).click()
|
||||
await expect.poll(() => textarea.isEnabled()).toBe(true)
|
||||
await textarea.evaluate((element) => {
|
||||
const binary = atob('iVBORw0KGgoAAAANSUhEUgAAAKAAAABaCAYAAAA/xl1SAAAAvklEQVR42u3SMQ0AAAjAMIyhELM4AAe8PD1qYFlk9cCXEAEDYkAwIAYEA2JAMCAGBANiQDAgBgQDYkAwIAYEA2JAMCAGBANiQDAgBgQDYkAwIAYEA2JAMCAGxIBCYEAMCAbEgGBADAgGxIBgQAwIBsSAYEAMCAbEgGBADAgGxIBgQAwIBsSAYEAMCAbEgGBADAgGxIAYEAyIAcGAGBAMiAHBgBgQDIgBwYAYEAyIAcGAGBAMiAHBgBgQDIgB4bYWLb6pnOb1xAAAAABJRU5ErkJggg==')
|
||||
const bytes = Uint8Array.from(binary, character => character.charCodeAt(0))
|
||||
const transfer = new DataTransfer()
|
||||
transfer.items.add(new File([bytes], 'dropped.png', { type: 'image/png' }))
|
||||
element.closest('[class*="card"]')?.dispatchEvent(new DragEvent('dragenter', {
|
||||
bubbles: true,
|
||||
cancelable: true,
|
||||
dataTransfer: transfer,
|
||||
}))
|
||||
})
|
||||
await page.getByRole('status').filter({ hasText: '松开以添加图片' }).waitFor({ state: 'visible' })
|
||||
await textarea.evaluate((element) => {
|
||||
const binary = atob('iVBORw0KGgoAAAANSUhEUgAAAKAAAABaCAYAAAA/xl1SAAAAvklEQVR42u3SMQ0AAAjAMIyhELM4AAe8PD1qYFlk9cCXEAEDYkAwIAYEA2JAMCAGBANiQDAgBgQDYkAwIAYEA2JAMCAGBANiQDAgBgQDYkAwIAYEA2JAMCAGxIBCYEAMCAbEgGBADAgGxIBgQAwIBsSAYEAMCAbEgGBADAgGxIBgQAwIBsSAYEAMCAbEgGBADAgGxIAYEAyIAcGAGBAMiAHBgBgQDIgBwYAYEAyIAcGAGBAMiAHBgBgQDIgB4bYWLb6pnOb1xAAAAABJRU5ErkJggg==')
|
||||
const bytes = Uint8Array.from(binary, character => character.charCodeAt(0))
|
||||
const transfer = new DataTransfer()
|
||||
transfer.items.add(new File([bytes], 'dropped.png', { type: 'image/png' }))
|
||||
element.closest('[class*="card"]')?.dispatchEvent(new DragEvent('drop', {
|
||||
bubbles: true,
|
||||
cancelable: true,
|
||||
dataTransfer: transfer,
|
||||
}))
|
||||
})
|
||||
await rail.waitFor({ state: 'visible' })
|
||||
await rail.getByAltText('dropped.png').waitFor({ state: 'visible' })
|
||||
await page.getByRole('button', { name: '发送' }).click()
|
||||
await rail.waitFor({ state: 'detached' })
|
||||
await expect.poll(() => page.getByTitle('双击查看原图').count()).toBeGreaterThanOrEqual(4)
|
||||
})
|
||||
|
||||
it('stayed clean: no page errors across the whole load chain', () => {
|
||||
expect(pageErrors).toEqual([])
|
||||
})
|
||||
|
||||
@@ -7,10 +7,14 @@ A service can be a core spine service, a swappable capability seam, or a bundle/
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
pkg_attachment["attachment"]
|
||||
svc_attachments["ctx.attachments<br/>Durable binary attachment storage"]
|
||||
pkg_attachment_local["attachment-local"]
|
||||
pkg_host_runtime["host-runtime"]
|
||||
pkg_llm_pi_ai["llm-pi-ai"]
|
||||
pkg_llm["llm"]
|
||||
svc_llm["ctx.llm<br/>LLM adapter registry"]
|
||||
pkg_llm_deepseek["llm-deepseek"]
|
||||
pkg_llm_pi_ai["llm-pi-ai"]
|
||||
pkg_llm_replay["llm-replay"]
|
||||
pkg_agent_loop["agent-loop"]
|
||||
pkg_compact_basic["compact-basic"]
|
||||
@@ -125,6 +129,8 @@ flowchart LR
|
||||
pkg_agent --> svc_agents
|
||||
pkg_agent_loop --> svc_agentLoop
|
||||
pkg_approval --> svc_approval
|
||||
pkg_attachment --> svc_attachments
|
||||
pkg_attachment_local --> svc_attachments
|
||||
pkg_bash --> svc_bash
|
||||
pkg_bash_local --> svc_bash
|
||||
pkg_bash_sandbox --> svc_bash
|
||||
@@ -189,6 +195,8 @@ flowchart LR
|
||||
svc_agents --> pkg_tui_demo
|
||||
svc_approval --> pkg_tool_bash
|
||||
svc_approval --> pkg_tools
|
||||
svc_attachments --> pkg_host_runtime
|
||||
svc_attachments --> pkg_llm_pi_ai
|
||||
svc_bash --> pkg_hooks_claude
|
||||
svc_bash --> pkg_hooks_codex
|
||||
svc_bash --> pkg_tool_bash
|
||||
@@ -263,6 +271,7 @@ flowchart LR
|
||||
|
||||
| ctx key | Role | Owner | Implementations | Direct consumers | Companion plugins | Note |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| `ctx.attachments` | `seam` | [`attachment`](../packages/attachment/attachment) | [`attachment-local`](../packages/attachment/attachment-local) | [`host-runtime`](../packages/host/runtime), [`llm-pi-ai`](../packages/llm/llm-pi-ai) | - | The host commits accepted images before session events; provider adapters resolve authorized durable references into provider-native content. |
|
||||
| `ctx.llm` | `seam` | [`llm`](../packages/llm/llm) | [`llm-deepseek`](../packages/llm/llm-deepseek), [`llm-pi-ai`](../packages/llm/llm-pi-ai), [`llm-replay`](../packages/support/llm-replay) | [`agent-loop`](../packages/core/agent-loop), [`compact-basic`](../packages/compact/compact-basic) | - | Adapters register provider implementations; the loop and compaction call the provider-neutral stream service. |
|
||||
| `ctx.tokenMeter` | `core` | [`token-meter`](../packages/llm/token-meter) | - | [`compact-basic`](../packages/compact/compact-basic) | - | Owns isolated per-session replay folds; pressure consumers share immutable revisioned measurements. |
|
||||
| `ctx.toolResultPrune` | `core` | [`compact-tool-result-prune`](../packages/compact/compact-tool-result-prune) | - | [`compact-basic`](../packages/compact/compact-basic) | - | Rewrites oversized current tool results through replayable single-node surface replacements before summary compaction. |
|
||||
|
||||
@@ -193,6 +193,26 @@ Depends on: [`AgentLoopConfig`](#deepseek-aidsh-agent-loop) · [`GoalDomainConfi
|
||||
|
||||
Source: [`packages/examples/agent-spine-demo/src/index.ts:87`](../packages/examples/agent-spine-demo/src/index.ts)
|
||||
|
||||
## `@deepseek-ai/dsh-attachment-local`
|
||||
|
||||
```ts config-catalog
|
||||
/** Local attachment backend configuration. */
|
||||
export interface Config {
|
||||
/** Explicit harness home; omitted follows `DSH_HOME`, then `~/.dsh`. */
|
||||
dshHome?: string
|
||||
/** Maximum encoded bytes accepted for one image. */
|
||||
maxImageBytes?: number
|
||||
/** Maximum image count accepted in one submitted message. */
|
||||
maxImagesPerMessage?: number
|
||||
/** Maximum aggregate encoded image bytes accepted in one submitted message. */
|
||||
maxMessageImageBytes?: number
|
||||
/** Maximum intrinsic width multiplied by height accepted for one image. */
|
||||
maxImagePixels?: number
|
||||
}
|
||||
```
|
||||
|
||||
Source: [`packages/attachment/attachment-local/src/index.ts:26`](../packages/attachment/attachment-local/src/index.ts)
|
||||
|
||||
## `@deepseek-ai/dsh-bash-local`
|
||||
|
||||
```ts config-catalog
|
||||
@@ -1881,6 +1901,7 @@ These load from a `cordis.yml` entry with no `config:` block; they declare no co
|
||||
|
||||
Abstract service classes — a deployment loads a concrete implementation package instead ([capability seams](../.agents/notes/implemented/architecture/2026-06-13-capability-seams.md)).
|
||||
|
||||
- `@deepseek-ai/dsh-attachment` — abstract `AttachmentStore` ([`packages/attachment/attachment/src/index.ts`](../packages/attachment/attachment/src/index.ts))
|
||||
- `@deepseek-ai/dsh-bash` — abstract `BashExecutor` ([`packages/bash/bash/src/index.ts`](../packages/bash/bash/src/index.ts))
|
||||
- `@deepseek-ai/dsh-code-runtime` — abstract `CodeRuntime` ([`packages/code-runtime/code-runtime/src/index.ts`](../packages/code-runtime/code-runtime/src/index.ts))
|
||||
- `@deepseek-ai/dsh-compact` — abstract `CompactService` ([`packages/compact/compact/src/index.ts`](../packages/compact/compact/src/index.ts))
|
||||
|
||||
@@ -248,6 +248,30 @@ Types: [ApprovalOutcome](../core-data-structures/approval.md) · [ApprovalReques
|
||||
|
||||
Source: [`packages/ui/user-approval/src/index.ts:213`](../../packages/ui/user-approval/src/index.ts)
|
||||
|
||||
## `ctx.attachments` — `AttachmentStore` (abstract seam)
|
||||
|
||||
Immutable binary attachment service. Implementations validate bytes before publishing a reference.
|
||||
|
||||
```ts cordis-catalog
|
||||
/**
|
||||
* Validate and durably commit one image before its owning session event is appended.
|
||||
* @param input - encoded bytes, declared media type, and optional display name.
|
||||
* @returns a durable content-addressed reference.
|
||||
*/
|
||||
abstract saveImage(input: SaveImageAttachment): Promise<ImageAttachmentRef>
|
||||
|
||||
/**
|
||||
* Read one image and verify that bytes still match the recorded reference.
|
||||
* @param ref - durable reference from the session log.
|
||||
* @returns the verified bytes and canonical reference.
|
||||
*/
|
||||
abstract readImage(ref: ImageAttachmentRef): Promise<StoredImageAttachment>
|
||||
```
|
||||
|
||||
Types: [ImageAttachmentRef](../core-data-structures/attachment.md) · [SaveImageAttachment](../core-data-structures/attachment.md) · [StoredImageAttachment](../core-data-structures/attachment.md)
|
||||
|
||||
Source: [`packages/attachment/attachment/src/index.ts:28`](../../packages/attachment/attachment/src/index.ts)
|
||||
|
||||
## `ctx.bash` — `BashExecutor` (abstract seam)
|
||||
|
||||
Abstract bash execution service. Subclass, implement the abstract methods, and load the subclass as a plugin — it registers as `ctx.bash` (one implementation per context; loading a second throws, which is cordis' standard duplicate-service behavior).
|
||||
@@ -1493,7 +1517,7 @@ estimateMessage(message: Message): number
|
||||
|
||||
Types: [EpochHeader](../core-data-structures/session.md) · [Message](../core-data-structures/core.md) · [Session](../core-data-structures/session.md) · [TokenMeasurement](../core-data-structures/token-meter.md)
|
||||
|
||||
Source: [`packages/llm/token-meter/src/index.ts:82`](../../packages/llm/token-meter/src/index.ts)
|
||||
Source: [`packages/llm/token-meter/src/index.ts:87`](../../packages/llm/token-meter/src/index.ts)
|
||||
|
||||
## `ctx.toolResultPrune` — `ToolResultPruneService`
|
||||
|
||||
|
||||
70
docs/core-data-structures/attachment.md
Normal file
70
docs/core-data-structures/attachment.md
Normal file
@@ -0,0 +1,70 @@
|
||||
# Durable Image Attachments
|
||||
|
||||
The attachment seam separates binary image ownership from the session log. A producer gives validated encoded bytes to [`ctx.attachments`](../cordis-catalog/services.md#ctxattachments); the service publishes an immutable content-addressed reference only after the object is durable. Session events and model-visible `ImageBlock`s contain that reference and metadata, never a browser object URL, host temporary path, provider URL, or base64 payload.
|
||||
|
||||
Unsent browser drafts may stay in memory and native clients may stage them in operating-system temporary storage. Once the host accepts a user message, its images move below `<DSH_HOME>/attachments/v1` before the user event is appended. Structured model image output follows the same persist-before-event rule.
|
||||
|
||||
Source: [`packages/attachment/attachment/src/types.ts`](../../packages/attachment/attachment/src/types.ts)
|
||||
|
||||
## Identity and verified metadata
|
||||
|
||||
`AttachmentId` is a branded opaque string. The local backend currently emits `sha256:<digest>`, but consumers must neither parse that representation nor derive a filesystem path from it.
|
||||
|
||||
```ts type-equiv
|
||||
/** Raster image formats accepted by the version-one attachment path. */
|
||||
type ImageMediaType = 'image/png' | 'image/jpeg' | 'image/webp' | 'image/gif'
|
||||
```
|
||||
|
||||
```ts type-equiv
|
||||
/** Durable, serializable metadata for one immutable image object. */
|
||||
interface ImageAttachmentRef {
|
||||
/** Opaque storage identifier; never a filesystem path or bearer URL. */
|
||||
attachmentId: AttachmentId
|
||||
/** Media type verified from the stored bytes. */
|
||||
mediaType: ImageMediaType
|
||||
/** Exact encoded byte length. */
|
||||
bytes: number
|
||||
/** Intrinsic encoded width in pixels. */
|
||||
width: number
|
||||
/** Intrinsic encoded height in pixels. */
|
||||
height: number
|
||||
/** Optional display name stripped of local path information. */
|
||||
name?: string
|
||||
}
|
||||
```
|
||||
|
||||
```ts type-equiv
|
||||
/** Deployment-resolved limits shared by upload consumers and UI preflight. */
|
||||
interface ImageAttachmentLimits {
|
||||
maxImageBytes: number
|
||||
maxImagesPerMessage: number
|
||||
maxMessageImageBytes: number
|
||||
maxImagePixels: number
|
||||
mediaTypes: readonly ImageMediaType[]
|
||||
}
|
||||
```
|
||||
|
||||
The reference records intrinsic dimensions and encoded length so clients can lay out history without decoding first, while every authoritative read still re-checks digest, media signature, dimensions, and metadata against the object.
|
||||
|
||||
## Commit and verified-read payloads
|
||||
|
||||
```ts type-equiv
|
||||
/** Request to validate and durably commit one image. */
|
||||
interface SaveImageAttachment {
|
||||
data: Uint8Array
|
||||
/** Caller-declared media type, checked against magic bytes. */
|
||||
mediaType: ImageMediaType
|
||||
/** Optional browser/provider display name; it is never interpreted as a path. */
|
||||
name?: string
|
||||
}
|
||||
```
|
||||
|
||||
```ts type-equiv
|
||||
/** Stored image bytes returned after reference and digest verification. */
|
||||
interface StoredImageAttachment {
|
||||
ref: ImageAttachmentRef
|
||||
data: Uint8Array
|
||||
}
|
||||
```
|
||||
|
||||
`saveImage()` validates bytes and atomically commits one object before returning its reference. `readImage()` accepts a reference from an authorized session path and returns bytes only after integrity verification. The service is deliberately retention-neutral: resumed and forked sessions may share objects, so reference-aware garbage collection is deferred rather than tied to any one session's deletion.
|
||||
@@ -28,6 +28,7 @@ Everything else is documented on a **sub-page**, not here. The rule that draws t
|
||||
| [tools.md](tools.md) | `ToolDefinition` full fields, the schema DSL, `ToolExecution`/`ToolResult`, tool-presentation UI types, and the guarded execution pipeline |
|
||||
| [user-interaction.md](user-interaction.md) | the UI-backed human question/answer seam: `AskUserQuestionRequest`, answer/options vocabulary, provider API, error taxonomy |
|
||||
| [approval.md](approval.md) | the one-shot user-approval seam: `ApprovalRequest`, `ApprovalOutcome`, per-session policy, audit and answerer contracts |
|
||||
| [attachment.md](attachment.md) | durable image identity and metadata, validation inputs, verified reads, and the `AttachmentStore` seam |
|
||||
| [bash.md](bash.md) | the bash executor seam: `BashExecRequest`/`Spec`, `BashRunResult`, background `BashProcess` handles |
|
||||
| [pty.md](pty.md) | persistent terminal ids, backend/session contracts, send readiness, bounded reads, and owner-visible snapshots |
|
||||
| [sandbox.md](sandbox.md) | per-session policy resolution and the process-confinement seam: file-effect modes, execution/provider policies, `ConfinedArgv`, enforcement and fail-closed errors |
|
||||
@@ -106,12 +107,13 @@ Source: [`packages/llm/llm/src/types.ts`](../../packages/llm/llm/src/types.ts)
|
||||
interface ContentBlockMap {
|
||||
'text': TextBlock
|
||||
'reasoning': ReasoningBlock
|
||||
'image': ImageBlock
|
||||
'tool-call': ToolCallBlock
|
||||
'tool-result': ToolResultBlock
|
||||
}
|
||||
```
|
||||
|
||||
The block interfaces (full fields in source): `TextBlock` (`text`), `ReasoningBlock` (thinking, distinct from visible text), `ToolCallBlock` (`id: CallId`, `name`, raw-JSON `arguments`), `ToolResultBlock` (`toolCallId`, nested `content: ContentBlock[]`, `isError?`). `ContentBlock = ContentBlockMap[ContentBlockType]`. The core set is limited to blocks every shipping path honors — multimodal content (images, audio, …) has no core block type; a feature that needs one adds it via the merge-extensible map together with the adapter/UI/compaction support that honors it.
|
||||
The block interfaces (full fields in source): `TextBlock` (`text`), `ReasoningBlock` (thinking, distinct from visible text), `ImageBlock` (a durable [image attachment](attachment.md) plus optional alternative text), `ToolCallBlock` (`id: CallId`, `name`, raw-JSON `arguments`), and `ToolResultBlock` (`toolCallId`, nested `content: ContentBlock[]`, `isError?`). `ContentBlock = ContentBlockMap[ContentBlockType]`. A new modality belongs in the merge-extensible map only when its adapter, UI, compaction, and durable replay paths honor it.
|
||||
|
||||
A `Message` is a role plus blocks. Loop-derived assistant messages carry their durable provider/model identity and optional adapter-private replay metadata:
|
||||
|
||||
@@ -192,6 +194,10 @@ interface LlmModelInfo {
|
||||
name: string
|
||||
/** Optional user-facing distinction from otherwise similar models. */
|
||||
description?: string
|
||||
/** Accepted request modalities; absent means unknown, while an explicit omission is negative capability. */
|
||||
inputModalities?: readonly ModelModality[]
|
||||
/** Structured response modalities; absent means unknown, while an explicit omission is negative capability. */
|
||||
outputModalities?: readonly ModelModality[]
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
@@ -208,6 +208,7 @@ declare abstract class LlmAdapter {
|
||||
interface ContentBlockMap {
|
||||
'text': TextBlock
|
||||
'reasoning': ReasoningBlock
|
||||
'image': ImageBlock
|
||||
'tool-call': ToolCallBlock
|
||||
'tool-result': ToolResultBlock
|
||||
}
|
||||
|
||||
@@ -197,6 +197,11 @@
|
||||
"src/**/*.ts"
|
||||
]
|
||||
},
|
||||
"packages/attachment/attachment": {
|
||||
"project": [
|
||||
"src/**/*.ts"
|
||||
]
|
||||
},
|
||||
"packages/util/timeout": {
|
||||
"entry": [
|
||||
"tests/**/*.spec.ts"
|
||||
|
||||
@@ -24,6 +24,7 @@ Packages live at `packages/<group>/<pkg>/`; groups are containers, while names r
|
||||
| [`tasks/`](tasks/README.md) | Generic background-task runtime and model-facing `task_*` control tools | Product — stable surface |
|
||||
| [`workflow/`](workflow/README.md) | Workflow capability family: the script-engine seam, worker-thread engine, and model-facing `workflow` and fresh-agent `ralph` tools | Product — stable surface |
|
||||
| [`web/`](web/README.md) | Web capability family: seam, search/fetch provider impls, and the model-facing web tools | Product — stable surface |
|
||||
| [`attachment/`](attachment/README.md) | Durable attachment seam and DSH_HOME backend | Product — stable surface |
|
||||
| [`spill/`](spill/README.md) | Spill capability family: storage seam, local impl, tool-result spill policy | Product — stable surface |
|
||||
| [`todo/`](todo/README.md) | Todo/planning family: the model-facing `todo_write` tool | Product — stable surface |
|
||||
| [`plan/`](plan/README.md) | Plan collaboration state with a direct entry command and reviewed exit | Product — stable surface |
|
||||
|
||||
10
packages/attachment/README.md
Normal file
10
packages/attachment/README.md
Normal file
@@ -0,0 +1,10 @@
|
||||
# attachment/ - durable attachment capability family
|
||||
|
||||
The durable binary attachment seam and its local filesystem implementation. Both are product packages.
|
||||
|
||||
| Package | Role | ctx key |
|
||||
|---|---|---|
|
||||
| `attachment/` | Immutable attachment references, image limits, and storage service | `ctx.attachments` |
|
||||
| `attachment-local/` | Content-addressed private storage below `DSH_HOME` | (registers on `ctx.attachments`) |
|
||||
|
||||
Unsent browser drafts are intentionally outside this capability. Bytes enter durable storage only when a user prompt is submitted or when a provider adapter commits structured model output.
|
||||
19
packages/attachment/attachment-local/README.md
Normal file
19
packages/attachment/attachment-local/README.md
Normal file
@@ -0,0 +1,19 @@
|
||||
# @deepseek-ai/dsh-attachment-local
|
||||
|
||||
The private local implementation of [`@deepseek-ai/dsh-attachment`](../attachment). Objects land at `<DSH_HOME>/attachments/v1/objects/<sha256-prefix>/<sha256>` and are addressed by an opaque `sha256:` id. Writes use a private staging directory, owner-only files, a synced temporary file, and an atomic exclusive hard-link publish; reads re-check the digest, media signature, dimensions, and logged metadata.
|
||||
|
||||
`DSH_HOME` resolves through the shared path policy: explicit config, `$DSH_HOME`, then `~/.dsh`. Session logs contain only the reference and verified metadata, never this host path.
|
||||
|
||||
## Model Experience
|
||||
|
||||
Indirectly, through durable replay of historical user images and structured model image output after restart and fork.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
None beyond the image block owned by the requesting adapter.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- Objects are retained indefinitely; reference-aware garbage collection is deferred.
|
||||
- The local backend assumes the host and provider adapter share this filesystem service.
|
||||
- Animated GIF metadata is validated from the logical screen; frame-level decoding policy is provider-owned.
|
||||
30
packages/attachment/attachment-local/package.json
Normal file
30
packages/attachment/attachment-local/package.json
Normal file
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-attachment-local",
|
||||
"description": "Private content-addressed DSH_HOME attachment storage",
|
||||
"version": "0.0.1",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "lib/index.js",
|
||||
"types": "lib/types/index.d.ts",
|
||||
"exports": {
|
||||
".": { "types": "./lib/types/index.d.ts", "default": "./lib/index.js" },
|
||||
"./invariant": { "types": "./lib/types/invariant.d.ts", "default": "./lib/invariant.js" },
|
||||
"./src/*": "./src/*",
|
||||
"./package.json": "./package.json"
|
||||
},
|
||||
"files": ["lib/index.js", "lib/invariant.js", "lib/types/**/*.d.ts", "lib/types/**/*.d.ts.map", "src"],
|
||||
"license": "BSD-3-Clause",
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/dsh-attachment": "^0.0.1",
|
||||
"@deepseek-ai/dsh-invariants": "^0.0.1",
|
||||
"@deepseek-ai/dsh-paths": "^0.0.1",
|
||||
"cordis": "^4.0.0-rc.6"
|
||||
},
|
||||
"dependencies": { "schemastery": "^3.18.0" },
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/dsh-attachment": "workspace:^",
|
||||
"@deepseek-ai/dsh-invariants": "workspace:^",
|
||||
"@deepseek-ai/dsh-paths": "workspace:^",
|
||||
"cordis": "^4.0.0-rc.6"
|
||||
}
|
||||
}
|
||||
101
packages/attachment/attachment-local/src/image.ts
Normal file
101
packages/attachment/attachment-local/src/image.ts
Normal file
@@ -0,0 +1,101 @@
|
||||
/** Minimal raster header validation used before bytes enter durable storage. */
|
||||
|
||||
import { AttachmentError } from '@deepseek-ai/dsh-attachment'
|
||||
import type { ImageMediaType } from '@deepseek-ai/dsh-attachment'
|
||||
|
||||
/** Decoded metadata from a supported image header. */
|
||||
export interface DetectedImage {
|
||||
mediaType: ImageMediaType
|
||||
width: number
|
||||
height: number
|
||||
}
|
||||
|
||||
function ascii(data: Uint8Array, start: number, value: string): boolean {
|
||||
if (data.length < start + value.length) return false
|
||||
for (let i = 0; i < value.length; i++) if (data[start + i] !== value.charCodeAt(i)) return false
|
||||
return true
|
||||
}
|
||||
|
||||
function u16be(data: Uint8Array, offset: number): number {
|
||||
return ((data[offset] ?? 0) << 8) | (data[offset + 1] ?? 0)
|
||||
}
|
||||
|
||||
function u16le(data: Uint8Array, offset: number): number {
|
||||
return (data[offset] ?? 0) | ((data[offset + 1] ?? 0) << 8)
|
||||
}
|
||||
|
||||
function u24le(data: Uint8Array, offset: number): number {
|
||||
return (data[offset] ?? 0) | ((data[offset + 1] ?? 0) << 8) | ((data[offset + 2] ?? 0) << 16)
|
||||
}
|
||||
|
||||
function u32be(data: Uint8Array, offset: number): number {
|
||||
return (((data[offset] ?? 0) * 0x1000000) + ((data[offset + 1] ?? 0) << 16)
|
||||
+ ((data[offset + 2] ?? 0) << 8) + (data[offset + 3] ?? 0)) >>> 0
|
||||
}
|
||||
|
||||
function u32le(data: Uint8Array, offset: number): number {
|
||||
return ((data[offset] ?? 0) + ((data[offset + 1] ?? 0) << 8)
|
||||
+ ((data[offset + 2] ?? 0) << 16) + ((data[offset + 3] ?? 0) * 0x1000000)) >>> 0
|
||||
}
|
||||
|
||||
function dimensions(width: number, height: number, mediaType: ImageMediaType): DetectedImage {
|
||||
if (width < 1 || height < 1) throw new AttachmentError('Image dimensions must be positive.', 'INVALID_IMAGE')
|
||||
return { mediaType, width, height }
|
||||
}
|
||||
|
||||
function jpeg(data: Uint8Array): DetectedImage | null {
|
||||
if (data[0] !== 0xff || data[1] !== 0xd8) return null
|
||||
const sof = new Set([0xc0, 0xc1, 0xc2, 0xc3, 0xc5, 0xc6, 0xc7, 0xc9, 0xca, 0xcb, 0xcd, 0xce, 0xcf])
|
||||
let offset = 2
|
||||
while (offset + 3 < data.length) {
|
||||
while (data[offset] === 0xff) offset++
|
||||
const marker = data[offset]
|
||||
if (marker === undefined || marker === 0xd9 || marker === 0xda) break
|
||||
if (marker === 0x01 || (marker >= 0xd0 && marker <= 0xd7)) {
|
||||
offset++
|
||||
continue
|
||||
}
|
||||
const length = u16be(data, offset + 1)
|
||||
if (length < 2 || offset + 1 + length > data.length) throw new AttachmentError('JPEG data is truncated.', 'INVALID_IMAGE')
|
||||
if (sof.has(marker)) {
|
||||
if (length < 7) throw new AttachmentError('JPEG dimensions are truncated.', 'INVALID_IMAGE')
|
||||
return dimensions(u16be(data, offset + 6), u16be(data, offset + 4), 'image/jpeg')
|
||||
}
|
||||
offset += length + 1
|
||||
}
|
||||
throw new AttachmentError('JPEG dimensions are missing.', 'INVALID_IMAGE')
|
||||
}
|
||||
|
||||
/**
|
||||
* Detect a supported raster type and intrinsic dimensions from encoded bytes.
|
||||
* @param data - complete encoded image bytes.
|
||||
* @returns verified format and dimensions.
|
||||
*/
|
||||
export function detectImage(data: Uint8Array): DetectedImage {
|
||||
if (data.length >= 24
|
||||
&& data[0] === 0x89 && ascii(data, 1, 'PNG\r\n\u001a\n') && ascii(data, 12, 'IHDR')) {
|
||||
return dimensions(u32be(data, 16), u32be(data, 20), 'image/png')
|
||||
}
|
||||
if (data.length >= 10 && (ascii(data, 0, 'GIF87a') || ascii(data, 0, 'GIF89a'))) {
|
||||
return dimensions(u16le(data, 6), u16le(data, 8), 'image/gif')
|
||||
}
|
||||
const detectedJpeg = jpeg(data)
|
||||
if (detectedJpeg !== null) return detectedJpeg
|
||||
if (data.length >= 30 && ascii(data, 0, 'RIFF') && ascii(data, 8, 'WEBP')) {
|
||||
const declaredLength = u32le(data, 4) + 8
|
||||
if (declaredLength > data.length) throw new AttachmentError('WebP data is truncated.', 'INVALID_IMAGE')
|
||||
if (ascii(data, 12, 'VP8X')) return dimensions(u24le(data, 24) + 1, u24le(data, 27) + 1, 'image/webp')
|
||||
if (ascii(data, 12, 'VP8L') && data[20] === 0x2f) {
|
||||
const b0 = data[21] ?? 0
|
||||
const b1 = data[22] ?? 0
|
||||
const b2 = data[23] ?? 0
|
||||
const b3 = data[24] ?? 0
|
||||
return dimensions(1 + b0 + ((b1 & 0x3f) << 8), 1 + (b1 >> 6) + (b2 << 2) + ((b3 & 0x0f) << 10), 'image/webp')
|
||||
}
|
||||
if (ascii(data, 12, 'VP8 ') && data[23] === 0x9d && data[24] === 0x01 && data[25] === 0x2a) {
|
||||
return dimensions(u16le(data, 26) & 0x3fff, u16le(data, 28) & 0x3fff, 'image/webp')
|
||||
}
|
||||
throw new AttachmentError('WebP dimensions are missing.', 'INVALID_IMAGE')
|
||||
}
|
||||
throw new AttachmentError('Unsupported or malformed image data.', 'INVALID_IMAGE')
|
||||
}
|
||||
74
packages/attachment/attachment-local/src/index.ts
Normal file
74
packages/attachment/attachment-local/src/index.ts
Normal file
@@ -0,0 +1,74 @@
|
||||
/** Local durable attachment backend rooted below `DSH_HOME`. @module @deepseek-ai/dsh-attachment-local */
|
||||
|
||||
import { join, resolve } from 'node:path'
|
||||
import { Context } from 'cordis'
|
||||
import z from 'schemastery'
|
||||
import { AttachmentStore } from '@deepseek-ai/dsh-attachment'
|
||||
import type { ImageAttachmentLimits, ImageAttachmentRef, SaveImageAttachment, StoredImageAttachment } from '@deepseek-ai/dsh-attachment'
|
||||
import { resolveDshHome } from '@deepseek-ai/dsh-paths'
|
||||
import { readImageFile, saveImageFile } from './store.ts'
|
||||
|
||||
export { detectImage } from './image.ts'
|
||||
export { readImageFile, saveImageFile } from './store.ts'
|
||||
export { AttachmentError } from '@deepseek-ai/dsh-attachment'
|
||||
export type { ImageAttachmentRef } from '@deepseek-ai/dsh-attachment'
|
||||
|
||||
/** Default maximum encoded bytes for one image. */
|
||||
export const DEFAULT_MAX_IMAGE_BYTES = 5 * 1024 * 1024
|
||||
/** Default maximum images in one prompt. */
|
||||
export const DEFAULT_MAX_IMAGES_PER_MESSAGE = 10
|
||||
/** Default maximum aggregate image bytes in one prompt. */
|
||||
export const DEFAULT_MAX_MESSAGE_IMAGE_BYTES = 20 * 1024 * 1024
|
||||
/** Default maximum intrinsic pixels for one image. */
|
||||
export const DEFAULT_MAX_IMAGE_PIXELS = 40_000_000
|
||||
|
||||
/** Local attachment backend configuration. */
|
||||
export interface Config {
|
||||
/** Explicit harness home; omitted follows `DSH_HOME`, then `~/.dsh`. */
|
||||
dshHome?: string
|
||||
/** Maximum encoded bytes accepted for one image. */
|
||||
maxImageBytes?: number
|
||||
/** Maximum image count accepted in one submitted message. */
|
||||
maxImagesPerMessage?: number
|
||||
/** Maximum aggregate encoded image bytes accepted in one submitted message. */
|
||||
maxMessageImageBytes?: number
|
||||
/** Maximum intrinsic width multiplied by height accepted for one image. */
|
||||
maxImagePixels?: number
|
||||
}
|
||||
|
||||
/** Persistent content-addressed local attachment store. */
|
||||
export class LocalAttachmentStore extends AttachmentStore {
|
||||
static Config: z<Config> = z.object({
|
||||
dshHome: z.string(),
|
||||
maxImageBytes: z.number().step(1).min(1).default(DEFAULT_MAX_IMAGE_BYTES),
|
||||
maxImagesPerMessage: z.number().step(1).min(1).default(DEFAULT_MAX_IMAGES_PER_MESSAGE),
|
||||
maxMessageImageBytes: z.number().step(1).min(1).default(DEFAULT_MAX_MESSAGE_IMAGE_BYTES),
|
||||
maxImagePixels: z.number().step(1).min(1).default(DEFAULT_MAX_IMAGE_PIXELS),
|
||||
})
|
||||
|
||||
/** Absolute versioned storage root. */
|
||||
readonly root: string
|
||||
readonly imageLimits: ImageAttachmentLimits
|
||||
|
||||
constructor(ctx: Context, config: Config) {
|
||||
super(ctx)
|
||||
this.root = resolve(join(resolveDshHome(config.dshHome), 'attachments', 'v1'))
|
||||
this.imageLimits = Object.freeze({
|
||||
maxImageBytes: config.maxImageBytes ?? DEFAULT_MAX_IMAGE_BYTES,
|
||||
maxImagesPerMessage: config.maxImagesPerMessage ?? DEFAULT_MAX_IMAGES_PER_MESSAGE,
|
||||
maxMessageImageBytes: config.maxMessageImageBytes ?? DEFAULT_MAX_MESSAGE_IMAGE_BYTES,
|
||||
maxImagePixels: config.maxImagePixels ?? DEFAULT_MAX_IMAGE_PIXELS,
|
||||
mediaTypes: Object.freeze(['image/png', 'image/jpeg', 'image/webp', 'image/gif'] as const),
|
||||
})
|
||||
}
|
||||
|
||||
async saveImage(input: SaveImageAttachment): Promise<ImageAttachmentRef> {
|
||||
return saveImageFile(this.root, input, this.imageLimits)
|
||||
}
|
||||
|
||||
async readImage(ref: ImageAttachmentRef): Promise<StoredImageAttachment> {
|
||||
return readImageFile(this.root, ref, this.imageLimits)
|
||||
}
|
||||
}
|
||||
|
||||
export default LocalAttachmentStore
|
||||
20
packages/attachment/attachment-local/src/invariant.ts
Normal file
20
packages/attachment/attachment-local/src/invariant.ts
Normal file
@@ -0,0 +1,20 @@
|
||||
/** Package-owned invariant companion for `@deepseek-ai/dsh-attachment-local`. @module @deepseek-ai/dsh-attachment-local/invariant */
|
||||
|
||||
/* jscpd:ignore-start */
|
||||
import type { Context } from 'cordis'
|
||||
import type { InvariantInstaller } from '@deepseek-ai/dsh-invariants'
|
||||
|
||||
const PACKAGE_NAME = '@deepseek-ai/dsh-attachment-local'
|
||||
/** Cordis companion plugin name. */
|
||||
export const name = 'attachment-local-invariant'
|
||||
/** Services required before package ownership can be reserved. */
|
||||
export const inject = ['invariants', 'attachments']
|
||||
/** No runtime invariant: immutable writes and verified reads are enforced directly at the backend boundary. */
|
||||
const install: InvariantInstaller = () => {}
|
||||
/**
|
||||
* Register the package invariant companion.
|
||||
* @param ctx - Cordis context carrying the invariant service.
|
||||
* @returns the registration disposer.
|
||||
*/
|
||||
export const apply = (ctx: Context): Promise<() => void> => Promise.resolve(ctx.invariants.register(PACKAGE_NAME, install))
|
||||
/* jscpd:ignore-end */
|
||||
121
packages/attachment/attachment-local/src/store.ts
Normal file
121
packages/attachment/attachment-local/src/store.ts
Normal file
@@ -0,0 +1,121 @@
|
||||
/** Content-addressed, owner-private local attachment storage. */
|
||||
|
||||
import { createHash, randomUUID } from 'node:crypto'
|
||||
import { constants } from 'node:fs'
|
||||
import { chmod, link, mkdir, open, readFile, unlink } from 'node:fs/promises'
|
||||
import { basename, join } from 'node:path'
|
||||
import {
|
||||
AttachmentError,
|
||||
AttachmentId,
|
||||
} from '@deepseek-ai/dsh-attachment'
|
||||
import type {
|
||||
ImageAttachmentLimits,
|
||||
ImageAttachmentRef,
|
||||
SaveImageAttachment,
|
||||
StoredImageAttachment,
|
||||
} from '@deepseek-ai/dsh-attachment'
|
||||
import { detectImage } from './image.ts'
|
||||
|
||||
const ID_PATTERN = /^sha256:([a-f0-9]{64})$/
|
||||
|
||||
function digest(data: Uint8Array): string {
|
||||
return createHash('sha256').update(data).digest('hex')
|
||||
}
|
||||
|
||||
function displayName(value: string | undefined): string | undefined {
|
||||
if (value === undefined) return undefined
|
||||
const clean = basename(value).replace(/[\u0000-\u001f\u007f]/g, '').trim().slice(0, 255)
|
||||
return clean === '' ? undefined : clean
|
||||
}
|
||||
|
||||
function objectPath(root: string, sha256: string): string {
|
||||
return join(root, 'objects', sha256.slice(0, 2), sha256)
|
||||
}
|
||||
|
||||
function ensureReference(ref: ImageAttachmentRef): string {
|
||||
const match = ID_PATTERN.exec(String(ref.attachmentId))
|
||||
if (match?.[1] === undefined) throw new AttachmentError('Attachment reference is invalid.', 'INVALID_ATTACHMENT_REF')
|
||||
return match[1]
|
||||
}
|
||||
|
||||
function validateMetadata(data: Uint8Array, declaredMediaType: ImageAttachmentRef['mediaType'], limits: ImageAttachmentLimits): Omit<ImageAttachmentRef, 'attachmentId' | 'name'> {
|
||||
if (data.byteLength === 0) throw new AttachmentError('Image is empty.', 'INVALID_IMAGE')
|
||||
if (data.byteLength > limits.maxImageBytes) throw new AttachmentError('Image exceeds the configured byte limit.', 'IMAGE_TOO_LARGE')
|
||||
const detected = detectImage(data)
|
||||
if (detected.mediaType !== declaredMediaType) throw new AttachmentError('Declared image type does not match its bytes.', 'IMAGE_TYPE_MISMATCH')
|
||||
if (detected.width * detected.height > limits.maxImagePixels) throw new AttachmentError('Image exceeds the configured decoded-pixel limit.', 'IMAGE_TOO_MANY_PIXELS')
|
||||
return { ...detected, bytes: data.byteLength }
|
||||
}
|
||||
|
||||
/**
|
||||
* Save and verify immutable image bytes below a versioned attachment root.
|
||||
* @param root - absolute `DSH_HOME/attachments/v1` root.
|
||||
* @param input - encoded bytes and declared metadata.
|
||||
* @param limits - resolved storage policy.
|
||||
* @returns durable content-addressed reference.
|
||||
*/
|
||||
export async function saveImageFile(root: string, input: SaveImageAttachment, limits: ImageAttachmentLimits): Promise<ImageAttachmentRef> {
|
||||
const metadata = validateMetadata(input.data, input.mediaType, limits)
|
||||
const sha256 = digest(input.data)
|
||||
const bucket = join(root, 'objects', sha256.slice(0, 2))
|
||||
const staging = join(root, 'tmp')
|
||||
await mkdir(bucket, { recursive: true, mode: 0o700 })
|
||||
await mkdir(staging, { recursive: true, mode: 0o700 })
|
||||
await chmod(bucket, 0o700)
|
||||
await chmod(staging, 0o700)
|
||||
const temporary = join(staging, randomUUID())
|
||||
const target = objectPath(root, sha256)
|
||||
let handle
|
||||
try {
|
||||
handle = await open(temporary, constants.O_CREAT | constants.O_EXCL | constants.O_WRONLY, 0o600)
|
||||
await handle.writeFile(input.data)
|
||||
await handle.sync()
|
||||
await handle.close()
|
||||
handle = undefined
|
||||
try {
|
||||
await link(temporary, target)
|
||||
} catch (error) {
|
||||
if (!(error instanceof Error && 'code' in error && error.code === 'EEXIST')) throw error
|
||||
const existing = new Uint8Array(await readFile(target))
|
||||
if (digest(existing) !== sha256) throw new AttachmentError('Stored attachment failed integrity verification.', 'ATTACHMENT_CORRUPT')
|
||||
}
|
||||
await unlink(temporary)
|
||||
} catch (error) {
|
||||
if (handle !== undefined) await handle.close().catch(() => { /* close failure is superseded by the storage failure */ })
|
||||
await unlink(temporary).catch((cleanupError: unknown) => {
|
||||
if (!(cleanupError instanceof Error && 'code' in cleanupError && cleanupError.code === 'ENOENT')) throw cleanupError
|
||||
})
|
||||
if (error instanceof AttachmentError) throw error
|
||||
throw new AttachmentError('Unable to persist image attachment.', 'ATTACHMENT_WRITE_FAILED', { cause: error })
|
||||
}
|
||||
const name = displayName(input.name)
|
||||
return {
|
||||
attachmentId: AttachmentId(`sha256:${sha256}`),
|
||||
...metadata,
|
||||
...(name !== undefined ? { name } : {}),
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Read and verify one content-addressed image.
|
||||
* @param root - absolute `DSH_HOME/attachments/v1` root.
|
||||
* @param ref - reference recorded in the session log.
|
||||
* @param limits - resolved storage policy.
|
||||
* @returns verified bytes and reference.
|
||||
*/
|
||||
export async function readImageFile(root: string, ref: ImageAttachmentRef, limits: ImageAttachmentLimits): Promise<StoredImageAttachment> {
|
||||
const sha256 = ensureReference(ref)
|
||||
let data: Uint8Array
|
||||
try {
|
||||
data = new Uint8Array(await readFile(objectPath(root, sha256)))
|
||||
} catch (error) {
|
||||
if (error instanceof Error && 'code' in error && error.code === 'ENOENT') throw new AttachmentError('Attachment object is missing.', 'ATTACHMENT_NOT_FOUND')
|
||||
throw new AttachmentError('Unable to read image attachment.', 'ATTACHMENT_READ_FAILED', { cause: error })
|
||||
}
|
||||
if (digest(data) !== sha256) throw new AttachmentError('Stored attachment failed integrity verification.', 'ATTACHMENT_CORRUPT')
|
||||
const metadata = validateMetadata(data, ref.mediaType, limits)
|
||||
if (metadata.bytes !== ref.bytes || metadata.width !== ref.width || metadata.height !== ref.height) {
|
||||
throw new AttachmentError('Stored attachment metadata does not match its reference.', 'ATTACHMENT_CORRUPT')
|
||||
}
|
||||
return { ref, data }
|
||||
}
|
||||
96
packages/attachment/attachment-local/tests/store.spec.ts
Normal file
96
packages/attachment/attachment-local/tests/store.spec.ts
Normal file
@@ -0,0 +1,96 @@
|
||||
import { createHash } from 'node:crypto'
|
||||
import { chmod, mkdir, readFile, stat, writeFile } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { mkdtemp, rm } from 'node:fs/promises'
|
||||
import { afterEach, describe, expect, it } from 'vitest'
|
||||
import type { ImageAttachmentLimits } from '@deepseek-ai/dsh-attachment'
|
||||
import { readImageFile, saveImageFile } from '../src/store.ts'
|
||||
|
||||
const PNG = Uint8Array.from(Buffer.from(
|
||||
'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII=',
|
||||
'base64',
|
||||
))
|
||||
|
||||
const LIMITS: ImageAttachmentLimits = {
|
||||
maxImageBytes: 1024,
|
||||
maxImagesPerMessage: 2,
|
||||
maxMessageImageBytes: 2048,
|
||||
maxImagePixels: 16,
|
||||
mediaTypes: ['image/png', 'image/jpeg', 'image/webp', 'image/gif'],
|
||||
}
|
||||
|
||||
const roots: string[] = []
|
||||
|
||||
async function root(): Promise<string> {
|
||||
const value = await mkdtemp(join(tmpdir(), 'dsh-attachment-'))
|
||||
roots.push(value)
|
||||
return join(value, 'attachments', 'v1')
|
||||
}
|
||||
|
||||
afterEach(async () => {
|
||||
await Promise.all(roots.splice(0).map(path => rm(path, { recursive: true, force: true })))
|
||||
})
|
||||
|
||||
describe('local attachment store', () => {
|
||||
it('publishes one private content-addressed object and deduplicates equal bytes', async () => {
|
||||
const storageRoot = await root()
|
||||
const first = await saveImageFile(storageRoot, {
|
||||
data: PNG, mediaType: 'image/png', name: '/private/tmp/pixel.png',
|
||||
}, LIMITS)
|
||||
const second = await saveImageFile(storageRoot, { data: PNG, mediaType: 'image/png' }, LIMITS)
|
||||
const sha256 = createHash('sha256').update(PNG).digest('hex')
|
||||
const object = join(storageRoot, 'objects', sha256.slice(0, 2), sha256)
|
||||
|
||||
expect(first).toEqual({
|
||||
attachmentId: `sha256:${sha256}`,
|
||||
mediaType: 'image/png',
|
||||
bytes: PNG.byteLength,
|
||||
width: 1,
|
||||
height: 1,
|
||||
name: 'pixel.png',
|
||||
})
|
||||
expect(second.attachmentId).toBe(first.attachmentId)
|
||||
expect(new Uint8Array(await readFile(object))).toEqual(PNG)
|
||||
expect((await stat(object)).mode & 0o777).toBe(0o600)
|
||||
expect((await stat(join(storageRoot, 'objects', sha256.slice(0, 2)))).mode & 0o777).toBe(0o700)
|
||||
await expect(readImageFile(storageRoot, first, LIMITS)).resolves.toEqual({ ref: first, data: PNG })
|
||||
})
|
||||
|
||||
it('rejects malformed bytes, mismatched declarations, byte limits, and decoded-pixel limits', async () => {
|
||||
const storageRoot = await root()
|
||||
await expect(saveImageFile(storageRoot, {
|
||||
data: Uint8Array.of(1, 2, 3), mediaType: 'image/png',
|
||||
}, LIMITS)).rejects.toMatchObject({ code: 'INVALID_IMAGE' })
|
||||
await expect(saveImageFile(storageRoot, {
|
||||
data: PNG, mediaType: 'image/jpeg',
|
||||
}, LIMITS)).rejects.toMatchObject({ code: 'IMAGE_TYPE_MISMATCH' })
|
||||
await expect(saveImageFile(storageRoot, {
|
||||
data: PNG, mediaType: 'image/png',
|
||||
}, { ...LIMITS, maxImageBytes: 1 })).rejects.toMatchObject({ code: 'IMAGE_TOO_LARGE' })
|
||||
|
||||
const wide = PNG.slice()
|
||||
wide.set([0, 0, 0, 5, 0, 0, 0, 5], 16)
|
||||
await expect(saveImageFile(storageRoot, {
|
||||
data: wide, mediaType: 'image/png',
|
||||
}, LIMITS)).rejects.toMatchObject({ code: 'IMAGE_TOO_MANY_PIXELS' })
|
||||
})
|
||||
|
||||
it('fails closed when an object is missing, corrupted, or addressed by an invalid reference', async () => {
|
||||
const storageRoot = await root()
|
||||
const ref = await saveImageFile(storageRoot, { data: PNG, mediaType: 'image/png' }, LIMITS)
|
||||
const sha256 = String(ref.attachmentId).slice('sha256:'.length)
|
||||
const object = join(storageRoot, 'objects', sha256.slice(0, 2), sha256)
|
||||
await chmod(object, 0o600)
|
||||
await writeFile(object, Uint8Array.of(1, 2, 3))
|
||||
await expect(readImageFile(storageRoot, ref, LIMITS))
|
||||
.rejects.toMatchObject({ code: 'ATTACHMENT_CORRUPT' })
|
||||
await expect(readImageFile(storageRoot, { ...ref, attachmentId: 'bad' as never }, LIMITS))
|
||||
.rejects.toMatchObject({ code: 'INVALID_ATTACHMENT_REF' })
|
||||
|
||||
const missingRoot = await root()
|
||||
await mkdir(missingRoot, { recursive: true })
|
||||
await expect(readImageFile(missingRoot, ref, LIMITS))
|
||||
.rejects.toMatchObject({ code: 'ATTACHMENT_NOT_FOUND' })
|
||||
})
|
||||
})
|
||||
12
packages/attachment/attachment-local/tsconfig.json
Normal file
12
packages/attachment/attachment-local/tsconfig.json
Normal file
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"extends": "../../../tsconfig.base.json",
|
||||
"compilerOptions": { "rootDir": "src", "outDir": "lib/types" },
|
||||
"include": ["src"],
|
||||
"references": [
|
||||
{ "path": "../../../vendor/cosmokit" },
|
||||
{ "path": "../../../vendor/cordis" },
|
||||
{ "path": "../attachment" },
|
||||
{ "path": "../../util/paths" },
|
||||
{ "path": "../../support/invariants" }
|
||||
]
|
||||
}
|
||||
19
packages/attachment/attachment/README.md
Normal file
19
packages/attachment/attachment/README.md
Normal file
@@ -0,0 +1,19 @@
|
||||
# @deepseek-ai/dsh-attachment
|
||||
|
||||
The durable attachment seam. `ctx.attachments` validates and atomically commits immutable image bytes, then returns a serializable `ImageAttachmentRef`; consumers never persist browser paths, object URLs, provider URLs, or base64 in session events.
|
||||
|
||||
Unsent composer images remain browser-owned temporary drafts. `saveImage` is called only at message submission or while committing structured provider output, before any model-visible session event is published. `readImage` verifies the content-addressed object against its logged metadata.
|
||||
|
||||
## Model Experience
|
||||
|
||||
Indirectly, through the role-neutral core `ImageBlock` and provider adapters that resolve its durable reference.
|
||||
|
||||
#### KV Cache effect
|
||||
|
||||
Adding an image changes the provider request and therefore invalidates the affected request suffix.
|
||||
|
||||
## Known Limitations and Deferred Work
|
||||
|
||||
- Version one accepts PNG, JPEG, WebP, and GIF only.
|
||||
- Retention and garbage collection are deferred because resumed and forked sessions may share immutable objects.
|
||||
- Generic files, audio, video, and persistent unsent drafts require separate lifecycle and provider contracts.
|
||||
27
packages/attachment/attachment/package.json
Normal file
27
packages/attachment/attachment/package.json
Normal file
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"name": "@deepseek-ai/dsh-attachment",
|
||||
"description": "Durable immutable attachment storage seam for the DeepSeek Harness",
|
||||
"version": "0.0.1",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"main": "lib/index.js",
|
||||
"types": "lib/types/index.d.ts",
|
||||
"exports": {
|
||||
".": { "types": "./lib/types/index.d.ts", "default": "./lib/index.js" },
|
||||
"./invariant": { "types": "./lib/types/invariant.d.ts", "default": "./lib/invariant.js" },
|
||||
"./src/*": "./src/*",
|
||||
"./package.json": "./package.json"
|
||||
},
|
||||
"files": ["lib/index.js", "lib/invariant.js", "lib/types/**/*.d.ts", "lib/types/**/*.d.ts.map", "src"],
|
||||
"license": "BSD-3-Clause",
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/dsh-brand": "^0.0.1",
|
||||
"@deepseek-ai/dsh-invariants": "^0.0.1",
|
||||
"cordis": "^4.0.0-rc.6"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/dsh-brand": "workspace:^",
|
||||
"@deepseek-ai/dsh-invariants": "workspace:^",
|
||||
"cordis": "^4.0.0-rc.6"
|
||||
}
|
||||
}
|
||||
51
packages/attachment/attachment/src/index.ts
Normal file
51
packages/attachment/attachment/src/index.ts
Normal file
@@ -0,0 +1,51 @@
|
||||
/** Durable attachment storage seam (`ctx.attachments`). @module @deepseek-ai/dsh-attachment */
|
||||
|
||||
import { Context, Service } from 'cordis'
|
||||
import type {
|
||||
ImageAttachmentLimits,
|
||||
ImageAttachmentRef,
|
||||
SaveImageAttachment,
|
||||
StoredImageAttachment,
|
||||
} from './types.ts'
|
||||
|
||||
export { AttachmentError, AttachmentId } from './types.ts'
|
||||
export type {
|
||||
AttachmentId as AttachmentIdType,
|
||||
ImageAttachmentLimits,
|
||||
ImageAttachmentRef,
|
||||
ImageMediaType,
|
||||
SaveImageAttachment,
|
||||
StoredImageAttachment,
|
||||
} from './types.ts'
|
||||
|
||||
declare module 'cordis' {
|
||||
interface Context {
|
||||
attachments: AttachmentStore
|
||||
}
|
||||
}
|
||||
|
||||
/** Immutable binary attachment service. Implementations validate bytes before publishing a reference. */
|
||||
export abstract class AttachmentStore extends Service {
|
||||
constructor(ctx: Context) {
|
||||
super(ctx, 'attachments')
|
||||
}
|
||||
|
||||
/** Deployment-resolved image policy used by authoritative and fast-path validation. */
|
||||
abstract readonly imageLimits: ImageAttachmentLimits
|
||||
|
||||
/**
|
||||
* Validate and durably commit one image before its owning session event is appended.
|
||||
* @param input - encoded bytes, declared media type, and optional display name.
|
||||
* @returns a durable content-addressed reference.
|
||||
*/
|
||||
abstract saveImage(input: SaveImageAttachment): Promise<ImageAttachmentRef>
|
||||
|
||||
/**
|
||||
* Read one image and verify that bytes still match the recorded reference.
|
||||
* @param ref - durable reference from the session log.
|
||||
* @returns the verified bytes and canonical reference.
|
||||
*/
|
||||
abstract readImage(ref: ImageAttachmentRef): Promise<StoredImageAttachment>
|
||||
}
|
||||
|
||||
export default AttachmentStore
|
||||
20
packages/attachment/attachment/src/invariant.ts
Normal file
20
packages/attachment/attachment/src/invariant.ts
Normal file
@@ -0,0 +1,20 @@
|
||||
/** Package-owned invariant companion for `@deepseek-ai/dsh-attachment`. @module @deepseek-ai/dsh-attachment/invariant */
|
||||
|
||||
/* jscpd:ignore-start */
|
||||
import type { Context } from 'cordis'
|
||||
import type { InvariantInstaller } from '@deepseek-ai/dsh-invariants'
|
||||
|
||||
const PACKAGE_NAME = '@deepseek-ai/dsh-attachment'
|
||||
/** Cordis companion plugin name. */
|
||||
export const name = 'attachment-invariant'
|
||||
/** Service required before package ownership can be reserved. */
|
||||
export const inject = ['invariants']
|
||||
/** No runtime invariant: this stateless seam owns types while implementations enforce immutable-store checks. */
|
||||
const install: InvariantInstaller = () => {}
|
||||
/**
|
||||
* Register the package invariant companion.
|
||||
* @param ctx - Cordis context carrying the invariant service.
|
||||
* @returns the registration disposer.
|
||||
*/
|
||||
export const apply = (ctx: Context): Promise<() => void> => Promise.resolve(ctx.invariants.register(PACKAGE_NAME, install))
|
||||
/* jscpd:ignore-end */
|
||||
75
packages/attachment/attachment/src/types.ts
Normal file
75
packages/attachment/attachment/src/types.ts
Normal file
@@ -0,0 +1,75 @@
|
||||
/** Durable attachment vocabulary. @module @deepseek-ai/dsh-attachment/types */
|
||||
|
||||
import type { Branded } from '@deepseek-ai/dsh-brand'
|
||||
|
||||
/** Opaque content-addressed identifier for one immutable attachment object. */
|
||||
export type AttachmentId = Branded<'AttachmentId'>
|
||||
|
||||
/**
|
||||
* Brand a validated storage identifier.
|
||||
* @param value - backend-produced opaque identifier.
|
||||
* @returns the branded identifier.
|
||||
*/
|
||||
export function AttachmentId(value: string): AttachmentId {
|
||||
return value as AttachmentId
|
||||
}
|
||||
|
||||
/** Raster image formats accepted by the version-one attachment path. */
|
||||
export type ImageMediaType = 'image/png' | 'image/jpeg' | 'image/webp' | 'image/gif'
|
||||
|
||||
/** Durable, serializable metadata for one immutable image object. */
|
||||
export interface ImageAttachmentRef {
|
||||
/** Opaque storage identifier; never a filesystem path or bearer URL. */
|
||||
attachmentId: AttachmentId
|
||||
/** Media type verified from the stored bytes. */
|
||||
mediaType: ImageMediaType
|
||||
/** Exact encoded byte length. */
|
||||
bytes: number
|
||||
/** Intrinsic encoded width in pixels. */
|
||||
width: number
|
||||
/** Intrinsic encoded height in pixels. */
|
||||
height: number
|
||||
/** Optional display name stripped of local path information. */
|
||||
name?: string
|
||||
}
|
||||
|
||||
/** Deployment-resolved limits shared by upload consumers and UI preflight. */
|
||||
export interface ImageAttachmentLimits {
|
||||
maxImageBytes: number
|
||||
maxImagesPerMessage: number
|
||||
maxMessageImageBytes: number
|
||||
maxImagePixels: number
|
||||
mediaTypes: readonly ImageMediaType[]
|
||||
}
|
||||
|
||||
/** Request to validate and durably commit one image. */
|
||||
export interface SaveImageAttachment {
|
||||
data: Uint8Array
|
||||
/** Caller-declared media type, checked against magic bytes. */
|
||||
mediaType: ImageMediaType
|
||||
/** Optional browser/provider display name; it is never interpreted as a path. */
|
||||
name?: string
|
||||
}
|
||||
|
||||
/** Stored image bytes returned after reference and digest verification. */
|
||||
export interface StoredImageAttachment {
|
||||
ref: ImageAttachmentRef
|
||||
data: Uint8Array
|
||||
}
|
||||
|
||||
/** Stable failures suitable for host RPC error mapping. */
|
||||
export class AttachmentError extends Error {
|
||||
/** Stable machine-routing failure code. */
|
||||
readonly code: string
|
||||
|
||||
/**
|
||||
* @param message - human-readable failure description without raw bytes or host paths.
|
||||
* @param code - stable machine-routing code.
|
||||
* @param options - optional chained cause.
|
||||
*/
|
||||
constructor(message: string, code: string, options?: ErrorOptions) {
|
||||
super(message, options)
|
||||
this.name = 'AttachmentError'
|
||||
this.code = code
|
||||
}
|
||||
}
|
||||
11
packages/attachment/attachment/tsconfig.json
Normal file
11
packages/attachment/attachment/tsconfig.json
Normal file
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"extends": "../../../tsconfig.base.json",
|
||||
"compilerOptions": { "rootDir": "src", "outDir": "lib/types" },
|
||||
"include": ["src"],
|
||||
"references": [
|
||||
{ "path": "../../../vendor/cosmokit" },
|
||||
{ "path": "../../../vendor/cordis" },
|
||||
{ "path": "../../util/brand" },
|
||||
{ "path": "../../support/invariants" }
|
||||
]
|
||||
}
|
||||
@@ -29,6 +29,7 @@
|
||||
},
|
||||
"license": "BSD-3-Clause",
|
||||
"dependencies": {
|
||||
"@deepseek-ai/dsh-attachment": "workspace:^",
|
||||
"@deepseek-ai/dsh-host-apiproxy": "workspace:^",
|
||||
"@deepseek-ai/dsh-llm": "workspace:^",
|
||||
"@deepseek-ai/dsh-session": "workspace:^",
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
// The ./api and ./client subpath exports are the browser-safe channels added for this.
|
||||
|
||||
export type {
|
||||
ApiProxy, SessionsApi, SessionSummary, HostApi, EventsApi, MuxFrame, HostFrame,
|
||||
ApiProxy, SessionsApi, SessionSummary, PromptContentPart, HostApi, EventsApi, MuxFrame, HostFrame,
|
||||
ApprovalResponsePayload, QuestionResponsePayload, HistoryEntry, ToolEventView,
|
||||
} from '@deepseek-ai/dsh-host-apiproxy/api'
|
||||
export type { ToolCallView, ToolResultView } from '@deepseek-ai/dsh-tools/presentation'
|
||||
|
||||
@@ -6,6 +6,7 @@
|
||||
// approval (placeholder-card material, subscribed-baseline-replay semantics: stable rpcId reuse).
|
||||
|
||||
import type { ContentBlock } from '@deepseek-ai/dsh-llm/types'
|
||||
import type { AttachmentIdType, ImageAttachmentRef } from '@deepseek-ai/dsh-attachment'
|
||||
import type { SessionEvent, SessionId } from '@deepseek-ai/dsh-session/types'
|
||||
import type {
|
||||
ApiProxy, ClientRequest, ClientResponse, HistoryEntry, HostFrame, MuxFrame, RpcReceipt,
|
||||
@@ -28,6 +29,16 @@ function sid(id: string): SessionId {
|
||||
return id as SessionId
|
||||
}
|
||||
|
||||
const FIXTURE_IMAGE_DATA = 'iVBORw0KGgoAAAANSUhEUgAAAKAAAABaCAYAAAA/xl1SAAAAvklEQVR42u3SMQ0AAAjAMIyhELM4AAe8PD1qYFlk9cCXEAEDYkAwIAYEA2JAMCAGBANiQDAgBgQDYkAwIAYEA2JAMCAGBANiQDAgBgQDYkAwIAYEA2JAMCAGxIBCYEAMCAbEgGBADAgGxIBgQAwIBsSAYEAMCAbEgGBADAgGxIBgQAwIBsSAYEAMCAbEgGBADAgGxIAYEAyIAcGAGBAMiAHBgBgQDIgBwYAYEAyIAcGAGBAMiAHBgBgQDIgB4bYWLb6pnOb1xAAAAABJRU5ErkJggg=='
|
||||
const FIXTURE_IMAGE_REF: ImageAttachmentRef = {
|
||||
attachmentId: 'fixture:image' as AttachmentIdType,
|
||||
mediaType: 'image/png',
|
||||
bytes: 68,
|
||||
width: 160,
|
||||
height: 90,
|
||||
name: 'fixture-image.png',
|
||||
}
|
||||
|
||||
/** fx-alpha history script: 60 turns (~130+ messages -> 3 pages at PAGE_MESSAGES=50),
|
||||
* mixing reasoning blocks / tool call+result / steering / context. */
|
||||
function buildAlphaLog(): SessionEvent[] {
|
||||
@@ -88,6 +99,12 @@ function buildAlphaLog(): SessionEvent[] {
|
||||
toolTurn(60, 'fx-bash', '{"command":"ls -la","cwd":"/tmp/fixture"}', 'total 2\ndrwxr-xr-x fixture\n-rw-r--r-- demo.txt')
|
||||
toolTurn(61, 'fx-write', '{"path":"notes/demo.txt","content":"hello fixture\\n"}', 'wrote notes/demo.txt')
|
||||
toolTurn(62, 'fx-note', '{"note":"三型卡验收样本"}', '已记录')
|
||||
push({ type: 'turn/start', data: { turn: 63, trigger: { kind: 'message', source: { kind: 'user' } } } })
|
||||
push({ type: 'user/message', surfaceOp: 'append', data: { content: [{ type: 'image', attachment: FIXTURE_IMAGE_REF }, ...text('历史用户图片')], source: { kind: 'user' } } })
|
||||
push({ type: 'step/start', data: { turn: 63, step: 0 } })
|
||||
push({ type: 'assistant/message', surfaceOp: 'append', data: { turn: 63, step: 0, content: [...text('结构化模型图片：'), { type: 'image', attachment: FIXTURE_IMAGE_REF }], provenance: { provider: 'fixture', model: 'fx-vision' } } })
|
||||
push({ type: 'step/end', data: { turn: 63, step: 0 } })
|
||||
push({ type: 'turn/end', data: { turn: 63, reason: { kind: 'completed' } } })
|
||||
return events as unknown as SessionEvent[]
|
||||
}
|
||||
|
||||
@@ -186,6 +203,18 @@ function pageOf(
|
||||
return { events, hasMore: start > 0 }
|
||||
}
|
||||
|
||||
/** Fixture mirror of host session-scoped attachment authorization. */
|
||||
function logReferencesAttachment(log: readonly SessionEvent[], attachmentId: string): boolean {
|
||||
const visit = (value: unknown): boolean => {
|
||||
if (Array.isArray(value)) return value.some(visit)
|
||||
if (typeof value !== 'object' || value === null) return false
|
||||
const record = value as Record<string, unknown>
|
||||
if (record.attachmentId === attachmentId) return true
|
||||
return Object.values(record).some(visit)
|
||||
}
|
||||
return log.some(event => visit(event.data))
|
||||
}
|
||||
|
||||
interface StreamConn<F> {
|
||||
push(envelope: RpcRequest<F>): void
|
||||
}
|
||||
@@ -243,7 +272,11 @@ export function createFixtureApi(): ApiProxy {
|
||||
{ sessionId: sid('fx-gamma'), updatedAt: Date.now() - 120_000, running: false, cwd: '/tmp/fixture' },
|
||||
]
|
||||
const logs = new Map<SessionId, SessionEvent[]>([[sid('fx-alpha'), buildAlphaLog()]])
|
||||
const nextTurn = new Map<SessionId, number>([[sid('fx-alpha'), 60]])
|
||||
const nextTurn = new Map<SessionId, number>([[sid('fx-alpha'), 64]])
|
||||
const attachments = new Map<string, { attachment: ImageAttachmentRef; data: string }>([[
|
||||
String(FIXTURE_IMAGE_REF.attachmentId),
|
||||
{ attachment: FIXTURE_IMAGE_REF, data: FIXTURE_IMAGE_DATA },
|
||||
]])
|
||||
let nextSession = 1
|
||||
let nextRpc = 1
|
||||
const mint = (): ReturnType<typeof RpcId> => RpcId(`fx-rpc-${nextRpc++}`)
|
||||
@@ -394,21 +427,44 @@ export function createFixtureApi(): ApiProxy {
|
||||
}
|
||||
summary.updatedAt = Date.now()
|
||||
const userText = content.map(b => (b.type === 'text' ? b.text : '')).join('')
|
||||
const durable: ContentBlock[] = content.map((block) => {
|
||||
if (block.type === 'text') return block
|
||||
const attachment: ImageAttachmentRef = {
|
||||
attachmentId: `fixture:${crypto.randomUUID()}` as AttachmentIdType,
|
||||
mediaType: block.mediaType,
|
||||
bytes: Math.max(1, Math.floor(block.data.length * 3 / 4) - (block.data.endsWith('==') ? 2 : block.data.endsWith('=') ? 1 : 0)),
|
||||
width: 160,
|
||||
height: 90,
|
||||
...block.name === undefined ? {} : { name: block.name },
|
||||
}
|
||||
attachments.set(String(attachment.attachmentId), { attachment, data: block.data })
|
||||
return { type: 'image', attachment }
|
||||
})
|
||||
if (mode === 'steer' && replays.has(id)) {
|
||||
// Steering: insert a steering message into the current turn; the replay continues.
|
||||
/* v8 ignore next -- the ?? arm needs a missing counter, but a live replay implies a prior prompt already set it. */
|
||||
const turn = (nextTurn.get(id) ?? 1) - 1
|
||||
append(id, { type: 'steering/message', surfaceOp: 'append', data: { turn, content, source: { kind: 'user' } } })
|
||||
append(id, { type: 'steering/message', surfaceOp: 'append', data: { turn, content: durable, source: { kind: 'user' } } })
|
||||
return ok(request, { accepted: true as const })
|
||||
}
|
||||
const turn = nextTurn.get(id) ?? 0
|
||||
nextTurn.set(id, turn + 1)
|
||||
setRunning(id, true)
|
||||
append(id, { type: 'turn/start', data: { turn, trigger: { kind: 'message', source: { kind: 'user' } } } })
|
||||
append(id, { type: 'user/message', surfaceOp: 'append', data: { content, source: { kind: 'user' } } })
|
||||
append(id, { type: 'user/message', surfaceOp: 'append', data: { content: durable, source: { kind: 'user' } } })
|
||||
startReply(id, turn, `回声：${userText}。这是 fixture 的流式回复，用于验证打字机增长与定稿切换。`)
|
||||
return ok(request, { accepted: true as const })
|
||||
},
|
||||
attachment: (request) => {
|
||||
const stored = attachments.get(String(request.payload.attachmentId))
|
||||
if (stored === undefined) {
|
||||
return err(request, { code: 'attachment-error', message: 'fixture attachment missing', details: { reason: 'ATTACHMENT_NOT_FOUND' } })
|
||||
}
|
||||
if (!logReferencesAttachment(logs.get(request.payload.sessionId) ?? [], String(request.payload.attachmentId))) {
|
||||
return err(request, { code: 'attachment-error', message: 'fixture attachment is not referenced by this session', details: { reason: 'ATTACHMENT_NOT_REFERENCED' } })
|
||||
}
|
||||
return ok(request, stored)
|
||||
},
|
||||
cancel: (request) => {
|
||||
const replay = replays.get(request.payload.sessionId)
|
||||
if (replay !== undefined) {
|
||||
@@ -421,7 +477,24 @@ export function createFixtureApi(): ApiProxy {
|
||||
},
|
||||
},
|
||||
host: {
|
||||
describe: request => ok(request, { version: '0.0.0-fixture', cwd: '/tmp/fixture', attachedSessions: 1 }),
|
||||
describe: request => ok(request, {
|
||||
version: '0.0.0-fixture',
|
||||
cwd: '/tmp/fixture',
|
||||
provider: 'fixture',
|
||||
model: 'fx-vision',
|
||||
activeModel: {
|
||||
provider: 'fixture', id: 'fx-vision', name: 'Fixture Vision',
|
||||
inputModalities: ['text', 'image'], outputModalities: ['text', 'image'],
|
||||
},
|
||||
imageLimits: {
|
||||
maxImageBytes: 5 * 1024 * 1024,
|
||||
maxImagesPerMessage: 10,
|
||||
maxMessageImageBytes: 20 * 1024 * 1024,
|
||||
maxImagePixels: 40_000_000,
|
||||
mediaTypes: ['image/png', 'image/jpeg', 'image/webp', 'image/gif'],
|
||||
},
|
||||
attachedSessions: 1,
|
||||
}),
|
||||
},
|
||||
events: {
|
||||
async *mux(_request, signal) {
|
||||
@@ -512,6 +585,7 @@ export class FixtureApiClient extends AbstractApiClient {
|
||||
case 'session.create': return this.api.sessions.create(request)
|
||||
case 'session.history': return this.api.sessions.history(request)
|
||||
case 'session.prompt': return this.api.sessions.prompt(request)
|
||||
case 'session.attachment': return this.api.sessions.attachment(request)
|
||||
case 'session.cancel': return this.api.sessions.cancel(request)
|
||||
case 'host.describe': return this.api.host.describe(request)
|
||||
}
|
||||
|
||||
@@ -14,7 +14,7 @@ import { WebApiClient } from './web-api-client.ts'
|
||||
|
||||
// ---- Contract re-exports (browser-safe apiproxy channels + core types) ----
|
||||
export type {
|
||||
ApiProxy, SessionsApi, SessionSummary, HostApi, EventsApi, MuxFrame, HostFrame,
|
||||
ApiProxy, SessionsApi, SessionSummary, PromptContentPart, HostApi, EventsApi, MuxFrame, HostFrame,
|
||||
ApprovalResponsePayload, QuestionResponsePayload, HistoryEntry, ToolEventView,
|
||||
ToolCallView, ToolResultView,
|
||||
RpcRequest, RpcResponse, RpcResult, RpcError, RpcErrorCode,
|
||||
|
||||
@@ -48,6 +48,8 @@ export class FakeApiClient implements IApiClient {
|
||||
() => Promise.resolve(ok({ events: [], hasMore: false }))
|
||||
|
||||
onPrompt: (payload: unknown) => Promise<RpcResponse<{ accepted: true }>> = () => Promise.resolve(ok({ accepted: true as const }))
|
||||
onAttachment: (payload: unknown) => Promise<RpcResponse<{ attachment: { attachmentId: never; mediaType: 'image/png'; bytes: number; width: number; height: number }; data: string }>> =
|
||||
() => Promise.resolve(ok({ attachment: { attachmentId: 'a' as never, mediaType: 'image/png', bytes: 1, width: 1, height: 1 }, data: 'AA==' }))
|
||||
onCancel: (payload: unknown) => Promise<RpcResponse<{ accepted: true }>> = () => Promise.resolve(ok({ accepted: true as const }))
|
||||
onDescribe: (payload: unknown) => Promise<RpcResponse<{ version: string; cwd: string; attachedSessions: number }>> =
|
||||
() => Promise.resolve(ok({ version: '0-fake', cwd: '/f', attachedSessions: 0 }))
|
||||
@@ -64,6 +66,7 @@ export class FakeApiClient implements IApiClient {
|
||||
history: (payload: { sessionId: SessionId; beforeSeq?: number; maxMessages?: number }) =>
|
||||
this.record('session.history', payload, this.onHistory(payload)),
|
||||
prompt: (payload: unknown) => this.record('session.prompt', payload, this.onPrompt(payload)),
|
||||
attachment: (payload: unknown) => this.record('session.attachment', payload, this.onAttachment(payload)),
|
||||
cancel: (payload: unknown) => this.record('session.cancel', payload, this.onCancel(payload)),
|
||||
}
|
||||
|
||||
|
||||
@@ -167,7 +167,7 @@ describe('createFixtureApi', () => {
|
||||
expect(second[1]?.rpcId).toBe(first[1]?.rpcId) // stable rpcId across replays (host replay semantics)
|
||||
})
|
||||
|
||||
it('steer with no replay in flight falls through to a fresh queued turn; non-text blocks stringify empty', async () => {
|
||||
it('steer with no replay in flight promotes image bytes to a session-scoped reference', async () => {
|
||||
const api = createFixtureApi()
|
||||
const abort = new AbortController()
|
||||
const framesPromise = collect<MuxFrame>(api.events.mux(req({}), abort.signal), abort,
|
||||
@@ -175,14 +175,36 @@ describe('createFixtureApi', () => {
|
||||
await new Promise(resolve => setTimeout(resolve, 10))
|
||||
const created = await api.sessions.create(req({}))
|
||||
if (!created.result.ok) throw new Error('create failed')
|
||||
// steer while idle + a non-text content block (covers the '' arm of the text join).
|
||||
// steer while idle + an image: the fixture mirrors the host's durable send boundary.
|
||||
await api.sessions.prompt(req({
|
||||
sessionId: created.result.value.sessionId, mode: 'steer' as const,
|
||||
content: [{ type: 'text' as const, text: '短' }, { type: 'image', data: 'x' } as never],
|
||||
content: [{ type: 'text' as const, text: '短' }, {
|
||||
type: 'image' as const,
|
||||
mediaType: 'image/png' as const,
|
||||
data: 'iVBORw0KGgoAAAANSUhEUgAAAKAAAABaCAYAAAA/xl1SAAAAvklEQVR42u3SMQ0AAAjAMIyhELM4AAe8PD1qYFlk9cCXEAEDYkAwIAYEA2JAMCAGBANiQDAgBgQDYkAwIAYEA2JAMCAGBANiQDAgBgQDYkAwIAYEA2JAMCAGxIBCYEAMCAbEgGBADAgGxIBgQAwIBsSAYEAMCAbEgGBADAgGxIBgQAwIBsSAYEAMCAbEgGBADAgGxIAYEAyIAcGAGBAMiAHBgBgQDIgBwYAYEAyIAcGAGBAMiAHBgBgQDIgB4bYWLb6pnOb1xAAAAABJRU5ErkJggg==',
|
||||
name: 'pixel.png',
|
||||
}],
|
||||
}))
|
||||
const frames = await framesPromise
|
||||
const types = frames.filter((f): f is Extract<MuxFrame, { type: 'session/event' }> => f.type === 'session/event').map(f => f.event.type)
|
||||
expect(types[0]).toBe('turn/start') // idle steer degraded to a queued turn, not a steering insert
|
||||
const user = frames.find((f): f is Extract<MuxFrame, { type: 'session/event' }> =>
|
||||
f.type === 'session/event' && f.event.type === 'user/message')
|
||||
const image = ((user?.event.data as { content?: { type: string; attachment?: { attachmentId: never } }[] } | undefined)?.content)
|
||||
?.find(block => block.type === 'image')
|
||||
expect(image?.attachment).toBeDefined()
|
||||
if (image?.attachment === undefined) throw new Error('fixture image missing')
|
||||
const loaded = await api.sessions.attachment(req({
|
||||
sessionId: created.result.value.sessionId,
|
||||
attachmentId: image.attachment.attachmentId,
|
||||
}))
|
||||
expect(loaded.result).toMatchObject({ ok: true, value: { attachment: { name: 'pixel.png' } } })
|
||||
const denied = await api.sessions.attachment(req({
|
||||
sessionId: sid('fx-beta'), attachmentId: image.attachment.attachmentId,
|
||||
}))
|
||||
expect(denied.result).toMatchObject({
|
||||
ok: false, error: { details: { reason: 'ATTACHMENT_NOT_REFERENCED' } },
|
||||
})
|
||||
})
|
||||
|
||||
it('gamma interval flip emits host/session-status and a running log-less session subscribes at lastSeq -1', async () => {
|
||||
@@ -306,6 +328,7 @@ describe('FixtureApiClient (protocol-level fake carrier)', () => {
|
||||
const id = created.result.value.sessionId
|
||||
expect((await client.sessions.history({ sessionId: id })).result.ok).toBe(true)
|
||||
expect((await client.sessions.prompt({ sessionId: id, mode: 'queue', content: [{ type: 'text', text: '嗨' }] })).result.ok).toBe(true)
|
||||
expect((await client.sessions.attachment({ sessionId: sid('fx-alpha'), attachmentId: 'fixture:image' as never })).result.ok).toBe(true)
|
||||
expect((await client.sessions.cancel({ sessionId: id })).result.ok).toBe(true)
|
||||
expect((await client.host.describe({})).result.ok).toBe(true)
|
||||
})
|
||||
|
||||
@@ -8,6 +8,9 @@
|
||||
"src"
|
||||
],
|
||||
"references": [
|
||||
{
|
||||
"path": "../../attachment/attachment"
|
||||
},
|
||||
{
|
||||
"path": "../../llm/llm"
|
||||
},
|
||||
|
||||
@@ -35,6 +35,7 @@
|
||||
},
|
||||
"license": "BSD-3-Clause",
|
||||
"dependencies": {
|
||||
"@deepseek-ai/dsh-attachment": "workspace:^",
|
||||
"@deepseek-ai/dsh-client-connection": "workspace:^",
|
||||
"@deepseek-ai/dsh-client-ui-slots": "workspace:^",
|
||||
"@deepseek-ai/dsh-llm": "workspace:^",
|
||||
|
||||
@@ -4,6 +4,7 @@
|
||||
// string here (narrow to real brands when convenient).
|
||||
|
||||
import type { ContentBlock } from '@deepseek-ai/dsh-llm/types'
|
||||
import type { ImageAttachmentRef } from '@deepseek-ai/dsh-attachment'
|
||||
import type { RpcError, RpcId, SessionId, ToolCallView, ToolResultView } from '@deepseek-ai/dsh-client-connection/client'
|
||||
|
||||
/** Assistant content blocks sorted by what the UI cares about
|
||||
@@ -11,6 +12,7 @@ import type { RpcError, RpcId, SessionId, ToolCallView, ToolResultView } from '@
|
||||
export type AssistantBlock =
|
||||
| { kind: 'text'; text: string }
|
||||
| { kind: 'reasoning'; text: string }
|
||||
| { kind: 'image'; attachment: ImageAttachmentRef; alt?: string }
|
||||
| { kind: 'tool-call'; callId: string; name: string; argsRaw: string }
|
||||
| { kind: 'other'; block: unknown }
|
||||
|
||||
@@ -32,6 +34,10 @@ export function toAssistantBlock(block: ContentBlock): AssistantBlock {
|
||||
switch (block.type) {
|
||||
case 'text': return { kind: 'text', text: block.text }
|
||||
case 'reasoning': return { kind: 'reasoning', text: block.text }
|
||||
case 'image': return {
|
||||
kind: 'image', attachment: block.attachment,
|
||||
...block.alt === undefined ? {} : { alt: block.alt },
|
||||
}
|
||||
case 'tool-call': return { kind: 'tool-call', callId: String(block.id), name: block.name, argsRaw: block.arguments }
|
||||
default: return { kind: 'other', block }
|
||||
}
|
||||
|
||||
@@ -3,9 +3,9 @@
|
||||
// created, they keep consuming mux frames in the background; React connects directly via
|
||||
// subscribe/getSnapshot.
|
||||
|
||||
import type { ContentBlock } from '@deepseek-ai/dsh-llm/types'
|
||||
import type { AttachmentIdType, ImageAttachmentRef } from '@deepseek-ai/dsh-attachment'
|
||||
import type { SessionEvent } from '@deepseek-ai/dsh-session/types'
|
||||
import type { HistoryEntry, IApiClient, MuxFrame, RpcError, RpcId, RpcResult, SessionId, ToolEventView } from '@deepseek-ai/dsh-client-connection/client'
|
||||
import type { HistoryEntry, IApiClient, MuxFrame, PromptContentPart, RpcError, RpcId, RpcResult, SessionId, ToolEventView } from '@deepseek-ai/dsh-client-connection/client'
|
||||
import { transportError } from '@deepseek-ai/dsh-client-connection/client'
|
||||
import type { ObservableSnapshot } from '../contract/store.ts'
|
||||
import type {
|
||||
@@ -82,11 +82,11 @@ export class Session implements ObservableSnapshot<ConversationSnapshot> {
|
||||
|
||||
/**
|
||||
* Send (queue/steer passed through 1:1); failures land in the snapshot's promptError.
|
||||
* @param content - core content blocks verbatim.
|
||||
* @param content - text plus browser-owned temporary image uploads.
|
||||
* @param mode - queue appends after the current turn; steer interrupts it.
|
||||
* @returns the prompt result (also mirrored into promptError on failure).
|
||||
*/
|
||||
async prompt(content: ContentBlock[], mode: 'queue' | 'steer'): Promise<RpcResult<{ accepted: true }>> {
|
||||
async prompt(content: PromptContentPart[], mode: 'queue' | 'steer'): Promise<RpcResult<{ accepted: true }>> {
|
||||
this.promptError = null
|
||||
this.lastAgentError = null
|
||||
this.notifier.markDirty()
|
||||
@@ -103,6 +103,23 @@ export class Session implements ObservableSnapshot<ConversationSnapshot> {
|
||||
return result
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve one image referenced by this session into browser-consumable bytes.
|
||||
* @param attachmentId - opaque id found in the folded session log.
|
||||
* @returns the authenticated reference and decoded bytes.
|
||||
*/
|
||||
async readAttachment(attachmentId: AttachmentIdType): Promise<RpcResult<{ attachment: ImageAttachmentRef; data: Uint8Array }>> {
|
||||
try {
|
||||
const result = (await this.api.sessions.attachment({ sessionId: this.sessionId, attachmentId })).result
|
||||
if (!result.ok) return result
|
||||
const binary = atob(result.value.data)
|
||||
const data = Uint8Array.from(binary, char => char.charCodeAt(0))
|
||||
return { ok: true, value: { attachment: result.value.attachment, data } }
|
||||
} catch (error) {
|
||||
return transportError(error)
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Stop: contract session.cancel 1:1; failures land in promptError (same error-strip display slot).
|
||||
* @returns the cancel result.
|
||||
|
||||
@@ -1,22 +1,30 @@
|
||||
/** Assistant block classifier (moved here with sessions/conversation.ts). */
|
||||
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { AttachmentId } from '@deepseek-ai/dsh-attachment'
|
||||
import type { ContentBlock } from '@deepseek-ai/dsh-client-connection/client'
|
||||
import { toAssistantBlock, toAssistantBlocks } from '../src/client/sessions/conversation.ts'
|
||||
|
||||
describe('toAssistantBlock', () => {
|
||||
it('classifies the four block shapes', () => {
|
||||
const attachment = {
|
||||
attachmentId: AttachmentId(`sha256:${'a'.repeat(64)}`),
|
||||
mediaType: 'image/png' as const,
|
||||
bytes: 68,
|
||||
width: 1,
|
||||
height: 1,
|
||||
}
|
||||
const blocks: ContentBlock[] = [
|
||||
{ type: 'text', text: '正文' },
|
||||
{ type: 'reasoning', text: '思考' },
|
||||
{ type: 'tool-call', id: 'c1', name: 'echo', arguments: '{}' } as ContentBlock,
|
||||
{ type: 'image', data: 'x' } as unknown as ContentBlock,
|
||||
{ type: 'image', attachment },
|
||||
]
|
||||
expect(toAssistantBlocks(blocks)).toEqual([
|
||||
{ kind: 'text', text: '正文' },
|
||||
{ kind: 'reasoning', text: '思考' },
|
||||
{ kind: 'tool-call', callId: 'c1', name: 'echo', argsRaw: '{}' },
|
||||
{ kind: 'other', block: blocks[3] },
|
||||
{ kind: 'image', attachment },
|
||||
])
|
||||
expect(toAssistantBlock(blocks[0] as ContentBlock)).toEqual({ kind: 'text', text: '正文' })
|
||||
})
|
||||
|
||||
@@ -51,6 +51,8 @@ export class FakeApiClient implements IApiClient {
|
||||
() => Promise.resolve(ok({ events: [], hasMore: false }))
|
||||
|
||||
onPrompt: (payload: unknown) => Promise<RpcResponse<{ accepted: true }>> = () => Promise.resolve(ok({ accepted: true as const }))
|
||||
onAttachment: (payload: unknown) => Promise<RpcResponse<{ attachment: { attachmentId: never; mediaType: 'image/png'; bytes: number; width: number; height: number }; data: string }>> =
|
||||
() => Promise.resolve(ok({ attachment: { attachmentId: 'a' as never, mediaType: 'image/png', bytes: 1, width: 1, height: 1 }, data: 'AA==' }))
|
||||
onCancel: (payload: unknown) => Promise<RpcResponse<{ accepted: true }>> = () => Promise.resolve(ok({ accepted: true as const }))
|
||||
onDescribe: (payload: unknown) => Promise<RpcResponse<{ version: string; cwd: string; attachedSessions: number }>> =
|
||||
() => Promise.resolve(ok({ version: '0-fake', cwd: '/f', attachedSessions: 0 }))
|
||||
@@ -67,6 +69,7 @@ export class FakeApiClient implements IApiClient {
|
||||
history: (payload: { sessionId: SessionId; beforeSeq?: number; maxMessages?: number }) =>
|
||||
this.record('session.history', payload, this.onHistory(payload)),
|
||||
prompt: (payload: unknown) => this.record('session.prompt', payload, this.onPrompt(payload)),
|
||||
attachment: (payload: unknown) => this.record('session.attachment', payload, this.onAttachment(payload)),
|
||||
cancel: (payload: unknown) => this.record('session.cancel', payload, this.onCancel(payload)),
|
||||
}
|
||||
|
||||
|
||||
@@ -239,6 +239,21 @@ describe('prompt and cancel errors', () => {
|
||||
expect(result.ok).toBe(false)
|
||||
expect(session.getSnapshot().promptError).toMatchObject({ op: 'stop', error: { code: 'internal' } })
|
||||
})
|
||||
|
||||
it('reads session-authorized attachment bytes and keeps the opaque id on the wire', async () => {
|
||||
const { api, session } = makeSession()
|
||||
const result = await session.readAttachment('attachment-1' as never)
|
||||
expect(result).toEqual({
|
||||
ok: true,
|
||||
value: {
|
||||
attachment: { attachmentId: 'a', mediaType: 'image/png', bytes: 1, width: 1, height: 1 },
|
||||
data: Uint8Array.of(0),
|
||||
},
|
||||
})
|
||||
expect(api.callsOf('session.attachment')).toEqual([{
|
||||
sessionId: SID, attachmentId: 'attachment-1',
|
||||
}])
|
||||
})
|
||||
})
|
||||
|
||||
describe('pending interactions', () => {
|
||||
|
||||
@@ -8,6 +8,9 @@
|
||||
"src"
|
||||
],
|
||||
"references": [
|
||||
{
|
||||
"path": "../../attachment/attachment"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/cordis"
|
||||
},
|
||||
|
||||
@@ -34,6 +34,7 @@
|
||||
},
|
||||
"license": "BSD-3-Clause",
|
||||
"dependencies": {
|
||||
"@deepseek-ai/dsh-attachment": "workspace:^",
|
||||
"@deepseek-ai/dsh-client-i18n": "workspace:^",
|
||||
"@deepseek-ai/dsh-client-runtime": "workspace:^",
|
||||
"@deepseek-ai/dsh-client-ui-layout": "workspace:^",
|
||||
|
||||
@@ -12,7 +12,9 @@ import type { SessionId, SessionsService, SlotsService } from '@deepseek-ai/dsh-
|
||||
import type { LayoutService } from '@deepseek-ai/dsh-client-ui-layout/client'
|
||||
import type { I18nService } from '@deepseek-ai/dsh-client-i18n/client'
|
||||
import type { SelectionTarget } from './contract/views.ts'
|
||||
import type { ConversationInjected, DetailsInjected, EmptyStateInjected } from './contract/slots.ts'
|
||||
import type {
|
||||
ComposerAttachment, ConversationInjected, DetailsInjected, EmptyStateInjected,
|
||||
} from './contract/slots.ts'
|
||||
import { createChatStore } from './stores.ts'
|
||||
import { ConversationService } from './service.ts'
|
||||
import { ToolViewRegistry } from './toolviews/registry.ts'
|
||||
@@ -92,15 +94,25 @@ export function apply(ctx: Context): void {
|
||||
subscribe: fn => conversation.subscribeViews(fn),
|
||||
version: () => conversation.viewsVersion(),
|
||||
},
|
||||
send: (text, mode) => {
|
||||
addImages: (files) => {
|
||||
const images = conversation.createDraftImages(files)
|
||||
actions.addImages(images.map(image => image.id))
|
||||
},
|
||||
removeImage: (id) => {
|
||||
conversation.releaseDraftImage(id)
|
||||
actions.removeImage(id)
|
||||
},
|
||||
draftImages: ids => conversation.draftImages(ids),
|
||||
send: (text, images: readonly ComposerAttachment[], mode) => {
|
||||
const trimmed = text.trim()
|
||||
if (trimmed === '') return
|
||||
if (trimmed === '' && images.length === 0) return
|
||||
// Optimistic clear with failure restore (choreography lives with the
|
||||
// sender; the business failure also lands in snapshot.promptError).
|
||||
// The store write path stays inside the declared actions set:
|
||||
// restoreDraft itself no-ops once the user typed something new.
|
||||
// The store write path stays inside the declared actions set.
|
||||
actions.clearDraft()
|
||||
void scoped.send(trimmed, mode).catch(() => { actions.restoreDraft(trimmed) })
|
||||
void scoped.send(trimmed, mode, images.map(image => image.file))
|
||||
.then(() => { conversation.releaseDraftImages(images) })
|
||||
.catch(() => { actions.restoreDraft(trimmed, images.map(image => image.id)) })
|
||||
},
|
||||
stop: () => {
|
||||
scoped.cancel().catch(() => {
|
||||
|
||||
@@ -8,6 +8,7 @@ import { memo } from 'react'
|
||||
import type { AssistantBlock } from '@deepseek-ai/dsh-client-runtime/client'
|
||||
import { IconThinkOutline14, JsonBlock, MessageText } from '@deepseek-ai/dsh-client-ui-primitives'
|
||||
import { ToolRow } from './ToolRow.tsx'
|
||||
import { ImageGallery, type ImageLoader } from './MessageImage.tsx'
|
||||
import css from './AssistantMarkdown.module.css'
|
||||
|
||||
export interface AssistantMarkdownProps {
|
||||
@@ -15,6 +16,7 @@ export interface AssistantMarkdownProps {
|
||||
streaming: boolean
|
||||
/** Frozen partial of an aborted turn: rendered with a 已停止 marker, no pulse. */
|
||||
interrupted?: boolean | undefined
|
||||
loadImage?: ImageLoader
|
||||
}
|
||||
|
||||
function firstLine(text: string): string {
|
||||
@@ -36,14 +38,17 @@ function ThinkRow({ text, running }: { text: string; running: boolean }) {
|
||||
)
|
||||
}
|
||||
|
||||
export const AssistantMarkdown = memo(function AssistantMarkdown({ blocks, streaming, interrupted }: AssistantMarkdownProps) {
|
||||
export const AssistantMarkdown = memo(function AssistantMarkdown({ blocks, streaming, interrupted, loadImage = unavailableImage }: AssistantMarkdownProps) {
|
||||
const last = blocks.length - 1
|
||||
const images = blocks.filter((block): block is Extract<AssistantBlock, { kind: 'image' }> => block.kind === 'image')
|
||||
return (
|
||||
<div className={css.root} data-streaming={streaming || undefined}>
|
||||
<ImageGallery images={images} load={loadImage} align="start" />
|
||||
{blocks.map((block, i) => {
|
||||
switch (block.kind) {
|
||||
case 'text': return <MessageText key={i} text={block.text} />
|
||||
case 'reasoning': return <ThinkRow key={i} text={block.text} running={streaming && i === last} />
|
||||
case 'image': return null
|
||||
// Tool-call heads render as tool rows in the chat view's grouping pass.
|
||||
case 'tool-call': return null
|
||||
default: return <JsonBlock key={i} label="未知内容块" payload={block.block} />
|
||||
@@ -54,3 +59,7 @@ export const AssistantMarkdown = memo(function AssistantMarkdown({ blocks, strea
|
||||
</div>
|
||||
)
|
||||
})
|
||||
|
||||
function unavailableImage(): Promise<string> {
|
||||
return Promise.reject(new Error('图片读取服务不可用'))
|
||||
}
|
||||
|
||||
@@ -11,8 +11,9 @@
|
||||
// map but only rows whose own selected bit flipped.
|
||||
|
||||
import {
|
||||
memo, useLayoutEffect, useMemo, useRef, useState, type FC, type ReactNode,
|
||||
memo, useCallback, useLayoutEffect, useMemo, useRef, useState, type FC, type ReactNode,
|
||||
} from 'react'
|
||||
import type { ImageAttachmentRef } from '@deepseek-ai/dsh-attachment'
|
||||
import type {
|
||||
ConversationNode, ConversationSnapshot, RunningToolCall, SessionId, ToolResultNode,
|
||||
} from '@deepseek-ai/dsh-client-runtime/client'
|
||||
@@ -24,6 +25,7 @@ import type { ToolViewResolver } from '../contract/toolview.ts'
|
||||
import { deriveChatFlow, type ChatFlowItem } from './chat-flow.ts'
|
||||
import { AssistantMarkdown } from './AssistantMarkdown.tsx'
|
||||
import { MessageItem } from './MessageItem.tsx'
|
||||
import type { ImageLoader } from './MessageImage.tsx'
|
||||
import { PendingCard } from './PendingCard.tsx'
|
||||
import { ToolViewOutlet } from './ToolViewOutlet.tsx'
|
||||
import css from './ChatView.module.css'
|
||||
@@ -32,6 +34,7 @@ import css from './ChatView.module.css'
|
||||
export interface ChatViewDeps {
|
||||
toolviews: ToolViewResolver
|
||||
t: Translate
|
||||
resolveImage?(sessionId: SessionId, attachment: ImageAttachmentRef): Promise<string>
|
||||
}
|
||||
|
||||
const FOLLOW_THRESHOLD = 24
|
||||
@@ -102,16 +105,17 @@ const ToolGroup = memo(function ToolGroup({ registry, sessionId, useSession, t,
|
||||
|
||||
/** The streaming partial, isolated so chunk batches re-render only this tail.
|
||||
* onGrow lets the scroll owner follow content the parent never re-renders for. */
|
||||
function StreamingTail({ useSession, onGrow }: {
|
||||
function StreamingTail({ useSession, onGrow, loadImage }: {
|
||||
useSession: UseConversation
|
||||
onGrow: () => void
|
||||
loadImage: ImageLoader
|
||||
}) {
|
||||
const partial = useSession((s) => s.partial)
|
||||
useLayoutEffect(() => {
|
||||
onGrow()
|
||||
})
|
||||
if (partial === null) return null
|
||||
return <AssistantMarkdown blocks={partial.blocks} streaming />
|
||||
return <AssistantMarkdown blocks={partial.blocks} streaming loadImage={loadImage} />
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -120,7 +124,7 @@ function StreamingTail({ useSession, onGrow }: {
|
||||
* @returns the ConvViewProps component registered as the chat view.
|
||||
*/
|
||||
export function createChatView(deps: ChatViewDeps): FC<ConvViewProps> {
|
||||
const { toolviews, t } = deps
|
||||
const { toolviews, t, resolveImage = unavailableImage } = deps
|
||||
|
||||
return function ChatView({ sessionId, useSession: useSessionWide, useStore, actions }: ConvViewProps) {
|
||||
const useSession = useSessionWide as UseConversation
|
||||
@@ -132,6 +136,10 @@ export function createChatView(deps: ChatViewDeps): FC<ConvViewProps> {
|
||||
const hasMore = useSession((s) => s.hasMore)
|
||||
const loadingOlder = useSession((s) => s.loadingOlder)
|
||||
const selectedCallId = useStore((s) => s.selection?.callId)
|
||||
const loadImage = useCallback<ImageLoader>(
|
||||
attachment => resolveImage(sessionId, attachment),
|
||||
[resolveImage, sessionId],
|
||||
)
|
||||
|
||||
const items = useMemo(() => deriveChatFlow(nodes), [nodes])
|
||||
|
||||
@@ -229,11 +237,11 @@ export function createChatView(deps: ChatViewDeps): FC<ConvViewProps> {
|
||||
}
|
||||
const node: ConversationNode = item.node
|
||||
if (node.kind === 'assistant') {
|
||||
return <AssistantMarkdown key={item.key} blocks={node.blocks} streaming={false} interrupted={node.interrupted} />
|
||||
return <AssistantMarkdown key={item.key} blocks={node.blocks} streaming={false} interrupted={node.interrupted} loadImage={loadImage} />
|
||||
}
|
||||
/* v8 ignore next -- tool-result never reaches here: deriveChatFlow folds them into groups. */
|
||||
if (node.kind === 'tool-result') return null
|
||||
return <MessageItem key={item.key} node={node} />
|
||||
return <MessageItem key={item.key} node={node} loadImage={loadImage} />
|
||||
}
|
||||
|
||||
return (
|
||||
@@ -250,7 +258,7 @@ export function createChatView(deps: ChatViewDeps): FC<ConvViewProps> {
|
||||
</div>
|
||||
)}
|
||||
{items.map(renderItem)}
|
||||
<StreamingTail useSession={useSession} onGrow={onGrow} />
|
||||
<StreamingTail useSession={useSession} onGrow={onGrow} loadImage={loadImage} />
|
||||
{runningCalls.length > 0 && (
|
||||
<div className={css.toolGroup}>
|
||||
{runningCalls.map((call) => (
|
||||
@@ -291,3 +299,7 @@ export function createChatView(deps: ChatViewDeps): FC<ConvViewProps> {
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
function unavailableImage(): Promise<string> {
|
||||
return Promise.reject(new Error('图片读取服务不可用'))
|
||||
}
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
.gallery {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 8px;
|
||||
width: min(240px, 100%);
|
||||
}
|
||||
|
||||
.gallery[data-align='end'] {
|
||||
justify-content: flex-end;
|
||||
align-self: flex-end;
|
||||
}
|
||||
|
||||
.gallery[data-align='start'] {
|
||||
justify-content: flex-start;
|
||||
align-self: flex-start;
|
||||
}
|
||||
|
||||
.frame {
|
||||
display: grid;
|
||||
flex: 0 0 auto;
|
||||
place-items: center;
|
||||
min-width: 44px;
|
||||
min-height: 44px;
|
||||
padding: 0;
|
||||
overflow: hidden;
|
||||
border: 1px solid var(--dsw-alias-border-l2-darkmode-thin);
|
||||
border-radius: 12px;
|
||||
background: var(--dsw-alias-interactive-bg-hover);
|
||||
cursor: zoom-in;
|
||||
}
|
||||
|
||||
.frame img {
|
||||
display: block;
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
object-fit: contain;
|
||||
}
|
||||
|
||||
.loading,
|
||||
.error {
|
||||
color: var(--dsw-alias-label-tertiary);
|
||||
font-size: 12px;
|
||||
line-height: 18px;
|
||||
}
|
||||
|
||||
.error {
|
||||
max-width: 240px;
|
||||
padding: 10px 12px;
|
||||
border: 1px solid var(--dsw-alias-border-l2-darkmode-thin);
|
||||
border-radius: 10px;
|
||||
background: var(--dsw-alias-interactive-bg-hover-danger);
|
||||
cursor: pointer;
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
import { useCallback, useEffect, useMemo, useState } from 'react'
|
||||
import type { ImageAttachmentRef } from '@deepseek-ai/dsh-attachment'
|
||||
import { ImageLightbox } from '../skeleton/ImageLightbox.tsx'
|
||||
import css from './MessageImage.module.css'
|
||||
|
||||
/** Loads a session-authorized durable image URL. */
|
||||
export type ImageLoader = (attachment: ImageAttachmentRef) => Promise<string>
|
||||
|
||||
/** Compact history renderer with retryable loading and double-click original preview. */
|
||||
export function MessageImage({ attachment, alt, load }: {
|
||||
attachment: ImageAttachmentRef
|
||||
alt?: string
|
||||
load: ImageLoader
|
||||
}) {
|
||||
const [src, setSrc] = useState<string | null>(null)
|
||||
const [error, setError] = useState(false)
|
||||
const [open, setOpen] = useState(false)
|
||||
const close = useCallback(() => { setOpen(false) }, [])
|
||||
const size = useMemo(() => {
|
||||
const scale = Math.min(1, 240 / attachment.width, 240 / attachment.height)
|
||||
return { width: Math.max(1, Math.round(attachment.width * scale)), height: Math.max(1, Math.round(attachment.height * scale)) }
|
||||
}, [attachment.height, attachment.width])
|
||||
|
||||
const request = useCallback(() => {
|
||||
setError(false)
|
||||
setSrc(null)
|
||||
void load(attachment).then(setSrc).catch(() => { setError(true) })
|
||||
}, [attachment, load])
|
||||
|
||||
useEffect(() => {
|
||||
let live = true
|
||||
setError(false)
|
||||
void load(attachment).then((url) => { if (live) setSrc(url) }).catch(() => { if (live) setError(true) })
|
||||
return () => { live = false }
|
||||
}, [attachment, load])
|
||||
|
||||
const label = alt ?? attachment.name ?? '图片'
|
||||
if (error) return <button type="button" className={css.error} onClick={request}>图片加载失败，点击重试</button>
|
||||
return (
|
||||
<>
|
||||
<button
|
||||
type="button"
|
||||
className={css.frame}
|
||||
style={size}
|
||||
title="双击查看原图"
|
||||
aria-label={`${label}，双击查看原图`}
|
||||
onDoubleClick={() => { if (src !== null) setOpen(true) }}
|
||||
>
|
||||
{src === null ? <span className={css.loading}>图片加载中…</span> : <img src={src} alt={label} />}
|
||||
</button>
|
||||
{open && src !== null && <ImageLightbox src={src} alt={label} onClose={close} />}
|
||||
</>
|
||||
)
|
||||
}
|
||||
|
||||
/** Wrapping image group shared by user and assistant history. */
|
||||
export function ImageGallery({ images, load, align }: {
|
||||
images: readonly { attachment: ImageAttachmentRef; alt?: string }[]
|
||||
load: ImageLoader
|
||||
align: 'start' | 'end'
|
||||
}) {
|
||||
if (images.length === 0) return null
|
||||
return (
|
||||
<div className={css.gallery} data-align={align}>
|
||||
{images.map((image, index) => (
|
||||
<MessageImage key={`${image.attachment.attachmentId}:${index}`} {...image} load={load} />
|
||||
))}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
@@ -7,9 +7,18 @@
|
||||
justify-content: flex-end;
|
||||
}
|
||||
|
||||
.userStack {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
align-items: flex-end;
|
||||
gap: 8px;
|
||||
min-width: 0;
|
||||
max-width: min(525px, 82%);
|
||||
}
|
||||
|
||||
.bubble {
|
||||
/* 525px cap inside the 736 column; percentage keeps narrow windows sane. */
|
||||
max-width: min(525px, 82%);
|
||||
max-width: 100%;
|
||||
background: var(--dsw-specific-bubble);
|
||||
border-radius: 22px;
|
||||
/* 44px single-line bubble: 24 line + 10 vertical padding each side. */
|
||||
|
||||
@@ -9,33 +9,49 @@ import type {
|
||||
} from '@deepseek-ai/dsh-client-runtime/client'
|
||||
import { JsonBlock, MessageText } from '@deepseek-ai/dsh-client-ui-primitives'
|
||||
import css from './MessageItem.module.css'
|
||||
import { ImageGallery, type ImageLoader } from './MessageImage.tsx'
|
||||
|
||||
export interface MessageItemProps {
|
||||
node: UserMessageNode | SteeringMessageNode | ContextMessageNode | UnknownSurfaceNode
|
||||
loadImage?: ImageLoader
|
||||
}
|
||||
|
||||
function contentText(content: readonly unknown[]): { text: string; rest: unknown[] } {
|
||||
type UserImage = Extract<UserMessageNode['content'][number], { type: 'image' }>
|
||||
|
||||
function contentParts(content: readonly unknown[]): {
|
||||
text: string
|
||||
images: { attachment: UserImage['attachment']; alt?: string }[]
|
||||
rest: unknown[]
|
||||
} {
|
||||
const texts: string[] = []
|
||||
const images: { attachment: UserImage['attachment']; alt?: string }[] = []
|
||||
const rest: unknown[] = []
|
||||
for (const block of content) {
|
||||
const b = block as { type?: string; text?: string }
|
||||
const b = block as { type?: string; text?: string; attachment?: unknown; alt?: string }
|
||||
if (b.type === 'text' && typeof b.text === 'string') texts.push(b.text)
|
||||
else if (b.type === 'image' && b.attachment !== undefined) {
|
||||
const image = b as UserImage
|
||||
images.push({ attachment: image.attachment, ...image.alt === undefined ? {} : { alt: image.alt } })
|
||||
}
|
||||
else rest.push(block)
|
||||
}
|
||||
return { text: texts.join(''), rest }
|
||||
return { text: texts.join(''), images, rest }
|
||||
}
|
||||
|
||||
export const MessageItem = memo(function MessageItem({ node }: MessageItemProps) {
|
||||
export const MessageItem = memo(function MessageItem({ node, loadImage = unavailableImage }: MessageItemProps) {
|
||||
switch (node.kind) {
|
||||
case 'user':
|
||||
case 'steering': {
|
||||
const { text, rest } = contentText(node.content)
|
||||
const { text, images, rest } = contentParts(node.content)
|
||||
return (
|
||||
<div className={css.userRow}>
|
||||
<div className={css.bubble}>
|
||||
{node.kind === 'steering' && <span className={css.badge}>插话</span>}
|
||||
<MessageText text={text} />
|
||||
{rest.map((block, i) => <JsonBlock key={i} label="附加内容块" payload={block} />)}
|
||||
<div className={css.userStack}>
|
||||
<ImageGallery images={images} load={loadImage} align="end" />
|
||||
{(text !== '' || rest.length > 0 || node.kind === 'steering') && <div className={css.bubble}>
|
||||
{node.kind === 'steering' && <span className={css.badge}>插话</span>}
|
||||
<MessageText text={text} />
|
||||
{rest.map((block, i) => <JsonBlock key={i} label="附加内容块" payload={block} />)}
|
||||
</div>}
|
||||
</div>
|
||||
</div>
|
||||
)
|
||||
@@ -54,3 +70,7 @@ export const MessageItem = memo(function MessageItem({ node }: MessageItemProps)
|
||||
)
|
||||
}
|
||||
})
|
||||
|
||||
function unavailableImage(): Promise<string> {
|
||||
return Promise.reject(new Error('图片读取服务不可用'))
|
||||
}
|
||||
|
||||
@@ -46,7 +46,11 @@ export function registerChat(deps: RegisterChatDeps): () => void {
|
||||
id: 'chat',
|
||||
label: 'Chat',
|
||||
order: 0,
|
||||
component: createChatView({ toolviews, t }),
|
||||
component: createChatView({
|
||||
toolviews,
|
||||
t,
|
||||
resolveImage: (sessionId, attachment) => conversation.resolveImage(sessionId, attachment),
|
||||
}),
|
||||
chrome: { footer: StatsLine },
|
||||
})
|
||||
}
|
||||
|
||||
@@ -12,6 +12,13 @@ import type { SessionId } from '@deepseek-ai/dsh-client-runtime/client'
|
||||
import type { createChatStore } from '../stores.ts'
|
||||
import type { SelectionTarget, ViewEntry } from './views.ts'
|
||||
|
||||
/** Browser-owned image that has not crossed the durable host boundary. */
|
||||
export interface ComposerAttachment {
|
||||
id: string
|
||||
file: File
|
||||
previewUrl: string
|
||||
}
|
||||
|
||||
/** The shared chat store handle type (apply constructs one; conversation and details both declare it). */
|
||||
export type ChatStore = ReturnType<typeof createChatStore>
|
||||
|
||||
@@ -29,8 +36,14 @@ export interface ConversationInjected {
|
||||
subscribe(fn: () => void): () => void
|
||||
version(): number
|
||||
}
|
||||
/** Create browser previews and append their ids through the declared store action. */
|
||||
addImages(files: readonly File[]): void
|
||||
/** Release one browser preview and remove its id through the declared store action. */
|
||||
removeImage(id: string): void
|
||||
/** Resolve ordered store ids to the browser-owned draft attachments still available this runtime. */
|
||||
draftImages(ids: readonly string[]): readonly ComposerAttachment[]
|
||||
/** Send choreography: trims, clears the draft optimistically, restores it on failure. */
|
||||
send(text: string, mode: 'queue' | 'steer'): void
|
||||
send(text: string, images: readonly ComposerAttachment[], mode: 'queue' | 'steer'): void
|
||||
/** Cancel the in-flight turn (failure surfaces via snapshot.promptError). */
|
||||
stop(): void
|
||||
/** Selection write + details panel opening in one gesture (store action + layout orchestration). */
|
||||
@@ -60,7 +73,12 @@ export type DetailsSlotProps = PropsRuntime<'details'> & PropsStore<ChatStore> &
|
||||
/** Injected share of the no-session empty-state slot. */
|
||||
export interface EmptyStateInjected {
|
||||
/** The create → navigate → first-send chain, in one service call. */
|
||||
startSession(opts: { cwd?: string; text: string; mode: 'queue' | 'steer' }): Promise<void>
|
||||
startSession(opts: {
|
||||
cwd?: string
|
||||
text: string
|
||||
images?: readonly File[]
|
||||
mode: 'queue' | 'steer'
|
||||
}): Promise<void>
|
||||
}
|
||||
|
||||
/** Full empty-state component props (root slot: no store; cwd options derive from useSessions in-component). */
|
||||
|
||||
@@ -69,6 +69,12 @@ export interface ChatStoreState {
|
||||
selection: SelectionTarget | null
|
||||
/** Composer draft (persisted; survives session switches and reloads). */
|
||||
draft: string
|
||||
/**
|
||||
* Ordered browser-draft attachment ids. The matching File/object-URL
|
||||
* objects stay in ConversationService because they are runtime-only; stale
|
||||
* persisted ids are pruned by ConversationRoot after a page reload.
|
||||
*/
|
||||
imageIds: string[]
|
||||
/** Active conversation view id; null falls back to the first registered view. */
|
||||
view: ViewId | null
|
||||
}
|
||||
|
||||
@@ -22,8 +22,27 @@ import type { Context } from 'cordis'
|
||||
// SessionsService tags contexts with — scopeOf then always returns undefined
|
||||
// in the browser while unit tests (single-instance path resolution) stay green.
|
||||
import { scopeOf } from '@deepseek-ai/dsh-client-runtime/client'
|
||||
import type { Session, SessionsService } from '@deepseek-ai/dsh-client-runtime/client'
|
||||
import type { Session, SessionId, SessionsService } from '@deepseek-ai/dsh-client-runtime/client'
|
||||
import type { ImageAttachmentRef, ImageMediaType } from '@deepseek-ai/dsh-attachment'
|
||||
import type { ViewEntry, ViewId } from './index.ts'
|
||||
import type { ComposerAttachment } from './contract/slots.ts'
|
||||
|
||||
/** Opaque wrapper keeps browser `File` internals outside persisted store state. */
|
||||
class BrowserDraftAttachment implements ComposerAttachment {
|
||||
readonly id: string
|
||||
readonly previewUrl: string
|
||||
readonly #file: File
|
||||
|
||||
constructor(file: File) {
|
||||
this.id = crypto.randomUUID()
|
||||
this.previewUrl = URL.createObjectURL(file)
|
||||
this.#file = file
|
||||
}
|
||||
|
||||
get file(): File {
|
||||
return this.#file
|
||||
}
|
||||
}
|
||||
|
||||
/** Mutable view-registry cell (plain object: mutation never crosses the tracker proxy). */
|
||||
interface ViewsState {
|
||||
@@ -36,6 +55,9 @@ interface ViewsState {
|
||||
|
||||
/** Scope-addressed conversation service (root singleton, provided as `conversation`). */
|
||||
export class ConversationService extends Service {
|
||||
private readonly draftAttachments = new Map<string, BrowserDraftAttachment>()
|
||||
private readonly imageUrls = new Map<string, Promise<string>>()
|
||||
private readonly createdImageUrls = new Set<string>()
|
||||
private readonly viewsState: ViewsState = {
|
||||
entries: new Map(), cache: null, tick: 0, listeners: new Set(),
|
||||
}
|
||||
@@ -46,6 +68,12 @@ export class ConversationService extends Service {
|
||||
*/
|
||||
constructor(ctx: Context) {
|
||||
super(ctx, 'conversation')
|
||||
ctx.effect(() => () => {
|
||||
for (const url of this.createdImageUrls) URL.revokeObjectURL(url)
|
||||
this.createdImageUrls.clear()
|
||||
this.draftAttachments.clear()
|
||||
this.imageUrls.clear()
|
||||
}, 'conversation attachment URL cache')
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -54,13 +82,98 @@ export class ConversationService extends Service {
|
||||
* exists for caller choreography (the composer restores the draft on it).
|
||||
* @param text - prompt text, sent verbatim as one text block.
|
||||
* @param mode - queue after the current turn, or steer into it.
|
||||
* @param images - browser-owned temporary images promoted by the host during this call.
|
||||
*/
|
||||
async send(text: string, mode: 'queue' | 'steer'): Promise<void> {
|
||||
async send(text: string, mode: 'queue' | 'steer', images: readonly File[] = []): Promise<void> {
|
||||
const session = this.scopedSession('send')
|
||||
const result = await session.prompt([{ type: 'text', text }], mode)
|
||||
const uploaded = await Promise.all(images.map(async file => ({
|
||||
type: 'image' as const,
|
||||
mediaType: imageMediaType(file.type),
|
||||
data: bytesToBase64(new Uint8Array(await file.arrayBuffer())),
|
||||
...(file.name === '' ? {} : { name: file.name }),
|
||||
})))
|
||||
const content = [...uploaded, ...(text === '' ? [] : [{ type: 'text' as const, text }])]
|
||||
const result = await session.prompt(content, mode)
|
||||
if (!result.ok) throw new Error(`conversation.send failed: ${result.error.code}: ${result.error.message}`)
|
||||
}
|
||||
|
||||
/**
|
||||
* Create runtime-only draft attachments and their object URLs.
|
||||
* @param files - browser-owned image files.
|
||||
* @returns ordered attachment descriptors whose ids may enter the chat store.
|
||||
*/
|
||||
createDraftImages(files: readonly File[]): readonly ComposerAttachment[] {
|
||||
return files.map((file) => {
|
||||
const attachment = new BrowserDraftAttachment(file)
|
||||
this.draftAttachments.set(attachment.id, attachment)
|
||||
this.createdImageUrls.add(attachment.previewUrl)
|
||||
return attachment
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve ordered store ids to runtime-owned draft attachments.
|
||||
* @param ids - ordered ids from the chat store.
|
||||
* @returns attachments still available in this browser runtime.
|
||||
*/
|
||||
draftImages(ids: readonly string[]): readonly ComposerAttachment[] {
|
||||
const attachments: ComposerAttachment[] = []
|
||||
for (const id of ids) {
|
||||
const attachment = this.draftAttachments.get(id)
|
||||
if (attachment !== undefined) attachments.push(attachment)
|
||||
}
|
||||
return attachments
|
||||
}
|
||||
|
||||
/**
|
||||
* Release one draft attachment preview.
|
||||
* @param id - draft-local attachment id.
|
||||
*/
|
||||
releaseDraftImage(id: string): void {
|
||||
const attachment = this.draftAttachments.get(id)
|
||||
if (attachment === undefined) return
|
||||
this.draftAttachments.delete(id)
|
||||
this.createdImageUrls.delete(attachment.previewUrl)
|
||||
revokePreview(attachment.previewUrl)
|
||||
}
|
||||
|
||||
/**
|
||||
* Release sent draft attachment previews.
|
||||
* @param attachments - successfully submitted attachments.
|
||||
*/
|
||||
releaseDraftImages(attachments: readonly ComposerAttachment[]): void {
|
||||
for (const attachment of attachments) this.releaseDraftImage(attachment.id)
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolve and cache one session-authorized historical image as an object URL.
|
||||
* @param sessionId - session whose durable log grants the read.
|
||||
* @param attachment - immutable reference from that log.
|
||||
* @returns a browser URL for inline and original-size display.
|
||||
*/
|
||||
resolveImage(sessionId: SessionId, attachment: ImageAttachmentRef): Promise<string> {
|
||||
const key = `${sessionId}:${attachment.attachmentId}`
|
||||
const cached = this.imageUrls.get(key)
|
||||
if (cached !== undefined) return cached
|
||||
const pending = this.requireSessions().manager.get(sessionId).readAttachment(attachment.attachmentId)
|
||||
.then((result) => {
|
||||
if (!result.ok) throw new Error(`${result.error.code}: ${result.error.message}`)
|
||||
if (typeof URL.createObjectURL !== 'function') {
|
||||
return `data:${result.value.attachment.mediaType};base64,${bytesToBase64(result.value.data)}`
|
||||
}
|
||||
const bytes = Uint8Array.from(result.value.data)
|
||||
const url = URL.createObjectURL(new Blob([bytes.buffer], { type: result.value.attachment.mediaType }))
|
||||
this.createdImageUrls.add(url)
|
||||
return url
|
||||
})
|
||||
.catch((error: unknown) => {
|
||||
this.imageUrls.delete(key)
|
||||
throw error
|
||||
})
|
||||
this.imageUrls.set(key, pending)
|
||||
return pending
|
||||
}
|
||||
|
||||
/** Cancel the scoped session's in-flight turn (failures land in promptError and reject, as in send). */
|
||||
async cancel(): Promise<void> {
|
||||
const session = this.scopedSession('cancel')
|
||||
@@ -132,7 +245,12 @@ export class ConversationService extends Service {
|
||||
* awaited through the RPC round trip).
|
||||
* @param opts - project directory, prompt text, and send mode.
|
||||
*/
|
||||
async startSession(opts: { cwd?: string; text: string; mode: 'queue' | 'steer' }): Promise<void> {
|
||||
async startSession(opts: {
|
||||
cwd?: string
|
||||
text: string
|
||||
images?: readonly File[]
|
||||
mode: 'queue' | 'steer'
|
||||
}): Promise<void> {
|
||||
const sessions = this.requireSessions()
|
||||
const id = await sessions.create(opts.cwd === undefined ? {} : { cwd: opts.cwd })
|
||||
// The manager notifier flushes per microtask; one await guarantees the
|
||||
@@ -146,7 +264,7 @@ export class ConversationService extends Service {
|
||||
// global store and still binds this service to the scoped ctx.
|
||||
const scopedConversation = scoped.get('conversation')
|
||||
if (scopedConversation === undefined) throw new Error('conversation.startSession: conversation service unavailable through the new scope')
|
||||
await scopedConversation.send(opts.text, opts.mode)
|
||||
await scopedConversation.send(opts.text, opts.mode, opts.images ?? [])
|
||||
}
|
||||
|
||||
/** Resolve the caller scope's Session or throw on root contexts. */
|
||||
@@ -173,3 +291,28 @@ function bumpViews(state: ViewsState): void {
|
||||
state.tick += 1
|
||||
for (const fn of [...state.listeners]) fn()
|
||||
}
|
||||
|
||||
function imageMediaType(value: string): ImageMediaType {
|
||||
switch (value) {
|
||||
case 'image/png':
|
||||
case 'image/jpeg':
|
||||
case 'image/webp':
|
||||
case 'image/gif':
|
||||
return value
|
||||
default:
|
||||
throw new Error(`不支持的图片格式：${value || '未知格式'}`)
|
||||
}
|
||||
}
|
||||
|
||||
function bytesToBase64(data: Uint8Array): string {
|
||||
let binary = ''
|
||||
const chunk = 0x8000
|
||||
for (let offset = 0; offset < data.length; offset += chunk) {
|
||||
binary += String.fromCharCode(...data.subarray(offset, offset + chunk))
|
||||
}
|
||||
return btoa(binary)
|
||||
}
|
||||
|
||||
function revokePreview(url: string): void {
|
||||
if (url.startsWith('blob:')) URL.revokeObjectURL(url)
|
||||
}
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
// Breadcrumbs derive from useSessions with a pure parentId walk; the active
|
||||
// view id lives in the chat store's `view` field (per-session by store scope).
|
||||
|
||||
import { useMemo, useSyncExternalStore, type ReactNode } from 'react'
|
||||
import { useEffect, useMemo, useSyncExternalStore, type ReactNode } from 'react'
|
||||
import clsx from 'clsx'
|
||||
import { shallowEqual } from '@deepseek-ai/dsh-client-runtime/client'
|
||||
import type { SessionId, SessionListState, SessionSummary } from '@deepseek-ai/dsh-client-runtime/client'
|
||||
@@ -36,7 +36,7 @@ function deriveAncestry(list: SessionListState, id: SessionId): readonly Session
|
||||
|
||||
export function ConversationRoot({
|
||||
sessionId, useSession, useSessions, useStore, actions,
|
||||
views, send, stop, openDetails, loadOlder, open,
|
||||
views, addImages, removeImage, draftImages, send, stop, openDetails, loadOlder, open,
|
||||
}: ConversationRootProps) {
|
||||
useSyncExternalStore(views.subscribe, views.version)
|
||||
const list = views.list()
|
||||
@@ -47,11 +47,22 @@ export function ConversationRoot({
|
||||
|
||||
const ancestry = useSessions(s => deriveAncestry(s, sessionId), shallowEqual)
|
||||
const draft = useStore(s => s.draft)
|
||||
const imageIds = useStore(s => s.imageIds)
|
||||
const attachments = useMemo(() => draftImages(imageIds), [draftImages, imageIds])
|
||||
const running = useSession(s => s.running)
|
||||
const removed = useSession(s => s.removed)
|
||||
const promptError = useSession(s => s.promptError)
|
||||
const turns = useSession(s => countTurns(s))
|
||||
|
||||
// Browser File/object-URL values are intentionally runtime-only. A reload
|
||||
// may rehydrate ids whose objects no longer exist; prune those ids through
|
||||
// the declared store action after the first render.
|
||||
useEffect(() => {
|
||||
if (attachments.length !== imageIds.length) {
|
||||
actions.pruneImages(attachments.map(attachment => attachment.id))
|
||||
}
|
||||
}, [actions, attachments, imageIds])
|
||||
|
||||
const error: InputBarError | null = promptError === null
|
||||
? null
|
||||
: { op: promptError.op, message: `${promptError.error.message}（${promptError.error.code}）` }
|
||||
@@ -128,12 +139,15 @@ export function ConversationRoot({
|
||||
|
||||
<InputBar
|
||||
draft={draft}
|
||||
attachments={attachments}
|
||||
running={running}
|
||||
disabled={removed}
|
||||
error={error}
|
||||
variant="composer"
|
||||
onDraftChange={actions.setDraft}
|
||||
onSend={(mode) => { send(draft, mode) }}
|
||||
onAddImages={addImages}
|
||||
onRemoveAttachment={removeImage}
|
||||
onSend={(mode) => { send(draft, attachments, mode) }}
|
||||
onStop={stop}
|
||||
/>
|
||||
</div>
|
||||
|
||||
@@ -6,10 +6,10 @@
|
||||
// §6) plus a free-form new-directory input; submit runs the startSession
|
||||
// chain (create → open → send) in one service call.
|
||||
|
||||
import { useMemo, useState } from 'react'
|
||||
import { useEffect, useMemo, useRef, useState } from 'react'
|
||||
import { FishLogo } from '@deepseek-ai/dsh-client-ui-primitives'
|
||||
import type { SessionListState } from '@deepseek-ai/dsh-client-runtime/client'
|
||||
import type { EmptyStateSlotProps } from '../contract/slots.ts'
|
||||
import type { ComposerAttachment, EmptyStateSlotProps } from '../contract/slots.ts'
|
||||
import { InputBar } from './InputBar.tsx'
|
||||
import type { InputBarError } from './InputBar.tsx'
|
||||
import css from './EmptyState.module.css'
|
||||
@@ -36,6 +36,9 @@ export function EmptyState({ useSessions, startSession }: EmptyStateProps) {
|
||||
// Local viewing state: the empty state owns no session, so its draft is
|
||||
// ephemeral by design (drafts are keyed by session id; there is none yet).
|
||||
const [draft, setDraft] = useState('')
|
||||
const [attachments, setAttachments] = useState<readonly ComposerAttachment[]>([])
|
||||
const attachmentsRef = useRef(attachments)
|
||||
attachmentsRef.current = attachments
|
||||
const [cwd, setCwd] = useState<string>('')
|
||||
const [custom, setCustom] = useState(false)
|
||||
const [sending, setSending] = useState(false)
|
||||
@@ -44,11 +47,16 @@ export function EmptyState({ useSessions, startSession }: EmptyStateProps) {
|
||||
const submit = (mode: 'queue' | 'steer'): void => {
|
||||
const text = draft.trim()
|
||||
/* v8 ignore next -- defensive: InputBar disables send while empty. */
|
||||
if (text === '' || sending) return
|
||||
if ((text === '' && attachments.length === 0) || sending) return
|
||||
setSending(true)
|
||||
setError(null)
|
||||
const chosen = cwd.trim()
|
||||
startSession({ text, mode, ...(chosen === '' ? {} : { cwd: chosen }) })
|
||||
startSession({
|
||||
text,
|
||||
...(attachments.length === 0 ? {} : { images: attachments.map(item => item.file) }),
|
||||
mode,
|
||||
...(chosen === '' ? {} : { cwd: chosen }),
|
||||
})
|
||||
.catch((reason: unknown) => {
|
||||
// The empty state survives failure with the draft intact (no session
|
||||
// exists to carry promptError; this is the only local error surface).
|
||||
@@ -58,6 +66,24 @@ export function EmptyState({ useSessions, startSession }: EmptyStateProps) {
|
||||
// Success needs no cleanup: the session selection swaps this slot out for the session body.
|
||||
}
|
||||
|
||||
useEffect(() => () => {
|
||||
for (const attachment of attachmentsRef.current) URL.revokeObjectURL(attachment.previewUrl)
|
||||
}, [])
|
||||
|
||||
const addImages = (files: readonly File[]): void => {
|
||||
setAttachments(current => [...current, ...files.map(file => ({
|
||||
id: crypto.randomUUID(), file, previewUrl: URL.createObjectURL(file),
|
||||
}))])
|
||||
}
|
||||
|
||||
const removeImage = (id: string): void => {
|
||||
setAttachments((current) => {
|
||||
const removed = current.find(item => item.id === id)
|
||||
if (removed !== undefined) URL.revokeObjectURL(removed.previewUrl)
|
||||
return current.filter(item => item.id !== id)
|
||||
})
|
||||
}
|
||||
|
||||
const picker = (
|
||||
<div className={css.picker}>
|
||||
{custom
|
||||
@@ -102,6 +128,7 @@ export function EmptyState({ useSessions, startSession }: EmptyStateProps) {
|
||||
</div>
|
||||
<InputBar
|
||||
draft={draft}
|
||||
attachments={attachments}
|
||||
running={false}
|
||||
disabled={sending}
|
||||
error={error}
|
||||
@@ -109,6 +136,8 @@ export function EmptyState({ useSessions, startSession }: EmptyStateProps) {
|
||||
placeholder="Message to run task, plan and build"
|
||||
accessory={picker}
|
||||
onDraftChange={setDraft}
|
||||
onAddImages={addImages}
|
||||
onRemoveAttachment={removeImage}
|
||||
onSend={submit}
|
||||
/* v8 ignore next -- structural noop: hero never passes running=true, so stop is unreachable. */
|
||||
onStop={() => {}}
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
.backdrop {
|
||||
position: fixed;
|
||||
inset: 0;
|
||||
z-index: 1000;
|
||||
display: grid;
|
||||
place-items: center;
|
||||
padding: 40px;
|
||||
background: color-mix(in srgb, var(--dsw-alias-label-primary) 74%, transparent);
|
||||
}
|
||||
|
||||
.image {
|
||||
max-width: min(100%, 1600px);
|
||||
max-height: calc(100vh - 80px);
|
||||
object-fit: contain;
|
||||
border-radius: 12px;
|
||||
background: var(--dsw-specific-input-major);
|
||||
box-shadow: var(--dsw-shadow-lv3);
|
||||
}
|
||||
|
||||
.close {
|
||||
position: fixed;
|
||||
top: 20px;
|
||||
right: 20px;
|
||||
display: grid;
|
||||
place-items: center;
|
||||
width: 36px;
|
||||
height: 36px;
|
||||
border: 1px solid var(--dsw-alias-border-l2-darkmode-thin);
|
||||
border-radius: 999px;
|
||||
background: var(--dsw-specific-input-major);
|
||||
color: var(--dsw-alias-label-primary);
|
||||
font-size: 24px;
|
||||
cursor: pointer;
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
import { useEffect, useRef } from 'react'
|
||||
import css from './ImageLightbox.module.css'
|
||||
|
||||
/** Document-level original-image preview opened by an explicit double-click. */
|
||||
export function ImageLightbox({ src, alt, onClose }: { src: string; alt: string; onClose(): void }) {
|
||||
const closeRef = useRef<HTMLButtonElement | null>(null)
|
||||
const restoreRef = useRef<HTMLElement | null>(null)
|
||||
|
||||
useEffect(() => {
|
||||
restoreRef.current = document.activeElement instanceof HTMLElement ? document.activeElement : null
|
||||
closeRef.current?.focus()
|
||||
const onKeyDown = (event: globalThis.KeyboardEvent): void => {
|
||||
if (event.key === 'Escape') onClose()
|
||||
}
|
||||
window.addEventListener('keydown', onKeyDown)
|
||||
return () => {
|
||||
window.removeEventListener('keydown', onKeyDown)
|
||||
restoreRef.current?.focus()
|
||||
}
|
||||
}, [onClose])
|
||||
|
||||
return (
|
||||
<div
|
||||
className={css.backdrop}
|
||||
role="dialog"
|
||||
aria-modal="true"
|
||||
aria-label="原图预览"
|
||||
onMouseDown={(event) => { if (event.target === event.currentTarget) onClose() }}
|
||||
>
|
||||
<img className={css.image} src={src} alt={alt} />
|
||||
<button ref={closeRef} type="button" className={css.close} aria-label="关闭原图预览" onClick={onClose}>×</button>
|
||||
</div>
|
||||
)
|
||||
}
|
||||
@@ -32,6 +32,7 @@
|
||||
}
|
||||
|
||||
.card {
|
||||
position: relative;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
/* figma Input 34:11458: 12px between the text area and the button row. */
|
||||
@@ -49,6 +50,25 @@
|
||||
line-height: 24px;
|
||||
}
|
||||
|
||||
.dragActive {
|
||||
border-color: var(--dsw-alias-state-business-primary);
|
||||
box-shadow: 0 0 0 2px color-mix(in srgb, var(--dsw-alias-state-business-primary) 24%, transparent), var(--dsw-shadow-lv2);
|
||||
}
|
||||
|
||||
.dropHint {
|
||||
position: absolute;
|
||||
z-index: 2;
|
||||
inset: 4px;
|
||||
display: grid;
|
||||
place-items: center;
|
||||
border-radius: 16px;
|
||||
background: color-mix(in srgb, var(--dsw-specific-input-major) 88%, var(--dsw-alias-state-business-primary));
|
||||
color: var(--dsw-alias-state-business-primary);
|
||||
font-size: 14px;
|
||||
font-weight: 600;
|
||||
pointer-events: none;
|
||||
}
|
||||
|
||||
/* New-session state rounds up (figma: r24 and a taller box). */
|
||||
.hero .card {
|
||||
border-radius: 24px;
|
||||
@@ -61,6 +81,57 @@
|
||||
padding: 10px 12px 0;
|
||||
}
|
||||
|
||||
.attachments {
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
min-width: 0;
|
||||
padding: 12px 12px 0;
|
||||
overflow-x: auto;
|
||||
overflow-y: hidden;
|
||||
}
|
||||
|
||||
.attachment {
|
||||
position: relative;
|
||||
flex: 0 0 72px;
|
||||
width: 72px;
|
||||
height: 72px;
|
||||
}
|
||||
|
||||
.thumbnail {
|
||||
width: 72px;
|
||||
height: 72px;
|
||||
padding: 0;
|
||||
overflow: hidden;
|
||||
border: 1px solid var(--dsw-alias-border-l2-darkmode-thin);
|
||||
border-radius: 12px;
|
||||
background: var(--dsw-alias-interactive-bg-hover);
|
||||
cursor: zoom-in;
|
||||
}
|
||||
|
||||
.thumbnail img {
|
||||
width: 100%;
|
||||
height: 100%;
|
||||
object-fit: cover;
|
||||
}
|
||||
|
||||
.remove {
|
||||
position: absolute;
|
||||
top: -6px;
|
||||
right: -6px;
|
||||
display: grid;
|
||||
place-items: center;
|
||||
width: 22px;
|
||||
height: 22px;
|
||||
padding: 0;
|
||||
border: 1px solid var(--dsw-specific-input-major);
|
||||
border-radius: 999px;
|
||||
background: var(--dsw-alias-label-primary);
|
||||
color: var(--dsw-specific-input-major);
|
||||
font-size: 16px;
|
||||
line-height: 1;
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
/* Mirror-div auto-grow wrapper: the hidden mirror is in normal flow and sets the height
|
||||
(min 2 lines / max 14 lines); the textarea rides it absolutely. Mirror and textarea
|
||||
MUST share font, line-height, padding and wrapping rules or heights diverge. */
|
||||
|
||||
@@ -5,11 +5,19 @@
|
||||
// LOCKS the input: textarea disabled with the draft visible, stop is the only
|
||||
// action; the turn ending re-enables and refocuses.
|
||||
|
||||
import { useEffect, useRef } from 'react'
|
||||
import type { KeyboardEvent, MouseEvent, ReactNode } from 'react'
|
||||
import { useCallback, useEffect, useRef, useState } from 'react'
|
||||
import type { ClipboardEvent, DragEvent, KeyboardEvent, MouseEvent, ReactNode } from 'react'
|
||||
import clsx from 'clsx'
|
||||
import type { ComposerAttachment } from '../contract/slots.ts'
|
||||
import { ImageLightbox } from './ImageLightbox.tsx'
|
||||
import css from './InputBar.module.css'
|
||||
|
||||
const IMAGE_MEDIA_TYPES = new Set(['image/png', 'image/jpeg', 'image/webp', 'image/gif'])
|
||||
|
||||
function supportedImages(files: Iterable<File>): File[] {
|
||||
return [...files].filter(file => IMAGE_MEDIA_TYPES.has(file.type))
|
||||
}
|
||||
|
||||
/** Prompt failure surface (mirrors the session snapshot's promptError shape). */
|
||||
export interface InputBarError {
|
||||
op: 'send' | 'stop'
|
||||
@@ -18,6 +26,7 @@ export interface InputBarError {
|
||||
|
||||
export interface InputBarProps {
|
||||
draft: string
|
||||
attachments?: readonly ComposerAttachment[]
|
||||
running: boolean
|
||||
disabled: boolean
|
||||
error: InputBarError | null
|
||||
@@ -27,15 +36,22 @@ export interface InputBarProps {
|
||||
/** Optional leading accessory row content (the empty state mounts its cwd picker here). */
|
||||
accessory?: ReactNode
|
||||
onDraftChange: (text: string) => void
|
||||
onAddImages?: (files: readonly File[]) => void
|
||||
onRemoveAttachment?: (id: string) => void
|
||||
onSend: (mode: 'queue' | 'steer') => void
|
||||
onStop: () => void
|
||||
}
|
||||
|
||||
export function InputBar({
|
||||
draft, running, disabled, error, variant, placeholder, accessory, onDraftChange, onSend, onStop,
|
||||
draft, attachments = [], running, disabled, error, variant, placeholder, accessory,
|
||||
onDraftChange, onAddImages = () => {}, onRemoveAttachment = () => {}, onSend, onStop,
|
||||
}: InputBarProps) {
|
||||
const empty = draft.trim() === ''
|
||||
const empty = draft.trim() === '' && attachments.length === 0
|
||||
const [preview, setPreview] = useState<ComposerAttachment | null>(null)
|
||||
const [dragActive, setDragActive] = useState(false)
|
||||
const [dropError, setDropError] = useState<string | null>(null)
|
||||
const inputRef = useRef<HTMLTextAreaElement | null>(null)
|
||||
const dragDepthRef = useRef(0)
|
||||
// IME guard: composition Enter picks a candidate, it must not send. The ref outlives renders;
|
||||
// clearing is deferred one tick because Safari delivers the closing keydown AFTER compositionend.
|
||||
const composingRef = useRef(false)
|
||||
@@ -72,6 +88,56 @@ export function InputBar({
|
||||
if (!empty && !locked) onSend('queue')
|
||||
}
|
||||
|
||||
const onPaste = (event: ClipboardEvent<HTMLTextAreaElement>): void => {
|
||||
const files = [...event.clipboardData.items]
|
||||
.filter(item => item.kind === 'file' && IMAGE_MEDIA_TYPES.has(item.type))
|
||||
.map(item => item.getAsFile())
|
||||
.filter((file): file is File => file !== null)
|
||||
if (files.length === 0) return
|
||||
event.preventDefault()
|
||||
setDropError(null)
|
||||
onAddImages(files)
|
||||
}
|
||||
|
||||
const onDragEnter = (event: DragEvent<HTMLDivElement>): void => {
|
||||
if (!event.dataTransfer.types.includes('Files')) return
|
||||
event.preventDefault()
|
||||
if (locked) return
|
||||
dragDepthRef.current += 1
|
||||
setDropError(null)
|
||||
setDragActive(true)
|
||||
}
|
||||
|
||||
const onDragOver = (event: DragEvent<HTMLDivElement>): void => {
|
||||
if (!event.dataTransfer.types.includes('Files')) return
|
||||
event.preventDefault()
|
||||
event.dataTransfer.dropEffect = locked ? 'none' : 'copy'
|
||||
}
|
||||
|
||||
const onDragLeave = (event: DragEvent<HTMLDivElement>): void => {
|
||||
if (!event.dataTransfer.types.includes('Files') || locked) return
|
||||
dragDepthRef.current = Math.max(0, dragDepthRef.current - 1)
|
||||
if (dragDepthRef.current === 0) setDragActive(false)
|
||||
}
|
||||
|
||||
const onDrop = (event: DragEvent<HTMLDivElement>): void => {
|
||||
if (!event.dataTransfer.types.includes('Files')) return
|
||||
event.preventDefault()
|
||||
dragDepthRef.current = 0
|
||||
setDragActive(false)
|
||||
if (locked) return
|
||||
const dropped = [...event.dataTransfer.files]
|
||||
const images = supportedImages(dropped)
|
||||
if (images.length === 0) {
|
||||
setDropError('暂仅支持 PNG、JPEG、WebP 和 GIF 图片')
|
||||
return
|
||||
}
|
||||
setDropError(images.length === dropped.length ? null : '已忽略不受支持的非图片文件')
|
||||
onAddImages(images)
|
||||
}
|
||||
|
||||
const closePreview = useCallback(() => { setPreview(null) }, [])
|
||||
|
||||
// Button presses steal focus from the textarea; suppress at mousedown so typing continues seamlessly.
|
||||
const keepFocus = (e: MouseEvent<HTMLButtonElement>): void => {
|
||||
e.preventDefault()
|
||||
@@ -95,8 +161,38 @@ export function InputBar({
|
||||
{error.op === 'stop' ? '停止失败' : '发送失败'}：{error.message}
|
||||
</div>
|
||||
)}
|
||||
<div className={css.card}>
|
||||
{dropError !== null && <div className={css.error}>{dropError}</div>}
|
||||
<div
|
||||
className={clsx(css.card, dragActive && css.dragActive)}
|
||||
onDragEnter={onDragEnter}
|
||||
onDragOver={onDragOver}
|
||||
onDragLeave={onDragLeave}
|
||||
onDrop={onDrop}
|
||||
>
|
||||
{dragActive && <div className={css.dropHint} role="status">松开以添加图片</div>}
|
||||
{accessory !== undefined && <div className={css.accessory}>{accessory}</div>}
|
||||
{attachments.length > 0 && (
|
||||
<div className={css.attachments} aria-label="待发送图片">
|
||||
{attachments.map(attachment => (
|
||||
<div key={attachment.id} className={css.attachment}>
|
||||
<button
|
||||
type="button"
|
||||
className={css.thumbnail}
|
||||
title="双击查看原图"
|
||||
onDoubleClick={() => { setPreview(attachment) }}
|
||||
>
|
||||
<img src={attachment.previewUrl} alt={attachment.file.name || '待发送图片'} />
|
||||
</button>
|
||||
<button
|
||||
type="button"
|
||||
className={css.remove}
|
||||
aria-label={`移除图片 ${attachment.file.name || ''}`}
|
||||
onClick={() => { onRemoveAttachment(attachment.id) }}
|
||||
>×</button>
|
||||
</div>
|
||||
))}
|
||||
</div>
|
||||
)}
|
||||
{/* Mirror-div auto-grow: the hidden mirror renders draft+'\n' and stretches the wrapper
|
||||
(min/max capped in CSS); the absolutely-positioned textarea rides its height. Counting
|
||||
rows by '\n' cannot see soft wraps. */}
|
||||
@@ -110,6 +206,7 @@ export function InputBar({
|
||||
rows={2}
|
||||
onChange={(e) => onDraftChange(e.target.value)}
|
||||
onKeyDown={onKeyDown}
|
||||
onPaste={onPaste}
|
||||
onCompositionStart={onCompositionStart}
|
||||
onCompositionEnd={onCompositionEnd}
|
||||
/>
|
||||
@@ -137,6 +234,7 @@ export function InputBar({
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
{preview !== null && <ImageLightbox src={preview.previewUrl} alt={preview.file.name || '原图'} onClose={closePreview} />}
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
||||
@@ -19,8 +19,11 @@ import type { ChatStoreState, SelectionTarget, ViewId } from './contract/views.t
|
||||
type ChatActions = {
|
||||
select: (draft: ChatStoreState, target: SelectionTarget | null) => void
|
||||
setDraft: (draft: ChatStoreState, text: string) => void
|
||||
addImages: (draft: ChatStoreState, ids: readonly string[]) => void
|
||||
removeImage: (draft: ChatStoreState, id: string) => void
|
||||
pruneImages: (draft: ChatStoreState, available: readonly string[]) => void
|
||||
clearDraft: (draft: ChatStoreState) => void
|
||||
restoreDraft: (draft: ChatStoreState, text: string) => void
|
||||
restoreDraft: (draft: ChatStoreState, text: string, imageIds: readonly string[]) => void
|
||||
setView: (draft: ChatStoreState, view: ViewId) => void
|
||||
}
|
||||
|
||||
@@ -37,15 +40,30 @@ export function createChatStore(): EngineStoreHandle<ChatStoreState, ChatActions
|
||||
// Anchored to the contract shape: views consume the store through
|
||||
// ConvViewProps' SnapshotSelectorHook<ChatStoreState>, so init and the
|
||||
// contract cannot drift.
|
||||
init: (): ChatStoreState => ({ selection: null, draft: '', view: null }),
|
||||
init: (): ChatStoreState => ({ selection: null, draft: '', imageIds: [], view: null }),
|
||||
persist: 'dsh.conversation.chat',
|
||||
actions: {
|
||||
select: (d, target: SelectionTarget | null) => { d.selection = target },
|
||||
setDraft: (d, text: string) => { d.draft = text },
|
||||
clearDraft: (d) => { d.draft = '' },
|
||||
// Optimistic-send failure restore: only when the user typed nothing new
|
||||
// since the clear (send choreography lives in the inject factory).
|
||||
restoreDraft: (d, text: string) => { if (d.draft === '') d.draft = text },
|
||||
addImages: (d, ids: readonly string[]) => { d.imageIds.push(...ids) },
|
||||
removeImage: (d, id: string) => {
|
||||
d.imageIds = d.imageIds.filter(candidate => candidate !== id)
|
||||
},
|
||||
pruneImages: (d, available: readonly string[]) => {
|
||||
const keep = new Set(available)
|
||||
d.imageIds = d.imageIds.filter(id => keep.has(id))
|
||||
},
|
||||
clearDraft: (d) => {
|
||||
d.draft = ''
|
||||
d.imageIds = []
|
||||
},
|
||||
// Optimistic-send failure restore keeps any newer typing/images while
|
||||
// restoring the submitted draft material that disappeared on clear.
|
||||
restoreDraft: (d, text: string, imageIds: readonly string[]) => {
|
||||
if (d.draft === '') d.draft = text
|
||||
const current = new Set(d.imageIds)
|
||||
d.imageIds = [...imageIds.filter(id => !current.has(id)), ...d.imageIds]
|
||||
},
|
||||
setView: (d, view: ViewId) => { d.view = view },
|
||||
},
|
||||
})
|
||||
|
||||
@@ -132,25 +132,25 @@ describe('conversation slot inject surface', () => {
|
||||
const { instance, injected } = b.conversationSurface(ROOT)
|
||||
// Whitespace-only: no send, and the (whitespace) draft is not cleared.
|
||||
instance.actions.setDraft(' ')
|
||||
injected.send(' ', 'queue')
|
||||
injected.send(' ', [], 'queue')
|
||||
expect(b.sessionFake.prompt).not.toHaveBeenCalled()
|
||||
expect(instance.store.getSnapshot().draft).toBe(' ')
|
||||
// Success: cleared and stays cleared.
|
||||
instance.actions.setDraft('hello')
|
||||
injected.send('hello', 'queue')
|
||||
injected.send('hello', [], 'queue')
|
||||
expect(instance.store.getSnapshot().draft).toBe('')
|
||||
await Promise.resolve()
|
||||
expect(b.sessionFake.prompt).toHaveBeenCalledWith([{ type: 'text', text: 'hello' }], 'queue')
|
||||
// Failure: restored (draft still empty when the rejection lands).
|
||||
b.sessionFake.prompt.mockResolvedValueOnce({ ok: false, error: { code: 'agent-busy', message: 'b' } })
|
||||
instance.actions.setDraft('retry me')
|
||||
injected.send('retry me', 'queue')
|
||||
injected.send('retry me', [], 'queue')
|
||||
await vi.waitFor(() => {
|
||||
expect(instance.store.getSnapshot().draft).toBe('retry me')
|
||||
})
|
||||
// Failure landing after new typing: no clobber (restoreDraft fills empty only).
|
||||
b.sessionFake.prompt.mockResolvedValueOnce({ ok: false, error: { code: 'agent-busy', message: 'b' } })
|
||||
injected.send('retry me', 'queue')
|
||||
injected.send('retry me', [], 'queue')
|
||||
instance.actions.setDraft('typed during flight')
|
||||
await new Promise(r => setTimeout(r, 0))
|
||||
expect(instance.store.getSnapshot().draft).toBe('typed during flight')
|
||||
|
||||
@@ -15,9 +15,9 @@ beforeEach(() => {
|
||||
})
|
||||
|
||||
describe('createChatStore', () => {
|
||||
it('init shape: empty selection/draft/view', () => {
|
||||
it('init shape: empty selection/draft/images/view', () => {
|
||||
const store = createChatStore().create()
|
||||
expect(store.store.getSnapshot()).toEqual({ selection: null, draft: '', view: null })
|
||||
expect(store.store.getSnapshot()).toEqual({ selection: null, draft: '', imageIds: [], view: null })
|
||||
})
|
||||
|
||||
it('actions cover the declared write set', () => {
|
||||
@@ -30,6 +30,11 @@ describe('createChatStore', () => {
|
||||
|
||||
store.actions.setDraft('hello')
|
||||
expect(store.store.getSnapshot().draft).toBe('hello')
|
||||
store.actions.addImages(['a', 'b'])
|
||||
store.actions.removeImage('a')
|
||||
expect(store.store.getSnapshot().imageIds).toEqual(['b'])
|
||||
store.actions.pruneImages([])
|
||||
expect(store.store.getSnapshot().imageIds).toEqual([])
|
||||
store.actions.clearDraft()
|
||||
expect(store.store.getSnapshot().draft).toBe('')
|
||||
|
||||
@@ -40,12 +45,15 @@ describe('createChatStore', () => {
|
||||
it('restoreDraft only fills an empty draft (optimistic-send rollback contract)', () => {
|
||||
const store = createChatStore().create()
|
||||
// Rollback path: draft was cleared by send, nothing typed since.
|
||||
store.actions.restoreDraft('failed text')
|
||||
store.actions.restoreDraft('failed text', ['old-image'])
|
||||
expect(store.store.getSnapshot().draft).toBe('failed text')
|
||||
expect(store.store.getSnapshot().imageIds).toEqual(['old-image'])
|
||||
// The user typed something new before the failure landed: keep theirs.
|
||||
store.actions.setDraft('newer input')
|
||||
store.actions.restoreDraft('stale text')
|
||||
store.actions.addImages(['new-image'])
|
||||
store.actions.restoreDraft('stale text', ['old-image'])
|
||||
expect(store.store.getSnapshot().draft).toBe('newer input')
|
||||
expect(store.store.getSnapshot().imageIds).toEqual(['old-image', 'new-image'])
|
||||
})
|
||||
|
||||
it('persists per scope key and rehydrates a fresh instance', () => {
|
||||
|
||||
@@ -129,3 +129,89 @@ describe('error strip and variants', () => {
|
||||
expect(view.container.querySelector('[class*="hero"]')).not.toBeNull()
|
||||
})
|
||||
})
|
||||
|
||||
describe('image draft rail', () => {
|
||||
it('collects supported clipboard images and leaves non-image clipboard data to the browser', () => {
|
||||
const onAddImages = vi.fn()
|
||||
const { textarea } = setup({ draft: '', onAddImages })
|
||||
const image = new File([Uint8Array.of(1, 2, 3)], 'pixel.png', { type: 'image/png' })
|
||||
const prevented = fireEvent.paste(textarea, {
|
||||
clipboardData: {
|
||||
items: [
|
||||
{ kind: 'string', type: 'text/plain', getAsFile: () => null },
|
||||
{ kind: 'file', type: 'image/png', getAsFile: () => image },
|
||||
],
|
||||
},
|
||||
})
|
||||
expect(prevented).toBe(false)
|
||||
expect(onAddImages).toHaveBeenCalledWith([image])
|
||||
|
||||
fireEvent.paste(textarea, {
|
||||
clipboardData: { items: [{ kind: 'file', type: 'video/mp4', getAsFile: () => image }] },
|
||||
})
|
||||
expect(onAddImages).toHaveBeenCalledTimes(1)
|
||||
})
|
||||
|
||||
it('accepts supported image drops, highlights the target, and prevents browser navigation', () => {
|
||||
const onAddImages = vi.fn()
|
||||
const { view } = setup({ draft: '', onAddImages })
|
||||
const card = view.container.querySelector('[class*="card"]')!
|
||||
const image = new File([Uint8Array.of(1, 2, 3)], 'dropped.png', { type: 'image/png' })
|
||||
const dataTransfer = {
|
||||
types: ['Files'],
|
||||
files: [image],
|
||||
dropEffect: 'none',
|
||||
}
|
||||
expect(fireEvent.dragEnter(card, { dataTransfer })).toBe(false)
|
||||
expect(view.getByRole('status').textContent).toContain('松开以添加图片')
|
||||
expect(fireEvent.dragOver(card, { dataTransfer })).toBe(false)
|
||||
expect(dataTransfer.dropEffect).toBe('copy')
|
||||
expect(fireEvent.drop(card, { dataTransfer })).toBe(false)
|
||||
expect(view.queryByRole('status')).toBeNull()
|
||||
expect(onAddImages).toHaveBeenCalledWith([image])
|
||||
})
|
||||
|
||||
it('ignores unsupported dropped files and refuses drops while locked', () => {
|
||||
const onAddImages = vi.fn()
|
||||
const { view } = setup({ draft: '', onAddImages })
|
||||
const card = view.container.querySelector('[class*="card"]')!
|
||||
const documentFile = new File(['hello'], 'notes.txt', { type: 'text/plain' })
|
||||
fireEvent.drop(card, {
|
||||
dataTransfer: { types: ['Files'], files: [documentFile], dropEffect: 'none' },
|
||||
})
|
||||
expect(view.getByText(/暂仅支持 PNG/)).toBeTruthy()
|
||||
expect(onAddImages).not.toHaveBeenCalled()
|
||||
|
||||
const image = new File([Uint8Array.of(1)], 'locked.png', { type: 'image/png' })
|
||||
const locked = setup({ draft: '', disabled: true, onAddImages })
|
||||
const lockedCard = locked.view.container.querySelector('[class*="card"]')!
|
||||
const dataTransfer = { types: ['Files'], files: [image], dropEffect: 'copy' }
|
||||
fireEvent.dragEnter(lockedCard, { dataTransfer })
|
||||
expect(locked.view.queryByRole('status')).toBeNull()
|
||||
fireEvent.dragOver(lockedCard, { dataTransfer })
|
||||
expect(dataTransfer.dropEffect).toBe('none')
|
||||
fireEvent.drop(lockedCard, { dataTransfer })
|
||||
expect(onAddImages).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('allows image-only send, removes a thumbnail, and opens original preview on double-click', () => {
|
||||
const file = new File([Uint8Array.of(1)], 'pixel.png', { type: 'image/png' })
|
||||
const attachment = { id: 'draft-1', file, previewUrl: 'blob:draft-1' }
|
||||
const onRemoveAttachment = vi.fn()
|
||||
const { view, textarea, props } = setup({
|
||||
draft: '', attachments: [attachment], onRemoveAttachment,
|
||||
})
|
||||
const send = view.getByRole('button', { name: '发送' }) as HTMLButtonElement
|
||||
expect(send.disabled).toBe(false)
|
||||
fireEvent.keyDown(textarea, { key: 'Enter' })
|
||||
expect(props.onSend).toHaveBeenCalledWith('queue')
|
||||
|
||||
fireEvent.click(view.getByRole('button', { name: '移除图片 pixel.png' }))
|
||||
expect(onRemoveAttachment).toHaveBeenCalledWith('draft-1')
|
||||
fireEvent.doubleClick(view.getByTitle('双击查看原图'))
|
||||
expect(view.getByRole('dialog', { name: '原图预览' })).toBeTruthy()
|
||||
expect(view.getAllByAltText('pixel.png').every(node => (node as HTMLImageElement).src.includes('blob:draft-1'))).toBe(true)
|
||||
fireEvent.keyDown(window, { key: 'Escape' })
|
||||
expect(view.queryByRole('dialog', { name: '原图预览' })).toBeNull()
|
||||
})
|
||||
})
|
||||
|
||||
44
packages/client/ui-conversation/tests/message-image.spec.tsx
Normal file
44
packages/client/ui-conversation/tests/message-image.spec.tsx
Normal file
@@ -0,0 +1,44 @@
|
||||
// @vitest-environment jsdom
|
||||
|
||||
import { afterEach, describe, expect, it, vi } from 'vitest'
|
||||
import { cleanup, fireEvent, render, waitFor } from '@testing-library/react'
|
||||
import { AttachmentId } from '@deepseek-ai/dsh-attachment'
|
||||
import { MessageImage } from '../src/client/chat/MessageImage.tsx'
|
||||
|
||||
afterEach(cleanup)
|
||||
|
||||
const attachment = {
|
||||
attachmentId: AttachmentId(`sha256:${'a'.repeat(64)}`),
|
||||
mediaType: 'image/png' as const,
|
||||
bytes: 68,
|
||||
width: 640,
|
||||
height: 320,
|
||||
name: 'history.png',
|
||||
}
|
||||
|
||||
describe('MessageImage', () => {
|
||||
it('loads a session-authorized URL, bounds the thumbnail, and double-clicks into the original', async () => {
|
||||
const load = vi.fn().mockResolvedValue('blob:history')
|
||||
const view = render(<MessageImage attachment={attachment} load={load} />)
|
||||
const frame = view.getByRole('button', { name: 'history.png，双击查看原图' })
|
||||
expect(frame.getAttribute('style')).toContain('width: 240px')
|
||||
expect(frame.getAttribute('style')).toContain('height: 120px')
|
||||
await waitFor(() => { expect(view.getByAltText('history.png')).toBeTruthy() })
|
||||
expect(load).toHaveBeenCalledWith(attachment)
|
||||
fireEvent.doubleClick(frame)
|
||||
expect(view.getByRole('dialog', { name: '原图预览' })).toBeTruthy()
|
||||
fireEvent.click(view.getByRole('button', { name: '关闭原图预览' }))
|
||||
expect(view.queryByRole('dialog', { name: '原图预览' })).toBeNull()
|
||||
})
|
||||
|
||||
it('surfaces a retry control when durable bytes cannot be read', async () => {
|
||||
const load = vi.fn()
|
||||
.mockRejectedValueOnce(new Error('offline'))
|
||||
.mockResolvedValueOnce('blob:retry')
|
||||
const view = render(<MessageImage attachment={attachment} load={load} />)
|
||||
const retry = await view.findByRole('button', { name: '图片加载失败，点击重试' })
|
||||
fireEvent.click(retry)
|
||||
await waitFor(() => { expect(view.getByAltText('history.png')).toBeTruthy() })
|
||||
expect(load).toHaveBeenCalledTimes(2)
|
||||
})
|
||||
})
|
||||
@@ -175,6 +175,6 @@ describe('selection survives on the store seat', () => {
|
||||
await flush()
|
||||
const reborn = storeFor(b, 'conversation', sid('s1'))
|
||||
expect(reborn).not.toBe(doomed)
|
||||
expect(reborn.store.getSnapshot()).toEqual({ selection: null, draft: '', view: null })
|
||||
expect(reborn.store.getSnapshot()).toEqual({ selection: null, draft: '', imageIds: [], view: null })
|
||||
})
|
||||
})
|
||||
|
||||
@@ -93,6 +93,29 @@ describe('send / cancel', () => {
|
||||
await expect(s.send('x', 'queue')).rejects.toThrow(/send failed: agent-busy: busy/)
|
||||
})
|
||||
|
||||
it('uploads temporary browser files as base64 image parts at the send boundary', async () => {
|
||||
const b = await bench()
|
||||
const file = new File([Uint8Array.of(1, 2, 3)], 'pixel.png', { type: 'image/png' })
|
||||
Object.defineProperty(file, 'arrayBuffer', {
|
||||
value: () => Promise.resolve(Uint8Array.of(1, 2, 3).buffer),
|
||||
})
|
||||
await b.scopedSvc(sid('s1')).send('describe', 'queue', [file])
|
||||
expect(b.sessionDoubles.get(sid('s1'))!.prompt).toHaveBeenCalledWith([
|
||||
{ type: 'image', mediaType: 'image/png', data: 'AQID', name: 'pixel.png' },
|
||||
{ type: 'text', text: 'describe' },
|
||||
], 'queue')
|
||||
})
|
||||
|
||||
it('rejects unsupported browser media before prompting the session', async () => {
|
||||
const b = await bench()
|
||||
const file = new File([Uint8Array.of(1)], 'clip.mp4', { type: 'video/mp4' })
|
||||
Object.defineProperty(file, 'arrayBuffer', {
|
||||
value: () => Promise.resolve(Uint8Array.of(1).buffer),
|
||||
})
|
||||
await expect(b.scopedSvc(sid('s1')).send('', 'queue', [file])).rejects.toThrow(/不支持的图片格式/)
|
||||
expect(b.sessionDoubles.get(sid('s1'))?.prompt).not.toHaveBeenCalled()
|
||||
})
|
||||
|
||||
it('cancel resolves on ok and throws the folded business error', async () => {
|
||||
const b = await bench()
|
||||
const s = b.scopedSvc(sid('s1'))
|
||||
|
||||
@@ -71,6 +71,9 @@ describe('ConversationRoot branches', () => {
|
||||
useStore={hookOf(chat)}
|
||||
actions={chat.actions}
|
||||
views={{ list: () => [chatEntry], subscribe: () => () => {}, version: () => 1 }}
|
||||
addImages={vi.fn()}
|
||||
removeImage={vi.fn()}
|
||||
draftImages={() => []}
|
||||
send={vi.fn()}
|
||||
stop={vi.fn()}
|
||||
openDetails={vi.fn()}
|
||||
@@ -129,6 +132,9 @@ describe('ConversationRoot branches', () => {
|
||||
useStore={hookOf(chat)}
|
||||
actions={chat.actions}
|
||||
views={{ list: () => [chatEntry], subscribe: () => () => {}, version: () => 1 }}
|
||||
addImages={vi.fn()}
|
||||
removeImage={vi.fn()}
|
||||
draftImages={() => []}
|
||||
send={vi.fn()}
|
||||
stop={vi.fn()}
|
||||
openDetails={vi.fn()}
|
||||
|
||||
@@ -121,6 +121,9 @@ describe('ConversationRoot', () => {
|
||||
subscribe: () => () => {},
|
||||
version: () => 1,
|
||||
}}
|
||||
addImages={vi.fn()}
|
||||
removeImage={vi.fn()}
|
||||
draftImages={() => []}
|
||||
send={send}
|
||||
stop={stop}
|
||||
openDetails={openDetails}
|
||||
@@ -183,7 +186,7 @@ describe('ConversationRoot', () => {
|
||||
// Typing goes through actions.setDraft into the shared store.
|
||||
expect(chat.store.getSnapshot().draft).toBe('hi')
|
||||
fireEvent.keyDown(box, { key: 'Enter' })
|
||||
expect(send).toHaveBeenCalledWith('hi', 'queue')
|
||||
expect(send).toHaveBeenCalledWith('hi', [], 'queue')
|
||||
})
|
||||
})
|
||||
|
||||
|
||||
@@ -8,6 +8,9 @@
|
||||
"src"
|
||||
],
|
||||
"references": [
|
||||
{
|
||||
"path": "../../attachment/attachment"
|
||||
},
|
||||
{
|
||||
"path": "../../../vendor/cordis"
|
||||
},
|
||||
|
||||
@@ -96,6 +96,9 @@ function mount(svc: ConversationService, nodes: ConversationSnapshot['nodes'] =
|
||||
subscribe: (fn) => svc.subscribeViews(fn),
|
||||
version: () => svc.viewsVersion(),
|
||||
}}
|
||||
addImages={vi.fn()}
|
||||
removeImage={vi.fn()}
|
||||
draftImages={() => []}
|
||||
send={vi.fn()}
|
||||
stop={vi.fn()}
|
||||
openDetails={vi.fn()}
|
||||
|
||||
@@ -152,6 +152,20 @@ export const SERVICE_API: readonly ServiceApiEntry[] = [
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
key: 'attachments',
|
||||
summary: 'Immutable binary attachment service.',
|
||||
methods: [
|
||||
{
|
||||
signature: 'abstract saveImage(input: SaveImageAttachment): Promise<ImageAttachmentRef>',
|
||||
jsDoc: '/**\n * Validate and durably commit one image before its owning session event is appended.\n * @param input - encoded bytes, declared media type, and optional display name.\n * @returns a durable content-addressed reference.\n */',
|
||||
},
|
||||
{
|
||||
signature: 'abstract readImage(ref: ImageAttachmentRef): Promise<StoredImageAttachment>',
|
||||
jsDoc: '/**\n * Read one image and verify that bytes still match the recorded reference.\n * @param ref - durable reference from the session log.\n * @returns the verified bytes and canonical reference.\n */',
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
key: 'bash',
|
||||
summary: 'Abstract bash execution service.',
|
||||
@@ -1195,6 +1209,10 @@ export const TYPE_API: readonly TypeApiEntry[] = [
|
||||
name: 'AssistantProvenance',
|
||||
declaration: 'export interface AssistantProvenance {\n provider: string;\n model: string;\n replayState?: unknown;\n}',
|
||||
},
|
||||
{
|
||||
name: 'AttachmentId',
|
||||
declaration: 'export type AttachmentId = Branded<\'AttachmentId\'>;',
|
||||
},
|
||||
{
|
||||
name: 'BashEnvContributor',
|
||||
declaration: 'export interface BashEnvContributor {\n name: string;\n variables: Readonly<Record<DshEnvironmentKey, BashEnvVariable>>;\n resolve(execution: ToolExecution): Readonly<Partial<Record<DshEnvironmentKey, string>>>;\n}',
|
||||
@@ -1309,7 +1327,7 @@ export const TYPE_API: readonly TypeApiEntry[] = [
|
||||
},
|
||||
{
|
||||
name: 'ContentBlockMap',
|
||||
declaration: 'export interface ContentBlockMap {\n \'text\': TextBlock;\n \'reasoning\': ReasoningBlock;\n \'tool-call\': ToolCallBlock;\n \'tool-result\': ToolResultBlock;\n}',
|
||||
declaration: 'export interface ContentBlockMap {\n \'text\': TextBlock;\n \'reasoning\': ReasoningBlock;\n \'image\': ImageBlock;\n \'tool-call\': ToolCallBlock;\n \'tool-result\': ToolResultBlock;\n}',
|
||||
},
|
||||
{
|
||||
name: 'ContentBlockType',
|
||||
@@ -1451,6 +1469,18 @@ export const TYPE_API: readonly TypeApiEntry[] = [
|
||||
name: 'HookContext',
|
||||
declaration: 'export interface HookContext {\n content: ContentBlock[];\n source: MessageSource;\n placement?: \'separate\' | \'prompt-prefix\';\n meta?: JsonValue;\n}',
|
||||
},
|
||||
{
|
||||
name: 'ImageAttachmentRef',
|
||||
declaration: 'export interface ImageAttachmentRef {\n attachmentId: AttachmentId;\n mediaType: ImageMediaType;\n bytes: number;\n width: number;\n height: number;\n name?: string;\n}',
|
||||
},
|
||||
{
|
||||
name: 'ImageBlock',
|
||||
declaration: 'export interface ImageBlock {\n type: \'image\';\n attachment: ImageAttachmentRef;\n alt?: string;\n}',
|
||||
},
|
||||
{
|
||||
name: 'ImageMediaType',
|
||||
declaration: 'export type ImageMediaType = \'image/png\' | \'image/jpeg\' | \'image/webp\' | \'image/gif\';',
|
||||
},
|
||||
{
|
||||
name: 'InjectOptions',
|
||||
declaration: 'export interface InjectOptions extends Omit<SendOptions, \'contexts\'> {\n meta?: JsonValue;\n}',
|
||||
@@ -1481,7 +1511,7 @@ export const TYPE_API: readonly TypeApiEntry[] = [
|
||||
},
|
||||
{
|
||||
name: 'LlmModelInfo',
|
||||
declaration: 'export interface LlmModelInfo {\n provider: string;\n id: string;\n name: string;\n description?: string;\n}',
|
||||
declaration: 'export interface LlmModelInfo {\n provider: string;\n id: string;\n name: string;\n description?: string;\n inputModalities?: readonly ModelModality[];\n outputModalities?: readonly ModelModality[];\n}',
|
||||
},
|
||||
{
|
||||
name: 'LlmProviderInfo',
|
||||
@@ -1499,6 +1529,14 @@ export const TYPE_API: readonly TypeApiEntry[] = [
|
||||
name: 'MessageSourceMap',
|
||||
declaration: 'export interface MessageSourceMap {\n user: {\n kind: \'user\';\n };\n plugin: {\n kind: \'plugin\';\n plugin: string;\n };\n}',
|
||||
},
|
||||
{
|
||||
name: 'ModelModality',
|
||||
declaration: 'export type ModelModality = ModelModalityMap[keyof ModelModalityMap];',
|
||||
},
|
||||
{
|
||||
name: 'ModelModalityMap',
|
||||
declaration: 'export interface ModelModalityMap {\n text: \'text\';\n image: \'image\';\n}',
|
||||
},
|
||||
{
|
||||
name: 'OutOfBandSessionEventMap',
|
||||
declaration: 'export interface OutOfBandSessionEventMap {\n}',
|
||||
@@ -1651,6 +1689,10 @@ export const TYPE_API: readonly TypeApiEntry[] = [
|
||||
name: 'SandboxPolicyRequest',
|
||||
declaration: 'export interface SandboxPolicyRequest {\n session?: Session;\n mode?: SandboxMode;\n}',
|
||||
},
|
||||
{
|
||||
name: 'SaveImageAttachment',
|
||||
declaration: 'export interface SaveImageAttachment {\n data: Uint8Array;\n mediaType: ImageMediaType;\n name?: string;\n}',
|
||||
},
|
||||
{
|
||||
name: 'SaveTextSpill',
|
||||
declaration: 'export interface SaveTextSpill {\n owner: SpillOwner;\n source: SpillSource;\n suggestedName: string;\n content: string;\n}',
|
||||
@@ -1827,6 +1869,10 @@ export const TYPE_API: readonly TypeApiEntry[] = [
|
||||
name: 'SpillSource',
|
||||
declaration: 'export interface SpillSource {\n toolName: string;\n callId: CallId;\n label: string;\n}',
|
||||
},
|
||||
{
|
||||
name: 'StoredImageAttachment',
|
||||
declaration: 'export interface StoredImageAttachment {\n ref: ImageAttachmentRef;\n data: Uint8Array;\n}',
|
||||
},
|
||||
{
|
||||
name: 'StreamChunk',
|
||||
declaration: 'export type StreamChunk = {\n type: \'block-start\';\n index: number;\n blockType: ContentBlockType;\n} | {\n type: \'text-delta\';\n index: number;\n text: string;\n} | {\n type: \'reasoning-delta\';\n index: number;\n text: string;\n} | {\n type: \'tool-call-delta\';\n index: number;\n id: CallId;\n name?: string;\n argumentsDelta: string;\n} | {\n type: \'block-end\';\n index: number;\n block: ContentBlock;\n} | {\n type: \'usage\';\n usage: TokenUsage;\n} | {\n type: \'finish\';\n reason: FinishReason;\n replayState?: unknown;\n};',
|
||||
|
||||
@@ -40,6 +40,7 @@
|
||||
],
|
||||
"license": "BSD-3-Clause",
|
||||
"dependencies": {
|
||||
"@deepseek-ai/dsh-attachment": "workspace:^",
|
||||
"@deepseek-ai/dsh-brand": "workspace:^",
|
||||
"@deepseek-ai/dsh-llm": "workspace:^",
|
||||
"@deepseek-ai/dsh-session": "workspace:^",
|
||||
|
||||
@@ -5,6 +5,9 @@
|
||||
import { z } from 'zod'
|
||||
import type { RequestPayload, ResponseValue } from './rpc-map.ts'
|
||||
import type { Wire } from './rpc.schema.ts'
|
||||
import { imageMediaTypeSchema } from './sessions.schema.ts'
|
||||
|
||||
const modalitySchema = z.union([z.literal('text'), z.literal('image')])
|
||||
|
||||
/** host.describe request payload (empty object literal). */
|
||||
export const hostDescribeRequestSchema = z.object({}) satisfies z.ZodType<Wire<RequestPayload<'host.describe'>>>
|
||||
@@ -15,5 +18,20 @@ export const hostDescribeValueSchema = z.object({
|
||||
cwd: z.string(),
|
||||
provider: z.string().optional(),
|
||||
model: z.string().optional(),
|
||||
activeModel: z.object({
|
||||
provider: z.string(),
|
||||
id: z.string(),
|
||||
name: z.string(),
|
||||
description: z.string().optional(),
|
||||
inputModalities: z.array(modalitySchema).optional(),
|
||||
outputModalities: z.array(modalitySchema).optional(),
|
||||
}).optional(),
|
||||
imageLimits: z.object({
|
||||
maxImageBytes: z.number().int().positive(),
|
||||
maxImagesPerMessage: z.number().int().positive(),
|
||||
maxMessageImageBytes: z.number().int().positive(),
|
||||
maxImagePixels: z.number().int().positive(),
|
||||
mediaTypes: z.array(imageMediaTypeSchema),
|
||||
}).optional(),
|
||||
attachedSessions: z.number().int().nonnegative(),
|
||||
}) satisfies z.ZodType<Wire<ResponseValue<'host.describe'>>>
|
||||
|
||||
@@ -4,6 +4,8 @@
|
||||
*/
|
||||
|
||||
import type { RpcRequest, RpcResponse } from './rpc.ts'
|
||||
import type { ImageAttachmentLimits } from '@deepseek-ai/dsh-attachment'
|
||||
import type { LlmModelInfo } from '@deepseek-ai/dsh-llm/types'
|
||||
|
||||
/** Host-level unary methods. */
|
||||
export interface HostApi {
|
||||
@@ -20,6 +22,10 @@ export interface HostApi {
|
||||
cwd: string
|
||||
provider?: string
|
||||
model?: string
|
||||
/** Catalog entry for the active route; absent means its capabilities are unknown. */
|
||||
activeModel?: LlmModelInfo
|
||||
/** Resolved authoritative image-upload limits. */
|
||||
imageLimits?: ImageAttachmentLimits
|
||||
attachedSessions: number
|
||||
}>>
|
||||
}
|
||||
|
||||
@@ -19,7 +19,7 @@ export interface ApiProxy {
|
||||
}
|
||||
|
||||
// ---- Domain interfaces and payload entities ----
|
||||
export type { HistoryEntry, SessionsApi, SessionSummary } from './sessions.ts'
|
||||
export type { HistoryEntry, PromptContentPart, SessionsApi, SessionSummary } from './sessions.ts'
|
||||
export type { HostApi } from './host.ts'
|
||||
export type { EventsApi, MuxFrame, HostFrame, ToolCallView, ToolEventView, ToolResultView } from './events.ts'
|
||||
export type { ApprovalResponsePayload } from './approvals.ts'
|
||||
|
||||
@@ -14,6 +14,7 @@ export interface RpcMethodMap {
|
||||
'session.create': SessionsApi['create']
|
||||
'session.history': SessionsApi['history']
|
||||
'session.prompt': SessionsApi['prompt']
|
||||
'session.attachment': SessionsApi['attachment']
|
||||
'session.cancel': SessionsApi['cancel']
|
||||
'host.describe': HostApi['describe']
|
||||
}
|
||||
|
||||
@@ -35,6 +35,7 @@ export const rpcErrorSchema: z.ZodType<RpcError> = z.discriminatedUnion('code',
|
||||
z.object({ code: z.literal('bad-request'), message: z.string(), details: z.object({ issues: z.array(z.custom<ZodIssue>()) }) }),
|
||||
z.object({ code: z.literal('session-not-found'), message: z.string(), details: z.object({ sessionId: z.string() }) }),
|
||||
z.object({ code: z.literal('agent-busy'), message: z.string(), details: z.object({ reason: z.string() }) }),
|
||||
z.object({ code: z.literal('attachment-error'), message: z.string(), details: z.object({ reason: z.string() }) }),
|
||||
z.object({ code: z.literal('internal'), message: z.string(), details: z.object({}) }),
|
||||
]) as unknown as z.ZodType<RpcError>
|
||||
|
||||
|
||||
@@ -32,6 +32,7 @@ export interface RpcErrorDetailsMap {
|
||||
'bad-request': { issues: ZodIssue[] }
|
||||
'session-not-found': { sessionId: SessionId }
|
||||
'agent-busy': { reason: string }
|
||||
'attachment-error': { reason: string }
|
||||
'internal': {}
|
||||
}
|
||||
|
||||
|
||||
@@ -11,6 +11,7 @@ import type { RequestPayload, ResponseValue } from './rpc-map.ts'
|
||||
import type { Wire } from './rpc.schema.ts'
|
||||
import type { HistoryEntry, SessionSummary } from './sessions.ts'
|
||||
import type { ToolEventView } from './events.ts'
|
||||
import type { AttachmentIdType, ImageAttachmentRef } from '@deepseek-ai/dsh-attachment'
|
||||
|
||||
/** SessionId: one brand cast after shape validation (the only cast point in this domain). */
|
||||
export const sessionIdSchema = z.string().min(1) as unknown as z.ZodType<SessionId>
|
||||
@@ -84,14 +85,25 @@ export const sessionHistoryValueSchema = z.object({
|
||||
hasMore: z.boolean(),
|
||||
}) satisfies z.ZodType<Wire<ResponseValue<'session.history'>>>
|
||||
|
||||
/** ContentBlock passthrough: core is merge-extensible — the type discriminant envelope is strict, the rest stays wide. */
|
||||
export const contentBlockSchema = z.looseObject({ type: z.string() })
|
||||
/** Raster image media types accepted by the version-one browser wire. */
|
||||
export const imageMediaTypeSchema = z.union([
|
||||
z.literal('image/png'),
|
||||
z.literal('image/jpeg'),
|
||||
z.literal('image/webp'),
|
||||
z.literal('image/gif'),
|
||||
])
|
||||
|
||||
/** Prompt wire content is intentionally narrower than merge-extensible durable core content. */
|
||||
export const promptContentPartSchema = z.discriminatedUnion('type', [
|
||||
z.object({ type: z.literal('text'), text: z.string() }),
|
||||
z.object({ type: z.literal('image'), mediaType: imageMediaTypeSchema, data: z.string(), name: z.string().optional() }),
|
||||
])
|
||||
|
||||
/** session.prompt request payload. */
|
||||
export const sessionPromptRequestSchema = z.object({
|
||||
sessionId: sessionIdSchema,
|
||||
mode: z.union([z.literal('queue'), z.literal('steer')]),
|
||||
content: z.array(contentBlockSchema),
|
||||
content: z.array(promptContentPartSchema),
|
||||
}) as unknown as z.ZodType<RequestPayload<'session.prompt'>>
|
||||
|
||||
/** session.prompt response value. */
|
||||
@@ -99,6 +111,31 @@ export const sessionPromptValueSchema = z.object({
|
||||
accepted: z.literal(true),
|
||||
}) satisfies z.ZodType<Wire<ResponseValue<'session.prompt'>>>
|
||||
|
||||
/** Opaque attachment id after string-shape validation. */
|
||||
export const attachmentIdSchema = z.string().min(1) as unknown as z.ZodType<AttachmentIdType>
|
||||
|
||||
/** Durable image reference returned from the authenticated session lookup. */
|
||||
export const imageAttachmentRefSchema = z.object({
|
||||
attachmentId: attachmentIdSchema,
|
||||
mediaType: imageMediaTypeSchema,
|
||||
bytes: z.number().int().positive(),
|
||||
width: z.number().int().positive(),
|
||||
height: z.number().int().positive(),
|
||||
name: z.string().optional(),
|
||||
}) as unknown as z.ZodType<ImageAttachmentRef>
|
||||
|
||||
/** session.attachment request payload. */
|
||||
export const sessionAttachmentRequestSchema = z.object({
|
||||
sessionId: sessionIdSchema,
|
||||
attachmentId: attachmentIdSchema,
|
||||
}) satisfies z.ZodType<Wire<RequestPayload<'session.attachment'>>>
|
||||
|
||||
/** session.attachment response value. */
|
||||
export const sessionAttachmentValueSchema = z.object({
|
||||
attachment: imageAttachmentRefSchema,
|
||||
data: z.string(),
|
||||
}) satisfies z.ZodType<Wire<ResponseValue<'session.attachment'>>>
|
||||
|
||||
/** session.cancel request payload. */
|
||||
export const sessionCancelRequestSchema = z.object({
|
||||
sessionId: sessionIdSchema,
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
* else references RequestPayload<'session.*'> / ResponseValue<'session.*'>.
|
||||
*/
|
||||
|
||||
import type { ContentBlock } from '@deepseek-ai/dsh-llm/types'
|
||||
import type { AttachmentIdType, ImageAttachmentRef, ImageMediaType } from '@deepseek-ai/dsh-attachment'
|
||||
import type { SessionEvent, SessionId } from '@deepseek-ai/dsh-session/types'
|
||||
import type { RpcId, RpcRequest, RpcResponse } from './rpc.ts'
|
||||
import type { ToolEventView } from './events.ts'
|
||||
@@ -44,6 +44,11 @@ export interface SessionSummary {
|
||||
cwd?: string
|
||||
}
|
||||
|
||||
/** Browser-submitted prompt content; image bytes are promoted to durable references by the host. */
|
||||
export type PromptContentPart =
|
||||
| { type: 'text'; text: string }
|
||||
| { type: 'image'; mediaType: ImageMediaType; data: string; name?: string }
|
||||
|
||||
/** Session-domain unary methods (the map keys session.* of RpcMethodMap). */
|
||||
export interface SessionsApi {
|
||||
/** Lists persisted sessions (updatedAt descending). v1 returns everything; cursor is a reserved seat, unimplemented. */
|
||||
@@ -64,10 +69,14 @@ export interface SessionsApi {
|
||||
history(request: RpcRequest<{ sessionId: SessionId; beforeSeq?: number; maxMessages?: number }>):
|
||||
Promise<RpcResponse<{ events: HistoryEntry[]; hasMore: boolean }>>
|
||||
|
||||
/** Sends a message. content is core's ContentBlock[] verbatim; mode maps 1:1 — queue→send, steer→steer. */
|
||||
prompt(request: RpcRequest<{ sessionId: SessionId; mode: 'queue' | 'steer'; content: ContentBlock[] }>):
|
||||
/** Sends text plus temporary base64 image uploads; the host persists images before calling the agent. */
|
||||
prompt(request: RpcRequest<{ sessionId: SessionId; mode: 'queue' | 'steer'; content: PromptContentPart[] }>):
|
||||
Promise<RpcResponse<{ accepted: true }>>
|
||||
|
||||
/** Reads one durable image after proving that this session's log references its id. */
|
||||
attachment(request: RpcRequest<{ sessionId: SessionId; attachmentId: AttachmentIdType }> ):
|
||||
Promise<RpcResponse<{ attachment: ImageAttachmentRef; data: string }>>
|
||||
|
||||
/** Stops: clears both FIFOs + aborts the current step (1:1 with agent.cancel). */
|
||||
cancel(request: RpcRequest<{ sessionId: SessionId }>): Promise<RpcResponse<{ accepted: true }>>
|
||||
}
|
||||
|
||||
@@ -16,6 +16,7 @@ import { hostFrameSchema, muxFrameSchema } from '../api/events.schema.ts'
|
||||
import { hostDescribeValueSchema } from '../api/host.schema.ts'
|
||||
import {
|
||||
sessionCancelValueSchema,
|
||||
sessionAttachmentValueSchema,
|
||||
sessionCreateValueSchema,
|
||||
sessionHistoryValueSchema,
|
||||
sessionListValueSchema,
|
||||
@@ -43,6 +44,7 @@ export interface IApiClient {
|
||||
create(payload: RequestPayload<'session.create'>, signal?: AbortSignal): Promise<RpcResponse<ResponseValue<'session.create'>>>
|
||||
history(payload: RequestPayload<'session.history'>, signal?: AbortSignal): Promise<RpcResponse<ResponseValue<'session.history'>>>
|
||||
prompt(payload: RequestPayload<'session.prompt'>, signal?: AbortSignal): Promise<RpcResponse<ResponseValue<'session.prompt'>>>
|
||||
attachment(payload: RequestPayload<'session.attachment'>, signal?: AbortSignal): Promise<RpcResponse<ResponseValue<'session.attachment'>>>
|
||||
cancel(payload: RequestPayload<'session.cancel'>, signal?: AbortSignal): Promise<RpcResponse<ResponseValue<'session.cancel'>>>
|
||||
}
|
||||
host: {
|
||||
@@ -65,6 +67,7 @@ const UNARY_VALUE_SCHEMAS: { [K in keyof RpcMethodMap]: z.ZodType<Wire<ResponseV
|
||||
'session.create': sessionCreateValueSchema,
|
||||
'session.history': sessionHistoryValueSchema,
|
||||
'session.prompt': sessionPromptValueSchema,
|
||||
'session.attachment': sessionAttachmentValueSchema,
|
||||
'session.cancel': sessionCancelValueSchema,
|
||||
'host.describe': hostDescribeValueSchema,
|
||||
}
|
||||
@@ -246,6 +249,7 @@ export abstract class AbstractApiClient implements IApiClient {
|
||||
create: (payload, signal) => this.callUnary('session.create', payload, signal),
|
||||
history: (payload, signal) => this.callUnary('session.history', payload, signal),
|
||||
prompt: (payload, signal) => this.callUnary('session.prompt', payload, signal),
|
||||
attachment: (payload, signal) => this.callUnary('session.attachment', payload, signal),
|
||||
cancel: (payload, signal) => this.callUnary('session.cancel', payload, signal),
|
||||
}
|
||||
|
||||
|
||||
@@ -16,6 +16,7 @@ import type { Wire } from '../api/rpc.schema.ts'
|
||||
import { clientRequestSchema, clientResponseSchema } from '../api/rpc.schema.ts'
|
||||
import {
|
||||
sessionCancelRequestSchema,
|
||||
sessionAttachmentRequestSchema,
|
||||
sessionCreateRequestSchema,
|
||||
sessionHistoryRequestSchema,
|
||||
sessionListRequestSchema,
|
||||
@@ -42,6 +43,7 @@ const UNARY_ROUTES: UnaryRoutes = {
|
||||
'session.create': { schema: sessionCreateRequestSchema, invoke: (api, r) => api.sessions.create(r) },
|
||||
'session.history': { schema: sessionHistoryRequestSchema, invoke: (api, r) => api.sessions.history(r) },
|
||||
'session.prompt': { schema: sessionPromptRequestSchema, invoke: (api, r) => api.sessions.prompt(r) },
|
||||
'session.attachment': { schema: sessionAttachmentRequestSchema, invoke: (api, r) => api.sessions.attachment(r) },
|
||||
'session.cancel': { schema: sessionCancelRequestSchema, invoke: (api, r) => api.sessions.cancel(r) },
|
||||
'host.describe': { schema: hostDescribeRequestSchema, invoke: (api, r) => api.host.describe(r) },
|
||||
}
|
||||
|
||||
@@ -30,6 +30,10 @@ function scriptedApi(overrides: {
|
||||
create: r => ok(r, { sessionId: sid('s-new') }),
|
||||
history: r => ok(r, { events: [], hasMore: false }),
|
||||
prompt: r => ok(r, { accepted: true as const }),
|
||||
attachment: r => ok(r, {
|
||||
attachment: { attachmentId: 'a' as never, mediaType: 'image/png', bytes: 1, width: 1, height: 1 },
|
||||
data: 'AA==',
|
||||
}),
|
||||
cancel: r => ok(r, { accepted: true as const }),
|
||||
...overrides.sessions,
|
||||
},
|
||||
|
||||
@@ -33,6 +33,12 @@ function fakeApi(overrides: Partial<{ muxFrames: MuxFrame[]; hostFrames: HostFra
|
||||
async prompt(request) {
|
||||
return { rpcId: request.rpcId, result: { ok: true, value: { accepted: true as const } } }
|
||||
},
|
||||
async attachment(request) {
|
||||
return {
|
||||
rpcId: request.rpcId,
|
||||
result: { ok: true, value: { attachment: { attachmentId: 'a' as never, mediaType: 'image/png' as const, bytes: 1, width: 1, height: 1 }, data: 'AA==' } },
|
||||
}
|
||||
},
|
||||
async cancel(request) {
|
||||
return { rpcId: request.rpcId, result: { ok: true, value: { accepted: true as const } } }
|
||||
},
|
||||
|
||||
@@ -6,7 +6,8 @@ import {
|
||||
} from '../src/api/rpc.schema.ts'
|
||||
import { z } from 'zod'
|
||||
import {
|
||||
contentBlockSchema, sessionCancelRequestSchema, sessionCancelValueSchema, sessionCreateRequestSchema,
|
||||
promptContentPartSchema, sessionAttachmentRequestSchema, sessionAttachmentValueSchema,
|
||||
sessionCancelRequestSchema, sessionCancelValueSchema, sessionCreateRequestSchema,
|
||||
sessionCreateValueSchema, sessionEventSchema, sessionHistoryRequestSchema, sessionHistoryValueSchema,
|
||||
sessionIdSchema, sessionListRequestSchema, sessionListValueSchema, sessionPromptRequestSchema,
|
||||
sessionPromptValueSchema, sessionSummarySchema,
|
||||
@@ -31,6 +32,7 @@ describe('rpcErrorSchema', () => {
|
||||
expect(rpcErrorSchema.parse({ code: 'bad-request', message: 'm', details: { issues: [] } }).code).toBe('bad-request')
|
||||
expect(rpcErrorSchema.parse({ code: 'session-not-found', message: 'm', details: { sessionId: 's' } }).code).toBe('session-not-found')
|
||||
expect(rpcErrorSchema.parse({ code: 'agent-busy', message: 'm', details: { reason: 'r' } }).code).toBe('agent-busy')
|
||||
expect(rpcErrorSchema.parse({ code: 'attachment-error', message: 'm', details: { reason: 'r' } }).code).toBe('attachment-error')
|
||||
expect(rpcErrorSchema.parse({ code: 'internal', message: 'm', details: {} }).code).toBe('internal')
|
||||
})
|
||||
|
||||
@@ -105,7 +107,20 @@ describe('sessions domain schemas', () => {
|
||||
expect(sessionPromptValueSchema.parse({ accepted: true }).accepted).toBe(true)
|
||||
expect(sessionCancelRequestSchema.parse({ sessionId: 's1' }).sessionId).toBe('s1')
|
||||
expect(sessionCancelValueSchema.parse({ accepted: true }).accepted).toBe(true)
|
||||
expect(contentBlockSchema.parse({ type: 'text', text: 'x', extra: 1 })).toMatchObject({ extra: 1 })
|
||||
expect(promptContentPartSchema.parse({ type: 'text', text: 'x', extra: 1 })).toEqual({ type: 'text', text: 'x' })
|
||||
expect(promptContentPartSchema.parse({
|
||||
type: 'image', mediaType: 'image/png', data: 'AA==', name: 'pixel.png',
|
||||
})).toMatchObject({ type: 'image', mediaType: 'image/png', name: 'pixel.png' })
|
||||
const attachment = {
|
||||
attachmentId: `sha256:${'a'.repeat(64)}`,
|
||||
mediaType: 'image/png' as const,
|
||||
bytes: 1,
|
||||
width: 1,
|
||||
height: 1,
|
||||
}
|
||||
expect(sessionAttachmentRequestSchema.parse({ sessionId: 's1', attachmentId: attachment.attachmentId }))
|
||||
.toMatchObject({ sessionId: 's1' })
|
||||
expect(sessionAttachmentValueSchema.parse({ attachment, data: 'AA==' }).attachment).toEqual(attachment)
|
||||
})
|
||||
})
|
||||
|
||||
|
||||
@@ -11,6 +11,9 @@
|
||||
{
|
||||
"path": "../../util/brand"
|
||||
},
|
||||
{
|
||||
"path": "../../attachment/attachment"
|
||||
},
|
||||
{
|
||||
"path": "../../llm/llm"
|
||||
},
|
||||
|
||||
@@ -31,6 +31,7 @@
|
||||
"@cordisjs/plugin-timer": "workspace:^",
|
||||
"@deepseek-ai/dsh-agent": "workspace:^",
|
||||
"@deepseek-ai/dsh-agent-loop": "workspace:^",
|
||||
"@deepseek-ai/dsh-attachment-local": "workspace:^",
|
||||
"@deepseek-ai/dsh-bash-local": "workspace:^",
|
||||
"@deepseek-ai/dsh-client-connection": "workspace:^",
|
||||
"@deepseek-ai/dsh-client-i18n": "workspace:^",
|
||||
@@ -46,6 +47,7 @@
|
||||
"@deepseek-ai/dsh-host-apiproxy": "workspace:^",
|
||||
"@deepseek-ai/dsh-llm": "workspace:^",
|
||||
"@deepseek-ai/dsh-llm-deepseek": "workspace:^",
|
||||
"@deepseek-ai/dsh-llm-pi-ai": "workspace:^",
|
||||
"@deepseek-ai/dsh-session": "workspace:^",
|
||||
"@deepseek-ai/dsh-session-persistence": "workspace:^",
|
||||
"@deepseek-ai/dsh-session-persistence-jsonl": "workspace:^",
|
||||
|
||||
@@ -9,10 +9,12 @@ import { randomUUID } from 'node:crypto'
|
||||
import { stat } from 'node:fs/promises'
|
||||
import type { Context } from 'cordis'
|
||||
import type { Agent, AgentStatus } from '@deepseek-ai/dsh-agent'
|
||||
import { AttachmentError } from '@deepseek-ai/dsh-attachment-local'
|
||||
import type { ImageAttachmentRef } from '@deepseek-ai/dsh-attachment-local'
|
||||
import type { ContentBlock, MessageSource } from '@deepseek-ai/dsh-llm'
|
||||
import type { Session, SessionEvent, SessionHeader, SessionId } from '@deepseek-ai/dsh-session'
|
||||
import type { SessionPersistence } from '@deepseek-ai/dsh-session-persistence'
|
||||
import type { ApiProxy, HistoryEntry, HostFrame, MuxFrame, SessionSummary, ToolEventView } from '@deepseek-ai/dsh-host-apiproxy/api'
|
||||
import type { ApiProxy, HistoryEntry, HostFrame, MuxFrame, PromptContentPart, SessionSummary, ToolEventView } from '@deepseek-ai/dsh-host-apiproxy/api'
|
||||
import type { ClientResponse, RpcError, RpcReceipt, RpcRequest, RpcResponse } from '@deepseek-ai/dsh-host-apiproxy/api/rpc'
|
||||
import { RpcId } from '@deepseek-ai/dsh-host-apiproxy/api/rpc'
|
||||
|
||||
@@ -22,6 +24,69 @@ const DEFAULT_MAX_MESSAGES = 50
|
||||
/** Surface message event types (the pagination counting unit). */
|
||||
const MESSAGE_TYPES = new Set(['user/message', 'assistant/message', 'steering/message'])
|
||||
|
||||
function decodeBase64(data: string): Uint8Array {
|
||||
if (data.length === 0 || data.length % 4 !== 0 || !/^(?:[A-Za-z0-9+/]{4})*(?:[A-Za-z0-9+/]{2}==|[A-Za-z0-9+/]{3}=)?$/.test(data)) {
|
||||
throw new AttachmentError('Image upload is not canonical base64.', 'INVALID_IMAGE_BASE64')
|
||||
}
|
||||
const decoded = Buffer.from(data, 'base64')
|
||||
if (decoded.toString('base64') !== data) throw new AttachmentError('Image upload is not canonical base64.', 'INVALID_IMAGE_BASE64')
|
||||
return new Uint8Array(decoded)
|
||||
}
|
||||
|
||||
async function durablePromptContent(ctx: Context, content: readonly PromptContentPart[]): Promise<ContentBlock[]> {
|
||||
const limits = ctx.attachments.imageLimits
|
||||
const prepared = content.map(part => part.type === 'text'
|
||||
? part
|
||||
: { part, data: decodeBase64(part.data) })
|
||||
const images = prepared.filter((part): part is Extract<typeof part, { data: Uint8Array }> => 'data' in part)
|
||||
if (images.length > limits.maxImagesPerMessage) {
|
||||
throw new AttachmentError('Prompt exceeds the configured image-count limit.', 'TOO_MANY_IMAGES')
|
||||
}
|
||||
const totalBytes = images.reduce((sum, image) => sum + image.data.byteLength, 0)
|
||||
if (totalBytes > limits.maxMessageImageBytes) {
|
||||
throw new AttachmentError('Prompt exceeds the configured aggregate image-byte limit.', 'IMAGES_TOO_LARGE')
|
||||
}
|
||||
return Promise.all(prepared.map(async (item): Promise<ContentBlock> => {
|
||||
if (!('data' in item)) return { type: 'text', text: item.text }
|
||||
const attachment = await ctx.attachments.saveImage({
|
||||
data: item.data,
|
||||
mediaType: item.part.mediaType,
|
||||
...item.part.name === undefined ? {} : { name: item.part.name },
|
||||
})
|
||||
return { type: 'image', attachment }
|
||||
}))
|
||||
}
|
||||
|
||||
function imageInContent(content: unknown, attachmentId: string): ImageAttachmentRef | undefined {
|
||||
if (!Array.isArray(content)) return undefined
|
||||
for (const value of content) {
|
||||
if (typeof value !== 'object' || value === null || Array.isArray(value)) continue
|
||||
const block = value as { type?: unknown; attachment?: unknown; content?: unknown }
|
||||
if (block.type === 'image' && typeof block.attachment === 'object' && block.attachment !== null) {
|
||||
const ref = block.attachment as ImageAttachmentRef
|
||||
if (String(ref.attachmentId) === attachmentId) return ref
|
||||
}
|
||||
if (block.type === 'tool-result') {
|
||||
const nested = imageInContent(block.content, attachmentId)
|
||||
if (nested !== undefined) return nested
|
||||
}
|
||||
}
|
||||
return undefined
|
||||
}
|
||||
|
||||
function referencedImage(events: readonly SessionEvent[], attachmentId: string): ImageAttachmentRef | undefined {
|
||||
for (const event of events) {
|
||||
const data = event.data as { content?: unknown; chunk?: { type?: unknown; block?: unknown } }
|
||||
const direct = imageInContent(data.content, attachmentId)
|
||||
if (direct !== undefined) return direct
|
||||
if (event.type === 'assistant/chunk' && data.chunk?.type === 'block-end') {
|
||||
const streamed = imageInContent([data.chunk.block], attachmentId)
|
||||
if (streamed !== undefined) return streamed
|
||||
}
|
||||
}
|
||||
return undefined
|
||||
}
|
||||
|
||||
/**
|
||||
* Message-boundary pagination: count maxMessages surface messages backwards from
|
||||
* the window tail; the cut is the starting seq of the oldest message group
|
||||
@@ -321,15 +386,52 @@ export function createApiProxy(ctx: Context, defaults: ApiProxyDefaults): ApiPro
|
||||
// The rpcId rides MessageSource into user/message (merge declaration in api/sessions.ts; provisional correlation).
|
||||
const source: MessageSource = { kind: 'user', rpcId: request.rpcId }
|
||||
try {
|
||||
if (mode === 'steer') agent.steer(content, { source })
|
||||
else agent.send(content, { source })
|
||||
if (content.some(part => part.type === 'image')) {
|
||||
const activeModel = (await ctx.llm.listModels(defaults.provider)).find(model => model.id === defaults.model)
|
||||
if (activeModel?.inputModalities !== undefined && !activeModel.inputModalities.includes('image')) {
|
||||
return err(request, {
|
||||
code: 'attachment-error',
|
||||
message: `Model "${defaults.model}" does not support image input.`,
|
||||
details: { reason: 'MODEL_DOES_NOT_SUPPORT_IMAGES' },
|
||||
})
|
||||
}
|
||||
}
|
||||
const durable = await durablePromptContent(ctx, content)
|
||||
if (mode === 'steer') agent.steer(durable, { source })
|
||||
else agent.send(durable, { source })
|
||||
} catch (error: unknown) {
|
||||
if (error instanceof AttachmentError) {
|
||||
return err(request, { code: 'attachment-error', message: error.message, details: { reason: error.code } })
|
||||
}
|
||||
// A synchronous throw from send/steer means disposed or invalid input; surface as agent-busy with the reason attached.
|
||||
return err(request, { code: 'agent-busy', message: 'prompt rejected', details: { reason: String(error) } })
|
||||
}
|
||||
return ok(request, { accepted: true as const })
|
||||
},
|
||||
|
||||
async attachment(request) {
|
||||
const { sessionId, attachmentId } = request.payload
|
||||
const found = await agentFor(sessionId)
|
||||
if ('error' in found) return err(request, found.error)
|
||||
const ref = referencedImage(found.agent.session.events, String(attachmentId))
|
||||
if (ref === undefined) {
|
||||
return err(request, {
|
||||
code: 'attachment-error',
|
||||
message: 'Image is not referenced by this session.',
|
||||
details: { reason: 'ATTACHMENT_NOT_REFERENCED' },
|
||||
})
|
||||
}
|
||||
try {
|
||||
const stored = await ctx.attachments.readImage(ref)
|
||||
return ok(request, { attachment: stored.ref, data: Buffer.from(stored.data).toString('base64') })
|
||||
} catch (error: unknown) {
|
||||
if (error instanceof AttachmentError) {
|
||||
return err(request, { code: 'attachment-error', message: error.message, details: { reason: error.code } })
|
||||
}
|
||||
return err(request, { code: 'internal', message: 'Unable to read image attachment.', details: {} })
|
||||
}
|
||||
},
|
||||
|
||||
cancel(request) {
|
||||
const { sessionId } = request.payload
|
||||
const agent = ctx.agents.get(sessionId)
|
||||
@@ -346,15 +448,21 @@ export function createApiProxy(ctx: Context, defaults: ApiProxyDefaults): ApiPro
|
||||
},
|
||||
|
||||
host: {
|
||||
describe(request) {
|
||||
async describe(request) {
|
||||
const activeModel = (await ctx.llm.listModels(defaults.provider)).find(model => model.id === defaults.model)
|
||||
// TODO(step2): version should read apps/cli's package.json; placeholder for now.
|
||||
return Promise.resolve(ok(request, {
|
||||
return ok(request, {
|
||||
version: '0.0.1',
|
||||
cwd: process.cwd(),
|
||||
provider: defaults.provider,
|
||||
model: defaults.model,
|
||||
...activeModel === undefined ? {} : { activeModel },
|
||||
imageLimits: {
|
||||
...ctx.attachments.imageLimits,
|
||||
mediaTypes: [...ctx.attachments.imageLimits.mediaTypes],
|
||||
},
|
||||
attachedSessions: ctx.agents.list().length,
|
||||
}))
|
||||
})
|
||||
},
|
||||
},
|
||||
|
||||
|
||||
@@ -7,6 +7,7 @@
|
||||
import { Context } from 'cordis'
|
||||
import Timer from '@cordisjs/plugin-timer'
|
||||
import LlmService from '@deepseek-ai/dsh-llm'
|
||||
import LocalAttachmentStore from '@deepseek-ai/dsh-attachment-local'
|
||||
import SessionStore from '@deepseek-ai/dsh-session'
|
||||
import SystemPrompt from '@deepseek-ai/dsh-system-prompt'
|
||||
import ToolRegistry from '@deepseek-ai/dsh-tools'
|
||||
@@ -14,6 +15,8 @@ import AgentRegistry from '@deepseek-ai/dsh-agent'
|
||||
import TaskService from '@deepseek-ai/dsh-tasks'
|
||||
import AgentLoop from '@deepseek-ai/dsh-agent-loop'
|
||||
import * as LlmDeepSeek from '@deepseek-ai/dsh-llm-deepseek'
|
||||
import * as LlmPiAi from '@deepseek-ai/dsh-llm-pi-ai'
|
||||
import type { PiAiProviderProfile } from '@deepseek-ai/dsh-llm-pi-ai'
|
||||
import SessionPersistenceJsonl from '@deepseek-ai/dsh-session-persistence-jsonl'
|
||||
import LocalBashExecutor from '@deepseek-ai/dsh-bash-local'
|
||||
import * as toolBash from '@deepseek-ai/dsh-tool-bash'
|
||||
@@ -42,10 +45,14 @@ import * as spillPolicy from '@deepseek-ai/dsh-spill-policy'
|
||||
export interface BootHostOptions {
|
||||
/** Root directory for JSONL session persistence. */
|
||||
persistenceRoot: string
|
||||
/** Explicit harness home for durable attachments; omitted follows DSH_HOME then ~/.dsh. */
|
||||
dshHome?: string
|
||||
/** Default provider route for created/resumed agents (defaults to 'deepseek', the only adapter bootHost registers). */
|
||||
provider?: string
|
||||
/** Default model id (defaults to 'deepseek-v4-flash', matching the demos). */
|
||||
model?: string
|
||||
/** Additional pi-ai provider routes available to visual-capable Web sessions. */
|
||||
piAiProviders?: PiAiProviderProfile[]
|
||||
/**
|
||||
* Default project directory for sessions created without an explicit cwd
|
||||
* (defaults to the host process working directory). A session's cwd is its
|
||||
@@ -88,6 +95,9 @@ export async function bootHost(options: BootHostOptions): Promise<HostHandle> {
|
||||
const ctx = new Context()
|
||||
await ctx.plugin(Timer)
|
||||
await ctx.plugin(LlmService)
|
||||
await ctx.plugin(LocalAttachmentStore, {
|
||||
...options.dshHome === undefined ? {} : { dshHome: options.dshHome },
|
||||
})
|
||||
await ctx.plugin(SessionStore)
|
||||
await ctx.plugin(SystemPrompt, { persona: '' })
|
||||
await ctx.plugin(ToolRegistry)
|
||||
@@ -95,6 +105,9 @@ export async function bootHost(options: BootHostOptions): Promise<HostHandle> {
|
||||
await ctx.plugin(TaskService)
|
||||
await ctx.plugin(AgentLoop, { agents: [] })
|
||||
await ctx.plugin(LlmDeepSeek, {})
|
||||
if (options.piAiProviders !== undefined && options.piAiProviders.length > 0) {
|
||||
await ctx.plugin(LlmPiAi, { providers: options.piAiProviders })
|
||||
}
|
||||
await ctx.plugin(SessionPersistenceJsonl, { root: options.persistenceRoot, compression: 'none' })
|
||||
await ctx.plugin(LocalBashExecutor, {})
|
||||
// Tool suite mirroring the demo:repl composition (repl-agent/cordis.yml +
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
import { mkdtempSync } from 'node:fs'
|
||||
import { existsSync, mkdtempSync, readFileSync } from 'node:fs'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join } from 'node:path'
|
||||
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
|
||||
import type { Context } from 'cordis'
|
||||
import type { Agent } from '@deepseek-ai/dsh-agent'
|
||||
import { agentEvents } from '@deepseek-ai/dsh-agent'
|
||||
import type { GenerateOptions, StreamChunk } from '@deepseek-ai/dsh-llm'
|
||||
import type { ContentBlock, GenerateOptions, LlmModelInfo, ModelModality, StreamChunk } from '@deepseek-ai/dsh-llm'
|
||||
import { LlmAdapter } from '@deepseek-ai/dsh-llm'
|
||||
import type { SessionId } from '@deepseek-ai/dsh-session'
|
||||
import type { HostFrame, MuxFrame } from '@deepseek-ai/dsh-host-apiproxy/api'
|
||||
@@ -15,10 +15,20 @@ import { bootHost, startHost, type HostHandle, type RunningHost } from '../src/i
|
||||
|
||||
/** Scripted adapter: each model call consumes the next chunk list; 'hang' streams then waits for abort. */
|
||||
class ScriptedAdapter extends LlmAdapter {
|
||||
constructor(private script: (StreamChunk[] | 'hang')[]) {
|
||||
constructor(
|
||||
private script: (StreamChunk[] | 'hang')[],
|
||||
private readonly inputModalities: readonly ModelModality[] = ['text', 'image'],
|
||||
) {
|
||||
super()
|
||||
}
|
||||
|
||||
override listModels(provider: string): Promise<readonly LlmModelInfo[]> {
|
||||
return Promise.resolve([{
|
||||
provider, id: 'test-model', name: 'test-model',
|
||||
inputModalities: this.inputModalities, outputModalities: ['text'],
|
||||
}])
|
||||
}
|
||||
|
||||
async * stream(options: GenerateOptions): AsyncIterable<StreamChunk> {
|
||||
const entry = this.script.shift()
|
||||
if (!entry) throw new Error('ScriptedAdapter: script exhausted')
|
||||
@@ -48,6 +58,8 @@ function request<P>(payload: P): RpcRequest<P> {
|
||||
}
|
||||
let nextRpc = 1
|
||||
|
||||
const PNG_BASE64 = 'iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk+A8AAQUBAScY42YAAAAASUVORK5CYII='
|
||||
|
||||
function waitForIdle(ctx: Context, agent: Agent): Promise<void> {
|
||||
return new Promise((resolve) => {
|
||||
const dispose = ctx.on('agent/status', (subject: Agent, status: string) => {
|
||||
@@ -171,14 +183,83 @@ describe('sessions.prompt / cancel', () => {
|
||||
})
|
||||
|
||||
it('maps a synchronous send throw to agent-busy', async () => {
|
||||
const { api } = await boot()
|
||||
const { api, ctx } = await boot()
|
||||
const { sessionId } = expectOk(await api.sessions.create(request({})))
|
||||
const poisoned = [{ type: 'text', text: 'x', bad: () => 1 }] as never
|
||||
const response = await api.sessions.prompt(request({ sessionId, mode: 'queue' as const, content: poisoned }))
|
||||
vi.spyOn(ctx.agents.get(sessionId) as Agent, 'send').mockImplementation(() => {
|
||||
throw new Error('disposed during prompt')
|
||||
})
|
||||
const response = await api.sessions.prompt(request({
|
||||
sessionId, mode: 'queue' as const, content: [{ type: 'text' as const, text: 'x' }],
|
||||
}))
|
||||
expect(response.result.ok).toBe(false)
|
||||
if (!response.result.ok) expect(response.result.error.code).toBe('agent-busy')
|
||||
})
|
||||
|
||||
it('persists uploaded bytes before the user event and serves them only through the owning session', async () => {
|
||||
const persistenceRoot = mkdtempSync(join(tmpdir(), 'dsh-image-session-'))
|
||||
const dshHome = mkdtempSync(join(tmpdir(), 'dsh-image-home-'))
|
||||
host = await startHost({
|
||||
boot: { persistenceRoot, dshHome, provider: 'scripted', model: 'test-model' },
|
||||
})
|
||||
host.ctx.llm.registerAdapter(['scripted'], new ScriptedAdapter([textResponse('seen')]))
|
||||
const { sessionId } = expectOk(await host.api.sessions.create(request({})))
|
||||
const agent = host.ctx.agents.get(sessionId) as Agent
|
||||
const idle = waitForIdle(host.ctx, agent)
|
||||
const response = await host.api.sessions.prompt(request({
|
||||
sessionId,
|
||||
mode: 'queue' as const,
|
||||
content: [
|
||||
{ type: 'text' as const, text: 'describe' },
|
||||
{ type: 'image' as const, mediaType: 'image/png' as const, data: PNG_BASE64, name: '/tmp/pixel.png' },
|
||||
],
|
||||
}))
|
||||
expectOk(response)
|
||||
await idle
|
||||
|
||||
const user = agent.session.events.find(event => event.type === 'user/message')
|
||||
const content = (user?.data as { content?: ContentBlock[] } | undefined)?.content ?? []
|
||||
const image = content.find(block => block.type === 'image')
|
||||
expect(image?.type).toBe('image')
|
||||
if (image?.type !== 'image') throw new Error('image block missing')
|
||||
expect(JSON.stringify(user)).not.toContain(PNG_BASE64)
|
||||
expect(image.attachment.name).toBe('pixel.png')
|
||||
const sha256 = String(image.attachment.attachmentId).slice('sha256:'.length)
|
||||
const object = join(dshHome, 'attachments', 'v1', 'objects', sha256.slice(0, 2), sha256)
|
||||
expect(existsSync(object)).toBe(true)
|
||||
expect(readFileSync(object).toString('base64')).toBe(PNG_BASE64)
|
||||
|
||||
const loaded = expectOk(await host.api.sessions.attachment(request({
|
||||
sessionId, attachmentId: image.attachment.attachmentId,
|
||||
})))
|
||||
expect(loaded).toEqual({ attachment: image.attachment, data: PNG_BASE64 })
|
||||
const { sessionId: other } = expectOk(await host.api.sessions.create(request({})))
|
||||
const denied = await host.api.sessions.attachment(request({
|
||||
sessionId: other, attachmentId: image.attachment.attachmentId,
|
||||
}))
|
||||
expect(denied.result).toMatchObject({
|
||||
ok: false, error: { code: 'attachment-error', details: { reason: 'ATTACHMENT_NOT_REFERENCED' } },
|
||||
})
|
||||
})
|
||||
|
||||
it('rejects images for an explicitly text-only model without creating a session event', async () => {
|
||||
const persistenceRoot = mkdtempSync(join(tmpdir(), 'dsh-text-session-'))
|
||||
const dshHome = mkdtempSync(join(tmpdir(), 'dsh-text-home-'))
|
||||
host = await startHost({
|
||||
boot: { persistenceRoot, dshHome, provider: 'scripted', model: 'test-model' },
|
||||
})
|
||||
host.ctx.llm.registerAdapter(['scripted'], new ScriptedAdapter([], ['text']))
|
||||
const { sessionId } = expectOk(await host.api.sessions.create(request({})))
|
||||
const response = await host.api.sessions.prompt(request({
|
||||
sessionId, mode: 'queue' as const,
|
||||
content: [{ type: 'image' as const, mediaType: 'image/png' as const, data: PNG_BASE64 }],
|
||||
}))
|
||||
expect(response.result).toMatchObject({
|
||||
ok: false, error: { code: 'attachment-error', details: { reason: 'MODEL_DOES_NOT_SUPPORT_IMAGES' } },
|
||||
})
|
||||
expect(host.ctx.agents.get(sessionId)?.session.events.some(event => event.type === 'user/message')).toBe(false)
|
||||
expect(existsSync(join(dshHome, 'attachments'))).toBe(false)
|
||||
})
|
||||
|
||||
it('cancels an attached agent and rejects an unattached one', async () => {
|
||||
const running = await boot(['hang'])
|
||||
const { api, ctx } = running
|
||||
|
||||
@@ -17,9 +17,15 @@
|
||||
{
|
||||
"path": "../../llm/llm"
|
||||
},
|
||||
{
|
||||
"path": "../../attachment/attachment-local"
|
||||
},
|
||||
{
|
||||
"path": "../../llm/llm-deepseek"
|
||||
},
|
||||
{
|
||||
"path": "../../llm/llm-pi-ai"
|
||||
},
|
||||
{
|
||||
"path": "../../core/session"
|
||||
},
|
||||
|
||||
@@ -116,6 +116,8 @@ export class DeepSeekAdapter extends LlmAdapter {
|
||||
id: model.id,
|
||||
name: model.name ?? model.id,
|
||||
...model.description === undefined ? {} : { description: model.description },
|
||||
inputModalities: ['text'],
|
||||
outputModalities: ['text'],
|
||||
})))
|
||||
}
|
||||
|
||||
|
||||
@@ -2,10 +2,12 @@
|
||||
* Serialize harness messages into DeepSeek chat completions. User text is joined; assistant text
|
||||
* becomes `content`, tool calls become `tool_calls`, and tool results become separate tool messages.
|
||||
* Assistant reasoning is replayed as `reasoning_content` only on tool-call turns, as required by
|
||||
* thinking-mode passback. Unknown declaration-merged block types are skipped rather than rejected.
|
||||
* thinking-mode passback. Core image blocks are rejected explicitly because this wire route is text-only;
|
||||
* unknown declaration-merged block types retain the adapter's documented extension fallback.
|
||||
* @module dsh-llm-deepseek/serialize
|
||||
*/
|
||||
|
||||
import { LlmError } from '@deepseek-ai/dsh-llm'
|
||||
import type { ContentBlock, GenerateOptions, Message } from '@deepseek-ai/dsh-llm'
|
||||
import type { WireMessage, WireRequest, WireTool } from './types.ts'
|
||||
|
||||
@@ -23,6 +25,16 @@ function flattenText(blocks: ContentBlock[]): string {
|
||||
.join('')
|
||||
}
|
||||
|
||||
/** Reject core image content before any text-flattening path can silently erase it. */
|
||||
function assertTextOnly(blocks: readonly ContentBlock[]): void {
|
||||
for (const block of blocks) {
|
||||
if (block.type === 'image') {
|
||||
throw new LlmError('The DeepSeek chat-completions adapter does not support image content.', 'UNSUPPORTED_CONTENT')
|
||||
}
|
||||
if (block.type === 'tool-result') assertTextOnly(block.content)
|
||||
}
|
||||
}
|
||||
|
||||
/** Serialize one assistant message (text + reasoning + tool calls). */
|
||||
function serializeAssistant(message: Message): WireMessage {
|
||||
const text = flattenText(message.content)
|
||||
@@ -68,6 +80,7 @@ function serializeAssistant(message: Message): WireMessage {
|
||||
export function serializeMessages(messages: Message[]): WireMessage[] {
|
||||
const wire: WireMessage[] = []
|
||||
for (const message of messages) {
|
||||
assertTextOnly(message.content)
|
||||
if (message.role === 'system') {
|
||||
wire.push({ role: 'system', content: flattenText(message.content) })
|
||||
continue
|
||||
|
||||
@@ -528,8 +528,8 @@ describe('plugin registration and config', () => {
|
||||
await ctx.plugin(LlmDeepSeek, { apiKey: 'k', baseURL: 'http://127.0.0.1:1' })
|
||||
expect(ctx.llm.listProviders()).toEqual([{ id: 'deepseek', name: 'DeepSeek' }])
|
||||
await expect(ctx.llm.listModels('deepseek')).resolves.toEqual([
|
||||
{ provider: 'deepseek', id: 'deepseek-v4-flash', name: 'deepseek-v4-flash' },
|
||||
{ provider: 'deepseek', id: 'deepseek-v4-pro', name: 'deepseek-v4-pro' },
|
||||
{ provider: 'deepseek', id: 'deepseek-v4-flash', name: 'deepseek-v4-flash', inputModalities: ['text'], outputModalities: ['text'] },
|
||||
{ provider: 'deepseek', id: 'deepseek-v4-pro', name: 'deepseek-v4-pro', inputModalities: ['text'], outputModalities: ['text'] },
|
||||
])
|
||||
await expect(ctx.llm.resolveModelContext('deepseek', 'deepseek-v4-flash'))
|
||||
.resolves.toEqual({ contextWindow: 128_000 })
|
||||
@@ -540,8 +540,8 @@ describe('plugin registration and config', () => {
|
||||
await ctx.plugin(LlmService)
|
||||
LlmDeepSeek.apply(ctx, { apiKey: 'k', baseURL: 'http://127.0.0.1:1' })
|
||||
await expect(ctx.llm.listModels('deepseek')).resolves.toEqual([
|
||||
{ provider: 'deepseek', id: 'deepseek-v4-flash', name: 'deepseek-v4-flash' },
|
||||
{ provider: 'deepseek', id: 'deepseek-v4-pro', name: 'deepseek-v4-pro' },
|
||||
{ provider: 'deepseek', id: 'deepseek-v4-flash', name: 'deepseek-v4-flash', inputModalities: ['text'], outputModalities: ['text'] },
|
||||
{ provider: 'deepseek', id: 'deepseek-v4-pro', name: 'deepseek-v4-pro', inputModalities: ['text'], outputModalities: ['text'] },
|
||||
])
|
||||
})
|
||||
|
||||
@@ -562,8 +562,8 @@ describe('plugin registration and config', () => {
|
||||
],
|
||||
})
|
||||
await expect(ctx.llm.listModels('deepseek')).resolves.toEqual([
|
||||
{ provider: 'deepseek', id: 'private-fast', name: 'private-fast' },
|
||||
{ provider: 'deepseek', id: 'private-reasoner', name: 'Private Reasoner', description: 'Higher reasoning budget' },
|
||||
{ provider: 'deepseek', id: 'private-fast', name: 'private-fast', inputModalities: ['text'], outputModalities: ['text'] },
|
||||
{ provider: 'deepseek', id: 'private-reasoner', name: 'Private Reasoner', description: 'Higher reasoning budget', inputModalities: ['text'], outputModalities: ['text'] },
|
||||
])
|
||||
await expect(ctx.llm.resolveModelContext('deepseek', 'private-fast'))
|
||||
.resolves.toEqual({ contextWindow: 32_000 })
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import { describe, expect, it } from 'vitest'
|
||||
import { CallId } from '@deepseek-ai/dsh-llm'
|
||||
import { AttachmentId } from '@deepseek-ai/dsh-attachment'
|
||||
import type { ContentBlock, GenerateOptions, Message } from '@deepseek-ai/dsh-llm'
|
||||
import { serializeMessages, serializeRequest } from '../src/serialize.ts'
|
||||
|
||||
@@ -123,6 +124,19 @@ describe('serializeMessages', () => {
|
||||
expect(wire).toEqual([{ role: 'user', content: 'see chart' }])
|
||||
})
|
||||
|
||||
it('rejects image blocks instead of silently flattening them away', () => {
|
||||
expect(() => serializeMessages([{
|
||||
role: 'user',
|
||||
content: [{
|
||||
type: 'image',
|
||||
attachment: {
|
||||
attachmentId: AttachmentId(`sha256:${'a'.repeat(64)}`),
|
||||
mediaType: 'image/png', bytes: 68, width: 1, height: 1,
|
||||
},
|
||||
}],
|
||||
}])).toThrow(expect.objectContaining({ code: 'UNSUPPORTED_CONTENT' }))
|
||||
})
|
||||
|
||||
it('emits an empty user message rather than dropping block-less messages', () => {
|
||||
const wire = serializeMessages([{ role: 'user', content: [] }])
|
||||
expect(wire).toEqual([{ role: 'user', content: '' }])
|
||||
|
||||
@@ -27,6 +27,7 @@
|
||||
],
|
||||
"license": "BSD-3-Clause",
|
||||
"peerDependencies": {
|
||||
"@deepseek-ai/dsh-attachment": "^0.0.1",
|
||||
"@deepseek-ai/dsh-invariants": "^0.0.1",
|
||||
"@deepseek-ai/dsh-llm": "^0.0.1",
|
||||
"@deepseek-ai/dsh-timeout": "^0.0.1",
|
||||
@@ -37,6 +38,7 @@
|
||||
"schemastery": "^3.18.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@deepseek-ai/dsh-attachment": "workspace:^",
|
||||
"@deepseek-ai/dsh-invariants": "workspace:^",
|
||||
"@deepseek-ai/dsh-llm": "workspace:^",
|
||||
"@deepseek-ai/dsh-llm-deepseek": "workspace:^",
|
||||
|
||||
@@ -12,6 +12,7 @@ import type {
|
||||
Model,
|
||||
SimpleStreamOptions,
|
||||
} from '@earendil-works/pi-ai'
|
||||
import type { AttachmentStore } from '@deepseek-ai/dsh-attachment'
|
||||
import { attributionHeaders, LlmAdapter, LlmError } from '@deepseek-ai/dsh-llm'
|
||||
import type { GenerateOptions, LlmModelContext, LlmModelInfo, StreamChunk } from '@deepseek-ai/dsh-llm'
|
||||
import { idleWatchdog, timeoutOf } from '@deepseek-ai/dsh-timeout'
|
||||
@@ -24,6 +25,8 @@ import { toStreamChunks } from './stream.ts'
|
||||
export interface PiAiAdapterOptions {
|
||||
/** Validated provider profiles this adapter instance owns. */
|
||||
profiles: readonly PiAiProviderProfile[]
|
||||
/** Durable image resolver used only when a request contains image references. */
|
||||
attachments?: AttachmentStore
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -69,10 +72,12 @@ function requestHeaders(headers: Readonly<Record<string, string>> | undefined):
|
||||
*/
|
||||
export class PiAiAdapter extends LlmAdapter {
|
||||
private readonly profiles: ReadonlyMap<string, ResolvedPiAiProviderProfile>
|
||||
private readonly attachments: AttachmentStore | undefined
|
||||
|
||||
constructor(options: PiAiAdapterOptions) {
|
||||
super()
|
||||
this.profiles = new Map(resolveProfiles(options.profiles).map(profile => [profile.provider, profile]))
|
||||
this.attachments = options.attachments
|
||||
}
|
||||
|
||||
override listModels(provider: string): Promise<readonly LlmModelInfo[]> {
|
||||
@@ -84,6 +89,8 @@ export class PiAiAdapter extends LlmAdapter {
|
||||
provider,
|
||||
id: model.id,
|
||||
name: model.name,
|
||||
inputModalities: [...model.input],
|
||||
outputModalities: ['text'],
|
||||
})))
|
||||
}
|
||||
|
||||
@@ -112,7 +119,6 @@ export class PiAiAdapter extends LlmAdapter {
|
||||
throw new LlmError(`pi-ai adapter does not own provider "${options.provider}"`, 'NO_ADAPTER')
|
||||
}
|
||||
const model = resolveModel(profile, options.model)
|
||||
|
||||
const consumer = new AbortController()
|
||||
const upstream = options.signal === undefined
|
||||
? consumer.signal
|
||||
@@ -121,7 +127,22 @@ export class PiAiAdapter extends LlmAdapter {
|
||||
using watchdog = idleWatchdog(upstream, streamIdleTimeoutMs, 'LLM_STREAM_IDLE_TIMEOUT')
|
||||
|
||||
try {
|
||||
const events = streamSimple(model, toPiContext(options), {
|
||||
const containsImage = options.messages.some((message) => {
|
||||
// The discriminant is part of same-process message validity and is read before content.
|
||||
void message.role
|
||||
return message.content.some(block => block.type === 'image'
|
||||
|| (block.type === 'tool-result' && block.content.some(piece => piece.type === 'image')))
|
||||
})
|
||||
if (containsImage && !model.input.includes('image')) {
|
||||
throw new LlmError(`pi-ai model "${model.id}" does not support image input`, 'UNSUPPORTED_CONTENT')
|
||||
}
|
||||
if (containsImage && this.attachments === undefined) {
|
||||
throw new LlmError('pi-ai image input requires the durable attachment service', 'UNSUPPORTED_CONTENT')
|
||||
}
|
||||
const context = this.attachments === undefined
|
||||
? toPiContext(options)
|
||||
: await toPiContext(options, this.attachments)
|
||||
const events = streamSimple(model, context, {
|
||||
...profileOptions(profile),
|
||||
...options.temperature === undefined ? {} : { temperature: options.temperature },
|
||||
...options.maxTokens === undefined ? {} : { maxTokens: options.maxTokens },
|
||||
|
||||
@@ -4,9 +4,10 @@
|
||||
* @module dsh-llm-pi-ai/context
|
||||
*/
|
||||
|
||||
import { CallId } from '@deepseek-ai/dsh-llm'
|
||||
import type { GenerateOptions, Message } from '@deepseek-ai/dsh-llm'
|
||||
import type { Context as PiContext, Message as PiMessage, Tool as PiTool } from '@earendil-works/pi-ai'
|
||||
import { CallId, LlmError } from '@deepseek-ai/dsh-llm'
|
||||
import type { ContentBlock, GenerateOptions, Message } from '@deepseek-ai/dsh-llm'
|
||||
import type { AttachmentStore } from '@deepseek-ai/dsh-attachment'
|
||||
import type { Context as PiContext, ImageContent, Message as PiMessage, TextContent, Tool as PiTool } from '@earendil-works/pi-ai'
|
||||
import { toPiAssistant } from './replay.ts'
|
||||
|
||||
/** Join the text blocks of a harness message. */
|
||||
@@ -17,20 +18,122 @@ function flattenText(message: Message): string {
|
||||
.join('')
|
||||
}
|
||||
|
||||
async function userContent(
|
||||
blocks: readonly ContentBlock[],
|
||||
attachments: AttachmentStore,
|
||||
): Promise<string | (TextContent | ImageContent)[]> {
|
||||
const content: (TextContent | ImageContent)[] = []
|
||||
for (const block of blocks) {
|
||||
switch (block.type) {
|
||||
case 'text':
|
||||
if (block.text.length > 0) content.push({ type: 'text', text: block.text })
|
||||
break
|
||||
case 'image': {
|
||||
const stored = await attachments.readImage(block.attachment)
|
||||
content.push({
|
||||
type: 'image',
|
||||
data: Buffer.from(stored.data).toString('base64'),
|
||||
mimeType: stored.ref.mediaType,
|
||||
})
|
||||
break
|
||||
}
|
||||
case 'tool-result':
|
||||
break
|
||||
default:
|
||||
// Other merge-extensible blocks are not user-input vocabulary for pi-ai.
|
||||
break
|
||||
}
|
||||
}
|
||||
if (content.every(block => block.type === 'text')) return content.map(block => block.text).join('')
|
||||
return content
|
||||
}
|
||||
|
||||
function toolsOf(options: GenerateOptions): PiTool[] | undefined {
|
||||
return options.tools?.map(tool => ({
|
||||
name: tool.name,
|
||||
description: tool.description,
|
||||
// ToolSchema.parameters is a JSON Schema object; pi-ai's TSchema
|
||||
// (TypeBox) is structurally JSON Schema, so it assigns directly.
|
||||
parameters: tool.parameters,
|
||||
}))
|
||||
}
|
||||
|
||||
/** Assemble the request-level pi-ai context envelope shared by both conversion paths. */
|
||||
function piContext(options: GenerateOptions, messages: PiMessage[]): PiContext {
|
||||
const tools = toolsOf(options)
|
||||
return {
|
||||
...options.system !== undefined ? { systemPrompt: options.system } : {},
|
||||
messages,
|
||||
...tools !== undefined && tools.length > 0 ? { tools } : {},
|
||||
}
|
||||
}
|
||||
|
||||
function textOnlyContext(options: GenerateOptions): PiContext {
|
||||
const toolNames = new Map<CallId, string>()
|
||||
const messages: PiMessage[] = []
|
||||
for (const message of options.messages) {
|
||||
if (message.content.some(block => block.type === 'image'
|
||||
|| (block.type === 'tool-result' && block.content.some(piece => piece.type === 'image')))) {
|
||||
throw new LlmError('pi-ai image conversion requires the durable attachment service', 'UNSUPPORTED_CONTENT')
|
||||
}
|
||||
if (message.role === 'system') {
|
||||
messages.push({ role: 'user', content: flattenText(message), timestamp: 0 })
|
||||
continue
|
||||
}
|
||||
if (message.role === 'assistant') {
|
||||
const assistant = toPiAssistant(message)
|
||||
for (const block of assistant.content) if (block.type === 'toolCall') toolNames.set(CallId(block.id), block.name)
|
||||
messages.push(assistant)
|
||||
continue
|
||||
}
|
||||
const text = flattenText(message)
|
||||
const results = message.content.filter(block => block.type === 'tool-result')
|
||||
if (text.length > 0 || results.length === 0) messages.push({ role: 'user', content: text, timestamp: 0 })
|
||||
for (const result of results) {
|
||||
messages.push({
|
||||
role: 'toolResult',
|
||||
toolCallId: result.toolCallId,
|
||||
toolName: toolNames.get(result.toolCallId) ?? 'unknown',
|
||||
content: [{
|
||||
type: 'text',
|
||||
text: result.content.filter(block => block.type === 'text').map(block => block.text).join('') || '(no output)',
|
||||
}],
|
||||
isError: result.isError ?? false,
|
||||
timestamp: 0,
|
||||
})
|
||||
}
|
||||
}
|
||||
return piContext(options, messages)
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert harness history to a pi-ai Context. Tool results need the tool
|
||||
* NAME (pi-ai's `toolName`), which the harness doesn't carry on the result
|
||||
* block — it is recovered from the preceding assistant tool-call with the
|
||||
* same id.
|
||||
* Convert text-only harness history to a synchronous pi-ai Context. Tool
|
||||
* result names are recovered from preceding assistant tool calls.
|
||||
* @param options - the harness request; `options.system` maps to pi-ai's single `systemPrompt` slot.
|
||||
* @returns the pi-ai context; `tools` is omitted entirely when the request declares none.
|
||||
* @returns the pi-ai context; `tools` is omitted when the request declares none.
|
||||
*/
|
||||
export function toPiContext(options: GenerateOptions): PiContext {
|
||||
export function toPiContext(options: GenerateOptions): PiContext
|
||||
/**
|
||||
* Convert harness history to a pi-ai Context while resolving durable images.
|
||||
* Tool result names are recovered from preceding assistant tool calls.
|
||||
* @param options - the harness request; `options.system` maps to pi-ai's single `systemPrompt` slot.
|
||||
* @param attachments - durable byte resolver for image references.
|
||||
* @returns the asynchronously resolved pi-ai context.
|
||||
*/
|
||||
export function toPiContext(options: GenerateOptions, attachments: AttachmentStore): Promise<PiContext>
|
||||
export function toPiContext(options: GenerateOptions, attachments?: AttachmentStore): PiContext | Promise<PiContext> {
|
||||
return attachments === undefined ? textOnlyContext(options) : toPiContextWithImages(options, attachments)
|
||||
}
|
||||
|
||||
async function toPiContextWithImages(options: GenerateOptions, attachments: AttachmentStore): Promise<PiContext> {
|
||||
const toolNames = new Map<CallId, string>()
|
||||
const messages: PiMessage[] = []
|
||||
|
||||
for (const message of options.messages) {
|
||||
if (message.role === 'system') {
|
||||
if (message.content.some(block => block.type === 'image')) {
|
||||
throw new LlmError('pi-ai cannot represent an image in an in-history system message', 'UNSUPPORTED_CONTENT')
|
||||
}
|
||||
// pi-ai has a single systemPrompt slot; in-history system messages are
|
||||
// folded into user messages to preserve order (rare in practice — the
|
||||
// harness sends the system prompt via options.system).
|
||||
@@ -46,40 +149,26 @@ export function toPiContext(options: GenerateOptions): PiContext {
|
||||
continue
|
||||
}
|
||||
// user role: text + tool results (each result becomes its own message).
|
||||
const text = flattenText(message)
|
||||
const regular = message.content.filter(block => block.type !== 'tool-result')
|
||||
const content = await userContent(regular, attachments)
|
||||
const results = message.content.filter(block => block.type === 'tool-result')
|
||||
if (text.length > 0 || results.length === 0) {
|
||||
messages.push({ role: 'user', content: text, timestamp: 0 })
|
||||
if (content.length > 0 || results.length === 0) {
|
||||
messages.push({ role: 'user', content, timestamp: 0 })
|
||||
}
|
||||
for (const result of results) {
|
||||
const resultContent = await userContent(result.content, attachments)
|
||||
messages.push({
|
||||
role: 'toolResult',
|
||||
toolCallId: result.toolCallId,
|
||||
toolName: toolNames.get(result.toolCallId) ?? 'unknown',
|
||||
content: [{
|
||||
type: 'text',
|
||||
text: result.content
|
||||
.filter(block => block.type === 'text')
|
||||
.map(block => block.text)
|
||||
.join('') || '(no output)',
|
||||
}],
|
||||
content: typeof resultContent === 'string'
|
||||
? [{ type: 'text', text: resultContent || '(no output)' }]
|
||||
: resultContent,
|
||||
isError: result.isError ?? false,
|
||||
timestamp: 0,
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
const tools: PiTool[] | undefined = options.tools?.map(tool => ({
|
||||
name: tool.name,
|
||||
description: tool.description,
|
||||
// ToolSchema.parameters is a JSON Schema object; pi-ai's TSchema
|
||||
// (TypeBox) is structurally JSON Schema, so it assigns directly.
|
||||
parameters: tool.parameters,
|
||||
}))
|
||||
|
||||
return {
|
||||
...options.system !== undefined ? { systemPrompt: options.system } : {},
|
||||
messages,
|
||||
...tools !== undefined && tools.length > 0 ? { tools } : {},
|
||||
}
|
||||
return piContext(options, messages)
|
||||
}
|
||||
|
||||
@@ -36,6 +36,10 @@ export const inject = ['llm']
|
||||
/** Register one generic pi-ai adapter for all configured provider routes. */
|
||||
export function apply(ctx: Context, config: Config): void {
|
||||
const profiles = resolveProfiles(config.providers)
|
||||
const adapter = new PiAiAdapter({ profiles })
|
||||
const attachments = ctx.get('attachments')
|
||||
const adapter = new PiAiAdapter({
|
||||
profiles,
|
||||
...(attachments === undefined ? {} : { attachments }),
|
||||
})
|
||||
ctx.llm.registerAdapter(profiles.map(entry => entry.provider), adapter)
|
||||
}
|
||||
|
||||
@@ -134,6 +134,8 @@ function foreignAssistant(message: Message): AssistantMessage {
|
||||
name: block.name,
|
||||
arguments: parseArguments(block.arguments),
|
||||
}); break
|
||||
case 'image':
|
||||
throw new LlmError('pi-ai chat history cannot represent structured assistant image output', 'UNSUPPORTED_CONTENT')
|
||||
default:
|
||||
// plugin-added block types are not representable in pi-ai.
|
||||
break
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user