diff --git a/.agents/notes/implemented/architecture/2026-07-30-package-manager-native-repository-cache.i18n.yaml b/.agents/notes/implemented/architecture/2026-07-30-package-manager-native-repository-cache.i18n.yaml new file mode 100644 index 0000000000..6c888c7cad --- /dev/null +++ b/.agents/notes/implemented/architecture/2026-07-30-package-manager-native-repository-cache.i18n.yaml @@ -0,0 +1,6 @@ +# Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each +# side as of the last confirmed-consistent state. Both languages carry equal authority; +# after editing either side, bring the other along and re-record with: +# pnpm run verify-translation-pairing --write .agents/notes/implemented/architecture/2026-07-30-package-manager-native-repository-cache.md +2026-07-30-package-manager-native-repository-cache.md: f8a6706065a936ca4a9abf2a50d266a60f09b252 +2026-07-30-package-manager-native-repository-cache.zh.md: b1fea3d655f8d7aeb466744dc27bbf4ba69993ec diff --git a/.agents/notes/implemented/architecture/2026-07-30-package-manager-native-repository-cache.md b/.agents/notes/implemented/architecture/2026-07-30-package-manager-native-repository-cache.md new file mode 100644 index 0000000000..f8a6706065 --- /dev/null +++ b/.agents/notes/implemented/architecture/2026-07-30-package-manager-native-repository-cache.md @@ -0,0 +1,47 @@ +# Agent Note: Package-manager-native repository cache + +Status: implemented + +English | [中文](2026-07-30-package-manager-native-repository-cache.zh.md) + +## Problem + +A standalone Harness app cannot rely on a developer-owned SDK project to declare and install repository dependencies. Loading a configured GitHub repository therefore needs a persistent fetch, preparation, and cache boundary, but implementing Git transport, hosted-source syntax, package preparation, and a content store inside DSH would duplicate a package manager. Requiring a separately installed package manager would make a config-only feature depend on host setup. + +The cache also needs an update identity. A mutable branch name cannot both remain permanently cached and reflect later commits without an independent refresh protocol. + +## Decision + +Vendored `@cordisjs/plugin-loader/repository` exports `RepositoryCache`, a generic Node-only package helper with no DSH plugin-format knowledge. Keeping it on a subpath prevents browser consumers of the Loader's main entry from traversing Node filesystem and child-process imports. The caller supplies a package-manager-native source specifier and a cache root. DSH-specific callers own accepted source syntax, path selection, and the cache-root location; the [SDK project dependency workflow](../../proposed/feature/2026-07-17-sdk-follow-up-capabilities.md#external-cordis-plugin-installation) remains a separate path owned by the developer project's selected package manager. + +The Loader carries an exact runtime dependency on `pnpm@11.7.0` and invokes that package's JavaScript entry with the current Node executable. It never discovers a global executable or delegates through Corepack. Each cache miss creates an isolated project with one dependency named `repository`; pnpm owns Git/GitHub resolution, fetching, its content-addressed store, dependency installation, and lifecycle scripts in the repository's dependency graph. + +The isolated workspace sets `dangerouslyAllowAllBuilds: true`. A configured repository and its dependency graph are trusted executable code: lifecycle scripts may run before DSH reads any declared assets. The child receives ordinary host process state needed by Git and pnpm, but ambient credential-shaped (`KEY`, `PASSWORD`, `SECRET`, `TOKEN`) variables are removed. No OAuth, token forwarding, or private-repository authentication contract is added. + +The SHA-256 of the exact specifier names the cache entry. Concurrent same-process requests share one task. Installation occurs in a sibling temporary directory; only a successful install with a package directory and marker is atomically renamed into the final key. Failed staging is removed, and a competing process's already-published valid entry wins. A later process validates the marker and package directory before returning the stable `node_modules/repository` path. + +An identical specifier permanently reuses its published entry. The caller changes the ref or another part of the specifier to request a new generation; the cache does not poll remotes, reinterpret mutable refs, expire entries, or garbage-collect old generations. + +## Alternatives considered + +**Implement GitHub download, archive extraction, preparation, and caching directly.** Rejected under the [dependency policy](../process/2026-07-26-dependencies-over-hand-rolling.md): pnpm already owns hosted Git syntax, Git execution, lifecycle policy, and a shared content store. A second resolver would add more code while still needing package semantics. + +**Require `pnpm` on `PATH` or invoke Corepack.** Rejected because changing one app config must be sufficient on every supported installation. Pinning and shipping the CLI also makes the preparation policy reviewable and independent of the host's package-manager version. + +**Resolve a branch or tag again on every startup.** Rejected because it turns startup into a network refresh, changes code without a config diff, and makes rollback depend on remote state. Explicit ref changes preserve auditability even when a user deliberately chooses a mutable ref. + +**Disable repository lifecycle scripts.** Rejected because common plugin repositories need a declarative `prepare` step to validate and package their plugin subdirectory. The trust boundary is explicit configuration of executable source, not an incomplete illusion that only static files can run. + +**Introduce a Cordis repository service.** Rejected because cache lookup has no runtime contribution registry or provider variation. A small helper lets the later host own Cordis lifecycle and HMR without adding a service seam prematurely. + +## Consequences + +- Standalone apps carry pnpm's approximately 18.6 MB unpacked runtime instead of requiring a global tool or owning a Git/package implementation. +- A repository author may use ordinary package preparation, and a malicious configured repository or dependency can execute code with the scrubbed child environment and the user's filesystem authority. +- Exact specifiers make startup deterministic after the first successful install; changing cached code requires a config/ref change. +- Failed installs leave no published cache entry and may be retried. Published corruption fails loud instead of silently reinstalling under the same identity. +- Cache generations consume disk until a future explicit cache-management policy removes them. + +## Testing + +`packages/ui/app-boot/tests/repository-cache.spec.ts` covers same-process single-flight, cross-instance cache reuse, exact-specifier separation, failed-stage cleanup and retry, and boundary validation. Its real local-Git case invokes the bundled pnpm, runs the fixture repository's `prepare` script, and reads the prepared file from the installed cache entry without network access. diff --git a/.agents/notes/implemented/architecture/2026-07-30-package-manager-native-repository-cache.zh.md b/.agents/notes/implemented/architecture/2026-07-30-package-manager-native-repository-cache.zh.md new file mode 100644 index 0000000000..b1fea3d655 --- /dev/null +++ b/.agents/notes/implemented/architecture/2026-07-30-package-manager-native-repository-cache.zh.md @@ -0,0 +1,47 @@ +# Agent Note: 包管理器原生仓库缓存 + +Status: implemented + +[English](2026-07-30-package-manager-native-repository-cache.md) | 中文 + +## 问题 + +独立运行的 Harness 应用不能依赖开发者自有的 SDK 工程来声明并安装仓库依赖。因此,加载配置中的 GitHub 仓库需要一道持久的获取、准备与缓存边界;但如果在 DSH 内实现 Git 传输、托管来源语法、包(package)准备流程和内容存储,就会重复实现包管理器。若要求用户另行安装包管理器,则只需修改配置即可使用的功能还会依赖宿主环境的额外配置。 + +缓存还需要明确更新标识。若没有独立的刷新协议,可变分支名无法既永久缓存,又反映后续 commit。 + +## 决策 + +vendor 中的 `@cordisjs/plugin-loader/repository` 导出 `RepositoryCache`:一个不包含 DSH 插件格式知识、仅限 Node 使用的通用包辅助工具。把它保留在子路径上,可以避免 Loader 主入口的浏览器消费方在解析依赖时遍历到 Node 文件系统和子进程 import。调用方提供包管理器原生的来源 specifier 和缓存根目录。DSH 专属调用方负责规定可接受的来源语法、路径选择与缓存根目录位置;[SDK 工程依赖工作流](../../proposed/feature/2026-07-17-sdk-follow-up-capabilities.md#external-cordis-plugin-installation)仍是另一条路径,由开发者工程选定的包管理器负责。 + +Loader 将 `pnpm@11.7.0` 作为固定版本的运行时依赖,并使用当前 Node 可执行文件调用该包的 JavaScript 入口。它绝不探测全局可执行文件,也不经 Corepack 调用。每次缓存未命中都会创建一个隔离工程,其中只有一个名为 `repository` 的依赖;Git 与 GitHub 来源的解析和获取、pnpm 自身的内容寻址 store、依赖安装,以及仓库依赖图中的生命周期脚本均由 pnpm 负责。 + +隔离工作区设置 `dangerouslyAllowAllBuilds: true`。用户配置的仓库及其依赖图都属于受信任的可执行代码:DSH 读取任何已声明资产之前,生命周期脚本就可能运行。子进程会收到 Git 与 pnpm 所需的常规宿主进程状态,但会移除环境中名称形似凭据(`KEY`、`PASSWORD`、`SECRET`、`TOKEN`)的变量。该机制不新增 OAuth、token 转发或私有仓库认证契约。 + +缓存项以精确 specifier 的 SHA-256 命名。同一进程内针对相同 specifier 的并发请求共享一项任务。安装在同级临时目录中进行;只有安装成功且存在包目录和标记时,系统才会把暂存目录原子重命名为最终键对应的目录。失败的暂存目录会被删除;如果另一进程已发布有效项,则以该项为准。后续进程会先校验标记与包目录,再返回稳定的 `node_modules/repository` 路径。 + +相同的 specifier 会永久复用已发布项。调用方通过修改 ref 或 specifier 的其他部分来请求新的缓存代次;缓存不会轮询远端、重新解释可变 ref、让条目过期,也不会垃圾回收旧代次。 + +## 曾考虑的替代方案 + +**直接实现 GitHub 下载、归档解压、准备与缓存。** 根据[依赖政策](../process/2026-07-26-dependencies-over-hand-rolling.md)不予采纳:pnpm 已负责托管 Git 语法、Git 执行、生命周期政策和共享内容存储。第二套解析器会增加更多代码,却仍需实现包语义。 + +**要求 `pnpm` 位于 `PATH` 上,或调用 Corepack。** 不予采纳:在每种受支持的安装形态中,只修改一份应用配置就必须足以启用该功能。固定并随应用分发 CLI(命令行界面)还能使准备政策可供评审,并与宿主的包管理器版本无关。 + +**每次启动都重新解析分支或 tag。** 不予采纳:这会把启动变成网络刷新,在配置 diff 未变化时更改代码,并让回滚依赖远端状态。即使用户有意选择可变 ref,显式修改 ref 仍能保持可审计性。 + +**禁用仓库生命周期脚本。** 不予采纳:常见插件仓库需要声明式 `prepare` 步骤来校验并打包插件子目录。信任边界是显式配置可执行来源,而不是营造一种不完整的假象,仿佛只有静态文件能够运行。 + +**引入 Cordis 仓库服务。** 不予采纳:缓存查找没有运行时贡献注册表,也不存在提供方变体。小型 helper 让后续宿主负责 Cordis 生命周期与 HMR(热模块替换),无需过早新增服务 seam。 + +## 后果 + +- 独立应用随附 pnpm 约 18.6 MB 的解压后运行时,不要求全局工具,也无需自行实现 Git 与包处理。 +- 仓库作者可以使用常规包准备流程;恶意的已配置仓库或依赖可以在经过上述清理的子进程环境中,以用户的文件系统权限执行代码。 +- 精确 specifier 使首次安装成功后的启动具有确定性;更改缓存代码必须修改配置或 ref。 +- 安装失败不会留下已发布缓存项,可以再次重试。已发布缓存损坏时会明确报错,而不会在同一标识下静默重装。 +- 缓存代次会持续占用磁盘,直到未来有明确的缓存管理政策将其移除。 + +## 测试 + +`packages/ui/app-boot/tests/repository-cache.spec.ts` 覆盖同进程 single-flight、跨实例缓存复用、精确 specifier 隔离、失败暂存清理与重试,以及边界校验。其真实本地 Git 用例会调用随附的 pnpm,运行 fixture(测试前置数据)仓库的 `prepare` 脚本,并在不访问网络的情况下,从已安装缓存项中读取准备后的文件。 diff --git a/.agents/notes/proposed/feature/2026-07-17-sdk-follow-up-capabilities.i18n.yaml b/.agents/notes/proposed/feature/2026-07-17-sdk-follow-up-capabilities.i18n.yaml index 8b70484312..fab3ecc2b1 100644 --- a/.agents/notes/proposed/feature/2026-07-17-sdk-follow-up-capabilities.i18n.yaml +++ b/.agents/notes/proposed/feature/2026-07-17-sdk-follow-up-capabilities.i18n.yaml @@ -1,6 +1,6 @@ # Bilingual-pair consistency record (docs/i18n/README.md): the git blob hash of each # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: -# pnpm run verify-translation-pairing --write -2026-07-17-sdk-follow-up-capabilities.md: 0f3ada6bdbb4ce933d14602cf59be9a51640e61c -2026-07-17-sdk-follow-up-capabilities.zh.md: d0d0b3e6bcdf192e64f003dc9f6e90cc2bdb060b +# pnpm run verify-translation-pairing --write .agents/notes/proposed/feature/2026-07-17-sdk-follow-up-capabilities.md +2026-07-17-sdk-follow-up-capabilities.md: 88d5d2f9bd1ce01c20177bcaee5bbe6b434bb978 +2026-07-17-sdk-follow-up-capabilities.zh.md: 998b7ec3cfddafe40537908fb61aa6d7e6f90418 diff --git a/.agents/notes/proposed/feature/2026-07-17-sdk-follow-up-capabilities.md b/.agents/notes/proposed/feature/2026-07-17-sdk-follow-up-capabilities.md index 0f3ada6bdb..88d5d2f9bd 100644 --- a/.agents/notes/proposed/feature/2026-07-17-sdk-follow-up-capabilities.md +++ b/.agents/notes/proposed/feature/2026-07-17-sdk-follow-up-capabilities.md @@ -47,6 +47,8 @@ The repository ships a thin `SKILL.md` that teaches an agent to construct the st The package manager owns source parsing, version or commit resolution, integrity data, lockfile updates, and any build policy. The SDK does not download or unpack a second copy through giget or pacote. An external plugin remains a dependency under `node_modules`; local plugin scaffolding remains a separate project-creation concern. +This proposal concerns dependencies of developer-owned SDK projects. Standalone app repository caching, its bundled-pnpm policy, and its explicit preparation trust boundary are owned by the [package-manager-native repository cache](../../implemented/architecture/2026-07-30-package-manager-native-repository-cache.md). + ## Launcher telemetry ### Consent and collection diff --git a/.agents/notes/proposed/feature/2026-07-17-sdk-follow-up-capabilities.zh.md b/.agents/notes/proposed/feature/2026-07-17-sdk-follow-up-capabilities.zh.md index d0d0b3e6bc..998b7ec3cf 100644 --- a/.agents/notes/proposed/feature/2026-07-17-sdk-follow-up-capabilities.zh.md +++ b/.agents/notes/proposed/feature/2026-07-17-sdk-follow-up-capabilities.zh.md @@ -47,6 +47,8 @@ Create 和 config 使用相同的功能计划形状。create 通过上述命令 包管理器负责来源解析、版本或 commit 解析、`integrity` 数据、lockfile 更新和构建策略。SDK 不再通过 giget 或 pacote 下载、解压第二份副本。外部插件是 `node_modules` 下的依赖;本地插件脚手架仍属于独立的工程创建问题。 +本提案只涉及开发者自有 SDK 工程的依赖。独立应用的仓库缓存、随应用捆绑 pnpm 的政策和显式的准备流程信任边界,均由[包管理器原生仓库缓存](../../implemented/architecture/2026-07-30-package-manager-native-repository-cache.md)负责。 + ## Launcher 遥测 ### Consent 与采集 diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index 754ae93d82..92ea0d2406 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -66,6 +66,7 @@ External packages that a workspace package resolves at runtime. `scripts/install | [`node-addon-require-builtin`](https://www.npmjs.com/package/node-addon-require-builtin) | MIT | | [`node-pty`](https://github.com/microsoft/node-pty) | MIT | | [`picomatch`](https://github.com/micromatch/picomatch) | MIT | +| [`pnpm`](https://github.com/pnpm/pnpm) | MIT | | [`react`](https://github.com/facebook/react) | MIT | | [`react-dom`](https://github.com/facebook/react) | MIT | | [`react-markdown`](https://github.com/remarkjs/react-markdown) | MIT | diff --git a/packages/ui/app-boot/tests/repository-cache.spec.ts b/packages/ui/app-boot/tests/repository-cache.spec.ts new file mode 100644 index 0000000000..33a223020f --- /dev/null +++ b/packages/ui/app-boot/tests/repository-cache.spec.ts @@ -0,0 +1,144 @@ +import { execFile } from 'node:child_process' +import { createHash } from 'node:crypto' +import { mkdtemp, mkdir, readFile, readdir, rm, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { pathToFileURL } from 'node:url' +import { promisify } from 'node:util' +import { afterEach, describe, expect, it, vi } from 'vitest' +import { BUNDLED_PNPM_VERSION, RepositoryCache, type RepositoryInstall } from '@cordisjs/plugin-loader/repository' + +const execFileAsync = promisify(execFile) +const roots: string[] = [] + +async function temporaryRoot(name: string): Promise { + const root = await mkdtemp(join(tmpdir(), `cordis-${name}-`)) + roots.push(root) + return root +} + +async function fakePackage(directory: string): Promise { + const target = join(directory, 'node_modules', 'repository') + await mkdir(target, { recursive: true }) + await writeFile(join(target, 'package.json'), '{"name":"fixture"}\n') +} + +afterEach(async () => { + vi.unstubAllEnvs() + await Promise.all(roots.splice(0).map(root => rm(root, { recursive: true, force: true }))) +}) + +describe('RepositoryCache', () => { + it('single-flights and permanently reuses an exact specifier', async () => { + const root = await temporaryRoot('repository-cache') + const calls: string[] = [] + const install: RepositoryInstall = async (directory) => { + calls.push(directory) + await fakePackage(directory) + } + const cache = new RepositoryCache(root, install) + const specifier = 'github:owner/repository#0123456789abcdef' + + const [first, concurrent] = await Promise.all([cache.resolve(specifier), cache.resolve(specifier)]) + expect(concurrent).toBe(first) + expect(calls).toHaveLength(1) + + const reopened = new RepositoryCache(root, async () => { throw new Error('cache miss') }) + expect(await reopened.resolve(specifier)).toBe(first) + expect(JSON.parse(await readFile(join(first, '..', '..', 'package.json'), 'utf8'))).toMatchObject({ + packageManager: `pnpm@${BUNDLED_PNPM_VERSION}`, + dependencies: { repository: specifier }, + }) + + const second = await cache.resolve('github:owner/repository#fedcba9876543210') + expect(second).not.toBe(first) + expect(calls).toHaveLength(2) + }) + + it('accepts the valid winner when independent cache instances race', async () => { + const root = await temporaryRoot('repository-race') + const bothStarted = Promise.withResolvers() + let starts = 0 + const install: RepositoryInstall = async (directory) => { + await fakePackage(directory) + starts += 1 + if (starts === 2) bothStarted.resolve(undefined) + await bothStarted.promise + } + const specifier = 'github:owner/repository#race' + + const [first, second] = await Promise.all([ + new RepositoryCache(root, install).resolve(specifier), + new RepositoryCache(root, install).resolve(specifier), + ]) + + expect(second).toBe(first) + expect(starts).toBe(2) + expect(await readdir(root)).toHaveLength(1) + }) + + it('removes a failed staging tree and permits an exact retry', async () => { + const root = await temporaryRoot('repository-retry') + let attempts = 0 + const cache = new RepositoryCache(root, async (directory) => { + attempts += 1 + if (attempts === 1) throw new Error('install failed') + await fakePackage(directory) + }) + + await expect(cache.resolve('github:owner/repository#ref')).rejects.toThrow('failed to prepare repository') + expect(await readdir(root)).toEqual([]) + await expect(cache.resolve('github:owner/repository#ref')).resolves.toContain('node_modules') + expect(attempts).toBe(2) + }) + + it('rejects empty or padded specifiers before touching the cache', async () => { + const root = await temporaryRoot('repository-input') + const cache = new RepositoryCache(root, fakePackage) + expect(() => cache.resolve('')).toThrow('non-empty unpadded string') + expect(() => cache.resolve(' github:owner/repository#ref')).toThrow('non-empty unpadded string') + await expect(readdir(root)).resolves.toEqual([]) + }) + + it('fails loud on a corrupt published marker instead of reinstalling it', async () => { + const root = await temporaryRoot('repository-corrupt') + const specifier = 'github:owner/repository#corrupt' + const key = createHash('sha256').update(specifier).digest('hex') + const entry = join(root, key) + await mkdir(join(entry, 'node_modules', 'repository'), { recursive: true }) + await writeFile(join(entry, '.repository-cache.json'), '{}\n') + const cache = new RepositoryCache(root, async () => { throw new Error('must not reinstall') }) + + await expect(cache.resolve(specifier)).rejects.toThrow('repository cache marker is invalid') + }) + + it('runs a Git dependency prepare script through the bundled pnpm', { timeout: 60_000 }, async () => { + const root = await temporaryRoot('repository-pnpm') + const repository = join(root, 'source') + await mkdir(repository) + await writeFile(join(repository, 'package.json'), `${JSON.stringify({ + name: 'repository-fixture', + version: '1.0.0', + scripts: { prepare: 'node prepare.mjs' }, + })}\n`) + await writeFile(join(repository, 'prepare.mjs'), [ + "import { writeFile } from 'node:fs/promises'", + "await writeFile('prepared.txt', `${process.env.REPOSITORY_TEST_VISIBLE ?? 'absent'}|${process.env.REPOSITORY_TEST_TOKEN ?? 'absent'}\\n`)", + '', + ].join('\n')) + await execFileAsync('git', ['init', '--quiet'], { cwd: repository }) + await execFileAsync('git', ['add', '.'], { cwd: repository }) + await execFileAsync('git', [ + '-c', 'user.name=Repository Fixture', + '-c', 'user.email=repository@example.invalid', + 'commit', '--quiet', '-m', 'fixture', + ], { cwd: repository }) + const { stdout } = await execFileAsync('git', ['rev-parse', 'HEAD'], { cwd: repository, encoding: 'utf8' }) + const specifier = `git+${pathToFileURL(repository).href}#${stdout.trim()}` + vi.stubEnv('REPOSITORY_TEST_VISIBLE', 'visible') + vi.stubEnv('REPOSITORY_TEST_TOKEN', 'hidden') + + const installed = await new RepositoryCache(join(root, 'cache')).resolve(specifier) + await expect(readFile(join(installed, 'prepared.txt'), 'utf8')).resolves.toBe('visible|absent\n') + }) +}) diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index cefd39d0ee..e7f8fd6148 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -6548,6 +6548,9 @@ importers: node-addon-require-builtin: specifier: ^0.1.3 version: 0.1.3 + pnpm: + specifier: 11.7.0 + version: 11.7.0 vendor/logger-console: dependencies: @@ -10951,6 +10954,11 @@ packages: engines: {node: '>=18'} hasBin: true + pnpm@11.7.0: + resolution: {integrity: sha512-GcyFLBIMcSV2DyRD7mvgyltA+fUFmN4aCaHxd1A+AQ5Xwjx3ZG4B52HeWb+HT7IqM5jDOrlpH8E+uUa28PTWIA==} + engines: {node: '>=22.13'} + hasBin: true + points-on-curve@0.2.0: resolution: {integrity: sha512-0mYKnYYe9ZcqMCWhUjItv/oHjvgEsfKvnUTg8sAtnHr3GVy7rGkXCb6d5cSyqrWqL4k81b9CPg3urd+T7aop3A==} @@ -16387,6 +16395,8 @@ snapshots: optionalDependencies: fsevents: 2.3.2 + pnpm@11.7.0: {} + points-on-curve@0.2.0: {} points-on-path@0.2.1: diff --git a/tsconfig.base.json b/tsconfig.base.json index 2d078202e7..94b60804e4 100644 --- a/tsconfig.base.json +++ b/tsconfig.base.json @@ -32,6 +32,7 @@ "cosmokit": ["./vendor/cosmokit/src"], "schemastery": ["./vendor/schemastery/src"], "@cordisjs/plugin-loader": ["./vendor/loader/src"], + "@cordisjs/plugin-loader/repository": ["./vendor/loader/src/repository.ts"], "@cordisjs/plugin-include": ["./vendor/include/src"], "@cordisjs/plugin-group": ["./vendor/group/src"], "@cordisjs/plugin-timer": ["./vendor/timer/src"], diff --git a/vendor/README.md b/vendor/README.md index 443c78e278..3872faa753 100644 --- a/vendor/README.md +++ b/vendor/README.md @@ -39,8 +39,9 @@ Keep this log exhaustive — every divergence from upstream must be listed. 7. **`cordis/src/*.ts` JSDoc enrichment**: added `@param`/`@returns` tags and contract documentation (disposal semantics, waterfall veto, bail conditions, error cases) across the public plugin-author surface — `Context` (class, statics, and the `Context` interface properties incl. `root`), `EventsService`, `Fiber`, `RegistryService`, `ReflectService`, `Service`, `LoggerService` and their `declare module './context.ts'` overloads. Comment-only; no code changes. Motivation: the website API-reference generator renders these docs and hard-errors on undocumented members. Retire this entry when the enrichment is upstreamed to the fork. 8. **Transactional Loader/Include config reconciliation**: Loader imports a changed entry name before disposal, awaits lifecycle settlement, and restores the previous plugin or config when candidate application fails. Loader settlement rechecks service-gated fibers after current tasks drain, rejects failures, and leaves fibers with absent dependencies pending. Group updates start candidates concurrently, await every outcome, undo changes and additions on failure, await removal, preserve programmatic option identity, and persist direct or tree-level mutations only after success. Include reads and validates detached candidate content, applies patches to a clone, reconciles the tree, and only then commits its cached content/data; direct refresh failures propagate for the caller to contain. A non-array parse is invalid, patches re-apply on every file or Include-config update, an omitted patch list clears the overlay, and initial content falls back to `initial` only on `ENOENT`. Covered by `packages/ui/app-boot/tests/config-reload.spec.ts` and `packages/host/webserver/tests/webserver.spec.ts`. 9. **`hmr/src/index.ts` exact config watching**: `registerConfig()` watches one absolute config path outside module roots, including a path under missing parents, serializes and coalesces refreshes, and returns an async disposer that closes the watcher and drains active work. Refresh failures are normalized to `Error`, logged, and broadcast through the parallel `hmr/config-update-failed` event; observer failures are contained. Config-file changes discovered by the ordinary HMR watcher use the same serialized path. Covered by `packages/ui/app-boot/tests/hmr-config.spec.ts`. -10. **Vendored Node-compatible TypeScript**: marked erased imports explicitly across `cordis`, `loader`, `include`, `hmr`, and `schemastery` so Node's native TypeScript transform does not request types as runtime exports. Schemastery's source uses an ESM default export and its package declares `type: module`; its built ESM/CJS entries retain explicit `.mjs`/`.cjs` extensions. -11. **`include/src/index.ts` patch-semantics export**: extracted the private `applyPatches` body into the exported pure function `applyEntryPatches(data, patches, warn)` (the method delegates to it) and exported the `!!js` YAML dialect as `entryListSchema`, so `dsh --dump-config` composes and prints exactly what the include would mount without booting a tree. Behavior-preserving for mounting; the extraction exists because config tooling must never reimplement (and drift from) the patch algorithm. `applyEntryPatches` also indexes each `insert`ed entry as it is added, so a later patch in the same list can configure or disable a row an earlier patch inserted; upstream built the id index once before the patch loop, leaving inserted rows silently unpatchable. That matters because `dsh` composes one shared base (`apps/cli/config/base.cordis.yml`) with a surface overlay, an optional `--config` overlay, and the personal `~/.dsh/config.yaml` as sibling patch lists at one include level — patches never cross an include boundary, so surface-only rows would otherwise be unreachable from user config. Covered by `packages/ui/app-boot/tests/config-reload.spec.ts`. +10. **`loader/src/repository.ts`, `loader/tsdown.config.ts`, and the `@cordisjs/plugin-loader/repository` export**: the Node-only `RepositoryCache` installs one exact dependency specifier through the bundled `pnpm@11.7.0`, single-flights callers, and atomically publishes only a prepared package plus marker under the specifier hash. The subpath stays out of the browser-reachable Loader entry. Identical specifiers permanently reuse that entry; callers change the ref/specifier for another generation. The isolated workspace permits dependency build scripts because a configured repository is executable code, while the child drops ambient credential-shaped variables. Covered by `packages/ui/app-boot/tests/repository-cache.spec.ts`, including a keyless local-Git prepare run through the bundled pnpm. +11. **Vendored Node-compatible TypeScript**: marked erased imports explicitly across `cordis`, `loader`, `include`, `hmr`, and `schemastery` so Node's native TypeScript transform does not request types as runtime exports. Schemastery's source uses an ESM default export and its package declares `type: module`; its built ESM/CJS entries retain explicit `.mjs`/`.cjs` extensions. +12. **`include/src/index.ts` patch-semantics export**: extracted the private `applyPatches` body into the exported pure function `applyEntryPatches(data, patches, warn)` (the method delegates to it) and exported the `!!js` YAML dialect as `entryListSchema`, so `dsh --dump-config` composes and prints exactly what the include would mount without booting a tree. Behavior-preserving for mounting; the extraction exists because config tooling must never reimplement (and drift from) the patch algorithm. `applyEntryPatches` also indexes each `insert`ed entry as it is added, so a later patch in the same list can configure or disable a row an earlier patch inserted; upstream built the id index once before the patch loop, leaving inserted rows silently unpatchable. That matters because `dsh` composes one shared base (`apps/cli/config/base.cordis.yml`) with a surface overlay, an optional `--config` overlay, and the personal `~/.dsh/config.yaml` as sibling patch lists at one include level — patches never cross an include boundary, so surface-only rows would otherwise be unreachable from user config. Covered by `packages/ui/app-boot/tests/config-reload.spec.ts`. ## Sync procedure diff --git a/vendor/loader/package.json b/vendor/loader/package.json index c7bbaf5176..e24e0657a6 100644 --- a/vendor/loader/package.json +++ b/vendor/loader/package.json @@ -11,11 +11,16 @@ "types": "./lib/types/index.d.ts", "default": "./lib/index.js" }, + "./repository": { + "types": "./lib/types/repository.d.ts", + "default": "./lib/repository.js" + }, "./src/*": "./src/*", "./package.json": "./package.json" }, "files": [ "lib/index.js", + "lib/repository.js", "lib/types/**/*.d.ts", "lib/types/**/*.d.ts.map", "src" @@ -32,6 +37,7 @@ } }, "dependencies": { - "cosmokit": "^1.8.1" + "cosmokit": "^1.8.1", + "pnpm": "11.7.0" } } diff --git a/vendor/loader/src/repository.ts b/vendor/loader/src/repository.ts new file mode 100644 index 0000000000..94a0c5cf16 --- /dev/null +++ b/vendor/loader/src/repository.ts @@ -0,0 +1,191 @@ +/** + * Exact-specifier repository packages installed through the Loader's bundled + * pnpm. The caller owns source validation and the cache root; this module owns + * isolated installation, single-flight reuse, and atomic cache publication. + */ + +import { spawn } from 'node:child_process' +import { createHash } from 'node:crypto' +import { mkdir, mkdtemp, readFile, rename, rm, stat, writeFile } from 'node:fs/promises' +import { createRequire } from 'node:module' +import { dirname, join, resolve } from 'node:path' + +/** Exact pnpm release shipped with the Loader for repository installation. */ +export const BUNDLED_PNPM_VERSION = '11.7.0' + +const DEPENDENCY_NAME = 'repository' +const MARKER_NAME = '.repository-cache.json' +const MAX_ERROR_OUTPUT = 32 * 1024 +const SENSITIVE_ENV_PATTERN = /KEY|PASSWORD|SECRET|TOKEN/i + +/** Injectable isolated-install boundary used by {@link RepositoryCache}. */ +export type RepositoryInstall = (directory: string) => Promise + +interface CacheMarker { + specifier: string +} + +function scrubEnvironment(environment: NodeJS.ProcessEnv = process.env): NodeJS.ProcessEnv { + return Object.fromEntries(Object.entries(environment).filter(([name]) => !SENSITIVE_ENV_PATTERN.test(name))) +} + +function appendOutput(current: string, chunk: Uint8Array): string { + const combined = current + Buffer.from(chunk).toString('utf8') + return combined.length <= MAX_ERROR_OUTPUT ? combined : combined.slice(-MAX_ERROR_OUTPUT) +} + +async function installWithBundledPnpm(directory: string): Promise { + const require = createRequire(import.meta.url) + const pnpmManifest = require.resolve('pnpm') + const pnpmBin = join(dirname(pnpmManifest), 'bin', 'pnpm.mjs') + let output = '' + const result = await new Promise<{ code: number | null; signal: NodeJS.Signals | null }>((resolve, reject) => { + const child = spawn(process.execPath, [ + pnpmBin, + 'install', + '--no-frozen-lockfile', + '--reporter=append-only', + ], { + cwd: directory, + env: scrubEnvironment(), + shell: false, + stdio: ['ignore', 'pipe', 'pipe'], + }) + child.stdout.on('data', (chunk: Uint8Array) => { output = appendOutput(output, chunk) }) + child.stderr.on('data', (chunk: Uint8Array) => { output = appendOutput(output, chunk) }) + child.once('error', reject) + child.once('close', (code, signal) => { resolve({ code, signal }) }) + }) + if (result.signal !== null) { + throw new Error(`bundled pnpm install was killed by ${result.signal}${output ? `\n${output.trimEnd()}` : ''}`) + } + if (result.code !== 0) { + throw new Error(`bundled pnpm install exited with code ${String(result.code)}${output ? `\n${output.trimEnd()}` : ''}`) + } +} + +function cacheKey(specifier: string): string { + return createHash('sha256').update(specifier).digest('hex') +} + +async function readCached(directory: string, specifier: string): Promise { + let content: string + try { + content = await readFile(join(directory, MARKER_NAME), 'utf8') + } catch (error) { + if ((error as NodeJS.ErrnoException).code === 'ENOENT') return + throw error + } + let parsed: unknown + try { + parsed = JSON.parse(content) as unknown + } catch (error) { + throw new Error(`repository cache marker is invalid: ${join(directory, MARKER_NAME)}`, { cause: error }) + } + if (typeof parsed !== 'object' || parsed === null || typeof (parsed as Partial).specifier !== 'string') { + throw new Error(`repository cache marker is invalid: ${join(directory, MARKER_NAME)}`) + } + const marker = parsed as CacheMarker + if (marker.specifier !== specifier) { + throw new Error(`repository cache key collision for ${JSON.stringify(specifier)}`) + } + const packageDirectory = join(directory, 'node_modules', DEPENDENCY_NAME) + let packageStat + try { + packageStat = await stat(packageDirectory) + } catch (error) { + throw new Error(`repository cache entry is incomplete: ${directory}`, { cause: error }) + } + if (!packageStat.isDirectory()) throw new Error(`repository cache package is not a directory: ${packageDirectory}`) + return packageDirectory +} + +async function removeStaging(directory: string, cause: unknown): Promise { + try { + await rm(directory, { recursive: true, force: true }) + } catch (cleanupError) { + throw new AggregateError([cause, cleanupError], `failed to clean repository staging directory ${directory}`) + } + throw cause +} + +/** + * Persistent exact-specifier package cache backed by bundled pnpm. + * + * One isolated project contains one dependency named `repository`. A successful + * install is atomically renamed into its SHA-256 key, so failed installs never + * become cache hits. The exact specifier is immutable: callers change the + * specifier (normally its Git ref) to request another generation. + */ +export class RepositoryCache { + /** Absolute directory containing immutable repository cache entries. */ + readonly directory: string + + private readonly tasks = new Map>() + + /** + * @param directory - caller-owned persistent cache root. + * @param install - isolated package installation boundary; defaults to the bundled pnpm. + */ + constructor(directory: string, private readonly install: RepositoryInstall = installWithBundledPnpm) { + this.directory = resolve(directory) + } + + /** + * Resolve one package-manager-native dependency specifier to its installed package directory. + * @param specifier - exact immutable dependency specifier used as the permanent cache identity. + * @returns the installed `repository` dependency directory. + * @throws when the specifier is empty/padded, installation fails, or a published cache entry is corrupt. + */ + resolve(specifier: string): Promise { + if (!specifier || specifier.trim() !== specifier) { + throw new TypeError('repository specifier must be a non-empty unpadded string') + } + const existing = this.tasks.get(specifier) + if (existing) return existing + const task = this.resolveUncached(specifier).finally(() => { + if (this.tasks.get(specifier) === task) this.tasks.delete(specifier) + }) + this.tasks.set(specifier, task) + return task + } + + private async resolveUncached(specifier: string): Promise { + const finalDirectory = join(this.directory, cacheKey(specifier)) + const cached = await readCached(finalDirectory, specifier) + if (cached) return cached + + await mkdir(this.directory, { recursive: true }) + const staging = await mkdtemp(join(this.directory, '.repository-')) + try { + await writeFile(join(staging, 'package.json'), `${JSON.stringify({ + name: 'cordis-repository-cache-entry', + private: true, + version: '0.0.0', + packageManager: `pnpm@${BUNDLED_PNPM_VERSION}`, + dependencies: { [DEPENDENCY_NAME]: specifier }, + }, undefined, 2)}\n`) + await writeFile(join(staging, 'pnpm-workspace.yaml'), [ + 'packages: []', + 'dangerouslyAllowAllBuilds: true', + '', + ].join('\n')) + await this.install(staging) + const packageDirectory = join(staging, 'node_modules', DEPENDENCY_NAME) + const packageStat = await stat(packageDirectory) + if (!packageStat.isDirectory()) throw new Error(`installed repository is not a directory: ${packageDirectory}`) + await writeFile(join(staging, MARKER_NAME), `${JSON.stringify({ specifier })}\n`) + try { + await rename(staging, finalDirectory) + } catch (error) { + const winner = await readCached(finalDirectory, specifier) + if (!winner) throw error + await rm(staging, { recursive: true, force: true }) + return winner + } + } catch (error) { + return removeStaging(staging, new Error(`failed to prepare repository ${JSON.stringify(specifier)}`, { cause: error })) + } + return (await readCached(finalDirectory, specifier))! + } +} diff --git a/vendor/loader/tsdown.config.ts b/vendor/loader/tsdown.config.ts new file mode 100644 index 0000000000..75e627cdd2 --- /dev/null +++ b/vendor/loader/tsdown.config.ts @@ -0,0 +1,18 @@ +import { defineConfig } from 'tsdown' + +/** Keep the browser-reachable Loader entry separate from the Node-only repository cache. */ +const shared = { + outDir: 'lib', + format: ['esm'], + platform: 'node', + target: 'es2024', + fixedExtension: false, + outputOptions: { codeSplitting: false }, + dts: false, + clean: false, +} as const + +export default defineConfig([ + { ...shared, entry: ['lib/types/index.js'] }, + { ...shared, entry: ['lib/types/repository.js'] }, +])