mirror of
https://github.com/deepseek-ai/deepseek-harness
synced 2026-08-15 21:04:50 +00:00
fix(i18n): harden prompt response handling
This commit is contained in:
@@ -2,5 +2,5 @@
|
||||
# side as of the last confirmed-consistent state. Both languages carry equal authority;
|
||||
# after editing either side, bring the other along and re-record with:
|
||||
# pnpm run verify-translation-pairing --write
|
||||
2026-07-23-translation-prompt-v4-contract.md: b288f14e784687ab194392cd94a872aa5324a812
|
||||
2026-07-23-translation-prompt-v4-contract.zh.md: 3b1bcba90dc6d4b2eea1e9b2c0af65db3ca0969b
|
||||
2026-07-23-translation-prompt-v4-contract.md: 3e1e51797aa3463c8db24d8657120434e6822789
|
||||
2026-07-23-translation-prompt-v4-contract.zh.md: 161d2b6cf3bd3499e3c505a178da40ce577ca797
|
||||
|
||||
@@ -10,15 +10,15 @@ Automated counterpart generation needs a stable prompt that reproduces the regis
|
||||
|
||||
## Decision
|
||||
|
||||
The committed [translation prompt](../../../../docs/i18n/translation-prompt.md) is the calibrated pipeline asset. Its renderer injects only the source language, target language, and current [terminology table](../../../../docs/i18n/terminology.md). The request assembler retains the source basename outside the model-visible prompt and places each reviewed whole-document pair into one bare-text user/assistant example turn before the real source document. The template may carry model-specific calibration rules, but those rules remain subordinate to the repository's binding pairing, terminology, structure, and emphasis contracts.
|
||||
The committed [translation prompt](../../../../docs/i18n/translation-prompt.md) is the calibrated pipeline asset. Its renderer injects only the source language, target language, and current [terminology table](../../../../docs/i18n/terminology.md), and rejects unknown, missing, or malformed placeholder syntax before assembling a request. The request assembler retains the source basename outside the model-visible prompt and places each reviewed whole-document pair into one bare-text user/assistant example turn before the real source document. The template may carry model-specific calibration rules, but those rules remain subordinate to the repository's binding pairing, terminology, structure, and emphasis contracts.
|
||||
|
||||
The response has three ordered top-level sections: `translation`, `review`, and `final`. The response consumer derives the target basename from the retained source context and mechanically inserts or corrects the language switcher in `final`. The parser requires each section exactly once, rejects content outside the envelope, and tolerates one outer `xml` Markdown fence because models sometimes echo the prompt's example fence.
|
||||
The response has three ordered top-level sections: `translation`, `review`, and `final`. The response consumer derives the target basename from the retained source context, preserves optional leading YAML frontmatter, and mechanically inserts or corrects the language switcher after the first H1 in `final`. The parser requires each section exactly once, rejects content outside the envelope, and tolerates one outer `xml` Markdown fence because models sometimes echo the prompt's example fence.
|
||||
|
||||
## Response framing
|
||||
|
||||
Section delimiter lines are reserved by the wire format. When a Markdown body line consists of a delimiter tag, possibly preceded by backslashes, the serializer and model add one leading backslash; the parser removes exactly one. This count-preserving escape round-trips both a literal delimiter and an already escaped delimiter without changing inline tag mentions.
|
||||
|
||||
The executable contract lives in [the renderer, request assembler, parser, and response consumer](../../../../scripts/translation-prompt.ts). Unit tests cover both directions, request order, target-path validation, strict section order and cardinality, fenced responses, inline tag mentions, delimiter lines inside Markdown bodies, and new-pair switcher correction. A keyless subprocess snapshot pins the assembled prompt and five reviewed example turns together with a recorded response consumed through the target-path correction.
|
||||
The executable contract lives in [the renderer, request assembler, parser, and response consumer](../../../../scripts/translation-prompt.ts). Unit tests cover both directions, request order, placeholder validation, target-path validation, strict section order and cardinality, fenced responses, inline tag mentions, delimiter lines inside Markdown bodies, and frontmatter-preserving new-pair switcher correction. A keyless subprocess snapshot pins the assembled prompt and five reviewed example turns together with a frontmatter-bearing recorded response consumed through the target-path correction.
|
||||
|
||||
## Alternatives considered
|
||||
|
||||
|
||||
@@ -10,15 +10,15 @@ Status: implemented
|
||||
|
||||
## 决策
|
||||
|
||||
提交入库的[翻译提示词](../../../../docs/i18n/translation-prompt.md)是经过校准的流水线资源。其渲染器仅注入源语言、目标语言和当前[术语表](../../../../docs/i18n/terminology.md)。请求组装器在模型可见的提示词之外保留源文件基本名,并在真正的源文档之前,将每组经评审的整篇文档对编排为一个纯文本 user/assistant 示例轮次。模板可以包含针对特定模型的校准规则,但这些规则必须服从仓库中具约束力的配对、术语、结构与强调格式契约。
|
||||
提交入库的[翻译提示词](../../../../docs/i18n/translation-prompt.md)是经过校准的流水线资源。其渲染器仅注入源语言、目标语言和当前[术语表](../../../../docs/i18n/terminology.md),并在组装请求前拒绝未知、缺失或语法格式错误的占位符。请求组装器在模型可见的提示词之外保留源文件基本名,并在真正的源文档之前,将每组经评审的整篇文档对编排为一个纯文本 user/assistant 示例轮次。模板可以包含针对特定模型的校准规则,但这些规则必须服从仓库中具约束力的配对、术语、结构与强调格式契约。
|
||||
|
||||
响应包含三个有序的顶层分段:`translation`、`review` 和 `final`。响应消费方根据保留的源文件上下文推导目标文件基本名,并以机械方式在 `final` 中插入或校正语言切换行。解析器要求每个分段恰好出现一次,拒绝封套之外的内容,并允许响应最外层有一层 `xml` Markdown 围栏,因为模型有时会照抄提示词中的示例围栏。
|
||||
响应包含三个有序的顶层分段:`translation`、`review` 和 `final`。响应消费方根据保留的源文件上下文推导目标文件基本名,保留文件开头可选的 YAML frontmatter,并以机械方式在 `final` 中第一个 H1 之后插入或校正语言切换行。解析器要求每个分段恰好出现一次,拒绝封套之外的内容,并允许响应最外层有一层 `xml` Markdown 围栏,因为模型有时会照抄提示词中的示例围栏。
|
||||
|
||||
## 响应封装格式
|
||||
|
||||
分段定界行由协议格式(wire format)保留。当 Markdown 正文中的某一行仅包含定界标签(前面可以带反斜杠)时,序列化器和模型会在行首再添加一个反斜杠;解析器则只移除一个。这种保留计数的转义方式让字面量定界标签与已转义的定界标签都能无损往返,同时不会改动行内提及的标签。
|
||||
|
||||
可执行契约由[渲染器、请求组装器、解析器和响应消费方](../../../../scripts/translation-prompt.ts)实现。单元测试覆盖两个翻译方向、请求顺序、目标路径校验、严格的分段顺序与数量约束、带围栏的响应、行内提及标签、Markdown 正文中的定界行,以及新配对的语言切换行校正。一个无密钥子进程快照锁定组装后的提示词、五个经评审的示例轮次,以及录制响应经目标路径校正后的消费结果。
|
||||
可执行契约由[渲染器、请求组装器、解析器和响应消费方](../../../../scripts/translation-prompt.ts)实现。单元测试覆盖两个翻译方向、请求顺序、占位符校验、目标路径校验、严格的分段顺序与数量约束、带围栏的响应、行内提及标签、Markdown 正文中的定界行,以及保留 YAML frontmatter 的新配对语言切换行校正。一个无密钥子进程快照锁定组装后的提示词、五个经评审的示例轮次,以及带 YAML frontmatter 的录制响应经目标路径校正后的消费结果。
|
||||
|
||||
## 考虑过的替代方案
|
||||
|
||||
|
||||
@@ -1,4 +1,8 @@
|
||||
<translation>
|
||||
---
|
||||
layout: doc
|
||||
---
|
||||
|
||||
# 快照说明
|
||||
|
||||
agent(智能体)执行一个步骤。
|
||||
@@ -9,6 +13,10 @@ agent(智能体)执行一个步骤。
|
||||
</review>
|
||||
|
||||
<final>
|
||||
---
|
||||
layout: doc
|
||||
---
|
||||
|
||||
# 快照说明
|
||||
|
||||
agent(智能体)执行一个步骤。
|
||||
|
||||
@@ -1,3 +1,7 @@
|
||||
---
|
||||
layout: doc
|
||||
---
|
||||
|
||||
# Snapshot note
|
||||
|
||||
The agent performs one step.
|
||||
|
||||
@@ -48,13 +48,13 @@
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "# Snapshot note\n\nThe agent performs one step.\n"
|
||||
"content": "---\nlayout: doc\n---\n\n# Snapshot note\n\nThe agent performs one step.\n"
|
||||
}
|
||||
]
|
||||
},
|
||||
"response": {
|
||||
"translation": "# 快照说明\n\nagent(智能体)执行一个步骤。",
|
||||
"translation": "---\nlayout: doc\n---\n\n# 快照说明\n\nagent(智能体)执行一个步骤。",
|
||||
"review": "- 无修正",
|
||||
"final": "# 快照说明\n\n[English](snapshot-note.md) | 中文\n\nagent(智能体)执行一个步骤。\n"
|
||||
"final": "---\nlayout: doc\n---\n\n# 快照说明\n\n[English](snapshot-note.md) | 中文\n\nagent(智能体)执行一个步骤。\n"
|
||||
}
|
||||
}
|
||||
|
||||
@@ -38,6 +38,17 @@ describe('translation prompt rendering', () => {
|
||||
expect(() => renderTranslationPrompt(missing, { sourceLanguage: 'English', sourceFilename: 'guide.md', terminology })).toThrow(/required placeholder/)
|
||||
})
|
||||
|
||||
it('rejects unmatched placeholder delimiters', () => {
|
||||
for (const delimiter of ['{{', '}}']) {
|
||||
const malformed = document.replace('Your task is to translate', `Your task ${delimiter} is to translate`)
|
||||
expect(() => renderTranslationPrompt(malformed, {
|
||||
sourceLanguage: 'English',
|
||||
sourceFilename: 'guide.md',
|
||||
terminology,
|
||||
})).toThrow(/malformed placeholder syntax/)
|
||||
}
|
||||
})
|
||||
|
||||
it('assembles bare few-shot turns before the real source document', () => {
|
||||
const request = renderTranslationRequest(document, {
|
||||
sourceLanguage: 'English',
|
||||
@@ -127,6 +138,46 @@ describe('translation response sections', () => {
|
||||
].join('\n'))
|
||||
})
|
||||
|
||||
it('preserves YAML frontmatter before inserting the target switcher', () => {
|
||||
const response = renderTranslationResponse({
|
||||
translation: '# 指南\n\n初稿。',
|
||||
review: '- 无修正',
|
||||
final: [
|
||||
'---',
|
||||
'layout: home',
|
||||
'---',
|
||||
'',
|
||||
'# 指南',
|
||||
'',
|
||||
'定稿。',
|
||||
].join('\n'),
|
||||
})
|
||||
expect(consumeTranslationResponse(response, { sourceLanguage: 'English', sourceFilename: 'guide.md' }).final).toBe([
|
||||
'---',
|
||||
'layout: home',
|
||||
'---',
|
||||
'',
|
||||
'# 指南',
|
||||
'',
|
||||
'[English](guide.md) | 中文',
|
||||
'',
|
||||
'定稿。',
|
||||
'',
|
||||
].join('\n'))
|
||||
})
|
||||
|
||||
it('rejects unterminated YAML frontmatter before the target H1', () => {
|
||||
const response = renderTranslationResponse({
|
||||
translation: '# 指南\n\n初稿。',
|
||||
review: '- 无修正',
|
||||
final: '---\nlayout: home\n\n# 指南\n\n定稿。',
|
||||
})
|
||||
expect(() => consumeTranslationResponse(response, {
|
||||
sourceLanguage: 'English',
|
||||
sourceFilename: 'guide.md',
|
||||
})).toThrow(/unterminated YAML frontmatter/)
|
||||
})
|
||||
|
||||
it('rejects a source filename that contradicts the translation direction', () => {
|
||||
expect(() => renderTranslationPrompt(document, {
|
||||
sourceLanguage: 'Chinese',
|
||||
|
||||
@@ -119,6 +119,10 @@ export function renderTranslationPrompt(document: string, input: TranslationProm
|
||||
terminology: input.terminology,
|
||||
}
|
||||
const template = extractTranslationPrompt(document)
|
||||
const placeholderFreeTemplate = template.replace(PLACEHOLDER, '')
|
||||
if (placeholderFreeTemplate.includes('{{') || placeholderFreeTemplate.includes('}}')) {
|
||||
throw new Error('translation prompt: template contains malformed placeholder syntax')
|
||||
}
|
||||
const names = [...template.matchAll(PLACEHOLDER)].map(match => match[1] ?? '')
|
||||
const unknown = names.filter(name => !TRANSLATION_PROMPT_PLACEHOLDERS.includes(name as TranslationPromptPlaceholder))
|
||||
if (unknown.length > 0) throw new Error(`translation prompt: unsupported placeholder(s): ${[...new Set(unknown)].join(', ')}`)
|
||||
@@ -215,16 +219,24 @@ export function parseTranslationResponse(text: string): TranslationResponse {
|
||||
function correctLanguageSwitcher(markdown: string, switcher: string): string {
|
||||
const lines = markdown.replaceAll('\r\n', '\n').split('\n')
|
||||
while (lines.at(-1) === '') lines.pop()
|
||||
if (!/^#\s+\S/.test(lines[0] ?? '')) {
|
||||
|
||||
let headingIndex = 0
|
||||
if (lines[0] === '---') {
|
||||
const frontmatterEnd = lines.indexOf('---', 1)
|
||||
if (frontmatterEnd === -1) throw new Error('translation response: final document has unterminated YAML frontmatter')
|
||||
headingIndex = frontmatterEnd + 1
|
||||
while (lines[headingIndex] === '') headingIndex++
|
||||
}
|
||||
if (!/^#\s+\S/.test(lines[headingIndex] ?? '')) {
|
||||
throw new Error('translation response: final document must start with an H1 heading')
|
||||
}
|
||||
|
||||
let contentStart = 1
|
||||
let contentStart = headingIndex + 1
|
||||
while (lines[contentStart] === '') contentStart++
|
||||
if (LANGUAGE_SWITCHER.test(lines[contentStart] ?? '')) contentStart++
|
||||
while (lines[contentStart] === '') contentStart++
|
||||
|
||||
const output = [lines[0] as string, '', switcher]
|
||||
const output = [...lines.slice(0, headingIndex), lines[headingIndex] as string, '', switcher]
|
||||
const content = lines.slice(contentStart)
|
||||
if (content.length > 0) output.push('', ...content)
|
||||
return `${output.join('\n')}\n`
|
||||
|
||||
@@ -71,8 +71,18 @@ try {
|
||||
throw new Error('reviewed examples are not assembled as system, example pairs, then source')
|
||||
}
|
||||
const consumed = consumeTranslationResponse(recordedResponse, englishInput)
|
||||
if (consumed.final.split('\n')[2] !== '[English](snapshot-note.md) | 中文') {
|
||||
throw new Error('recorded new-pair response does not receive the canonical target switcher')
|
||||
const expectedFinalPrefix = [
|
||||
'---',
|
||||
'layout: doc',
|
||||
'---',
|
||||
'',
|
||||
'# 快照说明',
|
||||
'',
|
||||
'[English](snapshot-note.md) | 中文',
|
||||
'',
|
||||
].join('\n')
|
||||
if (!consumed.final.startsWith(expectedFinalPrefix)) {
|
||||
throw new Error('recorded frontmatter response does not preserve metadata and receive the canonical target switcher')
|
||||
}
|
||||
|
||||
if (mode === '--snapshot') {
|
||||
|
||||
Reference in New Issue
Block a user