feat: GNS-2.1 close-loop enforcement — Issue lifecycle audit & PQS measurement
Based on audit of UniqueSoft/FutureWork (40 issues, 38 commits): - PQS baseline was 0.47 (Critical) due to agents not updating body checkboxes and not closing completed issues - After closing 3 orphans (#25,#38,#39) PQS improved to 0.66 (Adequate) Changes: - New rule: .kilo/rules/issue-close-loop.md — mandatory body checkbox updates + auto-close on completion - New script: scripts/issue-health-check.py — PQS calculator with --fix mode for auto-syncing checkboxes and closing orphans - Updated lead-developer & the-fixer: On Exit now includes checkbox update + auto-close steps, GNS_EVENT footer includes close_loop - Updated gns-agent-protocol.md: Exit Protocol expanded with 9 steps, close_loop field added to machine-readable footer - Updated orchestrator.md: Close-Loop Gate #6 added — mandatory checkbox verification before transitioning to next agent Target: PQS ≥ 0.85 after full agent adoption (Phases 2-3) Implements: UniqueSoft/APAW#129
This commit is contained in:
@@ -67,9 +67,11 @@ Tier 1 (Task Agent / Orchestrator-Mediated Cascade)
|
||||
- Do NOT call `task` tool directly (Tier 1)
|
||||
|
||||
### On Exit (MANDATORY)
|
||||
1. Update labels if needed (quality::*, phase::*)
|
||||
2. Post comment with result + GNS_EVENT footer
|
||||
3. Include `next_agent` recommendation
|
||||
1. **Update issue body checkboxes** — mark `[ ]` → `[x]` for completed criteria in the issue body via `PATCH /repos/{owner}/{repo}/issues/{n}`. NEVER only update checkboxes in comments — the issue body is the single source of truth.
|
||||
2. **Close issue if all checkboxes done** — if `all_checkboxes_done(body) == True`, close via `PATCH` with `{"state": "closed"}` and add `status::done` label.
|
||||
3. Update labels if needed (quality::*, phase::*)
|
||||
4. Post comment with result + GNS_EVENT footer (must include `close_loop` field)
|
||||
5. Include `next_agent` recommendation
|
||||
|
||||
### GNS Event Footer Template
|
||||
```markdown
|
||||
@@ -89,6 +91,13 @@ Tier 1 (Task Agent / Orchestrator-Mediated Cascade)
|
||||
},
|
||||
"next_agent": "{next_agent}",
|
||||
"estimated_next_tokens": {estimate},
|
||||
"close_loop": {
|
||||
"issue": {issue_number},
|
||||
"checkboxes_total": {total},
|
||||
"checkboxes_checked": {checked},
|
||||
"checkboxes_updated_in_body": true|false,
|
||||
"issue_closed": true|false
|
||||
},
|
||||
"timestamp": "{iso8601}"
|
||||
} -->
|
||||
```
|
||||
|
||||
@@ -144,11 +144,23 @@ Process manager. Distributes tasks between agents, monitors statuses, and switch
|
||||
|
||||
5. **Priorities:** Always check if the task is blocked by other Issues. If yes — suspend work and notify.
|
||||
|
||||
6. **Finalization:** Only you have the right to give Release Manager the command via Task tool with `subagent_type: "release-manager"` to prepare a release after receiving confirmation from Evaluator.
|
||||
6. **Close-Loop Gate (MANDATORY before transitioning to next agent):**
|
||||
After any agent reports task completion, the orchestrator MUST verify:
|
||||
1. Read issue body → count checkboxes (`- [ ]` = unchecked, `- [x]` = checked)
|
||||
2. If all checkboxes checked → auto-close issue (`PATCH {"state": "closed"}`) + add label `status::done`
|
||||
3. If checkboxes remain unchecked but agent claims completion → flag `quality::needs-fix`, return to agent
|
||||
4. If agent posted checkboxes in comments but not in body → sync from comments, then re-check
|
||||
5. Log result to `.kilo/logs/close-loop-audits.jsonl`:
|
||||
```jsonl
|
||||
{"ts":"2026-06-06T14:00:00Z","issue":42,"checkboxes_total":6,"checkboxes_checked":6,"auto_closed":true,"agent":"lead-developer"}
|
||||
```
|
||||
This is NOT optional. PQS (Prompt Quality Score) depends on correct close-loop behavior.
|
||||
|
||||
7. **Communication:** Your messages should be brief commands: "To: [Name]. Task: [ essence]. Context: [file reference]".
|
||||
7. **Finalization:** Only you have the right to give Release Manager the command via Task tool with `subagent_type: "release-manager"` to prepare a release after receiving confirmation from Evaluator.
|
||||
|
||||
8. **Context Budget Governance:**
|
||||
8. **Communication:** Your messages should be brief commands: "To: [Name]. Task: [ essence]. Context: [file reference]".
|
||||
|
||||
9. **Context Budget Governance:**
|
||||
Before spawning ANY agent, the orchestrator MUST calculate and enforce context window budget:
|
||||
- Read issue body → extract checkpoint YAML
|
||||
- If checkpoint `consumed` > 80% of `total`:
|
||||
|
||||
@@ -67,9 +67,11 @@ Tier 1 (Task Agent / Orchestrator-Mediated Cascade)
|
||||
- Do NOT call `task` tool directly (Tier 1)
|
||||
|
||||
### On Exit (MANDATORY)
|
||||
1. Update labels if needed (quality::*, phase::*)
|
||||
2. Post comment with result + GNS_EVENT footer
|
||||
3. Include `next_agent` recommendation
|
||||
1. **Update issue body checkboxes** — mark `[ ]` → `[x]` for completed criteria in the issue body via `PATCH /repos/{owner}/{repo}/issues/{n}`. NEVER only update checkboxes in comments — the issue body is the single source of truth.
|
||||
2. **Close issue if all checkboxes done** — if `all_checkboxes_done(body) == True`, close via `PATCH` with `{"state": "closed"}` and add `status::done` label.
|
||||
3. Update labels if needed (quality::*, phase::*)
|
||||
4. Post comment with result + GNS_EVENT footer (must include `close_loop` field)
|
||||
5. Include `next_agent` recommendation
|
||||
|
||||
### GNS Event Footer Template
|
||||
```markdown
|
||||
@@ -89,6 +91,13 @@ Tier 1 (Task Agent / Orchestrator-Mediated Cascade)
|
||||
},
|
||||
"next_agent": "{next_agent}",
|
||||
"estimated_next_tokens": {estimate},
|
||||
"close_loop": {
|
||||
"issue": {issue_number},
|
||||
"checkboxes_total": {total},
|
||||
"checkboxes_checked": {checked},
|
||||
"checkboxes_updated_in_body": true|false,
|
||||
"issue_closed": true|false
|
||||
},
|
||||
"timestamp": "{iso8601}"
|
||||
} -->
|
||||
```
|
||||
|
||||
@@ -30,11 +30,15 @@ During work:
|
||||
|
||||
Every agent MUST execute before terminating:
|
||||
|
||||
1. **Write Result Comment**: Structured markdown with machine-readable footer
|
||||
2. **Update Checkpoint**: Patch issue body with new checkpoint YAML
|
||||
3. **Update Labels**: Reflect new phase, quality, budget state
|
||||
4. **Set Assignee**: Hand off to next agent or self
|
||||
5. **Log Cascade**: If subagents were spawned, include cascade table
|
||||
1. **Update Issue Body Checkboxes**: Mark `[ ]` → `[x]` for each completed acceptance criterion in the issue body via `PATCH /repos/{owner}/{repo}/issues/{n}`. The issue body is the single source of truth — do NOT only update checkboxes in comments.
|
||||
2. **Auto-Close If Complete**: If `all_checkboxes_done(body) == True`, close the issue (`{"state": "closed"}`) and add `status::done` label. Post comment: `## ✅ Issue Auto-Closed — All acceptance criteria met`.
|
||||
3. **Partial Completion**: If some checkboxes remain, add `status::partial` label and post comment listing which criteria were completed and which remain.
|
||||
4. **Write Result Comment**: Structured markdown with machine-readable footer
|
||||
5. **Update Checkpoint**: Patch issue body with new checkpoint YAML
|
||||
6. **Update Labels**: Reflect new phase, quality, budget state
|
||||
7. **Set Assignee**: Hand off to next agent or self
|
||||
8. **Log Cascade**: If subagents were spawned, include cascade table
|
||||
9. **Include `close_loop` in GNS_EVENT footer**: Must contain `{issue, checkboxes_total, checkboxes_checked, checkboxes_updated_in_body, issue_closed}`
|
||||
|
||||
## Comment Format
|
||||
|
||||
@@ -95,6 +99,13 @@ Every agent MUST execute before terminating:
|
||||
"cascade_log": [
|
||||
{"agent": "history-miner", "task": "git search", "tokens": 1200, "verdict": "pass"}
|
||||
],
|
||||
"close_loop": {
|
||||
"issue": 42,
|
||||
"checkboxes_total": 6,
|
||||
"checkboxes_checked": 6,
|
||||
"checkboxes_updated_in_body": true,
|
||||
"issue_closed": true
|
||||
},
|
||||
"next_agent": "agent-architect",
|
||||
"estimated_next_tokens": 3000,
|
||||
"timestamp": "2026-05-08T20:00:00Z"
|
||||
|
||||
138
.kilo/rules/issue-close-loop.md
Normal file
138
.kilo/rules/issue-close-loop.md
Normal file
@@ -0,0 +1,138 @@
|
||||
# Issue Close-Loop Enforcement (GNS-2.1)
|
||||
|
||||
Every agent that completes a task MUST update the issue body checkboxes and close the issue if all criteria are met. Results in comments are NOT sufficient.
|
||||
|
||||
## Problem
|
||||
|
||||
Agents routinely:
|
||||
1. Post `[x]` checkboxes in **comments** but leave `[ ]` in the **issue body**
|
||||
2. Do not close issues after completing all acceptance criteria
|
||||
3. Do not map commits to specific acceptance criteria
|
||||
|
||||
This creates "orphan issues" — open but actually done.
|
||||
|
||||
## Core Rule: Body First, Comment Second
|
||||
|
||||
**The issue body is the single source of truth for task completion status.**
|
||||
|
||||
Comments describe WHAT was done. The body checkboxes show WHETHER it's done.
|
||||
|
||||
## Mandatory Exit Checklist
|
||||
|
||||
Every agent MUST execute these steps BEFORE posting the result comment:
|
||||
|
||||
### Step 1: Update Checkboxes in Issue Body
|
||||
|
||||
Use `update_issue_checkboxes` from `.kilo/shared/gitea-api.md`:
|
||||
|
||||
```python
|
||||
# Via shared API client
|
||||
from gitea_api import gitea_api, get_target_repo
|
||||
|
||||
issue = gitea_api(f"/repos/{target_repo}/issues/{issue_number}")
|
||||
body = issue['body']
|
||||
|
||||
# For each criterion the agent completed, update the body:
|
||||
# - [ ] Criterion text → - [x] Criterion text
|
||||
# Use precise text matching, NOT global regex replace.
|
||||
|
||||
gitea_api(f"/repos/{target_repo}/issues/{issue_number}", {"body": updated_body}, 'PATCH')
|
||||
```
|
||||
|
||||
**CRITICAL**: Only mark checkboxes that THIS agent actually completed. Do not mark checkboxes for work done by other agents unless verified.
|
||||
|
||||
### Step 2: Check If All Checkboxes Are Done
|
||||
|
||||
```python
|
||||
import re
|
||||
|
||||
def all_checkboxes_done(body):
|
||||
"""Return True if no unchecked checkboxes remain."""
|
||||
unchecked = re.findall(r'- \[ \]', body)
|
||||
unchecked += re.findall(r'\* \[ \]', body)
|
||||
return len(unchecked) == 0
|
||||
```
|
||||
|
||||
### Step 3: Auto-Close If Complete
|
||||
|
||||
If `all_checkboxes_done(body) == True`:
|
||||
1. Close the issue: `PATCH /repos/{owner}/{repo}/issues/{n}` with `{"state": "closed"}`
|
||||
2. Add comment: `## ✅ Issue Auto-Closed — All acceptance criteria met`
|
||||
3. Remove `status::in-progress` label, add `status::done` label
|
||||
|
||||
### Step 4: Partial Completion
|
||||
|
||||
If some checkboxes remain unchecked:
|
||||
1. Add label `status::partial` (if available) or keep `status::in-progress`
|
||||
2. Post comment listing which criteria were completed and which remain
|
||||
3. Recommend next agent for remaining work
|
||||
|
||||
## Commit-Criteria Mapping
|
||||
|
||||
Every commit from an agent MUST include a reference to which acceptance criteria it fulfills:
|
||||
|
||||
```
|
||||
feat: implement X endpoint
|
||||
|
||||
Implements: #42 criteria [1], [2]
|
||||
|
||||
- criterion 1 description
|
||||
- criterion 2 description
|
||||
```
|
||||
|
||||
This enables automated tracking of commit-to-criteria coverage.
|
||||
|
||||
## Prompt Quality Score (PQS)
|
||||
|
||||
Measured per repository per time period using `scripts/issue-health-check.py`:
|
||||
|
||||
```bash
|
||||
python3 scripts/issue-health-check.py --repo Owner/Repo [--since YYYY-MM-DD]
|
||||
```
|
||||
|
||||
### PQS Thresholds
|
||||
|
||||
| PQS | Rating | Action |
|
||||
|-----|--------|--------|
|
||||
| ≥ 0.85 | Excellent | No action needed |
|
||||
| 0.60–0.84 | Adequate | Review prompt quality |
|
||||
| 0.40–0.59 | Poor | Prompt optimization required |
|
||||
| < 0.40 | Critical | Immediate prompt rewrite |
|
||||
|
||||
## Issue Health Monitor
|
||||
|
||||
Automated periodic checks (every 24h via cron or manual):
|
||||
|
||||
| Check | Condition | Action |
|
||||
|-------|-----------|--------|
|
||||
| Orphan issue | Open > 7 days, all checkboxes in comments are `[x]` but body still has `[ ]` | Sync checkboxes to body, comment `🔧 Auto-synced checkboxes from comments` |
|
||||
| Stale issue | Open > 7 days, no comments in last 3 days | Add label `status::stale`, comment `⚠️ No activity for 7 days` |
|
||||
| Auto-close candidate | Open, all body checkboxes `[x]` | Close issue, comment `✅ Auto-closed: all criteria met` |
|
||||
| Unlinked commit | Agent commit without `Closes #` or `Implements:` | Post comment on likely issue with commit reference |
|
||||
|
||||
## GNS_EVENT Footer Addition
|
||||
|
||||
The GNS_EVENT footer MUST include checkbox state when an agent exits:
|
||||
|
||||
```html
|
||||
<!-- GNS_EVENT: {
|
||||
"type": "subagent_result",
|
||||
"agent": "lead-developer",
|
||||
...
|
||||
"close_loop": {
|
||||
"issue": 42,
|
||||
"checkboxes_total": 6,
|
||||
"checkboxes_checked": 6,
|
||||
"checkboxes_updated_in_body": true,
|
||||
"issue_closed": true
|
||||
}
|
||||
} -->
|
||||
```
|
||||
|
||||
## Prohibited Actions
|
||||
|
||||
- DO NOT post checklist results in comments WITHOUT also updating the issue body
|
||||
- DO NOT close an issue that has unchecked acceptance criteria
|
||||
- DO NOT mark a checkbox as done if you did not verify the work is complete
|
||||
- DO NOT skip the close-loop step when exiting an agent task
|
||||
- DO NOT use global regex `- [ ] → - [x]` — match specific criterion text only
|
||||
367
scripts/issue-health-check.py
Normal file
367
scripts/issue-health-check.py
Normal file
@@ -0,0 +1,367 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Issue Health Check & PQS Calculator (GNS-2.1)
|
||||
|
||||
Measures Prompt Quality Score (PQS) and detects orphan issues
|
||||
where work is done but checkboxes are not updated in the issue body.
|
||||
|
||||
Usage:
|
||||
python3 scripts/issue-health-check.py [--repo Owner/Repo] [--since YYYY-MM-DD] [--fix] [--verbose]
|
||||
|
||||
Options:
|
||||
--repo Target Gitea repo (default: auto-detect from git remote)
|
||||
--since Only analyze issues created after this date
|
||||
--fix Auto-fix: sync checkboxes from comments to body, auto-close completed issues
|
||||
--verbose Print detailed per-issue analysis
|
||||
|
||||
Environment:
|
||||
GITEA_API_URL — Gitea API base URL (default: https://git.softuniq.eu/api/v1)
|
||||
GITEA_TOKEN — API token (required)
|
||||
"""
|
||||
|
||||
import urllib.request
|
||||
import json
|
||||
import re
|
||||
import os
|
||||
import sys
|
||||
import subprocess
|
||||
from datetime import datetime, timedelta
|
||||
from collections import defaultdict
|
||||
|
||||
def get_target_repo():
|
||||
"""Detect target project from git remote."""
|
||||
try:
|
||||
result = subprocess.run(
|
||||
['git', 'remote', 'get-url', 'origin'],
|
||||
capture_output=True, text=True
|
||||
)
|
||||
remote_url = result.stdout.strip().rstrip('/')
|
||||
match = re.search(r'[:/]([^/]+/[^/]+?)(?:\.git)?$', remote_url)
|
||||
if match:
|
||||
return match.group(1)
|
||||
except Exception:
|
||||
pass
|
||||
return os.environ.get('GITEA_TARGET_REPO', 'UniqueSoft/APAW')
|
||||
|
||||
def gitea_api(path, data=None, method='GET', repo=None):
|
||||
"""Call Gitea API with error handling.
|
||||
|
||||
If repo is provided, path is treated as relative to /repos/{repo}/.
|
||||
If repo is empty string '', path is treated as absolute (already contains /repos/owner/repo).
|
||||
"""
|
||||
api_url = os.environ.get('GITEA_API_URL', 'https://git.softuniq.eu/api/v1')
|
||||
token = os.environ.get('GITEA_TOKEN', '')
|
||||
# Ensure path starts with / but doesn't double up
|
||||
if not path.startswith('/'):
|
||||
path = '/' + path
|
||||
|
||||
# If repo is explicitly '', the path already contains the full repo path
|
||||
# If repo is None, auto-detect and prepend
|
||||
if repo == '':
|
||||
# Path already contains the full repo path or is absolute
|
||||
url = f"{api_url}{path}"
|
||||
else:
|
||||
target_repo = repo or get_target_repo()
|
||||
url = f"{api_url}/repos/{target_repo}{path}"
|
||||
|
||||
headers = {'Content-Type': 'application/json'}
|
||||
if token:
|
||||
headers['Authorization'] = f'token {token}'
|
||||
req = urllib.request.Request(
|
||||
url,
|
||||
data=json.dumps(data).encode() if data else None,
|
||||
headers=headers,
|
||||
method=method
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=30) as r:
|
||||
response = r.read().decode()
|
||||
return json.loads(response) if response else {}
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code == 404:
|
||||
print(f" ⚠️ 404 Not Found: {url}")
|
||||
return [] if 'issues' in path or 'commits' in path else {}
|
||||
raise
|
||||
|
||||
def count_checkboxes(body):
|
||||
"""Count total checkboxes in issue body (checked + unchecked)."""
|
||||
if not body:
|
||||
return 0
|
||||
checked = len(re.findall(r'- \[x\]', body)) + len(re.findall(r'\* \[x\]', body))
|
||||
unchecked = len(re.findall(r'- \[ \]', body)) + len(re.findall(r'\* \[ \]', body))
|
||||
return checked + unchecked
|
||||
|
||||
def count_checked_in_body(body):
|
||||
"""Count checked checkboxes in issue body."""
|
||||
if not body:
|
||||
return 0
|
||||
return len(re.findall(r'- \[x\]', body)) + len(re.findall(r'\* \[x\]', body))
|
||||
|
||||
def count_unchecked_in_body(body):
|
||||
"""Count unchecked checkboxes in issue body."""
|
||||
if not body:
|
||||
return 0
|
||||
return len(re.findall(r'- \[ \]', body)) + len(re.findall(r'\* \[ \]', body))
|
||||
|
||||
def all_checkboxes_done(body):
|
||||
"""Return True if no unchecked checkboxes remain."""
|
||||
return count_unchecked_in_body(body) == 0
|
||||
|
||||
def get_comments(issue_number, repo=None):
|
||||
"""Get all comments for an issue."""
|
||||
target_repo = repo or get_target_repo()
|
||||
all_comments = []
|
||||
page = 1
|
||||
while True:
|
||||
try:
|
||||
comments = gitea_api(
|
||||
f"/repos/{target_repo}/issues/{issue_number}/comments?limit=50&page={page}",
|
||||
repo=''
|
||||
)
|
||||
except Exception:
|
||||
break
|
||||
if not comments:
|
||||
break
|
||||
all_comments.extend(comments)
|
||||
page += 1
|
||||
return all_comments
|
||||
|
||||
def find_checked_in_comments(comments):
|
||||
"""Find checkbox text that appears as [x] in comments but [ ] in body."""
|
||||
checked_texts = []
|
||||
for c in comments:
|
||||
body = c.get('body', '') or ''
|
||||
for match in re.finditer(r'- \[x\] (.+)', body):
|
||||
checked_texts.append(match.group(1).strip())
|
||||
return checked_texts
|
||||
|
||||
def sync_checkboxes_from_comments(issue_number, body, comments, repo=None):
|
||||
"""Sync [x] checkboxes from comments into the issue body."""
|
||||
target_repo = repo or get_target_repo()
|
||||
checked_in_comments = find_checked_in_comments(comments)
|
||||
updated = body
|
||||
sync_count = 0
|
||||
for text in checked_in_comments:
|
||||
# Escape special regex chars in the text
|
||||
escaped = re.escape(text)
|
||||
pattern = f'- \\[ \\] {escaped}'
|
||||
replacement = f'- [x] {text}'
|
||||
new_body = re.sub(pattern, replacement, updated, count=1)
|
||||
if new_body != updated:
|
||||
updated = new_body
|
||||
sync_count += 1
|
||||
if sync_count > 0:
|
||||
try:
|
||||
gitea_api(
|
||||
f"/repos/{target_repo}/issues/{issue_number}",
|
||||
{"body": updated},
|
||||
method='PATCH',
|
||||
repo=''
|
||||
)
|
||||
except Exception as e:
|
||||
print(f" ❌ Failed to sync checkboxes for #{issue_number}: {e}")
|
||||
return 0
|
||||
return sync_count
|
||||
|
||||
def calculate_pqs(repo, since_date=None):
|
||||
"""Calculate Prompt Quality Score for a repository."""
|
||||
# Fetch all issues (open + closed)
|
||||
open_issues = gitea_api(f"/repos/{repo}/issues?state=open&limit=50", repo='')
|
||||
closed_issues = gitea_api(f"/repos/{repo}/issues?state=closed&limit=50", repo='')
|
||||
all_issues = open_issues + closed_issues
|
||||
|
||||
if since_date:
|
||||
since = datetime.fromisoformat(since_date.replace('Z', '+00:00'))
|
||||
all_issues = [i for i in all_issues if datetime.fromisoformat(i['created_at'].replace('Z', '+00:00')) >= since]
|
||||
|
||||
total_checkboxes = 0
|
||||
checked_in_body = 0
|
||||
closed_correctly = 0
|
||||
total_with_criteria = 0
|
||||
orphan_issues = []
|
||||
|
||||
for i in all_issues:
|
||||
body = i.get('body', '') or ''
|
||||
cb_total = count_checkboxes(body)
|
||||
cb_checked = count_checked_in_body(body)
|
||||
|
||||
if cb_total > 0:
|
||||
total_checkboxes += cb_total
|
||||
checked_in_body += cb_checked
|
||||
total_with_criteria += 1
|
||||
if i['state'] == 'closed' and all_checkboxes_done(body):
|
||||
closed_correctly += 1
|
||||
if i['state'] == 'open' and all_checkboxes_done(body):
|
||||
orphan_issues.append({
|
||||
'number': i['number'],
|
||||
'title': i['title'],
|
||||
'reason': 'All checkboxes done but issue is open'
|
||||
})
|
||||
elif cb_total == 0 and i['state'] == 'open':
|
||||
# Skip orphan detection for issues without criteria in non-verbose mode
|
||||
# (would require API call per issue — too slow for large repos)
|
||||
pass
|
||||
|
||||
# Count agent commits with references
|
||||
try:
|
||||
commits = gitea_api(f"/repos/{repo}/commits?limit=100", repo='')
|
||||
agent_commits = [c for c in commits if c.get('commit', {}).get('author', {}).get('name') == 'Deploy Bot']
|
||||
commits_with_ref = sum(
|
||||
1 for c in agent_commits
|
||||
if 'Closes #' in c.get('commit', {}).get('message', '') or 'Implements:' in c.get('commit', {}).get('message', '')
|
||||
)
|
||||
total_agent_commits = max(len(agent_commits), 1)
|
||||
except Exception:
|
||||
commits_with_ref = 0
|
||||
total_agent_commits = 1
|
||||
|
||||
checkbox_score = checked_in_body / max(total_checkboxes, 1)
|
||||
close_score = closed_correctly / max(total_with_criteria, 1)
|
||||
commit_score = commits_with_ref / max(total_agent_commits, 1)
|
||||
pqs = round(checkbox_score * 0.4 + close_score * 0.4 + commit_score * 0.2, 2)
|
||||
|
||||
return {
|
||||
'pqs': pqs,
|
||||
'checkbox_score': round(checkbox_score, 2),
|
||||
'close_score': round(close_score, 2),
|
||||
'commit_score': round(commit_score, 2),
|
||||
'total_issues': len(all_issues),
|
||||
'total_checkboxes': total_checkboxes,
|
||||
'checked_in_body': checked_in_body,
|
||||
'total_with_criteria': total_with_criteria,
|
||||
'closed_correctly': closed_correctly,
|
||||
'agent_commits': total_agent_commits,
|
||||
'commits_with_ref': commits_with_ref,
|
||||
'orphan_issues': orphan_issues,
|
||||
}
|
||||
|
||||
def rate_pqs(pqs):
|
||||
"""Rate PQS score."""
|
||||
if pqs >= 0.85:
|
||||
return 'Excellent ✅'
|
||||
elif pqs >= 0.60:
|
||||
return 'Adequate 🟡'
|
||||
elif pqs >= 0.40:
|
||||
return 'Poor 🟠'
|
||||
else:
|
||||
return 'Critical ❌'
|
||||
|
||||
def main():
|
||||
import argparse
|
||||
parser = argparse.ArgumentParser(description='Issue Health Check & PQS Calculator')
|
||||
parser.add_argument('--repo', default=None, help='Target repo (default: auto-detect)')
|
||||
parser.add_argument('--since', default=None, help='Only analyze issues after this date (YYYY-MM-DD)')
|
||||
parser.add_argument('--fix', action='store_true', help='Auto-fix: sync checkboxes, auto-close completed issues')
|
||||
parser.add_argument('--verbose', action='store_true', help='Print detailed per-issue analysis')
|
||||
args = parser.parse_args()
|
||||
|
||||
repo = args.repo or get_target_repo()
|
||||
print(f"🔍 Analyzing repository: {repo}")
|
||||
print(f"{'=' * 60}")
|
||||
|
||||
# Calculate PQS
|
||||
result = calculate_pqs(repo, args.since)
|
||||
|
||||
print(f"\n📊 Prompt Quality Score (PQS)")
|
||||
print(f"{'=' * 60}")
|
||||
print(f"PQS: {result['pqs']} / 1.00 → {rate_pqs(result['pqs'])}")
|
||||
print(f"Checkbox Score: {result['checkbox_score']} ({result['checked_in_body']}/{result['total_checkboxes']} checked in body)")
|
||||
print(f"Close Score: {result['close_score']} ({result['closed_correctly']}/{result['total_with_criteria']} closed correctly)")
|
||||
print(f"Commit Score: {result['commit_score']} ({result['commits_with_ref']}/{result['agent_commits']} agent commits referenced)")
|
||||
print(f"Total Issues: {result['total_issues']}")
|
||||
|
||||
# Orphan issues
|
||||
if result['orphan_issues']:
|
||||
print(f"\n⚠️ Orphan Issues ({len(result['orphan_issues'])})")
|
||||
print(f"{'=' * 60}")
|
||||
for o in result['orphan_issues']:
|
||||
print(f" #{o['number']}: {o['title'][:60]} — {o['reason']}")
|
||||
|
||||
if args.verbose:
|
||||
# Detailed per-issue analysis
|
||||
print(f"\n📋 Detailed Issue Analysis")
|
||||
print(f"{'=' * 60}")
|
||||
open_issues = gitea_api(f"/repos/{repo}/issues?state=open&limit=50", repo='')
|
||||
closed_issues = gitea_api(f"/repos/{repo}/issues?state=closed&limit=50", repo='')
|
||||
all_issues = open_issues + closed_issues
|
||||
|
||||
for i in all_issues[:30]: # Limit to 30 for readability
|
||||
body = i.get('body', '') or ''
|
||||
cb_total = count_checkboxes(body)
|
||||
cb_checked = count_checked_in_body(body)
|
||||
cb_unchecked = count_unchecked_in_body(body)
|
||||
state_emoji = '🟢' if i['state'] == 'closed' else '🔴'
|
||||
print(f" {state_emoji} #{i['number']}: {i['title'][:50]} | checkboxes: {cb_checked}/{cb_total} | state: {i['state']}")
|
||||
|
||||
if cb_total > 0 and cb_unchecked > 0 and i['state'] == 'open':
|
||||
# Check comments for checkboxes
|
||||
comments = get_comments(i['number'], repo=repo)
|
||||
comment_checks = 0
|
||||
for c in comments:
|
||||
cbody = c.get('body', '') or ''
|
||||
comment_checks += len(re.findall(r'- \[x\]', cbody))
|
||||
if comment_checks > 0:
|
||||
print(f" ⚠️ {comment_checks} checkboxes in comments but {cb_unchecked} unchecked in body")
|
||||
|
||||
if args.fix:
|
||||
print(f"\n🔧 Auto-Fix: Syncing checkboxes and closing completed issues")
|
||||
print(f"{'=' * 60}")
|
||||
open_issues = gitea_api(f"/repos/{repo}/issues?state=open&limit=50", repo='')
|
||||
fixed_count = 0
|
||||
closed_count = 0
|
||||
|
||||
for i in open_issues:
|
||||
body = i.get('body', '') or ''
|
||||
cb_unchecked = count_unchecked_in_body(body)
|
||||
|
||||
if cb_unchecked == 0 and count_checkboxes(body) > 0:
|
||||
# All checkboxes done — auto-close
|
||||
print(f" ✅ Auto-closing #{i['number']}: {i['title'][:50]}")
|
||||
gitea_api(f"/repos/{repo}/issues/{i['number']}", {"state": "closed"}, method='PATCH', repo='')
|
||||
gitea_api(
|
||||
f"/repos/{repo}/issues/{i['number']}/comments",
|
||||
{"body": "## ✅ Issue Auto-Closed\n\nAll acceptance criteria checkboxes are checked."},
|
||||
method='POST',
|
||||
repo=''
|
||||
)
|
||||
closed_count += 1
|
||||
continue
|
||||
|
||||
# Try to sync checkboxes from comments
|
||||
comments = get_comments(i['number'], repo=repo)
|
||||
synced = sync_checkboxes_from_comments(i['number'], body, comments, repo=repo)
|
||||
if synced > 0:
|
||||
print(f" 🔧 Synced {synced} checkboxes in #{i['number']}: {i['title'][:50]}")
|
||||
fixed_count += synced
|
||||
|
||||
# Re-check after sync
|
||||
updated_issue = gitea_api(f"/repos/{repo}/issues/{i['number']}", repo='')
|
||||
updated_body = updated_issue.get('body', '') or ''
|
||||
if all_checkboxes_done(updated_body) and count_checkboxes(updated_body) > 0:
|
||||
print(f" ✅ Auto-closing #{i['number']} after sync")
|
||||
gitea_api(f"/repos/{repo}/issues/{i['number']}", {"state": "closed"}, method='PATCH', repo='')
|
||||
gitea_api(
|
||||
f"/repos/{repo}/issues/{i['number']}/comments",
|
||||
{"body": "## ✅ Issue Auto-Closed\n\nAll acceptance criteria checked after syncing from comments."},
|
||||
method='POST',
|
||||
repo=''
|
||||
)
|
||||
closed_count += 1
|
||||
|
||||
print(f"\n 📊 Fixed: {fixed_count} checkboxes synced, {closed_count} issues auto-closed")
|
||||
|
||||
print(f"\n{'=' * 60}")
|
||||
print(f"PQS Rating: {rate_pqs(result['pqs'])}")
|
||||
|
||||
# Return exit code based on PQS
|
||||
if result['pqs'] >= 0.85:
|
||||
sys.exit(0) # Excellent
|
||||
elif result['pqs'] >= 0.60:
|
||||
sys.exit(1) # Adequate
|
||||
elif result['pqs'] >= 0.40:
|
||||
sys.exit(2) # Poor
|
||||
else:
|
||||
sys.exit(3) # Critical
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
Reference in New Issue
Block a user