UniqueSoft/APAW
2
0
Fork 0
Code Issues 15 Pull Requests Actions Packages Projects Releases Wiki Activity

[Высокий] Перейти на эмпирическое назначение моделей — fitness-history.jsonl #126

New Issue
Open
opened 2026-05-28 13:39:28 +00:00 by NW · 0 comments
NW commented 2026-05-28 13:39:28 +00:00
Owner
Copy Link

Проблема

Три исследовательских источника дали противоречивые назначения за 72 часа:

  • 25 мая: priority-based (model-research-latest.json)
  • 27 мая: evolution.json → 18 агентов на kimi-k2.6
  • 28 мая: real-fit-report.json → все 36 агентов переназначены

real-fit методология ненадёжна:

  • Эвристическая генерация промптов (8 случайных строк из .md)
  • SWE-bench «unverifiable → null», hash fallback 55-80
  • Разброс оценок: code-skeptic 22.8 vs 91.2 (68 баллов)
  • Возможность USE_MOCK — данные могут быть синтетическими

Решение

Использовать .kilo/logs/fitness-history.jsonl и .kilo/logs/efficiency_score.json — реальные данные реальных прогонов пайплайна.

Acceptance Criteria

  • Скрипт scripts/compute-empirical-scores.py читает fitness-history.jsonl
  • Формула: empirical_score = avg(fitness × 10) по последним N прогонам агента
  • Сравнение: real-fit score vs empirical score для каждого агента
  • Если корреляция < 0.5 — real-fit признан невалидным
  • Если real-fit невалиден — откатить назначения к предыдущим validated моделям
  • Документация: EMPIRICAL_ASSIGNMENT.md

Эстимейт

Medium (10K tokens, ~45 мин)

Assignee

@system-analyst + @sdet-engineer

## Проблема Три исследовательских источника дали противоречивые назначения за 72 часа: - 25 мая: priority-based (`model-research-latest.json`) - 27 мая: `evolution.json` → 18 агентов на `kimi-k2.6` - 28 мая: `real-fit-report.json` → все 36 агентов переназначены `real-fit` методология ненадёжна: - Эвристическая генерация промптов (8 случайных строк из `.md`) - SWE-bench «unverifiable → null», hash fallback 55-80 - Разброс оценок: `code-skeptic` 22.8 vs 91.2 (68 баллов) - Возможность `USE_MOCK` — данные могут быть синтетическими ## Решение Использовать `.kilo/logs/fitness-history.jsonl` и `.kilo/logs/efficiency_score.json` — реальные данные реальных прогонов пайплайна. ## Acceptance Criteria - [ ] Скрипт `scripts/compute-empirical-scores.py` читает `fitness-history.jsonl` - [ ] Формула: `empirical_score = avg(fitness × 10)` по последним N прогонам агента - [ ] Сравнение: `real-fit score` vs `empirical score` для каждого агента - [ ] Если корреляция < 0.5 — `real-fit` признан невалидным - [ ] Если `real-fit` невалиден — откатить назначения к предыдущим validated моделям - [ ] Документация: `EMPIRICAL_ASSIGNMENT.md` ## Эстимейт Medium (10K tokens, ~45 мин) ## Assignee @system-analyst + @sdet-engineer
NW added this to the [Контроль качества] Остановить трэшинг, единый источник правды milestone 2026-05-28 13:39:28 +00:00
NW added the status::newpriority::hightype::enhancement labels 2026-05-28 13:39:28 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
agent::agent-architect
Owned by agent-architect
 agent::capability-analyst
Owned by capability-analyst
 agent::code-skeptic
Owned by code-skeptic
 agent::evaluator
Owned by evaluator
 agent::history-miner
Owned by history-miner
 agent::lead-developer
Owned by lead-developer
 agent::orchestrator
Owned by orchestrator
 agent::sdet-engineer
Owned by sdet-engineer
 agent::system-analyst
Owned by system-analyst
 agent::the-fixer
Owned by the-fixer
 budget::exhausted
Token budget exhausted
 budget::sufficient
Token budget sufficient
 budget::warning
Token budget low
 cascade::depth-0
No subagent calls
 cascade::depth-1
1-level subagent calls
 cascade::depth-2
2-level subagent calls
 cascade::depth-exceeded
Depth limit exceeded
 cascade::depth-n
Unlimited subagent calls
 dashboard::frozen
Dashboard feature freeze — no new features until thaw
 evolution::model-change
Model change evolution
 evolution::new-agent
New agent evolution
 evolution::new-skill
New skill evolution
 evolution::new-workflow
New workflow evolution
 evolution::prompt-opt
Prompt optimization evolution
 memory::checkpoint
Checkpoint stored
 memory::fresh
Checkpoint fresh
 memory::recoverable
Checkpoint recoverable
 memory::stale
Checkpoint stale
 permission::evolve-system
Can evolve system
 permission::read-only
Read-only access
 permission::violation
Security violation
 permission::write-code
Can write code
 permission::write-config
Can write config
 phase::awaiting-review
Agent awaits review
 phase::drafting-spec
Agent is drafting specification
 phase::executing
Agent is executing task
 phase::gathering-evidence
Agent is gathering data
 phase::refining-prompt
Agent is refining prompts
 phase::researching
Research phase — gathering evidence and solutions
 phase::verifying
Agent is verifying results
 priority::critical
Critical priority
 priority::high
High priority
 priority::low
Low priority
 priority::medium
Medium priority
 quality::blocked
Blocked by quality
 quality::fail
Quality check failed
 quality::needs-fix
Needs fixes
 quality::pass
Quality check passed
 size::l
Large (4-8 hours)
 size::m
Medium (2-4 hours)
 size::s
Small (1-2 hours)
 size::xl
Extra large (>8 hours)
 size::xs
Extra small (<1 hour)
 status::blocked
Blocked
 status::cancelled
Cancelled
 status::done
Completed
 status::in-progress
Work in progress
 status::new
New issue, not started
 status::planned
Planned for sprint
 status::review
Under review
 status::testing
In testing
 type::bug
Something is broken
 type::chore
Maintenance task
 type::documentation
Documentation
 type::enhancement
Improvement
 type::feature
New feature
 type::refactor
Code refactoring
 type::test
Testing
No Label priority::high status::new type::enhancement
Milestone
No items
No Milestone [Контроль качества] Остановить трэшинг, единый источник правды
Projects
Clear projects
No project
Assignees
Clear assignees
NW olilovedani
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: UniqueSoft/APAW#126
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?