Commit Graph

24 Commits

Author SHA1 Message Date
bboxwtf
a8a8ea1ee2 feat(swarm): autonomous agent containers, Swarm Manager with auto-stop, /nodes UI overhaul
## 1. Fix /nodes Swarm Status Display
- Add SwarmStatusBanner component: clear green/red/loading state
- Shows nodeId, managerAddr, isManager badge
- Error state explains what to check (docker.sock mount)
- Header now shows 'swarm unreachable — check gateway' vs 'active'
- swarmOk now checks nodeId presence, not just data existence

## 2. Autonomous Agent Container
- New docker/Dockerfile.agent — builds Go agent binary from gateway/cmd/agent/
- New gateway/cmd/agent/main.go — standalone HTTP microservice:
  * GET /health — liveness probe with idle time info
  * POST /task — receives task, forwards to Gateway orchestrator
  * GET /info  — agent metadata (id, hostname, gateway url)
  * Idle watchdog: calls /api/swarm/agents/{name}/stop after IdleTimeoutMinutes
  * Connects to Swarm overlay network (goclaw-net) → reaches DB/Gateway by DNS
  * Env: AGENT_ID, GATEWAY_URL, DATABASE_URL, IDLE_TIMEOUT_MINUTES

## 3. Swarm Manager Agent (auto-stop after 15min idle)
- New gateway/internal/api/swarm_manager.go:
  * SwarmManager goroutine checks every 60s
  * Scales idle GoClaw agent services to 0 replicas after 15 min
  * Tracks lastActivity from task UpdatedAt timestamps
- New REST endpoints in gateway:
  * GET  /api/swarm/agents           — list agents with idleMinutes
  * POST /api/swarm/agents/{name}/start — scale up agent
  * POST /api/swarm/agents/{name}/stop  — scale to 0
  * DELETE /api/swarm/services/{id}     — remove service permanently
- SwarmManager started as background goroutine in main.go with context cancel

## 4. Docker Client Enhancements
- Added NetworkAttachment type and Networks field to ServiceSpec
- CreateAgentServiceFull(opts) — supports overlay networks, custom labels
- CreateAgentService() delegates to CreateAgentServiceFull for backward compat
- RemoveService(id) — DELETE /v1.44/services/{id}
- GetServiceLastActivity(id) — finds latest task UpdatedAt for idle detection

## 5. tRPC & Gateway Proxy
- New functions: removeSwarmService, listSwarmAgents, startSwarmAgent, stopSwarmAgent
- SwarmAgentInfo type with idleMinutes, lastActivity, desiredReplicas
- createAgentService now accepts networks[] parameter
- New tRPC endpoints: nodes.removeService, nodes.listAgents, nodes.startAgent, nodes.stopAgent

## 6. Nodes.tsx UI Overhaul
- SwarmStatusBanner component at top — no more silent 'connecting…'
- New 'Agents' tab with AgentManagerRow: idle time, auto-stop warning, start/stop/remove buttons
- IdleColor coding: green < 5m, yellow 5-10m, red 10m+ with countdown to auto-stop
- ServiceRow: added Remove button with confirmation dialog
- RemoveConfirmDialog component
- DeployAgentDialog: added overlay networks field, default env includes GATEWAY_URL
- All queries refetch after agent start/stop/remove
2026-03-21 20:37:21 +00:00
bboxwtf
e228e7a655 fix(agents): provider pre-selection, magic-wand auto-fill, maxTokens from Ollama API
1. AgentDetailModal – fix provider not being pre-selected on edit open:
   - Add resolveProviderValue() that does exact → case-insensitive → partial
     match between stored provider string and connectedProviders list
   - Re-resolve provider in a second useEffect once providers load from API
   - Add safety-net SelectItem for stored value not found in providers list

2. AgentCreateModal – refactor Deploy Agent form:
   - Fix Provider + Model fields layout (grid-cols-2 with w-full truncate to
     prevent overflow/merging)
   - Add Wand2 'Auto-fill' button next to Agent Name field that calls
     agentCompiler.compile (existing LLM endpoint) with name+description as
     spec — fills role, model, temperature, systemPrompt automatically
   - Add Sparkles hint text explaining the magic wand functionality
   - Auto-select first provider/model when data loads
   - All fields use font-mono + proper label spacing

3. Both modals – MaxTokens auto-fill from Ollama API:
   - Add getOllamaModelInfo() in gateway-proxy.ts: calls Ollama /api/show,
     extracts {arch}.context_length from model_info, returns contextLength +
     parameterSize, family, quantization, capabilities
   - Add ollama.modelInfo tRPC query endpoint in routers.ts (input: modelId)
   - Both modals query trpc.ollama.modelInfo on model selection change
   - Auto-set maxTokens to context_length from API (262144 for kimi-k2.5 etc.)
   - Show 'max N from API' hint + clickable link to set full context window
   - Loading spinner while fetching model info
2026-03-21 19:41:15 +00:00
bboxwtf
c57d694236 feat(phase21): real Docker Swarm management — live nodes, services, tasks, host shell, agent deployment
## What's implemented

### Go Gateway — New /api/swarm/* endpoints (handlers.go + docker/client.go + db.go)
- GET  /api/swarm/info          — swarm state, manager address, join tokens
- GET  /api/swarm/nodes         — live node list (hostname, IP, CPU, RAM, role, labels)
- POST /api/swarm/nodes/{id}/label        — add/update node label
- POST /api/swarm/nodes/{id}/availability — set node availability (active|pause|drain)
- GET  /api/swarm/services       — all swarm services with replica counts
- POST /api/swarm/services/create — deploy a new agent as a swarm service
- GET  /api/swarm/services/{id}/tasks  — tasks per service (which node runs which replica)
- POST /api/swarm/services/{id}/scale  — scale replicas
- GET  /api/swarm/join-token    — worker/manager join command with token + manager addr
- POST /api/swarm/shell         — execute commands on the HOST via nsenter PID 1

### Docker client (client.go)
- ListServices, GetService, ScaleService, ListServiceTasks, CreateAgentService
- AddNodeLabel, UpdateNodeAvailability (patch node spec via Docker API)
- ExecOnHost (nsenter -t 1 → falls back to container scope)

### DB persistence (db.go)
- UpsertSwarmNodes — stores live node state to swarmNodes table
- UpsertSwarmTokens / GetSwarmTokens — persist join tokens
- Startup goroutine in main.go syncs tokens to DB on gateway start

### Node.js tRPC wrappers (routers.ts + gateway-proxy.ts)
- nodes.swarmInfo, nodes.list, nodes.services, nodes.serviceTasks
- nodes.scaleService, nodes.joinToken, nodes.execShell
- nodes.addNodeLabel, nodes.setAvailability, nodes.deployAgentService

### Frontend — Nodes.tsx (complete rewrite)
- Real swarm overview cards (nodes, managers, services, running tasks)
- Join token cards with copy button for worker & manager tokens
- Node cards with inline availability selector (active/pause/drain) + add-label form
- Services table with Scale dialog + Tasks drawer (replica → node mapping)
- Deploy Agent dialog (image, replicas, env vars, published port)
- Host Shell tab with command history and quick-command buttons

### docker-compose.yml
- gateway now runs with privileged: true + pid: host
  → nsenter can access the host PID namespace for real host-level shell execution

## Verified end-to-end
- GET /api/swarm/info returns manager addr + join tokens ✓
- GET /api/swarm/nodes returns node wsm (2 cores, 3.9 GB) ✓
- POST /api/swarm/services/create → deployed goclaw-test-agent (2 replicas) ✓
- GET /api/swarm/services/{id}/tasks returns task list with nodeId ✓
- POST /api/swarm/services/{id}/scale → scale to 0 ✓
- POST /api/swarm/shell {command:'docker node ls'} → real host output ✓
- tRPC chain: browser → control-center → gateway → docker.sock ✓
2026-03-21 17:23:32 +00:00
bboxwtf
471ca42835 feat(phase20): persistent background chat sessions — DB-backed polling architecture
ARCHITECTURE:
- Replace SSE stream (breaks on page reload) with DB-backed background sessions
- Go Gateway runs orchestrator in detached goroutine using context.Background()
  (survives HTTP disconnect, page reload, and laptop sleep/shutdown)
- Every SSE event (thinking/tool_call/delta/done/error) is persisted to chatEvents table
- Session lifecycle stored in chatSessions table (running→done/error)
- Frontend polls GET /api/orchestrator/getEvents every 1.5 s until status=done

DB CHANGES:
- Migration 0005_chat_sessions.sql: chatSessions + chatEvents tables
- schema.ts: TypeScript types for chatSessions and chatEvents
- db.go: ChatSessionRow and ChatEventRow structs with proper json tags (camelCase)
- db.go: CreateSession, AppendEvent, MarkSessionDone, GetSession, GetEvents, GetRecentSessions

GO GATEWAY:
- handlers.go: StartChatSession — creates DB session, launches goroutine, returns {sessionId} immediately
- handlers.go: GetChatSession, GetChatEvents, ListChatSessions handlers
- main.go: routes POST /api/chat/session, GET /api/chat/session/{id}, GET /api/chat/session/{id}/events, GET /api/chat/sessions
- JSON tags added to ChatSessionRow/ChatEventRow so Go returns camelCase to frontend

NODE.JS SERVER:
- gateway-proxy.ts: startChatSession, getChatSession, getChatEvents, listChatSessions functions
- routers.ts: orchestrator.startSession, .getSession, .getEvents, .listSessions tRPC procedures

FRONTEND:
- chatStore.ts: completely rewritten — uses background sessions + localStorage-based polling resume
  * send() calls orchestrator.startSession via tRPC (returns immediately)
  * Stores sessionId in localStorage (goclaw-pending-sessions)
  * Polls getEvents every 1.5 s, applies events to UI incrementally
  * On page reload: _resumePendingSessions() checks pending sessions and resumes polling
  * cancel() stops all active polls
- chatStore.ts: conversations persisted to localStorage (v3 key, survives page reload)
- Chat.tsx: updated status texts to 'Фоновая обработка…', 'Обработка в фоне…'

VERIFIED:
- POST /api/chat/session → {sessionId, status:'running'} in <100ms
- Poll events → thinking, delta('Привет!'), done after ~2s
- chatSessions table has rows with status=done, model, totalTokens
- Cyrillic stored correctly in UTF-8
- JSON fields are camelCase: id, sessionId, seq, eventType, content, toolName...
2026-03-21 16:50:44 +00:00
bboxwtf
73bfa99c67 feat(metrics): persist orchestrator call stats to agentMetrics + agentHistory
- db.go: added SaveMetric(MetricInput) and SaveHistory(HistoryInput) methods
  that write directly to MySQL; non-fatal (log-only on error)
- handlers.go (OrchestratorStream): after each SSE stream finishes, an async
  goroutine saves agentMetrics (agentId, requestId, tokens, processingTimeMs,
  model, toolsCalled, status) and agentHistory (userMessage, agentResponse);
  both error and success paths covered; orchAgentID resolved from DB
- routers.ts (agents.chat): saveMetric() called for both success and error paths
  in the Node.js direct-chat fallback (was only saving agentHistory before)
- Verified: agentMetrics row ID=2 shows processingTimeMs=2133, totalTokens=143,
  model=minimax-m2.7, Cyrillic text stored correctly as UTF-8
2026-03-21 16:17:15 +00:00
bboxwtf
1b6b8bc2cb feat(phase19): background chat store, UTF-8 SSE fix, DB-backed provider push to gateway
- Chat.tsx: rewritten to use global chatStore singleton — SSE connection survives
  page navigation; added StopCircle cancel button; scrolls only when near bottom
- chatStore.ts: new module-level singleton (EventTarget pattern) that holds all
  conversation/console state; TextDecoder with stream:true for correct UTF-8
- handlers.go (ProvidersReload): now accepts decrypted key in request body from
  Node.js so Go gateway can actually use the API key without sharing crypto logic
- providers.ts (activateProvider): sends decrypted key to gateway via
  notifyGatewayReload(); seedDefaultProvider also calls notifyGatewayReload()
- seed.ts: on startup, after seeding, pushes active provider to gateway with
  retry loop (5 retries × 3 s) to wait for gateway readiness
- index.ts (SSE proxy): TextDecoder('utf-8', {stream:true}) already correct;
  confirmed Cyrillic text arrives ungarbled (e.g. 'Привет!' not '??????????')
2026-03-21 04:12:45 +00:00
bboxwtf
981ab696b7 fix(seed): always run seedDefaultProvider regardless of agents count 2026-03-21 03:41:05 +00:00
bboxwtf
1ad62cf215 feat(phase18): DB-backed LLM providers, SSE streaming chat, left panel + console
Changes:
- drizzle/schema.ts: added llmProviders table (AES-256-GCM encrypted API keys)
- drizzle/0004_llm_providers.sql: migration for llmProviders
- server/providers.ts: full CRUD + AES-256-GCM encrypt/decrypt + seedDefaultProvider
- server/routers.ts: replaced hardcoded config.providers with DB-backed providers router;
  added providers.list/create/update/delete/activate tRPC endpoints
- server/seed.ts: calls seedDefaultProvider() on startup to seed from env if table empty
- server/_core/index.ts: added POST /api/orchestrator/stream SSE proxy route to Go Gateway
- gateway/internal/llm/client.go: added ChatStream (SSE) + UpdateCredentials
- gateway/internal/orchestrator/orchestrator.go: added ChatWithEvents (tool-call callbacks)
- gateway/internal/api/handlers.go: added OrchestratorStream (SSE) + ProvidersReload endpoints
- gateway/internal/db/db.go: added GetActiveProvider from llmProviders table
- gateway/cmd/gateway/main.go: registered /api/orchestrator/stream + /api/providers/reload routes
- client/src/pages/Chat.tsx: full rebuild — 3-panel layout (left: conversation list,
  centre: messages with SSE streaming + markdown, right: live tool-call console)
- client/src/pages/Settings.tsx: full rebuild — DB-backed provider CRUD (add/edit/activate/delete),
  no hardcoded keys, key shown masked from DB hint
2026-03-21 03:25:43 +00:00
bboxwtf
91684956bb fix(phase17): 401 auth, provider config from server, remove hardcoded PROVIDERS
Problems fixed:
1. 401 unauthorized on chat — OLLAMA_API_KEY was not set in containers
   - Created docker/.env with real API key
   - Added OLLAMA_BASE_URL + OLLAMA_API_KEY to control-center in docker-compose.yml

2. AgentDetailModal/AgentCreateModal showed hardcoded providers list
   (Ollama, OpenAI, Anthropic, Mistral, Groq) regardless of what is configured
   - Removed const PROVIDERS = [...] from both modals
   - Now loads providers via trpc.config.providers (server-side)
   - Only shows providers that are actually configured in env

3. Settings.tsx had API key hardcoded in frontend source code (security issue)
   - API key removed from frontend
   - New trpc.config.providers endpoint returns masked key (first 8 chars + ***)
   - Shows red warning badge 'NO KEY — chat will fail' if key is missing
   - Base URL read from server env, not hardcoded

New tRPC endpoint: config.providers
   - Returns list of configured providers with name, baseUrl, hasKey, maskedKey
   - Provider name auto-detected from URL (ollama.com → 'Ollama Cloud', etc.)
2026-03-21 02:55:05 +00:00
bboxwtf
62cedcdba5 feat(phase17): close technical debt — Dashboard real data, index.ts @deprecated, ADR streaming/auth
- Dashboard.tsx: removed 3 hardcoded mock constants (NODES/AGENTS/ACTIVITY_LOG)
  - Swarm Nodes panel: real data from trpc.nodes.list (swarm nodes or containers)
  - Container stats: live CPU%/MEM from trpc.nodes.stats, rendered as progress bars
  - Active Agents panel: real agents from trpc.agents.list with isActive/isSystem/model/role
  - Activity Feed: generated from active agents list (live agent names, models, timestamps)
  - Metric cards: real counts from trpc.dashboard.stats (uptime, nodes, agents, gateway)
  - All 3 panels have loading state (Loader2 spinner) and empty/error state
  - Hero banner subtitle uses real stats.nodes and stats.agents counts
  - Cluster Topology footer shows real uptime from dashboard.stats

- server/index.ts: documented as @deprecated legacy static-only entry point
  - Added JSDoc block explaining this file is NOT the production server
  - Points to server/_core/index.ts as the real server with tRPC/OAuth/seed
  - Added console.log WARNING on startup to prevent accidental use
  - File retained as historical artefact per Phase 17 decision

- todo.md: Phase 16 debt items closed as [x], Phase 17 section added
  - ADR-001: Streaming LLM — status DEFERRED, Phase 18 plan documented
    (Go Gateway stream:true + tRPC subscription + Chat.tsx EventSource)
  - ADR-002: Authentication — status ACCEPTED as internal tool
    (OAuth already partial; protectedProcedure path documented for future)
  - Phase 9 routers.ts orchestrator migration verified as complete
2026-03-21 02:47:59 +00:00
bboxwtf
f08513d9a5 fix(phase16): model validation & agent editor improvements
- AgentDetailModal: load real models from API with loading indicator;
  fallback to current agent model when API unavailable; show count badge
- AgentCreateModal: remove broken provider-filter on models list;
  add loading indicator and disabled state during fetch; show count badge
- gateway/orchestrator: add resolveModel() — validates desired model
  against LLM API before use; auto-fallback to first available model
  to prevent 401/404 errors (fixes glm-5 unauthorized in chat)
- gateway/orchestrator: add ModelWarning field to ChatResult struct
- gateway-proxy.ts: add modelWarning field to GatewayChatResult
- Chat.tsx: display modelWarning as amber badge next to model name
- todo.md: add Phase 16 section with bug fixes and tech debt notes
2026-03-21 02:10:17 +00:00
Manus
0959c90d36 Checkpoint: Fix: agents.list tRPC procedure now uses getAllAgents() instead of getUserAgents(SYSTEM_USER_ID=1). Root cause: seed creates agents with userId=0 but router queried userId=1. Added getAllAgents() and getSystemAgents() helpers. 86 tests pass. 2026-03-20 21:15:55 -04:00
Manus
16b101537c Checkpoint: Phase 14: Fixed hardcoded header metrics (UPTIME/NODES/AGENTS/CPU/MEM) — connected to real tRPC dashboard.stats endpoint with 30s polling. Fixed seed idempotency — now checks by isSystem=true instead of total count. Added dashboard.test.ts with 13 new tests. All 82 tests pass. 2026-03-20 21:00:51 -04:00
Manus
73a26d8a8a Checkpoint: Phase 13: Seed data for agents and orchestrator
- server/seed.ts: 6 default system agents (Orchestrator, Browser, Tool Builder, Agent Compiler, Coder, Researcher)
- Idempotent: runs only when agents table is empty
- Integrated into server/_core/index.ts startup
- server/seed.test.ts: 18 vitest tests, all pass
- Total: 69 tests pass (7 test files)
2026-03-20 20:39:08 -04:00
Manus
0dcae37a78 Checkpoint: Phase 12: Real-time Docker Swarm monitoring for /nodes page
Реализовано:
- gateway/internal/docker/client.go: Docker API клиент через unix socket (/var/run/docker.sock)
  - IsSwarmActive(), GetSwarmInfo(), ListNodes(), ListContainers(), GetContainerStats()
  - CalcCPUPercent() для расчёта CPU%
- gateway/internal/api/handlers.go: новые endpoints
  - GET /api/nodes: список Swarm нод или standalone Docker хост
  - GET /api/nodes/stats: live CPU/RAM статистика контейнеров
  - POST /api/tools/execute: выполнение инструментов
- gateway/cmd/gateway/main.go: зарегистрированы новые маршруты
- server/gateway-proxy.ts: добавлены getGatewayNodes() и getGatewayNodeStats()
- server/routers.ts: добавлен nodes router (nodes.list, nodes.stats)
- client/src/pages/Nodes.tsx: полностью переписан на реальные данные
  - Auto-refresh: 10s для нод, 15s для статистики контейнеров
  - Swarm mode: показывает все ноды кластера
  - Standalone mode: показывает локальный Docker хост + контейнеры
  - CPU/RAM gauges из реальных docker stats
  - Error state при недоступном Gateway
  - Loading skeleton
- server/nodes.test.ts: 14 новых vitest тестов
- Все 51 тест пройдены
2026-03-20 20:12:57 -04:00
Manus
2f87e18e85 Checkpoint: Phase 11 complete: Frontend connected to Go Gateway. All orchestrator/ollama tRPC calls go through gateway-proxy.ts with Node.js fallback. 37 vitest tests pass. End-to-end verified: chat, tool calling, health via Go Gateway. 2026-03-20 19:38:27 -04:00
Manus
02742f836c Checkpoint: Phase 9: Go Gateway — полный перенос оркестратора и tool executor на Go. Добавлены gateway/ (Go), docker/ (docker-compose + stack + Dockerfiles), server/gateway-proxy.ts 2026-03-20 18:43:49 -04:00
Manus
46e384c341 Checkpoint: Phase 8 Complete: Fix Orchestrator Chat
Исправлено:
- Chat.tsx: убрана хардкодированная модель "qwen2.5:7b" из мутации — теперь оркестратор использует модель из конфига БД (minimax-m2.7)
- Chat.tsx: добавлен Streamdown для markdown рендеринга ответов оркестратора
- Подтверждено: tool calling работает — команда "Покажи файлы проекта" вызывает file_list и возвращает структуру проекта
- Подтверждено: model в header показывает "minimax-m2.7" из БД
- TypeScript: 0 ошибок (pnpm tsc --noEmit)
- Тесты: 24/24 passed
2026-03-20 18:20:37 -04:00
Manus
7aa8eee2ca Checkpoint: Phase 7 complete: Orchestrator Agent добавлен в /agents с меткой CROWN/SYSTEM, кнопками Configure и Open Chat. /chat читает конфиг оркестратора из БД (модель, промпт, инструменты). AgentDetailModal поддерживает isOrchestrator. 24 теста пройдены. 2026-03-20 17:48:21 -04:00
Manus
c2fdfdbf72 true message 2026-03-20 17:34:20 -04:00
Manus
86a1ee9062 Checkpoint: Full Development Complete: All 4 Phases
## Phase 1 (Fixed): Agent Management UI
- Исправлена авторизация: agents переведены на publicProcedure
- AgentDetailModal: 5 вкладок (General, LLM Params, Tools, History, Stats)
- Полное редактирование: model, provider, temperature, topP, maxTokens, frequencyPenalty, presencePenalty, systemPrompt
- Управление allowedTools и allowedDomains через теги
- AgentCreateModal: создание агентов с выбором модели из Ollama API
- Кнопка Metrics на каждой карточке агента

## Phase 2+3: Tool Binding System
- server/tools.ts: реестр из 10 инструментов (http_get, http_post, shell_exec, file_read, file_write, docker_list, docker_exec, docker_logs, browser_navigate, browser_screenshot)
- Безопасное выполнение: проверка allowedTools агента, accessControl из БД
- tools.execute tRPC endpoint
- Tools.tsx: страница управления инструментами с тест-выполнением
- Добавлен пункт "Инструменты" в sidebar навигацию

## Phase 4: Metrics & History
- AgentMetrics.tsx: детальная страница метрик по агенту
- Request Timeline: bar chart по часам (success/error)
- Conversation Log: история диалогов с пагинацией
- Raw Metrics Table: все метрики с токенами и временем
- Time range selector: 6h/24h/48h/7d
- Маршрут /agents/:id/metrics

## Tests: 24/24 passed
- server/auth.logout.test.ts (1)
- server/agents.test.ts (7)
- server/tools.test.ts (13)
- server/ollama.test.ts (3)
2026-03-20 16:52:27 -04:00
Manus
159a89a156 true message 2026-03-20 16:39:29 -04:00
Manus
b18e6e244f Checkpoint: Интеграция реального Ollama Cloud API: серверный прокси (tRPC), Dashboard с live-статусом подключения и количеством моделей, Chat с реальными ответами LLM и выбором модели, Settings с живым списком 34 моделей. Все 4 vitest теста пройдены. 2026-03-20 16:03:01 -04:00
Manus
351be6cad6 Initial project bootstrap 2026-03-20 15:24:10 -04:00