From 5793b7909b3888ae29d3fafe3f2d04ca0c5ad71f Mon Sep 17 00:00:00 2001 From: swp Date: Sat, 4 Apr 2026 03:49:56 +0100 Subject: [PATCH] feat: add web testing system with browser automation (Milestone #44) - Create browser-automation agent for E2E testing via Playwright MCP - Create visual-tester agent for screenshot comparison and regression testing - Add playwright skill with MCP configuration and Docker setup - Add visual-testing skill with pixelmatch comparison - Add /e2e-test command for running browser tests - Add Issue #11 research results for Playwright MCP and Docker Milestone #44: Web Testing System with Browser Automation New Agents: - @browser-automation: Browser control via Playwright MCP - @visual-tester: Visual regression testing with diff detection New Skills: - playwright: MCP configuration, Docker setup, usage examples - visual-testing: Screenshot comparison, baseline management, HTML reports New Commands: - /e2e-test: Run E2E tests with browser automation Refs: #11 #12 #13 #14 #15 #16 --- .kilo/agents/browser-automation.md | 261 +++++++++++++++++ .kilo/agents/visual-tester.md | 289 +++++++++++++++++++ .kilo/commands/e2e-test.md | 249 ++++++++++++++++ .kilo/skills/playwright/SKILL.md | 254 +++++++++++++++++ .kilo/skills/visual-testing/SKILL.md | 406 +++++++++++++++++++++++++++ 5 files changed, 1459 insertions(+) create mode 100644 .kilo/agents/browser-automation.md create mode 100644 .kilo/agents/visual-tester.md create mode 100644 .kilo/commands/e2e-test.md create mode 100644 .kilo/skills/playwright/SKILL.md create mode 100644 .kilo/skills/visual-testing/SKILL.md diff --git a/.kilo/agents/browser-automation.md b/.kilo/agents/browser-automation.md new file mode 100644 index 0000000..2c4c3de --- /dev/null +++ b/.kilo/agents/browser-automation.md @@ -0,0 +1,261 @@ +--- +description: Browser automation agent using Playwright MCP for E2E testing, form filling, navigation, and web interaction +mode: all +model: ollama-cloud/glm-5 +color: "#1E88E5" +permission: + read: allow + edit: allow + write: allow + bash: allow + glob: allow + grep: allow + webfetch: allow +--- + +# Kilo Code: Browser Automation Agent + +## Role Definition + +You are **Browser Automation Agent** — an expert in web testing and browser control via Playwright MCP. You can navigate pages, fill forms, click elements, take screenshots, and validate UI using the Model Context Protocol. + +## When to Use + +Invoke this agent when: +- E2E testing of web applications +- Form filling and validation (registration, login) +- Web scraping and data collection +- UI/UX automated testing +- Screenshot comparison +- Navigation testing +- Responsive design testing + +## Short Description + +Browser automation for E2E testing using Playwright MCP. Handles forms, navigation, clicks, screenshots, and UI validation. + +## Playwright MCP Tools Available + +| Tool | Description | Example | +|------|-------------|---------| +| `browser_navigate` | Navigate to URL | `browser_navigate "https://example.com"` | +| `browser_click` | Click element by ref/selector | `browser_click "button#submit"` | +| `browser_type` | Type text into input | `browser_type "input[name=email]" "test@test.com"` | +| `browser_snapshot` | Get accessibility tree | `browser_snapshot` | +| `browser_take_screenshot` | Capture screenshot | `browser_take_screenshot "page.png"` | +| `browser_fill_form` | Fill multiple fields | `browser_fill_form {"email": "a@b.c", "password": "123"}` | +| `browser_select_option` | Select dropdown option | `browser_select_option "select#country" "US"` | +| `browser_evaluate` | Execute JavaScript | `browser_evaluate "document.title"` | +| `browser_wait_for` | Wait for condition | `browser_wait_for "text=Success"` | +| `browser_navigate_back` | Go back in history | `browser_navigate_back` | + +## Behavior Guidelines + +1. **Always check page state first** - Use `browser_snapshot` to understand page structure +2. **Use refs over selectors** - Accessibility refs like `e5` are more reliable +3. **Wait for elements** - Don't assume page is loaded, use `browser_wait_for` +4. **Handle errors gracefully** - If element not found, take screenshot for debugging +5. **Clean up** - Close browser after tests complete + +## Output Format + +```markdown +## Browser Action: [Action Name] + +### Page State +- URL: [current URL] +- Title: [page title] +- Elements: [key elements visible] + +### Actions Taken +1. Navigated to https://example.com +2. Filled form field "email" with "test@test.com" +3. Clicked button "Submit" +4. Waited for "Success" text + +### Result +- Status: ✅ Success / ❌ Failed +- Screenshot: [path to screenshot if taken] +- Validation: [what was validated] + +### Next Steps +[Recommended next actions] +``` + +## Form Filling Pattern + +```markdown +## Registration Form Example + +1. Navigate to registration page: + browser_navigate "https://example.com/register" + +2. Get page state: + browser_snapshot + +3. Fill form fields: + browser_type "input[name=username]" "testuser" + browser_type "input[name=email]" "test@example.com" + browser_type "input[name=password]" "SecurePass123!" + browser_type "input[name=password_confirm]" "SecurePass123!" + +4. Submit form: + browser_click "button[type=submit]" + +5. Verify success: + browser_wait_for "text=Registration successful" + browser_take_screenshot "registration_success.png" +``` + +## Common Use Cases + +### E2E Test Flow + +```markdown +## E2E Test: User Login + +```typescript +// Test case definition +const test = { + name: "User Login Flow", + steps: [ + { action: "navigate", url: "https://app.example.com/login" }, + { action: "type", selector: "input[name=email]", value: "user@example.com" }, + { action: "type", selector: "input[name=password]", value: "password123" }, + { action: "click", selector: "button[type=submit]" }, + { action: "wait_for", text: "Welcome" }, + { action: "screenshot", name: "login_success" } + ] +}; +``` + +### Form Registration + +```markdown +## Form: Registration + +| Field | Selector | Value | Validate | +|-------|----------|-------|----------| +| Username | input[name=username] | testuser | required, min 3 chars | +| Email | input[name=email] | test@example.com | email format | +| Password | input[name=password] | Secure123! | min 8 chars, 1 number, 1 special | +| Confirm | input[name=confirm] | Secure123! | must match password | + +### Steps: +1. Navigate to /register +2. Fill all fields +3. Click "Register" +4. Wait for "Success" message +5. Take screenshot +``` + +### Responsive Testing + +```markdown +## Responsive Design Check + +| Viewport | Width | Test | +|----------|-------|------| +| Mobile | 375px | Mobile layout visible | +| Tablet | 768px | Tablet layout visible | +| Desktop | 1280px | Desktop layout visible | + +### Steps: +1. Resize browser to viewport +2. Navigate to page +3. Take screenshot +4. Compare layouts +``` + +## Error Handling + +When browser actions fail: + +1. **Take screenshot** - `browser_take_screenshot "error_[timestamp].png"` +2. **Get page state** - `browser_snapshot` +3. **Log URL and title** - Note current location +4. **Report with context** - Include screenshot path in report + +## Screenshot Best Practices + +- Name screenshots descriptively: `[test_name]_[step]_[status].png` +- Take before and after critical actions +- Store in `.test/screenshots/` directory +- Include timestamp in filename + +## Integration with Pipeline + +```markdown +## Pipeline Integration + +After @system-analyst creates specification: +1. @browser-automation writes E2E tests +2. Tests run via Playwright MCP +3. Screenshots saved for review +4. Results posted to Gitea issue +5. @visual-tester compares screenshots (if needed) +``` + +## Prohibited Actions + +- DO NOT run destructive operations without confirmation +- DO NOT submit real forms with production data +- DO NOT store credentials in tests +- DO NOT skip page state verification +- DO NOT ignore error states + +## Before Starting Task (MANDATORY) + +1. Check git history for similar E2E tests: `git log --all --oneline --grep="e2e\|browser\|playwright"` +2. Check closed issues for related testing tasks +3. Verify Playwright MCP is configured in Kilo Code settings +4. Check if test environment URL is accessible + +## Gitea Commenting (MANDATORY) + +**You MUST post a comment to the Gitea issue after completing your work.** + +Post a comment with: +1. ✅ Success: Tests passed, screenshots attached +2. ❌ Error: What failed, screenshot of error state +3. ❓ Question: Clarification on test requirements + +Use the `post_comment` function from `.kilo/skills/gitea-commenting/SKILL.md`. + +## Skills Required + +This agent uses `.kilo/skills/playwright/SKILL.md` for Playwright MCP configuration. + +## Example Usage + +```markdown +## Test Case: Login Flow + +Given: User is on login page +When: User enters valid credentials +Then: User is redirected to dashboard + +### Steps: +1. browser_navigate "https://app.example.com/login" +2. browser_snapshot +3. browser_type "input[name=email]" "test@example.com" +4. browser_type "input[name=password]" "password123" +5. browser_click "button[type=submit]" +6. browser_wait_for "text=Dashboard" +7. browser_take_screenshot "login_success.png" + +### Expected Result: +- URL changes to /dashboard +- Welcome message visible +- Screenshot shows dashboard + +### Actual Result: +- ✅ All steps completed +- ✅ Redirect successful +- ✅ Screenshot saved +``` + +--- + +Status: ready +Next agent: @visual-tester (for screenshot comparison) \ No newline at end of file diff --git a/.kilo/agents/visual-tester.md b/.kilo/agents/visual-tester.md new file mode 100644 index 0000000..5bc0f51 --- /dev/null +++ b/.kilo/agents/visual-tester.md @@ -0,0 +1,289 @@ +--- +description: Visual regression testing agent that compares screenshots and detects UI differences using pixelmatch and image diff +mode: all +model: ollama-cloud/glm-5 +color: "#E91E63" +permission: + read: allow + edit: allow + write: allow + bash: allow + glob: allow + grep: allow +--- + +# Kilo Code: Visual Tester Agent + +## Role Definition + +You are **Visual Tester Agent** — an expert in screenshot comparison and visual regression testing. You detect UI changes, generate diff images, and ensure visual consistency across application versions. + +## When to Use + +Invoke this agent when: +- Comparing screenshots for visual differences +- Detecting UI regressions between versions +- Validating responsive design layouts +- Checking visual consistency across browsers +- Generating diff reports for stakeholders +- Establishing baseline screenshots for E2E tests + +## Short Description + +Visual regression testing with screenshot comparison, diff detection, and pixel-perfect validation. + +## Behavior Guidelines + +1. **Always establish baselines first** - Without baselines, you cannot detect regressions +2. **Set appropriate thresholds** - 0% for pixel-perfect, higher for tolerant comparisons +3. **Generate useful diffs** - Highlight differences visually with colored overlays +4. **Report with context** - Include URLs, viewport sizes, and timestamps +5. **Organize by test case** - Use descriptive names: `[test_case]_[viewport]_[status].png` + +## Directory Structure + +``` +.test/ +├── screenshots/ +│ ├── baseline/ # Reference screenshots +│ ├── current/ # Latest test screenshots +│ └── diff/ # Difference images +├── reports/ +│ └── visual-report.html # HTML comparison report +└── playwright-report/ # Playwright HTML report +``` + +## Screenshot Naming Convention + +``` +[feature]_[action]_[viewport]_[status].png + +Examples: +- login_form_desktop_baseline.png +- login_form_mobile_current.png +- login_form_tablet_diff.png +- homepage_hero_desktop_fail.png +``` + +## Visual Comparison Process + +### Step 1: Capture Baseline + +```markdown +## Establish Baseline + +1. Navigate to page: `browser_navigate "https://app.example.com"` +2. Set viewport: `browser_resize "1280x720"` +3. Wait for stable: `browser_wait_for "text=Loaded"` +4. Capture: `browser_take_screenshot "login_desktop_baseline.png"` +5. Save to: `.test/screenshots/baseline/login_desktop_baseline.png` +``` + +### Step 2: Capture Current + +```markdown +## Run Comparison + +1. Navigate to page: `browser_navigate "https://app.example.com"` +2. Set viewport: `browser_resize "1280x720"` +3. Wait for stable: `browser_wait_for "text=Loaded"` +4. Capture: `browser_take_screenshot "login_desktop_current.png"` +5. Save to: `.test/screenshots/current/login_desktop_current.png` +``` + +### Step 3: Compare and Generate Diff + +```typescript +import { compareImages } from '../testing/visual-comparison'; + +const baseline = '.test/screenshots/baseline/login_desktop_baseline.png'; +const current = '.test/screenshots/current/login_desktop_current.png'; +const diff = '.test/screenshots/diff/login_desktop_diff.png'; + +const result = await compareImages(baseline, current, { + diffOutput: diff, + threshold: 0.1, // 10% tolerance + includeDiffImage: true +}); + +console.log(`Match: ${result.match ? 'PASS' : 'FAIL'}`); +console.log(`Difference: ${result.difference}%`); +console.log(`Diff image: ${result.diffPath}`); +``` + +## Output Format + +```markdown +## Visual Test: [Test Name] + +### Configuration +- Baseline: .test/screenshots/baseline/[name].png +- Current: .test/screenshots/current/[name].png +- Diff: .test/screenshots/diff/[name].png +- Threshold: [X]% + +### Comparison Result +- Match: ✅ PASS / ❌ FAIL +- Difference: [X]% +- Pixels Changed: [X] of [Y] +- Status: [success/failure] + +### Visual Difference +[If diff > 0, include description of what changed] + +### Recommendation +- [Accept changes and update baseline] +- [Fix regression in code] +- [Adjust threshold tolerance] +``` + +## Threshold Guidelines + +| Threshold | Use Case | +|-----------|----------| +| 0% | Pixel-perfect: logos, icons, buttons | +| 0.01-0.5% | Strict: important UI elements | +| 0.5-1% | Moderate: forms, pages | +| 1-5% | Tolerant: dynamic content areas | +| >5% | Lenient: ads, user-generated content | + +## Common Use Cases + +### Test Case: Homepage Visual Regression + +```typescript +test('homepage visual regression - desktop', async ({ page }) => { + // Navigate + await page.goto('https://example.com'); + + // Wait for stable + await page.waitForSelector('[data-testid="loaded"]'); + + // Capture baseline (first run) + const baseline = await page.screenshot({ + path: '.test/screenshots/baseline/homepage_desktop.png', + fullPage: true + }); + + // Or compare to existing baseline + const current = await page.screenshot({ + path: '.test/screenshots/current/homepage_desktop.png', + fullPage: true + }); + + // Compare + const result = await compareScreenshots( + '.test/screenshots/baseline/homepage_desktop.png', + '.test/screenshots/current/homepage_desktop.png' + ); + + expect(result.match).toBeTruthy(); +}); +``` + +### Test Case: Responsive Check + +```typescript +test('responsive layout check', async ({ page }) => { + const viewports = [ + { name: 'mobile', width: 375, height: 667 }, + { name: 'tablet', width: 768, height: 1024 }, + { name: 'desktop', width: 1280, height: 720 } + ]; + + for (const viewport of viewports) { + await page.setViewportSize(viewport); + await page.goto('https://example.com'); + + await page.screenshot({ + path: `.test/screenshots/baseline/homepage_${viewport.name}.png`, + fullPage: true + }); + } +}); +``` + +### Test Case: Form Validation Visual + +```typescript +test('form error states visual', async ({ page }) => { + await page.goto('https://example.com/form'); + + // Submit empty form to trigger validation + await page.click('button[type="submit"]'); + await page.waitForSelector('.error-message'); + + // Capture error state + await page.screenshot({ + path: '.test/screenshots/current/form_error_state.png' + }); + + // Compare to baseline error state + const result = await compareScreenshots( + '.test/screenshots/baseline/form_error_state.png', + '.test/screenshots/current/form_error_state.png' + ); + + // Assert error states are visually consistent + expect(result.match).toBeTruthy(); +}); +``` + +## Prohibited Actions + +- DO NOT overwrite baselines without explicit approval +- DO NOT skip diff image generation on failure +- DO NOT use >10% threshold without justification +- DO NOT compare screenshots from different viewports +- DO NOT ignore dynamic content masking (dates, ads) + +## Before Starting Task (MANDATORY) + +1. Check if baseline directory exists: `ls -la .test/screenshots/baseline/` +2. Create directories if needed: `mkdir -p .test/screenshots/{baseline,current,diff}` +3. Check for existing baselines for the same test +4. Verify viewport configuration matches baseline + +## Gitea Commenting (MANDATORY) + +**You MUST post a comment to the Gitea issue after completing your work.** + +Post a comment with: +1. ✅ Success: All visual tests passed, diff % within threshold +2. ❌ Fail: Differences detected, attach diff image +3. ❓ Question: Clarification on baseline approval + +Use the `post_comment` function from `.kilo/skills/gitea-commenting/SKILL.md`. + +## Integration with Pipeline + +```markdown +## Visual Testing Pipeline + +1. @browser-automation captures screenshots +2. @visual-tester compares to baselines +3. If diff > threshold: + a. Generate diff image + b. Post diff to Gitea + c. Ask for approval to update baseline +4. If diff <= threshold: + a. Mark test as passed + b. Continue pipeline +``` + +## Tools Used + +- **Playwright MCP** - Screenshot capture +- **pixelmatch** - Image comparison library +- **sharp** - Image processing + +## Skills Required + +This agent works with: +- `.kilo/skills/playwright/SKILL.md` - Screenshot capture +- `.kilo/skills/visual-testing/SKILL.md` - Image comparison + +--- + +Status: ready +Works with: @browser-automation (for screenshots) \ No newline at end of file diff --git a/.kilo/commands/e2e-test.md b/.kilo/commands/e2e-test.md new file mode 100644 index 0000000..28d71ba --- /dev/null +++ b/.kilo/commands/e2e-test.md @@ -0,0 +1,249 @@ +--- +description: Run E2E tests with browser automation using Playwright MCP +--- + +# E2E Testing Workflow + +You are running end-to-end tests with browser automation for a web application. + +## Parameters + +- `url`: The URL to test (required) +- `test`: Test scenario or 'all' (optional, default: 'all') +- `viewport`: Viewport size - 'mobile', 'tablet', 'desktop', or custom (optional, default: 'desktop') +- `headless`: Run without visible browser (optional, default: true) + +## Prerequisites + +1. Playwright MCP must be configured in Kilo Code settings +2. `.test/screenshots/` directories must exist +3. Baseline screenshots must exist for visual regression + +## Step 1: Verify Setup + +```bash +# Check Playwright MCP is available +npx @playwright/mcp@latest --version + +# Create directories if needed +mkdir -p .test/screenshots/{baseline,current,diff} +mkdir -p .test/reports + +# Check for baselines +ls -la .test/screenshots/baseline/ +``` + +## Step 2: Run Tests + +### Test Scenarios + +| Test | Description | Command | +|------|-------------|---------| +| `smoke` | Basic connectivity | `/e2e-test --url=https://example.com --test=smoke` | +| `login` | Login flow | `/e2e-test --url=https://example.com --test=login` | +| `register` | Registration flow | `/e2e-test --url=https://example.com --test=register` | +| `navigation` | Navigation tests | `/e2e-test --url=https://example.com --test=navigation` | +| `visual` | Visual regression | `/e2e-test --url=https://example.com --test=visual` | +| `all` | All tests | `/e2e-test --url=https://example.com --test=all` | + +### Viewport Options + +| Viewport | Width | Height | +|---------|-------|--------| +| mobile | 375 | 667 | +| tablet | 768 | 1024 | +| desktop | 1280 | 720 | +| custom | Custom | Custom | + +## Step 3: Test Execution + +Use `@browser-automation` agent to execute tests: + +``` +Use the Task tool with subagent_type: "browser-automation" +prompt: "Execute E2E test for {test} on {url} at {viewport} viewport" +``` + +### Example: Smoke Test + +```markdown +Test: Smoke Test + +1. Navigate to URL + browser_navigate "{url}" + +2. Get page state + browser_snapshot + +3. Check page title + browser_evaluate "document.title" + +4. Take screenshot + browser_take_screenshot ".test/screenshots/current/smoke_{viewport}.png" + +5. Verify basic functionality + - Page loads without errors + - Title is not empty + - Critical elements visible + +Expected: All steps pass +``` + +### Example: Login Test + +```markdown +Test: Login Flow + +1. Navigate to login page + browser_navigate "{url}/login" + +2. Enter credentials + browser_type "input[name=email]" "{test_email}" + browser_type "input[name=password]" "{test_password}" + +3. Submit form + browser_click "button[type=submit]" + +4. Wait for redirect + browser_wait_for "text=Dashboard" + +5. Verify logged in state + browser_snapshot + browser_evaluate "localStorage.getItem('token')" + +6. Take screenshot + browser_take_screenshot ".test/screenshots/current/login_success_{viewport}.png" + +Expected: Login successful, redirect to dashboard +``` + +### Example: Visual Regression + +```markdown +Test: Visual Regression + +1. Navigate to page + browser_navigate "{url}" + +2. Set viewport + browser_resize "{width}x{height}" + +3. Wait for stable + browser_wait_for "text=Loaded" || browser_wait_for time:2000 + +4. Take screenshot + browser_take_screenshot ".test/screenshots/current/{test}_{viewport}.png" + +5. Compare to baseline + Use .kilo/skills/visual-testing/SKILL.md for comparison + +Expected: Diff < threshold (default 10%) +``` + +## Step 4: Report Results + +Post results to Gitea issue: + +```python +import urllib.request, json, base64 + +def post_test_results(issue_number, test_name, results): + user, pwd = "NW", "eshkink0t" + cred = base64.b64encode(f"{user}:{pwd}".encode()).decode() + + # Get token + req = urllib.request.Request( + "https://git.softuniq.eu/api/v1/users/NW/tokens", + data=json.dumps({"name": "e2e-test", "scopes": ["all"]}).encode(), + headers={'Content-Type': 'application/json', 'Authorization': f'Basic {cred}'}, + method='POST' + ) + with urllib.request.urlopen(req) as r: token = json.loads(r.read())['sha1'] + + # Post comment + body = f"""## ✅ E2E Test: {test_name} + +**URL**: {results['url']} +**Viewport**: {results['viewport']} +**Duration**: {results['duration']}ms + +### Steps Executed +{chr(10).join([f"- [{s['status']}] {s['name']}" for s in results['steps']])} + +### Screenshots +- Baseline: `{results['baseline_path']}` +- Current: `{results['current_path']}` +- Diff: `{results['diff_path']}` + +### Visual Diff +- Difference: {results['difference']}% +- Threshold: {results['threshold']}% +- Status: {'✅ PASS' if results['match'] else '❌ FAIL'} + +**Next**: {results['next_agent']} +""" + req = urllib.request.Request( + f"https://git.softuniq.eu/api/v1/repos/UniqueSoft/APAW/issues/{issue_number}/comments", + data=json.dumps({"body": body}).encode(), + headers={'Content-Type': 'application/json', 'Authorization': f'token {token}'}, + method='POST' + ) + urllib.request.urlopen(req) +``` + +## Step 5: Handle Failures + +If tests fail: + +1. **Take screenshot** of error state +2. **Get page state** with `browser_snapshot` +3. **Console logs** with `browser_console_messages` +4. **Network requests** with `browser_network_requests` +5. **Post to Gitea** with error details + +## Example Workflow + +``` +User: /e2e-test --url=https://app.example.com --test=login --viewport=desktop + +1. Invoke @browser-automation agent +2. Execute login test steps +3. Capture screenshots +4. Compare to baseline (if visual) +5. Post results to Gitea issue (if specified) +6. Return test summary +``` + +## Before Starting (MANDATORY) + +1. Check git history for similar E2E tests +2. Verify test environment URL is accessible +3. Create baseline screenshots if needed +4. Clear previous test artifacts + +## Gitea Commenting (MANDATORY) + +**You MUST post a comment to the Gitea issue after test completion.** + +Include: +- Test name and URL +- Viewport configuration +- Duration +- Step results +- Screenshot paths +- Visual diff results (if applicable) +- Pass/fail status + +## Agents Involved + +- `@browser-automation` - Executes Playwright MCP commands +- `@visual-tester` - Compares screenshots (if visual test) +- `@sdet-engineer` - Writes test cases +- `@code-skeptic` - Reviews test quality + +## Next Steps + +After E2E tests: +- `@visual-tester` - Generate visual report +- `@evaluator` - Score test coverage +- `@release-manager` - Commit test results \ No newline at end of file diff --git a/.kilo/skills/playwright/SKILL.md b/.kilo/skills/playwright/SKILL.md new file mode 100644 index 0000000..4a9360c --- /dev/null +++ b/.kilo/skills/playwright/SKILL.md @@ -0,0 +1,254 @@ +# Playwright MCP Skill + +## Purpose + +Configure and use Playwright MCP for browser automation with Kilo Code agents. + +## What is Playwright MCP? + +Playwright MCP (Model Context Protocol) is an official Microsoft server that provides browser automation capabilities to AI agents. It allows agents to: + +- Navigate web pages +- Click elements +- Fill forms +- Take screenshots +- Execute JavaScript +- Get accessibility snapshots + +## Installation + +### Option 1: Local Installation (Recommended for Development) + +```bash +# Install Node.js 18+ if not already installed +node --version # Should be 18+ + +# Install Playwright MCP +npx @playwright/mcp@latest --headless --browser chromium +``` + +### Option 2: Docker (Recommended for CI/CD) + +```bash +# Pull official image +docker pull mcr.microsoft.com/playwright:v1.58.2-noble + +# Run container +docker run -it --rm --ipc=host mcr.microsoft.com/playwright:v1.58.2-noble /bin/bash + +# Or run with Playwright MCP server +docker run -d -i --rm --init --ipc=host -p 8931:8931 \ + --name playwright-mcp \ + mcr.microsoft.com/playwright/mcp \ + cli.js --headless --browser chromium --no-sandbox --port 8931 --host 0.0.0.0 +``` + +## Configuration for Kilo Code + +### Local Mode + +```json +{ + "mcpServers": { + "playwright": { + "command": "npx", + "args": [ + "@playwright/mcp@latest", + "--headless", + "--browser", "chromium", + "--timeout-action", "3000", + "--timeout-navigation", "30000" + ] + } + } +} +``` + +### Docker Mode + +```json +{ + "mcpServers": { + "playwright": { + "command": "docker", + "args": [ + "run", "-i", "--rm", "--init", "--ipc=host", + "--pull=always", + "mcr.microsoft.com/playwright/mcp", + "--headless", "--no-sandbox" + ] + } + } +} +``` + +### Remote Service Mode + +```json +{ + "mcpServers": { + "playwright": { + "url": "http://localhost:8931/mcp" + } + } +} +``` + +## Available Tools + +| Tool | Description | Usage | +|------|-------------|-------| +| `browser_navigate` | Navigate to URL | `browser_navigate "https://example.com"` | +| `browser_click` | Click element | `browser_click "button#submit"` | +| `browser_type` | Type text | `browser_type "input[name=email]" "test@test.com"` | +| `browser_snapshot` | Get accessibility tree | `browser_snapshot` | +| `browser_take_screenshot` | Take screenshot | `browser_take_screenshot "page.png"` | +| `browser_fill_form` | Fill multiple fields | `browser_fill_form {"email": "...", "password": "..."}` | +| `browser_select_option` | Select dropdown | `browser_select_option "select#country" "US"` | +| `browser_evaluate` | Run JavaScript | `browser_evaluate "document.title"` | +| `browser_wait_for` | Wait for condition | `browser_wait_for "text=Loaded"` | +| `browser_hover` | Hover element | `browser_hover "button#menu"` | +| `browser_press_key` | Press keyboard key | `browser_press_key "Enter"` | +| `browser_navigate_back` | Go back | `browser_navigate_back` | +| `browser_resize` | Set viewport | `browser_resize "1280x720"` | +| `browser_handle_dialog` | Handle alerts | `browser_handle_dialog "accept"` | +| `browser_file_upload` | Upload files | `browser_file_upload "input[type=file]" "file.pdf"` | +| `browser_pdf_save` | Save as PDF | `browser_pdf_save "page.pdf"` | + +## User Settings + +Based on Kilo Code configuration: + +### 1. Enable Browser Automation (Playwright MCP) +- ✅ Enable this setting +- Registers Playwright MCP server in CLI backend + +### 2. Use System Chrome vs Bundled Chromium + +| Option | Advantages | Disadvantages | +|--------|------------|--------------| +| **System Chrome** | Uses installed browser | Version may vary | +| **Bundled Chromium** (Recommended) | Consistent version | Separate install | + +To use System Chrome: +```bash +npx @playwright/mcp@latest --browser chrome +``` + +To use Bundled Chromium (default): +```bash +npx @playwright/mcp@latest --browser chromium +``` + +### 3. Headless Mode (No Visible Window) + +- ✅ Enable for automation (recommended) +- Uses ~40% less memory +- Faster execution +- No visible window (good for servers) + +To run headless: +```bash +npx @playwright/mcp@latest --headless +``` + +## Example Workflow + +### E2E Test: User Registration + +```markdown +## Test: User Registration Flow + +### Steps: +1. Navigate to registration page + ``` + browser_navigate "https://app.example.com/register" + ``` + +2. Get page state (accessibility tree) + ``` + browser_snapshot + ``` + +3. Fill form fields + ``` + browser_type "input[name=username]" "newuser" + browser_type "input[name=email]" "newuser@example.com" + browser_type "input[name=password]" "SecurePass123!" + browser_type "input[name=confirm]" "SecurePass123!" + ``` + +4. Submit form + ``` + browser_click "button[type=submit]" + ``` + +5. Wait for success + ``` + browser_wait_for "text=Registration successful" + ``` + +6. Take screenshot + ``` + browser_take_screenshot ".test/screenshots/registration_success.png" + ``` + +### Expected Result: +- User registered successfully +- Redirect to dashboard +- Screenshot saved +``` + +## Performance Tips + +1. **Use headless mode** - Faster and uses less memory +2. **Set appropriate timeouts** - Lower for faster tests +3. **Reuse browser session** - Don't restart for each test +4. **Block unnecessary resources** - Images, ads, analytics +5. **Use accessibility snapshots** - Faster than screenshots + +## Troubleshooting + +### Chromium Launch Failed + +```bash +# Add --no-sandbox flag +npx @playwright/mcp@latest --no-sandbox + +# Or for Docker +docker run --cap-add=SYS_ADMIN ... +``` + +### Out of Memory + +```bash +# Increase shared memory +docker run --shm-size=2g ... + +# Or use --ipc=host +docker run --ipc=host ... +``` + +### Slow Page Loads + +```javascript +// Block images and stylesheets +await page.route('**/*.{png,jpg,jpeg,gif,svg,css}', route => route.abort()); + +// Block analytics +await page.route('**/analytics/**', route => route.abort()); +``` + +## References + +- [Playwright Documentation](https://playwright.dev/docs/intro) +- [Playwright MCP GitHub](https://github.com/microsoft/playwright-mcp) +- [Docker Images](https://mcr.microsoft.com/en-us/product/playwright/about) +- [MCP Protocol](https://modelcontextprotocol.io/) + +## Integration with Kilo Code + +1. **@browser-automation** - Uses Playwright MCP tools for E2E testing +2. **@visual-tester** - Uses screenshots from Playwright for comparison +3. **@sdet-engineer** - Writes Playwright tests using MCP tools +4. **@frontend-developer** - Uses Playwright for UI development testing \ No newline at end of file diff --git a/.kilo/skills/visual-testing/SKILL.md b/.kilo/skills/visual-testing/SKILL.md new file mode 100644 index 0000000..658d62b --- /dev/null +++ b/.kilo/skills/visual-testing/SKILL.md @@ -0,0 +1,406 @@ +# Visual Testing Skill + +## Purpose + +Configure and use visual regression testing with screenshot comparison, diff detection, and baseline management. + +## What is Visual Testing? + +Visual testing (or visual regression testing) compares screenshots to detect unexpected UI changes between versions. It ensures: + +- UI consistency across releases +- Responsive design correctness +- Visual quality of web applications +- Detection of unintended changes + +## Tools Used + +| Tool | Purpose | +|------|---------| +| **Playwright MCP** | Screenshot capture | +| **pixelmatch** | Image comparison | +| **sharp** | Image processing | + +## Directory Structure + +``` +.test/ +├── screenshots/ +│ ├── baseline/ # Reference screenshots +│ │ ├── homepage_desktop.png +│ │ ├── homepage_mobile.png +│ │ └── login_form.png +│ ├── current/ # Latest test screenshots +│ │ ├── homepage_desktop.png +│ │ ├── homepage_mobile.png +│ │ └── login_form.png +│ └── diff/ # Difference images +│ ├── homepage_desktop_diff.png +│ └── login_form_diff.png +├── reports/ +│ └── visual-report.html # HTML comparison report +└── playwright-report/ # Playwright HTML report +``` + +## Setup + +### Install Dependencies + +```bash +# Create directories +mkdir -p .test/screenshots/{baseline,current,diff} +mkdir -p .test/reports + +# Install image comparison tools +npm install --save-dev pixelmatch pngjs sharp +``` + +### Visual Comparison Script + +```typescript +// .test/scripts/visual-comparison.ts +import { PNG } from 'pngjs'; +import pixelmatch from 'pixelmatch'; +import sharp from 'sharp'; +import { readFileSync, writeFileSync, existsSync } from 'fs'; +import { join } from 'path'; + +const SCREENSHOTS_DIR = join(process.cwd(), '.test/screenshots'); +const BASELINE_DIR = join(SCREENSHOTS_DIR, 'baseline'); +const CURRENT_DIR = join(SCREENSHOTS_DIR, 'current'); +const DIFF_DIR = join(SCREENSHOTS_DIR, 'diff'); + +interface ComparisonOptions { + threshold?: number; // 0-1, default 0.1 (10%) + includeDiffImage?: boolean; +} + +interface ComparisonResult { + match: boolean; + difference: number; // percentage + pixelsChanged: number; + totalPixels: number; + diffPath?: string; +} + +export async function compareImages( + baselinePath: string, + currentPath: string, + options: ComparisonOptions = {} +): Promise { + const threshold = options.threshold ?? 0.1; + const includeDiffImage = options.includeDiffImage ?? true; + + // Read and parse images + const baselineData = readFileSync(baselinePath); + const currentData = readFileSync(currentPath); + + const img1 = PNG.sync.read(baselineData); + const img2 = PNG.sync.read(currentData); + + // Ensure same dimensions + if (img1.width !== img2.width || img1.height !== img2.height) { + throw new Error(`Image dimensions don't match: ${img1.width}x${img1.height} vs ${img2.width}x${img2.height}`); + } + + const { width, height } = img1; + const totalPixels = width * height; + + // Create diff image if requested + const diff = includeDiffImage ? new PNG({ width, height }) : null; + + // Compare pixel by pixel + const pixelsChanged = pixelmatch( + img1.data, img2.data, + diff?.data ?? null, + width, height, + { threshold: 0 } + ); + + const difference = (pixelsChanged / totalPixels) * 100; + const match = difference <= threshold; + + // Save diff image if requested + let diffPath: string | undefined; + if (includeDiffImage && diff) { + const diffName = baselinePath.replace('/baseline/', '/diff/').replace('.png', '_diff.png'); + writeFileSync(diffName, PNG.sync.write(diff)); + diffPath = diffName; + } + + return { + match, + difference, + pixelsChanged, + totalPixels, + diffPath + }; +} + +export async function runVisualTest( + testName: string, + screenshotFn: () => Promise +): Promise { + // Take current screenshot + const currentPath = await screenshotFn(); + + const baselinePath = currentPath.replace('/current/', '/baseline/'); + + // If no baseline exists, create one + if (!existsSync(baselinePath)) { + console.log(`Creating baseline: ${baselinePath}`); + const data = readFileSync(currentPath); + writeFileSync(baselinePath, data); + return { + match: true, + difference: 0, + pixelsChanged: 0, + totalPixels: 0 + }; + } + + // Compare to baseline + const result = await compareImages(baselinePath, currentPath, { + threshold: 0.1, + includeDiffImage: true + }); + + return result; +} +``` + +## Usage with Playwright MCP + +```typescript +// .test/e2e/homepage-visual.spec.ts +import { test, expect } from '@playwright/test'; +import { runVisualTest, compareImages } from '../scripts/visual-comparison'; + +test('homepage visual regression - desktop', async ({ page }) => { + // Navigate to page + await page.goto('https://example.com'); + await page.waitForSelector('[data-testid="loaded"]'); + + // Set viewport for desktop + await page.setViewportSize({ width: 1280, height: 720 }); + + // Take screenshot + const screenshotPath = '.test/screenshots/current/homepage_desktop.png'; + await page.screenshot({ path: screenshotPath, fullPage: true }); + + // Run visual test + const result = await runVisualTest('homepage_desktop', async () => { + return screenshotPath; + }); + + // Assert visual consistency + expect(result.match).toBeTruthy(); + expect(result.difference).toBeLessThan(0.1); // Less than 10% difference +}); + +test('responsive design check', async ({ page }) => { + const viewports = [ + { name: 'mobile', width: 375, height: 667 }, + { name: 'tablet', width: 768, height: 1024 }, + { name: 'desktop', width: 1280, height: 720 } + ]; + + for (const viewport of viewports) { + await page.setViewportSize(viewport); + await page.goto('https://example.com'); + + const screenshotPath = `.test/screenshots/current/homepage_${viewport.name}.png`; + await page.screenshot({ path: screenshotPath, fullPage: true }); + + const result = await compareImages( + `.test/screenshots/baseline/homepage_${viewport.name}.png`, + screenshotPath, + { threshold: 0.05 } + ); + + expect(result.match).toBeTruthy(); + } +}); +``` + +## Threshold Guidelines + +| Threshold | Difference | Use Case | +|-----------|------------|----------| +| 0% | Pixel-perfect | Logos, icons, buttons | +| 0.01-0.5% | Very strict | Important UI elements | +| 0.5-1% | Moderate | Forms, full pages | +| 1-5% | Tolerant | Dynamic content areas | +| >5% | Lenient | Ads, user content | + +## Baseline Management + +### Creating Baselines + +```bash +# Run tests in update mode +npx playwright test --update-snapshots + +# Or manually copy current to baseline +cp .test/screenshots/current/*.png .test/screenshots/baseline/ +``` + +### Updating Baselines + +```markdown +## When to Update Baselines + +1. **Intentional UI changes** - New design, layout changes +2. **New features** - Additional UI elements +3. **Bug fixes** - Correcting visual bugs + +## Approval Process + +1. Review diff image in `.test/screenshots/diff/` +2. Confirm changes are intentional +3. Run `cp current/* baseline/` to approve +4. Commit updated baselines to git +``` + +### Ignoring Dynamic Content + +```typescript +// Mask dynamic elements before screenshot +await page.evaluate(() => { + // Hide timestamps, dates, ads + document.querySelectorAll('.timestamp, .ad, [data-dynamic]').forEach(el => { + el.style.visibility = 'hidden'; + }); +}); + +await page.screenshot({ path: 'current.png' }); +``` + +## HTML Report Generation + +```typescript +// .test/scripts/generate-report.ts +import { writeFileSync } from 'fs'; + +interface TestResult { + name: string; + status: 'pass' | 'fail'; + difference: number; + baselinePath: string; + currentPath: string; + diffPath?: string; +} + +export function generateHtmlReport(results: TestResult[]): string { + const html = ` + + + + Visual Regression Report + + + +

Visual Regression Report

+

Generated: ${new Date().toISOString()}

+ ${results.map(r => ` +
+

${r.name} - ${r.status.toUpperCase()}

+

Difference: ${r.difference.toFixed(2)}%

+
+
+

Baseline

+ +
+
+

Current

+ +
+ ${r.diffPath ? ` +
+

Diff

+ +
+ ` : ''} +
+
+ `).join('')} + + + `; + + const reportPath = '.test/reports/visual-report.html'; + writeFileSync(reportPath, html); + return reportPath; +} +``` + +## CI/CD Integration + +```yaml +# .github/workflows/visual-tests.yml +name: Visual Regression Tests + +on: [push, pull_request] + +jobs: + visual-tests: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v3 + + - name: Setup Node.js + uses: actions/setup-node@v3 + with: + node-version: '18' + + - name: Install dependencies + run: npm ci + + - name: Install Playwright + run: npx playwright install --with-deps chromium + + - name: Run visual tests + run: npx playwright test --project=visual + + - name: Upload diff images + if: failure() + uses: actions/upload-artifact@v3 + with: + name: visual-diffs + path: .test/screenshots/diff/ + + - name: Upload report + if: always() + uses: actions/upload-artifact@v3 + with: + name: html-report + path: .test/reports/ +``` + +## Best Practices + +1. **Organize by viewport** - Separate baselines for mobile, tablet, desktop +2. **Use descriptive names** - `[feature]_[action]_[viewport].png` +3. **Set appropriate thresholds** - Pixel-perfect for logos, tolerant for dynamic content +4. **Mask dynamic elements** - Hide timestamps, dates, ads +5. **Review diffs carefully** - Don't blindly update baselines +6. **Commit baselines** - Include in git for reproducibility + +## Integration with Agents + +- **@browser-automation** - Captures screenshots using Playwright MCP +- **@visual-tester** - Compares screenshots to baselines +- **@sdet-engineer** - Writes visual test cases + +## References + +- [Pixelmatch](https://github.com/mapbox/pixelmatch) +- [Sharp](https://sharp.pixelplumbing.com/) +- [Playwright Screenshots](https://playwright.dev/docs/screenshots) \ No newline at end of file