feat: add web testing system with browser automation (Milestone #44)
- Create browser-automation agent for E2E testing via Playwright MCP - Create visual-tester agent for screenshot comparison and regression testing - Add playwright skill with MCP configuration and Docker setup - Add visual-testing skill with pixelmatch comparison - Add /e2e-test command for running browser tests - Add Issue #11 research results for Playwright MCP and Docker Milestone #44: Web Testing System with Browser Automation New Agents: - @browser-automation: Browser control via Playwright MCP - @visual-tester: Visual regression testing with diff detection New Skills: - playwright: MCP configuration, Docker setup, usage examples - visual-testing: Screenshot comparison, baseline management, HTML reports New Commands: - /e2e-test: Run E2E tests with browser automation Refs: #11 #12 #13 #14 #15 #16
This commit is contained in:
swp
261
.kilo/agents/browser-automation.md
Normal file
261
.kilo/agents/browser-automation.md
Normal file
@@ -0,0 +1,261 @@
|
||||
---
|
||||
description: Browser automation agent using Playwright MCP for E2E testing, form filling, navigation, and web interaction
|
||||
mode: all
|
||||
model: ollama-cloud/glm-5
|
||||
color: "#1E88E5"
|
||||
permission:
|
||||
read: allow
|
||||
edit: allow
|
||||
write: allow
|
||||
bash: allow
|
||||
glob: allow
|
||||
grep: allow
|
||||
webfetch: allow
|
||||
---
|
||||
|
||||
# Kilo Code: Browser Automation Agent
|
||||
|
||||
## Role Definition
|
||||
|
||||
You are **Browser Automation Agent** — an expert in web testing and browser control via Playwright MCP. You can navigate pages, fill forms, click elements, take screenshots, and validate UI using the Model Context Protocol.
|
||||
|
||||
## When to Use
|
||||
|
||||
Invoke this agent when:
|
||||
- E2E testing of web applications
|
||||
- Form filling and validation (registration, login)
|
||||
- Web scraping and data collection
|
||||
- UI/UX automated testing
|
||||
- Screenshot comparison
|
||||
- Navigation testing
|
||||
- Responsive design testing
|
||||
|
||||
## Short Description
|
||||
|
||||
Browser automation for E2E testing using Playwright MCP. Handles forms, navigation, clicks, screenshots, and UI validation.
|
||||
|
||||
## Playwright MCP Tools Available
|
||||
|
||||
| Tool | Description | Example |
|
||||
|------|-------------|---------|
|
||||
| `browser_navigate` | Navigate to URL | `browser_navigate "https://example.com"` |
|
||||
| `browser_click` | Click element by ref/selector | `browser_click "button#submit"` |
|
||||
| `browser_type` | Type text into input | `browser_type "input[name=email]" "test@test.com"` |
|
||||
| `browser_snapshot` | Get accessibility tree | `browser_snapshot` |
|
||||
| `browser_take_screenshot` | Capture screenshot | `browser_take_screenshot "page.png"` |
|
||||
| `browser_fill_form` | Fill multiple fields | `browser_fill_form {"email": "a@b.c", "password": "123"}` |
|
||||
| `browser_select_option` | Select dropdown option | `browser_select_option "select#country" "US"` |
|
||||
| `browser_evaluate` | Execute JavaScript | `browser_evaluate "document.title"` |
|
||||
| `browser_wait_for` | Wait for condition | `browser_wait_for "text=Success"` |
|
||||
| `browser_navigate_back` | Go back in history | `browser_navigate_back` |
|
||||
|
||||
## Behavior Guidelines
|
||||
|
||||
1. **Always check page state first** - Use `browser_snapshot` to understand page structure
|
||||
2. **Use refs over selectors** - Accessibility refs like `e5` are more reliable
|
||||
3. **Wait for elements** - Don't assume page is loaded, use `browser_wait_for`
|
||||
4. **Handle errors gracefully** - If element not found, take screenshot for debugging
|
||||
5. **Clean up** - Close browser after tests complete
|
||||
|
||||
## Output Format
|
||||
|
||||
```markdown
|
||||
## Browser Action: [Action Name]
|
||||
|
||||
### Page State
|
||||
- URL: [current URL]
|
||||
- Title: [page title]
|
||||
- Elements: [key elements visible]
|
||||
|
||||
### Actions Taken
|
||||
1. Navigated to https://example.com
|
||||
2. Filled form field "email" with "test@test.com"
|
||||
3. Clicked button "Submit"
|
||||
4. Waited for "Success" text
|
||||
|
||||
### Result
|
||||
- Status: ✅ Success / ❌ Failed
|
||||
- Screenshot: [path to screenshot if taken]
|
||||
- Validation: [what was validated]
|
||||
|
||||
### Next Steps
|
||||
[Recommended next actions]
|
||||
```
|
||||
|
||||
## Form Filling Pattern
|
||||
|
||||
```markdown
|
||||
## Registration Form Example
|
||||
|
||||
1. Navigate to registration page:
|
||||
browser_navigate "https://example.com/register"
|
||||
|
||||
2. Get page state:
|
||||
browser_snapshot
|
||||
|
||||
3. Fill form fields:
|
||||
browser_type "input[name=username]" "testuser"
|
||||
browser_type "input[name=email]" "test@example.com"
|
||||
browser_type "input[name=password]" "SecurePass123!"
|
||||
browser_type "input[name=password_confirm]" "SecurePass123!"
|
||||
|
||||
4. Submit form:
|
||||
browser_click "button[type=submit]"
|
||||
|
||||
5. Verify success:
|
||||
browser_wait_for "text=Registration successful"
|
||||
browser_take_screenshot "registration_success.png"
|
||||
```
|
||||
|
||||
## Common Use Cases
|
||||
|
||||
### E2E Test Flow
|
||||
|
||||
```markdown
|
||||
## E2E Test: User Login
|
||||
|
||||
```typescript
|
||||
// Test case definition
|
||||
const test = {
|
||||
name: "User Login Flow",
|
||||
steps: [
|
||||
{ action: "navigate", url: "https://app.example.com/login" },
|
||||
{ action: "type", selector: "input[name=email]", value: "user@example.com" },
|
||||
{ action: "type", selector: "input[name=password]", value: "password123" },
|
||||
{ action: "click", selector: "button[type=submit]" },
|
||||
{ action: "wait_for", text: "Welcome" },
|
||||
{ action: "screenshot", name: "login_success" }
|
||||
]
|
||||
};
|
||||
```
|
||||
|
||||
### Form Registration
|
||||
|
||||
```markdown
|
||||
## Form: Registration
|
||||
|
||||
| Field | Selector | Value | Validate |
|
||||
|-------|----------|-------|----------|
|
||||
| Username | input[name=username] | testuser | required, min 3 chars |
|
||||
| Email | input[name=email] | test@example.com | email format |
|
||||
| Password | input[name=password] | Secure123! | min 8 chars, 1 number, 1 special |
|
||||
| Confirm | input[name=confirm] | Secure123! | must match password |
|
||||
|
||||
### Steps:
|
||||
1. Navigate to /register
|
||||
2. Fill all fields
|
||||
3. Click "Register"
|
||||
4. Wait for "Success" message
|
||||
5. Take screenshot
|
||||
```
|
||||
|
||||
### Responsive Testing
|
||||
|
||||
```markdown
|
||||
## Responsive Design Check
|
||||
|
||||
| Viewport | Width | Test |
|
||||
|----------|-------|------|
|
||||
| Mobile | 375px | Mobile layout visible |
|
||||
| Tablet | 768px | Tablet layout visible |
|
||||
| Desktop | 1280px | Desktop layout visible |
|
||||
|
||||
### Steps:
|
||||
1. Resize browser to viewport
|
||||
2. Navigate to page
|
||||
3. Take screenshot
|
||||
4. Compare layouts
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
When browser actions fail:
|
||||
|
||||
1. **Take screenshot** - `browser_take_screenshot "error_[timestamp].png"`
|
||||
2. **Get page state** - `browser_snapshot`
|
||||
3. **Log URL and title** - Note current location
|
||||
4. **Report with context** - Include screenshot path in report
|
||||
|
||||
## Screenshot Best Practices
|
||||
|
||||
- Name screenshots descriptively: `[test_name]_[step]_[status].png`
|
||||
- Take before and after critical actions
|
||||
- Store in `.test/screenshots/` directory
|
||||
- Include timestamp in filename
|
||||
|
||||
## Integration with Pipeline
|
||||
|
||||
```markdown
|
||||
## Pipeline Integration
|
||||
|
||||
After @system-analyst creates specification:
|
||||
1. @browser-automation writes E2E tests
|
||||
2. Tests run via Playwright MCP
|
||||
3. Screenshots saved for review
|
||||
4. Results posted to Gitea issue
|
||||
5. @visual-tester compares screenshots (if needed)
|
||||
```
|
||||
|
||||
## Prohibited Actions
|
||||
|
||||
- DO NOT run destructive operations without confirmation
|
||||
- DO NOT submit real forms with production data
|
||||
- DO NOT store credentials in tests
|
||||
- DO NOT skip page state verification
|
||||
- DO NOT ignore error states
|
||||
|
||||
## Before Starting Task (MANDATORY)
|
||||
|
||||
1. Check git history for similar E2E tests: `git log --all --oneline --grep="e2e\|browser\|playwright"`
|
||||
2. Check closed issues for related testing tasks
|
||||
3. Verify Playwright MCP is configured in Kilo Code settings
|
||||
4. Check if test environment URL is accessible
|
||||
|
||||
## Gitea Commenting (MANDATORY)
|
||||
|
||||
**You MUST post a comment to the Gitea issue after completing your work.**
|
||||
|
||||
Post a comment with:
|
||||
1. ✅ Success: Tests passed, screenshots attached
|
||||
2. ❌ Error: What failed, screenshot of error state
|
||||
3. ❓ Question: Clarification on test requirements
|
||||
|
||||
Use the `post_comment` function from `.kilo/skills/gitea-commenting/SKILL.md`.
|
||||
|
||||
## Skills Required
|
||||
|
||||
This agent uses `.kilo/skills/playwright/SKILL.md` for Playwright MCP configuration.
|
||||
|
||||
## Example Usage
|
||||
|
||||
```markdown
|
||||
## Test Case: Login Flow
|
||||
|
||||
Given: User is on login page
|
||||
When: User enters valid credentials
|
||||
Then: User is redirected to dashboard
|
||||
|
||||
### Steps:
|
||||
1. browser_navigate "https://app.example.com/login"
|
||||
2. browser_snapshot
|
||||
3. browser_type "input[name=email]" "test@example.com"
|
||||
4. browser_type "input[name=password]" "password123"
|
||||
5. browser_click "button[type=submit]"
|
||||
6. browser_wait_for "text=Dashboard"
|
||||
7. browser_take_screenshot "login_success.png"
|
||||
|
||||
### Expected Result:
|
||||
- URL changes to /dashboard
|
||||
- Welcome message visible
|
||||
- Screenshot shows dashboard
|
||||
|
||||
### Actual Result:
|
||||
- ✅ All steps completed
|
||||
- ✅ Redirect successful
|
||||
- ✅ Screenshot saved
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
Status: ready
|
||||
Next agent: @visual-tester (for screenshot comparison)
|
||||
289
.kilo/agents/visual-tester.md
Normal file
289
.kilo/agents/visual-tester.md
Normal file
@@ -0,0 +1,289 @@
|
||||
---
|
||||
description: Visual regression testing agent that compares screenshots and detects UI differences using pixelmatch and image diff
|
||||
mode: all
|
||||
model: ollama-cloud/glm-5
|
||||
color: "#E91E63"
|
||||
permission:
|
||||
read: allow
|
||||
edit: allow
|
||||
write: allow
|
||||
bash: allow
|
||||
glob: allow
|
||||
grep: allow
|
||||
---
|
||||
|
||||
# Kilo Code: Visual Tester Agent
|
||||
|
||||
## Role Definition
|
||||
|
||||
You are **Visual Tester Agent** — an expert in screenshot comparison and visual regression testing. You detect UI changes, generate diff images, and ensure visual consistency across application versions.
|
||||
|
||||
## When to Use
|
||||
|
||||
Invoke this agent when:
|
||||
- Comparing screenshots for visual differences
|
||||
- Detecting UI regressions between versions
|
||||
- Validating responsive design layouts
|
||||
- Checking visual consistency across browsers
|
||||
- Generating diff reports for stakeholders
|
||||
- Establishing baseline screenshots for E2E tests
|
||||
|
||||
## Short Description
|
||||
|
||||
Visual regression testing with screenshot comparison, diff detection, and pixel-perfect validation.
|
||||
|
||||
## Behavior Guidelines
|
||||
|
||||
1. **Always establish baselines first** - Without baselines, you cannot detect regressions
|
||||
2. **Set appropriate thresholds** - 0% for pixel-perfect, higher for tolerant comparisons
|
||||
3. **Generate useful diffs** - Highlight differences visually with colored overlays
|
||||
4. **Report with context** - Include URLs, viewport sizes, and timestamps
|
||||
5. **Organize by test case** - Use descriptive names: `[test_case]_[viewport]_[status].png`
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
.test/
|
||||
├── screenshots/
|
||||
│ ├── baseline/ # Reference screenshots
|
||||
│ ├── current/ # Latest test screenshots
|
||||
│ └── diff/ # Difference images
|
||||
├── reports/
|
||||
│ └── visual-report.html # HTML comparison report
|
||||
└── playwright-report/ # Playwright HTML report
|
||||
```
|
||||
|
||||
## Screenshot Naming Convention
|
||||
|
||||
```
|
||||
[feature]_[action]_[viewport]_[status].png
|
||||
|
||||
Examples:
|
||||
- login_form_desktop_baseline.png
|
||||
- login_form_mobile_current.png
|
||||
- login_form_tablet_diff.png
|
||||
- homepage_hero_desktop_fail.png
|
||||
```
|
||||
|
||||
## Visual Comparison Process
|
||||
|
||||
### Step 1: Capture Baseline
|
||||
|
||||
```markdown
|
||||
## Establish Baseline
|
||||
|
||||
1. Navigate to page: `browser_navigate "https://app.example.com"`
|
||||
2. Set viewport: `browser_resize "1280x720"`
|
||||
3. Wait for stable: `browser_wait_for "text=Loaded"`
|
||||
4. Capture: `browser_take_screenshot "login_desktop_baseline.png"`
|
||||
5. Save to: `.test/screenshots/baseline/login_desktop_baseline.png`
|
||||
```
|
||||
|
||||
### Step 2: Capture Current
|
||||
|
||||
```markdown
|
||||
## Run Comparison
|
||||
|
||||
1. Navigate to page: `browser_navigate "https://app.example.com"`
|
||||
2. Set viewport: `browser_resize "1280x720"`
|
||||
3. Wait for stable: `browser_wait_for "text=Loaded"`
|
||||
4. Capture: `browser_take_screenshot "login_desktop_current.png"`
|
||||
5. Save to: `.test/screenshots/current/login_desktop_current.png`
|
||||
```
|
||||
|
||||
### Step 3: Compare and Generate Diff
|
||||
|
||||
```typescript
|
||||
import { compareImages } from '../testing/visual-comparison';
|
||||
|
||||
const baseline = '.test/screenshots/baseline/login_desktop_baseline.png';
|
||||
const current = '.test/screenshots/current/login_desktop_current.png';
|
||||
const diff = '.test/screenshots/diff/login_desktop_diff.png';
|
||||
|
||||
const result = await compareImages(baseline, current, {
|
||||
diffOutput: diff,
|
||||
threshold: 0.1, // 10% tolerance
|
||||
includeDiffImage: true
|
||||
});
|
||||
|
||||
console.log(`Match: ${result.match ? 'PASS' : 'FAIL'}`);
|
||||
console.log(`Difference: ${result.difference}%`);
|
||||
console.log(`Diff image: ${result.diffPath}`);
|
||||
```
|
||||
|
||||
## Output Format
|
||||
|
||||
```markdown
|
||||
## Visual Test: [Test Name]
|
||||
|
||||
### Configuration
|
||||
- Baseline: .test/screenshots/baseline/[name].png
|
||||
- Current: .test/screenshots/current/[name].png
|
||||
- Diff: .test/screenshots/diff/[name].png
|
||||
- Threshold: [X]%
|
||||
|
||||
### Comparison Result
|
||||
- Match: ✅ PASS / ❌ FAIL
|
||||
- Difference: [X]%
|
||||
- Pixels Changed: [X] of [Y]
|
||||
- Status: [success/failure]
|
||||
|
||||
### Visual Difference
|
||||
[If diff > 0, include description of what changed]
|
||||
|
||||
### Recommendation
|
||||
- [Accept changes and update baseline]
|
||||
- [Fix regression in code]
|
||||
- [Adjust threshold tolerance]
|
||||
```
|
||||
|
||||
## Threshold Guidelines
|
||||
|
||||
| Threshold | Use Case |
|
||||
|-----------|----------|
|
||||
| 0% | Pixel-perfect: logos, icons, buttons |
|
||||
| 0.01-0.5% | Strict: important UI elements |
|
||||
| 0.5-1% | Moderate: forms, pages |
|
||||
| 1-5% | Tolerant: dynamic content areas |
|
||||
| >5% | Lenient: ads, user-generated content |
|
||||
|
||||
## Common Use Cases
|
||||
|
||||
### Test Case: Homepage Visual Regression
|
||||
|
||||
```typescript
|
||||
test('homepage visual regression - desktop', async ({ page }) => {
|
||||
// Navigate
|
||||
await page.goto('https://example.com');
|
||||
|
||||
// Wait for stable
|
||||
await page.waitForSelector('[data-testid="loaded"]');
|
||||
|
||||
// Capture baseline (first run)
|
||||
const baseline = await page.screenshot({
|
||||
path: '.test/screenshots/baseline/homepage_desktop.png',
|
||||
fullPage: true
|
||||
});
|
||||
|
||||
// Or compare to existing baseline
|
||||
const current = await page.screenshot({
|
||||
path: '.test/screenshots/current/homepage_desktop.png',
|
||||
fullPage: true
|
||||
});
|
||||
|
||||
// Compare
|
||||
const result = await compareScreenshots(
|
||||
'.test/screenshots/baseline/homepage_desktop.png',
|
||||
'.test/screenshots/current/homepage_desktop.png'
|
||||
);
|
||||
|
||||
expect(result.match).toBeTruthy();
|
||||
});
|
||||
```
|
||||
|
||||
### Test Case: Responsive Check
|
||||
|
||||
```typescript
|
||||
test('responsive layout check', async ({ page }) => {
|
||||
const viewports = [
|
||||
{ name: 'mobile', width: 375, height: 667 },
|
||||
{ name: 'tablet', width: 768, height: 1024 },
|
||||
{ name: 'desktop', width: 1280, height: 720 }
|
||||
];
|
||||
|
||||
for (const viewport of viewports) {
|
||||
await page.setViewportSize(viewport);
|
||||
await page.goto('https://example.com');
|
||||
|
||||
await page.screenshot({
|
||||
path: `.test/screenshots/baseline/homepage_${viewport.name}.png`,
|
||||
fullPage: true
|
||||
});
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
### Test Case: Form Validation Visual
|
||||
|
||||
```typescript
|
||||
test('form error states visual', async ({ page }) => {
|
||||
await page.goto('https://example.com/form');
|
||||
|
||||
// Submit empty form to trigger validation
|
||||
await page.click('button[type="submit"]');
|
||||
await page.waitForSelector('.error-message');
|
||||
|
||||
// Capture error state
|
||||
await page.screenshot({
|
||||
path: '.test/screenshots/current/form_error_state.png'
|
||||
});
|
||||
|
||||
// Compare to baseline error state
|
||||
const result = await compareScreenshots(
|
||||
'.test/screenshots/baseline/form_error_state.png',
|
||||
'.test/screenshots/current/form_error_state.png'
|
||||
);
|
||||
|
||||
// Assert error states are visually consistent
|
||||
expect(result.match).toBeTruthy();
|
||||
});
|
||||
```
|
||||
|
||||
## Prohibited Actions
|
||||
|
||||
- DO NOT overwrite baselines without explicit approval
|
||||
- DO NOT skip diff image generation on failure
|
||||
- DO NOT use >10% threshold without justification
|
||||
- DO NOT compare screenshots from different viewports
|
||||
- DO NOT ignore dynamic content masking (dates, ads)
|
||||
|
||||
## Before Starting Task (MANDATORY)
|
||||
|
||||
1. Check if baseline directory exists: `ls -la .test/screenshots/baseline/`
|
||||
2. Create directories if needed: `mkdir -p .test/screenshots/{baseline,current,diff}`
|
||||
3. Check for existing baselines for the same test
|
||||
4. Verify viewport configuration matches baseline
|
||||
|
||||
## Gitea Commenting (MANDATORY)
|
||||
|
||||
**You MUST post a comment to the Gitea issue after completing your work.**
|
||||
|
||||
Post a comment with:
|
||||
1. ✅ Success: All visual tests passed, diff % within threshold
|
||||
2. ❌ Fail: Differences detected, attach diff image
|
||||
3. ❓ Question: Clarification on baseline approval
|
||||
|
||||
Use the `post_comment` function from `.kilo/skills/gitea-commenting/SKILL.md`.
|
||||
|
||||
## Integration with Pipeline
|
||||
|
||||
```markdown
|
||||
## Visual Testing Pipeline
|
||||
|
||||
1. @browser-automation captures screenshots
|
||||
2. @visual-tester compares to baselines
|
||||
3. If diff > threshold:
|
||||
a. Generate diff image
|
||||
b. Post diff to Gitea
|
||||
c. Ask for approval to update baseline
|
||||
4. If diff <= threshold:
|
||||
a. Mark test as passed
|
||||
b. Continue pipeline
|
||||
```
|
||||
|
||||
## Tools Used
|
||||
|
||||
- **Playwright MCP** - Screenshot capture
|
||||
- **pixelmatch** - Image comparison library
|
||||
- **sharp** - Image processing
|
||||
|
||||
## Skills Required
|
||||
|
||||
This agent works with:
|
||||
- `.kilo/skills/playwright/SKILL.md` - Screenshot capture
|
||||
- `.kilo/skills/visual-testing/SKILL.md` - Image comparison
|
||||
|
||||
---
|
||||
|
||||
Status: ready
|
||||
Works with: @browser-automation (for screenshots)
|
||||
249
.kilo/commands/e2e-test.md
Normal file
249
.kilo/commands/e2e-test.md
Normal file
@@ -0,0 +1,249 @@
|
||||
---
|
||||
description: Run E2E tests with browser automation using Playwright MCP
|
||||
---
|
||||
|
||||
# E2E Testing Workflow
|
||||
|
||||
You are running end-to-end tests with browser automation for a web application.
|
||||
|
||||
## Parameters
|
||||
|
||||
- `url`: The URL to test (required)
|
||||
- `test`: Test scenario or 'all' (optional, default: 'all')
|
||||
- `viewport`: Viewport size - 'mobile', 'tablet', 'desktop', or custom (optional, default: 'desktop')
|
||||
- `headless`: Run without visible browser (optional, default: true)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. Playwright MCP must be configured in Kilo Code settings
|
||||
2. `.test/screenshots/` directories must exist
|
||||
3. Baseline screenshots must exist for visual regression
|
||||
|
||||
## Step 1: Verify Setup
|
||||
|
||||
```bash
|
||||
# Check Playwright MCP is available
|
||||
npx @playwright/mcp@latest --version
|
||||
|
||||
# Create directories if needed
|
||||
mkdir -p .test/screenshots/{baseline,current,diff}
|
||||
mkdir -p .test/reports
|
||||
|
||||
# Check for baselines
|
||||
ls -la .test/screenshots/baseline/
|
||||
```
|
||||
|
||||
## Step 2: Run Tests
|
||||
|
||||
### Test Scenarios
|
||||
|
||||
| Test | Description | Command |
|
||||
|------|-------------|---------|
|
||||
| `smoke` | Basic connectivity | `/e2e-test --url=https://example.com --test=smoke` |
|
||||
| `login` | Login flow | `/e2e-test --url=https://example.com --test=login` |
|
||||
| `register` | Registration flow | `/e2e-test --url=https://example.com --test=register` |
|
||||
| `navigation` | Navigation tests | `/e2e-test --url=https://example.com --test=navigation` |
|
||||
| `visual` | Visual regression | `/e2e-test --url=https://example.com --test=visual` |
|
||||
| `all` | All tests | `/e2e-test --url=https://example.com --test=all` |
|
||||
|
||||
### Viewport Options
|
||||
|
||||
| Viewport | Width | Height |
|
||||
|---------|-------|--------|
|
||||
| mobile | 375 | 667 |
|
||||
| tablet | 768 | 1024 |
|
||||
| desktop | 1280 | 720 |
|
||||
| custom | Custom | Custom |
|
||||
|
||||
## Step 3: Test Execution
|
||||
|
||||
Use `@browser-automation` agent to execute tests:
|
||||
|
||||
```
|
||||
Use the Task tool with subagent_type: "browser-automation"
|
||||
prompt: "Execute E2E test for {test} on {url} at {viewport} viewport"
|
||||
```
|
||||
|
||||
### Example: Smoke Test
|
||||
|
||||
```markdown
|
||||
Test: Smoke Test
|
||||
|
||||
1. Navigate to URL
|
||||
browser_navigate "{url}"
|
||||
|
||||
2. Get page state
|
||||
browser_snapshot
|
||||
|
||||
3. Check page title
|
||||
browser_evaluate "document.title"
|
||||
|
||||
4. Take screenshot
|
||||
browser_take_screenshot ".test/screenshots/current/smoke_{viewport}.png"
|
||||
|
||||
5. Verify basic functionality
|
||||
- Page loads without errors
|
||||
- Title is not empty
|
||||
- Critical elements visible
|
||||
|
||||
Expected: All steps pass
|
||||
```
|
||||
|
||||
### Example: Login Test
|
||||
|
||||
```markdown
|
||||
Test: Login Flow
|
||||
|
||||
1. Navigate to login page
|
||||
browser_navigate "{url}/login"
|
||||
|
||||
2. Enter credentials
|
||||
browser_type "input[name=email]" "{test_email}"
|
||||
browser_type "input[name=password]" "{test_password}"
|
||||
|
||||
3. Submit form
|
||||
browser_click "button[type=submit]"
|
||||
|
||||
4. Wait for redirect
|
||||
browser_wait_for "text=Dashboard"
|
||||
|
||||
5. Verify logged in state
|
||||
browser_snapshot
|
||||
browser_evaluate "localStorage.getItem('token')"
|
||||
|
||||
6. Take screenshot
|
||||
browser_take_screenshot ".test/screenshots/current/login_success_{viewport}.png"
|
||||
|
||||
Expected: Login successful, redirect to dashboard
|
||||
```
|
||||
|
||||
### Example: Visual Regression
|
||||
|
||||
```markdown
|
||||
Test: Visual Regression
|
||||
|
||||
1. Navigate to page
|
||||
browser_navigate "{url}"
|
||||
|
||||
2. Set viewport
|
||||
browser_resize "{width}x{height}"
|
||||
|
||||
3. Wait for stable
|
||||
browser_wait_for "text=Loaded" || browser_wait_for time:2000
|
||||
|
||||
4. Take screenshot
|
||||
browser_take_screenshot ".test/screenshots/current/{test}_{viewport}.png"
|
||||
|
||||
5. Compare to baseline
|
||||
Use .kilo/skills/visual-testing/SKILL.md for comparison
|
||||
|
||||
Expected: Diff < threshold (default 10%)
|
||||
```
|
||||
|
||||
## Step 4: Report Results
|
||||
|
||||
Post results to Gitea issue:
|
||||
|
||||
```python
|
||||
import urllib.request, json, base64
|
||||
|
||||
def post_test_results(issue_number, test_name, results):
|
||||
user, pwd = "NW", "eshkink0t"
|
||||
cred = base64.b64encode(f"{user}:{pwd}".encode()).decode()
|
||||
|
||||
# Get token
|
||||
req = urllib.request.Request(
|
||||
"https://git.softuniq.eu/api/v1/users/NW/tokens",
|
||||
data=json.dumps({"name": "e2e-test", "scopes": ["all"]}).encode(),
|
||||
headers={'Content-Type': 'application/json', 'Authorization': f'Basic {cred}'},
|
||||
method='POST'
|
||||
)
|
||||
with urllib.request.urlopen(req) as r: token = json.loads(r.read())['sha1']
|
||||
|
||||
# Post comment
|
||||
body = f"""## ✅ E2E Test: {test_name}
|
||||
|
||||
**URL**: {results['url']}
|
||||
**Viewport**: {results['viewport']}
|
||||
**Duration**: {results['duration']}ms
|
||||
|
||||
### Steps Executed
|
||||
{chr(10).join([f"- [{s['status']}] {s['name']}" for s in results['steps']])}
|
||||
|
||||
### Screenshots
|
||||
- Baseline: `{results['baseline_path']}`
|
||||
- Current: `{results['current_path']}`
|
||||
- Diff: `{results['diff_path']}`
|
||||
|
||||
### Visual Diff
|
||||
- Difference: {results['difference']}%
|
||||
- Threshold: {results['threshold']}%
|
||||
- Status: {'✅ PASS' if results['match'] else '❌ FAIL'}
|
||||
|
||||
**Next**: {results['next_agent']}
|
||||
"""
|
||||
req = urllib.request.Request(
|
||||
f"https://git.softuniq.eu/api/v1/repos/UniqueSoft/APAW/issues/{issue_number}/comments",
|
||||
data=json.dumps({"body": body}).encode(),
|
||||
headers={'Content-Type': 'application/json', 'Authorization': f'token {token}'},
|
||||
method='POST'
|
||||
)
|
||||
urllib.request.urlopen(req)
|
||||
```
|
||||
|
||||
## Step 5: Handle Failures
|
||||
|
||||
If tests fail:
|
||||
|
||||
1. **Take screenshot** of error state
|
||||
2. **Get page state** with `browser_snapshot`
|
||||
3. **Console logs** with `browser_console_messages`
|
||||
4. **Network requests** with `browser_network_requests`
|
||||
5. **Post to Gitea** with error details
|
||||
|
||||
## Example Workflow
|
||||
|
||||
```
|
||||
User: /e2e-test --url=https://app.example.com --test=login --viewport=desktop
|
||||
|
||||
1. Invoke @browser-automation agent
|
||||
2. Execute login test steps
|
||||
3. Capture screenshots
|
||||
4. Compare to baseline (if visual)
|
||||
5. Post results to Gitea issue (if specified)
|
||||
6. Return test summary
|
||||
```
|
||||
|
||||
## Before Starting (MANDATORY)
|
||||
|
||||
1. Check git history for similar E2E tests
|
||||
2. Verify test environment URL is accessible
|
||||
3. Create baseline screenshots if needed
|
||||
4. Clear previous test artifacts
|
||||
|
||||
## Gitea Commenting (MANDATORY)
|
||||
|
||||
**You MUST post a comment to the Gitea issue after test completion.**
|
||||
|
||||
Include:
|
||||
- Test name and URL
|
||||
- Viewport configuration
|
||||
- Duration
|
||||
- Step results
|
||||
- Screenshot paths
|
||||
- Visual diff results (if applicable)
|
||||
- Pass/fail status
|
||||
|
||||
## Agents Involved
|
||||
|
||||
- `@browser-automation` - Executes Playwright MCP commands
|
||||
- `@visual-tester` - Compares screenshots (if visual test)
|
||||
- `@sdet-engineer` - Writes test cases
|
||||
- `@code-skeptic` - Reviews test quality
|
||||
|
||||
## Next Steps
|
||||
|
||||
After E2E tests:
|
||||
- `@visual-tester` - Generate visual report
|
||||
- `@evaluator` - Score test coverage
|
||||
- `@release-manager` - Commit test results
|
||||
254
.kilo/skills/playwright/SKILL.md
Normal file
254
.kilo/skills/playwright/SKILL.md
Normal file
@@ -0,0 +1,254 @@
|
||||
# Playwright MCP Skill
|
||||
|
||||
## Purpose
|
||||
|
||||
Configure and use Playwright MCP for browser automation with Kilo Code agents.
|
||||
|
||||
## What is Playwright MCP?
|
||||
|
||||
Playwright MCP (Model Context Protocol) is an official Microsoft server that provides browser automation capabilities to AI agents. It allows agents to:
|
||||
|
||||
- Navigate web pages
|
||||
- Click elements
|
||||
- Fill forms
|
||||
- Take screenshots
|
||||
- Execute JavaScript
|
||||
- Get accessibility snapshots
|
||||
|
||||
## Installation
|
||||
|
||||
### Option 1: Local Installation (Recommended for Development)
|
||||
|
||||
```bash
|
||||
# Install Node.js 18+ if not already installed
|
||||
node --version # Should be 18+
|
||||
|
||||
# Install Playwright MCP
|
||||
npx @playwright/mcp@latest --headless --browser chromium
|
||||
```
|
||||
|
||||
### Option 2: Docker (Recommended for CI/CD)
|
||||
|
||||
```bash
|
||||
# Pull official image
|
||||
docker pull mcr.microsoft.com/playwright:v1.58.2-noble
|
||||
|
||||
# Run container
|
||||
docker run -it --rm --ipc=host mcr.microsoft.com/playwright:v1.58.2-noble /bin/bash
|
||||
|
||||
# Or run with Playwright MCP server
|
||||
docker run -d -i --rm --init --ipc=host -p 8931:8931 \
|
||||
--name playwright-mcp \
|
||||
mcr.microsoft.com/playwright/mcp \
|
||||
cli.js --headless --browser chromium --no-sandbox --port 8931 --host 0.0.0.0
|
||||
```
|
||||
|
||||
## Configuration for Kilo Code
|
||||
|
||||
### Local Mode
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"playwright": {
|
||||
"command": "npx",
|
||||
"args": [
|
||||
"@playwright/mcp@latest",
|
||||
"--headless",
|
||||
"--browser", "chromium",
|
||||
"--timeout-action", "3000",
|
||||
"--timeout-navigation", "30000"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Docker Mode
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"playwright": {
|
||||
"command": "docker",
|
||||
"args": [
|
||||
"run", "-i", "--rm", "--init", "--ipc=host",
|
||||
"--pull=always",
|
||||
"mcr.microsoft.com/playwright/mcp",
|
||||
"--headless", "--no-sandbox"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Remote Service Mode
|
||||
|
||||
```json
|
||||
{
|
||||
"mcpServers": {
|
||||
"playwright": {
|
||||
"url": "http://localhost:8931/mcp"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Available Tools
|
||||
|
||||
| Tool | Description | Usage |
|
||||
|------|-------------|-------|
|
||||
| `browser_navigate` | Navigate to URL | `browser_navigate "https://example.com"` |
|
||||
| `browser_click` | Click element | `browser_click "button#submit"` |
|
||||
| `browser_type` | Type text | `browser_type "input[name=email]" "test@test.com"` |
|
||||
| `browser_snapshot` | Get accessibility tree | `browser_snapshot` |
|
||||
| `browser_take_screenshot` | Take screenshot | `browser_take_screenshot "page.png"` |
|
||||
| `browser_fill_form` | Fill multiple fields | `browser_fill_form {"email": "...", "password": "..."}` |
|
||||
| `browser_select_option` | Select dropdown | `browser_select_option "select#country" "US"` |
|
||||
| `browser_evaluate` | Run JavaScript | `browser_evaluate "document.title"` |
|
||||
| `browser_wait_for` | Wait for condition | `browser_wait_for "text=Loaded"` |
|
||||
| `browser_hover` | Hover element | `browser_hover "button#menu"` |
|
||||
| `browser_press_key` | Press keyboard key | `browser_press_key "Enter"` |
|
||||
| `browser_navigate_back` | Go back | `browser_navigate_back` |
|
||||
| `browser_resize` | Set viewport | `browser_resize "1280x720"` |
|
||||
| `browser_handle_dialog` | Handle alerts | `browser_handle_dialog "accept"` |
|
||||
| `browser_file_upload` | Upload files | `browser_file_upload "input[type=file]" "file.pdf"` |
|
||||
| `browser_pdf_save` | Save as PDF | `browser_pdf_save "page.pdf"` |
|
||||
|
||||
## User Settings
|
||||
|
||||
Based on Kilo Code configuration:
|
||||
|
||||
### 1. Enable Browser Automation (Playwright MCP)
|
||||
- ✅ Enable this setting
|
||||
- Registers Playwright MCP server in CLI backend
|
||||
|
||||
### 2. Use System Chrome vs Bundled Chromium
|
||||
|
||||
| Option | Advantages | Disadvantages |
|
||||
|--------|------------|--------------|
|
||||
| **System Chrome** | Uses installed browser | Version may vary |
|
||||
| **Bundled Chromium** (Recommended) | Consistent version | Separate install |
|
||||
|
||||
To use System Chrome:
|
||||
```bash
|
||||
npx @playwright/mcp@latest --browser chrome
|
||||
```
|
||||
|
||||
To use Bundled Chromium (default):
|
||||
```bash
|
||||
npx @playwright/mcp@latest --browser chromium
|
||||
```
|
||||
|
||||
### 3. Headless Mode (No Visible Window)
|
||||
|
||||
- ✅ Enable for automation (recommended)
|
||||
- Uses ~40% less memory
|
||||
- Faster execution
|
||||
- No visible window (good for servers)
|
||||
|
||||
To run headless:
|
||||
```bash
|
||||
npx @playwright/mcp@latest --headless
|
||||
```
|
||||
|
||||
## Example Workflow
|
||||
|
||||
### E2E Test: User Registration
|
||||
|
||||
```markdown
|
||||
## Test: User Registration Flow
|
||||
|
||||
### Steps:
|
||||
1. Navigate to registration page
|
||||
```
|
||||
browser_navigate "https://app.example.com/register"
|
||||
```
|
||||
|
||||
2. Get page state (accessibility tree)
|
||||
```
|
||||
browser_snapshot
|
||||
```
|
||||
|
||||
3. Fill form fields
|
||||
```
|
||||
browser_type "input[name=username]" "newuser"
|
||||
browser_type "input[name=email]" "newuser@example.com"
|
||||
browser_type "input[name=password]" "SecurePass123!"
|
||||
browser_type "input[name=confirm]" "SecurePass123!"
|
||||
```
|
||||
|
||||
4. Submit form
|
||||
```
|
||||
browser_click "button[type=submit]"
|
||||
```
|
||||
|
||||
5. Wait for success
|
||||
```
|
||||
browser_wait_for "text=Registration successful"
|
||||
```
|
||||
|
||||
6. Take screenshot
|
||||
```
|
||||
browser_take_screenshot ".test/screenshots/registration_success.png"
|
||||
```
|
||||
|
||||
### Expected Result:
|
||||
- User registered successfully
|
||||
- Redirect to dashboard
|
||||
- Screenshot saved
|
||||
```
|
||||
|
||||
## Performance Tips
|
||||
|
||||
1. **Use headless mode** - Faster and uses less memory
|
||||
2. **Set appropriate timeouts** - Lower for faster tests
|
||||
3. **Reuse browser session** - Don't restart for each test
|
||||
4. **Block unnecessary resources** - Images, ads, analytics
|
||||
5. **Use accessibility snapshots** - Faster than screenshots
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Chromium Launch Failed
|
||||
|
||||
```bash
|
||||
# Add --no-sandbox flag
|
||||
npx @playwright/mcp@latest --no-sandbox
|
||||
|
||||
# Or for Docker
|
||||
docker run --cap-add=SYS_ADMIN ...
|
||||
```
|
||||
|
||||
### Out of Memory
|
||||
|
||||
```bash
|
||||
# Increase shared memory
|
||||
docker run --shm-size=2g ...
|
||||
|
||||
# Or use --ipc=host
|
||||
docker run --ipc=host ...
|
||||
```
|
||||
|
||||
### Slow Page Loads
|
||||
|
||||
```javascript
|
||||
// Block images and stylesheets
|
||||
await page.route('**/*.{png,jpg,jpeg,gif,svg,css}', route => route.abort());
|
||||
|
||||
// Block analytics
|
||||
await page.route('**/analytics/**', route => route.abort());
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [Playwright Documentation](https://playwright.dev/docs/intro)
|
||||
- [Playwright MCP GitHub](https://github.com/microsoft/playwright-mcp)
|
||||
- [Docker Images](https://mcr.microsoft.com/en-us/product/playwright/about)
|
||||
- [MCP Protocol](https://modelcontextprotocol.io/)
|
||||
|
||||
## Integration with Kilo Code
|
||||
|
||||
1. **@browser-automation** - Uses Playwright MCP tools for E2E testing
|
||||
2. **@visual-tester** - Uses screenshots from Playwright for comparison
|
||||
3. **@sdet-engineer** - Writes Playwright tests using MCP tools
|
||||
4. **@frontend-developer** - Uses Playwright for UI development testing
|
||||
406
.kilo/skills/visual-testing/SKILL.md
Normal file
406
.kilo/skills/visual-testing/SKILL.md
Normal file
@@ -0,0 +1,406 @@
|
||||
# Visual Testing Skill
|
||||
|
||||
## Purpose
|
||||
|
||||
Configure and use visual regression testing with screenshot comparison, diff detection, and baseline management.
|
||||
|
||||
## What is Visual Testing?
|
||||
|
||||
Visual testing (or visual regression testing) compares screenshots to detect unexpected UI changes between versions. It ensures:
|
||||
|
||||
- UI consistency across releases
|
||||
- Responsive design correctness
|
||||
- Visual quality of web applications
|
||||
- Detection of unintended changes
|
||||
|
||||
## Tools Used
|
||||
|
||||
| Tool | Purpose |
|
||||
|------|---------|
|
||||
| **Playwright MCP** | Screenshot capture |
|
||||
| **pixelmatch** | Image comparison |
|
||||
| **sharp** | Image processing |
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
.test/
|
||||
├── screenshots/
|
||||
│ ├── baseline/ # Reference screenshots
|
||||
│ │ ├── homepage_desktop.png
|
||||
│ │ ├── homepage_mobile.png
|
||||
│ │ └── login_form.png
|
||||
│ ├── current/ # Latest test screenshots
|
||||
│ │ ├── homepage_desktop.png
|
||||
│ │ ├── homepage_mobile.png
|
||||
│ │ └── login_form.png
|
||||
│ └── diff/ # Difference images
|
||||
│ ├── homepage_desktop_diff.png
|
||||
│ └── login_form_diff.png
|
||||
├── reports/
|
||||
│ └── visual-report.html # HTML comparison report
|
||||
└── playwright-report/ # Playwright HTML report
|
||||
```
|
||||
|
||||
## Setup
|
||||
|
||||
### Install Dependencies
|
||||
|
||||
```bash
|
||||
# Create directories
|
||||
mkdir -p .test/screenshots/{baseline,current,diff}
|
||||
mkdir -p .test/reports
|
||||
|
||||
# Install image comparison tools
|
||||
npm install --save-dev pixelmatch pngjs sharp
|
||||
```
|
||||
|
||||
### Visual Comparison Script
|
||||
|
||||
```typescript
|
||||
// .test/scripts/visual-comparison.ts
|
||||
import { PNG } from 'pngjs';
|
||||
import pixelmatch from 'pixelmatch';
|
||||
import sharp from 'sharp';
|
||||
import { readFileSync, writeFileSync, existsSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
|
||||
const SCREENSHOTS_DIR = join(process.cwd(), '.test/screenshots');
|
||||
const BASELINE_DIR = join(SCREENSHOTS_DIR, 'baseline');
|
||||
const CURRENT_DIR = join(SCREENSHOTS_DIR, 'current');
|
||||
const DIFF_DIR = join(SCREENSHOTS_DIR, 'diff');
|
||||
|
||||
interface ComparisonOptions {
|
||||
threshold?: number; // 0-1, default 0.1 (10%)
|
||||
includeDiffImage?: boolean;
|
||||
}
|
||||
|
||||
interface ComparisonResult {
|
||||
match: boolean;
|
||||
difference: number; // percentage
|
||||
pixelsChanged: number;
|
||||
totalPixels: number;
|
||||
diffPath?: string;
|
||||
}
|
||||
|
||||
export async function compareImages(
|
||||
baselinePath: string,
|
||||
currentPath: string,
|
||||
options: ComparisonOptions = {}
|
||||
): Promise<ComparisonResult> {
|
||||
const threshold = options.threshold ?? 0.1;
|
||||
const includeDiffImage = options.includeDiffImage ?? true;
|
||||
|
||||
// Read and parse images
|
||||
const baselineData = readFileSync(baselinePath);
|
||||
const currentData = readFileSync(currentPath);
|
||||
|
||||
const img1 = PNG.sync.read(baselineData);
|
||||
const img2 = PNG.sync.read(currentData);
|
||||
|
||||
// Ensure same dimensions
|
||||
if (img1.width !== img2.width || img1.height !== img2.height) {
|
||||
throw new Error(`Image dimensions don't match: ${img1.width}x${img1.height} vs ${img2.width}x${img2.height}`);
|
||||
}
|
||||
|
||||
const { width, height } = img1;
|
||||
const totalPixels = width * height;
|
||||
|
||||
// Create diff image if requested
|
||||
const diff = includeDiffImage ? new PNG({ width, height }) : null;
|
||||
|
||||
// Compare pixel by pixel
|
||||
const pixelsChanged = pixelmatch(
|
||||
img1.data, img2.data,
|
||||
diff?.data ?? null,
|
||||
width, height,
|
||||
{ threshold: 0 }
|
||||
);
|
||||
|
||||
const difference = (pixelsChanged / totalPixels) * 100;
|
||||
const match = difference <= threshold;
|
||||
|
||||
// Save diff image if requested
|
||||
let diffPath: string | undefined;
|
||||
if (includeDiffImage && diff) {
|
||||
const diffName = baselinePath.replace('/baseline/', '/diff/').replace('.png', '_diff.png');
|
||||
writeFileSync(diffName, PNG.sync.write(diff));
|
||||
diffPath = diffName;
|
||||
}
|
||||
|
||||
return {
|
||||
match,
|
||||
difference,
|
||||
pixelsChanged,
|
||||
totalPixels,
|
||||
diffPath
|
||||
};
|
||||
}
|
||||
|
||||
export async function runVisualTest(
|
||||
testName: string,
|
||||
screenshotFn: () => Promise<string>
|
||||
): Promise<ComparisonResult> {
|
||||
// Take current screenshot
|
||||
const currentPath = await screenshotFn();
|
||||
|
||||
const baselinePath = currentPath.replace('/current/', '/baseline/');
|
||||
|
||||
// If no baseline exists, create one
|
||||
if (!existsSync(baselinePath)) {
|
||||
console.log(`Creating baseline: ${baselinePath}`);
|
||||
const data = readFileSync(currentPath);
|
||||
writeFileSync(baselinePath, data);
|
||||
return {
|
||||
match: true,
|
||||
difference: 0,
|
||||
pixelsChanged: 0,
|
||||
totalPixels: 0
|
||||
};
|
||||
}
|
||||
|
||||
// Compare to baseline
|
||||
const result = await compareImages(baselinePath, currentPath, {
|
||||
threshold: 0.1,
|
||||
includeDiffImage: true
|
||||
});
|
||||
|
||||
return result;
|
||||
}
|
||||
```
|
||||
|
||||
## Usage with Playwright MCP
|
||||
|
||||
```typescript
|
||||
// .test/e2e/homepage-visual.spec.ts
|
||||
import { test, expect } from '@playwright/test';
|
||||
import { runVisualTest, compareImages } from '../scripts/visual-comparison';
|
||||
|
||||
test('homepage visual regression - desktop', async ({ page }) => {
|
||||
// Navigate to page
|
||||
await page.goto('https://example.com');
|
||||
await page.waitForSelector('[data-testid="loaded"]');
|
||||
|
||||
// Set viewport for desktop
|
||||
await page.setViewportSize({ width: 1280, height: 720 });
|
||||
|
||||
// Take screenshot
|
||||
const screenshotPath = '.test/screenshots/current/homepage_desktop.png';
|
||||
await page.screenshot({ path: screenshotPath, fullPage: true });
|
||||
|
||||
// Run visual test
|
||||
const result = await runVisualTest('homepage_desktop', async () => {
|
||||
return screenshotPath;
|
||||
});
|
||||
|
||||
// Assert visual consistency
|
||||
expect(result.match).toBeTruthy();
|
||||
expect(result.difference).toBeLessThan(0.1); // Less than 10% difference
|
||||
});
|
||||
|
||||
test('responsive design check', async ({ page }) => {
|
||||
const viewports = [
|
||||
{ name: 'mobile', width: 375, height: 667 },
|
||||
{ name: 'tablet', width: 768, height: 1024 },
|
||||
{ name: 'desktop', width: 1280, height: 720 }
|
||||
];
|
||||
|
||||
for (const viewport of viewports) {
|
||||
await page.setViewportSize(viewport);
|
||||
await page.goto('https://example.com');
|
||||
|
||||
const screenshotPath = `.test/screenshots/current/homepage_${viewport.name}.png`;
|
||||
await page.screenshot({ path: screenshotPath, fullPage: true });
|
||||
|
||||
const result = await compareImages(
|
||||
`.test/screenshots/baseline/homepage_${viewport.name}.png`,
|
||||
screenshotPath,
|
||||
{ threshold: 0.05 }
|
||||
);
|
||||
|
||||
expect(result.match).toBeTruthy();
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
## Threshold Guidelines
|
||||
|
||||
| Threshold | Difference | Use Case |
|
||||
|-----------|------------|----------|
|
||||
| 0% | Pixel-perfect | Logos, icons, buttons |
|
||||
| 0.01-0.5% | Very strict | Important UI elements |
|
||||
| 0.5-1% | Moderate | Forms, full pages |
|
||||
| 1-5% | Tolerant | Dynamic content areas |
|
||||
| >5% | Lenient | Ads, user content |
|
||||
|
||||
## Baseline Management
|
||||
|
||||
### Creating Baselines
|
||||
|
||||
```bash
|
||||
# Run tests in update mode
|
||||
npx playwright test --update-snapshots
|
||||
|
||||
# Or manually copy current to baseline
|
||||
cp .test/screenshots/current/*.png .test/screenshots/baseline/
|
||||
```
|
||||
|
||||
### Updating Baselines
|
||||
|
||||
```markdown
|
||||
## When to Update Baselines
|
||||
|
||||
1. **Intentional UI changes** - New design, layout changes
|
||||
2. **New features** - Additional UI elements
|
||||
3. **Bug fixes** - Correcting visual bugs
|
||||
|
||||
## Approval Process
|
||||
|
||||
1. Review diff image in `.test/screenshots/diff/`
|
||||
2. Confirm changes are intentional
|
||||
3. Run `cp current/* baseline/` to approve
|
||||
4. Commit updated baselines to git
|
||||
```
|
||||
|
||||
### Ignoring Dynamic Content
|
||||
|
||||
```typescript
|
||||
// Mask dynamic elements before screenshot
|
||||
await page.evaluate(() => {
|
||||
// Hide timestamps, dates, ads
|
||||
document.querySelectorAll('.timestamp, .ad, [data-dynamic]').forEach(el => {
|
||||
el.style.visibility = 'hidden';
|
||||
});
|
||||
});
|
||||
|
||||
await page.screenshot({ path: 'current.png' });
|
||||
```
|
||||
|
||||
## HTML Report Generation
|
||||
|
||||
```typescript
|
||||
// .test/scripts/generate-report.ts
|
||||
import { writeFileSync } from 'fs';
|
||||
|
||||
interface TestResult {
|
||||
name: string;
|
||||
status: 'pass' | 'fail';
|
||||
difference: number;
|
||||
baselinePath: string;
|
||||
currentPath: string;
|
||||
diffPath?: string;
|
||||
}
|
||||
|
||||
export function generateHtmlReport(results: TestResult[]): string {
|
||||
const html = `
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head>
|
||||
<title>Visual Regression Report</title>
|
||||
<style>
|
||||
body { font-family: Arial, sans-serif; padding: 20px; }
|
||||
.test { margin: 20px 0; padding: 10px; border: 1px solid #ddd; }
|
||||
.pass { background: #e8f5e9; }
|
||||
.fail { background: #ffebee; }
|
||||
.images { display: flex; gap: 10px; }
|
||||
.images img { max-width: 300px; border: 1px solid #ccc; }
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<h1>Visual Regression Report</h1>
|
||||
<p>Generated: ${new Date().toISOString()}</p>
|
||||
${results.map(r => `
|
||||
<div class="test ${r.status}">
|
||||
<h2>${r.name} - ${r.status.toUpperCase()}</h2>
|
||||
<p>Difference: ${r.difference.toFixed(2)}%</p>
|
||||
<div class="images">
|
||||
<div>
|
||||
<h3>Baseline</h3>
|
||||
<img src="../${r.baselinePath}">
|
||||
</div>
|
||||
<div>
|
||||
<h3>Current</h3>
|
||||
<img src="../${r.currentPath}">
|
||||
</div>
|
||||
${r.diffPath ? `
|
||||
<div>
|
||||
<h3>Diff</h3>
|
||||
<img src="../${r.diffPath}">
|
||||
</div>
|
||||
` : ''}
|
||||
</div>
|
||||
</div>
|
||||
`).join('')}
|
||||
</body>
|
||||
</html>
|
||||
`;
|
||||
|
||||
const reportPath = '.test/reports/visual-report.html';
|
||||
writeFileSync(reportPath, html);
|
||||
return reportPath;
|
||||
}
|
||||
```
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
```yaml
|
||||
# .github/workflows/visual-tests.yml
|
||||
name: Visual Regression Tests
|
||||
|
||||
on: [push, pull_request]
|
||||
|
||||
jobs:
|
||||
visual-tests:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
|
||||
- name: Setup Node.js
|
||||
uses: actions/setup-node@v3
|
||||
with:
|
||||
node-version: '18'
|
||||
|
||||
- name: Install dependencies
|
||||
run: npm ci
|
||||
|
||||
- name: Install Playwright
|
||||
run: npx playwright install --with-deps chromium
|
||||
|
||||
- name: Run visual tests
|
||||
run: npx playwright test --project=visual
|
||||
|
||||
- name: Upload diff images
|
||||
if: failure()
|
||||
uses: actions/upload-artifact@v3
|
||||
with:
|
||||
name: visual-diffs
|
||||
path: .test/screenshots/diff/
|
||||
|
||||
- name: Upload report
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v3
|
||||
with:
|
||||
name: html-report
|
||||
path: .test/reports/
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Organize by viewport** - Separate baselines for mobile, tablet, desktop
|
||||
2. **Use descriptive names** - `[feature]_[action]_[viewport].png`
|
||||
3. **Set appropriate thresholds** - Pixel-perfect for logos, tolerant for dynamic content
|
||||
4. **Mask dynamic elements** - Hide timestamps, dates, ads
|
||||
5. **Review diffs carefully** - Don't blindly update baselines
|
||||
6. **Commit baselines** - Include in git for reproducibility
|
||||
|
||||
## Integration with Agents
|
||||
|
||||
- **@browser-automation** - Captures screenshots using Playwright MCP
|
||||
- **@visual-tester** - Compares screenshots to baselines
|
||||
- **@sdet-engineer** - Writes visual test cases
|
||||
|
||||
## References
|
||||
|
||||
- [Pixelmatch](https://github.com/mapbox/pixelmatch)
|
||||
- [Sharp](https://sharp.pixelplumbing.com/)
|
||||
- [Playwright Screenshots](https://playwright.dev/docs/screenshots)
|
||||
Reference in New Issue
Block a user