feat: add AI-powered visual regression testing infrastructure

- Add docker-compose.web-testing.yml with vlmkit service
- Add 5 test scripts: capture, compare, pipeline, console-monitor, link-checker
- Add vrt.config.json with 3 viewports + maskSelectors
- Add package.json with @mizchi/vlmkit and Node >=24
- Add tests/README.md with VRT documentation
- Update .gitignore for test artifacts

Based on APAW issue #144 / milestone #106
This commit is contained in:
NW
2026-06-30 21:09:30 +01:00
parent f2d4d6d3b1
commit 8d85776135
10 changed files with 1049 additions and 44 deletions

188
tests/README.md Normal file
View File

@@ -0,0 +1,188 @@
# Web Testing Suite
Автоматическое тестирование веб-приложений. Запускается целиком в Docker без зависимостей на хосте.
## Возможности
| Тест | Скрипт | Описание |
|------|--------|----------|
| **Visual Regression** | `visual-test-pipeline.js` | Скриншоты + pixelmatch + извлечение UI-элементов с bbox |
| **VLMKit AI VRT** | `@mizchi/vlmkit` | AI-powered pixel + computed-style + a11y tree diff |
| **Screenshot Capture** | `capture-screenshots.js` | Захват baseline/current скриншотов в 3 viewport |
| **Screenshot Compare** | `compare-screenshots.js` | Сравнение PNG через pixelmatch |
| **Console Errors** | `console-error-monitor-standalone.js` | Ловит JS ошибки, network 4xx/5xx, request failures |
| **Link Checking** | `link-checker.js` | Проверка ссылок на 404/500 |
## Быстрый старт
### Вариант 1: Docker Compose (рекомендуется)
```bash
# Полный визуальный пайплайн (захват + сравнение + элементы + ошибки)
docker compose -f docker/docker-compose.web-testing.yml run --rm \
-e TARGET_URL=https://example.com \
-e PAGES=/ \
visual-tester
# Только захват baseline-скриншотов
docker compose -f docker/docker-compose.web-testing.yml run --rm \
-e TARGET_URL=https://example.com \
screenshot-baseline
# Только захват текущих скриншотов
docker compose -f docker/docker-compose.web-testing.yml run --rm \
-e TARGET_URL=https://example.com \
screenshot-current
# Только сравнение (pixelmatch)
docker compose -f docker/docker-compose.web-testing.yml run --rm visual-compare
# Мониторинг консольных ошибок
docker compose -f docker/docker-compose.web-testing.yml run --rm \
-e TARGET_URL=https://example.com \
console-monitor
# VLMKit AI-powered visual regression (новый)
docker compose -f docker/docker-compose.web-testing.yml run --rm \
-e TARGET_URL=https://example.com \
-e PAGES=/,/login,/cart \
vlmkit
```
### Вариант 2: Прямой docker run
```bash
docker run --rm \
--add-host=host.docker.internal:host-gateway \
--shm-size=2g \
-v $(pwd)/tests:/app/tests \
-e TARGET_URL=https://example.com \
-e PAGES=/ \
mcr.microsoft.com/playwright:v1.52.0-noble \
sh -c "cd /app/tests && npm install --ignore-scripts && node scripts/visual-test-pipeline.js"
```
## Структура
```
tests/
├── scripts/
│ ├── visual-test-pipeline.js # Полный пайплайн (захват + сравнение + элементы)
│ ├── capture-screenshots.js # Захват скриншотов (baseline|current)
│ ├── compare-screenshots.js # Сравнение PNG (pixelmatch)
│ ├── console-error-monitor-standalone.js # Мониторинг console/network ошибок
│ ├── console-error-monitor.js # Мониторинг через Playwright MCP
│ └── link-checker.js # Проверка ссылок
├── visual/
│ ├── baseline/ # Эталонные скриншоты (git-tracked)
│ ├── current/ # Текущие скриншоты (gitignored)
│ └── diff/ # Diff-изображения (gitignored)
├── reports/ # JSON-отчёты (gitignored)
│ ├── visual-test-report.json
│ └── console-error-report.json
├── vrt.config.json # Конфигурация @mizchi/vlmkit
├── package.json
└── README.md
```
## Переменные окружения
| Переменная | По умолчанию | Описание |
|------------|--------------|----------|
| `TARGET_URL` | `http://host.docker.internal:3000` | URL тестируемого приложения |
| `PAGES` | `/,/admin/login` | Список путей через запятую |
| `PIXELMATCH_THRESHOLD` | `0.05` | Допустимый % отличий (5%) |
| `REPORTS_DIR` | `./reports` | Папка для отчётов |
| `VRT_CONFIG_PATH` | `./vrt.config.json` | Путь к конфигу @mizchi/vlmkit |
## Visual Regression — как работает
1. **Захват скриншотов** — Playwright открывает страницу в 3 viewport (mobile 375x667, tablet 768x1024, desktop 1280x720)
2. **Извлечение элементов** — обходит DOM, собирает bbox, tag, className, text, href для каждого видимого элемента
3. **Сравнение** — pixelmatch сравнивает текущие PNG с baseline, генерирует diff-изображение
4. **Отчёт** — JSON с результатами: элементы, console/network ошибки, diff-процент
### Baseline-скриншоты
При первом запуске без baseline скрипт автоматически создаёт их. Для обновления:
```bash
docker compose -f docker/docker-compose.web-testing.yml run --rm \
-e TARGET_URL=https://example.com screenshot-baseline
```
### Обнаруживаемые проблемы
- Наложение элементов (кнопка вне viewportа)
- Сдвиг шрифтов / неверные цвета
- Отсутствующие / лишние элементы
- Микро-кнопки (width < 10px)
- Console JS errors
- Network errors (4xx/5xx)
## Пример отчёта
```json
{
"summary": {
"screenshotsCaptured": 3,
"totalElements": 702,
"comparisonsPassed": 3,
"comparisonsFailed": 0,
"totalConsoleErrors": 0,
"totalNetworkErrors": 25
},
"elements": {
"homepage_desktop": [
{ "tag": "button", "text": "Buy Now", "bbox": {"x":318,"y":349,"width":644,"height":47} }
]
}
}
```
## VLMKit AI-Powered Visual Regression
Для запуска AI-анализа визуальных регрессий через `@mizchi/vlmkit`:
```bash
# Убедитесь, что Node.js >= 24
docker compose -f docker/docker-compose.web-testing.yml run --rm vlmkit
```
### Требования
- **Node.js 24+**: `@mizchi/vlmkit` требует Node.js версии 24 или выше.
- **pnpm**: Рекомендуется для установки `@mizchi/vlmkit`.
### Команды VLMKit
```bash
# Захват baseline-скриншотов
npx vlmkit snapshot --url http://localhost:3000 --pages /,/login,/cart --out ./visual/baseline
# Сравнение текущих скриншотов с baseline
npx vlmkit diff --baseline ./visual/baseline --current ./visual/current --out ./visual/diff
# Проверка доступности (a11y)
npx vlmkit check a11y --url http://localhost:3000 --pages /,/login,/cart
```
### Конфигурация
Настройки `vrt.config.json`:
- `baseUrl`: URL тестируемого приложения
- `pages`: Список страниц для тестирования
- `viewports`: Разрешения экранов (mobile, tablet, desktop)
- `maskSelectors`: Селекторы для скрытия динамического контента
- `thresholds`: Пороги для pixel, style и a11y проверок
- `ai.generateCssFix`: Автоматическая генерация CSS-фиксов (требует API ключ)
## Docker-образ
Используется `mcr.microsoft.com/playwright:v1.52.0-noble` — предустановленный Playwright с Chromium.
## See Also
- `docker/docker-compose.web-testing.yml` — Docker Compose конфигурация
- `.kilo/agents/visual-tester.md` — Агент визуального тестирования
- `.kilo/commands/e2e-test.md` — E2E workflow

35
tests/package.json Normal file
View File

@@ -0,0 +1,35 @@
{
"name": "apaw-web-testing",
"version": "2.0.0",
"description": "Web application testing suite for APAW - Visual regression, link checking, form testing, console error detection",
"main": "scripts/visual-test-pipeline.js",
"scripts": {
"test": "node scripts/visual-test-pipeline.js",
"test:visual": "node scripts/visual-test-pipeline.js",
"test:baseline": "node scripts/capture-screenshots.js baseline",
"test:current": "node scripts/capture-screenshots.js current",
"test:compare": "node scripts/compare-screenshots.js",
"test:console": "node scripts/console-error-monitor-standalone.js",
"test:links": "node scripts/link-checker.js"
},
"keywords": [
"web-testing",
"visual-regression",
"e2e",
"playwright",
"kilo-code"
],
"author": "APAW Team",
"license": "MIT",
"dependencies": {
"pixelmatch": "^5.3.0",
"playwright": "1.52.0",
"pngjs": "^7.0.0"
},
"devDependencies": {
"@mizchi/vlmkit": "latest"
},
"engines": {
"node": ">=24.0.0"
}
}

View File

@@ -0,0 +1,84 @@
const { chromium } = require('playwright');
const path = require('path');
const fs = require('fs');
const VIEWPORTS = {
mobile: { width: 375, height: 667 },
tablet: { width: 768, height: 1024 },
desktop: { width: 1280, height: 720 },
};
function slugify(pagePath) {
return pagePath.replace(/^\//, '').replace(/[^a-zA-Z0-9_-]/g, '_') || 'home';
}
async function capturePage(page, pagePath, viewportName, outputDir) {
const fullUrl = `${process.env.TARGET_URL}${pagePath}`;
const filename = `${slugify(pagePath)}_${viewportName}.png`;
const filepath = path.join(outputDir, filename);
await page.setViewportSize(VIEWPORTS[viewportName]);
await page.goto(fullUrl, { waitUntil: 'networkidle', timeout: 30000 });
await page.waitForTimeout(500);
await page.screenshot({ path: filepath, fullPage: false });
return { pagePath, viewport: viewportName, file: filename, saved: true };
}
async function main() {
const mode = process.argv[2];
if (!['baseline', 'current'].includes(mode)) {
console.error('Usage: node capture-screenshots.js <baseline|current>');
process.exit(1);
}
const targetUrl = process.env.TARGET_URL;
const pages = (process.env.PAGES || '/').split(',').map(p => p.trim()).filter(p => /^\/[A-Za-z0-9._\/-]*$/.test(p));
if (pages.length === 0) {
console.error('No valid pages in PAGES environment variable');
process.exit(1);
}
const outputDir = mode === 'baseline'
? (process.env.BASELINE_DIR || './screenshots/baseline')
: (process.env.CURRENT_DIR || './screenshots/current');
if (!targetUrl) {
console.error('TARGET_URL environment variable is required');
process.exit(1);
}
fs.mkdirSync(outputDir, { recursive: true });
const browser = await chromium.launch({ headless: true });
const context = await browser.newContext();
const page = await context.newPage();
const results = [];
try {
for (const pagePath of pages) {
for (const vpName of Object.keys(VIEWPORTS)) {
try {
const result = await capturePage(page, pagePath, vpName, outputDir);
results.push(result);
console.log(`Captured ${pagePath} @ ${vpName} -> ${result.file}`);
} catch (err) {
results.push({ pagePath, viewport: vpName, error: err.message, saved: false });
console.error(`Failed ${pagePath} @ ${vpName}: ${err.message}`);
}
}
}
} finally {
await browser.close();
}
const report = { mode, targetUrl, pages, outputDir, results };
const reportPath = path.join(outputDir, '..', `${mode}-capture-report.json`);
fs.mkdirSync(path.dirname(reportPath), { recursive: true });
fs.writeFileSync(reportPath, JSON.stringify(report, null, 2));
console.log(`Report saved to ${reportPath}`);
}
main().catch(err => {
console.error('Fatal error:', err);
process.exit(1);
});

View File

@@ -0,0 +1,107 @@
const pixelmatch = require('pixelmatch');
const { PNG } = require('pngjs');
const fs = require('fs');
const path = require('path');
const THRESHOLD = parseFloat(process.env.PIXELMATCH_THRESHOLD) || 0.05;
const BASELINE_DIR = process.env.BASELINE_DIR || './screenshots/baseline';
const CURRENT_DIR = process.env.CURRENT_DIR || './screenshots/current';
const DIFF_DIR = process.env.DIFF_DIR || './screenshots/diffs';
const REPORTS_DIR = process.env.REPORTS_DIR || './reports';
function readPNG(filepath) {
const data = fs.readFileSync(filepath);
return PNG.sync.read(data);
}
function compareImages(baselinePath, currentPath, diffPath) {
const img1 = readPNG(baselinePath);
const img2 = readPNG(currentPath);
const { width, height } = img1;
const diff = new PNG({ width, height });
const mismatchedPixels = pixelmatch(img1.data, img2.data, diff.data, width, height, {
threshold: THRESHOLD,
});
const totalPixels = width * height;
const mismatchPercent = (mismatchedPixels / totalPixels) * 100;
if (mismatchedPixels > 0) {
fs.mkdirSync(path.dirname(diffPath), { recursive: true });
fs.writeFileSync(diffPath, PNG.sync.write(diff));
}
return { mismatchedPixels, totalPixels, mismatchPercent, diffCreated: mismatchedPixels > 0 };
}
function findPNGFiles(dir) {
if (!fs.existsSync(dir)) return [];
return fs.readdirSync(dir).filter(f => f.endsWith('.png'));
}
async function main() {
const baselineFiles = findPNGFiles(BASELINE_DIR);
const currentFiles = findPNGFiles(CURRENT_DIR);
if (baselineFiles.length === 0) {
console.error(`No baseline screenshots found in ${BASELINE_DIR}`);
process.exit(1);
}
if (currentFiles.length === 0) {
console.error(`No current screenshots found in ${CURRENT_DIR}`);
process.exit(1);
}
const allFiles = [...new Set([...baselineFiles, ...currentFiles])];
const comparisons = [];
for (const file of allFiles) {
const baselinePath = path.join(BASELINE_DIR, file);
const currentPath = path.join(CURRENT_DIR, file);
if (!fs.existsSync(baselinePath)) {
comparisons.push({ file, status: 'missing_baseline' });
continue;
}
if (!fs.existsSync(currentPath)) {
comparisons.push({ file, status: 'missing_current' });
continue;
}
try {
const diffPath = path.join(DIFF_DIR, file);
const result = compareImages(baselinePath, currentPath, diffPath);
comparisons.push({ file, status: result.mismatchedPixels === 0 ? 'pass' : 'fail', ...result });
} catch (err) {
comparisons.push({ file, status: 'error', error: err.message });
}
}
const passed = comparisons.filter(c => c.status === 'pass').length;
const failed = comparisons.filter(c => c.status === 'fail').length;
const errors = comparisons.filter(c => c.status === 'error').length;
const missing = comparisons.filter(c => c.status === 'missing_baseline' || c.status === 'missing_current').length;
const report = {
timestamp: new Date().toISOString(),
threshold: THRESHOLD,
summary: { total: comparisons.length, passed, failed, errors, missing },
comparisons,
};
fs.mkdirSync(REPORTS_DIR, { recursive: true });
const reportPath = path.join(REPORTS_DIR, 'visual-test-report.json');
fs.writeFileSync(reportPath, JSON.stringify(report, null, 2));
console.log(`Report saved to ${reportPath}`);
console.log(`Passed: ${passed}, Failed: ${failed}, Errors: ${errors}, Missing: ${missing}`);
if (failed > 0 || errors > 0) process.exit(1);
}
main().catch(err => {
console.error('Fatal error:', err);
process.exit(1);
});

View File

@@ -0,0 +1,127 @@
const { chromium } = require('playwright');
const fs = require('fs');
const path = require('path');
const TARGET_URL = process.env.TARGET_URL;
const PAGES = (process.env.PAGES || '/').split(',').map(p => p.trim());
const REPORTS_DIR = process.env.REPORTS_DIR || './reports';
async function monitorPage(context, pagePath) {
const page = await context.newPage();
const fullUrl = `${TARGET_URL}${pagePath}`;
const consoleErrors = [];
const networkErrors = [];
page.on('console', msg => {
if (msg.type() === 'error') {
consoleErrors.push({
type: 'console_error',
text: msg.text(),
location: msg.location(),
timestamp: new Date().toISOString(),
});
}
});
page.on('pageerror', err => {
consoleErrors.push({
type: 'page_error',
text: err.message,
stack: err.stack,
timestamp: new Date().toISOString(),
});
});
page.on('requestfailed', req => {
networkErrors.push({
type: 'network_failure',
url: req.url(),
method: req.method(),
failure: req.failure()?.errorText || 'unknown',
resourceType: req.resourceType(),
timestamp: new Date().toISOString(),
});
});
page.on('response', res => {
if (res.status() >= 400) {
networkErrors.push({
type: 'http_error',
url: res.url(),
status: res.status(),
statusText: res.statusText(),
method: res.request().method(),
resourceType: res.request().resourceType(),
timestamp: new Date().toISOString(),
});
}
});
try {
await page.goto(fullUrl, { waitUntil: 'networkidle', timeout: 30000 });
await page.waitForTimeout(1000);
} finally {
await page.close();
}
return { pagePath, consoleErrors, networkErrors };
}
async function main() {
if (!TARGET_URL) {
console.error('TARGET_URL environment variable is required');
process.exit(1);
}
const browser = await chromium.launch({ headless: true });
const context = await browser.newContext();
const results = [];
try {
for (const pagePath of PAGES) {
try {
const result = await monitorPage(context, pagePath);
results.push(result);
console.log(`Checked ${pagePath}: ${result.consoleErrors.length} console errors, ${result.networkErrors.length} network errors`);
} catch (err) {
results.push({ pagePath, error: err.message, consoleErrors: [], networkErrors: [] });
console.error(`Failed ${pagePath}: ${err.message}`);
}
}
} finally {
await browser.close();
}
const totalConsoleErrors = results.reduce((sum, r) => sum + r.consoleErrors.length, 0);
const totalNetworkErrors = results.reduce((sum, r) => sum + r.networkErrors.length, 0);
const pagesWithErrors = results.filter(r => r.consoleErrors.length > 0 || r.networkErrors.length > 0).length;
const report = {
timestamp: new Date().toISOString(),
targetUrl: TARGET_URL,
pages: PAGES,
summary: {
totalPages: results.length,
pagesWithErrors,
totalConsoleErrors,
totalNetworkErrors,
hasErrors: totalConsoleErrors > 0 || totalNetworkErrors > 0,
},
results,
};
fs.mkdirSync(REPORTS_DIR, { recursive: true });
const reportPath = path.join(REPORTS_DIR, 'console-error-report.json');
fs.writeFileSync(reportPath, JSON.stringify(report, null, 2));
console.log(`Report saved to ${reportPath}`);
if (totalConsoleErrors > 0 || totalNetworkErrors > 0) {
console.error(`Found ${totalConsoleErrors} console errors and ${totalNetworkErrors} network errors across ${pagesWithErrors} page(s)`);
process.exit(1);
}
}
main().catch(err => {
console.error('Fatal error:', err);
process.exit(1);
});

View File

@@ -0,0 +1,83 @@
const { chromium } = require('playwright');
const fs = require('fs');
const path = require('path');
const TARGET_URL = process.env.TARGET_URL;
const PAGES = (process.env.PAGES || '/').split(',').map(p => p.trim());
const REPORTS_DIR = process.env.REPORTS_DIR || './reports';
async function collectLinks(page, pagePath) {
const fullUrl = `${TARGET_URL}${pagePath}`;
await page.goto(fullUrl, { waitUntil: 'networkidle', timeout: 30000 });
return page.evaluate(() => Array.from(document.querySelectorAll('a[href]'))
.map(a => a.href));
}
async function checkLink(context, url) {
try {
const resp = await context.request.fetch(url, { method: 'HEAD', timeout: 15000 });
return { url, status: resp.status(), ok: resp.ok() };
} catch (err) {
return { url, status: 0, ok: false, error: err.message };
}
}
async function main() {
if (!TARGET_URL) {
console.error('TARGET_URL environment variable is required');
process.exit(1);
}
const browser = await chromium.launch({ headless: true });
const context = await browser.newContext();
const results = [];
try {
for (const pagePath of PAGES) {
const page = await context.newPage();
try {
const links = await collectLinks(page, pagePath);
await page.close();
const checked = [];
for (const url of [...new Set(links)]) {
if (url.startsWith('mailto:') || url.startsWith('tel:')) continue;
checked.push(await checkLink(context, url));
}
const broken = checked.filter(c => !c.ok);
results.push({ pagePath, total: checked.length, broken: broken.length, brokenLinks: broken });
console.log(`${pagePath}: ${checked.length} links, ${broken.length} broken`);
} catch (err) {
results.push({ pagePath, error: err.message, total: 0, broken: 0, brokenLinks: [] });
console.error(`Failed ${pagePath}: ${err.message}`);
} finally {
await page.close().catch(() => {});
}
}
} finally {
await browser.close();
}
const totalBroken = results.reduce((s, r) => s + r.broken, 0);
const report = {
timestamp: new Date().toISOString(),
targetUrl: TARGET_URL,
pages: PAGES,
summary: { totalPages: results.length, totalBroken, hasBroken: totalBroken > 0 },
results,
};
fs.mkdirSync(REPORTS_DIR, { recursive: true });
const reportPath = path.join(REPORTS_DIR, 'link-check-report.json');
fs.writeFileSync(reportPath, JSON.stringify(report, null, 2));
console.log(`Report saved to ${reportPath}`);
if (totalBroken > 0) {
console.error(`Found ${totalBroken} broken links`);
process.exit(1);
}
}
main().catch(err => {
console.error('Fatal error:', err);
process.exit(1);
});

View File

@@ -0,0 +1,249 @@
const { chromium } = require('playwright');
const pixelmatch = require('pixelmatch');
const { PNG } = require('pngjs');
const fs = require('fs');
const path = require('path');
const VIEWPORTS = {
mobile: { width: 375, height: 667 },
tablet: { width: 768, height: 1024 },
desktop: { width: 1280, height: 720 },
};
const TARGET_URL = process.env.TARGET_URL;
const PAGES = (process.env.PAGES || '/').split(',').map(p => p.trim());
const BASELINE_DIR = process.env.BASELINE_DIR || './screenshots/baseline';
const CURRENT_DIR = process.env.CURRENT_DIR || './screenshots/current';
const DIFF_DIR = process.env.DIFF_DIR || './screenshots/diffs';
const REPORTS_DIR = process.env.REPORTS_DIR || './reports';
const THRESHOLD = parseFloat(process.env.PIXELMATCH_THRESHOLD) || 0.05;
function slugify(pagePath) {
return pagePath.replace(/^\//, '').replace(/[^a-zA-Z0-9_-]/g, '_') || 'home';
}
function readPNG(filepath) {
const data = fs.readFileSync(filepath);
return PNG.sync.read(data);
}
async function captureScreenshots(page, pagePath, viewportName, outputDir) {
const fullUrl = `${TARGET_URL}${pagePath}`;
const filename = `${slugify(pagePath)}_${viewportName}.png`;
const filepath = path.join(outputDir, filename);
await page.setViewportSize(VIEWPORTS[viewportName]);
await page.goto(fullUrl, { waitUntil: 'networkidle', timeout: 30000 });
await page.waitForTimeout(500);
await page.screenshot({ path: filepath, fullPage: false });
return { pagePath, viewport: viewportName, file: filename };
}
async function extractBBoxes(page) {
return page.evaluate(() => {
const els = document.querySelectorAll('[data-testid], [id], [class]');
return Array.from(els).slice(0, 50).map(el => {
const rect = el.getBoundingClientRect();
return {
tag: el.tagName,
id: el.id || null,
testid: el.getAttribute('data-testid') || null,
class: el.getAttribute('class')?.slice(0, 60) || null,
x: rect.x, y: rect.y, width: rect.width, height: rect.height,
visible: rect.width > 0 && rect.height > 0,
};
}).filter(b => b.visible);
});
}
function compareImages(baselinePath, currentPath, diffPath) {
const img1 = readPNG(baselinePath);
const img2 = readPNG(currentPath);
if (img1.width !== img2.width || img1.height !== img2.height) {
return {
mismatchedPixels: -1,
mismatchPercent: -1,
diffCreated: false,
dimensionMismatch: true,
baselineDimensions: { width: img1.width, height: img1.height },
currentDimensions: { width: img2.width, height: img2.height },
};
}
const diff = new PNG({ width: img1.width, height: img1.height });
const mismatchedPixels = pixelmatch(img1.data, img2.data, diff.data, img1.width, img1.height, {
threshold: THRESHOLD,
});
const mismatchPercent = (mismatchedPixels / (img1.width * img1.height)) * 100;
if (mismatchedPixels > 0) {
fs.mkdirSync(path.dirname(diffPath), { recursive: true });
fs.writeFileSync(diffPath, PNG.sync.write(diff));
}
return { mismatchedPixels, mismatchPercent, diffCreated: mismatchedPixels > 0 };
}
async function main() {
if (!TARGET_URL) {
console.error('TARGET_URL environment variable is required');
process.exit(1);
}
let hasBaselines = false;
try {
hasBaselines = fs.existsSync(BASELINE_DIR) && fs.statSync(BASELINE_DIR).isDirectory() && fs.readdirSync(BASELINE_DIR).some(f => f.endsWith('.png'));
} catch {}
const isBaselineMode = !hasBaselines;
if (isBaselineMode) {
console.log('No baselines found. Creating baseline screenshots...');
}
const outputDir = isBaselineMode ? BASELINE_DIR : CURRENT_DIR;
fs.mkdirSync(outputDir, { recursive: true });
const browser = await chromium.launch({ headless: true });
const context = await browser.newContext();
const page = await context.newPage();
const screenshots = [];
const allConsoleErrors = [];
const allNetworkErrors = [];
const allBBoxes = [];
let iterationConsoleErrors = [];
let iterationNetworkErrors = [];
page.on('console', msg => {
if (msg.type() === 'error') {
iterationConsoleErrors.push({ type: 'console_error', text: msg.text(), location: msg.location() });
}
});
page.on('pageerror', err => {
iterationConsoleErrors.push({ type: 'page_error', text: err.message, stack: err.stack });
});
page.on('requestfailed', req => {
iterationNetworkErrors.push({
type: 'network_error',
url: req.url(),
method: req.method(),
failure: req.failure()?.errorText || 'unknown',
});
});
page.on('response', res => {
if (res.status() >= 400) {
iterationNetworkErrors.push({
type: 'http_error',
url: res.url(),
status: res.status(),
method: res.request().method(),
});
}
});
try {
for (const pagePath of PAGES) {
for (const vpName of Object.keys(VIEWPORTS)) {
try {
iterationConsoleErrors = [];
iterationNetworkErrors = [];
const result = await captureScreenshots(page, pagePath, vpName, outputDir);
screenshots.push(result);
const bboxes = await extractBBoxes(page);
allBBoxes.push({ pagePath, viewport: vpName, elements: bboxes });
await page.waitForTimeout(200);
allConsoleErrors.push(...iterationConsoleErrors);
allNetworkErrors.push(...iterationNetworkErrors);
console.log(`Captured ${pagePath} @ ${vpName}`);
} catch (err) {
screenshots.push({ pagePath, viewport: vpName, error: err.message });
console.error(`Failed ${pagePath} @ ${vpName}: ${err.message}`);
}
}
}
} finally {
await browser.close();
}
const comparisons = [];
if (!isBaselineMode) {
const baselineFiles = fs.readdirSync(BASELINE_DIR).filter(f => f.endsWith('.png'));
const currentFiles = fs.readdirSync(CURRENT_DIR).filter(f => f.endsWith('.png'));
const allFiles = [...new Set([...baselineFiles, ...currentFiles])];
for (const file of allFiles) {
const baselinePath = path.join(BASELINE_DIR, file);
const currentPath = path.join(CURRENT_DIR, file);
if (!fs.existsSync(baselinePath)) {
comparisons.push({ file, status: 'missing_baseline' });
continue;
}
if (!fs.existsSync(currentPath)) {
comparisons.push({ file, status: 'missing_current' });
continue;
}
try {
const diffPath = path.join(DIFF_DIR, file);
const result = compareImages(baselinePath, currentPath, diffPath);
const status = result.dimensionMismatch ? 'error' : (result.mismatchedPixels === 0 ? 'pass' : 'fail');
comparisons.push({ file, status, ...result });
} catch (err) {
comparisons.push({ file, status: 'error', error: err.message });
}
}
}
const passed = comparisons.filter(c => c.status === 'pass').length;
const failed = comparisons.filter(c => c.status === 'fail').length;
const errors = comparisons.filter(c => c.status === 'error').length;
const missing = comparisons.filter(c => c.status === 'missing_baseline' || c.status === 'missing_current').length;
const report = {
timestamp: new Date().toISOString(),
mode: isBaselineMode ? 'baseline_creation' : 'comparison',
targetUrl: TARGET_URL,
pages: PAGES,
screenshots,
bboxes: allBBoxes,
bboxCount: allBBoxes.reduce((sum, b) => sum + b.elements.length, 0),
consoleErrors: allConsoleErrors,
networkErrors: allNetworkErrors,
visualComparison: comparisons.length > 0 ? {
threshold: THRESHOLD,
summary: { total: comparisons.length, passed, failed, errors, missing },
comparisons,
} : null,
};
fs.mkdirSync(REPORTS_DIR, { recursive: true });
const reportPath = path.join(REPORTS_DIR, 'visual-test-report.json');
fs.writeFileSync(reportPath, JSON.stringify(report, null, 2));
console.log(`Report saved to ${reportPath}`);
if (allConsoleErrors.length > 0) {
console.warn(`Found ${allConsoleErrors.length} console errors`);
}
if (allNetworkErrors.length > 0) {
console.warn(`Found ${allNetworkErrors.length} network errors`);
}
if (failed > 0 || errors > 0 || missing > 0) {
console.error(`Visual diffs: ${failed} failed, ${errors} errors, ${missing} missing`);
process.exit(1);
}
}
main().catch(err => {
console.error('Fatal error:', err);
process.exit(1);
});

19
tests/vrt.config.json Normal file
View File

@@ -0,0 +1,19 @@
{
"baseUrl": "http://host.docker.internal:3000",
"pages": ["/", "/admin/login"],
"viewports": {
"mobile": { "width": 375, "height": 667 },
"tablet": { "width": 768, "height": 1024 },
"desktop": { "width": 1280, "height": 720 }
},
"maskSelectors": [".timestamp", ".live-counter"],
"thresholds": {
"pixel": 0.05,
"style": 0.02,
"a11y": 0.0
},
"outputDir": "./visual",
"ai": {
"generateCssFix": false
}
}