mtw-e2e-runner

English ·

@matware/e2e-runner

The AI-native E2E test runner that writes, runs, and debugs tests for you.

E2E Runner lets you test your web app without writing test code. Tests are plain JSON — and you don't even have to write that yourself: just ask Claude Code.

🎬 Write a test by asking — then watch it run

The live dashboard while a suite runs — every step streams a screenshot into the feed, in real time.

With the built-in MCP server, creating a test is a conversation — no docs, no syntax to memorize:

> You: *Create an E2E test for the login flow and run it.*

>

> Claude Code: *writes the test, runs it in a real browser, and reports back —*

> ✅ `login-flow` passed in 2.3s · screenshot saved · no network errors.

Behind the scenes Claude just wrote and ran this. A test is just JSON — an ordered list of what a user does:

[
  { "name": "login-flow", "actions": [
    { "type": "goto", "value": "/login" },
    { "type": "type", "selector": "#email", "value": "user@test.com" },
    { "type": "type", "selector": "#password", "value": "secret" },
    { "type": "click", "text": "Sign In" },
    { "type": "assert_text", "text": "Welcome back" },
    { "type": "screenshot", "value": "logged-in.png" }
  ]}
]

No imports, no `describe`/`it`, no build step. If you can read it you can write it — or just ask.

Connect it to Claude Code (2 commands):

claude plugin marketplace add fastslack/mtw-e2e-runner
claude plugin install e2e-runner@matware

Now say *"create a test for X and run it"* — Claude gets 17 MCP tools, slash commands, and specialized agents.

> Using a different agent (Cursor, Codex, Copilot, 40+ more)? Install the skill: `npx skills add fastslack/mtw-e2e-runner`

📖 Contents

🚀 Install — it's tiny

npm install --save-dev @matware/e2e-runner
npx e2e-runner init        # scaffolds e2e/ with a sample test + config

Then pick how to run the browser. You don't need Docker unless you want the parallel pool:

Option 1 · Use the Chrome you already have — no Docker ⭐

Launch any Chromium browser with a debugging port, then point the runner at it:

google-chrome --headless=new --remote-debugging-port=9222 &   # or brave / chromium / msedge
CHROME_POOL_URL=http://localhost:9222 POOL_DRIVER=cdp npx e2e-runner run --all

Or bake it into `e2e.config.js` so you never repeat it:

export default {
  baseUrl: 'http://localhost:3000',     // your app — plain localhost, no docker hostname
  poolUrls: ['http://localhost:9222'],
  poolDriver: 'cdp',
};

Nothing to install beyond npm, and `baseUrl` is just `localhost` (the browser is on your machine).

Option 2 · Obscura — one tiny binary, no Docker

A single ~30 MB binary with built-in anti-detection. Install once, run it, point the runner at it:

obscura serve --port 9222 --stealth &
CHROME_POOL_URL=http://localhost:9222 POOL_DRIVER=obscura npx e2e-runner run --all

`npx e2e-runner pool start` (with `poolDriver: 'obscura'` in your config) prints the exact install command for your OS.

Option 3 · Docker pool — parallel, for CI & big suites

A shared, queue-managed Chrome pool that runs many tests at once:

npx e2e-runner run --all     # the first run auto-starts the Docker pool for you

Requires Docker. Set `baseUrl: 'http://host.docker.internal:3000'` so the containerized Chrome can reach your app.

Why host.docker.internal (Docker option only)?

With the Docker pool, Chrome runs inside a container, so `localhost` there means the container — not your machine. `host.docker.internal` bridges to your host. On Linux (Docker Engine, not Docker Desktop) add `--add-host=host.docker.internal:host-gateway`, or use your LAN IP. Options 1 & 2 don't have this — the browser is local, so plain `localhost` just works.

Write your first test

Open `e2e/tests/sample.json` — a flow is an ordered list of actions:

[
  { "name": "homepage loads", "actions": [
    { "type": "goto", "value": "/" },
    { "type": "assert_text", "text": "Welcome" },
    { "type": "screenshot", "value": "home.png" }
  ]}
]

Run it with `npx e2e-runner run --all`. Results — pass/fail, timing, screenshots, network errors — print to your terminal and to the web dashboard if it's open.

Add OpenCode (optional)

cp node_modules/@matware/e2e-runner/opencode.json ./
mkdir -p .opencode && cp -r node_modules/@matware/e2e-runner/.opencode/* .opencode/

See OPENCODE.md for details.

Updating

Each install method updates separately — bump the one(s) you use:

# npm dependency (per project)
npm install --save-dev @matware/e2e-runner@latest

# Claude Code plugin
claude plugin update e2e-runner@matware

# MCP-only install (npx caches the package — pin @latest to force a refresh)
claude mcp add --transport stdio --scope user e2e-runner \
  -- npx -y -p @matware/e2e-runner@latest e2e-runner-mcp

> [!NOTE]

> Two gotchas: (1) `npx` prefers a copy found in the project's `node_modules` over its own cache — if a project pins an old version, the MCP server and dashboard run that old version, so update the project dependency too. (2) Already-running processes keep the old code in memory: after updating, restart the dashboard and reconnect the MCP server (`/mcp` → `e2e-runner` → Reconnect, or restart your session).

✨ What you get

🧪 Zero-code tests — JSON files that anyone on your team can read and write. No JavaScript, no compilation, no framework lock-in.

🤖 AI-powered testing — Claude Code creates, executes, and debugs tests natively through 17 MCP tools. Ask it to "test the checkout flow" and it builds the JSON, runs it, and reports back.

🐛 Issue-to-Test pipeline — Paste a GitHub or GitLab issue URL. The runner fetches it, generates E2E tests, runs them, and tells you: *bug confirmed* or *not reproducible*.

👁️ Visual verification — Describe what the page should look like in plain English. The AI captures a screenshot and judges pass/fail against your description. No pixel-diffing setup needed.

🧠 Learning system — Tracks test stability across runs. Detects flaky tests, unstable selectors, slow APIs, and error patterns — then surfaces actionable insights.

⚡ Parallel execution — Run N tests simultaneously against a shared browser pool (browserless, raw CDP, Lightpanda, Obscura, or Steel). Serial mode available for tests that share state.

🎯 Pluggable browser drivers — Pick the engine that fits each test: real Chrome via browserless, Lightpanda or Obscura for fast lightweight runs, Steel for managed sessions. Set `driver` per test or override the whole run with `--driver`.

📊 Real-time dashboard — Live execution view, run history with pass-rate charts, screenshot gallery with hash-based search, expandable network request logs.

🔁 Smart retries — Test-level and action-level retries with configurable delays. Flaky tests are detected and flagged automatically.

📦 Reusable modules — Extract common flows (login, navigation, setup) into parameterized modules and reference them with `$use`.

🏗️ CI-ready — JUnit XML output, exit code 1 on failure, auto-captured error screenshots. Drop-in GitHub Actions example included.

🌐 Multi-project — One dashboard aggregates test results from all your projects. One Chrome pool serves them all.

🐳 Portable — Chrome runs in Docker, tests are JSON files in your repo. Works on any machine with Node.js and Docker.

✍️ Writing tests

Everything about authoring tests — the file format, the full action vocabulary, retries, state isolation, and reuse. Expand what you need:

Test format & file layout

Each `.json` file in `e2e/tests/` contains an array of tests. Each test has a `name` and sequential `actions`:

[
  {
    "name": "homepage-loads",
    "actions": [
      { "type": "goto", "value": "/" },
      { "type": "assert_visible", "selector": "body" },
      { "type": "assert_url", "value": "/" },
      { "type": "screenshot", "value": "homepage.png" }
    ]
  }
]

Suite files can have numeric prefixes for ordering (`01-auth.json`, `02-dashboard.json`). The `--suite` flag matches with or without the prefix, so `--suite auth` finds `01-auth.json`.

Action catalog — navigation, input & interaction

Click by text — when `click` uses `text` instead of `selector`, it searches across common interactive and content elements:

button, a, [role="button"], [role="tab"], [role="menuitem"], [role="option"],
[role="listitem"], div[class*="cursor"], span, li, td, th, label, p, h1-h6

{ "type": "click", "text": "Sign In" }

Assertions — verify text, elements, URLs, counts & network

Framework-aware actions — React/MUI without evaluate boilerplate

These actions handle common patterns in React/MUI apps that normally require verbose `evaluate` boilerplate:

// Before: 5 lines of evaluate boilerplate
{ "type": "evaluate", "value": "const input = document.querySelector('#search'); const nativeSet = Object.getOwnPropertyDescriptor(window.HTMLInputElement.prototype, 'value').set; nativeSet.call(input, 'term'); input.dispatchEvent(new Event('input', {bubbles: true})); input.dispatchEvent(new Event('change', {bubbles: true}));" }

// After: 1 action
{ "type": "type_react", "selector": "#search", "value": "term" }

Multi-tab actions — popups, OAuth windows & cross-tab flows

All subsequent actions run in the active tab:

{ "type": "click", "text": "Open report" }
{ "type": "wait_for_tab", "text": "report" }
{ "type": "assert_text", "text": "Quarterly results" }
{ "type": "close_tab" }

Retries & flaky detection

Test-level retry — retry an entire test on failure. Set globally via config or per-test:

{ "name": "flaky-test", "retries": 3, "timeout": 15000, "actions": [...] }

Tests that pass after retry are flagged as flaky in the report and learning system.

Action-level retry — retry a single action without rerunning the entire test. Useful for timing-sensitive clicks and waits:

{ "type": "click", "selector": "#dynamic-btn", "retries": 3 }
{ "type": "wait", "selector": ".lazy-loaded", "retries": 2 }

Set globally: `actionRetries` in config, `--action-retries ` CLI, or `ACTION_RETRIES` env var. Delay between retries: `actionRetryDelay` (default 500ms).

Serial tests — for tests that share state

Tests that share state (e.g., two tests modifying the same record) can race when running in parallel. Mark them as serial:

{ "name": "create-patient", "serial": true, "actions": [...] }
{ "name": "verify-patient-list", "serial": true, "actions": [...] }

Serial tests run one at a time after all parallel tests finish — preventing interference without slowing down independent tests.

Testing authenticated apps

The simplest approach — log in via the UI like a real user:

{
  "hooks": {
    "beforeEach": [
      { "type": "goto", "value": "/login" },
      { "type": "type", "selector": "#email", "value": "test@example.com" },
      { "type": "type", "selector": "#password", "value": "test-password" },
      { "type": "click", "text": "Sign In" },
      { "type": "wait", "selector": ".dashboard" }
    ]
  },
  "tests": [...]
}

For SPAs with JWT, skip the login form by injecting the token directly:

{ "type": "set_storage", "value": "accessToken=eyJhbGciOiJIUzI1NiIs..." }

Or set it globally in config:

// e2e.config.js
export default {
  authToken: 'eyJhbGciOiJIUzI1NiIs...',
  authStorageKey: 'accessToken',
};

Each test runs in a fresh browser context, so auth state is automatically clean between tests.

> More strategies: Cookie-based auth, HTTP header injection, OAuth/SSO bypasses, reusable auth modules, and role-based testing — see docs/authentication.md

Reusable modules — extract common flows with $use

Extract common flows into parameterized modules:

// e2e/modules/login.json
{
  "$module": "login",
  "description": "Log in via the UI login form",
  "params": {
    "email": { "required": true, "description": "User email" },
    "password": { "required": true, "description": "User password" }
  },
  "actions": [
    { "type": "goto", "value": "/login" },
    { "type": "type", "selector": "#email", "value": "{{email}}" },
    { "type": "type", "selector": "#password", "value": "{{password}}" },
    { "type": "click", "text": "Sign In" },
    { "type": "wait", "value": "2000" }
  ]
}

Use in tests:

{
  "name": "dashboard-loads",
  "actions": [
    { "$use": "login", "params": { "email": "user@test.com", "password": "secret" } },
    { "type": "assert_text", "text": "Dashboard" }
  ]
}

Modules support parameter validation (required params fail fast), conditional blocks (`{{#param}}...{{/param}}`), nested composition, and cycle detection.

Hooks — beforeAll / beforeEach / afterEach / afterAll

Run actions at lifecycle points. Define globally in config or per-suite:

{
  "hooks": {
    "beforeAll": [{ "type": "goto", "value": "/setup" }],
    "beforeEach": [{ "type": "goto", "value": "/" }],
    "afterEach": [{ "type": "screenshot", "value": "after.png" }],
    "afterAll": []
  },
  "tests": [...]
}

> Important: `beforeAll` runs on a separate browser page that is closed before tests start. Use `beforeEach` for state that tests need (cookies, localStorage, auth tokens).

Exclude patterns — skip drafts from --all

Skip exploratory or draft tests from `--all` runs:

// e2e.config.js
export default {
  exclude: ['explore-*', 'debug-*', 'draft-*'],
};

Individual suite runs (`--suite`) are not affected by exclude patterns.

🤖 AI integration

The whole point: your agent writes, runs, and verifies tests for you.

Claude Code — plugin install & MCP-only install

claude plugin marketplace add fastslack/mtw-e2e-runner
claude plugin install e2e-runner@matware

This gives Claude 17 MCP tools, a workflow skill, 4 slash commands (`/e2e-runner:run`, `/e2e-runner:create-test`, `/e2e-runner:verify-issue`, `/e2e-runner:capture`), and 3 specialized agents (test-analyzer, test-creator, test-improver).

MCP-only install (tools only, no skill/commands/agents):

claude mcp add --transport stdio --scope user e2e-runner \
  -- npx -y -p @matware/e2e-runner e2e-runner-mcp

OpenCode

cp node_modules/@matware/e2e-runner/opencode.json ./
mkdir -p .opencode && cp -r node_modules/@matware/e2e-runner/.opencode/* .opencode/

See OPENCODE.md for details.

The 17 MCP tools

> Pool start/stop are CLI-only — not exposed via MCP.

Visual verification — describe the page, AI judges it

Describe what the page should look like — AI judges pass/fail from screenshots:

{
  "name": "dashboard-loads",
  "expect": "Patient list with at least 3 rows, no error messages, sidebar with navigation links",
  "actions": [
    { "type": "goto", "value": "/dashboard" },
    { "type": "wait", "selector": ".patient-list" }
  ]
}

After test actions complete, the runner auto-captures a verification screenshot. The MCP response includes the screenshot hash — Claude Code retrieves it and visually verifies against your `expect` description. No API key required.

Issue-to-test — turn a bug report into a runnable test

Turn GitHub and GitLab issues into executable E2E tests. Paste an issue URL and get runnable tests — automatically.

How it works:

1. Fetch — Pulls issue details (title, body, labels) via `gh` or `glab` CLI

2. Generate — AI creates JSON test actions based on the issue description

3. Run — Optionally executes the tests immediately to verify if a bug is reproducible

# Fetch and display
e2e-runner issue https://github.com/owner/repo/issues/42

# Generate a test file via Claude API
e2e-runner issue https://github.com/owner/repo/issues/42 --generate

# Generate + run + report
e2e-runner issue https://github.com/owner/repo/issues/42 --verify
# -> "BUG CONFIRMED" or "NOT REPRODUCIBLE"

In Claude Code, just ask:

> "Fetch issue #42 and create E2E tests for it"

Bug verification logic: Generated tests assert the correct behavior. Test failure = bug confirmed. All tests pass = not reproducible.

Auth: GitHub requires `gh` CLI, GitLab requires `glab` CLI. Self-hosted GitLab is supported.

📊 Dashboard & insights

e2e-runner dashboard                  # Start on default port 8484
e2e-runner dashboard --port 9090      # Custom port

Web dashboard tour — live view, history, gallery, pool status

Live execution — monitor tests in real-time with step-by-step progress, durations, and active worker count.

Test suites — browse all suites across projects. Run a single suite or all tests with one click.

Run history — track pass-rate trends with the built-in chart. Click any row to expand full detail.

Run detail — PASS/FAIL badges, screenshot thumbnails with copyable hashes (`ss:77c28b5a`), formatted console errors, and network request logs.

Screenshot gallery — browse all captured screenshots with hash search (action, error, and verification captures).

Pool status — Chrome pool health: available slots, running sessions, memory pressure.

Learning system — flaky tests, unstable selectors, slow APIs

The runner learns from every test run — building knowledge about your test suite over time. Query insights via the `e2e_learnings` MCP tool:

Storage & export:

• SQLite (`~/.e2e-runner/dashboard.db`) — default, zero setup
• Neo4j knowledge graph — optional, for relationship-based analysis. Manage via `e2e_neo4j` MCP tool or `docker compose`
• Markdown report (`e2e/learnings.md`) — auto-generated after each run

Test narration: Each test run generates a human-readable narrative of what happened step by step, visible in the CLI output and the dashboard.

Network error handling — assertions, global flag, full logging

Explicit assertion — place `assert_no_network_errors` after critical page loads:

{ "type": "goto", "value": "/dashboard" },
{ "type": "wait", "selector": ".loaded" },
{ "type": "assert_no_network_errors" }

Global flag — set `failOnNetworkError: true` to automatically fail any test with network errors:

e2e-runner run --all --fail-on-network-error

When disabled (default), the runner still collects and reports network errors — the MCP response includes a warning when tests pass but have network errors.

Full network logging — all XHR/fetch requests are captured with URL, method, status, duration, request/response headers, and response body (truncated at 50KB). Viewable in the dashboard with expandable request detail rows.

MCP drill-down flow:

1. e2e_run          → compact networkSummary + runDbId
2. e2e_network_logs(runDbId)                     → all requests (url, method, status, duration)
3. e2e_network_logs(runDbId, errorsOnly: true)   → only failed requests
4. e2e_network_logs(runDbId, includeHeaders: true) → with headers
5. e2e_network_logs(runDbId, includeBodies: true)  → full request/response bodies

The `e2e_run` response stays compact (~5KB) regardless of how many requests were captured. Use `e2e_network_logs` with the returned `runDbId` to drill into details on demand.

Screenshot capture — snapshot any URL on demand

Capture screenshots of any URL on demand — no test suite required:

e2e-runner capture https://example.com
e2e-runner capture https://example.com --full-page --selector ".loaded" --delay 2000

Via MCP, the `e2e_capture` tool supports `authToken` and `authStorageKey` for authenticated pages — it injects the token into localStorage before navigating.

Every screenshot gets a deterministic hash (`ss:a3f2b1c9`). Use `e2e_screenshot` to retrieve any screenshot by hash — it returns the image with metadata (test name, step, type).

🌐 Browser drivers

The runner can talk to multiple browser engines through different drivers. The default is `auto` — it probes each pool URL and picks the right driver per pool.

Pick a driver per test / force one per run

{
  "tests": [
    {
      "name": "checkout flow (heavy JS, real Chrome)",
      "driver": "browserless",
      "actions": [...]
    },
    {
      "name": "scrape product page (lightweight)",
      "driver": "obscura",
      "fallbackDriver": "cdp",
      "actions": [...]
    }
  ]
}

`driver` is optional. If set, only pools whose detected driver matches become candidates. `fallbackDriver` is explicit opt-in — without it, a missing target driver fails the test with a clear message. Pool busyness does not trigger fallback; the runner waits inside the filtered set.

Force a driver for a whole run (CLI overrides win over per-test fields — useful for A/B benchmarks):

e2e-runner run --all --driver obscura
e2e-runner run --all --driver obscura --fallback-driver cdp

Running each driver locally

# browserless (default) — managed by `pool start`
e2e-runner pool start

# Lightpanda — pool start uses templates/docker-compose-lightpanda.yml
e2e-runner pool start                 # with poolDriver: 'lightpanda' in config

# Obscura — install the binary and run it yourself
curl -LO https://github.com/h4ckf0r0day/obscura/releases/latest/download/obscura-x86_64-linux.tar.gz
tar xzf obscura-x86_64-linux.tar.gz
./obscura serve --port 9222 --stealth
# then point the runner at it: poolUrls: ['http://localhost:9222'], poolDriver: 'obscura'

⚙️ CLI, config & CI

CLI commands

# Run tests
e2e-runner run --all                  # All suites
e2e-runner run --suite auth           # Single suite
e2e-runner run --tests path/to.json   # Specific file
e2e-runner run --inline ''      # Inline JSON

# Pool management (CLI only, not MCP)
e2e-runner pool start                 # Start Chrome container
e2e-runner pool stop                  # Stop Chrome container
e2e-runner pool status                # Check pool health

# Issue-to-test
e2e-runner issue                 # Fetch issue
e2e-runner issue  --generate     # Generate test via AI
e2e-runner issue  --verify       # Generate + run + report

# Dashboard
e2e-runner dashboard                  # Start web dashboard

# Other
e2e-runner list                       # List available suites
e2e-runner capture               # On-demand screenshot
e2e-runner init                       # Scaffold project

CLI options

Configuration — e2e.config.js & priority

Create `e2e.config.js` in your project root:

export default {
  baseUrl: 'http://host.docker.internal:3000',
  concurrency: 4,
  retries: 2,
  actionRetries: 1,
  testTimeout: 30000,
  outputFormat: 'both',
  failOnNetworkError: true,
  exclude: ['explore-*', 'debug-*'],

  hooks: {
    beforeEach: [{ type: 'goto', value: '/' }],
  },

  environments: {
    staging: { baseUrl: 'https://staging.example.com' },
    production: { baseUrl: 'https://example.com', concurrency: 5 },
  },
};

Config priority (highest wins):

1. CLI flags

2. Environment variables

3. Config file (`e2e.config.js` or `e2e.config.json`)

4. Defaults

When `--env ` is set, the matching profile overrides everything.

CI/CD — JUnit XML & GitHub Actions

e2e-runner run --all --output junit

jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci
      - run: npx e2e-runner pool start
      - run: npx e2e-runner run --all --output junit
      - uses: mikepenz/action-junit-report@v4
        if: always()
        with:
          report_paths: e2e/screenshots/junit.xml

Programmatic API

import { createRunner } from '@matware/e2e-runner';

const runner = await createRunner({ baseUrl: 'http://localhost:3000' });

const report = await runner.runAll();
const report = await runner.runSuite('auth');
const report = await runner.runFile('e2e/tests/login.json');
const report = await runner.runTests([
  { name: 'quick-check', actions: [{ type: 'goto', value: '/' }] },
]);

Requirements

• Node.js >= 20
• Docker — only for Option 3 (the parallel Chrome pool). Options 1 & 2 don't need it.

License

Copyright 2026 Matias Aguirre (fastslack) — Matware

Licensed under the Apache License, Version 2.0. See LICENSE for details.