DAZZLE Test Infrastructure¶

Comprehensive guide to testing in the DAZZLE codebase.

Overview¶

DAZZLE has a multi-layered test infrastructure with 5,000+ test functions across 100+ test files. Tests are organized by scope (unit, integration, E2E) and execution context (local, pre-commit, CI).

┌─────────────────────────────────────────────────────────────────┐
│                        Test Pyramid                              │
├─────────────────────────────────────────────────────────────────┤
│                     E2E Tests (Playwright)                       │
│              ┌───────────────────────────────────┐              │
│              │  Browser automation, full stack   │              │
│              └───────────────────────────────────┘              │
│                                                                  │
│                   Integration Tests                              │
│         ┌─────────────────────────────────────────────┐         │
│         │  Multi-component, runtime pipeline              │         │
│         └─────────────────────────────────────────────┘         │
│                                                                  │
│                        Unit Tests                                │
│  ┌───────────────────────────────────────────────────────────┐  │
│  │  Parser, IR, Linker, Validator, CLI, Code Generation      │  │
│  └───────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘

Quick Reference¶

What	Command	When
Fast local tests	`pytest tests/unit -m "not slow" -x`	During development
All unit tests	`pytest tests/unit/`	Before committing
Integration tests	`pytest tests/integration/`	Before PR
JavaScript tests	`npm test`	For UI changes
Pre-commit checks	`pre-commit run --all-files`	Auto on commit
Full CI simulation	`pytest -m "not e2e" --cov`	Manual verification

Test Categories¶

1. Unit Tests¶

Location: tests/unit/ Framework: pytest + pytest-asyncio Execution: Local, Pre-commit (fast), CI

Core functionality tests covering:

Area	Files	Description
Parser	`test_parser.py`	DSL parsing, syntax
IR	`test_ir.py`, `test_ir_e2e_types.py`	Intermediate representation
Linker	`test_linker.py`	Module linking, resolution
Validator	`test_validator.py`	Semantic validation
State Machines	`test_state_machine_parsing.py`	State/transition parsing
Access Control	`test_access_rules.py`, `test_access_evaluator.py`	Permissions
CLI	`test_cli.py`, `test_cli_bridge.py`	Command-line interface
Code Generation	`test_adapters.py`, `test_ejection.py`	Ejection adapters
Layout	`test_layout_ir.py`	UI layout IR
Accessibility	`test_accessibility_checker.py`, `test_wcag_mapping.py`	WCAG compliance

# Run all unit tests
pytest tests/unit/ -v

# Run fast tests only (excludes subprocess spawning)
pytest tests/unit/ -m "not slow" -v

# Run with coverage
pytest tests/unit/ --cov=src/dazzle --cov-report=html

# Run specific test file
pytest tests/unit/test_parser.py -v

# Run tests matching pattern
pytest tests/unit/ -k "test_entity" -v

2. Integration Tests¶

Location: tests/integration/ Framework: pytest Execution: CI (after unit tests pass)

Multi-component tests:

File	Description
`test_runtime_pipeline.py`	AppSpec → BackendSpec/UISpec conversion
`test_runtime_e2e.py`	Full runtime pipeline
`test_end_to_end.py`	Complete workflows
`test_archetype_examples.py`	Archetype expansion
`test_golden_master.py`	Snapshot/golden master testing
`test_ux_semantic_layer.py`	UX semantic layer

# Run all integration tests
pytest tests/integration/ -v

# Run with early exit on failure
pytest tests/integration/ -v --maxfail=1

3. E2E Tests¶

3.1 Playwright Tests (Local Only)¶

Location: tests/e2e/ Execution: Manual (not in CI)

Local-only Playwright browser tests for page load, JS errors, and screenshots. See tests/e2e/README.md for details.

3.2 Semantic DOM Contract Tests¶

Location: tests/e2e/test_semantic_dom_contract.py Execution: CI (semantic-e2e job)

Tests semantic HTML contracts and data-dazzle-* attributes:

pytest tests/e2e/test_semantic_dom_contract.py -v --timeout=60

3.3 Serve Validation Tests¶

Location: tests/integration/test_serve_validation.py Execution: CI (e2e-smoke job), pytest -m "not e2e"

Tests dazzle validate and AppSpec-to-BackendSpec conversion (no server needed):

pytest tests/integration/test_serve_validation.py -v

4. Runtime Tests¶

Backend Tests (`tests/unit/http_runtime/`)¶

Runtime tests live under tests/; the wheel ships no tests (ADR-0057, flense ship H).

File	Description
`test_runtime.py`	FastAPI initialization, routing
`test_query_builder.py`	Query construction
`test_state_machine.py`	State transitions
`test_file_storage.py`	File uploads
`test_auth.py`	Auth store, sessions, middleware, routes

Computed fields, invariants and the rich-text allowlist are covered by tests/unit/test_computed_evaluator.py, test_invariant_evaluator.py and test_richtext_allowlist_parity.py.

UI Tests (`tests/unit/page/`)¶

File	Description
`test_ui_spec.py`	UI spec generation

5. JavaScript Tests¶

Location: src/dazzle/page/runtime/static/js/*.test.js Framework: Vitest (jsdom environment) Execution: Pre-commit, CI (js-test job)

# Run JavaScript tests
npm test

# Watch mode for development
npm run test:watch

Example-Specific Tests¶

Simple Task (`examples/simple_task/tests/e2e/`)¶

File: test_simple_task_generated.py Execution: CI (example-e2e job)

Auto-generated E2E tests with 12 test cases:

cd examples/simple_task
pytest tests/e2e/test_simple_task_generated.py -v

Execution Contexts¶

Local Development¶

Fast feedback loop during development:

# Fastest - unit tests, skip slow
pytest tests/unit -m "not slow" -x --tb=short

# JavaScript tests
npm test

# Type checking
mypy src/dazzle/core src/dazzle/cli
npm run typecheck

Pre-Commit Hooks¶

Automatically runs on git commit:

Hook	Description
`ruff`	Python linting + formatting
`mypy`	Python type checking (src/ only)
`bandit`	Security checks
`eslint`	JavaScript linting
`tsc`	JavaScript type checking
`vitest`	JavaScript tests
`dsl-validate`	DSL validation
`pytest-fast`	Fast unit tests

# Manual pre-commit run
pre-commit run --all-files

# Skip hooks (emergency only)
git commit --no-verify

CI/GitHub Actions¶

Runs on push to main/develop and pull requests (see .github/workflows/ci.yml). Local mirrors: make ci-fast (Tier 0) and make ci-core (Tier 1) — full mapping in Local CI concordance.

Job	Runs	Description
`python-tests`	All PRs	Non-e2e tests (Python 3.12 / 3.13 / 3.14 matrix)
`lint`	All PRs	Ruff, CSS gates, bandit, coverage gate
`type-check`	All PRs	`mypy src/dazzle` with full extras on 3.12
`security-tests`	All PRs	JWT fuzz, RBAC matrix, pip-audit hard-fail
`integration`	After core	Integration tests
`e2e-smoke` / `e2e-runtime`	After core	Example validate + Postgres e2e
`guide-walks` / `contracts-gate` / `interaction-walks`	After core	UX walkers
`homebrew-validation`	All PRs	Formula validation (macOS)

Test Markers¶

Standard Markers (pyproject.toml)¶

@pytest.mark.unit          # Unit tests
@pytest.mark.integration   # Integration tests
@pytest.mark.slow          # Slow tests (subprocess, full validation)
@pytest.mark.e2e           # End-to-end tests

Priority Markers (E2E tests)¶

@pytest.mark.high_priority    # P0 - blocks PRs
@pytest.mark.medium_priority  # P1 - informational
@pytest.mark.low_priority     # P2 - experimental

Feature Markers¶

@pytest.mark.story         # User story tests
@pytest.mark.access_control # Permission tests
@pytest.mark.state_machine  # State transition tests
@pytest.mark.invariant      # Business rule tests
@pytest.mark.dev_controls    # Dev control tests
@pytest.mark.computed       # Computed field tests

Coverage¶

Configuration (pyproject.toml)¶

[tool.coverage.run]
source = ["src/dazzle"]
omit = ["*/tests/*", "*/__pycache__/*"]

Generate Coverage Report¶

# Terminal report
pytest --cov=src/dazzle --cov-report=term-missing

# HTML report (opens in browser)
pytest --cov=src/dazzle --cov-report=html
open htmlcov/index.html

# XML for CI upload
pytest --cov=src/dazzle --cov-report=xml

Test Dependencies¶

Python (install with `pip install -e ".[dev]"`)¶

pytest >= 7.4
pytest-asyncio >= 0.23
pytest-cov >= 4.1
pytest-xdist >= 3.3 (parallel execution)
pytest-timeout >= 2.2
hypothesis >= 6.82 (property-based testing)
syrupy >= 4.0 (snapshot testing)
playwright (E2E tests)
httpx (API testing)

JavaScript (install with `npm install`)¶

vitest ^2.0.0
jsdom ^25.0.0
typescript ^5.0.0
eslint ^9.0.0

System¶

PostgreSQL (local install or managed instance)
Playwright browsers (playwright install chromium)

Troubleshooting¶

Tests timing out¶

# Increase timeout
pytest tests/e2e/ -v --timeout=300

# Run with verbose output
pytest -vv -l

Flaky E2E tests¶

# Re-run failed tests
pytest --lf

# Run specific test in isolation
pytest tests/e2e/test_file.py::test_specific -v

Pre-commit failures¶

# See what failed
pre-commit run --all-files -v

# Fix ruff issues
ruff check src/ --fix
ruff format src/

# Skip specific hook (debug only)
SKIP=pytest-fast git commit -m "message"

E2E Runtime issues¶

# Run E2E tests locally
pytest tests/integration/test_runtime_e2e.py -v -m e2e --timeout=180

Adding New Tests¶

Unit Test¶

# tests/unit/test_new_feature.py
import pytest
from dazzle.core.new_feature import NewFeature

class TestNewFeature:
    def test_basic_functionality(self):
        """Test basic feature works."""
        feature = NewFeature()
        assert feature.do_something() == expected

    @pytest.mark.slow
    def test_expensive_operation(self):
        """Test that spawns subprocess."""
        ...

    @pytest.mark.asyncio
    async def test_async_operation(self):
        """Test async functionality."""
        result = await feature.async_method()
        assert result is not None

E2E Test (Playwright)¶

# examples/my_example/tests/e2e/test_my_example.py
import pytest
from playwright.sync_api import Page

@pytest.mark.e2e
class TestMyExample:
    def test_page_loads(self, page: Page, ui_url: str):
        """Test main page loads."""
        page.goto(ui_url)
        assert page.title() == "My Example"

    def test_create_entity(self, page: Page, ui_url: str):
        """Test entity creation."""
        page.goto(f"{ui_url}/entity/create")
        page.fill('[data-dazzle-field="name"]', "Test")
        page.click('[data-dazzle-action="save"]')
        # Assert success

Suite Distillation (test authoring feedback loop)¶

tests/audit/ holds the suite-distillation artifacts (strategy: docs/proposals/Suite Distillation Strategy.md). Before adding non-trivial tests, check redundancy_report.md (extend an existing @pytest.mark.parametrize cluster rather than adding a standalone test), cross_file_report.md (copy-pasted shapes spanning handler/parser files) and taxonomy_report.md (do not add implementation mirrors or tautologies). The suite runs ~20k tests at ~7 ms each, so count is cheap but attention-per-failure and cluster bloat are not.

Regenerate after large test additions:

python scripts/distill/classify.py && python scripts/distill/cluster.py && python scripts/distill/cross_file.py

The 19 MB classification.json stays gitignored; the five summary artifacts are committed. Run the suite locally with pytest -n auto --dist loadgroup -m "not e2e" (~2 min; per-worker Postgres databases are provisioned automatically when a DB URL is set). Tests that boot an app subprocess against a repo directory (not tmp_path) must carry an xdist_group pin naming that directory's cohort.

Summary¶

Layer	Location	Local	Pre-commit	CI
Unit	`tests/unit/`	✅	✅ (fast)	✅
Integration	`tests/integration/`	✅	❌	✅
E2E (Playwright)	`tests/e2e/`	✅	❌	❌ (local only)
E2E (Semantic)	`tests/e2e/test_semantic_dom_contract.py`	✅	❌	✅
Backend runtime	`tests/unit/http_runtime/`	✅	✅ (fast)	✅
UI runtime	`tests/unit/page/`	✅	✅ (fast)	✅
JavaScript	`*.test.js`	✅	✅	✅
Example E2E (P0)	`examples/*/tests/e2e/`	✅	❌	✅
Example E2E (P1/P2)	`examples/*/tests/e2e/`	✅	❌	⚠️ (main only)

Last updated: February 2026