North star: Every critical behavior is proven by the right layer(s): white-box unit, black-box integration, and Playwright E2E — all three in CI.

---

The three layers (non-negotiable roles)

┌─────────────────────────────────────────────────────────────┐
│  E2E (Playwright)                                           │
│  User journeys, browser UX, a11y smoke, visual shells       │
├─────────────────────────────────────────────────────────────┤
│  Integration (black-box HTTP + real DB matrix)              │
│  Full hooks, cookies, RBAC, multi-DB parity, API contracts  │
├─────────────────────────────────────────────────────────────┤
│  Unit (Vitest white-box)                                    │
│  Domain logic, widgets, hooks, dispatcher edges, utils      │
└─────────────────────────────────────────────────────────────┘

Layer	Tooling	Proves	Does not replace
1 — White-box unit	Vitest (bun run test:unit)	Pure logic, fail-closed dispatcher, widgets, permissions math	Real cookies, multi-DB, browser
2 — Black-box integration	HTTP + preview (test:integration / harness)	Full middleware chain, auth sessions, adapter parity	Click paths, layout a11y
3 — E2E Playwright	Playwright (test:e2e)	Real user journeys, shell UX, smoke product paths	Exhaustive API matrix, all adapters

Rule of placement

If the risk is…	Prefer layer
Wrong formula / permission ID / Valibot rule	Unit
Wrong status/cookie/tenant under real server	Integration
User cannot complete a product path	E2E

Do not push API matrices into Playwright. Do not claim E2E coverage from unit mocks.

Behaviour contracts (the gate that maps to real defects)

The three layers say where a proof lives; they do not say what a route must prove. The external audit of 2026-10-24 found ten user-visible defects in ~6 700 green tests, and nine of them lay outside the P0 line-coverage set. Two gates close that class:

Gate	Command	Rule	Current
Route behaviour	bun run check:routes	Every route scores four contracts — load-contract, failure-path, persistence (mutate → re-read), concurrency — from test evidence tied to the route module or URL. Applicability is per route: persistence only for endpoints with a mutation verb or pages with actions; concurrency only where requests share mutable state.	99/99 applicable cells across 37 routes (floor 99)
Unfailable assertions	bun run lint:trivial-assertions	Rejects literal tautologies, complementary arms (x === 200 \|\| x !== 200), one-of status unions with no negative case, and expect swallowed by try/catch. Exemption: // trivial-assert-ok: <reason>.	0 findings / 1 068 test files
Mutation (seams)	bun run test:mutation	Rewrites one seam guarantee at a time; the mapped suite must fail.	7/7 killed

Both static gates run inside bun run check, so a route without a failure-path test or a test that cannot fail now reds the fast tier.

What the gate cannot see: visual chrome, control sizes, tab semantics, transactionality across a file write, and adapter capability gaps (e.g. media search). Those need visual regression, a11y sweeps, fault injection and adapter-contract tests respectively.

Thin helpers fit (unit only)

Shared code: tests/unit/utils/mock-event.ts + rbac-matrix.ts (see API Unit Helpers).

Pattern	Helper	Layer
Build event + dispatch	invokeApi / expectApi	Unit
RBAC table (role × method × path → status)	runRbacMatrix	Unit
GraphQL auth gate	invokeGraphql	Unit
Multipart media	mockFormData + invokeApi	Unit
Namespace completeness	namespace-ownership.test.ts	Unit
Real cookies / multi-DB	safeFetch + integration suite	Integration
Product journey	Playwright + test ids	E2E

---

Recipes (copy patterns)

RBAC (unit → integration → E2E)

// Unit — dispatcher authz only
await runRbacMatrix([
  {
    name: "editor denied",
    method: "POST",
    path: "collections/posts",
    user: editor,
    roles: editorNoPerms,
    expectedStatus: 403,
  },
]);
// Integration — real session cookie as editor → same path returns 403
// E2E — editor UI does not show admin-only actions

Collections (unit → integration → E2E)

// Unit — missing tenant / unauth edges with mocks
await expectApi("GET", { path: "collections", user: admin, tenantId: null }, [400, 403]);
// Integration — full CRUD + publish filters on each DB
// E2E — open builder, save schema, create entry (collection-builder-flow helpers)

GraphQL

// Unit
await invokeGraphql("{ __typename }", {}, { user: null, bypass: false }); // 401
// Integration — tests/integration/api/graphql.test.ts
// E2E — do not re-test GraphQL in the browser

---

What “100%” means per layer

Layer 1 — Unit (white-box)

Inventory	100% means
Core widgets	Every core widget has unit tests (see widget-test-coverage)
Auth/permission helpers	Lockout, password strength, hasPermissionWithRoles, fail-closed maps
Hooks security	Defense-in-depth suite green (security-testing)
API dispatcher edges	Authn 401, authz 403, unknown namespace 404, multi-tenant 4xx (dispatcher-security-matrix, invokeApi)
Utils/domain modules	New public utils ship with unit tests in the same PR

Supporting metric: bun run test:unit -- --coverage (v8) — track thresholds on src/databases/auth, src/hooks, src/utils, src/widgets first; not all of src/.

Layer 2 — Integration (black-box)

Inventory	100% means
DB matrix	Same contract suite green on SQLite, MongoDB, MariaDB, PostgreSQL
Auth & users	Login, 2FA, lockout, tokens, user CRUD/batch HTTP
Content & media	Collections CRUD, media upload/delete, search
System	Settings, health, permissions, theme, cache
Security negatives	Rate limit, XSS/SQLi probes, setup gating after complete
API namespaces	Every production namespace in NAMESPACE_CONFIG has ≥1 happy + ≥1 fail path in unit or integration

Definition of done for multi-DB: no “SQLite-only green + known skew” left untracked — either fixed or filed with owner.

Layer 3 — E2E (Playwright)

Inventory	100% means
Setup	Wizard completes on CI DB path
Auth	Login + session + logout smoke
Collection builder	Create schema smoke + one entry path (stable test ids)
Config shell	Major config hubs load (not every toggle)
RBAC UI	Admin vs restricted affordance where UI enforces it
A11y smoke	Login/keyboard focus smoke green
Visual	Stable admin shells only (not volatile builder canvas)

Definition of done for E2E: every P0 product journey in the matrix has a non-flaky CI project — not every button in the admin.

See e2e-coverage-matrix for route-level gaps.

---

Seam coverage — why “100%” missed real bugs

A green three-layer suite can still miss a regression when a test proves a different contract than the UI sends, or stops before the state is actually persisted. Real escapes that slipped through all three layers recently hardened these three non-negotiable seam rules:

Identifier / contract parity

Feed the test exactly the values the UI sends. A unit test that passes a different identifier than the UI is proving a phantom contract — it can go green while the product is broken.

• Token edit/delete resolved the hashed token value shown in the admin list, but the API re-hashes on lookup, so the identifier could never resolve. The fix resolves by row _id, and delete fails loudly on deletedCount === 0.
• Token validity edits sent expiresInHours (not a column) and silently no-op’d. The UI now sends a concrete ISO expires.

State persistence assertions

For mutations, prove the side effect with create → reload → assert → cleanup. “A toast appeared” is not persistence — reload and re-assert before you call it covered.

Failure / alternate paths

Every mutation endpoint and route needs its fail path, not just the happy path: deletedCount === 0, an invalid invite token, a stale ?edit= id, and 500-vs-404 copy. A 500 must never render “page moved” — resolveErrorDisplay in src/utils/error-display.ts now makes “page moved” 404-only, and recovery actions are variant="outline".

Route coverage is now CI-enforced

tests/route-test-coverage.ts (APP_ROUTE_COVERAGE + verifyAppRouteCoverage()) is enforced by tests/unit/route-test-coverage.test.ts in bun run test:unit:

• Bidirectional inventory/file parity — every src/routes/(app) group has a manifest row and every row maps to a real group; adding a route without a row, or deleting/emptying its covering test, fails the gate.
• Layer-agnostic — a route is covered if any of unit / integration / E2E provides a test. Layer depth stays with tests/critical-test-paths.ts and the P0 lists below.

---

Completeness scorecard (how we track “100%”)

Use three scores; all three must hit target for “complete.”

Score	Formula	Target
Unit inventory	Critical unit modules with suite / inventory size	100%
Integration inventory	Contract domains green × 4 DBs / (domains × 4)	100% on CI matrix
E2E P0 journeys	P0 journeys green in CI / P0 list	100%

P0 E2E journeys (must stay green)

1. Setup wizard completes (CI)
2. Admin login → authenticated shell
3. Collection builder: open + new draft (+ save when stable)
4. Create/list entry on a collection (or smoke equivalent)
5. Logout / unauthenticated redirect
6. Permission denial UI or navigation for non-admin (where applicable)
7. A11y smoke on login

P0 API / integration domains (must stay green on all DBs)

1. Auth login/session
2. User CRUD + batch
3. Collections content CRUD
4. Media write/delete permissions
5. Setup gating (post-complete blocked)
6. Fail-closed unknown / forbidden routes
7. Multi-tenant isolation when MT enabled

---

How the three layers work together (example)

“Editor cannot publish but can create draft”

Layer	Test
Unit	Permission helper + dispatcher collections:write vs publish flag
Integration	Cookie as editor → POST entry 201, publish action 403
E2E	Editor UI hides/disables Publish; admin sees it

Missing any one layer leaves a gap (logic-only, HTTP-only, or UI-only).

---

Current baseline (order of magnitude)

Counts drift with every wave — test-status.mdx is the single source of truth for file/test counts.

Layer	Scale (approx.)	Health signal
Unit	see test-status.mdx	Local full unit green in a single run
Integration	see test-status.mdx	SQLite strong; multi-DB skew tracked
E2E	see test-status.mdx	Flakes on builder; smoke + journey hardening in progress

Gap themes

1. Multi-DB parity not yet “all green without asterisks” (Phase B)
2. E2E matrix has known ❌ rows (setup presets, some config edges) — Phase C; see E2E matrix
3. Namespace owners are now gated in unit (Phase D); some owners still unit-only until HTTP cases land

Ultra-smart additions (2026-07-18) — headless-safe, not E2E bloat:

Layer	Suite	Path
Unit	Session cookies (loopback Secure)	tests/unit/auth/sessions/session-cookies.test.ts
Unit	Client import boundary (collaboration / client roots)	tests/unit/live/client-import-boundary.test.ts
Unit	Testing API login dual-write	tests/unit/api/testing-login-cookie-contract.test.ts
Unit	Page guards + collectionbuilder load	page-guards · collectionbuilder-page-server
Integration	Session → (app) __data.json (18 admin shells)	tests/integration/api/session-page-load.test.ts
Integration	Webhooks HTTP happy + deny (P1 ADR domain)	tests/integration/api/webhooks.test.ts
Unit	Exhaustive client import-boundary walk	tests/unit/live/client-import-boundary.test.ts

Inventory & tiers: headless-test-inventory.mdx.

---

Operating rules toward 100%

1. PR rule: Touching behavior in inventory requires the matching layer test(s) in the same PR.
2. No layer shopping: Don’t skip integration because unit is green.
3. Flake = red: Flaky E2E counts as incomplete until hardened or replaced with a better layer.
4. Fixtures: @tests/harness only for tenants/users/roles.
5. API unit helpers: invokeApi / createMockRequestEvent — no second HTTP client for unit.
6. Local loop:
   • Unit: bun run test:unit
   • Integration: bun test --timeout 300000 tests/integration/
   • E2E: project-scoped Playwright (not full matrix every commit)

---

Commands (completeness suite)

# Layer 1 — white-box
bun run test:unit

# Optional supporting coverage (auth/hooks/utils focus over time)
bun run test:unit -- --coverage

# Layer 2 — black-box (day-to-day; needs build/)
bun test --timeout 300000 tests/integration/

# Layer 2 — matrix (pre-merge for adapter/auth work; Docker up)
DB_TYPE=postgresql bun test --timeout 300000 tests/integration/
DB_TYPE=mariadb bun test --timeout 300000 tests/integration/
DB_TYPE=mongodb bun test --timeout 300000 tests/integration/

# Layer 3 — E2E (CI projects / smoke)
bun run test:e2e
bun run test:e2e:quick -- --project=chromium

# Unit + SQLite integration (local health)
bun run test:doctor
bun run test:all

---

Roadmap to 100% (ordered — dependency-checked)

Order is intentional: unit edges + ownership first, then HTTP multi-DB truth, then browser P0, then CI reporting.

Phase	Focus	Exit criteria	Status (2026-07)
A	Unit inventory locked	Widgets + hooks security + dispatcher matrix green; API helpers standard	In progress
B	Integration multi-DB P0	P0 domains green on 4 DBs; batch/bulk HTTP covered	In progress (SQLite strong)
C	E2E P0 journeys	7 P0 journeys non-flaky in CI; builder test ids	In progress
D	Namespace ownership	Every NAMESPACE_CONFIG key has unit/integration owner	Gate live (namespace-ownership.test.ts)
E	Scorecard in CI	Inventory % (owners, P0 journeys), not vanity line %	Backlog

Why this order

1. A before B — Unit fail-closed/RBAC catches wiring bugs before multi-DB CI burn.
2. B before C — Browser journeys assume auth/API contracts; multi-DB is not a Playwright problem.
3. D with A — Ownership inventory fails the build when a namespace is added without tests (API_NAMESPACE_KEYS + NAMESPACE_OWNERS).
4. E last — Metrics without A–D definitions become vanity dashboards.
5. Never put bulk API matrices in E2E; never treat unit alone as multi-DB proof.

Backlog mapping

Item	Phase	Improved by
Multi-DB auth/permission green matrix	B	Run P0 checklist per adapter; track skew as bugs; contract.test.ts PermissionContract
GraphQL isolation	A + B	Unit resolvers + dispatcher 401; integration graphql.test.ts
Migrate remaining tests/unit/api/* → invokeApi	A	Opportunistic; ownership gate does not require full migration

Multi-DB P0 checklist (per adapter)

bun test --timeout 300000 tests/integration/api/
DB_TYPE=postgresql bun test --timeout 300000 tests/integration/api/
DB_TYPE=mariadb bun test --timeout 300000 tests/integration/api/
DB_TYPE=mongodb bun test --timeout 300000 tests/integration/api/

1. Login + session cookie
2. User batch block/unblock/delete
3. Collections auth negatives
4. Media permission negatives
5. Setup complete → setup API blocked
6. Fail-closed unknown/forbidden
7. GraphQL unauthenticated → 401

Namespace ownership (Phase D)

bun run test:unit -- tests/unit/api/namespace-ownership.test.ts

---

Related

• Testing Strategy
• Black-Box Testing
• API Testing
• API Unit Helpers
• E2E Coverage Matrix
• Test Status
• Security Testing

---

Last Updated: 2026-07-15