ADR: Testing 2026 — Pyramid, Fixtures, Soft-Skip Ban, Testids
Accepted and implemented: pyramid, seed-first E2E, soft-skip ban, shared CSRF client, fail-closed testing API, reference routes.
On this page
| Field | Value |
|---|---|
| Status | Accepted + Implemented (core admin rollout complete 2026-07-18) |
| Date | 2026-07-18 |
| Reference routes | Webhooks (canonical), automations, redirects, trash |
| Policy source | AGENTS.md (E2E & Control-Map Testing Policy) |
Context
Route hardening (2026-07) expanded control maps and data-testids quickly. That improved security/UX consistency but tilted the suite toward:
- Brittle shell E2E (“every testid exists”)
- Soft-skips when lists were empty (green CI, hidden regressions)
- Duplicated CSRF/fetch in Svelte pages
- Risk of treating
/api/testingseeds as a production-facing backdoor
This ADR locks how admin/product routes are tested and structured so UI refactors do not erase proof, empty installs cannot skip mutating journeys, and testing infrastructure stays fail-closed outside CI.
Decision (summary)
- Pyramid — unit-heavy; E2E only for golden outcome journeys
- Soft-skip ban on control-map rows — seed via
/api/testinginstead - Fixture-first E2E — login → seed (if needed) → journey → cleanup
- Testid policy — control-risk anchors only
- Shared mutation client —
fetchApialways attaches CSRF on non-GET - Testing API security — production-hard-closed; same gate as test bypass
- Type gate —
bun run checkrunstsc --noEmit+svelte-check(2026-10-18) - Composite DOM coverage — Tabs, dropdowns, confirm dialogs, SmartTable selection/sort, bulk Multibutton (2026-10-18)
- Local DB parity —
bun run test:paritymirrors the CI contract suites on every reachable engine (2026-10-18) - Explicit flake budget — retried-pass count/rate is a CI gate, not a retry side effect (2026-10-18)
- No force-click workarounds — E2E drives the real hit target; remaining
forceuses require a documented overlay reason (2026-10-18) - Composite Editorial Journeys & Zero-Error Trap — Real multi-widget lifecycle (
kitchen-sink.spec.ts), non-admin editor boundary (editor-boundary.spec.ts), real-browser vitality (browser-vitality.spec.ts), and client widget DOM testing (widget-inputs-interaction.test.ts) attached to the fail-on-error trap (zero-error-guard.ts) (2026-10-23) - Rolling flake trend + nightly repeat probe — the 7-day rate is the gate (a single 1/29 shard is noise); the 20 newest specs are re-run nightly with
--repeat-each=3to feed it (2026-10-23)
1. Testing pyramid (target ratio)
| Layer | Share (guidance) | Proves | Tools |
|---|---|---|---|
| Unit | ~62% | Pure utils, validation, page.server gates, *-api.ts clients |
Vitest |
| Component | ~8% | Real DOM events, Svelte 5 runes, disabled/ARIA/modal wiring | Vitest + jsdom (client build) |
| Integration | ~20% | HTTP + DB, RBAC, settings/webhooks services | Integration suite |
| E2E | ~10% | Golden journeys (outcome), sparse shell guards | Playwright |
“100% control map” means every control risk has the right layer(s)—not 100% of DOM nodes.
2. Soft-skip ban (control-map rows) — enforced
| Rule | Practice |
|---|---|
| Forbidden | test.skip because a list is empty, no sample data, or “install-specific empty state” |
| Required | Seed via POST /api/testing (tests/e2e/helpers/api.ts + handlers/testing.ts) |
| Allowed hard-fail | Optional external fixtures (Docker Postgres, real IdP) after seed returns structured 503 / unavailable code |
| Policy text | AGENTS.md |
3. Fixture-first E2E — enforced
login → seed (if needed) → UI journey → assert outcome → cleanup (finally)
| Prefer | Avoid |
|---|---|
| Create → reload → value still present | Asserting 12 decorative testids |
| Delete → name gone | Fixed waitForTimeout(800) |
| URL / API / list text | Soft-skip when empty |
Shared helpers (implemented):
| Helper | Path |
|---|---|
| Seeds | tests/e2e/helpers/api.ts — seedWebhook, seedAutomation, enablePlugin, … |
| Confirm modal | tests/e2e/helpers/confirm-modal.ts |
| Migration wizard | tests/e2e/helpers/migration-wizard.ts — ensureSmartImporterReady (hard-fail, no skip) |
Testing API actions (behind fail-closed gate only):
| Action | Purpose |
|---|---|
seed-webhook / delete-webhook |
Webhook list fixtures |
seed-expired-password-reset |
Past-expiry password_reset token (TOKEN_EXPIRED toast) |
seed-media-with-metadata |
Media rows with distinct metadata for ?jsonPath= |
outbox-emit / outbox-process-batch |
Transactional outbox fixtures |
outbox-tx-rollback |
Outbox row rolled back with failed SQL transaction |
plugin-storage-* |
Plugin storage create/get/list/delete |
seed-automation / delete-automation |
Automation list / search fixtures |
seed-trash / purge-trash |
Soft-deleted entry for restore golden E2E |
enable-plugin |
Core plugin enable for wizard E2E (pluginId sanitized) |
4. Testid policy — enforced on reference specs
| Use testids for | Do not require testids for |
|---|---|
| Primary actions (add/save/delete) | Every badge, icon, spacer |
| Modal root + critical fields | Entire field matrix when catalog varies |
| Loading/empty states used as wait conditions | Redundant duplicates of role+name |
Prefer getByRole / accessible name when stable.
5. Hydration gate — required before driving fresh pages
The root layout sets <html data-hydrated="true"> from its onMount, which only
runs after Svelte 5 finished the (synchronous) SSR hydration walk. Test helpers
wait on it via waitForHydration(page) (tests/e2e/helpers/auth.ts;
loginAsAdmin already gates its own navigations).
Why it is mandatory: auto-waiting locators cannot see this gap. SSR markup is present and stable, so a click resolves, fires and simply never reaches a handler — observed 2026-09-21 as 20–25 s timeouts whose element “was never found” even though it was in the DOM (dashboard widget menu, extensions tab switch, media-gallery folder dialog, upload-media tabs). Symptoms of a missing gate:
- the assertion times out on an element that SSR HTML clearly contains, and
- the page state did not change (e.g. the previous tab is still selected).
Call waitForHydration(page) after every spec-local page.goto that is
followed by an interaction (helpers such as openGallery, openUploadMedia,
goExtensions, goDashboard do this). A spec-local helper that navigates is the
usual place where the gate is forgotten.
5.1 Hydration helpers — tests/e2e/helpers/hydration.ts
A retry loop invented inside one spec is a bug in the helper layer. Use these
instead of copy-pasting expect(async () => { … }).toPass():
| Helper | Use when | Failure names |
|---|---|---|
runAfterHydration(page, action, { selector? }) |
one-shot interaction (click, upload) on a fresh page | the marker the page never reached, or the element that never attached |
selectAndWaitForUrl(page, select, value, pattern, opts?) |
a select change that should navigate |
the selected value, the select locator, the URL pattern |
fillUntil(page, locator, value, assertion, opts?) |
a fill whose binding drives dirty/validation state |
the value, the locator, the assertion that never passed |
All three wait on html[data-hydrated="true"] first, retry through Playwright’s
toPass budget (never waitForTimeout), and surface a message naming the
interaction that never took effect.
bun run scripts/lint/lint-e2e-hydration.ts --check fails on any direct
.dispatchEvent("change") in a tests/e2e spec; route it through the helpers,
or annotate // hydration-ok: <reason> on the same or the previous line. (The
fill → dirty-button rule is deliberately not automated: three-line proximity
cannot separate an unrelated fill from a dirty-flag assertion without false
positives, so fillUntil carries that intent explicitly.)
The 2026-09-21 upload gap in move-remote.spec.ts is closed: the spec drives
the header Upload button and waits on filechooser, which only fires once the
input’s handler is attached — a hydration gate in the user path, with no
synthetic re-dispatch.
5. Shared mutation client — implemented
| Piece | Location | Status |
|---|---|---|
| CSRF on mutations | fetchApi → clientJsonHeaders (src/utils/api.ts) |
✅ |
| Domain API clients | webhooks-api.ts, automations-api.ts, trash-api.ts |
✅ |
| Pure validation | webhooks-utils.ts, redirects-utils.ts, … |
✅ |
| Admin gates | +page.server.ts unit tests |
✅ |
| Access save / plugins / sync | fetchApi (no manual CSRF) |
✅ |
Canonical stack (copy for new routes):
+page.server.ts → admin gate (unit)
*-utils.ts → validate/filter (unit)
*-api.ts → list/create/update/delete via fetchApi (unit, mock fetch)
+page.svelte → UI + showConfirm only (no raw fetch / manual CSRF)
E2E → one golden create→list→reload→delete (or equivalent)
6. Security: testing API is not a production backdoor — hardened
Seed/reset under /api/testing exist only for CI/E2E.
| Control | Implementation | Proof |
|---|---|---|
| Production hard-close | NODE_ENV=production → 403 |
testing-api-gate.test.ts |
| Env allowlist | TEST_MODE / PLAYWRIGHT_TEST only — not bare NODE_ENV=test, and not BENCHMARK (benchmarks run production-mode with real sessions) |
same |
| Secret | Timing-safe x-test-secret vs TEST_API_SECRET |
same + route-access-audit |
| Authz bypass | Same production hard-gate in applyTestBypassFromRequest |
test-bypass.server.ts |
| Bundle strip | Vite testBackdoorStripperPlugin + verify-prod-build-backdoor |
unit + CI build |
| Shared entry | assertTestingApiAllowed used by handleTestingRoutes |
no weaker second gate |
Agents: Do not add seed routes outside this handler. Do not weaken the gate. Do not commit production secrets. Known e2e default secret strings are rejected outside an explicit test env.
Testing API vs Production Benchmarks & Compliance Audits
/api/testingis strictly for CI/E2E fixture orchestration and state resets. It is never available on production servers (NODE_ENV=production→ 403TESTING_DISABLED_PRODUCTION) and is stripped from deployable builds viatestBackdoorStripperPlugin.- Benchmarks and regulatory compliance suites must exercise authentic production APIs:
- Regulatory audits (e.g. GDPR Right to Erasure
right-to-be-forgotten-audit.test.ts) must never target/api/testing. - Instead, they target real endpoints (such as
POST /api/gdprwithaction: "erase"orPOST /api/user/batchwithaction: "delete") using real authenticated session cookies (benchmarkAuthHeaders()). - This ensures benchmarks measure authentic production latency without requiring
requireTestInfrastructureguards or being excluded from the multi-database benchmark matrix.
- Regulatory audits (e.g. GDPR Right to Erasure
Shared infrastructure
| Piece | Path |
|---|---|
| CSRF mutation client | src/utils/api.ts + src/utils/security/client-csrf.ts |
| Testing / bypass gate | src/utils/test-bypass.server.ts (assertTestingApiAllowed) |
| Testing handler | src/routes/api/[...path]/handlers/testing.ts |
| E2E seeds | tests/e2e/helpers/api.ts |
| Confirm helper | tests/e2e/helpers/confirm-modal.ts |
| Checkbox helper | tests/e2e/helpers/checkbox.ts — clicks label[for] for sr-only inputs |
| Gate unit tests | tests/unit/utils/testing-api-gate.test.ts |
| Prod strip proof | tests/unit/scripts/verify-prod-build-backdoor.test.ts |
| Smart Test Selector | scripts/test/test-smart.ts — unioned signals, synthetic edges, surgical diff-filtering |
| Component interaction | vitest.dom.config.ts + tests/unit/components/dom/ (client Svelte, jsdom) |
| Zero-mock OIDC peer | tests/integration/helpers/oidc-test-server.ts — real HTTP provider on 127.0.0.1:0 (discovery, /authorize, /token + PKCE, /jwks, logout-token minting) |
| Local DB parity | scripts/runs/run-local-db-matrix.ts — bun run test:parity (--docker, --only, --require-all) |
| Flake budget | scripts/check/check-flake-budget.ts — bun run flake:budget (reads Playwright JSON) |
| Flake trend | scripts/check/check-flake-trend.ts — rolling 7-day rate from tests/e2e/.flake-history.jsonl (--append, --check) |
| Nightly flake probe | scripts/runs/e2e-flake-nightly.ts — --repeat-each=3 over the 20 newest tests/e2e specs (--list to inspect) |
| Canonical fixtures | tests/harness — ADMIN_CREDENTIALS / USERS / TEST_PASSWORD (all layers) |
| Integration harness | tests/integration/harness.ts — singleton preview, private.test.ts, seed isolation |
| E2E CI-parity runner | bun run test:e2e → scripts/runs/run-e2e.ts (preview :4173, harness pre-check) — --dev for Vite :5173, --no-build to reuse a build |
A++ suite contract (2026-07-19)
Enterprise stability rules that every layer must honor:
| Rule | Unit | Integration | E2E |
|---|---|---|---|
| One identity | @tests/harness |
seed uses harness emails | helpers/auth.ts → harness |
| Right altitude | pure logic / gates | HTTP + multi-DB | golden journeys only |
| No soft-skip empty | n/a | n/a | seed or hard-fail |
| Crash isolation | n/a | detect socket death → restart | storageState fail-closed |
| Local = CI binary | Vitest | preview node build |
test:e2e = preview :4173 |
| P0 floors | bun run test:coverage thresholds (vitest.config.ts) |
multi-DB matrix in CI | P0 journeys in critical-test-paths.ts |
| Push smoke | smart unit + security | auto SQLite + local parity when src/databases changes |
not on push (CI-only) |
Commands (canonical):
bun run test:unit # Layer 1
bun run test:unit:components # Layer 1b — real DOM component interaction (jsdom)
bun run test:coverage # Layer 1 + P0 coverage floors (thresholds in vitest.config.ts)
bun run test:integration # Layer 2 (builds + multi-file)
bun run test:integration:smoke # Layer 2 quick SQLite
bun run test:parity # Layer 2b — contract suites on every reachable local engine
bun run test:e2e # Layer 3 CI-parity (preview :4173)
bun run test:e2e:dev # Layer 3 Vite :5173 (local only)
bun run flake:budget # Layer 3 flake gate (retried-pass budget)
6. Component interaction layer — implemented 2026-10-18
The unit suite runs under an SSR-forced Vitest config, so SSR string assertions
(render(Component).not.toThrow()) cannot prove that a control is wired. The DOM
suite closes that gap without Playwright:
| Piece | Path / command |
|---|---|
| Config | vitest.dom.config.ts — conditions: ["browser"], import.meta.env.SSR: "false", jsdom |
| Harness | tests/unit/components/dom/render.ts — renderComponent / click / type / select / press |
| Suites | Button, Checkbox, Toggle, Input/Select/Textarea, Modal, ModalPrompt, Tabs, Dropdown, ConfirmDialog, SmartTable saved views + selection/sort, TableIcons, Multibutton, System Settings, Access Management role modal, Workflow editor, Webhooks, Media Gallery folders, Collection Builder field editor + validation/conditions/permissions/preview (19 files / 118 tests) |
| Command | bun run test:unit:components (pre-commit + CI 04-unit) |
The default vitest.config.ts excludes tests/unit/components/dom/** — running
those files under the SSR config resolves svelte to its server entry and mount()
throws lifecycle_function_unavailable.
Rules:
- An interaction test must mount the component, act on real DOM (click/label/key), and assert the observable outcome — not just that render did not throw.
- Prefer
data-testidanchors on controls the E2E suite also drives (ModalPrompt now exposesmodal-prompt-input|confirm|cancel). A testid must be unique in the rendered tree:lint-duplicate-testids.tsfailsbun run checkwhen a route passes a literal testid to a component that already renders the same one (Playwright strict mode resolvesgetByTestId()to 2 elements — that is how the media gallery’s wrapperAdminCardbroke the Media & Dashboard shard). Suppress a deliberate reuse with atestid-allowcomment. - Keep the suite fast: it must stay under ~15 s locally (19 files / 118 tests currently run in ~13 s) so pre-commit can run it on every commit.
- Never mock a component’s own event wiring — mount it and click/keypress the real control.
7. Reliability gates — implemented 2026-10-18
Six gaps let regressions pass local hooks. Each now has an explicit gate:
| Gap | Gate | Where |
|---|---|---|
| Type drift | tsc --noEmit + svelte-check (errors and warnings gated) |
bun run typecheck, appended to check; CI 03-check |
| Composite UI untested | 20 jsdom suites (see §6) | bun run test:unit:components (pre-commit) |
| E2E workarounds hide bugs | Removed force from user-management, permission, profile-link and redirect-delete clicks; toggleCheckbox() clicks the visible label |
tests/e2e/routes/user/management.spec.ts, tests/e2e/helpers/checkbox.ts |
| DB parity was CI-only | bun run test:parity runs contract suites on every reachable engine; pre-push runs the core contract when src/databases/ changed |
scripts/runs/run-local-db-matrix.ts, .githooks/pre-push step 7b |
| Coverage floors too low | Raised to 72/72/60/70 (lines/functions/branches/statements) from measured 75.73/75.71/64.33/73.92 | vitest.config.ts |
| Security core untested | Real suites for 2FA, handle-security, submission replay, SSO/OIDC (discovery, JWKS, exchange, back-channel claims, logout), and OAuth state (109 tests) |
tests/unit/databases/two-factor-auth-service.test.ts, tests/unit/hooks/handle-security.test.ts, tests/unit/security/submission-guard.test.ts, tests/unit/auth/sso-session.test.ts, tests/unit/auth/google-auth.test.ts |
| Admin route forms | jsdom suites for System Settings (11), Access Management role modal (8), Workflow editor (6), Webhooks (10), Media Gallery folders (6), Collection Builder field editor (16), Collection Builder validation/conditions/permissions/preview (8), Entry editor create skeleton (2) | tests/unit/components/dom/system-settings-interaction.test.ts, .../access-management-interaction.test.ts, .../workflow-interaction.test.ts, .../webhooks-interaction.test.ts, .../media-folders-interaction.test.ts, .../collection-builder-fields-interaction.test.ts, .../collection-builder-conditions-interaction.test.ts, .../entry-fields-create-skeleton.test.ts |
| Mocks hid boundary bugs | oidc-live integration (11): real in-process IdP, real TCP/safeFetch/JWKS; the only seam is an exact-origin test allowlist that throws outside TEST_MODE |
tests/integration/auth/oidc-live.test.ts, tests/integration/helpers/oidc-test-server.ts, src/utils/egress-guard.ts (allowTestEgressOrigin) |
| Preview ≠ product gap | showIf/requiredIf are evaluated by the live entry form via isFieldVisible/isFieldRequired (unit-tested), not only by the builder preview; hidden fields also drop stale required errors |
src/widgets/desugar-field.ts, src/components/collection-display/fields.svelte, tests/unit/widgets/desugar-field.test.ts |
| Integrated page loops | Fresh-build smoke + pageerror/console capture; the create-entry journey asserts the title field appears. A DOM suite alone cannot catch a loop caused by field-identity churn across WidgetLoader |
tests/e2e/smoke.spec.ts, src/components/collection-display/widget-loader.svelte |
| Flake control reactive | ≤2 retried-pass per shard; a single retry is governed by --max-flaky (0 in pre-push), the percentage rate only applies when >1 retry and ≥20 executed |
scripts/check/check-flake-budget.ts, CI E2E steps; local Playwright writes the same JSON |
| Whole-code % invisible | Full-source unit report + client DOM report unioned best measurement per file (never summed: builds have incompatible statement maps) | bun run test:coverage:all, scripts/merge-coverage.ts, vitest.dom.config.ts |
| Route breadth unmeasured | 18 static admin routes must render a shell (HTTP < 400, no /login, no error page) |
tests/e2e/routes/config/route-breadth.spec.ts |
| Adapter-specific bulk bug | Real MariaDB contract run catches what a mock/SSR suite cannot: prepareUpdateValues omits the PK on updates, which the MariaDB bulk strategies need for the JOIN/CASE match |
tests/integration/databases/bulk-operations-contract.test.ts (20/20 on MariaDB) |
E2E actionability rule — do not use force: true on a normal in-page control. If a
control is obscured, that is a product bug (z-index/pointer interception). force is
reserved for documented animation/consent overlays; evaluate(el => el.click()) is
reserved for controls Playwright genuinely cannot hit and must include a reason.
E2E worker rule — the CI-parity runner (bun run test:e2e) defaults to one worker,
like every CI shard. Parallel spec files share one SQLite E2E database and race on
reset/seed, which shows up as retried-pass flakes; pass --workers=N only when you
accept that contention.
Flake measurement rule — local Playwright writes tests/playwright-results.json.
Retries are off locally by default; run with PLAYWRIGHT_RETRIES=1 when you want a
recovered test to be recorded as flaky, then bun run flake:budget applies the same
≤2 / ≤3 % budget CI uses.
Rolling flake rule — the per-run budget stays as a shard-local early warning, but the
gate is the rolling window: check-flake-trend.ts --append=<json> records
{ at, label, executed, flaky, failed } in tests/e2e/.flake-history.jsonl, and
--check fails when the 7-day window exceeds --max-flaky=0 / --max-rate=1. One flaky
test out of 29 executed (3.45 %) no longer fails a shard; a flake that keeps recurring
does. The history file is generated state — CI carries it with a cache keyed on the
workflow, never a commit.
Nightly repeat rule — e2e-flake-nightly.ts selects the N newest specs by
git log -1 --format=%ct (path ascending as tie-break, filesystem mtime for untracked
specs) and runs them through run-e2e.ts --no-build --repeat-each=3, so a 2-of-3 pass
arrives at the trend gate as flaky rather than as a green run.
Parity rule — an adapter change is not “done” on SQLite alone. Run
bun run test:parity (or test:parity:docker to boot the missing engines);
--require-all turns skipped engines into a failure.
Migration checklist (per admin route)
Use when adding or finishing a domain:
- Shared
fetchApiCSRF for mutations (global) - Extract pure utils + unit tests (reference domains)
- Extract
*-api.tsusingfetchApi(webhooks, automations, trash) - Page.server admin/permission unit tests (config admin surfaces)
- Testing seed + delete helpers where E2E needs fixtures
- Golden E2E journeys; soft-skips removed on control-map paths
- Control maps + test-status + this ADR updated
- Testing API fail-closed gate hardened + unit proof
Still incremental (not blockers for this ADR):
- Appearance / themes full
*-apimatrix + theme create→list→delete golden - Workflows builder API + page + golden E2E (
seed-workflow,/api/workflows) - Trash restore golden (
seed-trash/purge-trash) - Extensions widgets-api (list/status/uninstall via
fetchApi) - Unified Data Hub: always-on tests hard-assert; Postgres suite uses
handleOptionalInfraUnavailable(REQUIRE_OPTIONAL_INFRA=true→ hard-fail) - Residual soft-skips: dashboard reorder, collection entry status, media HTML5 drop → hard-fail
- Integration coverage: all 9 E2E-only namespaces now HTTP-covered (P0: 26/26 files)
- Integration runner: sequential isolation + auto-retry + stderr capture
- CI: centralized secrets (fork-safe) + server process traps
- Multi-tenancy E2E:
isolation.spec.tsgated behindMULTI_TENANT=true - OAuth E2E: separate Playwright project, gated behind
OAUTH_ENABLED=true - Builder E2E: consolidated 9 files → 1 (shell + golden)
- E2E reliability: 6
page.reload()→.toPass()polling - Multi-DB tracking:
db-summaryCI job with per-adapter dashboard - A++ identity: single harness credentials across unit/integration/E2E
- A++ integration crash:
serverCrasheddetection + full process restart - A++ E2E default:
test:e2e= CI-parity preview runner (not Vite) - A++ unit coverage floors on hooks/auth/security paths
- A++ push SQLite smoke when databases/plugins/hooks change
- Residual config/user goldens (2026-07-19): redirectsMV primary store; profile email-fallback update; automations card link open; workflow seed
name; access tokens panel effect loop fixed
Rollout status (complete as of 2026-07-18)
| Domain | Product / API | Unit | Golden / outcome E2E | Soft-skip free |
|---|---|---|---|---|
| Webhooks | webhooks-api + utils |
✅ | create→list→reload→delete | ✅ |
| Automations | automations-api |
✅ | builder create→list→edit→delete | ✅ |
| Redirects | server helpers + utils | ✅ | create→search→delete | ✅ |
| Trash | trash-api |
✅ | golden restore (seed-trash) |
✅ |
| Access management | save via fetchApi + website-tokens-api |
✅ gate + tokens | permissions toggle+save | ✅ |
| Appearance / themes | appearance-api |
✅ | create→list→delete + prefs shell | ✅ |
| Workflows | workflows-api + /api/workflows |
✅ | seed→load/save shell golden | ✅ |
| System settings | remotes + utils | ✅ | edit→save→reload (+ shell) | ✅ |
| Sync | plan/apply via fetchApi |
✅ gate | shell tabs/status | ✅ |
| Queue / Monitor | admin gates + testids | ✅ | operations shell + filters | ✅ |
| Extensions | plugins-api + widgets-api | ✅ | tabs smoke | ✅ |
| Migration wizard | enable-plugin + hard-fail ready | — | dry-run / import (plugin) | ✅ no empty skip |
| Collection builder structure | — | — | category persist (Quick Start if empty) | ✅ |
| Dashboard reorder | — | — | ≥2 widgets hard-fail | ✅ |
| Unified Data Hub | plugin + external DB | — | optional infra / 503 | external only |
| Multi-tenancy | isolation.spec.ts |
— | browser isolation (gated) | ✅ env-gated |
Consequences
Positive
- UI refactors less often break unit proofs.
- Empty CI installs cannot soft-skip mutation coverage.
- CSRF cannot be forgotten on a single page’s
fetch. - E2E stays smaller and outcome-focused.
- Seeds cannot run under
NODE_ENV=productioneven with flag + secret.
Trade-offs
- New domains still need a seed action before deep E2E.
- Remaining pages (appearance, workflows) migrate incrementally using the checklist.
12. Composite Editorial Journeys & Zero-Error Trap (2026-10-23)
To ensure that the CMS functions reliably in the browser for end users (rather than merely passing synthetic 1-field API tests), four real-browser workflow contracts are enforced:
- Kitchen Sink Master Collection (
tests/e2e/routes/collection-builder/kitchen-sink.spec.ts): Proves that multiple real-world widgets (Input,Number,Checkbox,Email) validate, save, hydrate on reload, update, and delete in one composite form. - Editor Boundary (
tests/e2e/routes/admin/editor-boundary.spec.ts): Proves non-superadmin users can perform permitted editorial actions while restricted admin configurations fail closed cleanly without client crashes. - Client Widget DOM Testing (
tests/unit/components/dom/widget-inputs-interaction.test.ts): Mounts Svelte 5 widget components in the client DOM to test two-way value bindings and validation store synchronization. - Zero-Error Trap (
tests/e2e/helpers/zero-error-guard.ts): Attaches browser listeners so any unhandledpageerror, unhandled rejection, non-allowlisted console error, or same-origin HTTP $\ge 500$ response fails the journey immediately. - Real-Browser Vitality (
tests/e2e/browser-vitality.spec.ts): Records Navigation Timing (domInteractive), input responsiveness, and SPA heap memory stability in Chromium viabun run benchmark:browser.
Related
- Testing Strategy
- Three-Layer Completeness
- Test Status dashboard
- Webhooks control map (canonical reference)
- Automations · Redirects · Sync/Trash
- AGENTS.md — E2E policy + Testing API security sections