Skip to content
SveltyCMS

Documentation

ADR: Testing 2026 — Pyramid, Fixtures, Soft-Skip Ban, Testids

Accepted and implemented: pyramid, seed-first E2E, soft-skip ban, shared CSRF client, fail-closed testing API, reference routes.

7/19/2026
10 min read Edit on GitHub
On this page
Field Value
Status Accepted + Implemented (core admin rollout complete 2026-07-18)
Date 2026-07-18
Reference routes Webhooks (canonical), automations, redirects, trash
Policy source AGENTS.md (E2E & Control-Map Testing Policy)

Context

Route hardening (2026-07) expanded control maps and data-testids quickly. That improved security/UX consistency but tilted the suite toward:

  • Brittle shell E2E (“every testid exists”)
  • Soft-skips when lists were empty (green CI, hidden regressions)
  • Duplicated CSRF/fetch in Svelte pages
  • Risk of treating /api/testing seeds as a production-facing backdoor

This ADR locks how admin/product routes are tested and structured so UI refactors do not erase proof, empty installs cannot skip mutating journeys, and testing infrastructure stays fail-closed outside CI.

Decision (summary)

  1. Pyramid — unit-heavy; E2E only for golden outcome journeys
  2. Soft-skip ban on control-map rows — seed via /api/testing instead
  3. Fixture-first E2E — login → seed (if needed) → journey → cleanup
  4. Testid policy — control-risk anchors only
  5. Shared mutation client — fetchApi always attaches CSRF on non-GET
  6. Testing API security — production-hard-closed; same gate as test bypass

1. Testing pyramid (target ratio)

Layer Share (guidance) Proves Tools
Unit ~70% Pure utils, validation, page.server gates, *-api.ts clients Vitest
Integration ~20% HTTP + DB, RBAC, settings/webhooks services Integration suite
E2E ~10% Golden journeys (outcome), sparse shell guards Playwright

“100% control map” means every control risk has the right layer(s)—not 100% of DOM nodes.

2. Soft-skip ban (control-map rows) — enforced

Rule Practice
Forbidden test.skip because a list is empty, no sample data, or “install-specific empty state”
Required Seed via POST /api/testing (tests/e2e/helpers/api.ts + handlers/testing.ts)
Allowed hard-fail Optional external fixtures (Docker Postgres, real IdP) after seed returns structured 503 / unavailable code
Policy text AGENTS.md

3. Fixture-first E2E — enforced

login → seed (if needed) → UI journey → assert outcome → cleanup (finally)
Prefer Avoid
Create → reload → value still present Asserting 12 decorative testids
Delete → name gone Fixed waitForTimeout(800)
URL / API / list text Soft-skip when empty

Shared helpers (implemented):

Helper Path
Seeds tests/e2e/helpers/api.ts — seedWebhook, seedAutomation, enablePlugin, …
Confirm modal tests/e2e/helpers/confirm-modal.ts
Migration wizard tests/e2e/helpers/migration-wizard.ts — ensureSmartImporterReady (hard-fail, no skip)

Testing API actions (behind fail-closed gate only):

Action Purpose
seed-webhook / delete-webhook Webhook list fixtures
seed-expired-password-reset Past-expiry password_reset token (TOKEN_EXPIRED toast)
seed-media-with-metadata Media rows with distinct metadata for ?jsonPath=
outbox-emit / outbox-process-batch Transactional outbox fixtures
outbox-tx-rollback Outbox row rolled back with failed SQL transaction
plugin-storage-* Plugin storage create/get/list/delete
seed-automation / delete-automation Automation list / search fixtures
seed-trash / purge-trash Soft-deleted entry for restore golden E2E
enable-plugin Core plugin enable for wizard E2E (pluginId sanitized)

4. Testid policy — enforced on reference specs

Use testids for Do not require testids for
Primary actions (add/save/delete) Every badge, icon, spacer
Modal root + critical fields Entire field matrix when catalog varies
Loading/empty states used as wait conditions Redundant duplicates of role+name

Prefer getByRole / accessible name when stable.

5. Shared mutation client — implemented

Piece Location Status
CSRF on mutations fetchApi → clientJsonHeaders (src/utils/api.ts) ✅
Domain API clients webhooks-api.ts, automations-api.ts, trash-api.ts ✅
Pure validation webhooks-utils.ts, redirects-utils.ts, … ✅
Admin gates +page.server.ts unit tests ✅
Access save / plugins / sync fetchApi (no manual CSRF) ✅

Canonical stack (copy for new routes):

+page.server.ts  →  admin gate (unit)
*-utils.ts       →  validate/filter (unit)
*-api.ts         →  list/create/update/delete via fetchApi (unit, mock fetch)
+page.svelte     →  UI + showConfirm only (no raw fetch / manual CSRF)
E2E              →  one golden create→list→reload→delete (or equivalent)

6. Security: testing API is not a production backdoor — hardened

Seed/reset under /api/testing exist only for CI/E2E.

Control Implementation Proof
Production hard-close NODE_ENV=production → 403 testing-api-gate.test.ts
Env allowlist TEST_MODE / PLAYWRIGHT_TEST / BENCHMARK / SVELTY_BENCHMARK_SUITE — not bare NODE_ENV=test same
Secret Timing-safe x-test-secret vs TEST_API_SECRET same + route-access-audit
Authz bypass Same production hard-gate in applyTestBypassFromRequest test-bypass.server.ts
Bundle strip Vite testBackdoorStripperPlugin + verify-prod-build-backdoor unit + CI build
Shared entry assertTestingApiAllowed used by handleTestingRoutes no weaker second gate

Agents: Do not add seed routes outside this handler. Do not weaken the gate. Do not commit production secrets. Known e2e default secret strings are rejected outside an explicit test env.

Shared infrastructure

Piece Path
CSRF mutation client src/utils/api.ts + src/utils/security/client-csrf.ts
Testing / bypass gate src/utils/test-bypass.server.ts (assertTestingApiAllowed)
Testing handler src/routes/api/[...path]/handlers/testing.ts
E2E seeds tests/e2e/helpers/api.ts
Confirm helper tests/e2e/helpers/confirm-modal.ts
Gate unit tests tests/unit/utils/testing-api-gate.test.ts
Prod strip proof tests/unit/scripts/verify-prod-build-backdoor.test.ts
Smart Test Selector scripts/test-smart.ts — unioned signals, synthetic edges, surgical diff-filtering
Canonical fixtures tests/harness — ADMIN_CREDENTIALS / USERS / TEST_PASSWORD (all layers)
Integration harness tests/integration/harness.ts — singleton preview, private.test.ts, seed isolation
E2E CI-parity runner bun run test:e2e → run-e2e-ci.ts (preview :4173, harness pre-check). Dev: run-e2e-dev.ts (strictPort :5173)

A++ suite contract (2026-07-19)

Enterprise stability rules that every layer must honor:

Rule Unit Integration E2E
One identity @tests/harness seed uses harness emails helpers/auth.ts → harness
Right altitude pure logic / gates HTTP + multi-DB golden journeys only
No soft-skip empty n/a n/a seed or hard-fail
Crash isolation n/a detect socket death → restart storageState fail-closed
Local = CI binary Vitest preview node build test:e2e = preview :4173
P0 floors test:unit:coverage thresholds multi-DB matrix in CI P0 journeys in critical-test-paths.ts
Push smoke smart unit + security auto SQLite smoke when `src/databases plugins hooks` change not on push (CI-only)

Commands (canonical):

bun run test:unit                 # Layer 1
bun run test:unit:coverage        # Layer 1 + P0 coverage floors
bun run test:integration          # Layer 2 (builds + multi-file)
bun run test:integration:smoke    # Layer 2 quick SQLite
bun run test:e2e                  # Layer 3 CI-parity (preview :4173)
bun run test:e2e:dev              # Layer 3 Vite :5173 (local only)

Migration checklist (per admin route)

Use when adding or finishing a domain:

  1. Shared fetchApi CSRF for mutations (global)
  2. Extract pure utils + unit tests (reference domains)
  3. Extract *-api.ts using fetchApi (webhooks, automations, trash)
  4. Page.server admin/permission unit tests (config admin surfaces)
  5. Testing seed + delete helpers where E2E needs fixtures
  6. Golden E2E journeys; soft-skips removed on control-map paths
  7. Control maps + test-status + this ADR updated
  8. Testing API fail-closed gate hardened + unit proof

Still incremental (not blockers for this ADR):

  • Appearance / themes full *-api matrix + theme create→list→delete golden
  • Workflows builder API + page + golden E2E (seed-workflow, /api/workflows)
  • Trash restore golden (seed-trash / purge-trash)
  • Extensions widgets-api (list/status/uninstall via fetchApi)
  • Unified Data Hub: always-on tests hard-assert; Postgres suite uses handleOptionalInfraUnavailable (REQUIRE_OPTIONAL_INFRA=true → hard-fail)
  • Residual soft-skips: dashboard reorder, collection entry status, media HTML5 drop → hard-fail
  • Integration coverage: all 9 E2E-only namespaces now HTTP-covered (P0: 26/26 files)
  • Integration runner: sequential isolation + auto-retry + stderr capture
  • CI: centralized secrets (fork-safe) + server process traps
  • Multi-tenancy E2E: isolation.spec.ts gated behind MULTI_TENANT=true
  • OAuth E2E: separate Playwright project, gated behind OAUTH_ENABLED=true
  • Builder E2E: consolidated 9 files → 1 (shell + golden)
  • E2E reliability: 6 page.reload() → .toPass() polling
  • Multi-DB tracking: db-summary CI job with per-adapter dashboard
  • A++ identity: single harness credentials across unit/integration/E2E
  • A++ integration crash: serverCrashed detection + full process restart
  • A++ E2E default: test:e2e = CI-parity preview runner (not Vite)
  • A++ unit coverage floors on hooks/auth/security paths
  • A++ push SQLite smoke when databases/plugins/hooks change
  • Residual config/user goldens (2026-07-19): redirectsMV primary store; profile email-fallback update; automations card link open; workflow seed name; access tokens panel effect loop fixed

Rollout status (complete as of 2026-07-18)

Domain Product / API Unit Golden / outcome E2E Soft-skip free
Webhooks webhooks-api + utils ✅ create→list→reload→delete ✅
Automations automations-api ✅ builder create→list→edit→delete ✅
Redirects server helpers + utils ✅ create→search→delete ✅
Trash trash-api ✅ golden restore (seed-trash) ✅
Access management save via fetchApi + website-tokens-api ✅ gate + tokens permissions toggle+save ✅
Appearance / themes appearance-api ✅ create→list→delete + prefs shell ✅
Workflows workflows-api + /api/workflows ✅ seed→load/save shell golden ✅
System settings remotes + utils ✅ edit→save→reload (+ shell) ✅
Sync plan/apply via fetchApi ✅ gate shell tabs/status ✅
Queue / Monitor admin gates + testids ✅ operations shell + filters ✅
Extensions plugins-api + widgets-api ✅ tabs smoke ✅
Migration wizard enable-plugin + hard-fail ready — dry-run / import (plugin) ✅ no empty skip
Collection builder structure — — category persist (Quick Start if empty) ✅
Dashboard reorder — — ≥2 widgets hard-fail ✅
Unified Data Hub plugin + external DB — optional infra / 503 external only
Multi-tenancy isolation.spec.ts — browser isolation (gated) ✅ env-gated

Consequences

Positive

  • UI refactors less often break unit proofs.
  • Empty CI installs cannot soft-skip mutation coverage.
  • CSRF cannot be forgotten on a single page’s fetch.
  • E2E stays smaller and outcome-focused.
  • Seeds cannot run under NODE_ENV=production even with flag + secret.

Trade-offs

  • New domains still need a seed action before deep E2E.
  • Remaining pages (appearance, workflows) migrate incrementally using the checklist.

Related

testingadre2earchitecturesecurity
Was this page helpful?