SveltyCMS separates user live collection data from benchmark and test fixtures at three layers: physical paths on disk, compile/runtime scanning, and GraphQL schema registration. This prevents benchmark runs from polluting config/collections/ or bloating the GraphQL schema with 150+ mock types.

Path Contract

Purpose	TypeScript source	Compiled output
User live data	config/collections/*.ts (root only)	.compiledCollections/*.js (root only)
Benchmark / test fixtures	config/collections/test/<workspace>/	.compiledCollections/test/<workspace>/

Workspaces

Benchmark modules use named workspaces under test/:

Workspace	Test file	Purpose
scan	tests/benchmarks/content-scan.test.ts	Self-healing scanner (150+ mock files)
stress	tests/benchmarks/content-scale-stress.test.ts	1,000-file scale stress
incremental	tests/benchmarks/content-incremental-reload.test.ts	Surgical vs full reload
integration	tests/integration/harness.ts + bun test tests/integration/	CI / local integration fixture

API helpers

import {
  getBenchmarkWorkspace,
  prepareBenchmarkCompiledWorkspace,
  cleanupBenchmarkCompiledWorkspace,
  cleanupAllBenchmarkWorkspaces,
} from "@utils/benchmark-paths";

tests/benchmarks/modules/benchmark-utils.ts re-exports these helpers and runs cleanupAllBenchmarkWorkspaces() in afterAll (except when BENCHMARK_MATRIX=1, where the matrix orchestrator owns cleanup).

Three-Layer Isolation

1. Physical (disk)

• Bootstrap (engine.server.ts): bench_*, mock_*, and test_* slugs regenerate under config/collections/test/, not the root.
• Benchmark filesystem tests write only under .compiledCollections/test/<workspace>/.
• Setup wizard (writePresetCollectionFiles with replaceAll: true): purges stale/benchmark debris before writing blog presets (posts.ts, authors.ts, categories.ts).

2. Compile & runtime scan

• compile.ts: Outside benchmark runtime (BENCHMARK_MODE, TEST_MODE, etc.), skips test/ sources and isBenchmarkArtifact() files — dev builds do not compile fixtures into the live tree.
• scanCompiledCollections: Skips benchmark artifacts when !isBenchmarkRuntime().
• Dev reconciler (scan-files.server.ts): Same skip outside benchmark mode.

3. GraphQL logical filter

• isMockScanCollection(): Always excluded from GraphQL schema — even when BENCHMARK=true. Prevents matrix HTTP 500 from registering 150+ mock_collection_* types.
• bench_* / test_*: Excluded in normal operation; included in BENCHMARK=true for relational audits (BenchmarkStable, benchmark_authors).
• allCollections query: Applies the same filter; returns full set only in benchmark mode (minus mock scan debris).

Benchmark Data: API vs Filesystem

Benchmark type	Collection source	Touches user paths?
Relational, REST, GraphQL API	setup-benchmarks.ts → POST /api/testing (LocalCMS)	No — DB + in-memory store only
Content scan / stress / incremental	Isolated .compiledCollections/test/ fixtures	No — workspace only
Matrix server boot	Reads user root if present; purges leaks first	Purge only — never writes user files

Matrix Lifecycle

1. Pre-audit (index.ts): purgeBenchmarkCollectionArtifacts() before any server starts.
2. Per-DB audit (runner.ts): Purge again before startServer().
3. Child tests: API_BASE_URL points at matrix server; setupBenchmarkServer() does not spawn a second server.
4. Exit / SIGINT (ConfigSafeguard.restore()): Removes config/private.test.ts and purges benchmark artifacts.

# Single benchmark via matrix (isolated temp DB + purge)
bun run scripts/benchmark-matrix/index.ts --db=sqlite --only=relational --no-build

Purge & Detection

purgeBenchmarkCollectionArtifacts() in preset-collections.server.ts:

1. Wipes entire config/collections/test/ and .compiledCollections/test/
2. Removes legacy root dirs (nested/, batch_bench/)
3. Deletes benchmark artifacts at root (bench_*, Mock Collection *, BenchmarkStable.ts, etc.)
4. Optional wipeAllSource: true on setup completion — full root wipe before preset install

isBenchmarkArtifact(fileName) — filesystem debris patterns.
isMockScanCollection(id, name) — GraphQL exclusion for scan/stress mocks.

Environment Flags (unified via isBenchmarkRuntime())

GraphQL registerCollections(), content scanners, and compile.ts all gate on isBenchmarkRuntime() — not a single env var. Any of these enables benchmark mode:

Variable	Typical source
BENCHMARK=true	benchmark-utils.ts, matrix buildServerEnv
BENCHMARK_MODE=1 or true	Matrix orchestrator, standalone benchmarks
BENCHMARK_STABLE=true	Matrix + relational audits
SVELTY_BENCHMARK_SUITE=true	Matrix index.ts
TEST_MODE=true	Integration runner, matrix child env

BENCHMARK_MATRIX=1 — child defers workspace cleanup to matrix orchestrator.

Standalone bun test tests/benchmarks/* sets all of the above in benchmark-utils.ts so GraphQL and scanners stay aligned.

One-Time Cleanup (polluted dev tree)

If an older benchmark run left Mock Collection *.js or bench_* files at the root of config/collections/ or .compiledCollections/:

# Purge filesystem + SQLite debris; preserves user files like posts.ts / authors.ts
bun -e "import { purgeBenchmarkCollectionArtifacts } from './src/routes/setup/preset-collections.server.ts'; console.log('Removed:', await purgeBenchmarkCollectionArtifacts())"

# Inspect SQLite pollution (collection_* tables + content_nodes)
bun scripts/inspect-mock-collections.ts

purgeBenchmarkCollectionArtifacts() also runs purgeBenchmarkDatabaseArtifacts() — drops stale collection_* tables and content_nodes rows (mock scan debris always; benchmark presets only on user/healing DBs, not benchmark_shared.sqlite).

Or complete setup with the blog preset (replaceAll: true) — that wipes stale root files before writing the three preset collections.

After cleanup, regenerate types if needed: bun x svelte-kit sync.

Resolved gaps

Item	Status
compile.ts benchmark filter	Done — skips test/ + isBenchmarkArtifact when !isBenchmarkRuntime()
mock_collection_* detection	Done — isBenchmarkArtifact + isMockScanCollection
ConfigSafeguard ad-hoc matching	Done — only purgeBenchmarkCollectionArtifacts()
Env var split GraphQL vs scanners	Done — isBenchmarkRuntime() everywhere
Docs outdated claims	Updated — this file + docs/tests/index.mdx

Troubleshooting

Symptom	Likely cause	Fix
158+ collections in GraphQL	Mock files at .compiledCollections/ root	purgeBenchmarkCollectionArtifacts() or complete setup with blog preset
Matrix relational HTTP 500	Mock scan types registered in GraphQL	Rebuild + run matrix (pre-purge + isMockScanCollection filter)
Mock Collection *.js in root	Old content-scan run before isolation	Delete root mocks; future runs use test/scan/
Unknown collection redirects	Redirect-manager plugin (DB-only)	Expected — not a wizard preset file

Related

• Testing Index
• CI Performance Metrics
• Black-Box Testing
• Performance Benchmarks