Data Operations Architecture
The unified data operations framework β configuration promotion, content packages, migrations, importers, backups, and content sync with a shared plan-first safety lifecycle.
On this page
The Data Operations framework is a unified, safety-first system for moving and transforming data across SveltyCMS environments. It consolidates six operation domains under a shared lifecycle with consistent audit logging, background job support, and identity matching.
π Six-Domain Separation
Data Operations span six distinct domains, each with its own namespace, handler, and permission model:
| # | Domain | Namespace | Handler | Purpose |
|---|---|---|---|---|
| 1 | Configuration Promotion | /api/config/* |
config.ts |
Move site structure (collections, settings, roles) between envs |
| 2 | Content Packages | /api/content-export/* /api/content-import/* |
content-transfer.ts |
Portable editorial record transfers with NDJSON streaming |
| 3 | Data Migrations | /api/migrations/* |
migrations.ts |
Transform stored data or schema state in place |
| 4 | External Importers | /api/importers/* |
importers.ts |
Import from WordPress, Drupal, CSV, JSON, and other sources |
| 5 | Backups | /api/backups/* |
backups.ts |
Disaster recovery archives with encrypted manifests and checksums |
| 6 | Content Sync | /api/content-sync/* |
content-sync.ts |
Explicit push/pull between configured environments |
π Shared Operation Lifecycle
All six domains follow a common validate β plan β apply β verify lifecycle. Each phase gates the next, ensuring no destructive operation proceeds without explicit confirmation:
Phase Details
| Phase | Responsibility | Failure Behavior |
|---|---|---|
| Validate | Check inputs, permissions, resource existence, schema compatibility | Abort with specific error |
| Snapshot | Capture pre-operation state for rollback reference | Warn, continue (best-effort) |
| Plan | Build an explicit operation list with risk assessment and blocked-reason enumeration | Return plan with warnings/blocks, no mutations |
| Confirm | Require explicit user consent for destructive plans (mirror, replace, deletes) |
Block until confirmed |
| Lock | Acquire a domain-specific operation lock to prevent concurrent mutations | Wait with timeout, fail if contention detected |
| Apply | Execute queued operations in dependency order, with per-operation error isolation | Roll back completed operations on failure |
| Audit | Write structured audit log entries with operation metadata, plan ID, and outcome | Fire-and-forget (non-blocking) |
| Verify | Run postcondition checks: resource counts, checksums, referential integrity | Report discrepancies, flag for manual review |
| Release | Release operation lock, clear temporary state, broadcast completion event | Always runs (even on prior failure) |
π‘οΈ Safety Modes
All mutation-capable domains support four safety modes that control merge behavior:
| Mode | Creates | Updates | Deletes | Rollback | Use Case |
|---|---|---|---|---|---|
add |
β | β | β | Trivial | Bootstrap: only add missing resources, never modify |
merge |
β | β | β | Reverse | Default. Safe promotion: add new, update existing |
mirror |
β | β | β | Snapshot | Full alignment: make target exactly match source |
replace |
β | β | β | Snapshot | Fresh install: drop all existing, import from source |
Destructive modes (mirror, replace) require explicit confirmation. Plans with these
modes carry risk: "destructive" and requiresConfirmation: true. The confirm phase will
block until the user explicitly acknowledges the plan. Snapshot-before-apply provides a
rollback path, but verify the plan output carefully before confirming.
Mode Selection by Domain
| Domain | Available Modes | Default |
|---|---|---|
| Configuration Promotion | add, merge, mirror, replace |
merge |
| Content Packages (import) | add, merge, mirror |
merge |
| Data Migrations | merge only |
merge |
| External Importers | add, merge |
add |
| Content Sync | merge, mirror |
merge |
π Identity Matching Priority
When moving resources between environments, the framework must match source entities to target entities. Identity resolution follows this priority chain:
| Priority | Match Key | Example | Used By |
|---|---|---|---|
| 1 | syncId |
Explicit UUID set during initial export | Configuration, Content |
| 2 | External ID | Source-system identifier (e.g., WordPress post ID) | Importers |
| 3 | Natural Key | Domain-unique composite (e.g., collection.name) |
Configuration |
| 4 | Manual Mapping | User-provided map in the plan confirmation step | All domains |
When no match is found, the entity is treated as new. When multiple candidates match, the operation is flagged for manual resolution and included in the planβs warnings.
π Background Job Integration
Large operations (1,000+ entities) are dispatched as background jobs via the Adaptive Job Scheduler:
Job thresholds:
| Domain | Inline Threshold | Background Above |
|---|---|---|
| Configuration Promotion | 100 entities | β |
| Content Packages (import) | 500 entities | β |
| Data Migrations | Always inline | β |
| External Importers | 100 entities | β |
| Backups (restore) | Always inline | β |
| Content Sync | 500 entities | β |
π Audit & Observability
Every data operation writes structured audit log entries with crypto-chained integrity (SHA-256 hash chain).
Audit Entry Schema
interface DataOperationAuditEntry {
timestamp: ISODateString;
domain: "config" | "content-package" | "migration" | "importer" | "backup" | "content-sync";
operation: "export" | "import" | "plan" | "apply" | "restore";
planId: string;
mode: "add" | "merge" | "mirror" | "replace";
actor: { userId: string; tenantId: string };
outcome: "success" | "partial" | "failure" | "rollback";
summary: {
created: number;
updated: number;
deleted: number;
skipped: number;
failed: number;
};
durationMs: number;
previousHash: string; // SHA-256 chain link
currentHash: string; // SHA-256 of this entry
}
Observability Requirements
| Signal | Implementation |
|---|---|
| Operation logs | Structured audit entries per domain |
| Plan diffs | Resource-level change enumeration |
| Duration metrics | Per-phase timing (validate, plan, apply) |
| Error counters | Per-domain failure tracking |
| Chain verification | GET /api/config/history with hash check |
| Completion events | SSE broadcast on operation finish |
π Service / Handler Patterns (Content Lists)
Data operations that mutate entries (content packages, importers, content sync, migrations) must stay consistent with how the CMS reads list data:
| Layer | Responsibility |
|---|---|
Handler (content-transfer.ts, importers.ts, β¦) |
Validate β plan β apply; permission-gated namespaces |
Service (CollectionService, domain services) |
Shared load/mutate logic; no raw DB outside adapters |
| Cache | After apply, cacheService.invalidateCollection(collectionId) clears all collection:{id}:query:* list variants (filtered pages, search, sort) |
| Admin UI | entry-list + createSmartFilter re-fetch via URL params β SSR loader β SWR |
Do not invalidate only a single page key. Filtered lists use query:{hash} fragments;
prefix invalidation is required so editors never see stale rows after import/sync.
See Cache System β Collection List Queries.
Related: Collection Filtering Platform Β· Content query params Β· entry-list
πΊοΈ Route Architecture
The Data Operations framework is dispatched through the Unified Gatekeeper (src/routes/api/[...path]/+server.ts):
api/
βββ config/
β βββ resources GET β List syncable resource types
β βββ status GET β Drift summary
β βββ export POST β DB β Filesystem
β βββ plan POST β Dry-run operation list
β βββ apply POST β Execute confirmed plan
β βββ history GET β Past operations
βββ config_sync GET β Deprecated alias β /api/config/status
βββ config-sync GET β Deprecated alias β /api/config/status
βββ content-export/
β βββ validate POST β Validate export selection
β βββ plan POST β Preview export contents
β βββ run POST β Execute export
β βββ download GET β Download exported package
β βββ jobs GET β Export job status
βββ content-import/
β βββ validate POST β Validate import package
β βββ plan POST β Preview import changes
β βββ apply POST β Execute import
β βββ jobs GET β Import job status
βββ migrations/
β βββ status GET β Pending migration list
β βββ history GET β Past migration runs
β βββ plan POST β Dry-run migration
β βββ apply POST β Execute migration
β βββ verify POST β Post-migration checks
βββ importers/
β βββ sources GET β Available import sources
β βββ validate POST β Validate import file
β βββ preview POST β Preview field mapping
β βββ run POST β Execute import
β βββ jobs GET β Import job status
βββ backups/
β βββ (list) GET β List backups
β βββ create POST β Create backup
β βββ validate POST β Validate backup integrity
β βββ restore-plan POST β Preview restore
β βββ restore POST β Execute restore
β βββ jobs GET β Backup job status
βββ content-sync/
βββ channels GET β List configured sync channels
βββ plan POST β Preview sync operations
βββ push POST β Push to target environment
βββ pull POST β Pull from source environment
βββ jobs GET β Sync job status
Permission Model
| Namespace | GET Permission | POST Permission |
|---|---|---|
config |
config:read |
config:write |
content-export |
content:read |
content:export |
content-import |
content:read |
content:import |
migrations |
migration:read |
migration:apply |
importers |
content:read |
content:import |
backups |
backup:read |
backup:create |
content-sync |
content:read |
content:sync |
All unmapped namespaces fail-closed with 403 Forbidden. Admin users bypass RBAC via the dispatcher fast-path.
π¦ File Format Plans
Configuration Sync Manifest
Deterministic JSON files for version-controlled site structure in /config/sync/:
/config/sync/
βββ config.manifest.json
βββ collections/
β βββ blog-posts.f47ac10b.json
βββ system/
β βββ widget.state.json
β βββ theme.state.json
β βββ webhooks.state.json
β βββ automations.state.json
βββ roles/
βββ editor.a1b2c3d4.json
Each file includes a deterministic checksum and UUID for cross-environment identity matching.
Content Package (.svelty-content-package)
Portable content bundles using NDJSON streaming:
package.svelty-content-package
βββ manifest.json β Metadata, source info, creation timestamp
βββ schema/ β Collection schemas referenced by content
β βββ {collection-name}.json
βββ data/
β βββ entries-0001.ndjson
β βββ entries-0002.ndjson
β βββ ...
βββ media/
βββ media-manifest.json
βββ files/ β Optional, for small transfers
Backup Archive (.svelty-backup)
Disaster-recovery archives with encryption and integrity verification:
backup-2026-07-10.svelty-backup
βββ backup.manifest.json β Timestamp, checksums, adapter type, version
βββ content/
β βββ {collection}/
β βββ entries.ndjson
βββ config/
β βββ snapshot.json
βββ media/
β βββ media.manifest.json
βββ signatures/
βββ sha256.checksums
ποΈ Current Implementation Status
| Domain | Handler | Core Logic | Status |
|---|---|---|---|
| Configuration Promotion | β | Partial (status, resources, plan, apply) | π‘ In progress |
| Content Packages | β | Stub (501) | π΄ Planned |
| Data Migrations | β | Stub (501) | π΄ Planned |
| External Importers | β | Hybrid (sources implemented) | π‘ Partial |
| Backups | β | Stub (501) | π΄ Planned |
| Content Sync | β | Stub (501) | π΄ Planned |
Legend: π’ Complete Β· π‘ In progress / Partial Β· π΄ Planned