Database Resilience
End-to-end resilience guide — boot retries, adapter hooks, APIs, metrics, and operations.
End-to-end database resilience is implemented across the engine, boot path, adapter disconnect hooks, REST APIs, and dashboard widgets.
Architecture
flowchart TD
Boot[db.ts ensureFullInitialization] --> Connect[connectDatabaseWithResilience]
Connect --> Retry[DatabaseResilience.executeWithRetry]
Retry --> Hooks[bindAdapterResilienceHooks]
Hooks --> Mongo[MongoDB disconnected/error]
Hooks --> Maria[MariaDB pool error]
Hooks --> PG[PostgreSQL onclose]
Mongo --> Schedule[scheduleAdapterReconnection]
Maria --> Schedule
PG --> Schedule
Schedule --> Attempt[attemptReconnection + admin email]
API[GET /api/database/*] --> Status[getSystemStatus]
API --> Pool[getPoolDiagnostics]
Implementation Status (Complete)
| Feature | Status | Key Artifacts |
|---|---|---|
| Boot connect with retry + circuit breaker | ✅ | resilience-integration.ts → db.ts |
| Adapter disconnect hooks | ✅ | Mongo disconnected/error, MariaDB pool error, PG onclose |
| Debounced self-healing reconnect | ✅ | scheduleAdapterReconnection() → attemptReconnection() |
| Admin email on persistent failure | ✅ | notifyAdminsOfDatabaseFailure() + database-failure.svelte |
| Unified system status | ✅ | getSystemStatus() |
| Pool diagnostics API | ✅ | GET /api/database/pool-diagnostics |
| Database status API | ✅ | GET /api/database/status |
| Log export API | ✅ | GET /api/logs/download (redacted) |
| Health endpoint resilience metrics | ✅ | GET /api/system/health includes resilience block |
| Redis connection retries | ✅ | cache-service.ts initializeL2() (reconnect strategy) |
| Self-healing adapter proxy | ✅ | createSelfHealingProxy in db.ts |
| Init promise locking | ✅ | INIT_PROMISE_KEY double-check in db.ts |
| SQL migrations | ✅ | runMigrations() per Drizzle adapter |
| Network DB connection queue gate | ✅ | NetworkDbQueueGate (core/network-db-queue-gate.ts) |
| Read-replica splitting & RYW consistency | ✅ | ReplicaRouterAdapter (core/replica-router.ts) |
| Dashboard pool widget | ✅ | database-pool-diagnostics.svelte |
APIs
GET /api/database/pool-diagnostics (admin)
Returns connection pool stats, utilization, health level, and scaling recommendations.
GET /api/database/status (admin)
Full getSystemStatus() payload: health latency ping, resilience metrics, pool diagnostics, system state machine, DB type.
GET /api/system/health (public liveness)
Fast connectivity check plus lightweight resilience counters (circuitState, totalRetries, …).
GET /api/logs/download (admin)
| Param | Values | Description |
|---|---|---|
type |
latest, all, archive |
Which log files to include |
format |
text, gzip |
Response format |
since |
ISO timestamp | Filter lines after date |
level |
error, warn, … |
Filter by log level |
Sensitive fields (password, token, secret, …) are redacted in exported lines.
Developer Usage
import { getDatabaseResilience } from "@src/databases/database-resilience";
import {
connectDatabaseWithResilience,
getSystemStatus,
} from "@src/databases/resilience-integration";
// Boot (already wired in db.ts)
await connectDatabaseWithResilience(adapter, "Database Boot (postgresql)");
// Custom idempotent operation
const resilience = getDatabaseResilience();
await resilience.executeWithRetry(async () => {
/* ... */
}, "My Operation");
// Dashboard / monitoring
const status = await getSystemStatus();Connection Pool Burst Gate (NetworkDbQueueGate)
For network-attached databases (PostgreSQL and MariaDB), sudden concurrent traffic spikes can saturate physical connection pools, resulting in OS socket drops, connection resets, and worker thread starvation.
NetworkDbQueueGate (src/databases/core/network-db-queue-gate.ts) provides in-app bounded concurrency ahead of driver pools:
- Bounded In-App Concurrency: Caps simultaneous active queries dispatched to the driver pool, buffering excess requests in a low-overhead queue rather than thrashing TCP sockets.
- AsyncLocalStorage Re-entrancy: Transaction contexts bypass outer concurrency gates automatically via
AsyncLocalStoragecontext tracking, eliminating nested-query deadlocks. - Fail-Fast Timers: Requests queued longer than configured timeout thresholds abort gracefully to prevent cascading request buildup.
Circuit Breaker & Adaptive Cooldown
- Threshold: 5 consecutive failures →
OPENfor 60s (configurable) - Recovery:
HALF_OPENprobe →CLOSEDon success - Adaptive: Rapid recovery (<5s) shrinks cooldown by 20% (floor 10s)
Read-Replica Splitting & Read-Your-Writes (RYW) Consistency
ReplicaRouterAdapter (src/databases/core/replica-router.ts) scales database read workloads horizontally by distributing queries across one or more read replicas while strictly maintaining transaction semantics and data consistency.
flowchart TD
Client[Client / SDK Request] --> Router[ReplicaRouterAdapter]
Router --> Check{Operation Type?}
Check -->|Write: insert / update / delete| Primary[(Primary DB)]
Primary --> Track[Update RYW Watermark]
Check -->|Read within Transaction| Primary
Check -->|Read: find* / count / aggregate| RYW{Recent Write by Client?}
RYW -->|Yes: Within RYW Window| Primary
RYW -->|No: Standard Read| Pool[Replica Pool: Round-Robin / Least-Conn]
Pool --> Replicas[(Read Replicas)]
Replicas -.->|Replica Failure / Circuit Open| Fallback[Failover to Primary]
Fallback --> Primary
Core Guarantees:
- Zero Single-Node Overhead: When no read replicas are configured (
readReplicasis empty),createReplicaRouterAdapter()returns the primary adapter directly without any proxy layer or routing overhead. - Read-Your-Writes (RYW) Consistency: Tracks per-client write watermarks across tenants via
ReadYourWritesTracker. When a client performs a write, subsequent reads by that client within thereadYourWritesWindowMswindow (default: 2,000 ms) are pinned to the primary database, eliminating stale reads caused by replication lag. - Transaction Pinning: Reads executed inside active transactions (
transaction.execute) automatically bypass replicas and run against the primary viaAsyncLocalStoragecontext tracking (replicaStorage). - Per-Replica Circuit Breaking: Each replica node tracks consecutive errors. After 3 consecutive failures, the replica enters a 15-second cooldown period and traffic routes to other healthy replicas or the primary.
- Seamless Primary Failover: If all replicas are unavailable or a replica query errors out, the router automatically fails over to the primary database to ensure high availability.
Future Roadmap
| Item | Status |
|---|---|
| Adaptive per-retry delay from historical success rate | 🔲 Planned |
| Distributed breaker coordination across instances | 🔲 Planned |
| Percentile latency trends (p50/p95/p99) | 🔲 Planned |
Testing
bun test tests/unit/databases/database-resilience.test.ts
bun test tests/unit/databases/resilience-integration.test.ts
bun test tests/integration/databases/resilience-load.test.ts
bun test tests/unit/hooks/defense-in-depth.test.tsRelated
- Core Infrastructure
- Database Methods
- Source:
src/databases/database-resilience.ts,src/databases/resilience-integration.ts
Last Updated: 2026-06-22 — End-to-end implementation complete (boot, hooks, APIs, status).
Was this page helpful?