4SO LabLifecycle after creation
Scale, backup, restore, upgrade and recovery are first-class actions.
DATABASE LIFECYCLE CONTROL PLANE
DBaaS EnterpriseEngine-aware database operations, not just provisioning
A lifecycle control plane for ten database engines with 1/2/3-node topologies, engine-native HA semantics, backup/restore, upgrade and runtime verification.
MySQLPostgreSQLRedis OSSValkeyMariaDBMongoDBClickHouseOpenSearchScyllaDBQdrant
DBaaS Enterprise components: one control plane and ten database engines
- 1Control PlaneAdmission, RBAC and durable operations for every engine; the Manager never joins the tenant data path.
- 2Durable operation stateEvery accepted operation is recorded in PostgreSQL before execution and handed to workers through a transactional outbox.
- 3Workers & automationWorkers hold an execution lease and source-manifest fence, then run Ansible on the database nodes.
- 4MySQLInnoDB ReplicaSet at 1–2 nodes, InnoDB Cluster / Group Replication at 3; MySQL Router runs on the DB nodes.
- 5PostgreSQLPatroni with streaming replication and etcd; PgBouncer and HAProxy colocated on the DB nodes.
- 6MariaDBMulti-primary Galera with HAProxy; three members provide quorum HA.
- 7Redis OSSReplication, Sentinel and HAProxy; two-node automatic failover is deliberately disabled.
- 8ValkeyThe same replication + Sentinel + HAProxy model behind a stable tenant FQDN.
- 9MongoDBDocument database; topology and HA follow its native semantics under the same control plane.
- 10ClickHouseColumnar analytics database (OLAP); topology and HA follow its native semantics under the same control plane.
- 11OpenSearchSearch and log analytics; topology and HA follow its native semantics under the same control plane.
- 12ScyllaDBWide-column database; topology and HA follow its native semantics under the same control plane.
- 13QdrantVector database; topology and HA follow its native semantics under the same control plane.
01PROBLEM & OUTCOME
Product problem
Provisioning is the easy part; operating the database is the product.
Replication, quorum, backup, restore, upgrade and failover differ by engine. DBaaS preserves those differences while presenting one durable operational surface.
No fake abstraction
InnoDB, Patroni, Sentinel and Galera keep their native quorum/election behavior.
Designed recovery
Promotion, fencing and rejoin rules exist before an incident.
Observed success
Membership, endpoint and client I/O must confirm the result.
02CAPABILITY SNAPSHOT
Capabilities
One operational surface, engine-specific mechanics.
The operator sees a coherent lifecycle while each engine keeps the coordination model it actually requires.
Create service
Engine, version, topology and nodes with preflight.
1 → 2 → 3
Add members without treating two nodes as automatic HA.
Data protection and recovery
Engine-native backup, integrity evidence and post-restore verification.
Controlled change
Supported version edge, backup gate and runtime read-back.
Promotion / routing
Engine-aware promotion or Galera routing reconciliation.
Stable client identity
Tenant FQDN remains stable across topology changes.
Real signals
Host/database telemetry and service-level probes.
TLS, RBAC and audit
Access and every operation stay controlled and traceable in the control plane.
03TOPOLOGY / DEPLOYMENT MODEL
Deployment model
Topology follows the engine, not a generic HA checkbox.
The relational and key-value profiles (MySQL, PostgreSQL, Redis, Valkey and MariaDB) start at one node, grow through two nodes without automatic failover, and reach automatic HA at three members using the engine's own coordination model.
| Profile | Engines | Failure behavior | Native mechanism |
|---|---|---|---|
| 1 node | MySQL · PostgreSQL · Redis · Valkey · MariaDB | Non-HA | Single member + stable endpoint |
| 2 nodes | MySQL · PostgreSQL · Redis · Valkey · MariaDB | Automatic failover disabled | Primary/Mirror or two-member Galera without majority |
| 3 nodes | MySQL · PostgreSQL · Redis · Valkey · MariaDB | Automatic HA | InnoDB Cluster · Patroni/etcd · Sentinel · Galera quorum |
04ARCHITECTURE
Architecture
Control Plane stays out of the tenant data path.
Intent and durable job state live in the manager; database runtime, membership and local routing stay on tenant database nodes.
ENTRYPanel / Product APITenant, project, engine, topology, permission and confirmation.
DURABLE STATEPostgreSQL operationsOperation, lock, correlation and audit persist before mutation.
EXECUTIONOutbox → RabbitMQ/Celery → AnsibleSerialized engine-specific execution on tenant DB nodes.
RUNTIMEDatabase nodes + local router/proxyThe serving system owns replication and routing truth.
| Boundary | Mechanism | Operational contract |
|---|---|---|
| Manager | Control plane only | Never a DB member, witness, quorum voter or failover target. |
| Stable endpoint | Tenant FQDN | Client identity survives topology changes. |
| Routing | Local Router/HAProxy on DB nodes | A client reaching any service node can still reach the correct role. |
| Coordination | Engine-native quorum/election | No external witness is added to invent HA. |
| Stage | Authority / Component | Technical contract |
|---|---|---|
| Admission | Panel / Product API | Engine, topology, permission and confirmation are checked; the request is only admitted. |
| Durable state | PostgreSQL | Operation and Job persist before mutation; only one active mutation is allowed per tenant; a second request is refused, not queued. |
| Dispatch | Transactional Outbox → RabbitMQ → Celery Worker | Work publication is coupled to transactional state and the worker receives serialized execution. |
| Execution fence | Execution lease + signed source manifest | The runtime tree must match expected release identity; drift blocks playbook execution. |
| Engine execution | Project-local Ansible | The native engine playbook runs only on tenant DB nodes; the Manager never enters the Data Plane. |
| Verification | Engine runtime + stable endpoint | Membership/quorum, replication, DNS/TLS and client read/write are read back from the real runtime. |
| Evidence | Job-scoped evidence | Result and correlation stay attached to the same Operation; HTTP acceptance never becomes completion by itself. |
05LIFECYCLE
Lifecycle
The lifecycle is one durable loop.
Every mutation is admitted, persisted, executed, read back and either verified or moved into an explicit recovery state.
01Createengine · version · topology
02Scaleprepare · join · sync
03Backupartifact · integrity
04Restorepreflight · restore · verify
05Upgradeedge · stage · read-back
06Failoverfence · promote / reconcile
07Rejoinmembership convergence
08Removesafe member removal
06ENGINE PROFILES
Engine profiles
Ten engines, each with its own native semantics, under one Control Plane.
The control plane is shared, but replication, election/quorum, local routing, backup and failover/recovery follow each engine's native architecture.
| Engine | Native topology / routing | 1 node | 2 nodes | 3 nodes | Data protection |
|---|---|---|---|---|---|
| MySQL | InnoDB ReplicaSet (1–2) · InnoDB Cluster / Group Replication (3) · MySQL Router on DB nodes | Non-HA | Manual promotion | Automatic HA | XtraBackup |
| PostgreSQL | Patroni + Streaming Replication · etcd + PgBouncer + HAProxy on DB nodes | Non-HA | Manual promotion | Patroni automatic HA | pg_basebackup |
| Redis OSS | Replication + Sentinel + HAProxy; Sentinel and proxy stay on DB nodes | Non-HA | Manual promotion | Sentinel automatic HA | RDB / AOF |
| Valkey | Replication + Sentinel + HAProxy; a separate engine, not a Redis alias | Non-HA | Manual promotion | Sentinel automatic HA | RDB / AOF |
| MariaDB | Galera (wsrep) + HAProxy; multi-primary cluster with product-routed write target | Single-member Galera | No automatic failover | Galera quorum HA | MariaBackup / Logical Dump |
| MongoDB | Document | Topology, HA and data protection follow the engine's own native semantics, under the same control plane and lifecycle. | |||
| ClickHouse | Columnar analytics (OLAP) | Topology, HA and data protection follow the engine's own native semantics, under the same control plane and lifecycle. | |||
| OpenSearch | Search & log analytics | Topology, HA and data protection follow the engine's own native semantics, under the same control plane and lifecycle. | |||
| ScyllaDB | Wide-column | Topology, HA and data protection follow the engine's own native semantics, under the same control plane and lifecycle. | |||
| Qdrant | Vector | Topology, HA and data protection follow the engine's own native semantics, under the same control plane and lifecycle. | |||
07ACTION & WORKFLOW CONTRACT
State and flows
Accepted is not succeeded.
One active mutation per tenant prevents overlapping topology changes; runtime evidence closes the job.
01AdmissionENGINE · TOPOLOGY · RBAC
02PersistDURABLE OPERATION
03QueueQUEUED
04ExecuteRUNNING
05Read-backVERIFYING
06TerminalSUCCEEDED / FAILED
FAILOVER PATH
Failover with fencing and quorum
The stable endpoint moves only after the new serving role is proven.
STABLE ENDPOINT
↓
PRIMARY
⇄
REPLICA
↓
QUORUM
+ FENCE
+ FENCE
Failure → membership read-back → fence → promotion → route update → client verify
RECOVERY FORK
Restore: promote or abort
A recovery point must verify before it can enter the serving path.
BACKUP + CHECKSUM
↓
RESTORE
VALID?
VALID?
↓
PROMOTE
RESTORED
RESTORED
↙ ↘
ABORT
ISOLATE
ISOLATE
Compatibility · ownership · disk · engine health · client read/write
RECONCILIATION LOOP
Rejoin / replace loop
Observed membership decides whether a node can rejoin or must be replaced.
DESIRED
MEMBERS
MEMBERS
→
RECONCILE
↓
READ
BACK
BACK
↑
OBSERVED
MEMBERS
MEMBERS
←
DRIFT /
FAILED NODE
FAILED NODE
Rejoin eligibility → re-seed if needed → membership sync → endpoint verify
Provision Service
From intent to service-level read-back.
01Engine / Version02Topology / Nodes03Preflight04Persist Job05Engine Automation06Runtime Verify
Failure / RecoveryMissing membership, endpoint or client I/O evidence keeps the job non-successful.
Scale 1→2→3
Grow without inventing two-node HA.
01Capacity Gate02Prepare Node03Join / Replicate04Sync05Quorum Check06Endpoint Verify
Failure / RecoveryPartial membership is reconciled before another mutation.
Backup → Restore
Protection is only useful when recovery verifies.
01Backup Admit02Native Backup03Integrity04Restore Preflight05Restore06Client I/O
Failure / RecoveryIncomplete or corrupt backup is never accepted as a recovery point.
Upgrade
Version change is staged and read back.
01Upgrade Edge02Backup Gate03Health Gate04Stage05Version Read-back06Post-check
Failure / RecoveryUnsupported edge or degraded health blocks continuation.
Failover / Recovery
Fencing and quorum precede promotion.
01Detect02Read Membership03Fence / Quorum04Promote / Reconcile05Route Update06Client Verify
Failure / RecoveryMinority partitions are not force-promoted.
ADDITIONAL OPERATIONAL FLOWS
Additional operational flows
Two Day-2 paths that matter as much as initial provisioning.
Rejoin Member
Return a recovered member without corrupting current membership.
01Detect Drift02Fence / Classify03Rejoin Eligibility04Re-seed if Needed05Join Membership06Catch-up07Endpoint Verify
Failure / RecoveryIf identity or replication position is unsafe, the node moves to replace instead of force-join.
Remove / Replace Node
Remove or replace hardware while protecting quorum and retained members.
01Change Intent02Quorum Gate03Drain / Fence04Remove Member05Prepare Replacement06Join / Sync07Runtime Verify
Failure / RecoveryRetained members are not sanitized and the old node identity is released only after confirmed removal.
| Action | Admission / Preconditions | Execution owner | Success criterion | Failure / recovery |
|---|---|---|---|---|
| Create Service | Valid tenant/project, engine/version, nodes | DBaaS worker + engine playbook | Membership + endpoint + client I/O | Failed job retains evidence; reconcile before retry |
| Scale | Healthy current cluster + exact new node | Serialized tenant mutation | Exact member count and replication/quorum | Partial join is reconciled |
| Backup | Compatible engine and destination | Engine-native backup | Artifact + integrity metadata | Incomplete backup is invalid |
| Restore | Verified backup + compatibility + maintenance gate | Cluster-locked restore | Service + replication + endpoint + I/O | Unknown outcome requires read-back |
| Upgrade | Supported edge + backup + health | Staged engine-aware upgrade | Exact version + health + client I/O | Unsupported/downgrade edge rejected; stops and recovers at the failed stage |
| Failover | Eligible topology + fencing/quorum | Promotion or Galera routing reconcile | New serving role + stable endpoint | Rejoin/reconcile remains separate |
08OPERATIONS CATALOG
Operator surface
DB operations become explicit product actions.
Operators work with lifecycle intent rather than memorized shell sequences.
Create database service
Engine-aware deployment with preflight and runtime verification.
Add member
Prepare, join, sync and verify topology.
Create recovery artifact
Scheduled backup with retention and verify; native backup with integrity evidence.
Recover service
Compatibility and service-level verification.
Change engine version
Supported edge and staged runtime read-back.
Move serving role
Planned switchover and failover with quorum/fencing-aware promotion or reconciliation.
Repair membership
Return a recovered node to valid membership.
Inspect service
Topology, replication, endpoint and telemetry state.
Users & databases
Create, grant, rotate passwords and drop through the same job path.
Certificate lifecycle
Certificate status, renewal, reissue and product CA rotation.
Apply tuning profile
Resource-aware configuration profiles applied with ordered, safe restarts.
MCP
Remote MCP without a parallel authority. OAuth 2.1 Authorization Code + PKCE: the Panel is the Authorization Server and the MCP Addon the Resource Server; grants bind to the existing mcp_clients, RBAC, scope and rate limits instead of a second access model.
Database lifecycle as a product, not a runbook.
The control plane stays consistent while each engine retains its real topology and failure semantics.