4SO LabServing must not depend on the Manager
The query path stays on the DNS node and its last verified state.
AUTHORITATIVE DNS / GLOBAL TRAFFIC
4SO GeoDNSDNS Control Plane with an Independent Serving Path
API-first management of DNS desired state, validation, immutable revisions, blue/green delivery, health-aware routing, DNSSEC and node lifecycle in a single process, while the query path remains independent of the Manager.
Authoritative DNSGlobal TrafficDNSSECBlue / Green
4SO GeoDNS components: a query path independent from the control path
- 1Resolver queryA query arrives from a resolver, carrying its ECS or resolver network.
- 2DNS nodednsdist → active PowerDNS blue/green runtime → local PostgreSQL state; no live Manager on the query path.
- 3Geo/ECS policyDeterministic priority over ECS, CIDR, ASN/organization and geography; arbitrary user Lua is rejected.
- 4Healthy regional endpointThe answer points to the healthy endpoint the policy selected for this query.
- 5Unhealthy endpointHealth is computed out of band; the node only reads a local versioned snapshot.
- 6ManagerFastAPI-backed Product API; PostgreSQL holds desired state, audit and the outbox.
- 7Immutable revisionsA worker and mTLS node agent stage the inactive color, read it back, then cut over atomically.
01PROBLEM & OUTCOME
Product problem
A small DNS change can have a large blast radius.
Authoritative DNS must be fast and always available, while its changes must be precise, auditable and reversible. GeoDNS separates the configuration path from the serving path.
No direct changes to the runtime
Records and policies first become a revision, then are staged and verified.
Traffic policy needs trustworthy data
Endpoint health and eligibility come from accepted observations, not from simulated browser state.
DNSSEC has its own lifecycle
Keys, signing, rollover and recovery must be as controlled as zone management.
02CAPABILITY SNAPSHOT
Capability snapshot
Zones, traffic and security in one control plane.
Beyond authoritative records, GeoDNS treats delivery, traffic management and node lifecycle as part of the same product.
Type-safe zones and records
Forward/reverse zones, typed RRsets and controlled import/export.
Immutable Revisions
Changesets, snapshot hashes and revisions for audit and rollback.
Blue/Green Delivery
Stage on the inactive color, read back state, then cut over atomically.
Global Traffic Management
Endpoints, pools, regions, health perspectives and answer policy.
DNSSEC lifecycle
Signing, DS evidence, rollover and restore/reconcile.
Secondary / Transfer
TSIG, AXFR/IXFR, catalog zones and secondary provider lifecycle.
DNS node operations
Onboard, drain, replace, remove and fleet rollout.
DNS Observability
Health, metrics, alerts, incidents and synthetic DNS probes.
4SO
The normal query path stays short: Resolver → DNS node → active runtime → local state. No regular query depends on the Manager or a cross-region database to be answered.
03DEPLOYMENT PROFILES
Deployment model
From one Manager and one DNS node to multiple regions.
The minimum product topology is simple, and HA or multi-region is not a prerequisite; nodes and regions are added incrementally.
| Profile | Control Plane | Serving Plane | Use case |
|---|---|---|---|
| Basic | 1 Manager | 1 DNS node | Simple, complete authoritative DNS setup |
| Multi-node | Central Manager | Multiple DNS nodes | Serving redundancy and controlled rollout |
| Multiple sites | Single control plane | Nodes across multiple sites | Failure-domain separation and proximity to resolvers |
| Multi-region | Central management with regional processes | Independent serving in multiple regions | Global traffic and broad resilience |
04ARCHITECTURE
Architecture
The configuration path is longer; the serving path is deliberately short.
A DNS change must pass through validation, revision and delivery. DNS answers, however, are served directly from the node's verified runtime.
CONTROLPanel / Product APIAuthentication, domains, RBAC and typed validation.
↓
AUTHORITYDesired state / Revision / JobTransactional state, immutable revisions, audit and evidence.
↓
DELIVERYWorker + Node Agent over mTLSStage, state read-back, verify and atomic cutover.
↓
SERVINGAuthoritative DNS nodednsdist and the active PowerDNS runtime with verified local state.
| Layer | Component | Responsibility | Fail-safe behavior |
|---|---|---|---|
| Product API | FastAPI + scoped RBAC | Zone/RRset, traffic policy, node lifecycle and typed validation | The browser never mutates the DNS runtime directly |
| Authority | PostgreSQL + transactional outbox | Desired state, audit, job/revision state | Mutation and event publication within one transactional boundary |
| Delivery | Worker + mTLS Node Agent | Stage inactive color, read-back, verify and atomic cutover | Failed delivery stops before serving |
| DNS Edge | dnsdist | Stable serving entry and selection of the active PowerDNS color | The query path does not depend on the Manager |
| Authoritative Runtime | PowerDNS blue/green + local PostgreSQL | Serves the last verified revision | A control-plane outage keeps DNS answering from the last healthy state |
| Security | DNSSEC · TSIG · transfer policy | Signing, rollover and zone-transfer trust | Secret material never enters Panel/Audit/Evidence |
TECH
Geo-routing & Health Contract. DNS decisions are read from the compiled policy and a local snapshot; the query path never runs a network health check at answer time.
| Domain | Input / Mechanism | Technical contract |
|---|---|---|
| Client locality | ECS or Recursive Resolver IP | Valid ECS is used when permitted; otherwise the resolver IP is the basis for selection. |
| Geo / network match | CIDR, ASN/Organization, Country/Subdivision/Continent/City | Precise overrides and geography are evaluated by numeric priority in deterministic order. |
| Native GeoIP | PowerDNS geoip backend + Country/ASN MMDB | GeoIP sits alongside gpgsql; zone data is not moved from the primary authority into a YAML/GeoIP backend. |
| Health | TCP/UDP/HTTP/HTTPS/TLS/DNS/gRPC/ICMP probes | Health is computed outside the query path, and states such as HEALTHY/DEGRADED/SUSPECT/UNHEALTHY are recorded in a versioned snapshot. |
| Policy compiler | Structured policy → trusted runtime material | Arbitrary Lua is not accepted; the compiler produces reproducible output with a hash bound to the revision. |
| Simulation | Resolver/ECS/time/health/dataset inputs | Rule match, selected pool/endpoint, fallback path, answer and TTL can be simulated before publish. |
| ECS cache isolation | Policy-aware cache keying | Answers that depend on the client network must not leak through the cache across unrelated ECS contexts. |
05LIFECYCLE
Lifecycle
Every DNS change has an observable path.
Going from edit to publish is more than a few clicks; validation, revision, stage, state read-back and recovery each have a defined state.
01EditEdit
02ValidationValidate
03Build revisionSeal
04StagingStage
05State read-backVerify
06PublishCutover
07MonitoringObserve
08Rollback / recoveryRecover
06DNS PROFILES
Capability profiles
Authoritative DNS is more than a zone editor.
Every layer, from authoritative data to global traffic and security, has its own operational profile.
Authoritative DNSZONES
Zone and RRset management with revisions.
- Forward / Reverse
- Typed RRsets
- Scheduled / templated changes
Traffic ManagementGTM
Answers based on endpoint and policy.
- Pools / Endpoints
- Geo / health policy
- Maintenance windows
DNS SecurityDNSSEC / TSIG
Trust and transfer security inside the product.
- DNSSEC signing / rollover
- TSIG
- Transfer policy
DNS node fleetDNS NODES
Lifecycle of serving nodes.
- Enroll / mTLS
- Drain / Replace
- Fleet rollout
DeliveryBLUE / GREEN
Safe, reversible publishing.
- Inactive staging
- State read-back validation
- Atomic cutover
RecoveryCONTROL PLANE
Protecting management state.
- Backup / Restore
- Release rollback
- Disaster recovery process
07ACTION & WORKFLOW CONTRACT
Job, cutover and recovery flow
A job is replayed only when its side-effect boundary is known.
Mutations are idempotent and project-scoped; node mutation, fleet rollout and certificate rotation never stay open concurrently on the same node. Worker ownership is fenced with a lease and an execution epoch.
01QueueQUEUED
02RetryRETRY
03LeaseRUNNING + EPOCH
04StageVERIFY / CUTOVER
05SuccessSUCCEEDED
06FailureFAILED / CANCELLED
07RollbackROLLED_BACK
BLUE / GREEN CUTOVER
Stage on the inactive color
DNS read-back happens before the active color is switched.
DNS QUERY ENTRY
↓
ACTIVE
BLUE
BLUE
⇄
STAGED
GREEN
GREEN
↓
DNS
VERIFY
VERIFY
revision → stage inactive → authoritative read-back → atomic cutover
A failed stage does not change active serving.
HEALTH BRANCH
Primary or fallback based on the snapshot
The query path never runs a network probe at request time.
CLIENT CONTEXT
ECS / RESOLVER
ECS / RESOLVER
↓
POLICY + HEALTH SNAPSHOT
↓
PRIMARY
HEALTHY?
HEALTHY?
PRIMARY
↙ ↘
FALLBACK
Health state is versioned; a stale snapshot can block policy.
DR LOOP
Readiness → Failover → Failback
DR actions are bound to the readiness hash and topology freeze.
PRIMARY
SITE
SITE
→
READINESS
GATE
GATE
↓
FENCE
+ HASH
+ HASH
↑
DR
SITE
SITE
←
FAILBACK
VERIFY
VERIFY
backup ready → approval → failover → serving verify → controlled failback
Stale readiness or unknown external state goes to reconcile.
Publish Zone Revision
The change is staged on the inactive color first.
01Validate02Build Revision03Compile Policy04Stage Inactive05DNS Read-back06Atomic Cutover
Failure / RecoveryA failed stage does not change the active query path.
Traffic Policy
Geo/network policy is simulated before publish.
01Policy Input02Priority Order03ECS / Resolver Context04Simulate05Compile06Publish
Failure / RecoveryArbitrary Lua and priority conflicts are rejected before serving.
Node Onboarding
A node joins the fleet only after trust and serving verification.
01Identity / mTLS02Preflight03Bootstrap04Register05Health06Rollout Eligible
Failure / RecoveryBootstrap failure does not turn into a blind retry.
DNSSEC Rotation
Key lifecycle runs with overlap and serving acknowledgement.
01Key Intent02Generate / Import03Stage04Overlap05Observe Signing06Retire Old
Failure / RecoverySecret material never enters public Panel/Audit/Evidence.
Restore / DR
Restore and site failover are bound to readiness evidence.
01Backup Ready02Preflight03Approval04Restore / Failover05Serving Verify06Commit / Failback
Failure / RecoveryUnknown external state → RECONCILE_REQUIRED, not blind replay.
ADDITIONAL OPERATIONAL FLOWS
Additional operational flows
Health and node replacement directly affect serving quality and need their own flows.
Health Snapshot → Routing
Health is collected outside the query hot path and turned into a versioned snapshot.
01Probe Policy02Schedule03Out-of-band Probe04Publish Snapshot05Compile Traffic Policy06Node Sync07DNS Verify
Failure / RecoveryStale or unknown health is never automatically assumed healthy and must not shift traffic without cause.
Drain / Replace DNS Node
The old node leaves only after the replacement is ready and serving is verified.
01Fleet Intent02Serving Capacity03Stop Eligibility04Bootstrap Replacement05Stage Active Revision06Serving Read-back07Remove Old Node
Failure / RecoveryUntil the replacement returns correct authoritative answers, removal of the old node does not reach its final step.
| Action / Flow | Admission / Preconditions | Execution / Lock | Success Criterion | Failure / Recovery |
|---|---|---|---|---|
| Publish Zone / RRset | Typed validation, revision and policy compile before delivery | Stage on inactive color → node read-back → atomic cutover | Authoritative answer/TTL and active revision confirmed from the serving path | A failed stage does not change serving; rollback to the preserved healthy color |
| Traffic Policy | Structured policy, priority and simulation without conflict | Compiler material is versioned; health snapshot is consumed outside the query path | Matched rule/pool/endpoint and synthetic DNS behavior match the revision | Arbitrary Lua is rejected; policy conflict fails before publish |
| Node Onboarding / Replace | Identity/mTLS, preflight and node mutation availability | Durable job with node lock; onboarding/day-2/fleet/cert-rotation overlap is rejected | Node observed healthy and rollout eligibility clear | Node onboarding failure does not become a generic retry; a fresh bootstrap may be required |
| DNSSEC / Certificate | Key/identity lifecycle and approval for sensitive operations | Stage/overlap/ack and owner-specific rotation workflow | Signing/serving or certificate observation matches the expected generation | Stale rotation/concurrent fleet mutation rejected; secret material removed from the public job projection |
| Control-plane Restore | Backup must be READY, repository-verified and carry snapshot/hash evidence | Preflight → approval → execution → cutover | RESTORED only after preflight binding and runtime verification | Worker interruption/unknown external state → RECONCILE_REQUIRED; blind retry prohibited |
| DR Failover / Failback | Readiness job, topology freeze, fencing, rollback and RPO acknowledgement required | The action is bound to readiness state/hash and profile revision | Active site/traffic/database state reaches the expected phase | Stale readiness or topology mismatch rejected; failback allowed only from FAILED_OVER |
FLOW
Retry classification is explicit to the operator: SAFE_TO_RETRY, RECONCILE_REQUIRED, FRESH_BOOTSTRAP_REQUIRED or FRESH_HEALTH_REQUEST_REQUIRED. Lease loss in disruptive workflows is never permission for blind replay.
08OPERATIONS CATALOG
Operations catalog
DNS operations are controlled in proportion to their blast radius.
Operators use the same product process and audit trail to change records, roll out nodes or manage DNSSEC.
Run DNS with serving speed and the rigor of a distributed system.
4SO GeoDNS controls changes without putting the Control Plane in the DNS answer path.