High availability¶
Nodes and workers¶
Two words, two levels:
- a node is a host, named by
GSLB_INSTANCE(dns-a,dns-b). The console, the VIP and the failure cases are expressed in nodes; - a worker is one process on a node.
GSLB_WORKERSruns several of them on one address so a node uses every core it has (see Capacity).
A save is reported as applied when every live worker of every node acknowledged it, so a worker still serving the previous configuration keeps the save pending — the strict choice. A worker silent for more than ten seconds is shown as such and stops blocking; it will pull the configuration when it comes back. One node with eight workers is one row in the fleet view, with the eight processes underneath it, and a backend reported up by one worker and down by another of the same node is flagged separately from a disagreement between two nodes.
Topology¶
+---------------------------+
console ---> | web (nginx) -> API | one management host
| YAML volume, SQLite |
+------------+--------------+
| HTTP, static token
+---------------+---------------+
| |
DNS node A DNS node B behind an
192.0.2.10:53 192.0.2.11:53 external VIP
\ /
+--------- 192.0.2.40 ---------+ (optional,
keepalived example)
The API is a single point of management, never of resolution: a DNS node never asks it for an answer, only for its configuration. If the API or the console is down, the nodes keep serving the configuration they already have.
How the nodes stay in sync¶
- Each node polls
GET /api/v1/config/rawevery second. When the hash changes it writes the YAML locally, reloads, and posts the outcome toPOST /api/v1/instances/<id>/reload. - It pushes its snapshot every two seconds and its drained events. A snapshot reports the hash it actually runs, and therefore also acknowledges it: after an API restart there is no need to wait for a configuration change before the next save can succeed.
- A save waits up to five seconds for every live node (contacted in the last ten seconds). A silent node is shown as such but does not block; it pulls the configuration when it comes back.
- Events carry
(instance, seq). The API stores one row per key, so a retried batch is ignored instead of counted twice. - Counters are per process. The API aggregates them: query counts add up, a
backend is healthy only if every node sees it healthy, percentiles keep the
worst node, and
started_atis the oldest process.
Environment of a DNS node¶
GSLB_API_URL=http://192.0.2.5:8088
GSLB_API_TOKEN=<same static token as the API>
GSLB_INSTANCE=dns-paris
GSLB_ADDR=192.0.2.10
GSLB_CONFIG=/config/gslb.yaml
GSLB_CONFIG lives on a tmpfs in the example compose: the API is the source of
truth and the file is re-pulled at every start. Without GSLB_API_URL the lot 2
file channel runs unchanged and no socket is opened.
The fleet in the console¶
- Instances page: every node with its bind addresses and port, the release it runs, its workers, its last contact, the hash it runs compared with the file (current / behind / mixed), its last reload (applied / refused, origin, time), its query count, the backends where nodes disagree on health, and the Check VIP button.
- Overview card: a compact summary of the same data.
- A save shows
Confirmed by N DNS instances.or names the node that refused.
A health disagreement is the first sign of a network split or a probe path that only one node can reach; check the network between nodes before touching the configuration.
VIP with keepalived (optional example)¶
The project ships no failover daemon. docker-compose.ha-vip.yml and
examples/keepalived/ show one way to front the nodes with a floating address.
Both nodes bind their own address and the VIP:
GSLB_ADDR=192.0.2.10,192.0.2.40
GSLB_HEALTHCHECK_ADDR=192.0.2.10
Prerequisites:
net.ipv4.ip_nonlocal_bind = 1on both hosts (examples/keepalived/10-nonlocal-bind.conf), so the backup may bind an address it does not own.- The VIP is outside the DHCP pool and unused.
- VRRP (IP protocol 112) is allowed between the hosts.
- The keepalived container runs with
NET_ADMIN,NET_RAW,SETUIDandSETGIDon the host network; it owns the address, not the DNS process. The last two are what lets it fork its health script: without them keepalived cannot run the check and keeps its last verdict, which is "succeeded". The script runs asnobody(script_user), never as root: dig needs no privilege. GSLB_VRRP_IFACEnames the interface that owns the node address. On a host where that address is on a bridge (br0), name the bridge: VRRP sent on a bridge port is dropped by reverse-path filtering.
The health script queries the node address, never the VIP: on the backup the
VIP belongs to the master, so checking it would report the master's process.
It asks heartbeat.<zone> TXT, which answers as long as the process runs, even
when every pool is down: a backend outage must not move the VIP. The value names
the node and the version (ok dns-a 0.10.5), so dig @<vip> heartbeat.<zone> TXT
also tells you which node owns the VIP. Older nodes have no such name; the
example's script falls back to _whoami.<zone> during an upgrade. The interval
is GSLB_VRRP_CHECK_INTERVAL (2 s by default, so six seconds to fail over). After three
failures the node lowers its priority and the other takes the VIP.
GSLB_VRRP_CHECK_PORT (53 by default) follows a DNS that listens elsewhere.
Declare the floating address on the API too (GSLB_VIP, and GSLB_VIP_PORT
when the DNS does not listen on 53) so the Instances page can show it and check
it — the GSLB never owns the lease, it only reports which node answers on it.
Preemption is keepalived's default: when the preferred node comes back it takes
the VIP again immediately, so a node that flaps moves the address on every flap.
GSLB_VRRP_NOPREEMPT=1 makes the move sticky — the current owner keeps the VIP
until it fails in turn — and is the safer choice when the preferred node is the
one being maintained.
Measured with two nodes of two workers each (bench/RESULTS.md, section H): the
VIP moved 2 s after the DNS of the master stopped, and came back 5 s
after it returned. A save applied by both nodes took 1.8 s, and went through
even when a worker had been killed a second earlier: it waits for the
replacement's acknowledgement.
A check that cannot run reports success — keepalived has no way to say
"unknown". examples/keepalived/README.md lists the three ways this check was
silently broken and how to verify it: stop the DNS container and watch for
VRRP_Script(chk_dns) failed in the keepalived log.
Failure table¶
| Failure | Effect |
|---|---|
| One DNS node down | The VIP health check removes it; the fleet shows it silent |
| The API is down | DNS keeps answering its last configuration; reloads wait, events buffer (bounded outbox) |
| The console is down | No dashboard; resolution and saves through the API are unaffected |
| The management host is lost | Restore the API volumes; the DNS nodes keep serving meanwhile |
| A node cannot reach the API | It keeps its configuration, logs warnings, and catches up when the API returns |
Rollback¶
The lot 2 single-host stack and its volumes can stay in place while the fleet is
tested. To go back: stop the HA projects, then start
docker-compose.lot2.yml again on the host that ran it. The VIP is released when
both keepalived containers stop.