Skip to content

High availability

Nodes and workers

Two words, two levels:

  • a node is a host, named by GSLB_INSTANCE (dns-a, dns-b). The console, the VIP and the failure cases are expressed in nodes;
  • a worker is one process on a node. GSLB_WORKERS runs several of them on one address so a node uses every core it has (see Capacity).

A save is reported as applied when every live worker of every node acknowledged it, so a worker still serving the previous configuration keeps the save pending — the strict choice. A worker silent for more than ten seconds is shown as such and stops blocking; it will pull the configuration when it comes back. One node with eight workers is one row in the fleet view, with the eight processes underneath it, and a backend reported up by one worker and down by another of the same node is flagged separately from a disagreement between two nodes.

Topology

                +---------------------------+
   console ---> | web (nginx) -> API        |   one management host
                | YAML volume, SQLite       |
                +------------+--------------+
                             | HTTP, static token
             +---------------+---------------+
             |                               |
        DNS node A                      DNS node B       behind an
        192.0.2.10:53                   192.0.2.11:53    external VIP
             \                               /
              +--------- 192.0.2.40 ---------+            (optional,
                        keepalived                          example)

The API is a single point of management, never of resolution: a DNS node never asks it for an answer, only for its configuration. If the API or the console is down, the nodes keep serving the configuration they already have.

How the nodes stay in sync

  • Each node polls GET /api/v1/config/raw every second. When the hash changes it writes the YAML locally, reloads, and posts the outcome to POST /api/v1/instances/<id>/reload.
  • It pushes its snapshot every two seconds and its drained events. A snapshot reports the hash it actually runs, and therefore also acknowledges it: after an API restart there is no need to wait for a configuration change before the next save can succeed.
  • A save waits up to five seconds for every live node (contacted in the last ten seconds). A silent node is shown as such but does not block; it pulls the configuration when it comes back.
  • Events carry (instance, seq). The API stores one row per key, so a retried batch is ignored instead of counted twice.
  • Counters are per process. The API aggregates them: query counts add up, a backend is healthy only if every node sees it healthy, percentiles keep the worst node, and started_at is the oldest process.

Environment of a DNS node

GSLB_API_URL=http://192.0.2.5:8088
GSLB_API_TOKEN=<same static token as the API>
GSLB_INSTANCE=dns-paris
GSLB_ADDR=192.0.2.10
GSLB_CONFIG=/config/gslb.yaml

GSLB_CONFIG lives on a tmpfs in the example compose: the API is the source of truth and the file is re-pulled at every start. Without GSLB_API_URL the lot 2 file channel runs unchanged and no socket is opened.

The fleet in the console

  • Instances page: every node with its bind addresses and port, the release it runs, its workers, its last contact, the hash it runs compared with the file (current / behind / mixed), its last reload (applied / refused, origin, time), its query count, the backends where nodes disagree on health, and the Check VIP button.
  • Overview card: a compact summary of the same data.
  • A save shows Confirmed by N DNS instances. or names the node that refused.

A health disagreement is the first sign of a network split or a probe path that only one node can reach; check the network between nodes before touching the configuration.

VIP with keepalived (optional example)

The project ships no failover daemon. docker-compose.ha-vip.yml and examples/keepalived/ show one way to front the nodes with a floating address.

Both nodes bind their own address and the VIP:

GSLB_ADDR=192.0.2.10,192.0.2.40
GSLB_HEALTHCHECK_ADDR=192.0.2.10

Prerequisites:

  1. net.ipv4.ip_nonlocal_bind = 1 on both hosts (examples/keepalived/10-nonlocal-bind.conf), so the backup may bind an address it does not own.
  2. The VIP is outside the DHCP pool and unused.
  3. VRRP (IP protocol 112) is allowed between the hosts.
  4. The keepalived container runs with NET_ADMIN, NET_RAW, SETUID and SETGID on the host network; it owns the address, not the DNS process. The last two are what lets it fork its health script: without them keepalived cannot run the check and keeps its last verdict, which is "succeeded". The script runs as nobody (script_user), never as root: dig needs no privilege.
  5. GSLB_VRRP_IFACE names the interface that owns the node address. On a host where that address is on a bridge (br0), name the bridge: VRRP sent on a bridge port is dropped by reverse-path filtering.

The health script queries the node address, never the VIP: on the backup the VIP belongs to the master, so checking it would report the master's process. It asks heartbeat.<zone> TXT, which answers as long as the process runs, even when every pool is down: a backend outage must not move the VIP. The value names the node and the version (ok dns-a 0.10.5), so dig @<vip> heartbeat.<zone> TXT also tells you which node owns the VIP. Older nodes have no such name; the example's script falls back to _whoami.<zone> during an upgrade. The interval is GSLB_VRRP_CHECK_INTERVAL (2 s by default, so six seconds to fail over). After three failures the node lowers its priority and the other takes the VIP. GSLB_VRRP_CHECK_PORT (53 by default) follows a DNS that listens elsewhere.

Declare the floating address on the API too (GSLB_VIP, and GSLB_VIP_PORT when the DNS does not listen on 53) so the Instances page can show it and check it — the GSLB never owns the lease, it only reports which node answers on it.

Preemption is keepalived's default: when the preferred node comes back it takes the VIP again immediately, so a node that flaps moves the address on every flap. GSLB_VRRP_NOPREEMPT=1 makes the move sticky — the current owner keeps the VIP until it fails in turn — and is the safer choice when the preferred node is the one being maintained.

Measured with two nodes of two workers each (bench/RESULTS.md, section H): the VIP moved 2 s after the DNS of the master stopped, and came back 5 s after it returned. A save applied by both nodes took 1.8 s, and went through even when a worker had been killed a second earlier: it waits for the replacement's acknowledgement.

A check that cannot run reports success — keepalived has no way to say "unknown". examples/keepalived/README.md lists the three ways this check was silently broken and how to verify it: stop the DNS container and watch for VRRP_Script(chk_dns) failed in the keepalived log.

Failure table

Failure Effect
One DNS node down The VIP health check removes it; the fleet shows it silent
The API is down DNS keeps answering its last configuration; reloads wait, events buffer (bounded outbox)
The console is down No dashboard; resolution and saves through the API are unaffected
The management host is lost Restore the API volumes; the DNS nodes keep serving meanwhile
A node cannot reach the API It keeps its configuration, logs warnings, and catches up when the API returns

Rollback

The lot 2 single-host stack and its volumes can stay in place while the fleet is tested. To go back: stop the HA projects, then start docker-compose.lot2.yml again on the host that ran it. The VIP is released when both keepalived containers stop.