Operations¶
Editing the configuration¶
The console is the normal path. Zones, Backends, Routing and Probes edit one working copy; the bar at the bottom summarises the pending changes, shows the canonical diff and only then writes. The save is applied when the DNS acknowledges it; on refusal the previous file is restored, so disk and running configuration never diverge. The structured form is the source, the YAML is its rendering.
Every applied configuration is kept in the History page: diff against the
current file, restore an earlier version. Restores go through the same
validate → write → acknowledge → roll back path. Each version records its
trigger — api for a save from the console, setup for the first
configuration, restore for a history restore, template when a probe
template was applied to the backends that follow it — and the account that
asked for it.
File mode¶
Send SIGHUP after editing the YAML on disk:
docker kill --signal=HUP gslb # or: make reload
The file is read, parsed and validated before anything is applied. A bad file keeps the previous configuration serving and logs an ERROR. Mount a directory, never a single file: a file mount pins the inode, so an atomic replacement breaks SIGHUP.
Fleet mode¶
Nodes poll the API every second; there is nothing to signal. A manual edit of a node's local YAML is replaced by the API copy at the next poll, because the API is the source of truth. SIGHUP still applies the local copy immediately for a diagnosis, then the next poll wins.
Retention¶
Routing events, health events and availability samples are purged daily.
GSLB_RETENTION_DAYS (default 7, minimum 1) bounds them; a typo falls back to
the default instead of disabling retention. Configuration versions are kept
indefinitely.
One answer is one routing row of about 162 bytes, so the volume is set by the query rate and nothing else:
| Node rate | Rows per day | /data per day |
Seven days |
|---|---|---|---|
| 10 q/s | 0.86 M | 140 MB | 1 GB |
| 100 q/s | 8.6 M | 1.4 GB | 10 GB |
| 1 000 q/s | 86 M | 14 GB | 100 GB |
| 10 000 q/s | 864 M | 140 GB | 1 TB |
The purge removes rows in a few seconds per million (bench/RESULTS.md,
section G) but SQLite does not shrink the file: the space is reused by the
next writes. Deleting a lot at once therefore does not give the disk back; run
VACUUM during a maintenance window if you need it.
The console at high rate¶
Every dashboard query is linear in the rows of its window, about 2 µs per row:
one million rows is 2.6 s for /api/stats, and the console polls every five
seconds. Below a few hundred queries per second the pages stay instant; above
that, prefer the short windows and lower GSLB_RETENTION_DAYS. A rate of
thousands of queries per second makes the journal a data warehouse problem:
keep the retention at a day or two, or query the SQLite file yourself with the
SQL you need.
The DNS side is not affected: it pushes its events over HTTP and drops (with a logged count) whatever its bounded queue cannot hold, so a slow console never slows an answer.
Backups¶
| What | Where | How |
|---|---|---|
| Configuration | /config volume (HA) or bind mount (single host) |
copy gslb.yaml |
| Statistics and history | /data/gslb.db |
sqlite3 <db> ".backup <copy>", or VACUUM INTO |
Never copy gslb.db alone while the API runs: the most recent commits live in
gslb.db-wal until a checkpoint, and the copy reads as empty. Use
sqlite3 .backup, VACUUM INTO, or stop the container.
Prove the pair before you need it: scripts/restore.sh stages a copy, checks it
and compares it with the recorded history — see
Disaster recovery. make restore-test proves the script
itself and runs in CI on every merge request.
The configuration history is also in SQLite and in the console; the YAML file remains the source of truth.
Probe templates and the links saying which backend follows which template live
in gslb.db only. Losing that file does not change what the DNS serves: every
backend keeps its probe: block in gslb.yaml. What is lost is the ability to
update several backends from one definition, and the history of how the
configuration was composed. Export the library from the Templates tab when you
want a copy outside the database.
If the database is lost¶
The DNS keeps answering: it reads gslb.yaml, never SQLite. The API restarts on
an empty file — the tables are created by the first migration step — and the
service is unchanged from a client's point of view.
What disappears with the file:
| Lost | Consequence | Recovery |
|---|---|---|
| Routing and health events | Statistics, journal, suggestions start empty; availability strips show a gap | None, it was a rolling window anyway |
| Configuration versions | No diff, no restore, and the History page is empty |
Keep a copy of gslb.yaml: it is the current state, and the next save starts a new history |
| Probe templates and links | Templates and the diverged/linked verdicts are gone; the probes themselves stay in the YAML | Re-import a JSON export (GET /api/v1/templates/export), then re-attach the followers |
Audit trail (actor, source_ip) |
Past saves are no longer attributable | Keep the database backed up if that record matters for compliance |
Recommended: GET /api/v1/templates/export monthly, and a VACUUM INTO copy of
the database before any upgrade. The configuration history is an audit trail,
not a configuration store — losing it never changes an answer.
Disaster recovery¶
Two files matter, and they do not play the same role:
| File | Holds | Losing it costs |
|---|---|---|
gslb.yaml |
the configuration the DNS serves; it is the source of truth | every answer: the DNS has nothing else to serve, and no history can rebuild a file nobody saved |
gslb.db |
statistics, journal, configuration history, probe templates and links | no answer: the API recreates the tables on an empty file; what is lost is the history, the charts and the template library |
The DNS reads the YAML and never SQLite, so a node with its YAML keeps serving whatever happened to the database (see If the database is lost).
Take a backup you can actually restore¶
# On the API host, into a directory that is not the live one.
mkdir -p /root/backup
docker exec gslb-api python -c \
"import sqlite3; c = sqlite3.connect('/data/gslb.db'); \
c.execute(\"VACUUM INTO '/data/gslb-copy.db'\"); c.close()"
docker cp gslb-api:/data/gslb-copy.db /root/backup/gslb.db
docker cp gslb-api:/config/gslb.yaml /root/backup/gslb.yaml
docker exec gslb-api rm -f /data/gslb-copy.db
sha256sum /root/backup/*
VACUUM INTO (or sqlite3 gslb.db ".backup copy.db") is not optional: a plain
copy of gslb.db while the API runs can miss the commits still in
gslb.db-wal, and the copy then reads as an empty history. The restore script
warns when it finds a non-empty -wal next to the backup, which is the sign
that the pair was copied the wrong way.
Keep the two files together. They are compared against each other, so a YAML from Monday with a database from Friday is not a restore, it is a guess.
Prove the backup before you need it¶
./scripts/restore.sh --yaml /root/backup/gslb.yaml \
--db /root/backup/gslb.db --into /tmp/restore-check
The script stages a copy and proves it; it never writes to /config or /data
and starts no service. make restore-test proves the script itself and runs in
CI on every merge request; the command above proves your backup.
| Option | Meaning |
|---|---|
--yaml PATH |
the gslb.yaml backup, required |
--db PATH |
the gslb.db backup; optional, since the YAML alone is enough to serve |
--into DIR |
the directory that receives the copies; must be empty unless --force |
--expect-sha SHA256 |
the SHA-256 the YAML must have (the 64 hex characters, sha256: prefix accepted) |
--force |
write into a directory that is not empty |
What it does, in order:
- copies the YAML (and the database, when given) into
--into, then reads the SHA-256 of the copy, not of the source; - runs
PRAGMA integrity_checkon the copied database — the only honest answer to "is this file damaged"; - runs
python -m gslb checkon the copied YAML, with the same interpreter (PYTHON) and the same release: a backup the DNS would refuse is a failure now, not at 3 a.m.; - compares the YAML with the configuration history. The newest recorded version
may be a refusal (
applied = false, rolled back), so the question is "which applied version is this YAML, and was anything refused after it". A refusal of the same bytes is only reported — the API rolled the file back to those bytes. A YAML that matches no recorded version, or only a refused one, fails: the two files do not describe the same configuration; - prints a JSON report (
ok, SHA, integrity, schema, matched version, refused versions, warnings, errors).
Exit codes: 0 the pair is consistent, 1 a check failed (damaged file, YAML
refused by the runtime, SHA mismatch, no matching version), 2 the command was
wrong (missing argument, non-empty target without --force, a target under
/config or /data).
Without --db the YAML is validated and the report says
history: absent — a legitimate restore of the configuration alone.
Put a backup back¶
- Stop the API, not the DNS: a DNS node keeps answering from the configuration it last acknowledged, so the service stays up during the restore.
- Put
gslb.yamlin/config(the API'sconfigvolume in HA, the bind mount in the single-host compose). The next save or SIGHUP makes the DNS pull it; an HA node pulls at boot and at every sync tick. - Put
gslb.dbin/data, owner UID 1000 (chown 1000:1000), then start the API.docker cpcreates the file as root, and a root-ownedgslb.dbis one SQLite (running as UID 1000) cannot write: the API would crash-loop with "attempt to write a readonly database". The migrations run forward on their own: an older schema is upgraded, never refused, so a 0.9.x file opens on a 0.11 release. - If you restored the YAML without the database, expect an empty history and an empty template library: the next save starts a new audit trail. Re-import the templates from their JSON export and re-attach the followers.
- Check it:
docker exec gslb-api python -m gslb checkvalidates the file in place, the Instances page shows every nodecurrent: true, anddig @<vip> heartbeat.<zone> TXTnames the node that holds the VIP.
Order matters only for the history: a YAML restored without its database serves immediately. The DNS never waits for SQLite.
In HA¶
| Where | What lives there | What a lost host means |
|---|---|---|
| API host | config and data volumes: the YAML and the whole history |
rebuild the host, restore the pair, the DNS keeps answering meanwhile |
| DNS node | a tmpfs copy of the YAML, and /state for its own counters |
nothing to restore: the node is stateless and refills from the API at boot |
| keepalived | the VIP lease | outside this project's restore path: chk-dns decides, see Split-brain |
So a DNS node disaster needs no backup at all — a fresh container with the
right GSLB_API_URL and token serves again in seconds. An API host disaster
needs the last pair; the DNS answers from its local copy while you rebuild it. Do
not restore the production database into a demo or a test host: the history is
the audit trail, and merging two is not something this project supports.
On the appliance¶
Both files live inside the VM, in the config and data Docker volumes. The API
image ships /app/scripts/restore.sh precisely so the proof runs where the files
are, with no clone to fetch. Compose needs the environment file: it is what
supplies GSLB_API_PASSWORD and the rest.
compose="docker compose --env-file /etc/gslb/gslb.env -f /opt/gslb/compose.yml"
# 1. Stage and prove the pair in a one-off container. The API service is
# read_only, so `docker cp` into it fails; a bind mount is the honest way to
# hand it the backup. The staged copy goes to a host directory owned by
# UID 1000, so the proof survives the one-off container.
chmod 0644 /root/backup/gslb.yaml /root/backup/gslb.db # the container is UID 1000
install -d -o 1000 -g 1000 /root/restore-checked
$compose run -T --rm --no-deps \
--volume /root/backup:/backup:ro \
--volume /root/restore-checked:/out \
gslb-api /app/scripts/restore.sh \
--yaml /backup/gslb.yaml --db /backup/gslb.db --into /out/stage
# The JSON report is on stdout; the staged pair is /root/restore-checked/stage.
# 2. Put the proved files in place with the API stopped. The copy runs in the
# service itself (UID 1000): `docker cp` would create the files as root, and
# SQLite could not open a root-owned database.
$compose stop gslb-api gslb
$compose run -T --rm --no-deps --volume /root/restore-checked/stage:/in:ro \
gslb-api cp /in/gslb.yaml /config/gslb.yaml
$compose run -T --rm --no-deps --volume /root/restore-checked/stage:/in:ro \
gslb-api cp /in/gslb.db /data/gslb.db
$compose start gslb-api gslb
docker exec gslb-api python -m gslb check
The one-off container is removed; the staged pair and the JSON proof stay on the
host in /root/restore-checked/.
Split-brain¶
The GSLB itself never replicates: the configuration is written in one place (the API), and every DNS node pulls it. A partition therefore has one shape, and it is worth stating plainly:
- A node cut off from the API keeps its last YAML and keeps answering. It does not fall back to an empty zone, it does not SERVFAIL. Clients routed to it keep getting an answer, possibly a stale one: the last configuration it acknowledged.
- Nothing diverges until someone saves. The API writes one file and waits
for every live node to acknowledge it. A partitioned node is
staleafter 10 s and no longer blocks the save: it is reported, not waited for. When it comes back it pulls the new file and acknowledges it — no merge, no reconciliation, last write wins by construction. - The VIP does not move because of a partition. keepalived decides, using
the
chk-dnsscript against the node address (seeexamples/keepalived/). A node that still answers DNS keeps the VIP even if it cannot reach the API. That is deliberate: an isolated node usually has a working data plane. - The console shows the gap, not a guess.
/api/v1/instancesgives each node itscurrentflag (hash running vs hash on disk),stale, anddisagreements(a backend up on one node, down on another — a probe or a network problem, not a configuration one). A node that stopped reporting for more than 10 s is listed asstale, never ascurrent: true.
Recovery: do not restart the API to "resync" — it holds the reference. When
the partition heals the node syncs by itself within one polling interval. Check
current: true on every node, then, if you suspect a node applied an older
file, save the configuration once more (or restore the wanted version from the
History page) and watch the acknowledgements: the save only reports success
when every live node confirmed the same hash.
The one real split-brain risk is two APIs writing the same zone: the design has a single management plane, so run one API. Two APIs on one database file are not supported (SQLite on a network filesystem corrupts under concurrent writers), and two APIs on two databases would each believe their own file.
Logs¶
- Text by default, unchanged from lot 1.
GSLB_LOG_FORMAT=jsonemits one JSON object per line, standard fields plus thekey=valuetokens already present in every message. Works for the DNS and for uvicorn, so a collector sees one format.- The DNS logs one DEBUG line per answer with
client=,qname=,pool=,ip=andrule=; DEBUG is lazy and off in production.
Reading the statistics¶
A query either reaches a pool or it does not. The console draws the pools on one
scale and states the rest underneath — 365 569 queries were not routed
(heartbeat 259 200, outside_zone 106 369) — because the two are not the same
thing: a node's liveness check, a client asking for a name outside the zone or a
type the zone does not serve, and a total outage all end with "no pool
answered". They stay in the journal and in by_pool as none, and
/api/v1/stats exposes the split as not_routed_reasons.
A liveness check at one query every two seconds is 43 200 queries a day per node
(heartbeat). On a site doing a few hundred queries a second it is invisible; on
a small one it can be most of the volume, which is exactly why it is labelled
and drawn apart rather than removed. The console gives it its own line
("N queries were liveness checks"), separate from the clients that reached no
backend: monitoring is not a misrouted client, and the two are worth telling
apart.
Monitoring¶
The console shows QPS, latency percentiles (p50/p95/p99 of the DNS answer
path), pool health, availability strips, fallback reasons, client /24
networks, unregistered client subnets (on the Routing page, as a rule to
create) and the fleet view. The API also computes the traffic-imbalance and
health-flapping detectors — /suggestions returns them — but the console draws
neither yet: query them directly, or treat them as an API feature.
The API exposes the same data under /api/v1; see API.
Prometheus¶
GET /metrics (canonical GET /api/v1/metrics) returns the text exposition
format 0.0.4. It is authenticated like every other data route: the
exposition names every pool and backend, which is a map of the private network.
Use the read-only token — it exists for exactly this.
# prometheus.yml
scrape_configs:
- job_name: gslb
scrape_interval: 30s
authorization:
type: Bearer
credentials_file: /etc/prometheus/gslb-ro-token
static_configs:
- targets: ["gslb-console.example.com:8080"]
Series: pygslb_queries_total{pool}, pygslb_queries_not_routed_total,
pygslb_queries_received_total, pygslb_backends_total{pool},
pygslb_backends_up{pool}, pygslb_backends_maintenance{pool},
pygslb_config_info{sha,actor,trigger},
pygslb_config_last_applied_timestamp_seconds, pygslb_reload_ack{instance},
pygslb_snapshot_age_seconds and pygslb_db_size_bytes.
none is not a pool: the queries that reached no backend are their own counter,
so a liveness check does not look like site traffic. No client address is ever
a label. A backend withdrawn by remove_when_degraded still counts in
pygslb_backends_up until its health flips: the DNS snapshot does not carry
that flag.
pygslb_config_info and pygslb_config_last_applied_timestamp_seconds describe
what the DNS acknowledged, read from /state/reload.json: a refused save
publishes neither (the refused hash is in the history, but it is not running),
and the database is only a fallback when no acknowledgement exists at all.
pygslb_config_info therefore carries one series per acknowledged hash; bound
it with --storage.tsdb.retention.time as for any info metric.
pygslb_reload_ack is one series per node in HA mode; in single-host mode it is
{instance="local"} = 1 only when the acknowledged hash equals the file on
disk, which is the machine-readable form of "drift". pygslb_snapshot_age_seconds
is absent while nothing has been published: 0 would claim a freshness no
node reported.
The DNS image still ships no exporter and no extra port: the API reads the
snapshot it already aggregates. There is still no Grafana bundle and no
/metrics on a DNS node.
Adding a DNS node¶
- Install the DNS image and the
docker-compose.ha-dns.ymlfile on the host. - Set
GSLB_INSTANCEto a unique name,GSLB_ADDRto the node address (and the VIP when a floating address is used),GSLB_API_URLandGSLB_API_TOKEN. - Start it. The node registers itself with its first snapshot (two seconds) and appears on the Instances page.
- Add it to the VIP peers (keepalived) if applicable, and to the load balancer.
Removing a node is the reverse: stop it, remove it from the peers. Its last state stays in the console until the API restarts (the registry is in memory) and its events stay in SQLite until retention.
Rotating the API token¶
- Generate a new token.
- Update
GSLB_API_TOKENon the API and restart it. - Update each DNS node's env and restart it; until then the node logs warnings and keeps serving its last configuration.
- The console session uses Basic or the same token and is unaffected.
Requiring TLS¶
Basic authentication sends the password base64-encoded, which protects nothing
on the wire. Since 0.11 the API refuses a credential that arrived over plain
HTTP, with a 403 that says what to do. Nothing to enable: it is the default.
Where TLS terminates¶
The console image is nginxinc/nginx-unprivileged, running as UID 101 with
cap_drop: ALL: it has no CAP_NET_BIND_SERVICE and cannot bind a port
below 1024. A listen 443 inside the container fails at startup
(bind() to 0.0.0.0:443 failed (13: Permission denied)).
So there is one nginx assignment in every deployment: a TLS server block that listens on 8443, published on the host as 443. It is required even when an upstream proxy terminates TLS, because that proxy must reach a TLS listener — there is no other one. Who holds the certificate changes; the block does not.
| Caller | URL it uses | Host mapping |
|---|---|---|
External proxy or load balancer terminating TLS, calling this nginx |
https://<console> (host 443) |
"443:8443" |
| Same, when the proxy is configured to call 8443 | https://<console>:8443 |
"8443:8443" |
| Proxy container in the same Docker network, terminating TLS | https://gslb-web:8443 |
none |
Browser, certificate on this nginx |
https://<console> (host 443) |
"443:8443" |
In the first three rows the certificate lives on the proxy, but the hop into
this nginx is still HTTPS: X-Forwarded-Proto is $scheme, computed from that
hop. A proxy configured for http://gslb-web:8080 produces X-Forwarded-Proto:
http and every authenticated call answers 403 — 8080 is not a TLS listener.
Add this block to web/nginx.conf (or mount your own file over
/etc/nginx/conf.d/default.conf, keeping the 8080 block too), and rebuild
gslb-web:
# TLS listener on 8443: the image runs as UID 101 without CAP_NET_BIND_SERVICE,
# so it cannot bind 443. docker-compose.tls.yml publishes this as 443:8443.
server {
listen 8443 ssl;
server_name _;
ssl_certificate /etc/nginx/certs/fullchain.pem;
ssl_certificate_key /etc/nginx/certs/privkey.pem;
ssl_protocols TLSv1.2 TLSv1.3;
root /usr/share/nginx/html;
index index.html;
location /api/ {
proxy_pass http://gslb-api:8000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 10s;
}
location = /metrics {
proxy_pass http://gslb-api:8000;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 10s;
}
location / {
try_files $uri /index.html;
}
}
docker-compose.tls.yml in the repository carries the host side: it publishes
443:8443 (and 8443:8443 in a comment, for a proxy that calls 8443) and mounts
the certificates read-only. The base file keeps publishing 8080: that plain
listener is not a TLS listener, but the image health check queries
http://127.0.0.1:8080/ inside the container, so it must keep answering.
GSLB_TLS_CERTS=./certs \
docker compose -f docker-compose.ha-api.yml -f docker-compose.tls.yml up -d
The port mapping alone does nothing: 443:8443 only publishes the listener that
the server block above creates. cap_add: NET_BIND_SERVICE would let the
container bind 443 directly, but it re-grants a capability this project drops on
purpose — 8443 costs nothing and keeps cap_drop: ALL.
The bundled web/nginx.conf listens on plain HTTP 8080 only. Its
X-Forwarded-Proto is $scheme, the scheme of the connection to this nginx:
an incoming X-Forwarded-Proto is overwritten, never forwarded, so a client
cannot claim HTTPS by writing a header. Consequently:
A proxy that terminates TLS and forwards plain HTTP to this nginx does not work:
$schemeishttp, the API sees plaintext and refuses. The hop must be HTTPS, to 8443.
# API container, whichever shape
GSLB_TRUST_PROXY=1
GSLB_TRUST_PROXY=1 is also what makes the API read X-Real-IP and
X-Forwarded-For for the audit trail. Without it those headers are ignored,
which is the point: anyone reaching the API directly could otherwise claim to be
on HTTPS and choose the address recorded. With it, the API must not be reachable
except through the proxy (the HA compose publishes no API port for that reason).
Not supported in 0.11: trusting the header from a specific peer only. That
would need GSLB_TRUST_PROXY to take a CIDR list of trusted proxies instead of a
boolean, and validating the forwarding chain hop by hop. Believing the header
half-way — when it comes from anywhere — is exactly the hole this version
closes, so it is documented as missing rather than approximated.
Settings, in the order the API applies them:
| Setting | Effect |
|---|---|
GSLB_REQUIRE_TLS=1 |
Refuse plaintext, always. Wins over the two below |
GSLB_REQUIRE_TLS=0 |
Allow plaintext. Deprecated alias of 0.10: a variable named "require" that means "allow" is a trap, use the next one |
GSLB_ALLOW_PLAINTEXT_AUTH=1 |
Allow plaintext. The documented lab switch; the demo compose sets it |
| nothing | Refuse plaintext |
A deployment that allows it logs a WARNING at startup
(authentication accepts plaintext HTTP ...). /healthz is never affected, so
the container health check keeps working on either scheme.
Upgrading from 0.10: an installation that relied on the old default (plain HTTP
accepted) now answers 403. Either make the hop to the console HTTPS with
GSLB_TRUST_PROXY=1, or set GSLB_ALLOW_PLAINTEXT_AUTH=1 for a lab. This is the
one breaking change of 0.11.0.
Who changed the DNS¶
Every saved, restored, setup or template-applied configuration is recorded with
its actor — the Basic user name, or token when the shared API token was used —
and the address the save came from. The History page shows both, in the
Applied by and Source columns, next to the trigger, the hash, whether the
reload was accepted, and a diff; the CSV export carries them too. Rows written
before these columns exist show unknown and —: the past has no author or
address to invent.
The address is the administrator's, not a DNS client's: the query journal only ever stores client networks reduced to a /24 (D13), while this one is kept indefinitely with the version it wrote, because it is the evidence an audit asks for. A refused reload is recorded with its address as well — that is the case an auditor actually wants to trace.
Behind a reverse proxy the socket peer is the proxy, so the real address is read
from X-Real-IP, or the last non-empty hop of X-Forwarded-For, but only
when GSLB_TRUST_PROXY=1 is set. Without it those headers are just text a client
can write, and they are ignored. The contract for the proxy: overwrite
X-Real-IP, append to X-Forwarded-For; with GSLB_TRUST_PROXY=1 the API must
not be reachable directly.
Fleet and VIP¶
Each DNS node publishes its bind addresses, port and release with its snapshot,
so the Instances page shows where it listens, which version it runs, its
workers, its configuration state and its last acknowledgement. The floating
address is only declared: set GSLB_VIP (and GSLB_VIP_PORT, default 53) on
the API. The GSLB does not own the lease, keepalived does. The Check VIP
button sends a real heartbeat.<zone> query to that address and reports which
node answered and with which version — the only honest check available to the
management plane. A node that stopped answering its heartbeat is a silent node,
not a healthy VIP holder.
The journal¶
The Journal page reads the routing events through /api/v1/events, which
filters and counts server-side: the page indicator shows the size of the
filtered result, not of the page. Pool labels in the filter are the ones the
DNS recorded — apex/pool when several zones share the process — and the
reason list comes from the decision code itself (gslb.decide.REASONS), so the
console cannot offer a value no row carries. Export CSV carries the same
filters and caps at 10 000 rows; narrow the range rather than raising the cap.
Read-only token¶
A dashboard or a monitoring probe does not need the credential that rewrites the zone. Set a second token and give it to them:
GSLB_API_RO_TOKEN=<random>
It authenticates every read (/stats, /config, the journal, the CSV exports)
and is refused on every write with 403 read-only credential. Writes keep
requiring GSLB_API_USER/GSLB_API_PASSWORD or GSLB_API_TOKEN. When only
the read-only token is configured, reads work and writes answer 403 writes are
disabled: no write credential is configured — fail closed, never anonymous.
OIDC¶
Without OIDC the API has one write credential for the whole console: the
audit trail names an account (admin, token, read-only), never a person.
With OIDC every operator signs in at your identity provider, the group they
belong to decides what they may do, and the audit trail records their address
and their e-mail.
It is optional and additive. Leave GSLB_OIDC_ISSUER unset (in the environment
and on the Identity page) and nothing changes: Basic and the tokens are the
only credentials, /auth/methods answers oidc: false, and /auth/login
answers 503.
Roles¶
| Role | May do |
|---|---|
reader |
Every GET, plus the pure computations: POST /config/validate, POST /probes/validate, POST /templates/{id}/preview |
operator |
A reader, plus POST /probes/test and POST /instances/vip-check |
admin |
An operator, plus every write: PUT /config, PUT /setup, a restore, every template mutation, and the DNS node push endpoints |
Machine identities keep their place: Basic and the write token are admin, the
read-only token is reader, and the DNS node token is the write token, so a
node keeps acknowledging configurations while OIDC is enabled. A role is a
minimum, never a grant by itself: an admin group member is also a reader.
No group matches any configured group → refused at the callback (403 no
GSLB role), unless GSLB_OIDC_DEFAULT_ROLE names one (reader, operator or
admin). A typo in that variable is ignored on purpose, so it locks people out
instead of silently inventing a role.
The console follows the same table: reader and operator see a
Read-only role banner on the editing pages, no Save and apply bar, no
Restore and no template action; operator keeps Test on a probe and
Check VIP. Every page stays readable for every role — the interface hides
what it cannot do, it never hides what it may read. The API is still the
authority: a console that outlived a role change gets a 403, which shows the
message and does not sign anyone out (only a 401 does).
Configuration¶
GSLB_OIDC_ISSUER=https://sso.example.com/realms/gslb
GSLB_OIDC_CLIENT_ID=gslb-console
GSLB_OIDC_CLIENT_SECRET=<secret from the provider>
GSLB_OIDC_REDIRECT_URI=https://gslb.example.com/api/v1/auth/callback
GSLB_OIDC_GROUPS_CLAIM=groups
GSLB_OIDC_ADMIN_GROUP=gslb-admins
GSLB_OIDC_OPERATOR_GROUP=gslb-operators
GSLB_OIDC_READER_GROUP=gslb-readers
# Optional: GSLB_OIDC_ALLOWED_DOMAINS=example.com GSLB_OIDC_DISPLAY_NAME=Microsoft
GSLB_SESSION_SECRET=<32+ random bytes>
GSLB_PUBLIC_ORIGIN=https://gslb.example.com
Only the two secrets have to stay here; every other value above can also be
typed on the Identity page, which wins over the file (GSLB_OIDC_ENABLED,
GSLB_OIDC_PRESET, GSLB_OIDC_ALLOWED_DOMAINS and GSLB_OIDC_DISPLAY_NAME
included). GSLB_OIDC_REDIRECT_URI may be left empty when
GSLB_PUBLIC_ORIGIN is set: the callback becomes
<origin>/api/v1/auth/callback.
Every variable above is already declared in the compose files this project
ships (docker-compose.ha-api.yml, docker-compose.lot2.yml and the
appliance). Filling the env file and restarting is enough: no compose file
needs editing. That is worth stating because docker compose --env-file only
substitutes ${...} — a variable the service's environment: block does not
list never reaches the container, with no error at all (the 0.11.3 defect,
fixed in 0.11.4). A variable left empty turns its feature off: with no
GSLB_OIDC_ISSUER, the API behaves exactly as it did in 0.11.2.
GSLB_OIDC_ISSUERis the switch, and the discovery document is read from<issuer>/.well-known/openid-configuration(cached five minutes). The document must name the same issuer, otherwise it is refused: a discovery document pointing elsewhere is a redirection attack, not a typo.GSLB_SESSION_SECRETsigns the session cookie. Without it sign-in answers503and logs an error — the API never invents a secret, which would sign everybody out at every restart or behind every worker. It is the one variable of this block with no safe default; every other one may stay empty.GSLB_PUBLIC_ORIGINis the console's public origin. It is compared with theOrigin(orReferer) of every cookie-authenticated write. Unset, the request's ownHostis the reference, which is right for a single hostname and wrong behind several.
Configuring from the console¶
Since 0.13 the provider can be configured in Settings → Identity instead of the environment file. The rule is one line:
a value saved in the console (SQLite) wins over the environment variable, which wins over the default.
Reading the settings is allowed to every role; saving, erasing and testing are
administrator actions. The page shows the effective values, the source of each
one (console, env or default), the callback URL to register at the
provider, and whether each secret is set. Back to the environment erases
every override — each erase is in the audit trail — and the environment becomes
the configuration again. A save applies to the next sign-in, without a restart.
Two variables are deliberately not editable there: GSLB_OIDC_CLIENT_SECRET
and GSLB_SESSION_SECRET. The database is backed up and restored, so a secret
stored in it would travel in every VACUUM INTO; the page reports
secret_configured / session_secret_configured and names the variable instead.
The redirect URI is computed from GSLB_PUBLIC_ORIGIN (or
GSLB_OIDC_REDIRECT_URI when pinned) and shown read-only with a copy button: a
typo in that string is the most common sign-in failure.
Test the connection calls POST /settings/identity/test and reports four
checks — the discovery document is reachable, it names the same issuer, a client
id is set, the callback is usable — with the raw detail under each. It stores
nothing and opens no session, so it is safe to run before a save.
Microsoft Entra ID¶
Exact values, in the order the portal asks for them:
- Entra admin center → Identity → Applications → App registrations → New
registration. Name it
GSLB console. - Supported account types: Accounts in this organizational directory only
(single tenant) unless you really need guests from other directories. The
issuer is then
https://login.microsoftonline.com/<tenant-id>/v2.0, where<tenant-id>is the Directory (tenant) ID on the app's Overview page. Multi-tenant useshttps://login.microsoftonline.com/common/v2.0, but then any Microsoft account can reach the sign-in page: restrict it with Allowed domains, and remember that group GUIDs from another tenant never match yours. - Redirect URI: platform Web, value
https://gslb.example.com/api/v1/auth/callback— the exact string the Identity page shows (copy it from there). Do not add a second spelling. - Certificates & secrets → Client secrets → New client secret. Copy the
Value (not the Secret ID) into
GSLB_OIDC_CLIENT_SECRETand restart the API. This is the only step done in the environment file. - Token configuration → Add groups claim → Security groups. The ID token
then carries the group object ids (GUIDs), not their names: paste the
GUID, not
gslb-admins, into the administrator/operator/reader fields. - App roles instead (recommended for large directories): App roles → Create
app role (
GSLB.Admin,GSLB.Operator,GSLB.Reader, allowed member type Users/Groups), assign the groups to those roles under Enterprise applications → Users and groups, then add a roles claim in Token configuration. Set Groups claim torolesand the three fields to the role values. - In the console: preset Microsoft Entra ID, the issuer from step 2, the Application (client) ID, the callback from step 3, then Test the connection and Save.
Two traps specific to Entra:
- Group overage. Above roughly 150 groups in the token (or 5 for implicit
flows), Microsoft replaces the list with
_claim_names: {"groups": …}. No group is then in the token; the console refuses the sign-in with a message about overage instead of granting a default role, because the token simply cannot say who the account is. Define app roles and use therolesclaim, or look the groups up through Graph. preferred_usernamemay be the only identity. Some tenants put noemailclaim in the ID token; the audit then recordspreferred_username, which is what the code already prefers afteremail.
Google¶
- Google Cloud console → APIs & Services → OAuth consent screen. Internal for a Workspace organisation, External otherwise (an External app in Testing only admits the test users you list).
- Scopes:
openid,email,profile. Nothing else is requested. - Credentials → Create credentials → OAuth client ID → Web application.
Authorised redirect URI:
https://gslb.example.com/api/v1/auth/callback, copied from the Identity page. Create, then copy the Client ID and Client secret. - Google sends no groups in the ID token. There is nothing to map, so a Google deployment must do one of these:
- set a default role (
readeris the safe one), or - restrict sign-in with Allowed domains (
example.com): the check reads thehdclaim of a Workspace account and the domain of the e-mail. Without either, every sign-in is refused with403— which is the fail-closed behaviour, not a bug. - In the console: preset Google (issuer
https://accounts.google.comand scopesopenid profile emailare filled in), the client id, then the default role or the allowed domains, Test the connection, Save.
Generic OIDC: Keycloak, Authentik, ADFS¶
Any issuer that serves /.well-known/openid-configuration and implements the
authorization-code flow with PKCE works: Keycloak (one realm URL per
deployment), Authentik (one application), ADFS 2016 or later
(https://adfs.example.com/adfs). The recipe is always the same:
- Create a confidential client whose client id is
gslb-console. - Register
https://gslb.example.com/api/v1/auth/callbackas the exact redirect URI; enable PKCE methodS256where the provider offers it. - Create three groups —
gslb-admins,gslb-operators,gslb-readers— and put each operator in the right one. - Make the provider emit the group list in the ID token, in a claim named
groups(or change Groups claim to the name it uses). Keycloak needs an explicit group-membership mapper for this; with Full group path on it writes/gslb-admins, and both spellings are accepted. - Put the client secret in
GSLB_OIDC_CLIENT_SECRET, then configure the issuer, client id and the three group names on the Identity page.
Active Directory on-premises alone cannot be used. The product speaks OIDC, never LDAP or Kerberos: an AD that only speaks LDAP needs an OIDC front end — Keycloak with the LDAP federation, Authentik, or ADFS — and the GSLB is configured against that front end.
The provider in examples/oidc/ is not a component of the product
It is a Keycloak in development mode, with a versioned realm and lab passwords, and it exists for one purpose: validating the sign-in mechanism end to end in a lab that has neither Microsoft nor Google. In production you connect the directory your organisation already owns; nothing here deploys an identity provider. That Keycloak speaks HTTP with lab secrets and must never be exposed to a network you do not control.
The shipped example still works unchanged: examples/oidc/ holds its realm,
compose file and README, and the same values can be typed in the environment or
on the Identity page.
The flow, and what is checked¶
GET /api/v1/auth/login?next=/ stores state, a nonce and the PKCE verifier
in a signed cookie (gslb_oidc, five minutes, HttpOnly, SameSite=Lax) and
redirects to the provider. GET /api/v1/auth/callback checks the state against
the cookie, exchanges the code with the verifier, then verifies the ID token:
- signature against the provider's JWKS, asymmetric algorithms only;
issis the provider we discovered,audis our client id;expandiatare current (60 s leeway for clock skew);nonceis the one this browser was given — a token replayed from another sign-in is refused.
Then the role is mapped, the session cookie is set and the browser is sent to
next. Only a local path is accepted (/..., never //host or
https://host): an open redirect in a sign-in page is a phishing page on your
own domain.
The userinfo endpoint is never called: one signed document decides the
identity, so two sources cannot disagree.
Sessions are stateless. gslb_session is signed, HttpOnly, SameSite=Lax,
Path=/, Max-Age=GSLB_SESSION_TTL (8 h by default) and Secure — except when
GSLB_ALLOW_PLAINTEXT_AUTH=1, which is the documented lab switch and relaxes it
so a plain-HTTP test deployment can work at all.
The console offers Sign in with SSO as its primary button as soon as
/auth/methods announces OIDC; the Basic and token form stays below as a
fallback, and disappears entirely on an instance that has neither. A successful
sign-in forgets any machine credential the tab had kept, because an
Authorization header wins over the session cookie at the API.
The role is frozen in the cookie until it expires. Removing someone from
gslb-admins does not demote the session they already hold; it applies at their
next sign-in. To cut a session short, rotate GSLB_SESSION_SECRET (every session
becomes invalid) or shorten GSLB_SESSION_TTL.
POST /api/v1/auth/logout erases the cookie and returns the
GSLB_OIDC_LOGOUT_URL the console may then open. Signing out of the provider is
the provider's business: without that variable the console returns to its own
sign-in page and the provider session stays open — GSLB_OIDC_LOGOUT_URL is the
only way to close both.
CSRF¶
A cookie travels with any request the browser makes, so an unsafe method
(POST, PUT, DELETE) authenticated by the session cookie must also carry:
X-Requested-With: gslb-console, a custom header a cross-origin form orfetchcannot add without a preflight the API never allows;- an
Origin(orReferer) matchingGSLB_PUBLIC_ORIGIN, or the request's ownHostwhen it is unset.
Requests authenticated by a header credential (Basic, a token) are exempt: no
browser sends those on someone else's behalf. SameSite=Lax is the browser's
half of the same defence, not a replacement for it.
Troubleshooting¶
| Symptom | Cause |
|---|---|
Console answers 403 on every call, $scheme is http |
The hop into the console nginx is not HTTPS; see Requiring TLS |
/auth/login answers 503 |
OIDC half-configured (ISSUER, CLIENT_ID, REDIRECT_URI), or GSLB_SESSION_SECRET missing |
| Callback says "the sign-in state is missing or has expired" | The gslb_oidc cookie was dropped (different host, five minutes elapsed, a proxy stripping cookies) |
403 no GSLB role |
The token carries no mapped group: check the group mapper and GSLB_OIDC_GROUPS_CLAIM |
401 the ID token claims are not valid |
Wrong audience (client id), expired token, clock skew beyond 60 s, or a nonce from another tab |
/auth/methods says oidc: false while the variables are set |
One of the three required values is empty, or the secret is missing |
403 the provider sent too many groups … (group overage) |
Microsoft replaced the group list: use app roles and the roles claim, or Graph |
403 this account is outside the allowed domains |
The sign-in succeeded but the address is not in the allowed list; remove it or add the domain |
The Identity page shows source: console after I edited the env file |
A console override wins; press Back to the environment to erase it |
A save answers 400 map at least one group to a role, or set a default role |
Nobody could sign in with that mapping; name a group or a default role |