Probes and health¶
Three states, and the distinction matters:
| State | Meaning |
|---|---|
up |
The probe concluded the backend answers |
down |
The probe concluded it does not |
unknown |
The probe could not conclude (timeout, supervision 5xx, bad JSON, missing env var) |
unknown is not DOWN. The backend keeps the serving decision it already had: a
supervision outage must never become a DNS outage.
Lot 1 probe (frozen)¶
healthcheck_url: https://192.0.2.10/health
verify_tls: false
Equivalent to an HTTP GET expecting a 2xx. A timeout or a connection error
counts as a failure here, because that is the lot 1 contract; fail_threshold
consecutive failures mark the backend down.
Typed probe¶
A backend declares healthcheck_url or a probe block, never both. The
block may be one endpoint or a list of up to five, tried in order; only when none
concludes does the backend stay unknown.
probe:
type: http
url: https://192.0.2.10/api/v3/status
method: GET
timeout: 2
verify_tls: false
headers: {X-Probe: gslb}
headers_env: {Authorization: PROBE_TOKEN} # secret by reference
expect:
status: [200]
path: state
allowed: [online, degraded]
Secrets never reach the YAML or /api/config: headers_env names environment
variables. body_json is inline and therefore not for secrets; it is refused on
GET and HEAD, and capped at 4 KB.
# TCP: the connection opens or it does not
probe:
type: tcp
port: 5432
host: 192.0.2.30 # optional, defaults to the backend ip
# PromQL against a Prometheus you already run
probe:
type: prometheus
url: http://prometheus.internal:9090
query: up{instance="192.0.2.10:9100"}
threshold: 1
comparison: ge # healthy while query >= 1
The project deploys no Prometheus server: Prometheus is a source the probe
polls. Since 0.11 the API does serve a scrape of its own state on an
authenticated GET /metrics (see Operations); it
needs no Prometheus to run. The same probe client works with VictoriaMetrics,
Mimir and Thanos.
Editing a probe in the console¶
Each backend row has a Probe button that opens a drawer: type, URL, method,
expected status, JSON path, secret header names, timeout, and up to five
endpoints tried in order. Saving sends the block to
POST /api/v1/probes/validate first, so the drawer refuses exactly what the
runtime would refuse, with the runtime's own message.
A backend declares a probe or a healthcheck_url, never both. When the
backend uses a health URL, the drawer says that saving a probe removes it.
Probe templates¶
A template is a probe definition kept in the console's database so several
backends can follow it. It is editorial only: the DNS reads the expanded
probe: blocks in gslb.yaml and never the database.
| Layer | Where | Losing it means |
|---|---|---|
| Operational | gslb.yaml (probe: blocks) |
nothing: that file is what the DNS serves |
| Editorial | SQLite (probetemplate, probetemplatelink) |
you can no longer propagate an edit; every backend keeps its probe |
The Probes page lists the library and edits it; the backends keep their own
probe: block in the YAML. Nothing about a template is written to the YAML
unless you apply it: editing a template that backends follow previews the diff
and the number of backends it would rewrite, then applies it through the normal
save path. Deleting a template is refused while a backend still follows it.
States¶
The API computes the state of every link from the running configuration, on each read, and never stores it — a stored verdict would go stale after a hand edit, a restore, or a SIGHUP.
- linked — the backend's
probe:block means exactly the template's, with the runtime defaults applied (method: GET,expect.status: [200],comparison: lt, a tcphostdefaulting to the backend address). - diverged — the backend was edited since, or carries no probe at all.
- orphaned — the backend named by the link no longer exists: renamed by hand in the file, or its pool was removed. The link is kept and shown as a warning.
A rename made in the form (zone, pool or backend) rewrites the links. A rename made by hand in the YAML leaves them orphaned, which is visible and repairable.
Attach, detach, delete¶
- Attach a backend copies the definition into the form and creates the link.
Save the configuration to make it true; until then the link reads
diverged, which is what it is. - Detach removes the link only: the backend keeps its probe.
- Deleting a template that still has links is refused, with the count. The operator detaches first; a definition deleted under its links would leave nothing to reapply.
Editing a template does not rewrite the configuration by itself: the linked backends keep the probe they have until it is applied to them.
Editing a template¶
An edit is a draft: nothing reaches the library or the file until the operator decides.
- Preview impact compares the draft with the configuration and reports the backends that follow the template (their probe equals the stored definition), the ones that diverge, the orphaned links, and the YAML diff. Nothing is written.
- Apply to N backends writes the draft to every follower through the ordinary save path — validation, atomic write, acknowledgement, rollback — and only then updates the library. A refused reload leaves the library on the old definition, which is exactly what the backends still carry.
- Save without applying is offered only when no backend follows the template: saving the new definition then changes nothing that is deployed. With followers, the old definition is what makes them followers; losing it would leave nothing to propagate.
A backend whose probe was edited by hand reads diverged and is never
overwritten by an apply. Reapply, on its link, is the explicit way to copy
the stored definition back onto that one backend.
A template application is recorded in History with the trigger template, along
with the account that asked for it.
Export and import¶
Export writes the templates as JSON, without links: a link names a
(zone, pool, backend) that exists in one configuration only. Import is a merge
by name, so importing the same file twice changes nothing.
Guard rails¶
Method whitelist (GET, POST, HEAD), no body on GET/HEAD, 4 KB body cap, at
most 16 headers and 5 endpoints, secrets only through *_env references.
Proactive checks¶
proactive:
type: prometheus
url: http://prometheus.internal:9090
query: node_load1{instance="192.0.2.10:9100"}
threshold: 8
comparison: gt
remove_when_degraded: false
Indicators are evaluated only while the primary probe answers. Every indicator
must agree before the backend is flagged degraded, and an inconclusive one
vetoes the flag. degraded is not DOWN: no DOWN event, no SERVFAIL. With
remove_when_degraded: false (the default) it keeps answering and the console
shows a badge; with true it leaves the rotation until its indicators calm
down.
Maintenance¶
maintenance: true
The backend stays configured, keeps being probed and is shown in the console, but it is never answered and produces no DOWN event. Clearing the flag restores it at the next reload with the health state it had meanwhile.