Skip to content

Probes and health

Three states, and the distinction matters:

State Meaning
up The probe concluded the backend answers
down The probe concluded it does not
unknown The probe could not conclude (timeout, supervision 5xx, bad JSON, missing env var)

unknown is not DOWN. The backend keeps the serving decision it already had: a supervision outage must never become a DNS outage.

Lot 1 probe (frozen)

healthcheck_url: https://192.0.2.10/health
verify_tls: false

Equivalent to an HTTP GET expecting a 2xx. A timeout or a connection error counts as a failure here, because that is the lot 1 contract; fail_threshold consecutive failures mark the backend down.

Typed probe

A backend declares healthcheck_url or a probe block, never both. The block may be one endpoint or a list of up to five, tried in order; only when none concludes does the backend stay unknown.

probe:
  type: http
  url: https://192.0.2.10/api/v3/status
  method: GET
  timeout: 2
  verify_tls: false
  headers: {X-Probe: gslb}
  headers_env: {Authorization: PROBE_TOKEN}   # secret by reference
  expect:
    status: [200]
    path: state
    allowed: [online, degraded]

Secrets never reach the YAML or /api/config: headers_env names environment variables. body_json is inline and therefore not for secrets; it is refused on GET and HEAD, and capped at 4 KB.

# TCP: the connection opens or it does not
probe:
  type: tcp
  port: 5432
  host: 192.0.2.30        # optional, defaults to the backend ip

# PromQL against a Prometheus you already run
probe:
  type: prometheus
  url: http://prometheus.internal:9090
  query: up{instance="192.0.2.10:9100"}
  threshold: 1
  comparison: ge          # healthy while query >= 1

The project deploys no Prometheus server: Prometheus is a source the probe polls. Since 0.11 the API does serve a scrape of its own state on an authenticated GET /metrics (see Operations); it needs no Prometheus to run. The same probe client works with VictoriaMetrics, Mimir and Thanos.

Editing a probe in the console

Each backend row has a Probe button that opens a drawer: type, URL, method, expected status, JSON path, secret header names, timeout, and up to five endpoints tried in order. Saving sends the block to POST /api/v1/probes/validate first, so the drawer refuses exactly what the runtime would refuse, with the runtime's own message.

A backend declares a probe or a healthcheck_url, never both. When the backend uses a health URL, the drawer says that saving a probe removes it.

Probe templates

A template is a probe definition kept in the console's database so several backends can follow it. It is editorial only: the DNS reads the expanded probe: blocks in gslb.yaml and never the database.

Layer Where Losing it means
Operational gslb.yaml (probe: blocks) nothing: that file is what the DNS serves
Editorial SQLite (probetemplate, probetemplatelink) you can no longer propagate an edit; every backend keeps its probe

The Probes page lists the library and edits it; the backends keep their own probe: block in the YAML. Nothing about a template is written to the YAML unless you apply it: editing a template that backends follow previews the diff and the number of backends it would rewrite, then applies it through the normal save path. Deleting a template is refused while a backend still follows it.

States

The API computes the state of every link from the running configuration, on each read, and never stores it — a stored verdict would go stale after a hand edit, a restore, or a SIGHUP.

  • linked — the backend's probe: block means exactly the template's, with the runtime defaults applied (method: GET, expect.status: [200], comparison: lt, a tcp host defaulting to the backend address).
  • diverged — the backend was edited since, or carries no probe at all.
  • orphaned — the backend named by the link no longer exists: renamed by hand in the file, or its pool was removed. The link is kept and shown as a warning.

A rename made in the form (zone, pool or backend) rewrites the links. A rename made by hand in the YAML leaves them orphaned, which is visible and repairable.

Attach, detach, delete

  • Attach a backend copies the definition into the form and creates the link. Save the configuration to make it true; until then the link reads diverged, which is what it is.
  • Detach removes the link only: the backend keeps its probe.
  • Deleting a template that still has links is refused, with the count. The operator detaches first; a definition deleted under its links would leave nothing to reapply.

Editing a template does not rewrite the configuration by itself: the linked backends keep the probe they have until it is applied to them.

Editing a template

An edit is a draft: nothing reaches the library or the file until the operator decides.

  1. Preview impact compares the draft with the configuration and reports the backends that follow the template (their probe equals the stored definition), the ones that diverge, the orphaned links, and the YAML diff. Nothing is written.
  2. Apply to N backends writes the draft to every follower through the ordinary save path — validation, atomic write, acknowledgement, rollback — and only then updates the library. A refused reload leaves the library on the old definition, which is exactly what the backends still carry.
  3. Save without applying is offered only when no backend follows the template: saving the new definition then changes nothing that is deployed. With followers, the old definition is what makes them followers; losing it would leave nothing to propagate.

A backend whose probe was edited by hand reads diverged and is never overwritten by an apply. Reapply, on its link, is the explicit way to copy the stored definition back onto that one backend.

A template application is recorded in History with the trigger template, along with the account that asked for it.

Export and import

Export writes the templates as JSON, without links: a link names a (zone, pool, backend) that exists in one configuration only. Import is a merge by name, so importing the same file twice changes nothing.

Guard rails

Method whitelist (GET, POST, HEAD), no body on GET/HEAD, 4 KB body cap, at most 16 headers and 5 endpoints, secrets only through *_env references.

Proactive checks

proactive:
  type: prometheus
  url: http://prometheus.internal:9090
  query: node_load1{instance="192.0.2.10:9100"}
  threshold: 8
  comparison: gt
remove_when_degraded: false

Indicators are evaluated only while the primary probe answers. Every indicator must agree before the backend is flagged degraded, and an inconclusive one vetoes the flag. degraded is not DOWN: no DOWN event, no SERVFAIL. With remove_when_degraded: false (the default) it keeps answering and the console shows a badge; with true it leaves the rotation until its indicators calm down.

Maintenance

maintenance: true

The backend stays configured, keeps being probed and is shown in the console, but it is never answered and produces no DOWN event. Clearing the flag restores it at the next reload with the health state it had meanwhile.