Observability

Every GantryCD binary exposes the same observability surface so you can wire it into whatever you already run.

Health probes

Point your load balancer and Kubernetes probes at these.

Metrics

The backend serves Prometheus metrics on METRICS_PORT (default :9090); runner groups serve them on their own METRICS_PORT when set. Metrics carry consistent snake_case labels (org_id, run_id, …) so they line up across dashboards.

Useful signals to alert on:

• gantrycd_run_deadline_cancellations — runs hitting the hard duration cap. A spike means RUN_TTL may be too low for your workloads.

• gantrycd_sso_oidc_discovery_requests{outcome} and gantrycd_sso_oidc_discovery_duration_seconds — OIDC issuer health.

• Runner-group connection status and dispatch failures.

• gantrycd_async_tasks_pending{task_type} and gantrycd_async_tasks_running{task_type} — unfinished work, and the part of it that is in flight, on the shared async-task queue (dependency-graph reconcile, dependent-trigger resolution, prefix delete, …). Every replica reports the same fleet-wide number, so aggregate with max() across replicas, never sum() — summing a global count multiplies it by the replica count. (gantrycd_async_task_slots is the one exception, and the reason the alert below sum()s it: that gauge is genuinely per-replica, so summing is what produces fleet capacity.) Running is a SUBSET of pending, not its complement — a task under a live lease is both queued work and running work. Pending’s complement is gantrycd_async_tasks_dead. Read them together: a climbing backlog with the slots busy is a capacity problem (add replicas, or raise GANTRYCD_TASK_PROCESSOR_CONCURRENCY); a climbing backlog with nothing running is a stuck queue.

• gantrycd_async_tasks_dead{task_type} — attempts exhausted AND not currently leased. The second half matters: the claim is what increments the attempt count, so a task on its final attempt sits at the threshold for exactly as long as a worker is running it, and counting it as dead would report it as abandoned at the moment it was most alive.

  Dead letters are pruned after GANTRYCD_ASYNC_TASK_RETENTION (7 days) by the hourly async-task-cleanup job — nothing else ever removes one. That window is the clock you are racing when this gauge goes non-zero: after it, the row’s last_error and payload are gone and the post-mortem has to come from the logs.

  Both scrapes are index-served. Measured at 200k dead rows on an all-visible table: the dead count is a parallel index-only scan (~170 buffers, zero heap fetches, via idx_async_tasks_dead), the pending count a BitmapOr over two partial indexes (~105 buffers). The retention prune dirties the visibility map, and until autovacuum catches up the planner drops back to a sequential scan — that is the sweep paying for itself, once an hour.

• gantrycd_async_task_oldest_due_seconds — how long the longest-waiting claimable task has been due. Unlabeled — one fleet-wide number on purpose.

  On its own it is not a starvation signal. A whole-fleet sweep (dependency-graph-rebuild, resource-index-rebuild) stamps every org’s task with the same run_at, so this gauge reads “time since the sweep started” until the last org drains — at ten thousand orgs that is legitimately minutes. That is healthy work in progress.

• gantrycd_async_task_slots — worker slots on THIS replica; sum() across replicas is fleet capacity. The one async-task gauge that is per-replica rather than a global count, and the denominator the starvation alert needs.

• gantrycd_async_task_lease_lost_total{task_type,reason} — leases that ended without their holder completing the task. expired_reclaimed means a holder was killed mid-handler (or ran past its ceiling without honouring its context) and another worker took the row over; fenced means a worker’s own write about its task was refused because the row had moved on. Either is a duplicate-execution risk the fencing then refused. Read the reasons apart: mostly expired_reclaimed points at workers dying or overrunning their ceilings, mostly fenced at tasks being cancelled or reclaimed out from under running handlers. See Async Task Queue.

  On a build from before PR #284, a claim could lease its whole candidate window and then run only one of those rows; the rest sat leased with no worker until their leases expired, each burning an attempt; only the row each later claim returned was reported as expired_reclaimed, so the counter undercounted them. Those rows counted as running, so the starvation alert’s capacity clause below was suppressed while nothing was being done. expired_reclaimed is recorded only when a later claim takes over an expired lease, so any cause — a parked row, a worker death, a handler that overran its ceiling without honouring cancellation — surfaces when the existing lease expires, up to one full lease after the event, and the timing does not tell them apart. The deployed build (pre-#284 or not), restarts, OOM kills, rollouts, and timeout outcomes are supporting evidence for which cause it was, not proof.

  Delete this note once no fleet you operate runs a pre-#284 build.

• gantrycd_deployments_unnotified_pending{status} — finished deployments whose notifications have not been handed to the outbox yet. A sweep drains this every minute, so healthy is at or near zero. A floor that never drains means some deployments can never be enqueued; because the sweep takes the oldest first and has no age limit, those stuck ones eventually fill every batch and newer deployments stop being announced at all. Nothing else reports this — once a deployment is enqueued it shows up as a delivery, but one that never gets that far is invisible.

• gantrycd_org_actions_total{org,kind,outcome} — API actions on org-scoped routes, split read/write and ok / client_error (4xx) / server_error (5xx). This is the only per-tenant error signal in metrics. gantrycd_http_* deliberately carries no org label (route × status is already wide), and WriteError logs 400 and 404 at Debug — so a collector running at info never sees a validation error or a 404 at all. Without this label “which customer is seeing errors” is answerable only by reading the database. Cardinality is bounded at org × 2 × 3.

  A per-org error ratio:

  sum by (org) (rate(gantrycd_org_actions_total{outcome="server_error"}[5m]))
    / sum by (org) (rate(gantrycd_org_actions_total[5m]))

• gantrycd_org_runs_per_hour_limit{org} — the org’s PURCHASED runs/hour budget, the denominator that turns gantrycd_org_run_dispatch_total{outcome} from a raw count into “how close is this customer to what they paid for”. It is the configured limit, not the live token balance: the balance lives in Redis and reading it per scrape would mean one Peek per org on the metrics path, while consumption over the trailing hour is already derivable from the counter. Refreshed by the business-inventory loop, so it carries the same 1-minute staleness budget as every other per-org gauge.

  sum by (org) (increase(gantrycd_org_run_dispatch_total{outcome="allowed"}[1h]))
    / max by (org) (gantrycd_org_runs_per_hour_limit)

• gantrycd_build_info{version} — always 1, one series per running build. Exists so a step change in any other series can be attributed to a rollout instead of guessed at; nothing else on /metrics names the build.

• gantrycd_runner_group_max_concurrent — slots a runner group is configured to run concurrently, the denominator for gantrycd_runner_group_active_runners. Occupied slots alone cannot distinguish a saturated group from an idle one. Emitted by the runner-group binary (so, like its siblings, only when METRICS_PORT is set), and its series carry otel_scope_* labels the backend’s do not.

• gantrycd_dependency_usage{org,budget} over gantrycd_dependency_budget{budget} — how close each org is to every cross-stack dependency cap, and gantrycd_dependency_budget_hits_total{org,budget} for the times one actually refused work (dependents that stopped auto-deploying, a dependency graph that degraded). The caps, what a user sees when each trips, and copyable alert expressions are in Cross-Stack Dependency Graph.

• gantrycd_explore_sweep_check_failures_total{org,check} — Insights checks (cycles / orphans / duplicates) the diagnostics sweep could not compute. The sweep publishes its findings row regardless, with the checks that did run and a fresh timestamp, so nothing else distinguishes a dimension that has quietly stopped being measured from one that is clean — this counter is the signal. A steady rate on duplicates for one org usually means its two org-wide index scans are timing out; check the autovacuum settings on resource_index / dependency_edges (docs/reference/resource_explorer_scaling.md).

• gantrycd_rebuild_stack_failures_total{org,task_type} — stacks a dependency graph or resource index rebuild could not process and deliberately skipped. Interrupted stacks are retried and are not counted. The rebuild still reports success, so a rebuild that processed almost none of an org is invisible without this counter. Alert on a sustained rate for one org and rebuild type (sum by (org, task_type) (increase(...[1h])) > 50), not on the odd stack.

• gantrycd_async_task_processed_total{task_type,outcome} and gantrycd_async_task_duration_seconds{task_type} — throughput, failure rate, and per-task latency of that queue. Pair the failure rate with the backlog gauge: a rising backlog with outcome="failure" points at the downstream store (Postgres / S3), not the slot count. outcome is success (the task is FINISHED and its row is gone), continued (a sliced handler committed one slice’s progress and handed the row back — it counts slices, which is what lets success be read as “tasks completed”), failure, timeout (the handler blew its registered ceiling — either it returned the deadline as an error, or it advanced a slice and was cut off, which for a sliced handler means a SINGLE UNIT of work — or one of its reads — outlasts the whole budget), cancelled (a rollout interrupted it with nothing saved and the lease was handed straight back; one that saved progress first is continued), or lost (the fence refused this worker’s settlement because the row had already moved on; whoever owns it now will run the task). Alert on failure|timeout only: continued is ordinary slicing, cancelled is expected on every deploy, and lost on every takeover. The dashboard’s failure-ratio denominator excludes those three, and its throughput panel counts success alone.

  One wrinkle: timeout also covers a sliced handler that made progress and was cut off by its ceiling. That is deliberate but is not an error, so a large org’s rebuild can contribute a low steady rate; read it with the Warn the processor logs for that case.

  A handler that FAILED at its ceiling is only recorded as timeout if it returned an error wrapping DeadlineExceeded; one that swallowed the cancellation and reported DONE is recorded as a success, because nothing else can tell the difference. One that swallowed it and returned CONTINUE is a timeout, then released rather than backed off — so it dead-letters only if it also failed to advance.

Three async-queue alert expressions worth copying:

Production alerts are Grafana-provisioned from the infra repo’s charts/grafana/values.yaml (alertingProvision.groups), so these three rules (AsyncTaskQueueStarving, AsyncTaskLeasesLost, AsyncTasksDeadLettered) have to be added there.

# Starvation: work has been due for five minutes AND the fleet is not fully busy.
#
# The capacity clause cuts both ways. Without it, every whole-fleet sweep trips the
# alert while draining normally — one sweep gives thousands of tasks the same
# run_at. But testing "nothing is running" instead would MISS the case this exists
# for: an orphaned lease counts as running for up to (ceiling + margin), an hour
# for prefix_delete, so one crashed worker would suppress the alert for an hour.
# One busy slot out of twelve is the shape to catch, and that needs a denominator.
#
# `or vector(0)` is load-bearing, not defensive. gantrycd_async_tasks_running is
# a GROUP BY, so with nothing running it exports no series and sum() is an empty
# vector — and `x and <empty>` is empty.
max(gantrycd_async_task_oldest_due_seconds) > 300
  and (sum(max by (task_type) (gantrycd_async_tasks_running)) or vector(0))
        < sum(gantrycd_async_task_slots)

# Leases are being lost faster than the odd crash explains. Every one of these is
# work that had to be re-run. expired_reclaimed means holders are dying
# mid-handler (check restarts and OOM kills); fenced means workers are finishing
# tasks whose rows had already moved on.
sum by (reason) (increase(gantrycd_async_task_lease_lost_total[15m])) > 3

# Tasks have given up. Nothing retries these, and retention deletes them after
# GANTRYCD_ASYNC_TASK_RETENTION (7 d) — that window is the clock you are racing.
# max by, not sum by: every replica reports the same global count, so summing
# multiplies it by the replica count and disagrees with the runbook query below.
max by (task_type) (gantrycd_async_tasks_dead) > 0

When it fires, the rows themselves are the runbook:

SELECT task_type, resource_org_id, resource_id, attempt_count, last_error, last_attempted_at
FROM async_tasks
WHERE attempt_count >= 20
  AND (lease_expires_at IS NULL OR lease_expires_at <= NOW())
ORDER BY last_attempted_at DESC;

last_error is the last failure recorded for each — the same message logged with its task id when it happened, so a task pruned past retention is still traceable in the logs. A per-org maintenance type (graph rebuild, index rebuild, diagnostics, seat-cap audit) needs no requeue: its next sweep enqueues a replacement, because a dead letter never suppresses one.

Tracing

Tracing is disabled unless OTEL_EXPORTER_OTLP_ENDPOINT or OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is set. The general endpoint is a collector base URL such as http://collector:4318; the traces endpoint includes the full path, such as http://collector:4318/v1/traces.

The SDK samples every trace by default. Keep that default when a collector makes the sampling decision. When exporting directly to a backend without tail sampling, limit trace volume at the application instead:

OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1

Other standard OTEL_* settings, including resource attributes and exporter headers, are honored by the SDK directly.

Logs

Logs are structured (slog). They go to stdout for your collector to pick up. Keep the snake_case attribute convention if you build queries on them — those keys double as metric labels and dashboard filters.