Runner Health Checking

Runner liveness is tracked at two levels:

• Runner-group liveness via heartbeat.
• Ephemeral-runner liveness via last_seen_at updates during ready / status / done calls.

Runner Group Heartbeat

POST /api/v1/runner-groups/heartbeat

The heartbeat loop is independent from poll. Heartbeat cadence is client-driven; the backend uses a 30-second connection-staleness threshold (connectionThreshold in pkg/domain/runner_group.go) to decide whether a group is currently connected. A group that hasn’t heartbeated within that window shows as disconnected, but is not deleted — heartbeat is a liveness signal, not a cleanup trigger.

Reported version

Every request a runner group makes carries its own gantrycd version in the X-GantryCD-Version header (contracts.VersionHeader). The heartbeat is where the backend records it, into runner_groups.last_poll_version — it rides the same UPDATE that stamps last_poll_at, so the version costs no extra write. A group that sends no header clears the column rather than leaving a stale value standing.

The API exposes it as version on the runner group, and gantrycli runner-groups list prints it as a VERSION column. Two cases read differently:

• Pull-based groups (self-hosted) report their own build — this is how you spot a group left behind on an old release.
• Backend-dispatched groups (github-actions) never poll, so they never report anything; the backend fills in its own version, because their dispatcher runs inside the backend process and is therefore always exactly that build.

A group that has never checked in reports null — it is not assumed to be running any particular version.

A tagged backend refuses an unversioned group

GET /api/v1/runner-groups/poll returns 403 when the backend is a tagged release and the polling group reports either dev or no version at all. Production is therefore never served by a build nobody can identify, and it takes no configuration to get that — the rule falls out of the two internal/version.Version stamps.

The rule is one-directional on purpose:

A dev backend serves anything, so local development is unaffected and a released group can still be pointed at a dev backend. A symmetric “the versions must match” rule would reject that normal case while protecting nobody.

The check lives on the poll handler, not in PollForWork. The built-in github-actions dispatcher claims work through that same service call without an HTTP request — it is the backend process, so it can never be unversioned. Putting the check in the service would strand every github-actions group on a released backend.

A refused group is reported with its own connection status, unversioned, everywhere a group’s status is shown — the API, gantrycli runner-groups list, and the UI. It is neither connected (it is being served nothing) nor disconnected (it is reachable and heartbeating), and the read path derives it from the same predicate the poll endpoint enforces, so the status cannot disagree with the behaviour. Backend-dispatched groups are never unversioned: they do not poll, and their version is the backend’s own.

Groups older than v0.0.2 predate the header and are refused for the same reason, which makes this an upgrade-ordering constraint: see Runner groups.

Two things this deliberately does not do. It does not compare release versions against each other: a v0.1.0 group polling a v9.0.0 backend is served, and the drift is something you see in the UI and act on, not something the backend breaks. And because the header is self-reported it stops accidents, not a determined operator — who already holds the ephemeral JWT and receives plaintext credentials at accept.

Ephemeral Runner Liveness

Ephemeral runners refresh last_seen_at through normal lifecycle calls (ready, status polling during a run, done). GetStatus throttles the refresh by lastSeenRefreshInterval = 10s (in internal/backend/services/runner_service.go) so quiet runs don’t appear stale and don’t hammer the primary DB.

Stale-Runner Cleanup

RunnerCleanupJob (internal/backend/jobs/runner_cleanup.go) force-fails runs whose runner accepted them (running) but then went silent — last_seen_at older than RUNNER_EXECUTION_TIMEOUT (default 1h) — and deletes the stale runner rows. Active runs are failed first so stacks aren’t blocked; the FK ON DELETE SET NULL on runs.assigned_runner_id handles the pointer cleanup automatically. It deliberately does not fail not-ready stubs or still-assigned (never-accepted) runs — those belong to the assignment-expiry and startup-deadline reapers below — but it does garbage-collect orphaned not-ready stubs by created_at (stubOrphanThreshold = 2 minutes) without recording a dispatch failure.

To short-circuit the long execution timeout for a lost runner, an operator can force-delete it (ForceDeleteRunner), which fails the active run and promotes the queue immediately. The runners UI surfaces this once a runner is “Offline” (last_seen_at older than 5 minutes).

Assignment Expiry And Startup Deadline

Poll only makes a tentative assignment. Two reapers cover the “never started” window, keyed off started_at staying NULL:

• Per-attempt re-queue — if a run is assigned but the ephemeral runner never accepts it before assigned_expires_at (RUNNER_ASSIGNMENT_TIMEOUT, default 2m), cleanup resets the run to pending so a fresh runner re-polls it. This is the main protection against a lost worker after poll but before accept; it can cycle many times.
• Overall startup deadline — a run that became eligible more than RUNNER_STARTUP_DEADLINE ago (default 2d, measured from first_pending_at, which is preserved across re-queue cycles) but was never accepted is failed as startup_timeout and its stack lane released.

Operational Meaning

Runner health is part of queue correctness, not just UI status: stale-runner cleanup frees blocked stacks, and expired-assignment cleanup returns work to the scheduler.

See also:

• runner_polling.md
• deployment_state_machine.md