Runner Polling

Work assignment is a two-phase protocol:

1. runner group poll claims work tentatively
2. ephemeral runner accept makes execution authoritative

This minimizes wasted credential generation and provides a clean recovery path when a poll succeeds but the worker never starts.

For github-actions runner groups the same protocol applies, except the “poll” is driven by an in-process dispatcher on the backend (not an external long-poll) and the “ephemeral runner” is a workflow run started via workflow_dispatch. See runner_groups_github_actions.md for the full flow.

Actors

• runner group coordinator: long-lived process
• ephemeral runner: short-lived per-run worker
• backend: source of truth for assignments, liveness, and credentials

Phase 1: Poll

Endpoint:

GET /api/v1/runner-groups/poll

High-level flow:

1. authenticate runner group JWT
2. find the oldest pending run whose runner selector is contained in the polling group’s labels (an empty selector falls back to the org’s default runner selector)
3. create a stub ephemeral-runner record
4. atomically assign the run to that runner
5. set assignment expiry
6. return run_id and runner_id

If no work matches, the endpoint returns no assignment.

Per-org runs/hour throttle

After a run is claimed (step 4, SetExclusiveAssignment won the SKIP-LOCKED race) but before the tx commits, the poll consumes one token from the org’s runs/hour dispatch budget — a shared Redis token bucket keyed on the org (PolicyOrgRunsPerHour, refill = the org’s purchased organizations.runs_per_hour_limit, burst = that rate × 24h). Consuming after the claim (not before) means groups that lost the race never spend a token, so the budget reflects runs actually dispatched, not polls. When the bucket is empty the claim is rolled back (run returns to pending, no stub runner persists) and PollForWork returns the sentinel ErrOrgDispatchThrottled:

• the HTTP poll handler maps it to a 204 (the self-hosted poller retries next tick), so a throttle is transparent to the runner-group client;
• the backend dispatcher (dispatchOne) treats it as not contended (unlike a genuine SKIP-LOCKED nil), so DrainPending stops instead of hot-looping every 10 ms until the bucket refills.

On a throttle the org is also deferred: organizations.dispatch_throttled_until is stamped to the bucket’s expected refill, and the backend-dispatch candidate query (ListDispatchCandidates) skips the whole org until then. Without this a throttled run — which rolls back to pending and frees no group capacity — would re-appear as a candidate every tick, wasting claim/rollback work and starving other orgs out of the bounded candidate batch. The stamp is a best-effort scheduling hint (a separate tx off the throttle path); the Redis bucket stays authoritative, so a stamp that is merely old only means the org is retried a little late — it decays against NOW().

A stamp that outlives its policy, though, is not benign, because the stamp encodes the refill of the rate in force when it was written. An org throttled at 1/h is deferred a full hour; raise it to 50/h — the operator’s usual response to a throttled backlog — and the bucket takes the new rate at once while the stamp goes on excluding the org from dispatch for the rest of that hour. The budget then reads as having room while the runs sit in the queue. So SetRunsPerHourLimit clears dispatch_throttled_until in the same write: a rate change invalidates the projection. Dropping it is safe — if the bucket is genuinely empty the next poll re-throttles and re-stamps at the new rate.

Cross-org fairness. The candidate query also skips any org already at its per-org in-flight cap (GANTRYCD_DISPATCH_MAX_PER_ORG active runners of the type). Together with the throttle defer this bounds each org’s dispatch to its cap, so a heavy org dispatches its cap in one tick then yields to peers — “heavy org waits, others don’t” — without a per-tick scan of the whole pending backlog. Two metrics expose when the dispatcher itself (not an org’s budget) is the bottleneck: gantrycd_dispatch_batch_saturated (full-batch ticks) and gantrycd_dispatch_oldest_dispatchable_age_seconds (dispatch-latency SLO). Both are recorded after the tick, over only the candidates it could actually serve — orgs the throttle rejected during the tick are dropped — so they signal “add dispatcher capacity,” not a purchased-budget throttle.

Excluding merely deferred orgs (the candidate query’s job) is not enough, and assuming it was is how a healthy dispatcher ended up paging: the deferral lapses the instant the bucket yields its next token, so for that one tick the org’s entire aged backlog is a candidate again — one run dispatches, the org re-defers, and the tick meanwhile reported the age of a run that is old because of the org’s purchased budget. With one sample per tick that is enough to pin the p95 latency alarm on for as long as the backlog exists. A throttled org’s queue depth belongs to gantrycd_org_run_dispatch_total{outcome="limited"}, which is the signal that actually means “this org needs a bigger budget.”

This meters how fast queued runs are handed out to an org’s runner groups; it does not gate run creation. The limiter fails open (a Redis error dispatches the run) and is a no-op when rate limiting is disabled instance-wide. Super-admins set the per-org rate from Backstage; a GET /api/v1/orgs/{org_id}/dispatch-budget read (surfaced as a banner on the org Queue page) lets org admins see when they are throttled and the time to the next slot. Default: new orgs start at 2/h (the column default in schema/tbl_organizations.hcl). The column is never grandfathered or backfilled, so changing that default moves only orgs created after the schema apply — orgs already in the DB keep whatever value they hold, including the 5/h they were created with before the default dropped. Operators raise per-org limits from Backstage as orgs are onboarded.

Selector Matching

Labels are key=value pairs, modelled on Kubernetes node selectors. Compatibility is JSONB containment:

• A stack carries a namespaced labels map. Keys under the reserved prefix gantrycd:runner-group: (prefix stripped) form the runner selector, snapshotted immutably onto each run as runs.runner_selector at creation. (gantrycd: is a reserved system prefix; other keys are free user labels with no matching role.)
• A runner group advertises plain capability labels in runner_groups.labels.
• Match rule: run.runner_selector <@ group.labels (every pair in the selector is present in the group’s labels). Runners carry no labels of their own.
• A run with an empty runner selector is served by the org’s optional organizations.default_runner_group_id (chosen explicitly, not label-matched). When no default is set, an empty-selector run may run on any group.

Matching is enforced in SQL, not by best-effort filtering in Go.

Stub Runner Creation

Poll creates a minimal runner record before the ephemeral process exists. This satisfies assignment foreign keys and gives the backend a concrete runner identity for the follow-up lifecycle.

The stub starts in not-ready.

Ready

Before accepting the run context, the ephemeral runner calls:

POST /api/v1/runners/{runner_id}/ready

This transitions the stub to a real ready runner and finalizes DB-backed runner metadata.

Phase 2: Accept

Endpoint:

POST /api/v1/runs/{run_id}/accept

Accept does the expensive work:

• verifies the run is still assigned to this runner
• checks assignment expiry
• generates backend/log/SCM/runtime credentials
• transitions run assigned → running
• transitions deployment into active execution state
• returns full RunContext

Status Polling During Execution

The ephemeral runner polls:

GET /api/v1/runs/{run_id}/status

Purposes:

• refresh liveness
• detect requested_cancellation

If cancellation is requested, the runner cancels its local context and later acknowledges cancellation explicitly.

Live Log Push During Execution

Alongside the status poll, the runner’s log uploader ticks every 2 seconds and pushes a snapshot of the active (unsealed) log chunk:

POST /api/v1/runs/{run_id}/logs/live

Body (contracts.LiveLogPushRequest): the active chunk’s index, the count of chunks durably uploaded to S3 (sealed_chunks, advanced only after a successful PUT — it never claims a chunk S3 doesn’t have), and the gzipped chunk bytes. The backend stores one record per run in Redis, fully replaced on every push; the web log APIs answer active-run metadata and active-chunk reads from it without touching S3.

Properties:

• Best-effort: push failures are logged on the runner and never fail the run; the endpoint always returns 204 for a well-formed push (without Redis the record is dropped — logs then appear chunk-by-chunk as they seal).
• Deduped: a tick where the chunk and sealed count are unchanged costs no HTTP call.
• Same auth as /status and /done: ephemeral runner JWT plus the path↔JWT run binding check.
• Lifecycle: /done deletes the record (the final flush made S3 authoritative); for runs whose runner went silent, CleanupInactiveRunners flushes the last snapshot to S3 as the final chunk before the record’s TTL (sized above RUNNER_EXECUTION_TIMEOUT) reaps it.

Sealed chunks never travel this path — they are PUT to S3 exactly once by the runner; see run_context_and_artifacts.md for the chunk storage contract.

Completion And Cleanup

After execution the runner calls:

• POST /api/v1/runs/{run_id}/done
• DELETE /api/v1/runners/{runner_id}/deregister

If the runner sees cancellation first, it also calls:

• POST /api/v1/runs/{run_id}/acknowledging-cancel

Which Run A Runner Holds

The binding between a runner and its run lives in exactly one column, runs.assigned_runner_id. A runner row carries no pointer back: the assigned_run_id / assigned_stack_id / assigned_deployment_id fields on the runner API type are derived per read, by joining the run whose assigned_runner_id is this runner and whose status is not terminal.

A runner holds a run when it is executing it (running, requested_cancellation, cancelling) or was dispatched for it and could still accept (assigned with an unexpired lease). That is narrower than “the run is not terminal”, and the difference is load-bearing: a lapsed lease returns the run to pending while keeping assigned_runner_id (see One Runner Per Run), so a non-terminal run does not imply a live claim.

deregister refuses only while the runner still holds a run. A runner whose run was skipped under it, or whose lease lapsed while it was booting, can always remove itself. That matters for more than tidiness: a runner that cannot deregister keeps its group’s concurrency slot and holds its abandoned run out of dispatch until a reaper removes it, up to RUNNER_EXECUTION_TIMEOUT later.

It also means a terminal run releases its runner everywhere at once, with no code path having to remember to clear anything. The copy that used to live on the runner row was cleared only on the /done path, so every abnormal termination left it dangling.

Failure Windows

The recovery path splits cleanly on one signal: runs.started_at. It is NULL until /accept succeeds, so a run that was assigned but never accepted (the worker never actually started) is handled differently from a run that accepted and was executing when its runner was lost. Three reapers, each keyed off a different timestamp, run on the 1-minute scheduler:

Reaping the runner row, not just the run

DeleteInactive sweeps runner rows on the same two-phase split, keyed on whether the runner ever accepted:

Neither startup state heartbeats, so last_seen_at cannot age them. ready used to sit on the execution cutoff, and that was the bug: a worker that signalled ready and then died — a lost /ready response, a node that vanished in the ready-to-accept gap — kept its row for a full hour. Because the one-runner-per-run guard reads a surviving row as a live claim, the run that assignment expiry had just requeued stayed undispatchable for that hour, and the group was charged a concurrency slot for a process that no longer existed. At max_concurrent_runs = 1 that stalls the whole group.

/ready is also idempotent on ready. It answers with an empty 204, so a lost response is indistinguishable to the runner from one that never committed, and it retries for 30 seconds. A strict not-ready → ready edge failed every replay, and the runner exited before installing its deregistration defer — creating the stranded row above. busy still conflicts: a runner that already accepted is not replaying.

When the accept response never arrives

/accept is authoritative and its transaction commits before the RunContext reaches the socket. If that response is reset, truncated, or lost to a backend exit in between, the run is running with the runner holding nothing it can execute — and it cannot be handed the same context again, because the context is not persisted, the API token exists only as a hash under a one-token-per-run constraint, and the credentials inside it were response secrets.

The runner’s retry is therefore the signal. A second /accept from the same runner for a run already running can only mean the first response was not delivered, so the backend fails that run immediately — the same terminal transition CleanupInactiveRunners would perform an hour later, on evidence instead of on silence — and returns 409 with code: "accept_response_undeliverable". The stack’s lane is released with it and the deployment can be retried.

This does not race a live execution: the runner only retries when it could not decode a context, and it discards the response it could not decode. The code matters because the client must distinguish this from the other 409s: a reassignment, an expired lease, or a run already terminal (next section).

The last one is a backstop, not a routine window: it is the only reaper that can see a run whose status has left pending/assigned without ever reaching running. RequestCancellation no longer creates that state (it terminates a never-started run outright — see Cancellation), but a run already in it matches none of the reapers above, its not-ready stub is excluded from CleanupInactiveRunners, and PromoteNextOnStack treats requested_cancellation as a live lock holder — so its stack’s lane would never release. Gating on a lapsed assignment lease is what keeps it from racing an /accept still in flight.

first_pending_at is stamped the first time a run becomes pending and is preserved across re-queue cycles, so the startup deadline is measured from the run’s original eligibility, not reset on every retry.

When the run is already terminal at accept

An ephemeral runner takes tens of seconds to boot (a GitHub Actions dispatch about a minute), and in that window the deployment can be skipped or the run cancelled. The runner then reaches /accept for a run that is discarded or cancelled. That is the outcome of someone’s decision, not a fault, so it is kept apart from every other rejection: the backend returns 409 with code: "run_already_terminal", logs it at INFO where every other conflict is a WARN, leaves the span unmarked, and the runner deregisters and exits 0.

Only those two statuses are benign. A run that is failure (a startup failure committed by an earlier accept whose response was lost) or startup_timeout (the reapers above gave up on this runner) is terminal too, but a fault: it gets an uncoded 409 naming the status, stays a warning, and the runner still exits 1.

Status is what decides, not the assignment columns. SkipDeployment leaves assigned_runner_id in place, a stub that deregisters nulls it through the FK, and ResetExpiredAssignments clears only the lease and keeps the runner id. So the remaining 409s key on status too, and each names its cause: pending means the lease expired and was reset; a different assigned_runner_id means the run was reassigned; an assigned run past assigned_expires_at means the reaper has not run yet. All of them are a runner that arrived too late, and both sides keep them a warning.

One Runner Per Run

When an assignment lapses, ResetExpiredAssignments clears the lease (assigned_at, assigned_expires_at) but deliberately keeps assigned_runner_id. The dispatched runner may still be cold-booting toward /accept, so both dispatch paths exclude a run whose assigned_runner_id names a live runner — ListDispatchCandidates for backend-dispatched groups and PollPendingRun for self-hosted long-polling ones. Without that, the run becomes servable again the instant its lease lapses and a second runner is sent, then a third — a herd all racing to accept a run only one of them can have.

The two predicates are written separately but must stay in step; a guard on only one path leaves the invariant half-held, which is the state this replaced.

The pointer is released by the ON DELETE SET NULL FK when the runner row goes away — either because the runner deregistered (the common case, and immediate) or because a reaper removed it. Nothing has to remember to clear it, and no reaper can forget. Note the reaper path is not prompt: a runner that reached ready is past the short stub cutoff and waits on RUNNER_EXECUTION_TIMEOUT (1h), so a runner that exits without deregistering holds its run out of dispatch for that long.

A non-null assigned_runner_id therefore does not mean a live assignment. Every reader gates on run status as well — /accept additionally requires status = 'assigned' and an unexpired lease, so a superseded runner cannot accept.

Important recovery cases:

• poll succeeded, the runner group cannot spin up the runner (github-actions workflow_dispatch rejected, self-hosted worker spawn failed, token signing failed):
  • the group reports it — synchronously inside the github-actions dispatcher, or via POST /api/v1/runner-groups/dispatch-failed for self-hosted groups
  • RunnerGroupService.ReportDispatchFailure fails the claimed run and its deployment, calls PromoteNextOnStack, deletes the stub runner, and records last_dispatch_error on the group (surfaced in the UI). This is a definitive failure the group knows about, so the run is failed, not re-queued — re-dispatching it would just fail again in a loop.
• poll succeeded, the runner is dispatched but silently never checks in:
  • CleanupExpiredAssignments resets the run to pending so another runner re-polls it (the orphaned not-ready stub is later garbage-collected by CleanupInactiveRunners, with no dispatch failure recorded — a vanished stub usually means a successful re-queue). If no runner ever starts the run within RUNNER_STARTUP_DEADLINE, CleanupStartupDeadline fails it as startup_timeout.
• runner accepts, then crashes mid-run:
  • CleanupInactiveRunners fails the run once the runner is silent past RUNNER_EXECUTION_TIMEOUT and releases the stack lane. To short-circuit the wait, an operator can force-delete the lost runner (RunnerGroupService.ForceDeleteRunner, eligible once the runner has been offline past the force gate), which fails the active run and promotes the queue immediately.
• runner group crashes after launching workers:
  • local launcher can rediscover running workers on restart

Launcher Notes

Launcher variants are separate from the protocol:

• local launcher starts OS processes
• Docker launcher starts containers
• Kubernetes launcher creates Pods (one per run; ownership tracked via the gantrycd.io/group-id + gantrycd.io/run-id labels so Discover can re-attach after a restart)
• GitHub launcher dispatches GitHub Actions jobs

All of them converge on the same runner-group and ephemeral-runner API surface.

See also:

• runner_auth.md
• run_context_and_artifacts.md