Deployment State Machine

This document is the canonical reference for deployment, stage, and run progression. Read it before changing queue behavior, cancellation, or runner-completion logic.

The Three State-Carrying Objects

• Deployment is the user-visible orchestration record.
• DeploymentStage is the current workflow step inside the deployment.
• Run is the unit executed by runners.

Stages are created just in time. A plan deployment does not pre-create the apply run; it creates a confirmation stage after a successful plan run, and the apply stage appears only after confirmation.

Status Vocabularies

Deployment Status

Non-terminal:

• queued
• pending
• planning
• applying
• waiting
• requested_cancellation
• cancelling

Terminal:

• finished
• failed
• cancelled
• skipped

Deployment Stage Phase

• waiting — stage pre-created but not yet active (e.g. apply stage created at confirmation time, before the deployment moves to applying).
• running — currently the active stage.
• requested_cancellation, cancelling — propagated from the active run.
• done, failed, cancelled, skipped — terminal.

Deployment Stage Kind

• plan_run
• apply_run
• destroy_run
• refresh_run
• raw_run — a raw-command deployment’s single stage (see “Raw-command deployments”)
• confirmation

plan_run, destroy_run and refresh_run are the plan-style kinds (DeploymentStageKind.IsPlanStyle): each runs tofu plan — with -destroy or -refresh-only for the latter two — under -lock=false and writes a planfile that a later apply_run consumes. A replan does not get a kind of its own: it re-runs the deployment’s own plan-style kind with Targets/Excludes/ Replaces set on the run, and the confirmation that produced it carries the replan decision.

The domain still has room for other stage kinds, but the active workflow package is built around the ones above.

Run Status

Non-terminal:

• queued
• pending
• assigned
• running
• requested_cancellation
• cancelling

Terminal:

• success
• failure
• cancelled
• timed_out
• startup_timeout
• discarded

Promotion requirements (PR plan/promote gates)

A stack may declare SCM-status conditions that gate two PR actions before they ever reach the flows below: running a PR’s read-only preview plan, and promoting that preview into an apply-capable deployment. The plan gate runs in SCMEventService.resolvePRDeploymentID and is re-evaluated by preview confirmation before it creates a run; the promote gate runs in DeploymentService.PromotePullRequest before the write transaction and returns a RequirementsNotMetError (HTTP 409) when unmet. Apply confirmation is not SCM-gated. Status is read on demand from the SCM; see docs/reference/promotion_requirements.md.

PR preview confirmation (stacks.pr_plan_mode)

A pull request’s read-only preview plan is not a passive read. tofu init fetches the modules and provider binaries the PR’s HCL names, and tofu plan then executes those providers — and any data source they expose — with the stack’s runtime credentials. A pull request is the one deployment origin whose code has not been reviewed and merged.

So by default (pr_plan_mode = 'manual') a PR preview is created parked on a confirmation stage, and a principal holding (stack, preview) decides whether it runs:

This is the only place in the system where a confirmation stage PRECEDES its run. Everywhere else a confirmation gates the apply of a plan the user has already read; here it gates the plan itself. Consequences worth knowing:

• The deployment is created with no run row at all — stage 1 is the confirmation (run_id IS NULL), and status is waiting. There is nothing for a runner to poll; the confirmation is what mints the run.
• waiting here blocks nothing: plan_unlocked never takes the stack lane. Contrast a waiting plan deployment, which holds the lane for as long as it waits.
• Confirming (ConfirmActionPlan) advances to pending, not planning — no runner has it yet. From there a confirmed preview is indistinguishable from one that never needed confirming.
• The gate has exactly one exit: confirm. There is deliberately no discard. An unconfirmed preview holds no lane, has no run, and burns no runner — so discarding it achieves precisely what ignoring it achieves, while dead-ending the pull request: PromotePullRequest requires a finished preview, and a skipped one can never become finished at that head SHA. A held preview resolves three ways that strand nothing: somebody confirms it, a new commit supersedes it, or the stack’s confirmation_timeout ages it out. Enforcing this takes two guards, not one — the apply confirmation mode check, and a mode check in SkipDeployment, because POST /skip is gated on (stack, confirm) and waiting is otherwise a skippable status. Without the second, the apply gate’s permission resolves the preview gate, which is exactly what the scope split exists to prevent.
• A skipped deployment publishes NO commit status. It looks like an omission and is a decision, and both alternatives were tried on this branch and were security bugs. success fails OPEN — an unapproved preview aging out of the gate goes green, so the winning move against the gate is to ignore it. failure permanently red-Xes a superseded SHA. Publishing nothing leaves the preview’s pending status in place, which correctly blocks a required check when the plan never ran. PR previews own gantrycd/<stack>/preview; lane deployments use gantrycd/<stack>, so promotion cannot overwrite a successful preview with pending. See internal/backend/scm/github/status.go.
• A parked preview has exactly one lifecycle, and every exit is accounted for. It is invisible in the PR view once the pointer moves on, no push reaches it once the PR closes, /skip refuses it, and a stack without a confirmation_timeout never ages it out — so an unretired preview is a standing offer to run a dead or refused pull request’s providers with the stack’s credentials. RetireUnconfirmedPRPreviews is therefore called from EVERY arm of the PR sync, including the closed-PR prune (which otherwise orphaned the preview rather than ending it — the worst hole the feature had). SCM processing is serialized per repository and base branch before fetching provider state, so an older worker cannot commit a stale snapshot after a newer worker. Retirement therefore removes every non-current parked preview, including the newer B deployment in an A -> B -> A force-push. It carries an honest reason — retracted when the stack’s plan requirements refused the PR, pull_request_closed when it closed — because telling the author “superseded by a newer commit” when no newer commit exists is the same misleading status this design closes elsewhere.
• The PR’s pointer is never trusted; GetIDByPROrigin is. That lookup excludes skipped rows and is consulted on every sync, so a dead pointer self-corrects into a fresh preview. Short-circuiting on the PR row’s own latest_deployment_id stranded the pull request the moment that pointer went dead (skip the preview, or clear the org queue, and no replacement was ever minted, while promote — which needs a finished preview — refused forever). It also stops a force-push back to a superseded SHA re-linking that SHA’s dead preview.
• A preview never drains the stack lane, even when it fails. failDeploymentAndCancelQueue skips FailQueuedRunsForStack and PromoteNextOnStack for plan_unlocked. Confirming a preview is the first path that routes a laneless deployment into the startup-failure check, and without the guard it would bulk-fail another deployment’s queued runs.
• The decision is derived inside CreateDeploymentInTx from the stack (domain.PRPreviewRequiresConfirmation), not passed in by callers — so retry re-asks rather than becoming a standing bypass for anyone holding (stack, deploy). Local plans (origin = local) and promoted PRs are excluded by the origin check.

Two routes, two permissions. The gate has its own endpoint precisely so it can carry its own permission — letting an unreviewed branch’s providers run with the stack’s credentials is a different decision from approving an apply of reviewed code:

The routes call separate service operations, and both cross-check the deployment mode so neither permission can reach the other’s gate.

Staleness. Each push to a PR moves its head SHA and mints a fresh preview. The previous head’s unconfirmed preview is auto-skipped (skipped_reason = superseded) in the same write transaction, via SupersedeUnconfirmedPRPreviews — unconditionally, not age-based: the PR has moved past that commit, so there is no age at which confirming it becomes right. A preview that was already confirmed and is running keeps running. And a stack with a confirmation_timeout (QueuePolicy) ages out an unanswered preview confirmation exactly as it ages out an unanswered apply confirmation — see docs/reference/queue_policy.md.

Distinct from promotion requirements above: those are SCM-status conditions deciding whether a preview exists at all; pr_plan_mode decides whether an existing preview runs now or waits for a person. They compose.

Creation Flow

DeploymentService.CreateDeployment is the entrypoint for user-triggered plan, destroy, and refresh deployments.

Inside one write transaction it:

1. loads the stack snapshot
2. creates the first run as queued
3. creates the deployment as queued
4. creates the first stage as running
5. calls RunRepository.PromoteNextOnStack

PromoteNextOnStack is the single queue primitive that:

• releases a stale lock if needed
• acquires the stack lock for the next queued deployment in queue order (prioritized_at NULLS LAST, created_at, id — see “Queue ordering and prioritization” below)
• promotes that deployment queued → pending
• promotes its first run queued → pending

Every write path that may free stack occupancy must end with the same primitive.

Queue ordering and prioritization

Both queue layers — per-stack promotion (PromoteNextOnStack) and runner selection (PollPendingRun, ListDispatchCandidates) — order waiting work by (prioritized_at ASC NULLS LAST, created_at ASC, id ASC). With no operator intervention prioritized_at is NULL everywhere and the order is plain FIFO.

POST .../deployments/{id}/prioritize (gated by (stack, prioritize)) sets deployments.prioritized_at = NOW() (and prioritized_by, the acting principal ID, for audit) and propagates the timestamp to the deployment’s queued, pending, and assigned runs — assigned included so a run whose tentative assignment expires returns to pending with its bump intact; /deprioritize clears both. Semantics:

• Bumped work goes first, FIFO by bump time among bumped peers. Re-clicking prioritize is a no-op that keeps the original bump timestamp — it cannot leapfrog other prioritized deployments (deprioritize-then-prioritize can).
• No preemption. A bump reorders waiting work only: the in-flight deployment on the stack and assigned/running runs are untouched, and the apply-run guard in PromoteNextOnStack still blocks promotion regardless of priority. A bump also cannot conjure runner capacity — it decides who gets the next free slot.
• Follow-up runs inherit the bump. Runs created when the workflow advances (confirm → apply, targeted replan) copy prioritized_at from the deployment row, so prioritizing a waiting or planning deployment pre-bumps its future runs. Re-queued runs (expired assignments) keep their bump.
• Retry does not inherit. A retried deployment is a brand-new queue entry with prioritized_at = NULL.
• A bump only outranks work competing for the same runner groups. This is structural, not a separate check: poll selection is already filtered to runs the polling group can serve, and dispatch candidates pair each run with a group that has room — a bumped run can never take a slot from a group that cannot serve it, and once its matching groups saturate it drops out of the candidate set instead of blocking others. The residual cross-group effects are bounded: at most one dispatch tick of delay at batch-limit boundaries, and per-org budget (MaxPerOrg) consumption order — which is intra-org arbitration, exactly what the permission grants.
• Priority is a strong hint, not a hard guarantee: FOR UPDATE SKIP LOCKED claim races across replicas keep ordering approximate at the margin, exactly as with plain FIFO.

Raw-command deployments

A raw-command deployment (DeploymentMode = raw) runs a free-text bash script on a fresh runner — an operator escape hatch for OpenTofu state surgery and one-off diagnostics. It is created through a dedicated endpoint (POST .../deployments/run-command) gated by the dangerous (stack, run_command) permission, never (stack, deploy).

Shape:

• A single raw_run stage (RunMode = raw); the script lives on deployments.raw_command. There is no confirmation, apply, or replan.
• It acquires the stack lane (UseStackLane = true) like plan/destroy/refresh, so raw deployments queue FIFO and serialize with every other lane-using deployment on the stack.
• On run completion the workflow (nextForRaw) maps the outcome through applyRunOutcome: success → finished, failure/timeout → failed, cancelled → cancelled — always releasing the lane via PromoteNextOnStack.
• At accept the run carries no plan artifact (WorkArtifacts = nil); the runner installs tofu, clones at the commit, writes the managed-backend config, runs tofu init, then executes the script as bash -x -c <script> with the stack’s cloud + state-backend credentials and GANTRYCD_PAT in the environment.
• Raw runs produce no dependency snapshot, so ReportRunCompletion skips the dependency-graph reconcile for them.
• Like every other deployment it pins deployments.commit_sha to the stack’s synced commit at creation, and the runner checks that exact SHA out. A raw command is not commit-less; the code on disk when the script ran is a matter of record.

Attribution and activity

Three different people can appear on one deployment, and conflating them is a recurring source of bugs:

deployments.created_by is set on every path with an authenticated caller (manual, raw, retry, promote, local) and on SCM pushes whose commit-author email resolves to exactly one verified user (see EmailAttributionResolver). It is NULL — meaningfully so — for PR preview plans, dependency triggers, and pushes from an unrecognized email.

deployment_events is an actor log, not a state log

Every principal-initiated action on a deployment appends a row: created, confirmed_apply / confirmed_replan / confirmed_skip, skipped, cancellation_requested, prioritized / deprioritized.

State transitions are deliberately absent. finished, failed and cancelled are outcomes of a run, not acts of a person, and they already live on deployments.status; duplicating them here would create a second source of truth that ReportRunCompletion and the self-heal job would both have to maintain. The UI synthesizes the terminal row from status + completed_at.

Two invariants:

• At most one event per deployment per write transaction. created_at defaults to NOW(), which in Postgres is transaction start time, and the trail is ordered by (created_at, id). SkipAllDeployments writes one row each for many deployments, never two for one.
• actor_id is NULL only on created. Every other kind is reachable solely through an authz-gated handler, so a principal is always present; the service rejects the write otherwise.

Cancellation is a request, not an act

Cancelling has three moments and only the first has a person behind it:

1. RequestCancellation — a principal asks. This is what cancellation_requested records.
2. AcknowledgeCancellation — the runner observes requested_cancellation on its in-band status poll and stops. Called with a runner JWT, not a principal.
3. Terminal cancelled status, set on run completion.

There is no cancelled event, because nobody cancels: the runner does, and a person asked. There is likewise no system-initiated cancellation path — the queue cascade (failDeploymentAndCancelQueue) fails queued runs.

Stage actors are derived, not stored

runs carries no actor column. deployment_events.stage_id binds a confirmed_* event to the confirmation stage it resolved, and domain.AttachStageActors derives the rest structurally:

• the first stage (plan / destroy / refresh / raw) belongs to the creator;
• a confirmation belongs to the principal whose confirmed_* event names it;
• any run stage following a confirmation belongs to the nearest preceding confirmation, and inherits its decision as well as its actor — approving a plan is the act that applies it, and the inherited replan decision is what marks a narrowed re-run as a replan.

Any other stage is left unattributed rather than guessed at.

Audit

run_command, apply confirmation, cancellation requests and skips also emit on the deployment-audit: slog channel (internal/backend/services/audit.go), carrying actor_id and actor_kind. The raw command’s script is never logged — operators paste secrets into it.

Local deployments

A local deployment (OriginType = local) plans a tarball of a user’s local working tree (gantrycli local-plan) instead of an SCM checkout. It is created through LocalDeploymentService.CreateLocalDeployment — the trigger half of a two-phase flow (see Run context and artifacts for the upload reservation, token, and storage details) — and gated by the dedicated (stack, local_deploy) permission, never (stack, deploy).

Shape:

• Always Mode = plan_unlocked (enforced by Deployment.Validate and a DB CHECK): one plan run, no confirmation/apply/replan, never takes the stack lane, runs in parallel with the lane like a PR preview plan.
• deployments.local_source_key / local_source_sha256 snapshot the uploaded tarball’s object key and checksum. At accept the run carries RunContext.LocalSource (scoped GET credentials for exactly that key) and SCMCredentials.Type = "none" — no SCM token is minted for a run of unreviewed code. The runner verifies the tarball’s SHA-256 before extracting.
• Commit fields are the client-reported HEAD snapshot (plus a (dirty) marker when the working tree had uncommitted changes) — provenance metadata for display, not a commit the SCM necessarily knows. SCM status publishing is therefore skipped for the local origin.
• Not promotable: promotion is keyed on pull_requests rows, which local deployments never have.
• Not retryable: RetryDeployment rejects the local origin with ConflictError — the tarball is reaped once the deployment is terminal, so there is nothing faithful to re-run. Re-run gantrycli local-plan instead.
• Skip-all (BatchDiscardSkippableDeployments) excludes every plan_unlocked deployment (PR previews and local plans alike), so a bulk skip of the lane queue never discards another user’s local plan.

Stack Lock Model

The stack lock is owned by the currently active deployment, not by the runner group.

Meaning:

• at most one deployment per stack can occupy the active lane
• younger deployments on the same stack stay queued
• queue ordering (prioritized first, then FIFO) is maintained by SQL, not by in-memory queues

State-unlocked runs do not hold or block the stack lane. PR plan deployments (plan_unlocked mode) always have state_locked = false on their runs. PromoteNextOnStack and FailQueuedRunsForStack both filter on state_locked = true, so an active unlocked run is invisible to the queue and does not prevent a new queued deployment from being promoted.

Important invariant:

• after any terminal or discard-like transition, the same write transaction must attempt promotion of the next queued deployment
• plan_unlocked deployments never call PromoteNextOnStack because they never hold the stack lane

This is heavily exercised in internal/backend/services/deployment_service_queue_test.go.

Poll, Assignment, And Accept

Runner-group poll only claims work tentatively:

1. the next label-compatible pending run in queue order (prioritized first, then oldest) is selected
2. a stub runner is created in not-ready
3. the run becomes assigned
4. assigned_expires_at is set (RUNNER_ASSIGNMENT_TIMEOUT, default 5m — sized above runner cold-boot so a slow-booting runner still accepts before expiry)

No deployment/stage transition happens yet. For backend-dispatched groups, a pending run that already has a live runner assigned (a not-yet-reaped stub, or an accepted runner) is not re-dispatched — ListDispatchCandidates serializes dispatch to one in-flight runner per run, so a re-queued run can’t accumulate a herd of superseded runners all racing to accept.

The authoritative transition happens on ephemeral-runner accept:

• run assigned → running
• deployment pending → planning or applying
• stage remains running
• started_at is set (this is the boundary between the “never started” and “lost while running” recovery paths)
• credentials and RunContext are generated at this point

Startup vs. execution timeouts

A run carries first_pending_at, stamped the first time it becomes pending and preserved across re-queue cycles. Three reapers key off three distinct signals so a runner that never started and a runner lost mid-run are handled differently (see Runner Polling):

• Per-attempt re-queue (assigned_expires_at, RUNNER_ASSIGNMENT_TIMEOUT): an unaccepted assigned run is reset to pending so a fresh runner re-polls it, incrementing assignment_attempts. first_pending_at is preserved, started_at stays NULL.
• Assignment-attempt cap (assignment_attempts, RUNNER_MAX_ASSIGNMENT_ATTEMPTS, default 5): a run that has now been assigned that many times without ever accepting is failed as startup_timeout instead of re-queued again. This is a count-based bound on a wedged accept loop — a runner that polls, is assigned, but never calls /accept (an assignment livelock, a broken runner image) — that gives up in minutes rather than burning runners until the wall-clock startup deadline below. A run merely waiting in pending for capacity never increments the counter, so this never shortens a legitimate wait.
• Overall startup deadline (first_pending_at, RUNNER_STARTUP_DEADLINE, default 48h): a run that was never accepted (started_at IS NULL) within the deadline is failed as startup_timeout and its lane released. Deliberately generous — a runner group may legitimately take a long time to come online.
• Execution timeout (last_seen_at, RUNNER_EXECUTION_TIMEOUT, default 1h): an accepted running run whose runner went silent is force-failed. An operator can short-circuit the wait with ForceDeleteRunner, which fails the active run and promotes the queue immediately.

Completion Flow

DeploymentService.ReportRunCompletion is the authoritative write path for runner completion.

It:

• validates the runner/run assignment inside the write transaction
• sets the run terminal status
• advances workflow state through internal/backend/services/workflow/
• updates the stage phase
• updates deployment status
• creates any next stage/run if the workflow requires it
• calls PromoteNextOnStack when stack occupancy may have been freed

Examples:

• successful plan run, plan has changes:
  • run success, DoneRequest.PlanSummary.HasChanges = true
  • plan stage done
  • deployment waiting
  • confirmation stage created
• successful plan run, plan has changes, stack auto-approves (see below):
  • plan stage done
  • deployment applying directly — no confirmation stage created; the apply run is created immediately (same path as a user’s apply confirmation, reusing the saved plan via findLastPlanRunID)
  • applies to plan_run only (narrowed replans included — they are plan_runs)
• successful plan run, plan reports no changes (the short-circuit):
  • run success, DoneRequest.PlanSummary.HasChanges = false
  • plan stage done
  • deployment finished directly — no confirmation stage created
  • stack lock released, next queued deployment promoted
  • applies symmetrically to every plan-style kind: plan_run, destroy_run, refresh_run
• replan at the confirmation stage (ConfirmActionReplan):
  • confirmation stage done, a new plan_run stage is created (RunModePlan)
  • the new run carries the user’s selection: Targets or Excludes (mutually exclusive — OpenTofu forbids combining -target/-exclude), plus an independent Replaces list (-replace=<addr>, repeatable). At least one of the three must be non-empty.
  • the runner materialises targets/excludes as -target-file=…/-exclude-file=… and emits one -replace=… per replace address (plan-style modes only)
  • runs.targets/excludes/replaces is what clients read the narrowing from: the stage response carries all three plus plan_run_id, so the apply that replays a narrowed planfile — and stores no targeting of its own — can be shown as narrowed too. The confirmed_replan event records the same addresses, but that is the DECISION trail; the run is what executed.
  • that plan completes and loops back through the same confirmation logic, so a deployment can be replanned repeatedly before apply/skip
• confirmed apply run succeeds:
  • apply stage done
  • deployment finished
  • next queued deployment may be promoted
• destroy or refresh success:
  • deployment usually ends directly in finished

Auto-approve (skip confirmation)

A stack may opt into applying a successful plan without the manual confirmation stage, via the per-stack auto_approve_mode column (Stack.AutoApproveMode):

• manual (default) — the confirmation gate stays; behaves exactly as above.
• auto — every successful plan with changes applies immediately, including plans that destroy or replace resources.
• non_destructive — applies immediately only when the plan has no destroy and no replace actions (PlanCounts.Destroy == 0 && Replace == 0); a destructive plan falls back to the manual confirmation stage. Import/Move/Forget/Read are not treated as destructive.

Scope and mechanics:

• Plan mode only. The decision is scoped to DeploymentModePlan in DeploymentService.ReportRunCompletion and honored in workflow.planRunCompleted (via Event.AutoApprove). destroy, refresh, raw, and read-only PR-preview plans (plan_unlocked) are never auto-approved. Applies to plan_run completions, a narrowed replan’s included.

• The decision is AutoApproveMode.Decide(summary), evaluated from the plan’s PlanSummary counts — which the backend already holds in memory at completion (never persisted). It returns both the outcome and a reason, because an opted-in stack that does not auto-apply has to be able to say why:

  mode	counts	outcome	reason
  auto	measured	apply	—
  auto	unmeasured	apply	blind
  non_destructive	measured, no destroy/replace	apply	—
  non_destructive	measured, destroy or replace	suppress	destructive
  non_destructive	unmeasured	suppress	incomplete
  manual	any	suppress	— (no policy was set)

  auto deliberately still applies when the counts are unknown — auto means auto — but the approval is flagged blind rather than silently treated as measured. non_destructive fails safe instead: destroy == 0 is only evidence of safety if every resource was examined (see PlanSummary.Gaps). ShouldAutoApprove remains as Decide(…).Approve for callers that need only the bit.

• When it fires, the plan workflow returns the same NextStage{apply_run, RunModeApply} a user’s apply confirmation produces; applyAdvanceResult creates the apply run (reusing the saved plan via findLastPlanRunID) and the deployment moves straight to applying. The startup-failure guard and stack-lock semantics are unchanged. SCM status publishes applying, never “awaiting approval”.

• Read at completion time (the current stack setting), not snapshotted onto the deployment.

• Both outcomes are recorded in the durable activity trail, as actorless, stage-less deployment_events rows written in the same tx as the transition (so the trail can never diverge from it):

  event	when	details
  auto_applied	the policy applied the plan	{mode, destroy, replace}, or {mode, blind: true}
  auto_approve_suppressed	an opted-in stack withheld the apply	{mode, reason}

  auto_applied is the trail’s counterpart to a human’s confirmed_apply — the timeline shows “Applied automatically” with no approver. A blind approval carries no counts at all, only the flag: writing destroy: 0 for a plan nobody measured would present a partial tally as fact, the same failure the honest-counts work exists to prevent.

  auto_approve_suppressed exists because the absence of an event is not an explanation. Without it, a non_destructive stack that suddenly parks at a confirmation gate is indistinguishable from a regression. A manual stack writes neither row — it never opted in, so nothing was withheld.

  A deployment_auto_approved slog line (channel deployment-audit:) is also emitted for SIEM, mirroring how manual apply emits both a confirmed_apply event and a deployment_confirmed slog line.

• Enabling it is gated by (stack, confirm): because auto-approve delegates apply-without-review, the create/update stack handlers require (stack, confirm) — not just (stack, create)/(stack, update) — to set a non-manual mode. Disabling (→manual) or editing other config on an auto stack needs only update.

• Orthogonal to promotion requirements (those gate PR plan/promote, pre-plan) and to PR promotion (still a deliberate manual action).

Dependent-trigger resolution (post-success hook)

When the completed stage is an apply_run of an eligible deployment (domain.Deployment.EligibleForDependentTriggers — shared with the detail handler’s section gate so the two can never drift) and the workflow result is DeploymentDone{Finished}, ReportRunCompletion also resolves the stack’s dependent triggers (the trigger= field of gantrycd:dependency: comments — semantics in dependency_graph.md). Both eligibility guards are load-bearing: the mode must be plan because destroy and refresh workflows also finish on an apply_run stage and a destroyed parent must not redeploy its dependents; and PR origins (pull_request, promoted_pull_request) are excluded because a promoted-PR apply deploys PR code rather than the configured branch — mirroring their exclusion from graph reconciliation. Plan-only finishes (no changes) never complete an apply_run, so they never qualify; raw and PR-plan deployments cannot reach this state at all.

Mechanics, mirroring the notification pattern:

• In the same write tx as the terminal transition, a dependent_trigger_resolve async task is enqueued (durable across a crash).
• Wake on commit (AsyncTaskPendingChannel pg_notify): the enqueue statement itself fires the notification, so an idle worker slot is woken the moment the transaction commits — and never for one that rolls back. The resolution always runs inside the task pipeline — never inline on the runner’s completion request (graph-store latency must not become runner-protocol latency), and never in a detached goroutine. It runs under an exclusive lease, so two replicas cannot resolve the same deployment’s dependents at once (see async_tasks.md).
• Idempotency via two claims: deployments.dependents_resolved_at (snapshot written once) and deployment_dependent_triggers. triggered_deployment_id IS NULL under FOR UPDATE (each dependent triggered at most once per parent deployment). The task re-scans untriggered always rows, so a crash between snapshot and trigger converges.
• Changeset awareness: when any edge is resource-scoped, resolution loads the changeset of the plan the apply executed — the last done apply_run stage’s run records its plan run (runs.plan_run_id, the targeted replan’s narrowed plan included), whose plan_changes section is read through the cached LogsService path — and filters those edges against it; a child whose every edge filtered out keeps an informational skipped_reason row that never auto-triggers. A missing or corrupt artifact fails open (everything changed, all edges fire); a transient storage error fails the resolution so the task/sweep retries it precisely. Matching semantics in dependency_graph.md → Resolution rules.
• Incarnation pinning: each snapshot row records the dependent stack’s created_at observed at resolution; the name join and both trigger paths match on it, so a stack deleted and recreated under the same id never inherits old parents’ triggers.
• Auto-trigger pre-flight: the system path runs the same cloud credentials pre-flight as the Deploy button before creating each child — broken credentials surface as trigger_error + the failure notification, not as a healthy-looking child that dies at startup.
• Each auto-triggered child is created through CreateDeploymentInTx in its own write tx, serialized on the child’s stack row lock — a normal plan deployment on the child’s synced commit, origin_type = dependency, queued into the child’s lane with PromoteNextOnStack deciding queued vs pending. A per-child failure (dependent stack deleted, commit not synced) is recorded on its row (trigger_error), fires the dependent_trigger_failed notification once, and never fails the others; the parent page’s button doubles as the retry.
• A child with multiple parents gets one deployment per firing parent, each claiming only that parent’s row — there is no cross-parent wait. The child’s stack lane serializes them; an age-based supersede_after queue policy can additionally collapse a queued child past its threshold. Full semantics in dependency_graph.md → Resolution rules.
• Retrying a dependency-origin deployment repoints the parent snapshot rows at the replacement (RepointTriggered), so parent pages follow the live attempt.

The no-changes short-circuit is driven by OpenTofu’s -detailed-exitcode (0 → no diff, 2 → diff present). After a successful plan the runner makes a single tofu show -json plan.tfplan call and feeds the bytes to the pkg/tofuinspect library, which derives the whole digest from that one document: a domain.PlanSummary + per-resource change list (with a self-rendered diff body), the prior-state resource inventory, and the configuration dependency graph. The summary is sent inline with /done; the workflow refines the outcome to OutcomeSuccessNoChanges via workflow.RunOutcomeFromCompletion, which preConfirmationOutcome maps to DeploymentDone{Finished, ReleaseLock: true}.

Why JSON (not log parsing): the rendered plan log isn’t a stable contract — phrasing shifts across versions, output blocks share indentation with resource diffs, and field-level sensitivity has to be re-derived from the rendered text. tofu show -json is the published machine-readable form (explicit action enum, output_changes map, structured sensitivity/unknown descriptors). The pkg/tofuinspect library — built on github.com/hashicorp/terraform-json — parses it and renders the per-resource diff itself, so there is no second tofu show text call and no regex slicing; the output is deterministic and byte-stable across the OpenTofu versions we support (golden-tested in pkg/tofuinspect/scenario_test.go). The exit code is still authoritative for HasChanges.

Counts are only meaningful with their gaps. PlanSummary.Gaps lists every reason the tally does not describe the whole plan, and PlanSummary.CountsComplete() is exactly “no gaps” — so the flag can never disagree with its reasons. A zero destroy count is not evidence that nothing will be destroyed:

analysis_failed replaces the old silent behaviour: the runner still reports empty counts so the exit-code short-circuit keeps working, but the gap makes the absence of data explicit instead of indistinguishable from a measured “no changes”. AutoApproveMode.ShouldAutoApprove requires CountsComplete() for non_destructive, so an unmeasured plan falls back to manual confirmation rather than auto-applying.

Narrowing is the one gap the plan document cannot reveal — Terraform’s complete field is absent from OpenTofu’s show -json entirely (verified through 1.12.1) — so the runner declares it from the -target/-exclude flags it passed (tofuinspect.WithNarrowed).

Every surface that shows counts must show their gaps. A partial tally rendered as bare numbers is worse than no tally: destroy: 0 from a plan that never examined the resource reads exactly like a measured zero. So:

PlanSummary.GapReasons() returns the distinct reasons (one reason can attach to many resources) and PlanGapReason.Label() renders each in plain language. An unrecognised reason — a newer runner than this backend — labels as a generic “incomplete analysis” rather than leaking its raw identifier to a user. The webhook provider forwards the summary as JSON, so it carries gaps verbatim with no rendering decision to make.

Import/move and drift. An import or move is not an action: OpenTofu records it alongside whatever the object is doing, so they live on PlanChangeEntry.StateOps with PlanCounts.Import/.Move as overlapping tallies (an imported-and-updated resource counts in both — Total() is a non-empty indicator, not an entry count). A change whose only content is such an operation carries ChangeTypeNoOp; it plans as a no-op while OpenTofu exits 2, so dropping it would empty the review surface for a plan that does mutate state. Dependent-trigger resolution therefore filters no-ops out of its changed paths (appliedPlanChangedPaths) — a move changes no attribute a dependent reads, and firing on it would spend run budget for nothing. Refresh-detected drift is kept out of the change list and the counts entirely (it already happened, and no apply will perform it).

Plan Analysis Artifacts

A plan-style run (plan, destroy, refresh) uploads two objects next to its logs, under LogsStorage.Prefix, same retention as metadata.json and the chunk objects. An apply has no plan diff and no summary, so it uploads only analysis.json, carrying the post-apply inventory and the dependency snapshot. A raw run uploads neither.

plan-summary.json — domain.PlanSummary (counts + has_changes); also echoed inline on /done so the workflow can decide without an S3 read. It is its own object because the UI loads it eagerly on every visible plan stage; folding it into the analysis would make the cheapest read on the deployment view pay for the most expensive payload we store.

analysis.json — contracts.RunAnalysis, an envelope of four nilable sections, every one of them fetched lazily or consumed once by a background job:

• plan_changes — domain.PlanChanges (per-resource action + self-rendered diff body); fetched lazily by the UI when the user expands the plan stage, and by dependent-trigger resolution to filter resource-scoped edges against what the apply actually changed.
• state_resources — domain.StateInventory (the resources in state: prior state for a plan run, post-apply state for an apply run, each with its recorded depends_on and its full attribute values); lazy-loaded by the UI’s resource-inventory panel, where a row opens the resource’s attributes/provider detail. Values OpenTofu marks sensitive are masked at the runner (replaced by (sensitive value), guided by the tofu show -json sensitivity descriptor) before the artifact is written, so plaintext secrets never reach S3, the API, or the UI; a parallel sensitive_values tree lets the UI badge the masked fields. This masks exactly what OpenTofu marks sensitive (provider schema + sensitivity propagation) — the same set tofu show hides. A secret stored in an attribute the provider does not declare sensitive is not masked; this surface is no more revealing than the tfstate already in S3, but it is not a guarantee that no secret is ever displayed.
• resource_graph — domain.ResourceGraph (the intra-stack resource dependency graph derived from the configuration: explicit depends_on plus inferred references); lazy-loaded by the UI’s dependency panel. This is OpenTofu’s own dependency graph surfaced as data — it does not drive cross-stack dependent triggers (those come from gantrycd:dependency: comments; see dependency_graph.md).
• dependencies — contracts.DependencyDiscovery (the stack’s complete cross-stack dependency snapshot); consumed once by the backend at run completion to reconcile the dependency graph. No UI route.

Both objects are write-once and immutable. They are served by /api/v1/orgs/{org}/stacks/{stack}/runs/{run}/logs/{plan-summary, plan-changes,state-resources,resource-graph} with Cache-Control: private, max-age=86400, immutable. The summary is read from its own object and its own LogsCache (Redis) entry. One fetch of analysis.json decodes it and populates the cache entry for every cold section it carries, so the first lazily-opened panel pays for the rest; poisoned entries fail open to S3.

The resource inventory is also surfaced at the stack level — the stack’s “current state” — by GET /api/v1/orgs/{org}/stacks/{stack}/resources (RunHandlers.GetStackResources → LogsService.GetStackStateResources). It resolves the run that last touched the stack’s state — a success, or an apply that failed part-way, which still changed real infrastructure — via RunRepository.RecentAuthoritativeRuns under AuthorityLiveState (ordered by completed_at, since an apply mutates state on completion) and reads that run’s state_resources section from analysis.json plus its source_run_id. A run that uploaded no inventory is walked past rather than served: a nil section means “this run tells us nothing”, so the endpoint falls back to the newest of the next few state-bearing runs that carries one, and reports that run as the source. Because the resolved run advances over time this endpoint is not immutably cacheable, so unlike the per-run artifact routes it sets no long-lived Cache-Control. It is gated by stack access (WithUserCanAccessStack), like the dependency graph.

“State-bearing” is a mode filter, not an origin one (DeploymentMode.DefinesStackCurrentState: plan, destroy, refresh). A stack has exactly one tfstate — paths.StateKey takes no branch, PR, or deployment component — so every lane-taking OpenTofu run observes and mutates the same state whatever branch it checked out. A promoted-PR apply is therefore authoritative for the Resources tab, as is a dependency-origin apply. What is excluded is plan_unlocked (PR previews and local-plan, which never take the stack lane and could read state mid-apply) and raw (runs no OpenTofu and uploads no inventory — including it would resolve the tab to a run with no artifact and blank it).

This is deliberately wider than the dependency graph’s filter, which is origin-based (DeploymentOriginType.RunsConfiguredBranch): the graph is derived from the checked-out HCL, so a PR head must never rewrite it. Index ← state; graph ← source. The same predicate gates the run-completion enqueue (ReportRunCompletion), the per-stack tab, and the 12h rebuild sweep — they must agree, or the sweep reverts what the enqueue wrote.

The endpoint serves two shapes off the one Redis-cached inventory so the heavy attribute payload only travels for resources the user actually opens:

• bare collection → a lean list (metadata only — no attribute values), carrying the stack’s outputs for the Outputs tab. (The Resources tab lists from the Explore index instead — see Resource Explorer.)
• ?address=A&address=B → the full detail (attribute values included) of just those resources, filtered server-side.

The client asks for the specific resource(s) it needs (a row click, or a node in the Dependencies graph) and the backend does the lookup — the frontend never pulls a whole stack’s attributes to show one resource.

The work prefix (work/orgs/{org}/stacks/{stack}/runs/{run}/) is intentionally kept short-lived — it only holds plan.tfplan for the next apply. The analysis artifacts live with the logs because they are the post-rendered, long-retention view of what the user saw in the log stream.

Best-effort upload + coherence invariant. The runner treats both PUTs as best-effort: a transient S3 failure is logged and swallowed, not propagated as a run failure — the authoritative HasChanges signal already travels inline on /done, so the backend workflow never depends on these objects existing. The worst case is “neither object in S3”, which the UI handles by not rendering the analysis panel.

Upload order: analysis.json first, plan-summary.json second, abandoning the summary if the analysis failed. That yields one coherence invariant — if plan-summary.json exists, the changes it summarizes exist too — so the UI checks the summary and never has to defend against “summary present, changes missing”. Within the analysis, coherence is structural: one write, so no reader sees half of it.

A consequence worth naming: the four cold sections now succeed or fail together. A partial upload used to leave some artifacts behind; it no longer can, and one failed PUT skips both the dependency-graph and the resource-index refresh for that run. Both are backstopped by their periodic rebuild sweeps.

Nil is not empty. A nil section means the producer never ran or never succeeded; a non-nil section is authoritative even when empty. Both derived stores reconcile destructively — ReplaceStackEdges and ReplaceStackResources replace a stack’s rows wholesale — so reading a nil section as an empty one would let a single best-effort upload failure delete every dependency edge, or every indexed resource, the stack has. Conversely a present-but-empty section is how a stack’s last gantrycd:dependency: comment, or its last resource, is removed. See contracts.RunAnalysis.

plan_changes is a single section today; pagination is deferred until real plans grow large enough to warrant it.

Cancellation

Cancellation has two shapes, split on whether a runner ever accepted the run (the /accept boundary — see Runner Polling → Failure Windows). RequestCancellation picks between them.

The run is running — a request, acknowledged by the runner

1. user/API requests cancellation
2. runner acknowledges cancellation before reporting a terminal status

Transitions:

• deployment/run move to requested_cancellation
• ephemeral runner polls status, sees the request, and cancels its local context
• runner calls POST /api/v1/runs/{run_id}/acknowledging-cancel
• deployment/run move to cancelling
• runner later reports cancelled

Skipping the acknowledgement step would lose the distinction between “requested” and “actively shutting down”.

The runner wins the race — a plan that succeeded anyway

A runner can finish a plan microseconds before it polls and sees the request, so requested_cancellation + a reported success is a normal outcome, not a fault. Two rules apply, and they point in opposite directions on purpose:

• The run stays success. It genuinely succeeded, and its plan artifact is still readable (ComputeFinalStatus deliberately lets a reported success through — see its test). Retconning it to cancelled would discard a result the runner actually produced.
• The deployment ends cancelled. ReportRunCompletion rewrites OutcomeSuccess / OutcomeSuccessNoChanges to OutcomeCancelled for a plan-style stage whose run is in requested_cancellation, so preConfirmationOutcome finishes it as DeploymentDone{Cancelled, ReleaseLock: true}.

Without the second rule the deployment carried on: an auto-approve stack started an apply of a deployment the user had already cancelled, and a manual stack parked a confirmation prompt for one. A no-change plan reported finished.

The rewrite is scoped to plan-style stages (DeploymentStageKind.IsPlanStyle). An apply/destroy/refresh that already succeeded has mutated real infrastructure, so reporting its deployment cancelled would be the lie in the other direction — those keep their normal outcome. Once the runner has acknowledged (cancelling), the earlier and stricter rule applies instead: the final status is forced to cancelled regardless of what the runner reports, because acknowledgement is irrevocable.

How the timeline names each shape

A cancelled deployment has four distinct endings, and only one of them involves a runner acknowledging anything. The activity trail’s terminal row distinguishes them (cancellationNote in DeploymentActivity.tsx) from data already on the wire — stage kind, run status, and started_at — with nothing extra persisted:

The order matters: the first two are checked before the start-time test, because a plan that succeeded also has a started_at. Only the two plan kinds count for the withheld case, matching IsPlanStyle — a cancelled destroy/refresh keeps its normal outcome, so “apply withheld” would describe a stage that workflow never has. started_at is stamped at /accept, the same moment a run becomes running, so its absence is exactly “no runner ever took this”.

The run is pending or assigned — terminated outright

Nothing ever accepted the run: it is waiting for a poll, or assigned to a stub runner that never booted. There is no runner to acknowledge anything, so the request is the cancellation. In the same write tx the run, its non-terminal stages, and the deployment all go terminal cancelled, and PromoteNextOnStack releases the lane. The runner (if any) is left to the existing GC, exactly as SkipDeployment leaves it: taking the run terminal is enough, because a runner’s assignment is derived from its run’s status. It stops counting as busy at once, can deregister itself, and CleanupInactiveRunners reaps it on age if it never comes back.

This branch is load-bearing, not a nicety. Parking a never-started run in requested_cancellation to await an ack that can never come strands it forever:

The stub is then GC’d, the FK nulls assigned_runner_id, and even the last reaper can never match again — while PromoteNextOnStack counts requested_cancellation as a legitimate lock holder, so the stack’s lane would stay locked forever. CleanupStrandedCancellations (see Recovery Jobs) is the backstop for rows already in that state.

It also closes a hole: confirming an apply moves the deployment to applying immediately while creating the apply run as pending. That deployment is not skippable (applying is not a skippable status), and before this its pending run was not cancellable either — it had no exit at all.

Note the branch keys on the run status, not on started_at alone. A run leaves assigned for running only at /accept, which is also what stamps started_at, so the two agree — but branching on the status cannot misfire on a live run, whereas a running row with a NULL started_at would be terminated out from under its runner.

There is still no cancelled event: a cancellation_requested row records the person who asked, and the terminal row is synthesized from status + completed_at.

Run-Deadline Cancellation (Hard Run-Duration Cap)

A run holds short-lived credentials minted at accept time (state backend, logs, artifacts, cloud runtime). All of them share one TTL (RUN_TTL, default 1h) and one shutdown grace (RUN_CREDENTIAL_GRACE_PERIOD, default 10m, capped at RUN_TTL/2):

• every credential is minted for TTL + grace;
• the backend stamps RunContext.RunDeadline at TTL − grace and RunContext.RunGracePeriod = grace. The deadline is always present, even when storage credentials do not expire; an earlier credential expiry can shorten it;
• at RunDeadline the runner interrupts the run — it cancels the run context, which SIGTERMs the tofu process group;
• if the run has not exited RunGracePeriod later (i.e. at TTL), the runner hard-kills the process group with SIGKILL.

No backend job or timer is involved. This is a hard cap on run duration: the run is interrupted at TTL − grace and killed by TTL, while the credentials — valid a further grace beyond TTL — comfortably outlast the kill, so the final state push and log upload always run with live credentials. The runner reports the run cancelled with cancellation_reason = run_deadline, and the backend records it as the gantrycd_run_deadline_cancellations metric. The reason names the deadline rather than the credential because RUN_TTL always sets it and an earlier credential expiry can only bring it forward. A runner that fails to do any of this is still caught by the stale-runner path below.

Skip Behavior

Skip applies only while the deployment has not entered irreversible execution.

Implemented paths include:

• skipping a queued or pending deployment
• skipping while the deployment is waiting
• bulk skipping skippable deployments on a stack incarnation

Skipping:

• updates deployment/stage/run records to skipped/discarded forms
• releases stack occupancy when appropriate
• ends with PromoteNextOnStack

Cancelling the whole runner queue (operator escape hatch)

When runner capacity goes away — a group is down, its credentials are broken, the org’s runs/hour budget is spent — runs pile up in pending on the org Queue page waiting for a poll that never comes, and the only other exits are skipping deployments one at a time or waiting out RUNNER_STARTUP_DEADLINE (48h). POST /api/v1/orgs/{org_id}/queue/cancel-pending (DeploymentService.CancelPendingRuns → DeploymentRepository.BatchDiscardPendingRuns) discards them in one write tx.

The predicate is exactly deployments.status = 'pending' AND runs.status = 'pending':

• assigned runs are not touched. A group has claimed them and a runner is booting; the assignment reapers own that window.
• plan_unlocked IS included — the opposite of BatchDiscardSkippableDeployments. PR previews and local plans never take the stack lane, so the per-stack “skip all” excludes them; but they queue for the same runner groups, so a wedged queue contains them and clearing it must clear them.
• skipped_reason stays NULL. That column means automatic skip (it drives the UI’s reason pill and the auto-skip notification); a user pressing a button stays silent like every other manual skip. Each discarded deployment gets a skipped deployment_events row instead, with {bulk, source: queue_cancel}.

Scope is resolved as name patterns from the caller’s (stack, confirm) grants — the route names no stack, so there is nothing for the middleware to check; it is the same decide-then-fetch shape the queue view uses for (stack, read). A principal who may skip on no stack gets Forbidden rather than a misleading “0 cancelled”.

Two properties worth internalising:

• The re-assert is on the RUN, not the deployment. A concurrent PollForWork claim moves runs.status pending → assigned and leaves deployments.status at pending, so re-checking the deployment (what the other batch-skip methods do) would not see the race at all and would skip a deployment whose run a runner is about to execute. The batch discards the runs first, re-asserting status = 'pending', and skips only the deployments whose run actually discarded.
• One press need not empty the queue. Discarding a pending deployment frees its stack’s lane, and the mandatory PromoteNextOnStack pulls that stack’s next queued deployment into pending — a fresh runner-queue entry. The queue refills by at most one row per backlogged stack and the operator presses again; it converges. Discarding the lane backlog too would be a far larger blast radius than “cancel what is waiting for a runner”, so it is deliberately not done. The confirm dialog says so.

Automatic skip (per-stack queue policy)

A stack may declare a queue policy that auto-skips stale lane work — a confirmation timeout, a queued timeout, and a supersede-on-enqueue age. These skip only queued / waiting deployments (never in-flight pending / planning / applying), exclude plan_unlocked, record a skipped_reason (superseded / confirmation_timeout / queued_timeout), and end with PromoteNextOnStack like every other skip. The timeouts run from the deployment-timeout recovery job; supersede runs inline in CreateDeploymentInTx. Full semantics in Queue Policy.

Retry

DeploymentService.RetryDeployment re-runs a terminal deployment as a brand-new one against the same commit. It is the only user-facing entrypoint that copies origin state forward.

Inside one write transaction it:

1. loads the original deployment, scoped by (orgID, stackID, deploymentID)
2. rejects with ConflictError if the original is not terminal (finished | failed | cancelled | skipped)
3. loads the stack snapshot
4. invokes CreateDeploymentInTx with the original’s Mode, CommitInfo, OriginType, OriginPRNumber, OriginHeadSHA, OriginSCMEventID; UseStackLane = !mode.IsReadOnly() so plan_unlocked stays off the lane while plan/destroy/refresh enqueue normally
5. for OriginType=pull_request, repoints pull_requests.latest_deployment_id at the new deployment via PullRequestRepository.UpdateLatestDeploymentByPRNumber — so the pull-requests view surfaces the new run

Notes:

• The original is unchanged. There is no “retried_from” backlink on the new row; the audit trail lives in the deployment list ordered by created_at.
• The PR (stack, pr_number, head_sha) index is non-unique by design (see schema/tbl_deployments.hcl): retry would otherwise be impossible. SCM-event dedup relies on the PR row’s latest_deployment_id plus GetIDByPROrigin returning the most recent match — not on a uniqueness constraint.
• Rapid double-clicks / multi-tab races are guarded by a server-side debounce: a retry is rejected with ConflictError if another deployment for the same (stack, commit_sha) was created within retryDebounceWindow (5s) — excluding the original being retried so a freshly-failed deployment stays retryable.
• PR-origin retries are rejected with ConflictError when the source deployment is no longer the PR’s latest_deployment_id (a newer deployment exists for the PR), or when the PR row has been deleted entirely. This prevents the retry from silently detaching the PR view from whichever deployment is actually current. The UI mirrors this rule by disabling the Retry button on stale PR deployments.
• Local-origin deployments are rejected with ConflictError: their source tarball is reaped once the deployment is terminal (see Local deployments). The UI hides the Retry button for them.

Startup Failures And Unsupported Providers

Cloud-provider support is checked twice:

• pre-flight read during CreateDeployment for fast feedback
• authoritative check inside the write transaction during startup-sensitive paths

If the provider is unsupported when execution is being prepared:

• the active run is marked with a startup failure message
• queued/pending state-locked runs on the same stack incarnation are failed in bulk (state-unlocked PR plan runs are unaffected and fail individually when a runner accepts them)
• the deployment is failed
• PromoteNextOnStack is still called to release the lock and preserve invariants (except for plan_unlocked deployments, which never held the lock)

This avoids repeated promote-fail loops across a broken queue.

Runner Spin-Up Failures

When a runner group claims a run via poll but cannot bring the runner online (github-actions workflow_dispatch rejected, self-hosted worker spawn failed, assignment-token signing failed), RunnerGroupService.ReportDispatchFailure:

• fails the claimed run and only that run (FailRunForRunner) plus its deployment — unlike the unsupported-provider path, the rest of the stack’s queue is left intact, since a dispatch failure is not necessarily stack-wide
• calls PromoteNextOnStack so the next queued deployment gets its turn
• deletes the not-ready stub runner so it never surfaces in the UI
• records last_dispatch_error on the runner group for operator visibility

This explicit path fails the run rather than re-queuing it: re-dispatching a run whose runner provably cannot be created just fails again on the next tick — a tight loop. The github-actions dispatcher reports synchronously; self-hosted groups report via POST /api/v1/runner-groups/dispatch-failed. A stub that is dispatched but silently never checks in is different — there is no definitive signal it will keep failing — so it is re-queued for a fresh runner and only given up on at the overall startup deadline (startup_timeout); see the startup vs. execution timeout split above.

Cancellation Recovery

If a runner never acknowledges cancellation (process crashed, network gone), RunnerCleanupJob deletes the stale runner once last_seen_at exceeds RUNNER_EXECUTION_TIMEOUT, fails any runs assigned to it, and calls PromoteNextOnStack. An operator can short-circuit this with ForceDeleteRunner. The deployment exits requested_cancellation via the same completion path as a normal failure. See runner_healthcheck.md.

Recovery Jobs

Background work in internal/backend/jobs/ keeps queues healthy:

• runner_cleanup.go — five reapers: CleanupExpiredAssignments (re-queue unaccepted runs, and give up as startup_timeout once a run hits the assignment-attempt cap), CleanupStartupDeadline (give up on never-started runs as startup_timeout), CleanupInactiveRunners (fail accepted-then-silent runs and delete stale runners), CleanupOrphanedRuns (discard a never-accepted run left non-terminal under a deployment that already went terminal — the self-heal backstop for a stranded run the cancel/skip cascade missed), and CleanupStrandedCancellations (cancel a never-started run parked in requested_cancellation/cancelling awaiting an ack no runner can send — see Cancellation: it is the one state the other four cannot see, and it locks a stack lane forever. Gated on the tentative assignment having lapsed, so it never races an /accept in flight. RequestCancellation no longer produces this state; the reaper exists for rows already in it).
• queue_recovery.go — PromoteAllStuckStacks re-promotes stacks stuck in pending after a crash between transitions, and refreshes the summary each promotion invalidates (queued→pending is a raw-SQL deployment write like any other).
• deployment_timeout.go — the per-stack queue-policy aging reaper: auto-skips queued deployments past their stack’s queued timeout and waiting deployments past their confirmation timeout, releasing each freed lane. See Queue Policy.
• deployment_status_sync.go — reconciles deployment status from the latest stage phase when they have drifted.
• stack_summary_reconcile.go — repairs any stack whose denormalized stacks.latest_deployment_* summary disagrees with the deployments table. See Derived state.
• schedule_expiry.go, session_cleanup.go, sso_cleanup.go, membership_expiry.go — adjacent housekeeping.

Derived state and its reconcilers

stacks.latest_deployment_* is a six-field projection used by the stack list. Deployment write paths refresh it in their transaction; stack-summary-reconcile independently recomputes it as a backstop for a missed refresh or concurrent lost update.

Three properties are load-bearing and easy to break:

• The stack summary is queried, not merely rendered. The stacks list filters, sorts and keyset-paginates over latest_deployment_*, and the list’s “Failed” status filter is latest_deployment_status = 'failed'. A stale summary therefore hides a failed stack rather than just mislabeling it, and keeps the web client polling indefinitely (it polls while any loaded stack is non-terminal).
• The reconciler scans every stack, including terminal summaries, and compares all six old fields before writing. A wrongly-terminal summary must remain visible to the sweep. Under READ COMMITTED, the source CTE is not recomputed after waiting on a row lock, so checking the complete old tuple prevents the job from overwriting a newer concurrent refresh.
• RefreshStackLatestDeployment is fail-safe on the incarnation. stack_created_at is matched against the stack row, not just used inside the LATERAL. A caller passing a stale or zero timestamp is a no-op; without the guard it would match no deployment and silently NULL the whole summary, dropping the stack out of every status filter at once.

Any new path that changes the projected deployment must refresh the summary. Every repair is logged at ERROR so missed paths and concurrency races remain visible.

See also: async_tasks.md for the durable async task queue (object-store cleanup, graph/index reconciles, dependent-trigger resolution).