SCM deployment status

GantryCD publishes each deployment’s live state back to the source-control platform as a projection of the deployment state machine. It reuses the org’s existing SCM integration for credentials and repository matching, and mirrors the notification outbox — with one change: a coalescing outbox keyed on the deployment, so a burst of transitions becomes a single in-flight publish and a stale state can never overwrite a newer one.

The user-facing behaviour (what shows up, required GitHub App permissions) is documented in Deployment status.

Provider surface

The neutral vocabulary is domain.SCMDeploymentStatusState (queued | in_progress | success | failure | error | cancelled | inactive). domain.MapDeploymentStatusToSCMState is the single source of truth mapping a deployment status to it (e.g. waiting → in_progress with an “Awaiting approval” description).

Two methods on the scm.Provider interface (internal/backend/scm/provider.go) carry it:

• SupportsStatusPublishing() bool — the worker skips (and settles the row as a no-op) when a provider can’t publish, so the always-on policy degrades gracefully.
• PublishDeploymentStatus(ctx, PublishStatusInput) (PublishStatusResult, error) — surfaces the state for a commit (and optionally its PR). PriorRef carries the opaque handle the provider returned last time (e.g. the GitHub Deployment id); the returned ExternalRef is persisted and fed back on the next transition. Unrecoverable failures are wrapped in scm.PermanentError so the worker dead-letters them.

The GitHub implementation (internal/backend/scm/github/status.go) posts both a commit status (POST /repos/{o}/{r}/statuses/{sha}, contexts gantrycd/<stack> for lane deployments and gantrycd/<stack>/preview for PR previews, last-write-wins) and a Deployment object created on the fly (POST .../deployments, auto_merge:false, required_contexts:[]) plus a deployment status. inactive (skipped) is recorded on the Deployment but not posted as a commit status, to avoid a red ✗ on a never-run deployment. A 401/403/404 is classified permanent; 429/5xx stay transient.

Coalescing outbox

scm_status_deliveries (schema/tbl_scm_status_deliveries.hcl) holds at most one row per deployment (unique on deployment_id). It snapshots repository_url, commit_sha, pr_number, environment (stack name) and target_url at enqueue, so dispatch never re-reads a possibly-renamed stack. The matching integration is resolved by the worker (by repo URL), so the hot deployment path pays only one stack read plus the upsert.

• Enqueue (SCMStatusDeliveryRepository.Enqueue) is an INSERT … ON CONFLICT (deployment_id) DO UPDATE that advances desired_status and bumps desired_seq (a per-row revision counter). A row mid-publish (processing) keeps its status; otherwise it is (re)set to pending. It is emitted in the same write transaction as the deployment’s UpdateStatus, and emits the wake pg_notify inside that transaction — Postgres delivers the NOTIFY only on commit and drops it on rollback.
• MarkProcessed stamps published_status + external_ref only when desired_seq still matches the claimed value. If a newer transition coalesced in mid-publish, the stamp is skipped and the row is re-pended (RependStale), so the newer state publishes next — this is why a slow worker can never leave a stale state on the SCM.
• Settled rows are kept (not deleted) so external_ref / published_status persist; the cleanup job prunes terminal-processed and dead rows past retention.

Hooks and coverage

DeploymentService.enqueueSCMStatus is called in the same write tx right after each status-changing UpdateStatus: deployment create (queued), run accept (planning/applying), workflow advance (applyAdvanceResult, both the next-stage and terminal branches), skip, and the startup-failure paths. The requested_cancellation and cancelling transitions are intentionally not hooked — they map to the same in_progress SCM state already published, so re-posting would be a no-op.

Two transitions write deployment status via raw SQL outside the service — FailRunsForInactiveRunners and FailRunsPastStartupDeadline (the runner-cleanup jobs) — plus SkipAllDeployments. ReconcileDrift is the safety net: a single UPDATE … FROM deployments that re-pends every settled row whose deployment’s live status maps to a different SCM state than what was last published. The comparison is on the mapped state (a SQL CASE mirroring MapDeploymentStatusToSCMState), so a same-state raw transition — e.g. the queued → pending promotion done by PromoteNextOnStack — does not cause a redundant publish.

Listener, worker, and jobs

The backend listens on db.SCMStatusPendingChannel (scm_status_pending) and calls SCMStatusService.ProcessPending, which recovers stale processing rows, claims a batch (FOR UPDATE SKIP LOCKED), resolves each row’s integration, publishes, and settles. Three scheduler jobs back it up:

Resilience

• Transient GitHub failures (5xx, network, and rate limits) are retried up to 8 times with an exponential backoff (1m, 2m, 4m … clamped to 1h), so the budget spans ~2 hours before the row dead-letters — long enough to outlast a typical GitHub incident. Rate limits (429, or a secondary-rate-limit 403, on the API or the installation-token endpoint) are detected and the worker honours GitHub’s Retry-After / X-RateLimit-Reset window when it is longer than the scheduled backoff (clamped to 1h).
• A plain 403 (“Resource not accessible by integration”) is retried, not dead-lettered: it usually means a missing GitHub App permission — operator-fixable. 401 (bad token) and 404 (deleted repo) still dead-letter on the first attempt.
• dead is not final. Any later transition on the deployment revives the row via Enqueue’s upsert — but a deployment whose last transition was terminal has no later transition, so that alone would leave the commit showing whatever was published before (typically an in_progress check that never clears, blocking a required-check PR forever). scm-status-reconcile therefore also revives a drifted dead row: once it has rested SCMStatusDeadRevivalRest (2h) since its last attempt and while its attempt_count is still under the ceiling (20). Each revival buys exactly one publish attempt (the row keeps its attempt_count, so it re-deads on the next failure), so a permanently-broken integration (deleted repo, uninstalled App) is retried a bounded number of times (~a day) and then left dead for good. GANTRYCD_SCM_STATUS_DELIVERY_RETENTION must exceed one rest or the cleanup job prunes a resting dead row before it can be revived; Config.Validate rejects a shorter window.
• A vanished integration is not a silent success. If a deployment published via a resolved integration and that integration is then deleted, repointed, or its match config breaks, the delivery dead-letters loudly rather than settling as a no-op — otherwise the commit would freeze at the last-published state, invisible to the drift reconciler. Once the integration is restored the revival republishes it. The discriminator is the row’s resolved integration_id, not whether a status was stamped: a delivery in an org that never had a matching integration keeps settling quietly (no backlog, no alert) across every transition, however many.
• The outbox row’s identity is immutable. repository_url, commit_sha, pr_number and is_preview are fixed when the deployment is created and are never re-derived from the stack on a later transition. The stack is mutable (an operator can repoint it at another repository, or delete and recreate it under the same deterministic id) while the deployment is not — re-reading them would retarget an in-flight delivery at a repository its commit does not exist in. A deployment whose stack incarnation has been superseded is skipped at enqueue entirely.
• Crashes mid-publish are recovered: the claim is reset from processing after 5 min and retried. Delivery is at-least-once and idempotent (commit statuses are last-write-wins; deployment statuses are append-only).
• Known limitation: because the created GitHub Deployment id is persisted only when the publish settles, a hard crash in the narrow window between creating the Deployment object and settling can, on retry, create a second Deployment object for the same ref+env (GitHub auto-marks the older one inactive on success). This is cosmetic — an extra Environments-timeline entry — and was accepted rather than guarded, to avoid an extra API call on every deployment’s first publish.

Configuration

Retry budget and timing live in internal/backend/services/scm_status_service.go (scmStatusMaxRetries, scmStatusRetryDelay, scmStatusStaleProcessing), matching the notification worker.