Notifications

Notifications announce deployment executions to external systems through a pluggable provider surface, delivered asynchronously with retries. The async machinery mirrors the SCM event inbox; the provider/registry shape mirrors the other integration surfaces.

Provider surface

internal/backend/notify defines Provider and a static Registry (keyed by provider type, panics on duplicate keys at construction). Concrete providers live in subpackages (internal/backend/notify/slack, internal/backend/notify/webhook) and are registered once in cmd/backend:

notifyRegistry := notify.NewRegistry(
    notifyslack.NewProvider(slackclient.NewClient(cfg.SlackAPIBaseURL)),
    notifywebhook.NewProvider(webhookclient.NewClient()),
)

The provider code is global and stateless; per-organization configuration and secrets are passed in as arguments, so one Slack provider serves every org with each org’s own token.

type Provider interface {
    ProviderType() string
    ValidateConfig(raw json.RawMessage) error      // non-secret provider_config shape
    ValidateAuthConfig(raw json.RawMessage) error  // secret auth_config shape
    MergeAuthConfig(current, incoming json.RawMessage) (json.RawMessage, error)
    Send(ctx, integration, authConfig json.RawMessage, n domain.Notification) error
}

A new provider type is added by implementing Provider, registering it, and listing the type in pkg/domain — no change to the outbox, the worker, or the deployment hook.

Slack message layout

Every Slack message is posted as one attachment, whatever the event family. The attachment is what carries color, and the coloured stripe down the left edge is the only signal a reader takes in before reading a word — a channel of identical white blocks makes a failure indistinguishable from a success while scrolling. Colour is never the sole carrier: the headline’s emoji and its wording repeat it, and the attachment’s fallback carries it into push notifications, where there is no stripe at all. The request sends no top-level text — Slack would render it as a visible line above the attachment, duplicating the headline.

A message is two blocks, the same two for every event, so a reader learns the shape once (internal/backend/notify/slack):

The subject always comes first and is the only part that links, so the eye lands on the stack (or repository, or report) in the same place every time. Stripped of its bold markers and link syntax, the headline is the fallback — one function returns colour, fallback and blocks together, so the two can no longer drift. The digest is the deliberate exception: its headline is a title so it can be recognised on sight, while its fallback narrates the report.

Slack’s header block is not used, and neither is an actions row. The header block’s 18px type on its own line, above a two-column field grid whose entries were appended conditionally, made every message about twice as tall as what it had to say — and no value ever landed in the same place twice. The button went the same way: on the events that merely report something it linked to a page nobody was being asked to open, which the subject’s own link does at no cost. Only deployment_awaiting_confirmation and preview_awaiting_confirmation keep one, as the section’s accessory so it shares the headline’s row. That makes them the only messages in a channel with a box in it — which is exactly the signal they should be sending.

The stripe palette is four colours — success #2eb886, failure #e01e5a, attention #eca336 (only the two awaiting-confirmation events, the ones that ask the reader to act), neutral #868f96. The findings digest is green only when the graph was checked and every count is zero: not having looked is not the same as being clean.

Two renderings are deliberately conditional:

• Plan counts follow the event’s tense. A finished, non-read-only deployment “destroyed” resources; anything else — pending, failed, or a plan_unlocked preview — has them “to destroy”. Reporting a pending destroy of five resources in the past tense is the misreading this avoids, on the one event that asks the reader to act. When the counts are incomplete the line says Change counts unavailable and names every distinct PlanGapReason, because +0 ~0 -0 reads as “nothing will change” for a plan that may destroy what it never examined.
• The mode is translated, not printed. plan renders as “deploy” and plan_unlocked as “preview”; the raw constants tell a reader nothing.

One event links from its context line instead of its headline: dependent_trigger_failed names the dependent stack that never started, while the notification’s URL is the parent’s deployment. Hanging one off the other would send the reader somewhere they did not ask to go, so the headline stays plain and the context reads “after applying <parent>”.

Per-org configuration

notification_integrations (one row per org integration; managed by NotificationIntegrationService) stores:

• events (jsonb) — subscribed event types, e.g. ["deployment_succeeded", ...].
• stack_label_selector (jsonb) — only matched when a stack’s labels are a superset.
• changes_only (bool) — drop deployment notifications whose plan is measured and reports no changes. Suppression requires evidence: a deployment with no plan summary still sends, and the check reads HasChanges (sourced from -detailed-exitcode) rather than the counts, which are empty for want of data whenever the plan analysis failed.
• findings_digest_schedule (restricted cron, UTC) plus findings_digest_last_sent_at and findings_digest_last_counts — the dependency-diagnostics report’s cadence and its state. See the findings-digest section below.
• provider_config (jsonb) — provider-specific non-secret config (Slack: default channel
  • the stack-label key that overrides it).
• auth_config_ciphertext (bytea) — provider-specific secret, envelope-encrypted with the platform data cipher and AAD-bound to (org_id, integration_id) (Slack: {"bot_token": ...}). Write-only at the API layer: only the delivery worker decrypts it (GetForDispatch); list/get never hydrate it.

This is the same split the SCM integration uses (plaintext match_config vs encrypted auth_config_ciphertext).

Delivery outbox

notification_deliveries is a durable outbox — one row per (integration, event). It mirrors scm_events:

• status ∈ pending | processing | failed | dead; index on (status, run_at). There is no success status: a delivered row is deleted, so the table only ever holds deliveries that are in flight or have failed. Every operator surface built on it is therefore a dead-letter view, not a delivery history — do not label it one.
• event_type carries no CHECK constraint. Its vocabulary is domain.AllNotificationEventTypes, enforced by NotificationDelivery.Validate(). A second copy in SQL has to be hand-synced with every new event type, and going stale is exactly how dependent_trigger_failed and deployment_auto_skipped came to fail their enqueue transactions (and, on the reconcile path, retry forever). The guard is instead an integration test that enqueues every value in the domain list.
• payload (jsonb) is a self-contained snapshot decoded into domain.Notification: an envelope (event type, org, occurred_at, stack id+name, stack labels snapshot, deep link) plus one optional detail per event family — today only domain.DeploymentDetail (deployment id, status, mode, commit, plan-summary, origin/PR). Routing reads the envelope alone, so a family with no detail type still label-matches. The provider renders and routes from the snapshot without re-reading the database, so it stays correct even if the stack changes later.

ClaimPending claims due pending/failed rows with FOR UPDATE SKIP LOCKED, flips them to processing, and increments attempt_count. On success the row is deleted (MarkProcessed); on error MarkFailed sets failed with a backoff run_at, or dead once attempt_count >= notificationMaxRetries. Dead rows are kept so failures are visible (see the deliveries endpoints below) and pruned after a retention window by the notification-delivery-cleanup job (DeleteOldDead).

Requeue returns a dead row to pending for the manual retry endpoint. It resets attempt_count to 0 (and clears last_error) because MarkFailed compares the post-claim count against the retry budget — a dead row left at its exhausted count would dead-letter again on the first attempt, making the retry a no-op. Only dead rows qualify: a pending or failed row is already due for a claim and a processing row is mid-dispatch, so requeueing either races the worker for no gain.

Marks are scoped to the claim. MarkProcessed and MarkFailed both carry AND status = 'processing' AND attempt_count = <claimed> and report whether they stamped; attempt_count is the claim token, since ClaimPending increments it. A worker whose batch outlived notificationStaleProcessing is a straggler: RecoverStaleProcessing re-pended its rows and another worker re-claimed them, so its outcome no longer describes them. Its mark is refused (the caller logs and records no telemetry) instead of deleting a row that is mid-flight elsewhere, or burying an operator’s fresh requeue under an error that predates their fix. This mirrors scm_status_deliveries, where the claim token is desired_seq.

ProcessPending also stops using a claim before it can be stolen: ClaimPending stamps last_attempted_at once for the whole batch, and serial dispatch can outlive the window (notificationBatchSize rows × the 15s provider timeout against a blackholed target exceeds 5 minutes, and the LISTEN-driven sweep carries no deadline). At half the stale window the loop abandons the remaining rows, which stay claimed until recovery re-pends them — unfinished, not failed, the same treatment as a cancelled sweep. Together these mean no row is sent on a claim that could already have been recovered.

A disabled integration does not send. enabled is consulted at fan-out, so a disabled integration is never enqueued for — but rows queued before it was disabled used to keep delivering, and the retry endpoint made that reachable on demand. processOne now treats enabled = false as a notify.PermanentError, so such a row dead-letters with “notification integration is disabled” instead of sending. It dead-letters rather than being dropped because a dropped row is indistinguishable from a delivered one, and a retry would appear to vanish.

Permanent failures dead-letter immediately. A provider may wrap an unrecoverable error (bad token, missing channel, …) in notify.PermanentError; the worker then calls MarkFailed with maxRetries=0 so the row goes straight to dead instead of burning the retry budget. The Slack provider classifies known-permanent Slack errors (channel_not_found, invalid_auth, …) and HTTP 4xx (except 429) as permanent; 429 and 5xx stay retryable.

RecoverStaleProcessing resets rows stuck in processing past a timeout (worker crashed between claim and mark) back to pending; ProcessPending runs it at the start of every sweep so no delivery is stranded.

Every dispatch records the gantrycd_notification_deliveries counter tagged by outcome (sent / failed / dead) for dashboards and dead-letter alerting.

Listener and worker

• db.NotificationPendingChannel (notification_pending) is a Postgres LISTEN/NOTIFY channel. After an enqueue transaction commits, NotificationService.NotifyPending fires pg_notify; the backend listener (cmd/backend/runtime.go) wakes and calls ProcessPending.
• The notification-processor scheduler job runs ProcessPending every minute as a safety net — missing a notification is harmless.
• ProcessPending recovers stale processing rows, claims a batch, then for each delivery loads the integration with its decrypted secret (closing the read tx before any network I/O), selects the provider from the registry, and calls Send.

Non-outcome deployment events

deployment_awaiting_confirmation is hooked in applyAdvanceResult, where a completed run advances a deployment onto a confirmation stage — the apply gate. preview_awaiting_confirmation is hooked in CreateDeploymentInTx, which opens with a confirmation stage when PRPreviewRequiresConfirmation holds (a plan_unlocked PR preview on a stack whose pr_plan_mode is manual) — the preview gate.

The preview gate is reached once per push to every open PR on such a stack, and retireStalePreviews supersedes the previous one each time. That is why it is a separate event rather than a second trigger of deployment_awaiting_confirmation: the subscription is the opt-out. An origin_filter on the integration was the earlier idea and was dropped for the same reason the whole preview family exists — a preview is not a deployment, and the event name should say which one arrived.

The awaiting-confirmation pair, run_command_executed and pull_request_promoted announce a deployment reaching a state rather than finishing, and go through NotificationService.EnqueueDeploymentEvent in the caller’s write transaction.

They carry no marker, unlike the completion path. Each transition happens exactly once by construction — a deployment parks on a confirmation stage once, a command is submitted once, a pull request is promoted once — so sharing the caller’s transaction is both the exactly-once guarantee and what stops a rolled-back transition announcing itself. notifications_enqueued_at exists only because several code paths can drive a deployment terminal.

run_command_executed carries command_bytes and never the command. Arbitrary commands run on a runner holding the stack’s credentials, and operators paste secrets into them — which is why the audit channel records only the length too. An integration test asserts the raw stored payload contains no part of the command, so a field added later that happened to carry it would fail rather than leak.

Coverage: marker + reconcile

A deployment can reach a terminal state through several code paths — a run reporting completion, a self-heal job failing a stuck run, or a startup failure — so the notification trigger is keyed off the persisted status, not a single call site:

• deployments.notifications_enqueued_at (nullable) marks whether a deployment’s notifications have been enqueued. MarkNotificationsEnqueued sets it conditionally (only if NULL) and reports whether the caller won the claim, making enqueue idempotent and race-safe.
• Fast path: DeploymentService.ReportRunCompletion calls NotificationService.EnqueueDeploymentCompletion after commit (best-effort) when the workflow returns DeploymentDone, for low latency.
• Safety net: the notification-reconcile scheduler job (ReconcilePending) claims terminal deployments whose marker is still NULL (ListUnnotifiedTerminal) and enqueues them. This covers every path the fast hook missed — self-heal/startup failures, or a crash in the post-commit window — within a minute.

Apply completions carry the plan they executed. The runner produces a PlanSummary only for plan-style runs (ExecuteResult.PlanSummary is nil for apply and raw), so the summary handed to ReportRunCompletion is nil for every plan → confirm → apply deployment — the no-op plan that short-circuits before any apply was the only case that ever reported counts. DeploymentService.appliedPlanSummary fills the gap: it resolves AppliedPlanRunID (a targeted replan’s narrowed plan included) and reads that run’s plan-summary.json. It runs post-commit, on a short read tx plus a cached blob fetch, and every failure path returns nil — the notification simply goes out without counts, exactly as before.

Note this is best-effort in a way the marker makes one-shot: the fast path has already claimed notifications_enqueued_at, so a failed artifact read is not retried by the reconcile sweep. Counts are advisory, so that is an accepted trade rather than an oversight.

EnqueueDeploymentCompletion claims the marker, maps the terminal status to an event type (finished→deployment_succeeded, failed→deployment_failed, cancelled→deployment_cancelled; skipped with a skipped_reason — an automatic skip, not a user’s — maps to deployment_auto_skipped; a user-initiated skip sends nothing). A plan_unlocked deployment maps to the preview_* twin of each, so an integration subscribes to deployments and previews independently — domain.NotificationEventTypeForDeployment is the single place that decides. It then loads the stack for labels, and writes one delivery per subscribed, label-matching integration — all in one transaction, so the marker and the deliveries commit together. A pre-existing-history backfill migration marks old deployments so enabling the feature does not retro-notify them.

Findings digest

A per-integration scheduled report, not an event. notification-findings-digest (every 15 minutes) calls ProcessFindingsDigests, which finds integrations subscribed to explore_findings_changed whose schedule has come round since they were last reported to, and sends each the org’s current ExploreFindings.Counts().

Why scheduled rather than change-triggered: these findings are a standing backlog, and a change-triggered version is silent about an org carrying the same twelve cycles for months — exactly the org that needs telling. The job interval is the resolution of the schedule, not the schedule: it bounds how late a report lands, so “Mondays at 10:00” may arrive by 10:15.

State lives on notification_integrations: findings_digest_schedule (a restricted cron expression, no CHECK), findings_digest_last_sent_at and findings_digest_last_counts. The stamp and the delivery commit in one transaction — a stamp without a delivery silently skips an occurrence, a delivery without a stamp repeats on the next tick.

pkg/cronspec parses <minute> <hour> * * <day-of-week> in UTC. Minute and hour must be fixed and the other two fields must be *, which makes a sub-daily report unrepresentable rather than merely rejected by a rule someone can loosen. That restriction is also why the parser is small enough to own instead of taking a cron dependency.

It is its own package rather than more of pkg/domain, which holds entities and their validation: a cron dialect is neither, and it knows nothing about what it schedules. It returns plain errors naming the offending field, and NotificationIntegration.Validate turns that into a validation failure against findings_digest_schedule — cronspec knows the expression is wrong, only the domain knows what the field is called.

Due-ness is evaluated in Go, not SQL: the repository returns subscribers with their last-sent time and the service compares PreviousFire(now) against it. PreviousFire returns the most recent occurrence at or before now rather than a next-fire time, so a tick that arrives late — or after the scheduler was stopped — still sees the same occurrence and sends, instead of stepping over it. An integration that has never reported is anchored on its updated_at, so subscribing on a Wednesday to a Monday report waits for Monday; a later edit re-anchors, which can defer one report.

Three properties are load-bearing:

• Clean orgs are skipped and NOT stamped (FindingsCounts.HasFindings). A clean org stays quiet, and because nothing was recorded it reports on the next tick after something appears rather than waiting out the interval.
• An unchecked dimension is never treated as zero. CyclesComplete / OrphansComplete say whether each check ran; an over-budget read skips its own detection, so that count is unknown. Duplicates (index-derived) are always measured. The message names what was not checked.
• The first report carries no previous — a zeroed one would read as “everything just appeared”.

Operator endpoints

• GET /api/v1/orgs/{org_id}/notification-deliveries — the dead-letter view: stored deliveries newest first, with status, attempt_count, last_error, and the deployment_id / stack_id / stack_name decoded from the payload snapshot so a failure says what went unannounced. Successful deliveries are absent by construction (the row is deleted on send), so an empty list means “nothing is failing”, not “nothing was sent”. Gated by (notification_integration, read).

• ?status= (repeatable) narrows the list to those states; omitted means every stored row. An unrecognised value is a 400 rather than being ignored, so a typo cannot render as “nothing is failing”. ?limit= defaults to 50 and caps at 200; the web panel asks for one more row than it renders so it can say the list is a floor rather than truncating in silence.

• POST /api/v1/orgs/{org_id}/notification-deliveries/{delivery_id}/retry — re-queues one dead-lettered delivery (Requeue) and fires NotifyPending so the worker sends it immediately. It answers {"outcome": "requeued" | "already_queued"}. A retry’s postcondition is “this delivery is queued to send again”, so a row that is already queued — a double-click, two tabs, or a row mid-dispatch — satisfies it and is reported as already_queued without being touched: resetting a pending or failed row would race the worker for a claim it is about to take. Only a delivery that does not exist in the org is a 404. Gated as (notification_integration, update) against the owning integration’s name, which GetNameByDeliveryID resolves from the delivery id — otherwise a patterned grant could be sidestepped by addressing the delivery instead of the integration. That resolution also joins on d.org_id = i.org_id, because the FK to notification_integrations is single-column and cannot enforce org agreement.

  Three cases return 404, and they are not all alike:

  • a denial and a delivery that does not exist are byte-identical, produced by the middleware from the same notification delivery not found message. That is the property scopedDecision promises, and it survives only because two string literals in two files agree — TestRetryNotificationDelivery_DenialIsIndistinguishableFromAMissingDelivery compares the bodies so a rename cannot quietly break it.
  • a row that exists but is not dead answers from the handler instead, so it is JSON rather than text/plain (handler errors are JSON by convention; the middleware uses http.Error). A caller can therefore tell that case apart. It reveals only that some delivery exists for an integration whose name already matches their update grant — inside the blast radius of a permission they hold — so it is documented rather than papered over.

• POST /api/v1/orgs/{org_id}/notification-integrations/{id}/test — sends a synthetic notification through the provider immediately (bypassing the outbox) to verify the token/channel. It writes no delivery row, so the HTTP response is the only result; a test never shows up in the dead-letter view.

  The payload is built for the integration’s first subscribed event in canonical order, or for ?event_type= when given. It used to always be a deployment_succeeded, which meant an integration set up purely for stack changes or the diagnostics report was verified by a message it would never receive — the rendering and routing under test were not the ones in use. buildTestNotification fabricates a plausible detail per family, so an operator sees the shape of the real thing.

Configuration

• SLACK_API_BASE_URL — Slack Web API root (default https://slack.com/api); override for tests/proxies. Bot tokens are per-org, not set here.
• GANTRYCD_BACKEND_PUBLIC_BASE_URL — used to build the deep link in notifications; links are omitted when unset.
• GANTRYCD_NOTIFICATION_DELIVERY_RETENTION — how long dead-lettered deliveries are kept before the cleanup job prunes them (default 30 days).