Async Task Queue

The async task queue is the durable pipeline for follow-up work that must survive a crash after the database state it belongs to has already committed: cleaning up an object-store prefix a deleted stack left behind, reconciling the cross-stack dependency graph from a finished run, resolving a deployment’s dependent triggers, auditing an org’s seat cap.

It is orthogonal to the deployment state machine. Tasks are enqueued inside a business transaction and drained later; nothing in a request or a run waits on them.

Task model

Rows live in async_tasks. The processor (internal/backend/jobs/task_processor.go) is generic — a task type is added by calling RegisterHandler in cmd/backend/scheduler.go and nothing else. Registered today:

Worker slots

Every replica runs GANTRYCD_TASK_PROCESSOR_CONCURRENCY (default 4) worker slots for the process lifetime. There is no scheduled tick and no batch. A slot:

1. discovers candidate ids on the read lane — run_at <= NOW(), attempt_count < 20, not currently leased, oldest first, 20 at a time;
2. claims exactly one of them on the write lane, in a single statement;
3. runs its handler for exactly the lease it took;
4. completes it, and goes straight back to step 1.

Fleet capacity is therefore replicas x slots, and adding a replica adds throughput.

The two-step shape exists so the polling half is a read. How much that buys depends on your deployment, and the honest version is worth stating:

• With a read replica configured (DATABASE_URL_READ), idle polling never touches the primary at all — which is the point.
• Without one — and the chart leaves DATABASE_URL_READ unset by default, so the read pool IS the primary — discovery is simply a second, cheaper round trip to the same server. Measured on that default, the split runs about 1.6 read queries/s against 0.4 zero-row updates/s for a single-statement equivalent: four times the query count, all of it cheap reads instead of write transactions.

The split is kept for the replica deployments it does help. On a replica, one further cost: a task enqueued at time T may not have replicated when the NOTIFY wakes a slot, so the claim rounds up to the next 5 s poll.

Nothing in the processor orders one resource’s tasks. It does not need to: the handlers that care compare the snapshot a task carries against the one already published and drop the older (ReplaceStackEdges, ReplaceStackStateProjection, PutFindings). Those guards are correct across replicas; a processor-side ordering would only ever have held within one replica’s batch.

The lease

Between claim and dequeue, exactly one worker owns a task.

One claim statement leases exactly one row, which is why the row-picking half is a WITH picked AS MATERIALIZED (…) rather than a FROM (…) subquery: a subquery can be rescanned once per outer tuple of a nested loop, and each rescan takes another row. TestAsyncTaskRepository_ClaimLeasesExactlyOneRow pins it.

• The claim stamps a random lease_token and sets lease_expires_at = NOW() + <that task type's registered ceiling> + 30s. It takes the row FOR UPDATE SKIP LOCKED, so two slots handed the same candidate list take different rows rather than racing for one.
• The handler runs under a context whose deadline is that returned lease_expires_at minus the margin — derived from the database’s own stamp, not budgeted from the ceiling and a local clock. So the row stays owned for half a minute after the handler must have stopped, and a claim that was itself slow shortens the handler’s budget instead of pushing it past its lease.
• If that deadline is already past when the claim returns (the claim outlasted the whole ceiling), the handler is not started at all: it would run beyond its lease from the first instant, with another worker free to take the row. Recorded as timeout with no latency sample, then backed off like any other failure.
• Every write about the task is fenced on its token: Dequeue, RecordFailure, AdvanceCursor and the release all carry AND lease_token = $tok. They do not all react the same way to a zero-row match: Dequeue and RecordFailure return a NotFoundError so the caller rolls back, while AdvanceCursor and the release simply report that they wrote nothing — a lease that already lapsed needs no releasing, and a slice that lost its lease has nothing to record. Dequeue also matches an unleased row when the caller passes an empty token, which is how the two inline cleanup paths remove a task they enqueued and completed themselves.
• All claim-eligibility comparisons happen in SQL against the database NOW() — whether a row is claimable, dead, or live is decided entirely in the database’s own frame. The single exception is the handler deadline below, which is derived from the returned stamp and is safe only because of the margin.

The handler contract: done, continue, or error

Handlers never touch the queue row. A handler returns (done bool, err error) and the processor writes the consequence. Three states, and nothing else:

Only a sliced handler returns continue, and only after committing a cursor that means the next slice starts somewhere new. That commit is AdvanceCursor, the one queue write a handler makes: it rewrites the payload, sets run_at = NOW(), resets attempt_count, and clears the lease in the same statement. So the row is unleased and due atomically with the progress describing it, and the processor’s release on continue is a safety net rather than the mechanism. That net matching a row is how a slice which advanced nothing is detected — it still gets released, and it is logged at Warn, because its next attempt will repeat exactly what this one did.

A refused settlement is treated as “this worker no longer owns the row” and nothing is written. That is only correct because a handler that returns has no work left in any of the three states: it finished, it committed its progress, or it failed.

A sliced handler whose AdvanceCursor is refused must return an ERROR, not continue. Its lease was taken over, so returning continue would have it release the row the new holder is running. The error routes through the processor’s own fenced settlement, which is refused in turn and recorded as lost.

A read that fails part-way through a slice

A sliced handler reads its work in pages, and any page can fail. What happens depends on whether the slice had already committed anything:

• After progress: it saves its cursor and returns continue, and the failure is a Warn. AdvanceCursor has already cleared the lease, so an error would meet a fenced RecordFailure and be recorded as lost.
• On the first page, with nothing committed: an ordinary error. Backoff, last_error, attempt retained, dead-letter at 20.

The first case leaves no trace in the data — the cursor write nulls last_error and resets attempt_count — so the log line is the whole signal. Search for slice stopped early on a page read; it carries org_id, cursor and pages_read.

A continue is counted as continued, not success — a slice is not a finished task — and sits outside the failure|timeout alert set. One returned with the handler context already past its deadline is recorded as timeout instead. That is a property of the deadline, not of intent: any (false, nil) at the deadline lands there.

Why the margin, and what the outboxes actually do

The margin and the derivation cover two different failures, which is why both are there:

Budgeting the ceiling from the moment the claim returned would get the second one wrong: the handler’s clock would start late and could outlive its own lease. Deducting the observed round trip instead does not fix it either — that correction is the same order as the clock offset it is competing with, and it covers only a fast claim. Deriving from the stamp needs no measurement at all.

This is the one place a database timestamp is compared against the application clock. It is safe only because of the margin: realistic skew is milliseconds, and the margin is thirty seconds.

The SCM-event and notification outboxes do not stamp a window on the row — they use a status flag plus a periodic reclaim sweep. But they keep the same kind of deliberate gap between “how long I may work” and “how long I own this”: the SCM sweep’s reclaim threshold is set “comfortably above the longest healthy processing time”, and the notification worker stops using its claim at half its window. This queue writes the window onto the row instead of inferring it from a flag, and keeps that gap as an explicit margin.

And the deadline is not what protects the row — the token fence is. The deadline and the margin make “the handler stops before the row is claimable” true by construction; the fence refuses the write of a worker that overran regardless.

What it costs, and what it does not cover

A worker is killed. Its row stays leased until ceiling + 30s elapses, so that sum is the crash-recovery time for the type: a prefix_delete orphaned by an OOM kill waits an hour and a half-minute, explore_diagnostics thirty and a half, the two rebuilds ten and a half, everything else five and a half. That is the deliberate trade for having no renewal machinery, and it is why the ceilings are per type rather than one shared number.

A rollout. Not the same thing: shutdown cancels in-flight handlers and explicitly releases their leases, so another replica picks the work up within a poll interval. The ceiling only bounds the ungraceful case.

A handler that ignores its context. This is the one case the model does not cover, and it is worth being plain about. Such a handler keeps running past the ceiling; its lease lapses; another worker re-runs the same task alongside it; and its slot is never returned, so the replica quietly loses capacity — repeatedly, as each re-run leaks another slot.

No handler does that today. Object-store calls go through the aws-sdk default client, which honours ctx; every database call goes through pgx, which honours ctx and sits under the jobs lane’s statement_timeout. But a new handler that spins without checking ctx would reintroduce exactly the duplicate execution this queue exists to prevent, and nothing in the queue can stop it.

At-least-once, and what the fence actually protects

Delivery is at-least-once, as it was before leases existed. The fence protects the queue row and nothing else: the processor’s settlement is a write transaction of its own, and most handlers do their real work in the Explore database or in object storage — different stores entirely — so a worker that lost its lease may already have committed its business write before the settlement is refused.

What leases changed is that the duplicate window is now bounded by a ceiling and visible on a counter, instead of unbounded and silent. Handlers that must survive a re-run still do so by their own means: the projection handlers carry generation guards that drop a snapshot older than the one published, and prefix_delete is idempotent for a correctly-selected prefix. The pre-existing key-reuse and split-backend cases are unaffected by this.

Per-handler maximum run time

Every RegisterHandler call carries a ceiling, set at roughly ten times the handler’s expected worst case. It is the handler’s context deadline, and it is what the lease is derived from — the lease is ceiling + 30s, which makes ceiling + 30s the crash-recovery wait for the type. It exists to catch a hung handler, never to budget a working one: a wedged handler would otherwise hold its slot until the process restarted.

Worst cases marked measured come from the load tests on branches loadtest-inference-scale (docs/reference/load_test_inference_2026-08.md, up to 10k stacks) and loadtest-inference-10m (docs/reference/load_test_inference_10m_2026-08.md, the 20k runs), plus the design note on branch design-chunked-rebuild (docs/reference/chunked_rebuild_design.md). None of those branches is merged, so none of those paths exists on main.

The ratios are deliberately not uniform, and a flat 2–3× rule would be worse than useless here.

• The two rebuild walks run in resumable two-minute slices, so an ordinary run is a slice. 10 minutes covers the slice plus one of the two things that can overrun it — a pathological stack, or a page read the jobs lane bounds at 5 minutes — and not both. The expected overrun is the stack, and resource_index_reindex / dependency_graph_reconcile run that same per-stack work under a 5-minute ceiling.

  Slicing is what removed the size cliff. Before it the whole org ran under one 30-minute ceiling, and run S4 of docs/reference/load_test_inference_10m_2026-08.md (branch loadtest-inference-10m) was still going at 600 s for 20,000 stacks, an eighth into a convergence pass that revisits nearly every stack — putting the index rebuild’s cliff near 34k stacks, past which an org dead-lettered rather than running slowly. The graph rebuild’s own figure was 46 s measured at 10k stacks (branch loadtest-inference-scale, docs/reference/load_test_inference_2026-08.md), taken against synthetic analysis.json files carrying no dependency section, so it is a floor; that report puts production nearer the index rebuild’s cost (~5×). Same shape of cliff, further out. The remaining exception is the final reverse reap: it still materializes the org’s live stack ids and runs one (index) or two (graph) whole-org deletes. It has not been sliced or separately load-tested.

  Slicing does not make dead-lettering impossible. The claim burns an attempt and only AdvanceCursor resets it, so every path returning before the cursor is saved accumulates them: a failing reap, a first-page read that keeps failing, the not-advancing assertion. A re-claimed slice resumes from the last persisted cursor, and a dead-lettered row is not liveRowPredicate-live, so the next sweep enqueues a fresh rebuild.

• explore_diagnostics is sized for a topology, not for its 9-second measurement. Under a transaction pooler the jobs lane’s session statement_timeout does not apply, and this ceiling is then the only bound on its org-wide scans.

• explore_purge and seat_cap_audit sit far above 10× simply because five minutes is a sensible floor; 10× of ten seconds is not a ceiling anyone could act on.

A ceiling hit is recorded as outcome timeout. What happens next depends on what the handler returned:

• an error wrapping its context’s DeadlineExceeded — an ordinary failure: backoff, retry, dead-letter at 20 attempts. Handlers that cannot make progress should return ctx.Err();
• continue (false, nil) — recorded as timeout and logged, then released rather than backed off. Whether it can dead-letter depends on whether the handler advanced: both rebuilds only continue after AdvanceCursor, which resets the attempt count, so they cannot; one that continued without advancing keeps the attempt and reaches 20, which the processor warns about;
• done (true, nil) — recorded as success. A handler that swallowed the cancellation and reported completion is indistinguishable from one that finished.

Retries, attempts and dead letters

attempt_count is incremented by the claim, not by the outcome. That single choice is what makes crash handling free: a worker that dies mid-handler runs no code at all, but the next claim that takes over its expired lease burns the attempt anyway, so a task that reliably kills its worker stops at AsyncTaskMaxAttempts (20) instead of looping forever.

• A handler error records last_error, pushes run_at forward by a capped exponential backoff (15 s doubling to 10 min), and clears the lease so the backed-off row is claimable again at run_at rather than at lease expiry. A failing task drops to the tail of the queue instead of head-of-line-blocking unrelated work.
• At 20 attempts the row dead-letters: it stops being discovered, stays in the table for debugging, and is counted on gantrycd_async_tasks_dead. A dead letter is history, not pending work — it never suppresses a later enqueue for the same resource.
• A slice that advances resets attempt_count to zero (AdvanceCursor). The counter has to mean “consecutive slices that got nowhere”, or a rebuild taking four slices would spend four of its twenty attempts on succeeding. A slice that cannot advance does not call it, so a permanently stuck task still dead-letters on schedule.

Retention

Dead letters are the only rows nothing else removes — a successful task is deleted outright and a failing one is rewritten in place — so without a bound the table grows forever for an org with a permanently failing type. The hourly async-task-cleanup job prunes them past GANTRYCD_ASYNC_TASK_RETENTION (7 days by default), measured from last_attempted_at.

What that costs: past the window the row’s last_error and payload are gone, so a post-mortem older than a week has to come from the logs instead — the failure was logged with its task id when it happened, and log retention outlives this. It is the same trade the SCM-event and notification outboxes already make.

The predicate is the same deadRowPredicate the gauge uses, so a task at the attempt threshold under a live lease is never pruned: that row is on its final attempt with a worker on it, not abandoned.

A rollout costs each in-flight task one attempt, deliberately not refunded — and not for the reason it looks like. A worker that dies never reaches the release path at all. The attempt is kept because from the row’s point of view a rollout is indistinguishable from any other interruption: refunding would make a task that is always mid-flight at rollout immortal, its attempt count pinned, never dead-lettered, and invisible to the starvation alert because something is always running it. Resetting on progress — what the cursor advance does — is the right semantics for a genuinely long task.

Waking up

The wake pg_notify (async_task_pending) is fired by the repository’s enqueue statements themselves, inside the enqueuing transaction. So it is delivered exactly when the work becomes real, never for an enqueue that rolls back, and no caller has to remember to send it. PostgreSQL collapses identical notifications within one transaction, so a thousand-row enqueue costs one wake.

Every replica listens on one wake channel, which releases a single idle slot; that slot re-signals the channel as soon as it has claimed something, so a burst cascades across the pool one slot at a time. Waking every slot at once instead would put all of them into a write transaction on the primary for one enqueued task — four per replica, twelve across three, eleven of which find nothing.

An idle slot otherwise re-checks every 5 s, on a single ticker shared by the replica’s slots — a ticker delivers each tick to exactly one receiver, so a poll wakes one slot rather than all of them. Per-slot tickers created in the same instant stayed phase-locked for the process lifetime, so every poll woke all four and (slots − available rows) of them opened an empty write transaction.

The poll is the backstop, and where a read replica is configured it is load-bearing rather than paranoid: discovery runs against the replica, so a NOTIFY-woken slot may look before the row has replicated and see nothing.

Failure and shutdown

A replica dies. Its leases simply lapse — at each task type’s own ceiling, so five minutes for most types and up to an hour for prefix_delete. The next claim on any replica takes the row over, burning an attempt as it does.

A replica is rolled. In-flight handlers are cancelled and their leases explicitly released on a detached context, so another replica picks the work up within a poll interval — seconds, not the whole lease. If the release itself fails, expiry still gets there. The interrupted run is recorded as cancelled when the handler saved nothing and continued when it saved progress first; both are deliberately outside the failure|timeout alert set, so a rollout pages nobody.

A release does not refund the attempt the claim burnt. Not because a dying worker would exploit a refund — a dying worker never reaches the release at all — but because from the row’s point of view a rollout is indistinguishable from any other interruption. Refunding would make a task that is always mid-flight at rollout immortal: its attempt count pinned, never dead-lettered, and invisible to the starvation alert because something is always running it. (Resetting on progress, which is what the cursor advance does, is the right semantics for a genuinely long task.)

So a shutdown on a task’s final attempt retires it without anything ever having failed, and the release stamps last_error in that one case — otherwise the resulting dead letter would carry no explanation at all.

Upgrading to this design. For the duration of one rolling upgrade, replicas still on the previous version run the lease-blind listing and can execute a task a new replica holds a lease on — the pre-existing duplicate-execution bug, for one rollout. Old replicas’ work is unaffected otherwise: they ignore the new columns.

Prefix-delete safety

Before deleting, prefix_delete verifies the target prefix does not belong to a currently live stack, and reports done without deleting if it does. The check rests on StorageKey differing between a deleted stack and its recreation — which is not guaranteed: the key is generated at second granularity (stack_service.go), and a recreate 80 ms later was measured receiving the SAME key. That is a pre-existing hazard which this queue neither creates nor fixes.

explore_purge applies a similar rule but not the same one, deliberately: it reads the current owner from the PRIMARY, because a read replica that has not yet replayed the stack DELETE would report the row as live and retire the purge without doing it. prefix_delete’s equivalent check goes through the read lane.

Prefix-delete tasks are enqueued with a 30-minute run_at delay, long enough that a large rename cannot still be in progress when the cleanup fires.

Watching it

See Observability for the full list and the alert expressions. The four that answer different questions:

• gantrycd_async_tasks_pending{task_type} — unfinished work: still retryable, or on its final attempt and currently held by a worker.
• gantrycd_async_tasks_running{task_type} — work under a live lease. This is a subset of pending, not its complement; a task in flight is both. The complement of pending is gantrycd_async_tasks_dead.
• gantrycd_async_task_oldest_due_seconds — how long the longest-waiting claimable task has been due.
• gantrycd_async_task_slots — worker slots per replica; sum() is fleet capacity, and it is the denominator the starvation alert needs.

A high oldest-due on its own is not starvation: a whole-fleet sweep gives every org’s task the same run_at, so the gauge reads “time since the sweep started” until the last org drains, and at ten thousand orgs that is legitimately minutes. Starvation is work sitting due while the fleet is not even fully busy:

max(gantrycd_async_task_oldest_due_seconds) > 300
  and (sum(max by (task_type) (gantrycd_async_tasks_running)) or vector(0))
        < sum(gantrycd_async_task_slots)

The capacity clause matters in both directions. Without it, every whole-fleet sweep trips the alert while draining normally. But comparing against nothing running instead would miss the case the alert exists for: an orphaned lease counts as running for up to (ceiling + margin) — an hour for prefix_delete — so one crashed worker would suppress the alert for an hour. One busy slot out of twelve is the shape to catch, and that needs a denominator.

or vector(0) is required rather than tidy: gantrycd_async_tasks_running is a GROUP BY, so with nothing running it exports no series and sum() is an empty vector.

Production alerts are Grafana-provisioned from the infra repo’s charts/grafana/values.yaml (alertingProvision.groups), so the three async rules have to be added there.

Related

• Object-store key conventions: internal/backend/services/storage/paths.
• Queue recovery for stuck deployments is not this system — see deployment_state_machine.md.