Cross-Stack Dependency Graph
A derived graph of which stacks and resources depend on which others, used to
surface circular dependencies across stack boundaries and to drive dependent
redeploys. Stored as a single Postgres edge table in the separate
gantrycd_explore database, off the critical primary. Circular dependencies are
found by an in-process strongly-connected-component (SCC) pass — see
Cycle detection at scale for the theory
and the benchmarks behind that choice.
- Repository:
internal/backend/repositories/dependency_edge_repo.go— theDependencyEdgeRepositoryinterface +PostgresDependencyEdgeRepositoryimpl. - Schema:
schema-explore/tbl_dependency_edges.hcl(one row per directed edge). - Domain types:
pkg/domain/dependency_graph.go; SCC inpkg/domain/dependency_cycles.go. - Config: shares the explore database (
EXPLORE_DATABASE_URL), which is required — the backend aborts boot without it. Availability is separate: a valid but unreachable explore DB does not block startup, and the graph paths error at query time (see Wiring status).
For a zoomed-out view of how groups of stacks connect (rather than individual
stacks/resources), see Stack Group Topology, which
aggregates these same edges by a gantrycd:group:<view> label.
- Discovery (runner-side):
internal/runner/discovery/+ the contract inpkg/contracts/dependencies.go. - Consumer (backend):
DependencyGraphServicereconciles after run completion (asynchronously) and serves the per-stack query (GET /api/v1/orgs/{org}/stacks/{stack}/dependencies).
End-to-end flow
- Runner scans
gantrycd:dependency:comments and uploads the snapshot in the run’sanalysis.json(dependenciessection). - Enqueue at completion (
DependencyGraphService.EnqueueReconcile, called inside the run-completion write transaction): durably enqueues adependency_graph_reconcileasync task (run-keyed) so it commits atomically with run completion and survives a crash. PR-plan / promoted-PR runs are not enqueued. The edge store is never touched on the completion path. The enqueue itself fires theAsyncTaskPendingChannelNOTIFY when it commits, so the reconcile drains in seconds. - Async reconcile (
TaskProcessorJob→DependencyGraphService.HandleReconcileTask→ReconcileFromRun, best-effort): fetches the snapshot, maps it to edges,ReplaceStackEdges, then reports done. It runs on one of the replica’s worker slots (GANTRYCD_TASK_PROCESSOR_CONCURRENCY, default 4) under an exclusive lease, so no two replicas reconcile the same stack at once. Ordering across a stack’s snapshots is not the processor’s job:ReplaceStackEdgescarries a generation and drops a snapshot older than the one already published, which is correct across replicas in a way a per-replica FIFO never was. A failure (e.g. the explore DB briefly down) backs the task off for a later retry. - Query API (
GetStackDependencies→DependencyGraphService.GetStackGraph): returns the stack’s bounded dependency neighborhood as{nodes, edges}— nodes carrykind/id/stack(grouped into per-stack boxes by the UI) plusin_cycle, and edges reference node keys and carryon_cycle. Cycle membership is decided server-side and stamped on, so the client renders rather than re-deriving it, and every component the neighborhood renders is flagged — not only the one the central stack belongs to. Two reads. The first is still ORG-WIDE, and it is worth saying plainly: the reduced pair set (OrgStackPairs) scans the org’s edges to group them into distinct stack pairs — ~159-205 ms for an org of 1,500,000 edges reducing to 6,000 pairs (measured, index-only). It is what gives the in-memory BFS its adjacency and the Tarjan SCC pass its components, and it is shared with the cycle banner, trigger demotion and topology. What the page no longer does is read the org’s edge list:EdgesAmongStacksfetches resource-level edges for the stacks the BFS reached and nothing else (~5-12 ms for a 17-stack neighborhood in that same org). Results are redacted per viewer (see Authorization & redaction). The cheap cycles-only counterpart (GET …/dependency-cycles→GetStackCycle) backs the stack-view warning banner. - UI — the stack’s “Dependencies” tab (
web/src/components/stacks/DependencyGraph.tsx, React Flow + dagre, lazy-loaded). The model is deliberately minimal: each stack is one box, the only things inside a box are its resource rows, and a dependency on a stack as a whole attaches to the box header itself (no stand-in “whole stack” row). Boxes are laid out by their inter-stack edges with each dependency to the left of what depends on it, so an arrow points leftward from a dependent to the dependency it relies on (org-management ← vpc-prod-billing) — the direction a Terraformfrom_*reference reads. The central stack is highlighted, and a box/row/edge on a cycle is flagged red with the cycles listed as warnings. A stack the viewer cannot read is drawn as a dashed, locked “Restricted stack” placeholder (its rows likewise locked) so the topology — and any cycle through it — stays visible without revealing its identity. The data→graph layout is the pure, unit-testeddependencyGraphLayout.ts.
Rebuild (backstage + periodic sweep)
The per-run reconcile keeps the graph current, but the graph store is derived
and can be wiped/recreated. The backstage per-org “Rebuild Explore metadata”
action (POST /api/backstage/v1/orgs/{org}/rebuild-metadata →
BackstageHandlers.RebuildExploreMetadata) requests the org’s Explore
rebuilds on demand. It enqueues three independent async tasks — dependency
graph, resource index, and ComputeOrgFindings — and returns
202 {"status":"enqueued"}.
It is not a coordinated refresh, and it guarantees less than its name suggests:
- the three tasks are separately deduplicated, so a diagnostics task already pending can run before the graph and index rebuilds this call enqueued;
- the processor moves on to diagnostics even if an earlier rebuild failed;
- no follow-up diagnostics task is scheduled after the rebuilds land.
So it shortens the wait for the next sweep rather than bounding it, and Insights
is not current when the call returns. No HTTP endpoint recomputes findings
synchronously — the only synchronous caller of ComputeOrgFindings is the
sample-dependencies operator CLI (cmd/cli/operator/advanced/sampledeps),
which is a seeding tool, not an operator lever for a real org.
A single orchestration task running the three in sequence would be both simpler
and actually able to promise what the name implies.
The rebuild itself makes the org’s graph self-healing in both directions:
-
Forward push — re-derive each stack’s edges from its latest successful configured-branch run (
RunRepository.LatestAuthoritativeRunsAfterunderAuthorityDeclaredSource, best-effort per run). A run that fails to reconcile is logged at error level and skipped, and the cursor moves past it, so one unreadable stack cannot pin the org. The origin filter (DeploymentOriginType.RunsConfiguredBranch: manual, push, dependency) must stay in lockstep with the inlineEnqueueReconcileguard — if the sweep admits fewer origins than the enqueue does, it silently reverts the edges the enqueue just wrote, on a 12h cycle.Successful is meant literally here, and it is where this authority parts company with the resource index. The index follows a failed
applytoo, because a partial apply changed real infrastructure (see Resource Explorer). The graph does not: it is derived from the checked-out HCL, which a failed apply did not change — its declaration is identical to the run that planned the same commit — and the run-completion reconcile only fires for successful runs, so admitting the failure here would make the sweep write edges nothing else writes. -
Reverse reap — garbage-collect edges owned by stacks that no longer exist in Postgres (
StackRepository.ListIDsByOrgis the authoritative live set →DependencyEdgeRepository.ReapStacksNotIn, aDELETE … WHERE dependent_stack_id NOT IN (live)). The push only touches stacks that still produce a run, so it can never remove these on its own; this closes the gap left when a stack delete’s best-effortDeleteStackEdgesnever ran. A read error fails the slice rather than passing an empty set (which would wipe the graph) — so a transient DB blip can never be read as “no stacks”, and a reap that never succeeds dead-letters instead of reporting a rebuild that did not happen.The reap deletes only edges owned by a dead stack (its dependent endpoint). An edge that merely points at a missing stack is a dangling reference, kept on purpose so the explore diagnostics can surface it as an orphan — not reaped.
The rebuild runs unattended as an eventual-consistency backstop. A
dependency-graph-rebuild scheduler job fires every 12h and calls
DependencyGraphService.EnqueueRebuildAll, which enqueues one org-keyed
dependency_graph_rebuild async task per active org — skipping any org whose
prior task is still unfinished, which includes one a worker is running on its
final attempt (EnqueueOrgTaskIfNotPending evaluates the check inside its own
INSERT), so a slow rebuild never piles up across sweeps.
The TaskProcessorJob drains those tasks via HandleRebuildTask, one
two-minute slice per claim. A slice pages the org by keyset in the database’s
stack-id order (never re-checked in Go — the two collations disagree on
hyphenated slugs), writes how far it got into the task’s own payload and reports
continue; that write clears the lease and resets the attempts in the same
statement, so any slot on any replica takes the next slice. Only the slice that
finds the walk exhausted reaps and reports done. The per-stack walk therefore no
longer has a whole-org deadline cliff: the 10-minute ceiling bounds one slice,
not one org. The final reverse reap remains whole-org; its residual scaling risk
is documented in Resource Explorer: Filtering & Scale.
An older payload reads as absent and the org restarts from its first
stack, which is always safe because a rebuild is idempotent. A newer payload is
refused without being rewritten: an older replica cannot safely interpret a
cursor shape written during a rolling upgrade. The payload carries a cursor and
nothing else — declared edges come from the stack’s own snapshot, so unlike the
resource index there is nothing to revisit once other stacks land. What a
slice does when a page read fails part-way, and why a lost lease is an error
rather than a continue, is the
handler contract in Async Task Queue.
This corrects drift the run-completion reconcile missed: the edge store being recreated, a dropped reconcile task, or an orphan left by a failed delete-time cleanup. Like every graph path it needs the explore database; if that database is unreachable the task fails and the queue retries it with backoff.
Lifecycle cleanup
Nothing else reclaims graph data, so deletes prune it directly. The prune is
durable: StackService enqueues an explore_purge async task inside the
same write transaction as the stack write, then attempts the removal inline as
a fast path and dequeues the task only when it succeeds. A purge failure therefore
never fails the stack write (the stores are lower-SLA) and never silently
leaves rows behind either — the task processor retries it, instead of the rows
lingering until the 12h reap. The inline dequeue is fenced on an EMPTY lease, so
it can only remove a task no worker has claimed; if one has, that worker owns
finishing it. (Enqueueing is a plain INSERT in that same
transaction, so if THAT fails the stack write fails with it, as it should.)
A task that exhausts its retries dead-letters, and those rows then wait for the
reap exactly as they used to.
That matters because leftover rows are not merely stale display data: they
still count as a live participant in cross-stack hazard detection, so a stack
that no longer exists can close a cycle for a live one — showing a circular
dependency nobody earned and demoting that stack’s always triggers to
manual, silently stopping its automatic redeploys — and can qualify a cloud
identity as a duplicate only one live stack actually holds.
Only the vacate events are durable (delete, rename away). When a stack is
created onto a reused slug or renamed into one, the purge clears a PREVIOUS
occupant’s rows while the new stack is live, and the stores carry no owner
incarnation to tell the two apart — so a retry that landed after the new stack
published would delete its data. Those paths stay inline best-effort, and their
window closes on the new stack’s first successful run, which replaces the owned
rows wholesale. The handler enforces the same rule: it skips (and retires the
task) once a live stack holds the id again, mirroring the prefix_delete
handler’s storage-key check.
The stores cleared on each event:
- Stack delete (
StackService.DeleteStack→DependencyGraphService.RemoveStack→DeleteStackEdges): deletes every edge touching the stack at either endpoint. Edges where the stack was only a dependency target then reappear when the owning stacks next reconcile. - Org hard-delete (
OrganizationService.HardDeleteOrg→DependencyGraphService.RemoveOrg→DeleteOrg): deletes every edge for the org.
Discovery (runner-side)
The graph is populated from gantrycd:dependency: comments authors write in
their OpenTofu code. On every plan/apply/destroy/refresh, once the repo
is fetched the ephemeral runner kicks off the scan in a background goroutine
so it overlaps the OpenTofu init/plan/apply instead of delaying it; it resolves
the comments against the current stack and, at the end of the run, writes
the snapshot into the dependencies section of analysis.json next to the run
logs (best-effort, like the rest of that object). The backend consumes it and
reconciles the stack’s edges via ReplaceStackEdges.
Comment grammar
A gantrycd:dependency: directive behind a # or // line comment (block
comments unsupported):
(#|//) gantrycd:dependency: key=value [key=value ...]
Values are bare (run to the next space) or double-quoted with \"/\\ escapes
(templates need this). A second #/// after the directive ends it, so
// gantrycd:dependency: from_stack=a // note (or a repeated directive after
it) keeps only the first part — but a #/// inside a quoted value is kept.
Field order is irrelevant.
dependency fields (identifiers are stack IDs and OpenTofu resource
addresses):
| field | meaning |
|---|---|
from_stack / from_stack_tpl | target stack ID (exactly one required) |
from_resource / from_resource_tpl | resource in the target stack (optional; absent ⇒ whole stack). May be an output.<name> reference — the companion to the gantrycd_state_outputs data source, for a stack that consumes another’s output. Scopes triggering too: the dependent only reacts when this resource (or output) changed in the applied plan — see Dependent triggers |
resource | resource in the current stack that depends (optional; absent ⇒ the whole stack depends) |
stack / stack_regex | only apply when the current stack ID matches (optional) |
trigger | redeploy policy when from_stack successfully applies: always, manual (default), or never — see Dependent triggers |
trigger is a closed enum with no _tpl variant by design: a redeploy
policy must be static and reproducible, never derived from rendered stack
metadata. An invalid value rejects the whole comment like any other field.
Two comments that resolve to the same edge but disagree on trigger are
de-duplicated by combining their policies (manual > never > always), so
file order never picks the winner.
When resource is omitted, the comment adopts the block it sits directly on top
of — the next non-blank, non-comment line — so these are equivalent:
// gantrycd:dependency: from_stack=aws-vpc-dev-euw1 resource=module.this
// gantrycd:dependency: from_stack=aws-vpc-dev-euw1
module "this" { ... }
module "x" ⇒ module.x, resource "t" "n" ⇒ t.n, data "t" "n" ⇒
data.t.n. Only that one next line is inspected; if it is not a
resource/data/module block (or there is none), the dependent falls back to the
stack itself (a stack-to-stack dependency).
*_tpl fields are Go templates over {{ .stack.ID }}, {{ .stack.Name }},
{{ .stack.Labels.<key> }}, with a curated allowlist of pure sprig string
functions (no env/now/random/crypto, and no count-based repeat — output
must be deterministic and bounded). The current-stack Name/Labels reach the
runner via RunContext fields.
resource-identity comments
The same scanner handles a second verb, the configuration-side twin of the
gantrycd:resource-identity-id = "null" tag:
(#|//) gantrycd:resource-identity: [resource=<address>] ignore=<true|false>
| field | meaning |
|---|---|
resource | address in the current stack: a resource (with or without its instance key), a data source, or a whole module.x. Optional; absent ⇒ the block the comment sits directly on top of. A comment above nothing is an error — there is no whole-stack form (that is default_tags). |
ignore | required. true: gantrycd asserts no cloud identity for the address — a managed resource is never an owner (no duplicate finding, no inferred dependency onto it) and a data source is never a reader (no inferred dependency from it). false: re-asserts inside a broader ignore=true (the most specific covering directive wins). |
It exists for the addresses a tag cannot reach: resource types that take no
tags, and data sources, whose recorded tags are the object’s, not the
author’s. Two directives for one address are an error naming the first. The
directives ride in the same dependencies snapshot and the backend applies
them to the state projection of the same run — so a run whose configuration
could not be scanned at all (nil snapshot) projects its state without them
until the next successful scan, and says so in the log. Holding the projection
back instead would freeze the stack’s resources and inferred edges over a
missing comment section. A promoted pull request’s apply is a real apply: its
comments apply to its own state.
Rules and guarantees
- Best-effort, never blocks the run. A scan/upload failure is logged and
swallowed. The channel
startDependencyDiscoveryreturns carries a*contracts.DependencyDiscovery, so “unknown” is representable — which is what the next rule turns on. - Reading the configuration and parsing a comment fail differently. If the
scan cannot read what the stack declares —
ReadDirfails, a file will not open, a line exceeds the 1 MiB scanner limit, or the scan panics — it reports nil, unknown. It learned nothing about the stack, and by the authoritative-snapshot rule below, the subset that happened to parse would delete every edge declared by the files it never read. A comment that was read and rejected is the opposite case: it is a fact about the configuration, so the snapshot stays authoritative and the comment becomes anErrorsentry. Keeping a staletrigger=alwaysbecause an author mistyped one comment is the worse failure, which is why parse errors do not suppress the snapshot. - All-or-nothing per comment. A malformed field, unknown key, bad template,
or invalid regex makes that comment an
Errorsentry — never a partial dependency, never a default. Astack/stack_regexthat simply doesn’t match the current stack is a silent skip (not an error). Each entry is printed to the run log asfile:line: message; the comment text itself is not echoed. - Hostile input is rejected, not recorded. Resolved identifiers are
charset-validated (stack ID
[A-Za-z0-9._-], resource address adds.[]"), so a value carrying shell/template injection, spaces, or control characters becomes an error rather than data. Field length and rendered-output length are capped. - No self-references. A directive that resolves to the very stack or
resource that declares it is a meaningless self-loop and becomes an
Errorsentry, so the author sees it. The backend drops any that slip through anyway. - Complete snapshot. The contract is uploaded even when empty, so the backend can reconcile a stack down to zero edges.
- A missing snapshot is not an empty one.
ReplaceStackEdgesis destructive: it replaces a stack’s edges wholesale. The upload is best-effort, so a transient storage failure is expected — and if that were read as “this stack has no dependencies”, one failed PUT would delete every edge the stack has. So thedependenciessection is a nilable pointer: nil (or noanalysis.jsonat all) means unknown andReconcileFromRunreturns without touching the graph; a non-nil section is authoritative even when its list is empty, and that is what clears a stack’s lastgantrycd:dependency:comment. The resource index applies the same rule tostate_resources. Seecontracts.RunAnalysisandrun_analysis_absence_integration_test.go.
Known limitations (deliberate)
- Scanning is line-based, so a
# gantrycd:dependency:inside an HCL string/heredoc is a false positive. Accepted for simplicity. - Scope is the working directory’s top-level
.tf/.tofufiles only (non-recursive, mirroring how OpenTofu loads a root module). Comments in nested module directories are not scanned — put them in the root module. - The injection charset rejects a few legitimate-but-unusual resource addresses
— notably index keys containing spaces (
data.x["my key"]). This is a deliberate trade of completeness for a tight guard; such a comment becomes an error.
Why a separate, lower-SLA store
This is an analytical feature, not part of the deployment critical path. Treating it as a derived index buys a lot of simplicity:
- Postgres stays the source of truth. The graph is rebuildable from runs; staleness, downtime, and full recreation are all acceptable. There is no distributed transaction with the primary and no drift problem to police.
- It runs off the critical DB. The edges live in the
gantrycd_exploredatabase (alongside the resource index), not the primary that runs deployments, so the feature carries a smaller SLA without dragging anything else down. - The graph is small. Edges track cross-stack coupling — declared comments,
plus one edge per cross-stack data source — not the
resource count. The SCC pass consumes the graph already reduced to distinct
stack PAIRS (
OrgStackPairs), which is smaller again: a million resource-level rows between the same handful of stacks is a few thousand pairs. This is what lets cycle detection run in-process in Go rather than needing a graph engine — see Cycle detection at scale for the measurements (a dedicated graph DB was worse at complex cycles, because the deployed engine has no SCC primitive and its closed-walk explodes on tangled graphs).
The seam (swapping stores later)
This follows the normal repository pattern: callers depend on the
DependencyEdgeRepository interface; the only store-specific code is
PostgresDependencyEdgeRepository. Everything in and out is a pkg/domain type.
Like the resource-index repo it owns its explore-database pool rather than taking
a caller transaction — the explore database is a distinct connection that core
write paths cannot thread their transaction through.
Data model
dependency_edges (schema-explore/tbl_dependency_edges.hcl), one row per
directed edge, org-scoped, no foreign keys (a derived, recreatable store):
- Endpoints are stored structurally:
dependent_{kind,id,stack_id}anddependency_{kind,id,stack_id}.kindisstackorresource; a stack node hasid == stack_id, a resource node’sidis its OpenTofu address scoped by its owningstack_id(two stacks containing the same addressmodule.thisare distinct nodes).GraphEntity.Key()(stack:<id>/resource:<stack>:<id>) is the in-memory node identity used by the SCC/BFS. - One
triggercolumn (always|manual|never), defaulting tomanual. - One
discoveredboolean marking provenance:truefor an edge gantrycd inferred from the cloud objects the stack reads,falsefor one a human declared (see Inferred dependencies). - Primary key is the full edge tuple, ordered so
(org_id, dependent_stack_id)is a prefix — that backs the reconcile delete and any per-dependent scan.discoveredsits immediately after it, so declared and inferred rows are disjoint sets each writer can delete-and-replace without touching the other’s. - Two further indexes:
(org_id, dependency_stack_id)backs inbound (dependent-trigger) lookups, and a partial(org_id, dependent_stack_id, dependency_stack_id) WHERE dependent_stack_id <> dependency_stack_idcovers the reduced stack-pair read behind cycle detection and topology. Reordering either one is a DROP + CREATE, and Atlas does not emitCONCURRENTLY: the CREATE holds aSHARElock ondependency_edgesfor its duration, blocking every reindex still writing edges — including from old pods mid-rollout. On a large explore store, apply it in a quiet window rather than as part of the deploy. CHECKs enforce the kind/trigger enums and the stack-node identity.
Idempotent per-stack reconciliation
ReplaceStackEdges(org, stackID, edges) is the write primitive for declared
edges, and the only one. Callers pass the complete declared edge set
originating from a stack every time, so repeated calls converge. Its delete is
scoped to NOT discovered and its inserts stamp false, so it never touches the
inferred rows the state projection owns — and the
reverse holds, scoped the other way.
Edge ownership rule: an edge belongs to the stack of its dependent (source)
endpoint. Replacing stackID deletes every row whose dependent_stack_id is
stackID, then bulk-inserts the supplied edges (deduplicated) with
dependent_stack_id bound to stackID — so the rows are always covered by the
delete and the result is a pure function of the input. Edges where stackID is
only a dependency target are owned by other stacks and left untouched.
One transaction. The delete + insert run in a single write transaction, so a
reader never sees the stack mid-rewrite and a crash cannot leave edges
deleted-but-not-recreated. A transaction-scoped pg_advisory_xact_lock on
(org, stack) serialises concurrent reconciles of the same stack (the task
processor can claim two), so they queue rather than racing into a PK conflict.
Invalid edges are dropped, not fatal. The service validates each edge
(DependencyEdge.Validate — well-formed endpoints, no self-loop) before the
write and drops + logs any that fail, rather than aborting the reconcile — one
bad comment can’t freeze a stack’s dependency view.
Cycle detection
A cycle is a stack-level fact. A stack applies atomically, so the deadlock is
“A must be applied after B and B after A” — whichever resources carry it. And the
resources cannot carry it alone: the hop that closes the loop runs inside a
stack and is never an edge. platform reads infra’s cluster while infra
reads platform’s role — four distinct resource nodes, two disjoint edges, no
resource-level loop, yet neither stack can be deployed first.
So CyclesFromStackPairs (pkg/domain/dependency_cycles.go) works on the graph
already collapsed onto owning stacks (StackPairsFromEdges does that for the one
caller still holding rows), then decomposes into strongly-connected components
(Tarjan): a circular dependency is exactly an SCC of more than one stack. One
O(V+E) pass over the org’s edges yields each entangled group once, and a
DependencyCycle is simply that group’s sorted StackIDs.
There is deliberately no path. At stack granularity a single loop through a
component can omit members that are equally circular — for {A⇄B, A⇄C} any loop
names two of the three — so a stack could be told it is in a cycle whose reported
path leaves it out. A component of one stack is never a cycle, which is also what
makes an intra-stack declaration (legal, and not a deployment-ordering statement)
collapse harmlessly instead of reporting its stack as circular.
The per-stack cycle warning reads the graph already reduced to distinct
cross-stack pairs (OrgStackPairs) and filters the SCC result to the stack; the
org-wide explore diagnostics run the same SCC over the edge set they load for
orphans anyway, falling back to the pair read only when that edge set is over
budget. There is one algorithm and no depth bound — SCC detects cycles
of any length.
Last-known rows remain part of the graph. Trigger reads return freshness once per child and use it only to demote that child’s automation, so an unreadable scan does not change the org’s cycle answer.
Every read is LIMIT-capped so the backend never materialises an unbounded
graph: OrgStackPairs at maxOrgStackPairs, OrgEdges (the diagnostics sweep)
at maxOrgEdges, EdgesAmongStacks (one page’s neighborhood) at
maxNeighborhoodEdges. The last two are both 200,000 and deliberately separate
constants — one is about an org, the other about a page, and a single shared
number drifts into meaning whichever the reader assumed.
maxOrgStackPairs is now the org-shape ceiling for all four surfaces that
ask what depends on what: the cycle banner, trigger demotion, the topology view,
and the Dependencies tab’s neighborhood walk. It was raised from 50,000 to
200,000 when the tab moved onto this read, because at the old number an org of
6,000 stacks with 10 declared dependencies each — 60,000 pairs, 60,000 edges —
rendered before the move and would have shown one box after it. 200,000 pairs is
~8 MB materialised, and the limit decides whether the question is answered, not
what asking costs: that 60,000-pair org groups in ~16 ms at either limit, and
200,000 pairs in ~75 ms. The raise is a strict improvement for the three
surfaces that already shared this read.
The Dependencies view is deliberately not bounded by the org’s edge count: it
reads the edges among the stacks it draws, so maxNeighborhoodEdges bounds one
neighborhood there (and is spent on the owner-side scan, which is the work).
An org can hold many times that number of edges and still render each of its
pages. Loading the whole org to keep the slice touching one stack is what made a
large org serve every stack as a lone box.
The response is capped separately, and it has to be: the read cap bounds what the backend loads, not what it returns. The neighborhood read keeps every edge between the stacks it reached, and edges are resource-level — so a hub topology (every app declaring many resource dependencies on one shared stack) put the org’s whole edge set into one page load. Measured before the caps: a 200k-edge org returned 51 MB and 202k React Flow nodes for a single stack’s view, with only 101 stacks in the component. The dangerous axis is edge multiplicity between few stacks, not stack count, and generated HCL produces exactly that.
capNeighborhoodEdges bounds it at maxResponseEdges, and is a no-op on any
neighborhood that already fits — the ordinary org keeps every edge and sees no
notice. Over budget it fills fairly: one edge per (dependent, dependency)
stack pair first, then widens each pair up to maxPairEdges. Filling greedily
in arrival order instead spends the budget on whichever stacks come first and
returns nothing for the rest, and since the node set is derived from these
edges, a dropped pair is a stack box that silently disappears — including,
potentially, the stack whose page it is.
Order is imposed rather than inherited: the edge read has no ORDER BY, so
arrival order is heap order and shifts when a stack reconciles and rewrites its
rows. Without the sort, two loads of one page could disagree about which stacks
exist. Within a pair, edges carrying a non-manual trigger sort first — an
always subscription is the one thing a reader cannot infer from the picture.
TotalEdges and TotalStacks are both counted on the UNTRIMMED neighborhood,
the same order the diagnostics sweep uses for its hazard sets and for the same
reason: a display cap must not be able to change what the totals say. The view
reports “showing N of M dependencies”, and when a neighborhood holds more stack
pairs than the whole edge budget — the only case where stacks are lost too — it
says that as well instead of implying the drawing is complete.
(For the reduced read the cap bounds memory; the ordered index lets the scan stop
early once enough distinct prefixes exist, but a duplicate-heavy org under budget
still scans its raw edges.) A pathological org over either cap degrades gracefully
rather than risking a serving-pod OOM: the per-stack graph serves the central
stack only — now reachable only when the ORG’S SHAPE is over maxOrgStackPairs,
or that one neighborhood’s own edges are over maxNeighborhoodEdges — and the
diagnostics sweep reports each check separately. Cycles and
orphans normally come out of the SAME edge read, so an org within
maxOrgEdges gets both from one snapshot and they cannot disagree. The
reduced pair read is the FALLBACK: only when the edge read is over budget does
the sweep take it, so such an org loses orphans (orphans_complete=false) but
keeps its cycles — and loses cycles too only when the pair read is over budget
as well. The cycle
warning reports unknown rather than nothing, and while cycle status cannot be
determined, newly resolved, unskipped always rows fail closed to manual —
“could not tell” must not read as “no cycle” (see
Cycle demotion). See
Cycle detection at scale for why this beats
the closed-walk it replaced.
Orphan reference detection
The explore diagnostics flag declared dependencies whose endpoint does not
currently exist — a cross-store anti-join of graph edges against the live
stacks (core PG) and the resource index (explore DB), in
ExploreDiagnosticsService.detectOrphans. Declared only, literally:
inferred edges are skipped, because a finding here
asks someone to correct a reference and nobody wrote those. A whole-stack
endpoint is checked
against the live/indexed stack sets; a resource endpoint is checked by
address, and how that address is resolved depends on its shape:
- A resource instance (
aws_sns_topic.main,module.network.aws_vpc.this) must match an indexed row’s(stack_id, address)exactly (ResourceIndexRepository.ResourceExists). - A whole-module reference (
module.network, or nestedmodule.network.module.subnet) points at the module unit, not one address. It exists when any indexed resource lives inside that module subtree (ModuleExists— a hierarchical-coverage match: the resource address continues the module path at a.or[boundary, the same coverage rule the changeset filter uses).domain.ModuleReferencePathclassifies the shape: an address that is purelymodule.<name>steps is a module reference; anything else is not. - An output reference (
output.service_endpoint) is a stack-level value the resource index never holds (it stores resources only). It is resolved against the producing stack’s applied state instead: an output exists once its stack has indexed resources, so it only reads as missing (always pending — a removed output can’t be detected without output rows to anti-join, and a gone stack is already caught above) while the stack has no applied state yet.domain.IsOutputReferenceclassifies theoutput.<name>shape.
This distinction is why module.network and output.service_endpoint are not
orphans merely because no resource is literally addressed that way. A malformed
or type-only address — a bare resource type with no instance name
(aws_sns_topic) or a dangling module keyword (...module) — is neither a
module nor an output reference, so it keeps exact-match semantics and stays
flagged: it is not a real reference. A flagged reference reports its
missing_kind ("output" / "module" / "resource" / "stack") so the UI
message reads accurately.
Inferred dependencies
Not every edge is declared. When a data source in one stack reads a cloud object a managed resource in another stack owns, gantrycd records the dependency itself — nobody writes a comment, and the two stacks never mention each other in HCL.
# stack "account" owns it
resource "aws_acm_certificate" "cloudfront_wildcard" { ... }
# stack "taloscluster-ovh-eks-1" reads it — and now depends on "account"
data "aws_acm_certificate" "wildcard" { domain = "..." }
The evidence is resource identity: the same globally-anchored value that backs duplicate detection. Two rows describe the same object when their id, realm and resource type all match, so the match key is exactly that, against managed rows of the same org in other stacks. Every matching owner yields its own edge — when two stacks manage one object (a duplicate) the reader really does depend on both, and collapsing that would name a winner the evidence does not.
Only a provider with a mounted identity rule can be inferred from, and today that is AWS alone — resources from any other provider assert no identity, so they never match and never produce an inferred edge.
-
Resource-scoped on both ends. The dependent endpoint is the data source’s address, the dependency endpoint is the owner’s. So the parent’s changeset scopes it exactly like a declared
from_resource=does: the dependent is offered a trigger only when the applied plan actually changed the resource it reads (see Dependent triggers). -
Always
manual. An inference never asks for automation;alwaysis only ever a human’s declaration. It does still carry a policy, andmanualcombines: a declaredalwayson one of the parent’s resources beside an inference on another — both matching the applied changeset — resolves tomanual, because manual is the safe middle ground when the evidence disagrees. The row says so, naming the inferred address, rather than going quiet (domain.HeldByInferenceFormat). Declaring a trigger for that resource is the fix. -
A declaration wins trigger behavior, by address coverage.
domain.EffectiveTriggerEdgesignores an inferredmanualpolicy when a declared edge for the same (dependent stack, dependency stack) has a dependency address that covers the inferred one — the sameResourceAddressCoversPathrelation changeset filtering uses. So: the same address, an ancestor of it (a declaration onmodule.certsspeaks formodule.certs.aws_acm_certificate.this["prod"]), or the dependency stack as a whole. A sibling resource does not override it; that would let an unrelated declaration change this dependency’s trigger behavior.This precedence is applied only inside trigger resolution, before policy combination and changeset filtering. It is not applied to graph or diagnostic reads: the inferred dependency remains a fact even when a broader declaration governs how it triggers.
Storage retains the declared and inferred rows independently. The Dependencies response (
MergeDependencyGraphEdges) unions rows only when their endpoints are exactly equal and reports provenance asdeclared,inferred, orboth. Coverage is not equality, so a module- or stack-scoped declaration that governs a resource-level inference does not merge with it: the view draws two arrows between the same pair of stacks, onedeclaredat the declared scope and oneinferredat the resource scope. That is deliberate — the declaration and the inference are different statements about different endpoints — but it is why a stack pair can show more arrows than a reader expects. -
Counted like any other edge for cycle detection, the graph view, and the org topology. Inferred evidence renders dotted; an exact union says
declared + inferredand retains the declared trigger policy. -
Never an orphan, never unsynced. An orphan finding asks someone to fix a reference they wrote, and nobody wrote these. Nor can an inference be “unsynced”: it only exists because the reader’s lookup and a managed owner were both indexed, so a target that later reads as absent — deleted stack, removed resource, or a stack with no indexed state — is the normal lag between an owner changing and the reader’s next reindex withdrawing the edge. Explore → Insights therefore builds both its Orphan and Unsynced references from declared edges only (the historical findings payload stores both under
orphans, distinguished bypending=true). The accepted trade-off: an owner importing a resource into state is not visible as an inferred edge until the periodic rebuild republishes the metadata. -
Bounded. Both raw addresses of an inferred edge sit in one btree primary key, and a pair past Postgres’ 2704-byte maximum fails the INSERT — which shares the state-projection transaction, so it would wedge that stack’s whole Explore projection on every retry. Either address over
maxInferredEdgeAddressLenrefuses the inferred-edge update (logged) and retains the last complete set, the same trademaxDeclaredIdentityLenmakes for an oversized declared id. Fan-out is data rows × owners, so it scales with the resource fleet rather than with anything a human configured.maxInferredEdgesPerStackbounds all three stages: the identifiable data reads collected before SQL, the cross-stack owner rows SQL may return (LIMIT cap+1), and edge expansion itself. No stage publishes its arbitrary prefix. The latest resource rows still advance, the last complete inferred edges remain visible, and the state generation marks them evidence-unknown so a narroweralwayscannot fire and a narrowernevercannot hide a possible manual trigger (that escalation says “manual deploy offered”, not “auto-trigger disabled” — nothing was withheld). A declared whole-stack policy still wins: it already governs every possible inferred dependency in that parent stack, so incompleteness cannot weaken it.
Who writes them, and when
The reader stack’s reindex does, in the same transaction as its index rows
and under the same state generation:
ResourceIndexService.ReindexFromRun computes each data row’s identity
transiently, resolves the distinct keys through one
ResourceIndexRepository.ManagedByIdentities query, and hands
domain.InferEdges’ output to ReplaceStackStateProjection. Passing an empty
set is how a stack that dropped its last cross-stack data source loses its
inferred edges.
Transient is deliberate: resource_index.cloud_identity_id stays NULL for
data rows, because a stored data-side identity would group the reader with the
resource it reads and report every cross-stack lookup as a duplicate. The
identity is derived, matched, and dropped.
The 12h resource-index rebuild uses a bounded two-pass order. The first, stack-id-ordered pass indexes every stack and records which ones made an identifiable data lookup. Once the org’s owner rows are all present, the second pass revisits only those readers. That makes an empty or partially restored Explore store converge in one sweep regardless of whether a reader sorts before its owner, without doubling work for owner-only stacks. There is no inference job or persistent staging projection of its own. The same cadence is the bound on the owner side during ordinary operation: an object newly managed by another stack produces an edge once the reader next reindexes.
Both passes are resumable: the rebuild runs in two-minute slices carrying
phase, cursor and the reader set in the task’s own payload. “Visits only the
readers” holds only below the payload’s 1,000-id cap — past it the second pass
walks the whole org. See
Resource Explorer: Filtering & Scale.
Declared and state projections have separate generations. A failed configuration read marks only declarations last-known; bounded or unstoreable inference marks only state. Trigger resolution reads both once per child because either can change the policy result.
Rolling upgrade. Pods predating inference may delete inferred edges during a reindex and do not enforce the newer freshness or budget holds. A mixed-version fleet can therefore briefly auto-trigger a dependent that a new pod would hold for review. This is accepted without a compatibility gate.
What this is not
gantrycd briefly inferred edges from gantrycd_state_outputs lookups, by data
source type name plus a syntactically valid stack_id. That was withdrawn
and stays withdrawn. Applied state records the producer’s organization, stack and
run, but NOT which gantrycd backend served them, and the provider lets any stack
point at any endpoint — so two installations sharing a database lineage (a
staging clone, the far side of a migration) hold identical ids, and a stack
reading the other one’s outputs produced an edge that looked local and was false.
It could not deploy anything unattended, but it rendered a Deploy button on the
wrong parent’s page and could close a false cycle. “Needs a human to click” is
not harmless.
An ARN cannot fail that way: it names a real object, not an installation. What it
can fail on is an unverified endpoint — an AWS-compatible API emits well-formed
ARNs and the endpoint appears nowhere in state — which is what
gantrycd:resource-identity-realm is for, and its remaining bound is that it
must be declared. See
Resource identity.
Re-enabling gantrycd_state_outputs discovery still needs a stable backend realm
identifier recorded by the provider and verified on read. Until then, a
gantrycd:dependency: comment remains the way to express a dependency gantrycd
cannot see in state — an output reference, an ordering constraint, or any
relationship that leaves no shared cloud object behind.
Dependent triggers
The trigger= field turns the graph from purely informational into a
(carefully bounded) automation surface: when a plan-mode, non-PR-origin
deployment finishes with a successful apply_run stage, the backend resolves
the stack’s direct dependents and either redeploys them automatically (always),
offers a button on the parent deployment’s page (manual, the default), or just
lists them (never). Full lifecycle details (qualifying condition, idempotency
claims, async task) live in deployment_state_machine.md — this section owns the
resolution semantics.
Resolution rules
DirectDependents(org, stackID) returns the stack’s inbound star
(InboundCrossStackEdges): every edge from another stack’s entity onto the stack
or one of its resources (intra-stack edges excluded). The execution read returns
only whole per-dependent-stack edge sets: at most 10,000 edges for one child
and 50,000 in aggregate. A child outside either budget is not dropped and no
arbitrary prefix is evaluated; it gets a durable manual row explaining that
the relationship was too large to resolve automatically. Oversized children do
not consume the aggregate budget, so one pathological stack does not hide
ordinary dependents that sort after it. This bounds deployment-path memory while
preserving the operator’s Deploy button. The irreducible output remains one
stack id and one snapshot row per direct dependent; the cap bounds detailed edge
materialization, not the number of real relationships the product must show.
Complete edge sets are grouped by dependent stack and each group collapses
to one effective policy (domain.ResolveDependentTrigger):
-
Changeset filtering. An edge that names a parent resource (
from_resource=) only counts when the applied plan actually changed that resource. The apply’s plan-changes artifact (resolved through the apply run’splan_run_id, so a targeted replan’s narrowed plan is used as-is) is matched against the edge address: equal, or a hierarchical prefix ending at a.or[boundary —module.vpccoversmodule.vpc.aws_subnet.a[0],aws_instance.webcoversaws_instance.web[0], but notaws_instance.webserver; index keys compare exactly with no quote normalization, andoutput.<name>entries match output-scoped addresses. Every changeset entry counts, whatever its change type. Edges depending on the whole stack always count — depending on the stack means “anything in it” — and an edge set with no resource-scoped edges never reads the artifact at all. Filtering happens before the policies combine, so aneveredge whose resource did not change no longer vetoes analwaysedge whose resource did. A child whose every edge filtered out keeps an informational skipped row (skipped_reason), with its policy resolved from the unfiltered edges so an all-neverchild still reads (and gates the button) asnever; skipped rows never auto-trigger and never enter automation, but the Deploy button doubles as “deploy anyway” and clears the skip. An unreadable or incomplete artifact does not filter dependencies away, but it also cannot justify unattended automation: a resource-scopedalwaysis retained as a manually actionable row with an explicit changeset-unknown reason. Transient storage errors make resolution retry via the async task / reconcile sweep. Nothing re-reads artifacts later: triggers fire at resolution time, so the decision and the evidence share one moment.Why incompleteness is checked, not just readability. An analysis that failed still uploads
changes: []— a well-formed, empty change list that is byte-identical to “this apply changed nothing”. Trusting it filters away every resource-scoped edge, so an apply that really did change things would silently drop the dependents it should have triggered. Readability cannot tell the two apart; onlyPlanSummary.CountsComplete()can, so the summary is consulted before the detail (appliedPlanChangedPaths). The same applies to a narrowed (-target) apply: its changeset says nothing about the resources it never examined. A complete analysis with an empty changeset is real evidence that nothing changed, and still filters normally. -
Declared policy precedence on two explicit scope axes. Declared and inferred edges remain independently stored and visible in the graph; only trigger behavior is narrowed here.
- On the dependency (parent) side, within the same child/parent stack
pair, a declaration suppresses an inferred
manualwhen its target is the same address, an ancestor module, or the whole parent stack. A sibling resource is not covered. - On the dependent (child) side, if any surviving declaration scopes the
policy to the whole current child stack (a comment with no
resource=), those stack-level policies govern instead of child-resource policies. The inbound read is already for one parent stack, so this cannot suppress evidence from a different dependency root.
These rules let a human override automation deliberately without making the underlying inferred relationship disappear from cycle detection, diagnostics, or the graph.
- On the dependency (parent) side, within the same child/parent stack
pair, a declaration suppresses an inferred
-
Combining. Within the surviving set:
manualif at least one ismanual, elseneverif at least one isnever, elsealways. Manual is the safe middle ground when declarations disagree;neveronly suppresses automation when nothing asked to be consulted. -
Demotion. At resolution time, an
alwaysdependent that was not skipped by changeset filtering is demoted tomanualwith a stored reason, surfaced as a warning next to the (still present) trigger button. The reasons below are evaluated in this order and the first match wins;demotionForinpkg/domain/dependent_trigger_resolution.gocarries the same order. Facts about what already ran come before inferences from the derived graph.# Condition Stored reason 1 The dependent is already on this deployment’s cascade_stack_path— the stacks the cascade that produced it has deployed, carried on the row and inherited hop by hop. A redeploy would re-enter a stack the cascade has been through, whatever the graph says about cycles. There is no depth bound: the path is read, not walked.…: this stack has already been deployed by this cascade— The child’s complete inbound edge set exceeded the deployment-path budget. This is decided before edge policy can be evaluated; the causal cascade reason above still wins when known. …: this stack has too many dependency edges on the applied stack to resolve automatically2 The dependent’s declared edges are last-known: its own latest run could not read its dependency declarations, so the alwayswas authored by an earlier revision that may since have dropped or changed it.…: this stack's dependency declarations could not be read on its latest run3 The dependent’s inferred edges are last-known because its latest state exceeded an inference safety bound or contained unstoreable evidence. A narrower policy resolves conservatively to manual: an omitted inferred sibling could holdalwaysdown or make a resource-levelnevermanual. A declared whole-stack policy is exempt because it covers every inferred address in that parent.…: this stack's inferred dependencies could not be fully determined from its latest state4 The alwaysrests on a resource-scoped edge and the applied plan’s changeset is unreadable, so the resource match cannot be shown to have happened.…: the applied plan's changed resources could not be determined5 The dependent appears on a dependency cycle through the parent ( GetStackCycle, SCC — no depth bound, so loop length does not matter).…: circular dependency6 Cycle detection could not run at all because the org’s graph exceeded maxOrgStackPairs. “Could not tell” is not “no cycle”, so this fails closed.…: circular dependency status could not be determinedEvery row is the same principle: absence of evidence is not evidence. Each costs a click rather than a deployment.
Nothing is re-checked later, because nothing waits: triggers fire at resolution time, in the same pass that computed these demotions.
There is no human-approval backstop behind this. A triggered child is an ordinary plan deployment and follows the dependent stack’s own auto-approve policy, so on an auto-approve stack a missed loop would apply unattended, repeatedly. Demotion is the only thing preventing that.
The rule is blunt on purpose. What demotes is an
alwayswhose endpoints share a stored SCC — whatever kind of edge closes that component, including an inferred one (see Inferred dependencies). It does not try to establish that the loop could actually execute automatically end to end: amanualorneverhop would break the chain. That proof would need provenance- and policy-aware cycle detection, and the graph it would run over is asynchronously derived and deliberately retains dangling edges, so it cannot support that precision. Refusing automation inside anything that looks circular costs a human click; getting it wrong the other way costs an unattended redeploy chain.A skipped row is left alone: every automatic path already excludes it (
skipped_reason IS NULLin the work list), so demoting it would add a cycle warning to a row that already says nothing it depends on changed.Demotion applies at RESOLUTION time, which is also when triggers fire — there is no later execution to drift from the decision.
-
No multi-parent wait — each parent fires its own child. An
alwaysdependent of several stacks gets one dependency deployment per parent apply, immediately, each linked to the row of the parent that caused it. Converging a diamond (A → {B, C} → D) into “one redeploy” is the job of D’s own stack lane: the children queue there like any deployments, run back-to-back, and D converges on the last one. Asupersede_afterqueue policy (age-based, off by default) can additionally collapse a queued child that has already waited longer than its threshold — parents applying seconds apart do not trip it, and are not meant to.An earlier design parked the trigger instead: a deferred row waited for every other parent to settle, and a minute-ly sweep fired completed joins. It was removed deliberately. A stored executable promise ages — the dependent can edit its declarations, grow a cycle, or change generations while the row waits — and keeping the stored verdict honest grew a revalidation pass, a tri-state verdict, and an authorize-then-fire generation guard, each patching the previous (dependency review, items 23/30/65). Firing immediately deletes the promise and the machinery with it. The honest bill for that: a fan-in of N parents applying together queues up to N children (each a plan; on an auto-approve stack, an apply; on a manual-approve stack, a confirmation prompt each) where the old design ran one — a plan or apply against a half-settled world is corrected by the next parent’s trigger. One class is NOT self-correcting: a parent whose infrastructure changes through a path that fires no trigger of its own (a destroy, a promoted-PR apply) overlapping another parent’s apply can leave the dependent applied against outputs that then changed, with nothing re-firing it — the old sweep converged that window, and now only the dependent’s next ordinary trigger or its Deploy button does. The
dependent-trigger-reconcilescheduler job remains, doing only its first duty: resolving finished deployments whose resolution never ran at all (NULL marker — a crash, or completion by a pre-feature instance during a rolling upgrade). -
No coalescing. A firing trigger always creates a fresh deployment — it never links an in-flight or queued dependency deployment, which would tie this parent’s trigger to a plan that may predate this parent’s outputs. Concurrent triggers serialize on the child’s stack row lock.
Each snapshot row also pins the dependent stack’s incarnation
(dependent_stack_created_at, observed at resolution): a stack deleted and
recreated under the same id reads as “stack deleted” on old parents’ pages and
both trigger paths refuse it — the new incarnation never declared those
dependencies. The auto path runs the same cloud pre-flight as the Deploy
button before creating each child, so broken credentials surface as a trigger
failure instead of a doomed deployment.
The resolved per-dependent policies are snapshotted into Postgres
(deployment_dependent_triggers, one row per dependent stack per parent
deployment) — the parent deployment’s page renders that durable point-in-time
snapshot, not the live graph. Each row carries the resolved trigger, any
demotion reason, the changeset-filtering skip reason when nothing the dependent
consumes changed, and — once triggered — the child deployment id, which doubles
as the trigger-once claim.
Each row also records which resources put the dependent there (triggering_resources,
jsonb): pairs of the parent resource the applied plan changed and the dependent’s
own address that reads it, each marked declared or inferred (never both,
unlike the row-level evidence: a declaration on an address suppresses the
inference for that same address before the pairs are derived). They are derived
only from the edges that survived changeset filtering, and only when the
changeset was readable — the ones that actually fired — and are sorted,
deduplicated, and capped at three pairs with the remainder counted, since the
value is rendered verbatim on the parent’s page (“Because account changed
aws_acm_certificate.cloudfront_wildcard["gantrycd-prod"], which this stack reads
via data.aws_acm_certificate.wildcard.”). A dependent that depends on the whole
parent stack records that marker instead of pairs: “anything in that stack” is the
relationship, and naming the resources that happened to change would read as the
reason.
The column is NULL on every path that did not establish a cause, because the sentence it produces asserts that a resource changed and only the changeset establishes that: the resolution-budget row, a skipped row, an unreadable changeset (reason 4 above — policy falls back to the unfiltered edges, which is right for policy and a lie as provenance), and a fully-filtered child whose inferred set is incomplete (the same fallback, where an omitted sibling may be the real cause). The API additionally withholds it on a redacted row along with the rest of that stack’s identity, because the dependent-side addresses are its configuration.
Triggered deployments
A trigger (automatic or button) creates a normal plan deployment on the
dependent stack at the dependent’s own synced commit: it queues into the
dependent’s stack lane (PromoteNextOnStack decides queued vs pending) and
plans. What happens next follows the dependent stack’s own auto-approve
policy, exactly like any other plan deployment: a stack set to auto applies
without confirmation. There is no origin filter on that check, which is why
cycle demotion is the thing standing between a loop and an unattended
redeploy chain. Provenance is recorded as
origin_type = dependency with origin_dependency_deployment_id/stack_id
pointing back at the parent, so the child page links back and the parent row
links forward.
The button (POST …/deployments/{id}/dependents/{dependent_stack_id}/deploy)
is gated on (stack, deploy) on the dependent stack — pressing it is exactly
“create a deployment there” — plus (stack, read) on the parent. A dependent
already triggered conflicts (no re-trigger; the row links to the child
instead); a never row is rejected; a row whose earlier auto-trigger failed
(dependent stack deleted, commit not synced) is retryable — the button
doubles as the retry and clears the recorded error. A human pressing Deploy
wants a plan of the world as it is now, so the button always creates a fresh
deployment, never links an in-flight one — the same rule the automatic path
follows.
Promotion and the configured branch
A trigger runs the dependent’s configured branch, and promotion deliberately does not move it. Promoting a pull request creates a plan/apply deployment at the PR head; the apply is state-bearing (it replaces the stack’s resource rows and the inferred edges derived from them) but it does not run the configured branch, so its dependency discovery is rejected and the stack’s declared edges stay as the last configured-branch run left them.
After a promotion, “current” therefore means two different things for the same stack, and both are intentional:
| describes | |
|---|---|
| live infrastructure, outputs, resource rows, inferred edges | the promoted PR commit |
| declared dependency policy, and what a trigger deploys | the configured branch |
Inferred edges make the promoted head influence trigger outcomes. They are
state-derived, so the promoted commit writes them — a pull request that merely
adds a data source plants a real inferred edge, before anyone has merged it.
The dependent then appears on the parent’s page and can be offered a Deploy
button for a dependency its configured branch does not have yet; and if that
dependent also declares an always on another of the parent’s resources, the
new inference combines with it and holds the automation to manual (the row
says so). So that a reader can tell these rows apart from a human-declared
dependency, the snapshot records its evidence (declared, inferred, or
both, read from the raw edges before declaration precedence; NULL when the
edge set exceeded the resolution budget), and the Deploy row says “inferred
from live state” and that Deploy runs the configured branch. Removing a data source in a promoted PR does the mirror image: the
edge is withdrawn while the configured branch still reads the object.
That follows from the split above rather than contradicting it — inferred edges describe what the stack is, not what it declares — and it self-corrects the moment a configured-branch run reindexes. It is called out because it is the one way a PR that changes no policy can change a trigger outcome.
The consequence is real and it belongs to the user. If a parent applies
after a promotion, the dependent’s always enqueues the configured branch,
which does not contain the promoted change — so the deployment replaces promoted
C1 with configured C0. On a dependent that auto-approves, that happens without
another human looking at it.
GantryCD does not arbitrate that. A trigger takes the dependent’s lane like any
other deployment and obeys its queue and approval policy, so whether the
promotion or the trigger runs last is decided by ordering the operator can see,
pause, cancel or reorder. Whoever combines promotion, trigger=always and
automatic approval on one stack owns that interaction; the product’s job is to
make it legible, not to withhold the automation they configured.
The deployment history is where that is legible. A stack page carried a banner for this state at one point — it named the promoting pull request and said the next configured-branch deployment would replace it — and it was removed: the condition it reported is normal between promoting a fix and merging it back, and it cleared only when a configured-branch apply landed. A merge that plans to no changes, or one whose deployment is still waiting to be approved, applies nothing, so the banner outlived the divergence it described and read as noise on a healthy stack. The deployment list already names each deployment’s origin and pull request.
An earlier implementation went further and demoted the trigger here. It was removed: the check read the latest qualifying run, which cannot tell whether the promotion or the trigger is ahead in the lane. It fired when a promotion resolved first, stayed silent when the trigger resolved first and was merely queued behind the promotion, and cleared itself when any configured-branch refresh or plan became the latest run — fencing some interleavings, missing others, and overriding declared policy in the ones it caught. See item 37 of the dependency review.
Retries follow. Retrying a dependency-origin child deployment repoints every snapshot row that linked the old deployment at its replacement, so parent pages track the live attempt instead of a dead one.
Failures notify. A failed auto-trigger fires the
dependent_trigger_failed notification event — once per underlying problem:
the failure claims every untriggered always row pointing at the dependent,
across parent deployments, so N waiting parents yield one alert, not N —
through the org’s notification integrations, anchored to the parent deployment
with the dependent stack and reason attached. Broken automation should not wait
for someone to open the parent page.
Dependent rows whose stack the viewer cannot read are redacted to the same
~restricted-N tokens as the graph view (name, links, and errors withheld;
the policy badge survives). The Dependencies graph view renders the per-edge
policy as a small badge on non-default edges (always/never; manual stays
unlabelled) and in the edge focus panel, so a subscription is visible before
anything ever fires.
Budgets and how to watch them
Every cap above is a cliff: under it nothing changes, over it a feature silently degrades. Three metrics make the approach visible so raising a cap is a planned change rather than an incident.
budget label | Cap | What a user sees when it trips |
|---|---|---|
org_graph_edges | maxOrgEdges = 200,000 | The diagnostics sweep stops reporting orphan references — that is a statement about every edge in the org, so no smaller read answers it. Cycles survive: the sweep falls back to the reduced pair read, and only loses them past org_stack_pairs below. Since #278 this cap no longer touches the Dependencies view. |
neighborhood_edges | maxNeighborhoodEdges = 200,000 | One stack’s Dependencies view renders the central stack alone, which reads as “this stack has no dependencies”. Per page, not per org, so it has no usage gauge — the reading would be one number per (org, stack) neighborhood and no scrape-time query yields it. Watched through hits only. |
org_stack_pairs | maxOrgStackPairs = 200,000 | Cycle status is unknown, and demotion fails closed: every always dependent in the org is demoted to manual. Topology drops its connections, and the Dependencies view — whose neighborhood walk reads the same pairs — renders the central stack alone. |
inbound_edges_per_child | maxDirectDependentEdgesPerStack = 10,000 | One noisy dependent of the deployed stack is demoted to manual with “auto-trigger disabled: this stack has too many dependency edges on the applied stack to resolve automatically”, instead of auto-deploying. |
inbound_edges_total | maxDirectDependentEdges = 50,000 | The same, for dependents past the aggregate budget of one parent’s fan-out. |
inferred_edges_per_stack | maxInferredEdgesPerStack = 5,000 | One stack keeps its last complete inferred edges; new inference is not published and its evidence is marked unknown. This cap is watched through hits only — it has no usage gauge. It bounds three stages of inference and only the last leaves rows behind, so a stored count would report a stack jammed at the reads stage as near zero: healthy-looking, and refused on every reindex. The hits counter observes every stage. |
-
gantrycd_dependency_budget{budget}— the cap itself, read straight from the constant (services.DependencyBudgets). Noorglabel. Publishing it means a dashboard never hard-codes the number, so an alert stays honest after a cap is raised. -
gantrycd_dependency_usage{org,budget}— how close one org is now, for the four of the six budgets that have an honest, org-wide reading (inferred_edges_per_stackandneighborhood_edgesare the exceptions above). Read on the same scrape asgantrycd_explore_dependencies_total(RegisterExploreInventory), one statement for all orgs.org_graph_edgesis the org’s total edge count;org_stack_pairsits distinct cross-stack pairs;inbound_edges_totalthe largest budgeted inbound star on any one stack (children already rejected by the per-child cap contribute nothing, exactly as the deployment path counts it);inbound_edges_per_childthe largest bundle from one child onto one parent. These are org-wide maxima on purpose — there is deliberately no per-stack label.Cost, measured warm at
work_mem=4MBand32MB(index-only either way, zero temp I/O in all four):Table Distinct pairs Usage read The edge count on the same scrape 1,089,000 edges 14,600 93–209 ms 78–122 ms 6,760,000 edges 270,400 2.0–2.8 s 0.87–1.03 s The first row is scrape-cheap. The second is not, and no rewrite fixes it — the cost is the 270,400-group aggregation, not the query’s shape. If an org ever approaches that pair count, the follow-up is to stop reading this on the scrape path: have the periodic dependency-graph rebuild sweep compute the numbers once per org and persist them as a usage row, and let the metrics callback read that row back. Not built — a stored row is only worth its staleness once the live read hurts.
-
gantrycd_dependency_budget_hits_total{org,budget}— a counter of the moments a cap actually refused work, incremented once per event (a truncated read, a resolution that dropped dependents, a stack held at last-known), never once per row. The two inbound caps share the labelinbound_edges: the read reports its incomplete children without saying which of the two excluded each one, and the two usage gauges above already answer “which am I near”.
Alert rules live in the infra repo. The two worth copying:
# Approaching a cap: over 30% of budget. Each replica reports the same
# scrape-time reading, so aggregate before dividing.
max by (org, budget) (gantrycd_dependency_usage)
/ on (budget) group_left max by (budget) (gantrycd_dependency_budget)
> 0.3
# A cap actually refused work in the last hour. Any value above zero is a
# user-visible degradation someone should see.
sum by (org, budget) (increase(gantrycd_dependency_budget_hits_total[1h])) > 0
The rebuild’s own runtime
The caps above bound how much a read may return. Nothing above bounds how LONG the periodic rebuilds take, and that used to be its own cliff.
Both rebuilds now run in resumable two-minute slices, so one run is a slice rather than an org and each is registered with a 10-minute ceiling — simultaneously its handler deadline, its lease, and its crash-recovery wait (see Async Task Queue for the model and the measurements). The per-stack walks no longer grow one claim with org size. What can still reach the ceiling is a single stack whose own work outlasts the budget, or the final whole-org reverse reap, which is not yet sliced — see Resource Explorer: Filtering & Scale.
They are watched through gantrycd_async_task_duration_seconds{task_type}, whose
boundaries reach 1800 from when the ceiling was thirty minutes:
# A rebuild RUN took over 5 minutes. Either one is now a SLICE at 2.5x its
# two-minute budget — a single stack or read that overran, not a slow org.
# Counted past a bucket rather than with a quantile: the sweep runs twice a day,
# so a rate() window is usually empty and histogram_quantile returns NaN.
sum(increase(gantrycd_async_task_duration_seconds_bucket{
task_type=~"resource_index_rebuild|dependency_graph_rebuild", le="+Inf"}[6h]))
- sum(increase(gantrycd_async_task_duration_seconds_bucket{
task_type=~"resource_index_rebuild|dependency_graph_rebuild", le="300"}[6h]))
> 0
No org label: the duration histogram is bucketed by task_type alone, and
adding org would multiply series by bucket count.
The backstop changed. This rule was introduced alongside
gantrycd_job_executions_total{outcome=~"failure|timeout"} firing as
job_name="task-processor". That scheduler job no longer exists — the queue runs
continuous worker slots, not a periodic batch — so that alert can never fire for
async tasks again. What replaces it, per task type rather than for a whole tick:
gantrycd_async_task_processed_total{outcome="timeout"} when a handler hits its
ceiling,
and AsyncTasksDeadLettered once one has given up for good.
Authorization & redaction
The …/dependencies and …/dependency-cycles routes authorize read on the
central stack only. But a neighborhood — and especially a cycle — can reach
stacks the viewer has no grant on, and node keys carry stack IDs and resource
addresses. Dropping those stacks would distort the topology and could hide a
cycle, so instead the handler redacts them:
StackService.ReadableStackIDs(principal, org, ids)resolves which referenced stacks the viewer may read (an unscoped/super-admin grant short-circuits to all-readable; otherwise each id is resolved to its name and checked with the canonical per-itemAllowed, so pattern semantics are never re-implemented).- A
dependencyRedactor(internal/backend/handlers/dependency_redaction.go) rewrites every entity owned by an unreadable stack into a stable, opaque placeholder —~restricted-Nfor the stack,~hidden-Mfor its resources — consistently across nodes, edges, and cycle keys, so the shape (including any cycle through a hidden stack) is preserved while names/addresses are not.~cannot occur in a real stack id or resource address, so a placeholder can never collide with a real identifier, and the frontend detects the prefix (isRedactedStack/isRedactedDependencyKeyinweb/src/utils/dependencyKeys.ts) to render a lock. The backend/frontend token prefixes must stay in sync.
Wiring status
The edge repository is built in buildRepos over the explore database pool and
held on appRepos.dependencyEdges — always non-nil, because
EXPLORE_DATABASE_URL is required and buildExploreDatabase aborts boot
without it. Configuration and availability are gated differently: an unset or
unparseable URL is fatal at startup, while a valid-but-unreachable database is
not (the pool connects lazily and does not ping), so the graph paths degrade at
query time and the async queue’s backoff/dead-letter carries the retries (see
Datastores). The async handlers are registered on the
TaskProcessorJob in cmd/backend/scheduler.go (TaskTypeDependencyReconcile →
HandleReconcileTask; TaskTypeDependencyGraphRebuild → HandleRebuildTask),
and the 12h dependency-graph-rebuild scheduler job (EnqueueRebuildAll) is
registered alongside the other recurring scans there. The full pipeline above is
wired: the runner-side discovery scan, the DependencyGraphService
enqueue/reconcile/query/rebuild/cleanup paths, the periodic rebuild sweep, and
the Dependencies-tab UI all consume it.