State Administration

Three advanced, individually-permissioned operations let an operator recover a stack whose Terraform state is wedged: force-unlock a stale state lock, overwrite the state with an uploaded file, and restore the state to a stored prior version. All act directly on the managed state backend from the backend process — no runner, no deployment — so they are synchronous and can preview the lock holder / current state / version list before mutating.

When you need them

• Force-unlock: a run crashed mid-apply and left the S3 backend lockfile behind. With use_lockfile = true (always set — see Runner runtime credentials), the lock is a plain object at <state_key>.tflock. Once orphaned, every subsequent plan/apply fails with “state locked” until the lockfile is removed.
• Overwrite state: the live state is corrupt or rolled back and must be replaced with a known-good terraform.tfstate.
• Restore a prior version: a bad apply moved the state forward and you want to roll back to a known-good earlier version without hunting for a copy to upload. Backed by S3 object versioning (see below).

Before this surface existed the only recourse was a free-text Run Command (tofu force-unlock … / tofu state push), which requires the broad (stack, run_command) permission and offers no visibility. These operations are narrower and safer.

Permissions

Each operation has its own action on the stack resource and its own built-in role, so they can be delegated independently of deploy and run_command:

restore_state is separate from overwrite_state on purpose: restoring a known, gantrycd-written prior version is narrower and lower-risk than uploading arbitrary bytes, so it can be delegated more freely. admin ((*, *)) and stack-admin ((stack, *)) hold all three via their wildcards. See the Permissions Catalog.

Managed backends only

These operations work only for stacks with a gantrycd-managed backend (managed_backend = true). The backend mutates the state object and lockfile directly, which it can only do for the backend it provisions and whose bucket layout it owns (paths.StateKey). For a stack with managed_backend = false (a bring-your-own backend) gantrycd does not know where the state lives, so it cannot force-unlock, overwrite (state push), or restore it — the API returns 422 (StateNotManagedError) and the UI hides the actions.

Version-restore has a second prerequisite: the backend must be able to enumerate object versions. The S3 backend can, but only when bucket versioning is enabled on the state bucket. When it isn’t, the version list returns versioning_enabled: false (an empty list, not an error) and the UI prompts to enable it; a backend that can’t address versions at all returns 422 (StateVersioningNotSupportedError) and the UI hides the Restore tab. Enabling versioning on the bucket is an infrastructure concern (e.g. an aws_s3_bucket_versioning resource), configured outside this repo.

For unmanaged backends, do the equivalent through a Run Command raw deployment instead: the runner clones the repo and runs tofu init against your backend, so tofu force-unlock <LOCK_ID> and tofu state push <file> operate on it directly. That path is gated by the broader (stack, run_command) permission — see the run_command notes in the Permissions Catalog and raw deployments in the Deployment State Machine.

API

All six routes are session-authed and gated by the matching permission (internal/backend/server/routes_state.go). The read previews require the same manage permission as the mutation they precede — they are part of the dangerous flow, not general read.

Safety rules (StateAdminService)

internal/backend/services/state_admin_service.go enforces:

• Force-unlock requires lock_id to match the lockfile currently present (re-read at the moment of unlock) — a mismatch returns 409 so a stale click can’t break a freshly-acquired lock. Unlocking when nothing is locked is a 409.
• Overwrite validates the uploaded bytes parse as a Terraform state envelope (a non-zero version) before writing anything, backs the current state up to …/history/terraform.tfstate.<timestamp>.backup (the response returns the backup key), and refuses while the state is locked (force-unlock first).
• Restore fetches the chosen version’s bytes, validates them as a state envelope (so a delete marker or any non-state object is rejected), backs up the current state to the same …/history/… key, then writes the chosen bytes as a new latest version. Because no version is ever deleted, a restore is itself reversible — you can restore the version you just replaced. It reuses the overwrite write path, so the same lock/lane guards apply.
• Overwrite and restore refuse while the state is locked (force-unlock first). All three refuse while a gantrycd deployment occupies the stack lane (stacks.locked_by set) — the lock may be legitimately held by a live apply, so cancel the deployment instead. The stale-lock case has no active deployment, so the lane is free.

Every mutation is logged via contextutil.GetLogger (actor, stack, lock ID / backup key, old→new serial) as the audit trail.