Runner Runtime Credentials

Accepted runs receive short-lived credential bundles inside RunContext. They are generated at accept time, never at poll, so credentials aren’t wasted on assignments that expire before the worker starts.

Credential Families In RunContext

The pkg/contracts.RunContext carries four independent concerns plus an optional artifact bundle:

Runtime Credentials

RuntimeCredentials carries one entry per credential, not per integration and not per provider: a stack whose one AWS integration lists two roles, plus a GCP integration, produces three entries. How many entries an integration yields is that provider’s business — see cloud_integrations.md.

type RuntimeCredential struct {
    Provider  string          // "aws", "gcp"
    Name      string          // optional — the stack's handle for this credential
    Config    json.RawMessage // the provider's own credential shape
    ExpiresAt *time.Time
}

Name is the provider’s, not the wire format’s. AWS sets it, because a stack with several roles must be able to say which one a given provider "aws" block uses. GCP leaves it empty, because the google provider reads one ambient token and there is nothing to select between — a name there would be a field nobody reads, kept only to satisfy a struct.

Config is opaque to everything but the provider that produced it and the runner-side materializer that consumes it — the same shape StorageConfig uses. Onboarding a cloud provider never touches this struct.

ExpiresAt is hoisted out of the blob because the run lifecycle depends on it: RunContext.EarliestCredentialExpiry() folds the minimum across every credential family, and an earlier expiry shortens RunDeadline. The configured RUN_TTL bounds the deadline independently, including when storage credentials do not expire. A run cannot outlive its credentials by construction — which is why minting static credentials up-front at accept is correct here, rather than a lazy/refreshable scheme.

Materialization

Before any subprocess starts, the runner groups the credentials by provider and hands each group to that provider’s materializer (runtimeCredentialMaterializers). Files are written under <home>/.gantrycd-runner/runtime/<run-id>/, 0700 dir / 0600 file, removed when the run ends.

• AWS — every role is written into one shared-credentials file, one [name] section each, and AWS_SHARED_CREDENTIALS_FILE points at it. The stack selects one the way it would on a workstation: provider "aws" { profile = "network" }; a role named default serves a bare provider "aws" {}. A file rather than AWS_ACCESS_KEY_ID: with several roles there is no single “the” credential to put in the environment, and it keeps the secrets out of the env dumps and set -x traces that hooks and raw runs can emit into the (unmasked) run log. GantryCD writes no region — region resolution stays where it is locally: the provider block, or AWS_REGION.

  AWS_PROFILE is not set, at any cardinality. Naming a lone role as ambient would make it the identity behind everything that reaches AWS without asking for a profile — a bare provider "aws" {}, an unmanaged stack’s own backend "s3" {}, a terraform_remote_state data source, a hook shelling out to the aws CLI — on the strength of a count, and silently. Which role those should use is the stack’s to state. The rule is AWS’s own instead: a role named default is the default, because the SDK resolves that profile itself. Those consumers authenticated from AWS_ACCESS_KEY_ID before this change; a stack that relies on them now names a role default.

  This runs on every run, including one whose stack has no AWS integration at all — and then both files are written empty. That is the case the pinning exists for: a stack with no AWS identity is precisely the one whose provider "aws" {} would otherwise find the runner host’s. Two more variables are pinned alongside it, so that a profile the stack names but GantryCD did not write — or names not at all — resolves to nothing, loudly, rather than to an ambient identity. AWS_CONFIG_FILE points at an empty file the run owns — otherwise the SDK reads the runner host’s ~/.aws/config (on the local and self-hosted groups $HOME is the operator’s), where a same-named [profile x] section can contribute credential_process, role_arn, or endpoint_url to our profile. AWS_EC2_METADATA_DISABLED=true removes the instance-role fallback: a named profile that doesn’t exist already errors inside the SDK, but a bare provider "aws" {} on a stack with no default role would otherwise find no [default] section and walk the chain to IMDS, running the stack under the runner node’s identity. This is a misconfiguration guard, not a security boundary — the run executes the stack’s own code and can reach IMDS by other means; the node’s IMDS hop limit or a runner NetworkPolicy is the actual control.

• GCP — the impersonated access token is exported as GOOGLE_OAUTH_ACCESS_TOKEN, which the google/google-beta providers and gcloud read directly. No file, and no name — the google provider reads one ambient token. The materializer expects exactly one credential; a stack can hold only one GCP integration and its config names a single service account, so more than one is a backend bug rather than user input.

  Note the asymmetry with AWS, which is deliberate but has a cost worth knowing: AWS credentials are kept in a file partly so an env dump or a set -x trace does not leak them into the (unmasked) run log, whereas the GCP token is in the environment. A lifecycle hook that runs env / printenv / set will print the live GCP token into the log, where anyone who can read the run’s logs — not only someone who can submit code to the stack — can capture it for its ~1h lifetime. This does not widen the code-author boundary (a stack’s own code already holds the token), only the log-viewer one. Run-log access for a stack should be treated as equivalent to holding that stack’s GCP token until the token is delivered by file (as AWS is) or subprocess output is secret-masked.

A credential for a provider the runner doesn’t know is a hard error, not a skip: running anyway — without the credentials we failed to understand — would execute the stack against whatever ambient identity the runner host happens to carry.

Credentials on disk, and who can read them

The credentials are written under <home>/.gantrycd-runner/runtime/<run-id>/, 0700 dir / 0600 file, and removed when the run ends.

Those modes exclude other OS users. They do not exclude other runs by the same user — and they cannot: a run executes arbitrary OpenTofu and arbitrary bash (lifecycle hooks) as that user, so anything the user can read, the run can read. A runner is therefore assumed to be ephemeral and to have the machine to itself for the length of one run, which the Kubernetes and Docker groups give it by construction (a fresh container per run).

That assumption is what makes the on-disk file safe, and it is not a property of the file: on a host shared between concurrent runs, the same user could read another run’s credentials out of /proc/<pid>/environ just as easily as off the disk. Keeping secrets in the environment would buy nothing. So the rule is about the runner, not the storage: do not point a runner group that shares a machine and an OS user between runs at stacks whose contributors are not allowed to deploy each other’s.

Git Configuration

SCMConfig.GitConfig carries the git configuration the run must use, as ordered (section, subsection, key, value) entries. The backend’s SCM provider builds them and the runner writes them verbatim — the runner synthesises no entry of its own and knows nothing about GitHub, exactly as it already knows nothing about it for CloneURL. A second SCM provider supplies different entries and touches no runner code.

It exists because the clone is not the run’s only git traffic. tofu init resolving a private module derives its own URL from the module source and would reach GitHub anonymously; GitHub’s entry attaches the installation token to every github.com request as an Authorization header:

[http "https://github.com/"]
	extraHeader = Authorization: Basic <base64("x-access-token:<token>")>

A header, not a token-in-URL rewrite (url."https://x-access-token:<token>@github.com/".insteadOf = https://github.com/ would also work). The URL form puts the live token where git prints it: on the git subprocess’s argv, and — the moment a stack sets GIT_TRACE=1 or GIT_CURL_VERBOSE=1 to debug a fetch — in cleartext in the run log, which a read-only log viewer who cannot deploy could then harvest. git redacts the Authorization header from those same traces by default and never puts it on argv, so the header form keeps the token out of both. (This is why the runner also pins GIT_TRACE_REDACT=1 above the stack’s env — a stack cannot switch that redaction off to spill its own token into a log its lower-privileged readers can see. A run can still exfiltrate its token from code, which holds it by construction; the pin closes the env-var-only path.)

http.<url>.extraHeader is matched by git against the request URL by scheme, host, and path prefix — so it reaches https://github.com/… (including the github.com/acme/mod shorthand, which go-getter expands to https://github.com/acme/mod.git), is not sent to a lookalike host like github.com.evil.com, and is dropped on a redirect to any other host. Only HTTPS is covered — an SSH module source keeps resolving through the runner’s key, because redirecting it at the token would break the repositories the App is not installed on. And because an installation token reaches one account, a private module in a different GitHub organization is not covered. Nor are non-git sources: an https://…/archive.zip module (fetched over plain HTTP by go-getter, not git) and a private OpenTofu registry module (authenticated by TF_TOKEN_*, not git config) get no token from this.

Why a file, and why it is pinned

The runner writes the entries to <home>/.gantrycd-runner/runtime/<run-id>/gitconfig (0600, removed with the rest of the run’s credential dir) and exports GIT_CONFIG_GLOBAL at it.

Not git config --global. That writes $HOME/.gitconfig, and $HOME is only the run’s own on the Kubernetes and Docker launchers. On the local launcher the runner is a subprocess on a long-lived operator machine holding that person’s HOME: --global would deposit a live GitHub token in their personal ~/.gitconfig, leave it there after the run, and let two concurrent runs overwrite each other’s. The env var reaches every git process the run spawns without writing anything the run does not own.

Not an env-var-borne token either (GIT_CONFIG_COUNT / GIT_CONFIG_KEY_0 would work). Subprocess output is not masked, so a token in the environment is one env dump or set -x trace away from the run log — the same weakness the GCP path has and the AWS file avoids. Rendering the file directly also keeps the token out of argv, which any process on the host can read.

The file is written and GIT_CONFIG_GLOBAL is pinned on every run, including one with no token at all — a public repo, a local deployment — and then the file is empty. That case is the reason the pin exists, not one it tolerates: GIT_CONFIG_GLOBAL replaces the host’s ~/.gitconfig, so pinning it unconditionally is what stops the operator’s git configuration from reaching into a run. Otherwise a run’s git behaviour would depend on which machine it landed on: an insteadOf or a credential.helper in that operator’s config would silently lend the run an identity GantryCD never granted it, and a stack that resolved its modules on one runner would fail on the next. This is the same reasoning behind the empty AWS_CONFIG_FILE above.

The clone is pinned at the same file, but keeps the host environment otherwise (it runs before the run’s curated environment is built, and it is the host’s proxy and TLS settings that get a self-hosted runner out to the SCM at all).

The system config (/etc/gitconfig) is deliberately left alone: it belongs to the runner image or the machine build, not to a per-user identity that varies by launcher.

A stack needing git configuration beyond the rewrite can append to $GIT_CONFIG_GLOBAL from a pre_init hook — the variable is in the hook’s environment and the file is writable.

Expiry

SCM credentials stay out of EarliestCredentialExpiry and so do not shorten RunDeadline. They gate only the run’s git traffic, and that is over once init has resolved the modules; folding them in would cut a long apply short over a credential apply does not use. The cost is that a git fetch late in a run (a lifecycle hook, a raw command) can find the token expired — it fails loudly when it does, and the installation-token cache refuses to hand out a token with under 30 minutes left, so init always gets a live one.

Provider Coverage

• SCM: GitHub App integrations mint short-lived installation tokens, used both to clone and — through the GitConfig rewrite above — to fetch private modules during tofu init. If no SCM integration matches, the runner is told Type: "none" and treats the repo as public.
• Terraform backend: S3 only.
• Logs / artifacts storage: S3 only.
• Runtime credentials: AWS (STS AssumeRole) and GCP (IAM Credentials service-account impersonation). Azure is not implemented — an Azure integration cannot be persisted today. See cloud_integrations.md.

S3 Key Layout

Backend, logs, and artifact keys are constructed by helpers in internal/backend/services/credentials/s3keys.go and delivered to the runner via the Bucket / Key / Prefix fields in RunContext. The runner does not synthesise keys.

S3 Per-Run Credentials

On the AWS path each storage concern (state / logs / artifacts) mints per-run credentials with sts:AssumeRole on <CONCERN>_S3_ROLE_ARN, passing an inline session policy scoped to that run’s S3 prefix. Because AssumeRole permits role chaining, the assuming principal may be a static IAM user, an EKS IRSA role, or an instance profile. Non-AWS S3 (RustFS/MinIO) has no STS endpoint: setting <CONCERN>_S3_SKIP_STS=true hands the concern’s static keys straight to the runner instead (full-bucket access, no expiry). See Configuration. Runtime credentials for Terraform cloud providers also use sts:AssumeRole against the customer’s role — one call per role in the stack’s AWS integration, fanned out — and they cannot be served by static S3 config. Each of those calls additionally accepts an optional stack-authored inline session policy (that role’s inline_policy), passed as the Policy so a shared role can be scoped down per stack — the same Policy mechanism the S3 concerns use, but authored by the stack owner rather than synthesised. See cloud_integrations.md.

The state concern’s session policy is additionally scoped by run mode: only apply (and the raw escape hatch) receive s3:PutObject / s3:DeleteObject, because only those mutate state. Plan-style runs — plan, destroy-plan, refresh-only, and so every PR preview and local-plan — get s3:GetObject only, so an unreviewed plan can read state to diff against but cannot overwrite or delete it (that stays behind the dedicated overwrite_state / unlock_state permissions). With STATE_S3_SKIP_STS=true this per-mode scoping is lost (static keys carry full-bucket access) — a known trade-off of running state off AWS.

State credentials and saved plans

The runner never inlines the state credentials into the OpenTofu backend config. OpenTofu serialises the resolved backend configuration into the saved plan file, so inline access_key / secret_key / token would be frozen into plan.tfplan at plan time — and the apply run (tofu apply plan.tfplan, a separate run with its own freshly minted write-scoped credentials) would then reconnect to the backend with the plan run’s read-only, short-lived credentials and fail to take the state lock.

Instead the runner writes the credentials to an AWS shared-credentials file under a gantrycd-state profile and the backend config references it via profile + shared_credentials_files. Only the profile name and file path are serialised into the plan; the credentials are re-read from the file at apply time, so the apply run’s own write-scoped credentials are used. Two constraints make this work:

• Stable, private, cross-OS path. The path is <home>/.gantrycd-runner/<state-scope>/state-aws-credentials: os.UserHomeDir() (works on every OS and keeps the secret out of world-writable /tmp) plus a <state-scope> hashed from the state object key. That key is fixed per stack incarnation, so the scope is identical for that stack’s plan and apply runs (the path baked into the saved plan resolves in the apply pod) yet distinct across stacks. It is not derived from os.TempDir()/$TMPDIR (two pods could resolve those differently) nor from a per-run temp dir. A plain 0600 file removed at run end (rather than a Linux-only /dev/shm tmpfs) keeps the runner portable. The apply pod overwrites it with its own credentials before tofu apply.
• The state identity is separate from the stack’s. The backend block names both its profile (gantrycd-state) and its shared_credentials_files, so it reads only its own file — never the stack’s runtime credentials, which live in a different file under AWS_SHARED_CREDENTIALS_FILE. That file separation is the whole isolation: a stack cannot reach the state credentials even if it names one of its own roles gantrycd-state, so no role-name prefix needs to be reserved.

The S3 access role (<CONCERN>_S3_ROLE_ARN)

The operator provisions one IAM role per concern (or one shared role) with:

• Trust policy — allows the backend’s own principal to sts:AssumeRole: the IRSA role ARN on EKS, the IAM-user ARN when running under static keys.
• Permissions policy — the broad grant the per-run session policy narrows down, e.g. s3:GetObject, s3:PutObject, s3:DeleteObject, s3:ListBucket on the concern’s bucket. Minted credentials are the intersection of this and the session policy.
• MaxSessionDuration >= RUN_TTL + grace. Note the role-chaining cap: when the backend principal is itself an assumed role (IRSA, instance profile), AssumeRole is capped at 1 hour regardless — keep RUN_TTL + grace <= 1h in that case.

The backend principal additionally needs an identity policy granting sts:AssumeRole on the role’s ARN.

References

• run_context_and_artifacts.md
• cloud_integrations.md
• scm_and_pull_requests.md