GitHub-Actions runner groups

GantryCD supports two runner-group types:

Both types share the same poll/accept/done HTTP protocol — the only difference is who plays the runner role and how it gets started.

For github-actions groups, gantrycd holds a backend-only ECDSA signing key per group. When a queued run’s required labels match the group’s labels, the in-process dispatcher mints a short-lived ephemeral runner JWT, calls GitHub’s workflow_dispatch API on the configured workflow, and the workflow uses the JWT to drive the rest of the runner protocol against gantrycd.

Why this exists

Self-hosted runners require infrastructure the operator owns: a process, an IAM identity, network egress to gantrycd, log forwarding, supervision. For teams that already run their CI on GitHub Actions, that’s duplicated work — they’d rather run gantrycd jobs alongside their PR checks on the same hosted runners. A github-actions group is the built-in answer: zero gantrycd-side infra, identical security model.

What the operator does

1. Create a GitHub App for the target repo with these permissions:
   • Repository → Actions: read and write (to call workflow_dispatch).
   • Install the App on the repo you’ll dispatch to. Note the App ID and download the App private key (.pem).
2. Commit a workflow file to .github/workflows/gantrycd-runner.yaml (see Workflow template below).
3. Create the runner group in gantrycd, pointing at the App + workflow:

   gantrycli runner-groups create \
     --type github-actions \
     --name infra-prod \
     --label production \
     --label terraform \
     --github-repo acme/infra \
     --github-workflow gantrycd-runner.yaml \
     --github-workflow-ref main \
     --github-app-id 123456 \
     --github-app-private-key-file ./gantrycd-app.pem \
     --max-concurrent 5

That’s the entire setup. Queueing a deployment whose stack runner_labels ⊆ {production, terraform} will now fire the workflow.

Workflow template

Copy this verbatim into .github/workflows/gantrycd-runner.yaml in the repo the App is installed on. Nothing in it names a gantrycd version: the runner group tells the workflow which release to run, so this file does not change when you upgrade.

The workflow does three things:

1. Download the runner binary for the release the dispatching group is itself running, supplied as the runner-version input.
2. Install tenv, the OpenTofu/Terraform version manager the runner shells out to (tenv opentofu install <version> then tofu). Its release tarball ships the tofu/terraform proxy binaries, so putting it on PATH is all the runner needs to resolve tofu.
3. Run the binary with the two env vars it reads — BACKEND_URL (from the gantrycd-url input) and EPHEMERAL_RUNNER_JWT (from the assignment-token input). The run_id/runner_id are read from the JWT claims, so no other configuration is required.

name: GantryCD Runner
# Carries the correlation-id so the dispatcher can find this run by display_title.
run-name: gantrycd-${{ inputs.correlation-id }}
on:
  workflow_dispatch:
    inputs:
      assignment-token:
        required: true
      # Supplied by the gantrycd dispatcher from GANTRYCD_API_PUBLIC_ENDPOINT.
      gantrycd-url:
        required: true
      # Opaque <group-id>/<run-id> used for run-name correlation; must be
      # declared or workflow_dispatch returns 422 "Unexpected inputs provided".
      correlation-id:
        required: true
      # The gantrycd release the dispatching runner group is itself running.
      # The group always sends it, so the runner it starts is the same build as
      # the group that started it and the two cannot disagree on the protocol.
      runner-version:
        required: true

# Third-party pins only. The gantrycd version is not pinned here — it arrives
# with the dispatch.
env:
  TENV_VERSION: v4.12.2

jobs:
  run:
    # ubuntu-slim is a lightweight Docker-based runner; swap to ubuntu-latest for
    # more capacity (CPU/RAM/disk), or to an arm64 label for arm64. Every
    # download below derives its architecture from this label in step 1, so the
    # label is the only thing an arch change touches.
    runs-on: ubuntu-slim
    # Sized above the backend's run budget so the runner — not GitHub — ends the
    # run: it self-cancels at RUN_TTL minus one grace period (50m at defaults),
    # then has RUN_CREDENTIAL_GRACE_PERIOD (10m) to shut down and call
    # done/deregister. This cap is the backstop; keep it > RUN_TTL. Setting it
    # equal to RUN_TTL leaves no margin and GitHub kills the job mid-shutdown.
    timeout-minutes: 75
    steps:
      # 1. Derive everything the later steps need from the runs-on label. The
      #    three projects below name the same two architectures three different
      #    ways, which is exactly why this is resolved once instead of written
      #    into three URLs that then drift apart:
      #
      #      cosign           cosign-linux-amd64  / cosign-linux-arm64
      #      gantrycd runner  linux-amd64         / linux-arm64
      #      tenv             Linux_x86_64        / Linux_arm64
      #
      #    An unrecognised arch fails here rather than 404-ing three steps later.
      #
      #    TOOLS_DIR holds the tenv CLI and its tofu/terraform proxies, and goes
      #    on PATH. It cannot be /usr/local/bin once it is cached: that dir is
      #    system-owned and full of unrelated binaries, so caching it would drag
      #    them along. The runner passes its own PATH to tofu verbatim, so the
      #    proxies resolve here just the same. TENV_ROOT is pinned explicitly so
      #    the cache path in step 4 provably matches where tenv writes, rather
      #    than relying on its default.
      - name: Resolve architecture and tool paths
        env:
          HOST_ARCH: ${{ runner.arch }}
        run: |
          set -euo pipefail
          case "${HOST_ARCH}" in
            X64)   arch=amd64; tenv_arch=x86_64 ;;
            ARM64) arch=arm64; tenv_arch=arm64  ;;
            *) echo "unsupported runner.arch: ${HOST_ARCH}" >&2; exit 1 ;;
          esac
          TOOLS_DIR="${HOME}/.cache/gantrycd-tools"
          mkdir -p "${TOOLS_DIR}/tenv"
          printf '%s\n' "${TOOLS_DIR}/tenv" >> "${GITHUB_PATH}"
          {
            echo "ARCH=${arch}"
            echo "TENV_ARCH=${tenv_arch}"
            echo "TOOLS_DIR=${TOOLS_DIR}"
            echo "TENV_ROOT=${HOME}/.tenv"
          } >> "${GITHUB_ENV}"

      # 2. Install cosign, used to verify the signatures on release artifacts.
      #    -f matters: without it curl writes the 404 body to the file and the
      #    step goes on to chmod +x an HTML error page.
      - name: Install cosign
        run: |
          set -euo pipefail
          curl -fsSL -o cosign \
            "https://github.com/sigstore/cosign/releases/latest/download/cosign-linux-${ARCH}"
          sudo install -m 0755 cosign /usr/local/bin/cosign
          rm cosign

      # 3. Pull the runner binary for the release the group runs. The tar.gz
      #    holds a single `runner` executable; extract it into the workspace.
      #    The input goes through the environment rather than into the script
      #    body, so its value is never expanded as shell.
      - name: Download runner binary
        env:
          RUNNER_VERSION: ${{ inputs.runner-version }}
        run: |
          set -euo pipefail
          curl -fsSL -o runner.tar.gz \
            "https://github.com/gantrycd/gantrycd/releases/download/${RUNNER_VERSION}/runner-${RUNNER_VERSION}-linux-${ARCH}.tar.gz"
          tar -xzf runner.tar.gz
          chmod +x runner

      # 4. One cache for the whole OpenTofu runtime, not for tenv alone: the
      #    tenv CLI and its tofu/terraform proxies, plus TENV_ROOT, where
      #    `tenv opentofu install <version>` puts the OpenTofu builds. The CLI is
      #    the small part — those builds are what makes this worth caching.
      #
      #    The key carries runner.arch and TENV_VERSION, not just the OS. Both
      #    parts of the cache are arch-specific native binaries, so without arch
      #    an arm64 run restores an amd64 toolchain; without TENV_VERSION,
      #    bumping tenv restores the old CLI under the new label.
      #
      #    run_id + restore-keys is the accumulate pattern: the primary key can
      #    never hit (run_id is unique per run), so every run reseeds from the
      #    newest matching entry and saves an updated one. That is deliberate —
      #    stacks pin different OpenTofu versions, and they need to pile up rather
      #    than evict each other.
      - name: Cache the OpenTofu runtime
        uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
        with:
          path: |
            ~/.cache/gantrycd-tools/tenv
            ~/.tenv
          key: ${{ runner.os }}-${{ runner.arch }}-tofu-runtime-${{ env.TENV_VERSION }}-${{ github.run_id }}
          restore-keys: |
            ${{ runner.os }}-${{ runner.arch }}-tofu-runtime-${{ env.TENV_VERSION }}-

      # 5. Install the tenv CLI if the cache did not carry it. Gated on the
      #    BINARY, not on cache-hit: the run_id key above never hits its primary
      #    key, so cache-hit is 'false' even after a successful restore-keys hit,
      #    and gating on it would redownload every run.
      - name: Install tenv
        run: |
          set -euo pipefail
          if [ -x "${TOOLS_DIR}/tenv/tenv" ]; then
            echo "tenv ${TENV_VERSION} restored from cache"
          else
            curl -fsSL "https://github.com/tofuutils/tenv/releases/download/${TENV_VERSION}/tenv_${TENV_VERSION}_Linux_${TENV_ARCH}.tar.gz" \
              | tar -xz -C "${TOOLS_DIR}/tenv"
          fi
          command -v tenv

      # 6. Drive the runner protocol against the backend.
      - name: Run the ephemeral runner
        env:
          BACKEND_URL:          ${{ inputs.gantrycd-url }}
          EPHEMERAL_RUNNER_JWT: ${{ inputs.assignment-token }}
        run: ./runner

What the dispatcher supplies

Nothing in this file is a gantrycd version or URL you maintain — all four inputs come from the dispatching group:

• gantrycd-url — read from GANTRYCD_API_PUBLIC_ENDPOINT, so the same workflow works across deployments and survives a backend URL change (e.g. a new dev tunnel) with no edit. It is required for github-actions runner groups; the dispatcher refuses to dispatch without it.
• correlation-id — an opaque <group-id>/<run-id> the workflow echoes into run-name as gantrycd-<correlation-id>. The list-runs API returns a run’s display_title but not its workflow_dispatch inputs, so the run-name is the only way the dispatcher correlates a dispatched run back — to poll it to completion and to re-discover in-progress runs after a restart. Both pieces are mandatory: drop the input and workflow_dispatch fails with 422 Unexpected inputs provided; drop the run-name and the dispatcher never finds the run it started. The runner itself does not consume it.
• runner-version — the group’s own version (internal/version.Version), not a configured value. The runner is therefore always the same build as the group that started it, so the two can never disagree about the wire contract. For a backend-dispatched group the dispatcher runs inside the backend, so the runner matches the backend too.

A runner group built from source reports dev and has no release to point at, so it refuses to dispatch rather than sending a version that cannot be downloaded. The reason lands on the group’s last_dispatch_error, visible in the UI. Develop against the local, docker, or kubernetes launchers, which run a locally built binary or image.

Upgrading an existing workflow

runner-version was added in v0.0.2, and the template that preceded it pinned the runner itself with a GANTRYCD_RUNNER_VERSION env var. Replace the whole workflow file when you upgrade past v0.0.1. A dispatch carrying an input the workflow does not declare fails with 422 Unexpected inputs provided, so an old file stops accepting work the moment the backend starts sending the version — the failure lands on the group’s last_dispatch_error.

This is the only upgrade step github-actions groups have; there is no separate process to update, because the backend dispatches them itself. Pull-based groups have the opposite problem — their binary must be upgraded no later than the backend — see Runner groups.

Architecture

   run enters 'pending'                ┌─────────────────────────────────────┐
   (create / promote /                 │  dispatcher tick — DispatchPending   │
    expiry-reset)                      │                                      │
       │                               │  1. query up to GANTRYCD_DISPATCH_     │
       │ pg_notify('run_pending')      │     BATCH_SIZE pending runs a         │
       ▼                               │     github-actions group can serve   │
   ┌───────────────┐  Wake()           │     (LATERAL join, label match)      │
   │ run_pending   │ ───────────────▶  │  2. group candidates by org; orgs    │
   │ LISTEN, every │                   │     run in parallel, each org seq.:  │
   │ replica       │                   │     a. caps check                    │
   └───────────────┘                   │     b. PollForWork → claim 1 run     │
   ┌───────────────┐  every 30s        │        (stub runner + exclusive      │
   │ scheduler job │ ───────────────▶  │         assignment, SKIP LOCKED)     │
   │ (leader, also │                   │     c. sign ephemeral runner JWT     │
   │  = heartbeat) │                   │     d. workflow_dispatch with JWT    │
   └───────────────┘                   └─────────────────────────────────────┘
                                                        │
                                                        ▼
                          GitHub starts the workflow → runner @ group's version
                          → standard runner protocol (ready → accept → done)

Event-driven. The dispatcher does not poll. When a run enters pending (creation, PromoteNextOnStack, or assignment-expiry reset) the run repository fires pg_notify('run_pending') inside that write transaction. Every replica holds a LISTEN on the channel; the notification calls Wake(), which signals a per-replica event worker to run a dispatch tick. An idle deployment does no dispatcher database work at all.

A leader-elected scheduler job (github-actions-dispatcher, every 30 s) is the failover: it calls DispatchAll to drain the entire pending backlog in case a NOTIFY was missed (notifications are dropped if no replica is listening at that instant — reconnects, restarts), and, by being claimed, it keeps the connection-status heartbeat (background_job_schedules.last_started_at) fresh.

Query-driven tick. ListDispatchCandidates(type) — a LATERAL join from pending runs to a matching group of the requested type — returns only runs that actually have somewhere to go, and skips groups already at their per-group cap (runner_group_configs.max_concurrent_runs) so a run matching several groups is paired with one that has room. The query is type-agnostic — the type is the only per-provider input. Cost scales with the pending-run backlog, not the number of orgs or groups. GANTRYCD_DISPATCH_BATCH_SIZE (default 256) bounds a single tick; DispatchAll then re-runs ticks until one comes back short of a full batch, so both the event worker and the failover job fully drain the backlog rather than handling one batch. Within a tick, candidates are grouped by org and worked in parallel.

Runs on every replica. The event worker is not DB-locked — every replica dispatches concurrently, so throughput scales horizontally with replica count. This is safe because the claim (SetExclusiveAssignment, FOR UPDATE SKIP LOCKED) guarantees each run is dispatched exactly once no matter how many replicas race for it. The trade-off: concurrency caps are best-effort — two replicas can both pass the cap check before either claims, so a cap may transiently overshoot by up to (replicas − 1). Caps are throttles, not safety limits, and the overshoot self-corrects on the next tick.

github-actions groups never long-poll, so they have no last_poll_at of their own. Their connection status is derived from the dispatcher scheduler job’s last run (background_job_schedules): connected while that job is fresh, disconnected once it goes stale. The dispatcher writes nothing per group.

The run_pending channel is keyed only on “a run became pending”, not on runner-group type — any future backend-dispatched runner-group type reacts to the same notification.

Per-group keypair

Every runner group has an ECDSA P-256 keypair. For self-hosted groups, gantrycd returns the private key once and the operator copies it to the runner. For github-actions groups, gantrycd generates the keypair, stores both halves, and the operator never sees either:

• runner_groups.public_key — verifies the ephemeral JWT presented by the workflow (same path as today’s runners).
• runner_group_configs.signing_private_key_ciphertext — encrypted at rest with the platform data cipher (GANTRYCD_DATA_ENCRYPTION_PROVIDER, see data_encryption.md); used by the dispatch path (RunnerGroupService.DrainPending) to sign assignment tokens (generically, for every backend-dispatched type). AAD (runner_group_configs:signing/<id>) binds it to the runner_group_id, so a row swap fails to decrypt.

The GitHub App credentials (app_id + private key) live inside the type-specific config blob runner_group_configs.config_ciphertext, encrypted with its own AAD (runner_group_configs:config/<id>). The signing key and the config blob are never decrypted into the same buffer.

Concurrency caps

Two gates run before each claim:

• Per group: runner_group_configs.max_concurrent_runs (default 5). Counts runners in not-ready / ready / busy for this group.
• Per org: GANTRYCD_DISPATCH_MAX_PER_ORG env var (default 20, set to 0 to disable). Counts runners across every github-actions group in the org.

Both checks happen inside the same read tx; the claim is a separate write tx using the existing SKIP LOCKED machinery, so two backend pods racing on the same run is safe (one wins, the other moves on).

Failure modes

There is no special “cancel the GitHub workflow” path — when a gantrycd run is cancelled mid-flight, the ephemeral runner observes requested_cancellation=true on its periodic /runs/{id}/status GET and shuts itself down.

Job timeout vs the run budget

timeout-minutes and RUN_TTL bound the same run from opposite ends, and only one of them can end it cleanly. The runner must always be the one that stops the run; GitHub’s timeout is a backstop for a runner that has stopped responding at all.

Two knobs set the run’s own budget (both backend-side — see Configuration):

So at defaults the runner self-cancels at 50m and is done by 60m. The job must outlive that:

timeout-minutes  >  RUN_TTL        (in minutes)

Note it is > RUN_TTL, not > RUN_TTL + grace — the grace is subtracted from the TTL to get the self-cancel instant, not added after it. RUN_TTL=1h therefore needs timeout-minutes above 60; the templates use 75, which leaves room for the workflow’s own setup steps (checkout, Go install, tenv) that run before the budget starts.

Change them together. Raising RUN_TTL without raising timeout-minutes is the failure this guidance exists to prevent: GitHub kills the job mid-shutdown, so the runner never reports and the run is only reclaimed later by the heartbeat-timeout path — losing the logs and plan artifact it was in the middle of uploading. Setting timeout-minutes equal to RUN_TTL has the same effect with zero margin.

A run bounded by an earlier credential expiry (a short AWS MaxSessionDuration) finishes sooner than the TTL, never later, so it needs no separate allowance here — see Runner runtime credentials.

Storage endpoints for off-network runners

A github-actions workflow runs on GitHub’s infrastructure, not inside your network. The state, logs, and artifacts S3 credentials gantrycd hands the runner carry an endpoint — and if that endpoint is a cluster-internal address (http://minio.internal:9000), the workflow cannot reach it and runs fail at the first S3 call.

Set <CONCERN>_S3_PUBLIC_ENDPOINT (for STATE, LOGS, ARTIFACTS) to a publicly-reachable address. The backend keeps dialing <CONCERN>_S3_ENDPOINT for its own reads; only the credentials handed to runners carry the public endpoint. When PUBLIC_ENDPOINT is unset it defaults to ENDPOINT, so single-network deployments need no extra config. See Configuration.

Adding a new managed-runner type

Backend-dispatched runner-group types share one generic dispatch path on RunnerGroupService; each type plugs in as a small pkg/extensions/runnergroup implementation. The split:

• pkg/extensions/runnergroup — the dependency-light extension point. Dispatcher is the single backend hook (Dispatch(ctx, config, LaunchSpec) error); Config is the per-type operator config (Validate/Redacted/MaxConcurrentRuns), produced by a free Unmarshal. LaunchSpec (group/run/runner IDs, the minted runner token, the backend callback URL) is what every dispatched runner receives.
• pkg/extensions/runnergroup/<type> (e.g. gha) — the per-type implementation: a Config struct + Unmarshal, plus a Dispatcher that decodes the stored config blob and brings one runner online from the LaunchSpec. No SQL, no config table, no signing — those are generic.
• internal/backend/services/runnergroup — the Registry of Type{Name, Unmarshal, Dispatcher} values. A Type with a Dispatcher is backend-dispatched (signing key retained server-side); one without (self-hosted) is external/pull-based.
• RunnerGroupService dispatch path (runner_group_dispatch.go) — the generic engine, enabled via WithDispatch(...). DrainPending(type) runs the candidate query (RunRepository.ListDispatchCandidates(type)), the per-group + per-org cap arithmetic, the claim path (PollForWork), the ephemeral-runner JWT signing (from the shared runner_group_configs table), and the Dispatcher.Dispatch call. DispatcherStatus(type) backs connection status; DispatcherJobName(type) is the derived failover/heartbeat job key "<type>-dispatcher".
• cmd/backend runtime + scheduler — the runtime starts one coalescing wake worker per DispatchedTypes() entry (the run_pending NOTIFY fans to every worker’s wake channel), each calling DrainPending; the scheduler registers one leader-elected failover job per type. Cancellation is not propagated upstream — it flows through the backend run state and the ephemeral runner observes it.

To add GitLab CI / Argo / BuildKite / etc.:

1. Implement Config + Unmarshal + Dispatcher in pkg/extensions/runnergroup/<type>/.
2. Register one more runnergroup.Type{Name, Unmarshal, Dispatcher} in cmd/backend/services.go. The shared runner_group_configs table, the generic dispatch path, the runtime wake workers, the scheduler failover job, and the handler connection-status read all pick it up with no further changes — no new table, repo, candidate query, service, or engine.

The rest of the system — runner state machine, RunContext, log streaming, deployment lifecycle, and the downstream HTTP runner protocol — is shared and unchanged.

API surface

App private key is write-only. It is encrypted at rest with the platform data cipher and never returned by GET/LIST. Any PUT that includes github_config must re-supply the private key (the UI prompts for it on every edit; CLI/Terraform callers must pass it explicitly).

Example: update workflow ref + concurrency

curl -X PUT https://gantrycd.example/api/v1/orgs/$ORG/runner-groups/$GID \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "labels": ["production", "terraform"],
    "github_config": {
      "repository_owner": "acme",
      "repository_name":  "infra",
      "workflow_file":    "gantrycd-runner.yaml",
      "workflow_ref":     "release",
      "max_concurrent_runs": 10,
      "app_id":      123456,
      "private_key": "-----BEGIN RSA PRIVATE KEY-----\n…\n-----END RSA PRIVATE KEY-----"
    }
  }'

See also

• runner_polling.md — the shared poll/accept protocol.
• runner_runtime_credentials.md — what RunContext carries at accept time.
• deployment_state_machine.md — how runs transition through states.