Scope GitHub Actions concurrency to your Terraform state, not your branch

We turned on auto-merge for Dependabot security PRs. Within a day, deploys started failing with a GCS 412 Precondition Failed on the Terraform state lock.

The cause is mechanical: the sweeper merges several PRs for the same service back to back, every merge pushes main, every push to main starts a deploy workflow, and every deploy runs terraform apply against the same state object. Two apply runs, one lock, one loser.

concurrency is the right tool. But the obvious way to apply it is wrong in three separate ways, and we hit all three before landing on an eleven-line block. This is what we learned.

The primitive

concurrency: group: <any string or expression> cancel-in-progress: false queue: single # default 

Two facts about the runtime matter more than the syntax:

• A concurrency group holds at most one running and one pending entry. Group names are case-insensitive and scoped to the repository.
• queue: single (the default) means a newly arriving run cancels whatever is sitting in the pending slot and takes its place. cancel-in-progress: false protects the running entry. It does nothing for the pending one. queue: max raises the pending capacity to 100 and processes FIFO, but it cannot be combined with cancel-in-progress: true — that is a workflow validation error.

That second point is the one people get wrong, and it is the difference between "every deploy eventually runs" and "only the latest deploy runs."

Rule 1: the group key is the lock identity

The reflex is group: ${{ github.workflow }}-${{ github.ref }}. That is the right key for PR CI, where the thing you are protecting is a branch's build. It is the wrong key for Terraform, where the thing you are protecting is a state object.

Two different workflows — a push to main and a workflow_dispatch — touch the same state and must serialize. Two pushes to different branches that deploy different services must not. The ref tells you neither.

Our backend is GCS with a per-service prefix and a Terraform workspace per environment:

# my-service/terraform/main.tf backend "gcs" { bucket = "tf-state-example" prefix = "terraform/my-service" } 

terraform workspace select "${ENVIRONMENT}" # dev | prod 

So the lock identity is prefix × workspace = service × environment. That is exactly what the group key encodes:

# .github/workflows/my-service.yml deploy: runs-on: ubuntu-latest concurrency: group: tfstate-my-service-${{ github.event_name == 'workflow_dispatch' && inputs.environment || startsWith(github.ref, 'refs/tags/') && 'prod' || 'dev' }} cancel-in-progress: false if: | (github.event_name == 'push' && (github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/my-service-v'))) || github.event_name == 'workflow_dispatch' 

The environment is in the group on purpose, not by accident. On a release we push a tag and merge to main at nearly the same moment: the tag run deploys prod, the main run deploys dev. Those are different workspaces, therefore different state objects, therefore no lock contention. Keeping environment in the key lets them run in parallel. We have the receipt — on one release the prod tag run and the main run started three seconds apart and both succeeded.

Collapse that to a single group: tfstate-my-service and you have just serialized two deploys that never needed to be serialized, doubling release wall-clock for no benefit.

Rule 2: cancel-in-progress: false is mandatory, and it is not enough

Killing a terraform apply mid-flight is the worst outcome available. The provider is in the middle of API calls; the process dies between "resource created" and "state written"; now your state and your infrastructure disagree and someone is doing a manual terraform import on a Friday. cancel-in-progress: false is non-negotiable here.

But remember the pending slot. With the default queue: single, if deploy A is applying and B and C both arrive, B gets evicted by C and never runs.

For us that is fine, and it is worth being explicit about why rather than just accepting it: on main, commit C contains commit B. Running only C converges to the same infrastructure. The same holds for tags, where a later tag includes the earlier one.

If your deploys are not cumulative — per-tenant applies, migrations that must each execute, anything where skipping an intermediate run loses work — then queue: max is what you want, and you accept the FIFO backlog that comes with it.

Rule 3: keep the read-only job out of the group

This is where we burned the most time.

The workflow has two Terraform jobs: plan (a reusable workflow, PR diff preview) and deploy (apply). The first instinct is that both touch state, so both belong in the group.

Attempt 1 put concurrency on both. The group expressions did not match, because the reusable terraform-plan.yml did not receive the environment input and always planned the dev workspace even when you dispatched with environment: prod.

Attempt 2 fixed that properly — added an environment input to the reusable workflow, threaded inputs.environment through, and the two group expressions became identical. The stated justification was that plan and deploy run concurrently within the same run, since deploy has no needs: plan, and a run-level concurrency cannot protect against same-run contention.

That justification was simply false. plan has needs: build, and build only runs on feature branches:

build: if: | github.event_name == 'push' && !startsWith(github.ref, 'refs/heads/main') && !startsWith(github.ref, 'refs/tags/') 

So plan never executes on a main push, a tag, or a dispatch — which is to say, it never shares a run with deploy. The entire premise was wrong, and we had modified a shared reusable workflow to satisfy it.

The real contention is cross-run: a feature branch's plan against some other run's apply. And since plan is read-only, that is solved at the Terraform layer instead:

# my-service/scripts/ci.sh # plan is read-only. In CI it can run alongside deploy's terraform apply, # so it does not take the state lock (-lock=false). Taking it fails with GCS 412. terraform plan -lock=false -var="image_tag=${TAG}" 

Now, the decisive argument against the concurrency-group approach for plan, which is the part worth stealing:

A group keyed on service × environment spans every branch. Put plan in it and a push on branch X can evict branch Y's pending plan. A cancelled job reports CANCELLED — and our auto-merge gate treats CANCELLED exactly like FAILURE:

elif (.conclusion == "FAILURE" or .conclusion == "TIMED_OUT" or .conclusion == "CANCELLED" or .conclusion == "ACTION_REQUIRED" or .conclusion == "STARTUP_FAILURE") then "failure" 

So the concurrency control you added to protect deploys would turn unrelated PRs red and block the very auto-merge sweep that started this whole investigation. Serialization primitives that are correct for a mutating job are actively harmful when applied to a read-only one sharing the same key.

The end state is one concurrency block per workflow, on deploy only. The reusable plan workflow went back to having no environment input at all.

Workflow level or job level?

Top-level concurrency groups the entire workflow run. Job-level groups that one job. Both are valid schema and both can be present, evaluated independently.

For a build-plan-deploy pipeline, job level is the only correct choice. The run as a whole is not the unit that contends for the lock — one specific job is. Hoisting it to the top would drag build and plan into the same serialization domain and reproduce exactly the cross-branch cancellation problem above.

Division of labor with the backend lock

Keep both. They fail differently.

The backend lock is correctness; it catches the human running apply locally. concurrency is ergonomics; it converts "the second run errors out" into "the second run waits its turn," which is what keeps a back-to-back merge sweep green.

Summary

• Key the group on what holds the lock — backend prefix × workspace — not on github.ref.
• Keep environment in the key so dev and prod deploys stay parallel. They are separate state objects.
• cancel-in-progress: false always, for anything running apply.
• Know that it does not protect the pending slot. Default queue: single evicts it. Reach for queue: max only if your deploys are not cumulative.
• Put read-only plan outside the group and give it -lock=false. A cross-branch group will cancel pending CI, and a cancelled check reads as a failure to every merge gate you own.
• Job level, not workflow level.