Here is a failure most teams have seen at least once. Two pull requests are open against main. Both have green checks. Both get approved. Both merge within an hour of each other. Main fails to build.
Nobody did anything wrong. Each change was tested, just not against the code it ended up next to.
How it happens
CI on a pull request usually tests the branch, or the branch merged with main as main looked when the pipeline ran. Once one pull request merges, every other open pull request was tested against a version of main that no longer exists.
Most of the time that doesn't matter. Sometimes it does:
- Renames. Pull request 1 renames
getUsertofetchUserand updates every caller. Pull request 2, opened earlier, adds a new call togetUser. Git sees no conflict because the two changes touch different lines. The merged code calls a function that's gone. - Changed contracts. One change makes a field required; another adds a code path that doesn't set it.
- Config and schema. One change removes a config key or migrates a table column; another starts reading it.
- Tests that relied on old behaviour. A test added in one branch passes against old main and fails against new main.
These are sometimes called semantic merge conflicts: Git merges the text cleanly, but the program no longer works.
Why AI agents make it more common
The chance of this goes up with the number of changes open at the same time and how long they stay open. AI coding agents push on both. A team that used to have a handful of open pull requests can have many more, each produced quickly and each tested in isolation. Agents also tend to make broad, mechanical changes (renames, signature changes, refactors) which are exactly the kind that break code elsewhere.
What "tested" should mean
The fix is a rule: the code that lands on main should have passed your tests in exactly that state. Not the branch on its own, not the branch merged with an older main.
There are a few established ways to get there.
Require branches to be up to date. Many Git hosts can require a branch to include the latest main before merging. It works, but every merge forces every other open branch to update and re-run CI, which gets slow when many changes are in flight.
Merge queues. GitHub's merge queue adds pull requests to a queue and creates a temporary branch containing the latest target branch plus the changes ahead of it in the queue. Required checks run on that combined state. If you use GitHub Actions, your workflows need to run on the merge_group event as well as pull_request, or the required checks never report and the merge doesn't happen.
Merge trains. GitLab's merge trains (Premium and Ultimate tiers) do the same thing: each merge request's pipeline runs on its changes combined with the target branch and the merge requests ahead of it in the train. If one fails, it drops out and the pipelines behind it restart without it.
Re-test on change. Whatever mechanism you use, a new commit on the branch or a moved target should invalidate the earlier result. A green check from an hour ago describes code from an hour ago.
A checklist for your setup
- Do your required checks run on the merged result, or only on the branch?
- If main moves after a check passes, does the check run again before merge?
- Does a new push to a pull request reset its checks and its approval?
- Can an agent or bot merge a change whose checks ran on an earlier commit?
- When a check errors or times out, does the merge stop, or go through?
If the answer to the last one is "it goes through," fix that first.
How AI CodeOps handles it
In AI CodeOps, AI agents plan, build, review and test changes on your repository, and the release gate decides whether a pull request opens. Your own tests run in a sandbox on the exact commit that will merge. If that commit wasn't tested, or the tests didn't pass on it, the pull request doesn't open. The AI review of the real diff must have no open high or critical findings, and a person must approve. If any step can't complete, the gate fails closed and nothing merges.
Every change also gets a signed evidence pack recording which commit was tested, what the review found and who approved it, so "were the tests run on what shipped?" has a direct answer.
Merge queues and merge trains solve the ordering problem on your Git host; a gate like this makes sure an AI-written change only gets that far with evidence attached. Used together, green means what people assume it means.
AI CodeOps works with GitHub, GitLab, Bitbucket, Gitea and Forgejo, including self-hosted. Free during the open beta, and free forever for non-commercial open-source projects: https://aicodeops.dev

