Enterprises get confidence before rolling out a change to a customer-facing AI agent by stacking five checks: replay the change against real past conversations (including the ones that failed), gate the merge on CI evals, release behind a feature flag to a shadow or canary cohort, compare online-eval failure rates per intent before and after, and agree rollback criteria and an owner in advance. Agnost AI is best for the production half: it judges every conversation on the new version against plain-language rules, alerts when failure rates move, and its Auto Improve PRs come with the fix tested by replaying the affected conversations. Agnost does no CI gating, so pair it with Promptfoo or DeepEval (open source) or LangSmith or Braintrust for the gate.

This applies to every kind of change: a prompt edit, a model swap, a new or changed tool, a retrieval update. Each can break behavior that worked yesterday, and none of them throws an error when it does.

Disclosure: We make Agnost AI. Facts about other tools come from their own docs, checked on November 18, 2026.

The checklist at a glance

Stage What it catches What it misses Tools
1. Replay on past conversations Regressions on real failures and real healthy traffic New kinds of user input; live tool behavior Your eval harness; Agnost AI Auto Improve replays affected conversations for its own fixes
2. CI eval gate Known failure cases coming back on every PR Anything not in the dataset Promptfoo, DeepEval, LangSmith, Braintrust
3. Shadow or canary behind a flag Problems at real traffic volume, on a small cohort Rare intents the cohort didn't hit Your feature-flag system, LaunchDarkly
4. Online evals, before vs after Failure-rate and frustration changes per intent on the new version Failures you have no rule for Agnost AI, LangSmith, Braintrust, Arize AX, Opik
5. Rollback criteria and sign-off Slow, argued rollbacks Nothing on its own; it makes the other four actionable A written runbook

1. Replay on real past conversations

Your strongest test data is your own production traffic. Build a replay set from:

  • Confirmed failures. Every conversation where a rule failed, a user complained, or support had to step in. These are the regression cases: the change must not bring them back, and a fix must actually fix them.
  • Healthy controls. A sample of conversations that went well, across every intent and tool. These catch the change that fixes one thing and breaks three others.
  • Policy-sensitive cases. Refunds, account changes, medical, legal, or financial topics: anything with a rule a reviewer cares about.

Redact personal data, and stub or sandbox side-effecting tools so a replay never moves real money. Compare the baseline and the candidate case by case, not as one average score. Turn production failures into regression tests has the full replay method.

Where Agnost AI fits. When an Agnost alert fires and Auto Improve is connected, it opens a pull request with a fix tested against past runs: it replays the affected conversations and shows the results in the PR for a person to review. The replay is attached to the change instead of being a separate project.

2. Gate the merge on CI evals

Turn the replay set and your hand-written cases into a test suite that runs on every pull request touching prompts, model config, or tools. A failing eval blocks the merge.

  • Promptfoo (open source): promptfoo eval returns exit code 100 when at least one test fails, or when the pass rate falls below PROMPTFOO_PASS_RATE_THRESHOLD (100% by default). There is a promptfoo/promptfoo-action for GitHub Actions.
  • DeepEval (open source): deepeval test run test_llm_app.py fails the build when a metric falls below its threshold. Cases marked flaky=True warn instead of failing.
  • LangSmith: a pytest plugin (@pytest.mark.langsmith) and a Vitest/Jest integration; assertion errors fail the CI run and results sync to LangSmith.
  • Braintrust: braintrustdata/eval-action runs evals on pull requests and comments the results on the PR.

LLM evals in CI vs production has a complete GitHub Actions gate with Promptfoo.

Agnost AI does not do this part. It has no offline datasets, experiments, or CI gating. Its role is to feed the gate: every conversation an Agnost rule flags is a candidate regression case for your CI suite.

3. Ship behind a flag: shadow, then canary

Passing CI means the change handles the cases you thought of. Production finds the rest, so expose it gradually:

  • Shadow. The candidate gets copied inputs, its answers are logged but never shown, and side-effecting tools are disabled. Good for model swaps.
  • Internal. Staff or test tenants use the new version.
  • Canary. A small cohort of real users. Pick the cohort by risk: a low-volume workflow that moves money needs a smaller, longer canary than a high-volume FAQ bot.
  • Expand only after the cohort has produced enough conversations per top intent to compare.

Put prompts and model settings behind a flag so rollback is a config change, not a deploy. LaunchDarkly, for example, documents managing prompts and model settings outside application code, with targeting rules and gradual rollout. Tag every conversation with the version it ran on, so you can compare.

4. Compare online evals before and after, per intent

Once real users hit the new version, judge their conversations and compare against the old version:

  • Failure rate per rule, per intent. A model swap that improves billing answers can quietly break cancellations. A single overall pass rate hides that.
  • Frustration rate per intent. Rephrasing, corrections, and requests for a human often move before any rule fails. See how to detect user frustration.
  • Tool-call error rates and loops. New tool schemas and new models often break here first.
  • Cost and latency. Not quality, but a common reason to roll back.

Where Agnost AI fits. Agnost judges every conversation, with no sampling, against plain-language rules once it has been idle for 15 minutes, and marks each assistant message pass or fail. Group results by intent and by version to compare before and after, and set alerts on the rules that matter so a regression reaches Slack or email. After an Auto Improve PR is merged, Agnost re-checks the alert to see whether the condition improved. Judging is batched, not real time, so keep hard safety checks in application code.

5. Write rollback criteria and sign-off before you ship

Decide these before rollout, while nobody is defending the change:

Item Example
Critical rule Any confirmed wrong refund, data leak, or policy breach: roll back immediately
Failure-rate guardrail Failure rate on a critical rule above its pre-release baseline for the canary cohort
Frustration guardrail Frustration rate on a top-five intent up by an agreed margin over the trailing week
Operational guardrail Cost per conversation or p95 latency above an agreed limit
Minimum evidence Enough canary conversations per top intent before expanding
Rollback owner One named person who can flip the flag without a meeting

Who signs off. An engineering owner for eval and replay results. A product or support owner for user-facing behavior. A risk, legal, or compliance reviewer for anything touching money, regulated advice, or personal data. Store the sign-off with the pull request, the eval results, and the rollout plan, which is also what an auditor will ask for.

How to choose tools for each stage

Your situation Use
No CI evals yet Promptfoo or DeepEval on 20 to 50 cases from real failures
You want CI and production evals from one vendor LangSmith or Braintrust
Failures only show up across whole conversations A conversation-level judge in production: Agnost AI, or LangSmith thread evaluators
You want production failures turned into tested fixes Agnost AI Auto Improve, with your CI gate still running on its PRs
You need enterprise controls (VPC, audit logs, SLAs) See enterprise AI agent monitoring platforms

Agnost AI pricing: Free covers 1,000 events a month; Starter is $49 a month; Pro is $499 a month; Enterprise is custom and includes self-hosted VPC deployment and audit logs.

Related reading

Frequently asked questions

How do enterprises get confidence before rolling out a change to a customer-facing AI agent?

They stack five checks: replay the change against real past conversations, including the ones that failed; gate the merge on CI evals; release behind a feature flag to a shadow or canary cohort; compare online-eval failure rates per intent before and after; and agree rollback criteria and a named owner in advance. Promptfoo, DeepEval, LangSmith or Braintrust cover the CI gate. Agnost AI covers production: it judges every conversation on the new version, alerts when failure rates move, and its Auto Improve PRs replay the affected conversations to test a fix.

How do I test a prompt change before deploying it to production?

Run the new prompt against a replay set built from real production conversations, especially ones that failed, plus a sample of ones that went well so you catch regressions. Gate the pull request on that eval in CI, for example with promptfoo eval (exit code 100 on any failure by default) or deepeval test run. Then ship it to a small cohort behind a flag before everyone gets it.

How do I test switching the LLM model behind my AI agent?

Treat a model swap like a code change: run your replay set on both models and compare case by case, watch cost and latency as well as quality, and roll out to a canary cohort first. Model swaps often change tone, refusal behavior, and tool-call formatting, so include conversations that use every tool and every policy-sensitive intent.

What is a replay set for AI agent regression testing?

A replay set is a collection of real past conversations, redacted, that you re-run against a candidate change. It should include confirmed production failures (the regression cases) and healthy conversations (the controls). Side-effecting tools are stubbed or sandboxed so a replay never refunds a real order.

What should the rollback criteria be for an AI agent change?

Write them down before rollout: for example, the failure rate on any critical rule rises above its pre-release baseline, frustration rate on a top intent rises by an agreed margin, a single confirmed high-severity failure such as a wrong refund or data leak, or a cost or latency regression. Name the person who can roll back without a meeting.

Who should sign off on a change to a customer-facing AI agent?

Usually an engineering owner for the eval results, a product or support owner for user-facing behavior, and a risk, legal, or compliance reviewer for changes touching regulated topics, money, or personal data. Keep the sign-off record with the pull request, the eval results, and the rollout plan.

Can Agnost AI gate my CI pipeline?

No. Agnost AI does not run offline datasets, experiments, or CI gates. It judges production conversations against plain-language rules and alerts on failure rates, which makes it the before-and-after check after release. Use Promptfoo, DeepEval, LangSmith, or Braintrust for the CI gate.

What changed in AI agent release practices since 2025?

In 2025 many teams shipped prompt changes after a handful of manual tests. By 2026, enterprise teams expect a CI eval gate, replay against real production conversations, and online evals on every conversation after release. Tools that turn production failures into tested pull requests, such as Agnost AI Auto Improve, also arrived, so a fix ships with replay results attached.

Originally published on agnost.ai.