Friday AI Question, round two. This one is about a moment every AI team hits weekly:

Someone changes the system prompt. What happens before it reaches users?

Pick one (and tell me in the comments):

  • A) Edit it in production and watch for complaints.
  • B) Try a handful of examples by hand. If it "looks better", ship it.
  • C) Run a saved eval set on every prompt change before merge, and compare against the current version.
  • D) C, plus continuous evaluation: mine production logs for new cases and grow the eval set over time.

My lean: C as the minimum, D as the goal. Here's why.

B has a name, and it's an anti-pattern

OpenAI's Evaluation best practices guide lists its anti-patterns plainly. One of them is "vibe-based evals": using "it seems like it's working" as an evaluation strategy, or waiting until you ship before implementing any evals.

That is option B. A prompt change that fixes the few examples you tried can quietly break the ones you didn't. Models are nondeterministic, so the same input can give a different output, which the guide says makes traditional software testing insufficient on its own.

Another anti-pattern on the same list: biased design, meaning eval datasets that don't reproduce production traffic patterns. Hand-picked examples are almost always biased toward the case you were just fixing.

What C looks like in practice

Treat the prompt like code and the eval set like its test suite:

  1. Define success first. Anthropic's guide on building evals asks for criteria that are specific and measurable. "Classify sentiment well" is bad; their example of good is an F1 of at least 0.85 on a held-out test set, as an improvement over the current baseline.
  2. Keep a held-out set of real inputs with expected outputs or pass/fail criteria. Include edge cases: Anthropic's list includes irrelevant or nonexistent input, overly long input, harmful input, and ambiguous cases where even humans would disagree.
  3. Automate the grading. Both guides push for automated scoring: exact match, string match, code-graded checks, or an LLM grader. Anthropic even says to prioritize volume over quality: more questions with slightly lower-signal automated grading beats fewer hand-graded ones.
  4. Compare against the baseline. Run the old prompt and the new prompt on the same set. Ship only if the new one is at least as good on the metrics you defined.

If you use an LLM judge, calibrate it. OpenAI's guide recommends validating judge agreement against human labels, and warns about position bias and verbosity bias.

Why D is the goal

OpenAI's eval process ends with step 5: continuously evaluate. Run evals on every change, monitor the app for new cases of nondeterminism, and grow the eval set over time. Their tip list includes "log everything" so you can mine your logs for good eval cases.

That's the difference between C and D: C protects you from regressions you already know about. D keeps finding the ones you don't.

A small timely note

If your CI currently leans on OpenAI's hosted Evals platform, check its deprecation page. The best-practices guide says Evals became read-only for existing users on October 31, 2026, and the platform is scheduled to shut down on November 30, 2026. Your eval data and criteria are the valuable part. Keep them in your repo, not only in a vendor dashboard.

A minimal checklist

  • Prompts live in version control, and every change is a reviewable diff.
  • A saved eval set runs automatically on every prompt change.
  • Old vs new prompt scored on the same set, with a pass threshold.
  • Edge cases in the set, not just happy paths.
  • LLM judge (if any) checked against human labels.
  • New failures from production get added back to the set.

So: A, B, C or D? And if you're on C or D, how many cases are in your eval set today?

References