This week we ran our free repository check on openai/openai-quickstart-python. It still calls gpt-3.5-turbo in 4 places, a model OpenAI stopped serving on October 23, 2026.

That's not a dig at OpenAI. It's how this happens in almost every codebase, including ones with good test coverage.

The suite is green. The pull request merges. The deploy goes out. And every request to the LLM comes back with an error like model_not_found, because the model ID in the config was retired the night before.

The tests didn't fail because the tests never talk to the model.

Why mocking hides it

Mocking the LLM client is the right decision:

@patch("app.llm.client.chat.completions.create") def test_summary(mock_create): mock_create.return_value = fake_response("A short summary.") assert summarize("long text") == "A short summary."

It's fast, it's free and it's deterministic. But look at what it checks: your code's handling of a response. It never checks that the request would be accepted. The model ID is just a string that goes into a mock and disappears.

So when OpenAI retired gpt-4-0613 on October 23, a repository still sending it stayed green right up until real traffic hit it.

Three cheap checks

1. Put model IDs in one place.

# config.py SUMMARY_MODEL = "gpt-5.6-sol" CLASSIFIER_MODEL = "claude-sonnet-5-5"

If model names are scattered across ten files, no check will find them all.

2. Check them against shutdown dates in CI, not against the API.

Vendors publish retirement dates months ahead. A CI step that reads your config and compares it to a list of dates fails the build before the date, with no API key and no network call to the model. (Versyka's GitHub Action does this, and the retirement calendar has the dates as JSON if you'd rather script it.)

3. One real smoke call per model on deploy.

for model in {SUMMARY_MODEL, CLASSIFIER_MODEL}: client.chat.completions.create(model=model, max_tokens=1, messages=[{"role": "user", "content": "ping"}])

A one-token request per model costs almost nothing and fails the deploy, not the user.

The next date

Claude Sonnet 4.5's dated ID, claude-sonnet-4-5-20250929, shuts down on November 30. If it's in your repository, that's a good one to test these checks on.

Where do the model names live in your codebase: one config file, or scattered through the code?