AI search answers change every day for the same question: in our monitoring of five AI engines, a brand named in today's answer was named again in tomorrow's answer only 67% of the time at the median, and the two days' brand lists had a median Jaccard overlap of 0.40. After a week the overlap falls to 0.33 and then stays near 0.29 to 0.33 out to 28 days, so most of the change is day-to-day noise around a stable pool of brands, with a slow drift on top.

  • ChatGPT and Bing AI Mode change least, Google changes most. The median share of brands kept from one day to the next was 75% on ChatGPT and Bing AI Mode, 63% on Gemini, 57% on Google AI Mode and 50% on Google AI Overview.
  • An identical answer is rare. Only 14% of consecutive-day pairs named exactly the same set of brands.
  • The pool is stable while each draw varies. On each engine with four weeks of data, overlap after 28 days is at most 0.1 below overlap after 7 days, so a brand that drops out of today's answer often comes back.

How quickly do AI search answers change for the same question?

We ran a fixed set of prompts once a day on each engine from October 17 to November 15, 2026, and compared each answer with the answer to the same prompt on the same engine on a later day. The prompts are buyer questions about search, SERP and AI-tracking APIs, and "brands" are the vendors those answers name, matched against a fixed list of vendor names.

Two measures, both per pair of answers:

  • Brands kept: of the brands named on day t, the share also named on day t + k.
  • Jaccard overlap: brands named on both days divided by brands named on either day. 1 is the same list, 0 is no brand in common.

Day to day (k = 1), medians with the middle half in brackets:

Engine Brands kept Jaccard overlap Same brand list
ChatGPT 75% (50% to 100%) 0.50 (0.25 to 0.75) 15%
Bing AI Mode 75% (50% to 100%) 0.50 (0.20 to 0.80) 19%
Gemini 63% (38% to 86%) 0.40 (0.20 to 0.63) 11%
Google AI Mode 57% (17% to 100%) 0.33 (0 to 0.67) 19%
Google AI Overview 50% (0% to 100%) 0.29 (0 to 0.57) 11%
All five 67% (33% to 100%) 0.40 (0 to 0.67) 14%

The wide ranges matter as much as the medians. On Google AI Overview and AI Mode, the lower quartile of brands kept is 0% to 17%: in a quarter of day pairs, the answer named almost none of the brands it named the day before. Bing AI Mode joined our monitoring on November 7, so its row covers 9 days.

Do AI answers drift over weeks, or only change day to day?

Mostly the second. If answers drifted steadily, overlap would keep falling the further apart two answers are. It does not:

Days apart Brands kept Jaccard overlap Same brand list
1 67% 0.40 14%
3 60% 0.36 12%
7 60% 0.33 10%
14 50% 0.33 8%
28 60% 0.29 8%

Medians over all five engines. Bing AI Mode has 9 days, so it adds pairs only to the rows up to 7 days.

The Jaccard overlap drops from 0.40 to 0.33 in the first week and then flattens. Per engine, the drop from 1 day to 28 days is 0.29 to 0.22 on Google AI Overview, 0.33 to 0.20 on AI Mode, 0.50 to 0.40 on ChatGPT and 0.40 to 0.33 on Gemini. So each engine draws from a pool of brands that changes slowly, and each answer is a different sample from that pool.

This has a direct consequence for tracking. A brand missing from one answer has not lost its place. The useful number is the share of answers that name it over many runs, and a change in that share is real only when it is larger than the run-to-run noise. A separate post covers how many runs a mention rate needs before a trend is real.

Why do answers for the same prompt change?

Three causes are visible in the data, and none needs a model update:

  1. Retrieval changes. AI Overview, AI Mode and ChatGPT search the web before they answer. A different set of retrieved pages gives a different set of brands.
  2. Sampling. The same model with the same sources can write a different answer, because generation is random.
  3. Routing. Engines pick the model per request. On ChatGPT, the model that answers a logged-out session is chosen by ChatGPT.

Our data shows how much answers change, and it cannot separate these causes. Each prompt ran once a day, so two answers a few minutes apart, which isolate sampling from retrieval changes, are not in this data set.

How do you measure answer change on your own prompts?

Run the same prompt twice on the same engine and compare what the two answers name or cite. The example below uses the Gemini endpoint of cloro's API, which returns the answer text and its sources as JSON, and computes the Jaccard overlap of the cited hostnames:

import requests from urllib.parse import urlparse API = "https://api.cloro.dev/v1/monitor/gemini" HEADERS = {"Authorization": "Bearer YOUR_API_KEY"} PROMPT = "How quickly do AI search answers change for the same question?" def cited_hosts(): r = requests.post(API, headers=HEADERS, json={"prompt": PROMPT, "country": "US"}) r.raise_for_status() sources = r.json()["result"].get("sources", []) return {urlparse(s["url"]).hostname.removeprefix("www.") for s in sources} a, b = cited_hosts(), cited_hosts() union = a | b print(f"Jaccard overlap: {len(a & b) / len(union):.2f}" if union else "no sources") 

Two runs give one pair. For a number you can compare with the tables above, collect one answer per day for at least two weeks and take the median over all pairs. The open-source geo-tracker schedules the runs and stores every answer in Postgres. To compare brands instead of hostnames, match your list of brand names against the answer text, with link text removed so that a citation does not count as a mention.

How was this checked?

  • Data. Our own monitoring of a fixed set of prompts, one run per prompt, engine and day, October 17 to November 15, 2026, through logged-out sessions. Google AI Overview pairs count only when both days showed an AI Overview.
  • Brands. A deterministic, word-bounded match against a fixed list of about 90 vendor names in the API market, on the answer text with links removed. Brands outside that list are not counted.
  • Pairs. All pairs of days k apart for the same prompt and engine where both days had an answer. Brands kept needs at least one brand on day t. Jaccard overlap needs at least one brand on either day.
  • Limits. This measures named brands. Overlap of cited pages is a separate measure that this data set does not include. The prompts are in one market, API buyers. Other markets can change faster or slower. Pairs on one engine are not independent of each other, so the ranges describe the spread of pairs, not a confidence interval.

This post is published by cloro, which sells an API that returns AI engine answers as structured data, and the answers above were collected through it.