AI Answer Volatility

AI answer volatility is the degree to which an AI engine's response to the same prompt changes from one run to the next. Ask an identical question twice and the brands named, their order, the cited sources and even the length of the list can all differ. This is expected behavior: generation is probabilistic, live retrieval runs against a web that keeps changing, and models and their search layers are updated continuously.

The clearest public measurement comes from SparkToro and Gumshoe. In November and December 2025, about 600 volunteers ran 12 prompts through ChatGPT, Claude and Google's AI Overviews or AI Mode a combined 2,961 times. They found less than a 1 in 100 chance that ChatGPT or Google's AI would return the same list of brands in two responses, and less than 1 in 1,000 for the same list in the same order. The same study found that how often a brand appears across many runs is far more stable: in tight categories, the leading brands showed up in most responses even as their positions moved.

That is why prompt monitoring reports rates, not ranks. A single check is a sample of one. Sound practice is to freeze the prompt set so the denominator stays stable, run each prompt repeatedly across models and days, report appearance rates such as mention rate and citation rate with their sample sizes, and treat answer position as an average rather than a fixed slot. Gumshoe's follow-up analysis estimated that one prompt on one model needs about 100 runs to pin visibility down to within plus or minus 10 points, and about 400 runs for plus or minus 5. Spreading runs across many related prompts and personas gets a stable overall number more cheaply than repeating one prompt.

Volatility also helps separate noise from real change. Movement on a handful of prompts within the usual range is noise. A sharp drop across many unrelated prompts on one platform that persists for several cycles usually points to a change on the engine's side, such as a model release or a shift in how it picks sources; the ChatGPT citation drop data shows one such step change. Setup changes cause false alarms too. Adding new prompts or models where a brand is weak lowers an average score the day they start running, even though no engine changed its mind.

High volatility on a specific prompt often means the engine has no settled answer and no source it treats as definitive. That is usually an opening for a clear, well-sourced page rather than a reason to stop tracking the prompt. It also explains why Promptwatch's Visibility Score averages across all analyzed responses, counting answers that leave a brand out as zero, instead of reporting any single answer.