Ask ChatGPT to recommend the best project management software, wait ten minutes, ask again. The list shifts. A brand that appeared third is gone. A new one leads. Nothing about the market changed in those ten minutes. What changed is the model rolling the dice again, because that is literally what it is built to do.
Key Takeaways
- AI models generate answers by sampling from a probability distribution over possible next words, not by retrieving a fixed record, so identical prompts can produce different outputs.
- Variation is heaviest in phrasing and ordering, and much lighter around which brands get named at all, because strong citation signals narrow the model’s options before sampling even starts.
- Brands should optimize for showing up inside the narrowed set of likely answers, not for controlling the exact wording of any single response.
The Model Is Not Looking Anything Up
A search engine has an index. Type a query and it retrieves matching documents, ranks them, returns the same list to everyone who asks the same thing at the same time. That is deterministic by design.
A large language model does not work that way. It predicts one token at a time, each token chosen from a probability distribution over everything that could plausibly come next given the text so far. Ask it to recommend a CRM and it is not pulling a stored answer labeled “best CRM.” It is calculating, at each word, which continuation is statistically likely given its training and whatever it retrieved for that query, then sampling from that distribution rather than always taking the single highest-probability option.
That sampling step is the whole story. If the model always picked the top-probability token, outputs would be far more repetitive and also noticeably worse, since the single most likely next word is not always the best one for a full, coherent answer. Introducing controlled randomness produces language that reads naturally. It also means two runs of the same prompt can diverge from the very first sentence.
Temperature, Top-p, and the Dial Nobody Outside the Model Sees
Every major model exposes, internally or through its API, a setting usually called temperature, sometimes paired with a related setting called top-p. Temperature controls how sharply the model favors high-probability tokens versus giving lower-probability ones a real chance. A temperature near zero makes output close to deterministic, the model almost always picks the most likely word. Push it higher and the distribution flattens, so less probable words get selected more often.
Consumer chat products run somewhere in the middle by default, not at zero. That is a deliberate choice. A temperature of zero produces flat, repetitive, sometimes oddly stilted prose. A bit of randomness produces answers that feel conversational and varied, which is what most users want from a chat product. The tradeoff is that the same input prompt no longer maps to the same output.
Top-p works alongside temperature by capping the pool of candidate tokens to the smallest set whose combined probability crosses a threshold, then sampling within that narrowed set. Change either setting and the same underlying model behaves differently. Neither setting is visible to the person typing a question into ChatGPT or Gemini, which is why the variation looks mysterious from the outside even though it is fully mechanical from the inside.
The randomness is not a bug in the system. It is the setting the system runs on by default.
What Actually Stays Stable Across Runs
Here is the part that matters for brands trying to show up in these answers: not everything is up for grabs each time.
The exact sentence structure, the specific adjectives, the order items appear in a list, these shift run to run. What tends to hold steadier is which entities get pulled into the answer in the first place. A brand with strong presence across the sources a model draws on, structured data that clearly identifies what it is and does, and consistent third-party coverage of the same facts, sits inside a narrower, higher-probability slice of the distribution before sampling even starts. Random variation still happens inside that slice. It happens less often outside it.
This is the difference between Share of Voice and Share of Citation. A brand mentioned constantly across the web can still have a near-zero chance of being the entity a model actually names and sources, because mentions alone do not shape the narrowed set of candidates a model is sampling from. Citations, structured data, and independent agreement across sources do that shaping. That is also why citations compound. Once a model has cited a brand as a source for one query, the surrounding context and pattern make that same brand more likely to surface for adjacent queries too.
Why the Same Prompt on Different Engines Diverges Even More
Run identical prompts against ChatGPT, Gemini, Claude, Perplexity, and Grok and the variation compounds, because each model is drawing from a different pool before sampling ever begins. ChatGPT pulls from across the open web. Gemini cross-references Search, YouTube and Scholar. Claude leans on high-authority publications and documentation. Perplexity cites its sources inline, which makes its variation more visible since you can see exactly which sources shifted. Grok reads the real-time social web, so its answer to the same prompt an hour later can reflect a conversation that had not happened yet.
That is stacked randomness. Each engine has its own training-data presence, built slowly through broad coverage over time, and its own retrieval presence, built through content that is structured and current enough to surface right now. A brand can be well represented in one layer on one engine and nearly invisible in the other layer on a different engine. Tracking a single model and assuming the pattern generalizes is how teams miss half the picture.
growth in AI brand presence for NewDay USA, tracked weekly across five engines over six months, precisely because single-engine snapshots miss how much answers vary model to model.See the case study
What To Actually Do About It
Stop measuring AEO by screenshotting one answer and treating it as proof of anything. A single run tells you almost nothing, because a single run is one sample from a distribution that resets each time. Run the same prompt set repeatedly, across the engines your buyers actually use, on a set schedule, and track how often your brand appears rather than what exact words surrounded it.
Then work on the two layers that narrow the distribution in your favor. Shore up retrieval presence with structured data, current content, and clear entity definitions the model can parse without ambiguity. Build training-data presence with independent, consistent coverage across sources that agree with each other, since that agreement is what forms real trust rather than a domain authority score. Neither move guarantees any single answer. Both moves shift the odds so that, run after run, your brand keeps landing inside the set the model is sampling from.
FAQs
Why Does ChatGPT Give a Different Answer to the Same Question?
Because it generates each response by sampling from a probability distribution over likely next words rather than retrieving a fixed, stored answer. A setting called temperature controls how much randomness is allowed in that sampling, and consumer chat products run with temperature above zero by default, which produces natural-sounding but non-identical output on repeated runs.
Can You Turn Off the Randomness?
Through an API, yes, by setting temperature close to zero, which makes output close to deterministic. Consumer-facing products like the ChatGPT or Gemini web apps do not expose that setting to end users, so the variation is baked into the default experience.
Does This Mean AEO Tracking Is Pointless Since Answers Keep Changing?
No. It means tracking a single answer is pointless. Tracking how often a brand appears across many runs of the same prompt set, over time and across engines, reveals a real pattern even though any individual answer is somewhat random.
Why Do Different AI Models Give Even More Different Answers Than the Same Model Twice?
Each model draws on a different pool of sources before it ever starts sampling. Gemini cross-references Search, YouTube and Scholar, Claude leans on high-authority publications and documentation, Grok reads the real-time social web. Different source pools plus independent randomness in each model compounds the variation between engines.
