Why the Same Prompt Gives You a Different Answer

Paste the same prompt into the same model twice and you will usually get two different texts. Most people treat this as a flaw — a sign the tool is unreliable, or that they phrased something wrong. It is neither. Variation is designed into the system at the most basic level, it survives even when you turn every knob to “deterministic,” and it carries a consequence almost nobody states out loud: if the same prompt does not produce the same work, then the prompt did not determine the work. That matters far beyond the annoyance of an inconsistent draft. It is the quiet technical fact sitting underneath every argument about who — or what — authored a piece of AI-generated text.

This article does two things. First, it explains, with the measurements, why the output moves when the input does not. Then it follows that fact where it actually leads, which is not to “write better prompts” but to a problem for anyone claiming the prompt makes them the author. The mechanism is the easy part. The consequence is the part that gets skipped.

The machine is built to roll dice

A language model does not “write” a sentence. At each step it produces a probability distribution over every possible next token — thousands of candidates, each with a score — and then samples one from that distribution. The sampling is the whole point. If you always took the single highest-probability token, you would get what the field calls degenerate text: Ari Holtzman and colleagues showed that maximization-based decoding like beam search produces output that is bland, repetitive, and prone to looping, and their proposed fix was to sample from the “nucleus” — the smallest set of top tokens whose probabilities sum to a threshold p — rather than always grabbing the mode (Holtzman et al., 2020). Nucleus sampling, also called top-p, became the default for open-ended generation precisely because it reintroduces controlled randomness. The fluency you like is bought with dice.

Two parameters govern how loaded those dice are. Temperature flattens or sharpens the distribution: higher temperature spreads probability toward unlikely tokens, lower temperature concentrates it on the frontrunners. Top-p and top-k clip the tail so the model never samples from the long list of near-zero garbage. Sergio Bermejo, building a system that deliberately generated multiple variants of each text, put the relationship plainly: variations in hyperparameters “such as temperature, top_k, and top_p directly affect the diversity and quality of generated outputs” (Bermejo, 2024, p. 5). His architecture ran several models “with different hyperparameter settings (temperature, top_k, and top_p)” specifically “to maximize diversity in outputs” and “obtain diverse responses with different characteristics” (Bermejo, 2024, p. 4). Read that backwards and you have your answer: diversity is not a side effect the engineers are fighting. It is a dial they turn up on purpose, because a tool that returns an identical paragraph every time is a worse writing tool, not a better one.

So the first reason the same prompt gives a different answer is the simplest. The default generation mode is stochastic by design. You are not observing a malfunction; you are observing the feature working.

Turn the temperature to zero and it still drifts

Here is where it gets strange. Set temperature to 0, which should make the model always pick the top token — pure greedy decoding, no dice at all — and the output still moves between runs. This is the finding that breaks most people’s mental model.

Berk Atil and colleagues tested exactly this. They fixed the temperature at 0, set top-p to 1, pinned the random seed, and ran five models — GPT-3.5 Turbo, GPT-4o, Llama-3-70B, Llama-3-8B, and Mixtral-8x7B — across eight tasks drawn from the BBH and MMLU benchmarks, ten times each (Atil et al., 2025). Under those nominally deterministic settings, accuracy still swung by up to 15% across repeated runs, and the gap between a model’s best-possible and worst-possible performance reached 70% (Atil et al., 2025). Their blunt conclusion: “none of the LLMs consistently delivers repeatable accuracy across all tasks, much less identical output strings,” and an LLM “rarely produces the same response ten times given the same input” (Atil et al., 2025). The same prompt, the same temperature of zero, the same seed — and the measured score wandered.

Why? The authors noticed a telling detail: when they ran the small Llama-3-8B model locally on their own GPUs without serving optimizations, it was deterministic (Atil et al., 2025). The non-determinism was coming from how the models were served at scale, not from the math of greedy decoding itself.

Horace He and the team at Thinking Machines Lab pinned down the exact culprit in 2025, and it is worth getting right because the usual explanation is wrong. The common story blames floating-point rounding plus concurrency — the idea that GPU threads finish in an unpredictable order and tiny rounding errors accumulate differently each time. He’s team showed that story “doesn’t reveal the full picture,” and that concurrency and atomic adds “end up being completely uninvolved in LLM inference nondeterminism” (He, 2025). Running the same matrix multiplication on the same data repeatedly, they note, “will always provide bitwise equal results” (He, 2025). The real cause is batch invariance — or the lack of it. The server batches your request together with everyone else’s, and the exact numerical result for your tokens depends on how many other requests happened to be in the batch, because the kernel reduces numbers in a different order at different batch sizes. “The load (and thus batch-size) nondeterministically varies,” they write, so the same prompt computed under a light load and a heavy load takes slightly different numerical paths (He, 2025).

The demonstration is vivid. Running Qwen3-235B on the prompt “Tell me about Richard Feynman” at temperature 0, one thousand times, produced 80 distinct completions out of 1,000 (He, 2025). All 1,000 were identical for the first 102 tokens; they first diverged at token 103, where — after the shared text “Feynman was born on May 11, 1918, in” — 992 continued with “Queens, New York” and 8 continued with “New York City” (He, 2025). One accident of batching, 103 tokens deep, and the texts fork. When the team enforced batch-invariant kernels, all 1,000 completions became identical (He, 2025) — proof that determinism is achievable, and equally that the systems you actually use do not bother, because the fix costs real performance and the variation is, again, not considered a bug worth paying to kill.

Temperature changes the variety, not necessarily the quality

A tempting misreading of all this is “so crank the temperature to zero for important work and at least it’ll be as consistent as possible.” The evidence complicates even that. Matthew Renze tested nine LLMs across five prompting techniques while raising the sampling temperature, and found that “changes in temperature from 0.0 to 1.0 do not have a statistically significant impact on LLM performance for problem-solving tasks,” a result that held “across LLMs, prompt-engineering techniques, and problem domains” (Renze, 2024, p. 7346). Temperature changes how much the outputs vary from each other. It does not reliably change how good they are. The low-temperature run is not the “correct” answer and the high-temperature run the corrupted one — they are both valid samples from the same distribution, and on measured accuracy they come out about even.

Put the three findings together and the picture is complete. At normal settings the model samples on purpose (Holtzman et al., 2020; Bermejo, 2024, p. 5). At zero temperature the serving infrastructure reintroduces variation through batching (Atil et al., 2025; He, 2025). And turning the randomness down does not buy you a more authoritative answer anyway (Renze, 2024, p. 7346). Variation is not a layer you can peel off. It is the material the text is made of.

This is also why a prompt you optimized is such a slippery asset — a theme that runs across the prompting pillar. A phrase that landed a great result once was partly riding a sample you cannot reproduce, which is a cousin of the problem that prompts do not transfer between tasks and models: in both cases you mistook a lucky local outcome for a stable property of your instruction. It is also why the brittleness that undoes imported reasoning tricks — the way a technique can work on math and fall apart on prose — rhymes with run-to-run drift. The input is doing less of the determining than it feels like.

The consequence nobody writes about

Now the part that actually matters. Almost every piece on this topic stops at “here’s how to reduce the variation.” But sit with what the variation is, and it reaches straight into the question of authorship.

Copyright protects a specific expressive work — this arrangement of these words — not an idea, a topic, or an instruction. When a person writes, the words on the page are the direct, traceable result of their choices; change the writer’s intent and the sentences change with it. That tight link between the author’s choice and the resulting expression is the thing authorship doctrine is built to reward.

Non-determinism severs that link. If I give a model a prompt and it can return 80 different thousand-token essays — all from the same instruction, the same temperature, the same seed (He, 2025) — then my prompt did not specify any one of those essays. It specified a distribution, and a stochastic process standing between me and the page selected which sample became the work. The particular sentence you are reading, versus the 79 others the system could just as well have emitted, was not chosen by the prompter. “Queens, New York” rather than “New York City” was decided by batch load on a server (He, 2025), not by anyone’s expressive intent. The clause the prompt author most wants to claim as theirs — the exact words — is precisely the clause the prompt did not determine.

This is the bridge from a prompting problem to a legal one, and it is why the mechanism is worth understanding before anyone argues about ownership. The strongest claim a heavy prompter can make is “I caused this text.” The measurements say the honest version is narrower: I constrained a distribution from which a machine sampled this text, and could as easily have sampled a different one. That is a real contribution, but it is not the same kind of contribution as choosing the words — and the gap between those two is where disclosure obligations and authorship disputes actually live. If you are weighing how much of an AI draft is yours to claim, start there; the questions compound quickly, which is why they get their own pillar on AI, copyright, and disclosure.

None of this makes AI writing worthless or unusable. It makes one specific claim false — the claim that a good prompt is the work, that the author of the instruction is therefore the author of the output. The same prompt gives you a different answer because the prompt was never the answer. It was the question, asked of a machine that answers a slightly different way every time, and knowing that is the difference between using the tool honestly and overstating what you did with it.