Why you might not get the same answer twice

If two people ask Claira (or any AI tool built on a large language model) the same question about the same document, they may get answers that are worded differently — and sometimes structured differently too. This is normal, expected behavior for this category of technology, not a malfunction or a sign that one answer is "wrong."

The short version

The key things to know:

• Language models generate text probabilistically, so some run-to-run variation is built into how they work.
• Vague, open-ended requests (like "summarize this") leave a lot of room for variation. Specific, structured instructions (like Claira's relevance prompts) leave very little.
• Variation shows up in wording and emphasis, not in facts or decisions. A well-constructed relevance prompt should produce the same call on the same document, even if two summaries of that document read differently.

The rest of this article explains why, in plain terms.

LLMs are very smart autocomplete

A large language model (LLM) is not a database that looks up "the answer" to your question. At its core, it is a system trained to predict what text should come next, one small chunk (a "token," roughly a word or part of a word) at a time — Anthropic's own glossary describes its models as "pretrained to predict the next word, given the previous context of text in the document" (Anthropic, Glossary).

A useful mental model: it works like the autocomplete on your phone, except instead of predicting the next word from the last two or three words you typed, it predicts it from everything in the conversation so far — your instructions, the document, and everything it has already written in its reply — drawing on patterns learned from an enormous amount of text. That is why the same underlying mechanism can draft an email, answer a legal question, or summarize a deposition transcript.

The important consequence: at each step, the model isn't choosing between "the right word" and "wrong words." It's choosing among many plausible next words, each with a probability attached. "The agreement was executed on..." could plausibly continue with "March," "the," or "or about" — all reasonable, all leading the sentence somewhere slightly different.

Where the variation comes from

1. Sampling: the model rolls weighted dice

When generating text, models typically don't just take the single most probable next word every time — doing so tends to produce repetitive, stilted text. Instead, they sample from the most plausible options, weighted by probability. A setting called temperature controls how adventurous this sampling is: "Higher temperatures lead to more creative and diverse outputs... Lower temperatures result in more conservative and deterministic outputs" (Anthropic, Glossary). Google describes the same mechanism for its Gemini models, alongside related settings (top-k, top-p) that shape which candidate words are even in the running (Google, Gemini API prompting guide).

Because of sampling, two runs of the same request can diverge at any word. And once they diverge — one summary opens with the parties, the other with the dispute — everything after that point is being predicted from a different context, so the two answers continue to drift apart.

2. Even "deterministic" settings aren't fully deterministic

You might expect that turning the randomness setting to zero would guarantee identical outputs. In practice it doesn't. OpenAI's own documentation describes its reproducibility features as making outputs "mostly consistent," not guaranteed (OpenAI Cookbook, reproducible outputs). Research testing five major LLMs across eight tasks with all randomness settings minimized still found measurable run-to-run differences, concluding that none of the models "consistently delivers repeatable... identical output strings" across all tasks (Atil et al., 2024, arXiv).

Why? The engineering reasons are deep in the plumbing: research from Thinking Machines Lab showed that a major culprit is that the numerical calculations inside an AI server can come out microscopically different depending on how many other users' requests are being processed on the same hardware at the same moment — and a microscopic difference in probabilities can occasionally flip which word gets picked (Thinking Machines Lab, "Defeating Nondeterminism in LLM Inference," 2025).

You don't need to remember the mechanics. The takeaway is: identical input does not guarantee identical output, by design and by the physics of the infrastructure.

3. Tiny differences in the input compound

Two people rarely ask a question in exactly the same way. "Summarize this document," "Can you summarize this?", and "Give me a summary" are the same request to a human — but to a next-word predictor, each one sets up a slightly different context, which shifts the probabilities of everything that follows. Academic research has found that models can be surprisingly sensitive even to trivial formatting choices in a prompt — changes as small as punctuation or spacing produced performance swings of up to 76 accuracy points in some evaluated tasks (Sclar et al., ICLR 2024, arXiv).

4. Models themselves change over time

AI providers update their models. A Stanford and UC Berkeley study documented that the same model service, asked the same questions months apart, could behave substantially differently (Chen, Zaharia & Zou, 2023, arXiv). So "I asked the same thing last month and got a different answer" can also reflect a model update, not just sampling.

Vague prompts vary; structured prompts converge

Here is the practical heart of the matter.

"Summarize this document" is a wide-open request. It doesn't say how long the summary should be, what to prioritize, whether to focus on parties, dates, legal issues, or narrative, or what format to use. Every one of those unstated decisions is left to the probabilistic word-by-word process described above. Two runs will make those micro-decisions differently, so two summaries of the same document can legitimately differ in length, emphasis, structure, and wording — while both being accurate.

A structured prompt closes down the space of acceptable answers. When an instruction specifies exactly what question to answer, what criteria to apply, what to ignore, and what format the answer must take, there is far less room for the model's word-by-word choices to lead anywhere different. This is precisely how relevance prompts in Claira are designed: they pose a specific, well-defined question about each document and constrain the form of the answer. The narrower the question, the more the probability distribution collapses onto a single answer — and the more consistent the output becomes run to run.

What this means for review work

Summaries may vary. Decisions shouldn't. For binary or categorical determinations — is this document relevant to the defined issue, yes or no — a well-constructed prompt should yield the same decision on the same document across runs and across users, even when the explanation accompanying that decision is worded differently each time. The wording of the reasoning is generated text (variable); the decision is the answer to a narrow, structured question (stable).

If a relevance decision does flip between runs on the same document, treat that as signal, not noise. It usually means the document is a genuine edge case relative to the prompt's criteria, or the prompt has an ambiguity worth tightening. Flipping decisions are a prompt-drafting issue to investigate — unlike differently-worded summaries, which are just the technology behaving normally.

Two differently worded answers can both be right. When comparing outputs, compare substance, not phrasing. Ask: do both answers identify the same key facts? Do they reach the same conclusion? Divergent wording with consistent substance is expected; divergent substance is what warrants a second look.

Frequently asked questions

Someone else got a different summary of the same document. Whose is correct?

Very likely both. Check whether the two summaries disagree on any fact or conclusion. If they only differ in emphasis, structure, or wording, that's ordinary sampling variation. If they disagree on substance, review the document — and consider whether the question asked was specific enough.

Can't the randomness just be turned off?

Not entirely. Even at the most conservative settings, run-to-run variation can occur for infrastructure reasons (Thinking Machines Lab, 2025; Atil et al., 2024). More importantly, chasing word-for-word identical output is usually the wrong goal — consistent substance and decisions, achieved through well-structured prompts, is the goal that matters for review work.

How do I get more consistent outputs?

Make the request more specific. State what to focus on, what to exclude, how long the answer should be, and what format to use. In Claira, prefer running documents through a defined, structured prompt over ad hoc open-ended questions whenever consistency matters. See Understanding Results and the Prompting Overview.

Does variation mean the AI is unreliable?

No — it means the output is generated rather than retrieved. Reliability for review purposes comes from constraining the question so that the space of acceptable answers is narrow, then validating that decisions are stable. That is what structured prompting is for.

Related

• Understanding Results — how to read, evaluate, and act on Claira's output.
• Prompting Overview and Custom Prompts — write prompts that constrain the answer.
• Relevance with confidence — a structured relevance template that returns a stable decision plus a worded rationale.

References

• Anthropic, Glossary (temperature, pretraining, and token prediction): platform.claude.com/docs/en/about-claude/glossary
• Google, Gemini API — Prompt design strategies (temperature, top-k, top-p): ai.google.dev/gemini-api/docs/prompting-strategies
• OpenAI, Cookbook: How to make your completions outputs consistent with the seed parameter: developers.openai.com/cookbook/examples/reproducible_outputs_with_the_seed_parameter
• Thinking Machines Lab, Defeating Nondeterminism in LLM Inference (2025): thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference
• Atil, B., Baldwin, B., et al., Non-Determinism of "Deterministic" LLM Settings (2024), arXiv:2408.04667: arxiv.org/abs/2408.04667
• Sclar, M., Choi, Y., Tsvetkov, Y., Suhr, A., Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design (ICLR 2024), arXiv:2310.11324: arxiv.org/abs/2310.11324
• Chen, L., Zaharia, M., Zou, J., How Is ChatGPT's Behavior Changing over Time? (Stanford / UC Berkeley, 2023), arXiv:2307.09009: arxiv.org/abs/2307.09009

Need help? Contact us at support@claira.to.