Lesson 2 of 10 · 9 min read
Deterministic AI, defined
In one paragraph
"Deterministic AI" describes AI-built systems whose production behaviour is deterministic: the same input always produces the same output, because the decision path runs as fixed, reviewed logic rather than a fresh model generation on every request.
In this lesson
- Define deterministic AI in one sentence
- Name four things deterministic AI is not
- Use the core vocabulary of the course: decision path, build time, run time, deterministic execution
The video version of this lesson is in production. The full lesson is below.
The last lesson ended with a problem: language models can give different answers to the same question, and a business owns every one of them. This lesson gives the problem’s opposite a name.
The definition, unpacked
“Deterministic AI” describes AI-built systems whose production behaviour is deterministic: the same input always produces the same output, because the decision path runs as fixed, reviewed logic rather than a fresh model generation on every request.
Three parts of that sentence do the work.
“AI-built systems.” AI is in the picture. In this approach, language models do a lot of the engineering: they read policies and documents, draft logic, and generate the tests that check it.
“The same input always produces the same output.” Ask the system the same question tomorrow, or a thousand times, and the answer doesn’t change. That is what makes it testable, supportable and auditable.
“The decision path runs as fixed, reviewed logic.” The decision path is the part of a system that decides what a customer is told or what happens to their case: approve or decline, the price, the refund, the next step. In a deterministic system, that path is code a person has reviewed and versioned, not something a model improvises on each request.
What deterministic AI is not
The term is easy to misread, so it’s worth being explicit.
- It’s not “no AI.” Models still do much of the work. They just do it before release, under review.
- It’s not temperature 0. Turning a model’s randomness down reduces variation, but production systems can still drift (batching, model updates, context changes). A setting is not a guarantee. The next section explains why.
- It’s not error-free. Deterministic logic can still be wrong. The difference is that a mistake is repeatable: it shows up the same way every time, so it can be found, fixed and covered by a test.
- It’s not a product you buy. It’s a property of how a system is built. Anyone who claims it should be able to show it.
Why temperature 0 is not deterministic
Temperature 0 does not make a language model deterministic. It makes the model choose its highest-scoring next token at every step instead of sampling, which removes the deliberate randomness, but the scores themselves can still differ slightly from one run to the next. When they do, the output can change.
Where the differences come from:
- Floating-point arithmetic. Computers add decimal numbers with limited precision, so
(a + b) + cis not always exactly equal toa + (b + c). A GPU adds millions of numbers per token, and the order it uses can change the result in the last digits. - Batching on shared servers. A hosted model processes your request alongside other people’s. Many GPU kernels pick their calculation order based on batch size, and batch size depends on traffic you can’t see or control. The same prompt can therefore produce slightly different scores at different times of day.
- Near-ties. Most of the time one token wins clearly and tiny differences don’t matter. When two tokens are almost equally likely, a difference in the last digit flips the choice, and because each token is generated from the ones before it, the rest of the answer can go a different way.
- Mixture-of-experts routing. Some large models send each token to a few specialised sub-networks (“experts”). In some serving setups, which expert handles a token can depend on what else is in the batch, which adds another source of variation.
- Provider-side changes. Hosted models are updated, re-deployed on different hardware or served with different software, sometimes behind the same model name.
- Context drift. A different retrieved document, a timestamp in the prompt or a slightly longer conversation history is a different input, even when the question looks the same.
A seed doesn’t close the gap either. Where an API offers a seed parameter, it fixes the random sampler, not the arithmetic, and providers that offer one describe it as best effort rather than a guarantee.
How to make LLM outputs reproducible
There is a ladder of options, from “mostly the same” to “the same every time”:
| Lever | What it does | What it doesn’t do |
|---|---|---|
| Temperature 0 (greedy decoding) | Removes deliberate sampling randomness | Stop batching, hardware or model changes shifting the output |
| Seed parameter, where offered | Fixes the sampler’s random choices | Fix the arithmetic; documented as best effort |
| Pinned, dated model version | Stops silent upgrades behind an alias | Prevent provider-side serving changes |
| Fixed prompt, context and settings | Removes input drift you control | Remove variation inside the model call |
| Strict output schema, validated in code | Makes the shape of every answer predictable and checkable | Make the content identical |
| Caching outputs | Replays the exact stored answer for the exact same input | Help with any input not seen before |
| Self-hosting with deterministic settings | Can make outputs bit-for-bit repeatable with fixed hardware, software and batching | Come free: it usually costs speed and operational effort |
| Compiling the decision into code | The model writes the logic at build time; reviewed code runs it, so the same input gives the same output | Suit open-ended language work, where a live model is the right tool |
The first six levers make a model’s outputs more consistent. Only the last one takes the model out of the decision itself, which is why, for decisions that must repeat, it is the dependable option. That approach is compiled AI, the subject of lesson 3. For the short version of how deterministic and generative AI compare, see What is deterministic AI?.
To check where you stand, run the same input ten to twenty times in fresh sessions and compare the parts of the answer that matter. If they differ even once, treat the workflow as not reproducible.
Where the AI goes instead
If the model isn’t making the decision at run time, where is it? Mostly at build time: the phase when software is designed, written and tested. Run time is when it serves real customers. The approach of using models at build time and deterministic code at run time is called compiled AI, and it’s the subject of lesson 3.
Models can also have bounded roles at run time. For example, a model can read a messy document and fill a strict form, which code then checks, with a person taking over when something doesn’t fit. How much of that is appropriate is a judgement this course gives you a framework for in lesson 4.
Deterministic, generative, agentic
You’ll hear three words used loosely. Here is how this course uses them:
| Term | Good at | Can go wrong | Belongs |
|---|---|---|---|
| Generative | Drafting, summarising, conversation | Different answers to the same question | Where a person reviews the output |
| Agentic | Multi-step tasks across tools | Errors compound, and actions are taken on them | Reversible, low-value tasks with hard limits |
| Deterministic | Decisions that must repeat and be explained | Misses cases nobody anticipated | Anywhere customers, auditors or regulators rely on the answer |
Most real systems mix all three. The skill is putting each where it belongs.
The vocabulary for the rest of the course
These terms come up in every lesson. Each one is defined in the glossary.
- Run-to-run variance: different outputs for the same input (lesson 1).
- Decision path: the part of a system that decides outcomes.
- Deterministic execution: running that path as fixed, reviewed, versioned code.
- Build time / run time: when software is made versus when it serves users.
- Golden dataset: real examples with answers your team agrees are correct, used to measure accuracy (lesson 5).
Try it yourself: the one-question test
Take any AI-assisted workflow in your organisation and ask:
“If we ran this decision again tomorrow with exactly the same input, would we get the same result, and could we show why?”
If the answer to either half is “no” or “not sure”, the decision path isn’t deterministic yet. That isn’t automatically a problem. It is a choice your team should be making deliberately rather than by default.
What comes next
Lesson 3 shows how teams get deterministic behaviour without giving up AI: using language models to write the logic, then running the logic, not the model.
Key takeaways
- Deterministic AI is about the decision path: the part of a system that decides what a customer is told or what happens to their case.
- It doesn't mean no AI. Models still do much of the engineering; they just don't improvise the decision every time.
- Temperature 0 is a setting, not a guarantee. Determinism comes from how the system is built.
- Seeds, pinned model versions, schemas and caching make LLM outputs more reproducible; compiling the decision into reviewed code is what makes it repeatable.
- Deterministic doesn't mean error-free. It means mistakes are repeatable, so they can be found, fixed and tested before release.
Check yourself
Pick an answer, then open the card to compare.
-
1. Which statement best describes deterministic AI?
- A.An AI system that uses no language models
- B.An AI-built system whose decision path gives the same output for the same input
- C.A language model running at temperature 0
Show the answer
B. An AI-built system whose decision path gives the same output for the same input The definition is about production behaviour on the decision path. Models can still be used to build the system.
-
2. A team says their chatbot is deterministic because they set temperature to 0. What's the problem?
- A.Nothing; that makes it deterministic
- B.Temperature 0 reduces variation but doesn't guarantee identical outputs in production
- C.Temperature 0 makes answers less accurate
Show the answer
B. Temperature 0 reduces variation but doesn't guarantee identical outputs in production Batching, model updates and context changes can still shift outputs. Determinism comes from fixed logic in the decision path.
-
3. What is a decision path?
- A.The steps a user clicks through in an app
- B.The part of a system that decides what a customer is told or what happens to their case
- C.The training data of a model
Show the answer
B. The part of a system that decides what a customer is told or what happens to their case Deterministic AI focuses on the part of the system where consistency matters most: the decision.
Common questions
Why isn't temperature 0 deterministic?
Temperature 0 makes a model pick its highest-scoring next token, but the scores themselves can differ slightly between runs. Floating-point arithmetic is not associative, so the order in which a GPU adds numbers changes the result in the last digits, and on hosted APIs that order can depend on how many other requests are batched with yours. When two candidate tokens are nearly tied, a tiny difference flips the choice, and every later token follows from it. Provider-side model updates and small changes in context add more variation.
How do I make LLM outputs reproducible?
Pin a dated model version, keep the prompt, context and settings fixed, use a seed where the API offers one (providers describe it as best effort), constrain the output to a schema that code validates, and cache any output you must be able to replay exactly. If you host the model yourself, fixing hardware, software versions and batching, with deterministic kernel settings, can make outputs repeatable at some cost in speed. For decisions that must never vary, have the model write the logic at build time and run the reviewed code instead.
Is deterministic AI a product I can buy?
No. It's a property of how a system is built: whether its decision path runs as fixed, reviewed logic. Any vendor that claims it has to show it, for example by running the same inputs many times and getting identical outputs.
If the logic is fixed, how does the system improve?
Through new versions. When rules or data change, the logic is updated, re-tested and released, like any software change. You always know which version made which decision.
Where do language models still fit?
At build time, writing code, extracting rules and generating tests, and in bounded roles at run time, such as reading an unstructured document into a strict format that code then checks. Lesson 4 covers how much model belongs where.
How is this different from traditional rules engines?
The output can look similar: explicit, testable logic. The difference is how it is built. Language models do much of the drafting and testing, which makes the logic faster to create and easier to update.