Cost math: compiled vs runtime LLM

Reliability is the main reason to compile the decision path. Cost is often the reason it gets approved. This lesson gives you a simple way to compare the two approaches for any workflow, using your own numbers.

Two cost shapes

Runtime model. Every transaction calls a model, so you pay tokens and compute on every case. The total grows in a straight line with volume: twice the cases, twice the bill.

Compiled logic. You pay up front, for the engineering to build, review and test the logic (with models doing much of the drafting). After that, each case costs ordinary compute, which is close to nothing per transaction. The total is almost flat as volume grows.

The two lines cross at the break-even volume.

The break-even formula

Break-even volume = up-front build cost ÷ (runtime cost per case − compiled cost per case)

An illustrative example, with numbers chosen for easy arithmetic rather than taken from any real project: if building and testing the compiled version costs 10,000, a model call costs 0.02 per case, and compiled execution costs effectively nothing, break-even is 10,000 ÷ 0.02 = 500,000 cases. Past that volume, every case is cheaper compiled; below it, the runtime model is cheaper on cost alone.

Use your own figures: real build estimates, your provider’s actual per-call cost at your prompt sizes, and your real monthly volume.

Why token-only figures look so different

An April 2026 research preprint on compiled AI reported break-even after about 17 transactions and 57 times fewer tokens at 1,000 transactions. That comparison counts tokens: the one-off tokens used to generate the code against the tokens a model uses on every call. It is a fair technical measure, and it shows how quickly runtime token costs add up.

For a business decision, also count the engineering time to review, test and maintain the compiled logic. That’s why a full-cost break-even sits much higher than a token-only one. Both numbers are honest; they answer different questions.

Four factors, weighed together

Factor	Favours runtime model	Favours compiled
Volume	Low, occasional use	High, steady volume
Change rate	Rules change daily	Rules change monthly or quarterly
Latency	Seconds are acceptable	Real-time or high throughput
Stakes	A person reviews every output	Customers, money or regulators rely on the answer

Stakes often decide it before cost does: if a decision has to be repeatable and explainable, a runtime model in the decision path is the wrong design at any price.

The costs people forget

• Evaluation and monitoring. Both approaches need a golden dataset and regression runs (lesson 5). Runtime models also need watching for drift when providers update them.
• Re-compiling. Each rule change means a new build-and-test cycle. Frequent change raises the compiled line.
• Incidents. A wrong answer that reaches customers has a cost (refunds, complaints, regulators) that rarely appears in a per-call price.

Try it yourself: a five-minute estimate

For one workflow, write down:

1. Monthly volume of decisions.
2. Your current cost per decision with a runtime model (from your provider’s bill ÷ volume).
3. A rough estimate to build and test a compiled version.
4. How often the rules change per year.

Divide (3) by (2) to get break-even volume, and compare it with (1) × 12. Then check the stakes column. If the decision must be repeatable, the cost question is about when, not whether.

What comes next

The next three lessons turn from engineering to accountability: who is liable for what an AI says (lesson 7), what regulators have penalised (lesson 8), and what the EU AI Act requires (lesson 9).