The 20-Second Version
A company ships an internal AI assistant. The pilot is cheap, people love it, adoption climbs — and then the monthly invoice lands with a thud. The instinct is to blame "the model." Usually that's the wrong suspect. AI cost is not simply the price of a model. It is the price of everything the application repeatedly asks the model to read, remember, retrieve, reason over, generate, and sometimes redo. Get the architecture right and the same helpful assistant costs a fraction as much, answers faster, and scales. This piece explains why — first in plain English, then under the hood — and gives you a checklist to take to your next budget meeting.
It always begins the same charming way. Someone in the company wires up an internal assistant — call it a helpful colleague who never sleeps. The pilot runs on a handful of enthusiasts and a rounding-error budget. Leadership is delighted. A memo goes out. Adoption takes off.
Then behaviour changes, quietly. People stop asking one-line questions and start pasting entire contracts. Conversations that used to be three messages become forty. Someone builds a "just draft the whole quarterly report" workflow. Another team wires the assistant into a nightly job. Everyone is thrilled. Nobody is watching the meter.
And then the invoice arrives, and it does not whisper. In the next finance review the CFO looks up from the spreadsheet and asks the question that launched a thousand architecture reviews:
It's a fair question, delivered with a raised eyebrow. The chatbot did not ask for a salary, health insurance, or a corner office. It never once mentioned equity. But somewhere between the pilot and the platform, it developed an impressive, insatiable appetite — for tokens.
The Consultant Who Rereads the Whole Handbook
Here's the analogy to keep in your head for the rest of this article, because it explains almost everything.
Imagine every question you ask the AI is you sending a consultant into a meeting to give you one answer. This consultant is sharp and fast. But your company has a peculiar rule: before the consultant is allowed to say a single word, they must silently reread a stack of material you hand them at the door. Today's stack is the full employee handbook, the complete transcript of every meeting you've had this week, a pile of documents someone thought might be relevant, the printout from a database query, and three pages of instructions on how to behave.
The consultant reads all of it, every time, then gives you a crisp two-paragraph answer. And here's the part that stings: you are billed for the reading, not just the answer. You asked one short question. You paid for a small library.
That is, more or less, how a modern AI application works. The user types a sentence. The application hands the model a stack. The bill reflects the stack.
1. The AI Bill Is More Than the User's Question
When someone types "summarise this and suggest next steps," it feels like a small request. Behind the curtain, the application may be assembling and sending a great deal more before the model reads a single word the user actually wrote. A typical request can carry:
- System instructions — the standing rules about tone, format, and what not to do.
- The user's prompt — the one sentence they actually typed.
- Conversation history — everything said earlier in the chat, often resent in full on every turn.
- Retrieved content — passages pulled from a knowledge base to ground the answer (this is "RAG," more on it soon).
- Uploaded documents — the contract, the deck, the spreadsheet someone dropped in.
- Tool and API results — the output of a search, a database lookup, a calculation.
- Images, audio, or other inputs — anything beyond plain text.
- The model's own output — the answer it writes back.
- Retries, failed calls, and agent loops — the work you never see because it happened between your question and your answer.
- Reasoning or internal-compute charges, where a model or provider bills for "thinking" steps.
The user typed one sentence. The application sent thousands of tokens. The gap between those two numbers is where budgets quietly go to die.
Most chat models are stateless: the model does not "remember" the previous turn. To create the illusion of memory, the application reconstructs the whole conversation and resends it on every turn. Turn 20 doesn't send one new message — it re-sends turns 1 through 19 plus the system prompt plus any retrieved context, then adds the new question. Cost per turn therefore tends to grow as the conversation grows, unless you actively manage it with truncation, summarisation, or a sliding window. The user experiences one smooth thread; the billing system experiences twenty increasingly heavy requests.
2. Tokens: The Meter Behind the Machine
Models don't read words the way we do. They read tokens — small chunks of text. A token might be a whole short word, part of a longer word, a space, or a piece of punctuation. Roughly speaking, common English words often land near one token each, but that is a loose rule of thumb, not a law. Longer or unusual words split into several tokens. Code, symbols, and many non-English languages tokenise differently, sometimes far less efficiently. The same sentence can cost a different number of tokens on a different model. So treat every token figure in this article as an illustrative estimate, never a universal fact.
What matters for the bill is that almost everything is metered in tokens — what goes in and what comes out. Here is a fresh, deliberately round example. Imagine an internal assistant at a mid-sized firm. Numbers below are illustrative:
| Quantity (illustrative) | Value | How it adds up |
|---|---|---|
| Input tokens per request | 7,500 | System + history + retrieved context + question |
| Output tokens per request | 600 | The answer the model writes back |
| Requests per employee / day | 20 | A moderately active user |
| Active employees | 500 | A single business unit |
| Requests per day | 10,000 | 20 × 500 |
| Input tokens per day | 75,000,000 | 10,000 × 7,500 |
| Output tokens per day | 6,000,000 | 10,000 × 600 |
| Total tokens / month (22 working days) | ≈ 1.78 billion | 1.65B input + 132M output |
Nearly two billion tokens a month — for a single 500-person unit doing nothing exotic. Notice the shape of it: input dwarfs output by more than ten to one. The model's clever answer is the small part. The material it reads to produce that answer is the large part. That is the whole game, in one table.
Input and output tokens are usually priced separately, and output is often the more expensive of the two per token — which makes it tempting to squeeze output. But look at the ratio above: because input volume is so much larger, input is frequently where the real money is. Providers also differ on how they bill cached input, "reasoning" tokens, and multimodal inputs (an image or audio clip is converted into a token-equivalent that varies by model). Rates change over time and vary by provider and model tier, so anchor your own numbers to the official pricing on the day you calculate — never to a figure you half-remember from a blog post.
3. The Context Window Is a Desk, Not a Target
You'll hear a lot about the context window — the maximum amount of text a model can consider at once. Think of it as the size of the model's working desk. A bigger desk can hold more material for a single question. That sounds like pure upside, and vendors market it that way.
But capacity is not a consumption target. A larger desk does not make the worker smarter; it just means you can pile more on it. Dumping 150 files onto the desk for every question doesn't produce a better answer. It produces a higher bill, a slower response, and a real risk that the one crucial paragraph gets lost in the clutter — the model distracted by ninety-eight things that didn't matter.
Four different quantities get confused here, and keeping them apart is worth money:
- Maximum context capacity — how much the desk can hold. A spec sheet number.
- Actual tokens processed — how much you actually put on the desk this request. A billing number.
- Useful context — the part that genuinely helps answer the question.
- Irrelevant context — the part you paid to process that changed nothing, or made things worse.
Two effects compound as you fill the window. First, cost and latency scale with the tokens actually processed, not with the window's ceiling. Second, models can exhibit a "lost-in-the-middle" tendency — evidence buried in the centre of a very long context is sometimes weighted less reliably than material at the start or end. So over-stuffing can degrade quality and raise cost simultaneously. The engineering goal is high signal density: the fewest tokens that still contain everything the answer needs.
4. Why the Bill Suddenly Explodes
Costs rarely creep. They lurch — usually right after a success. Here are the multipliers that turn a tidy pilot into a runaway line item:
Adoption & history
More users, and longer conversations that resend the entire thread on every single turn. Success itself is a cost multiplier.
Overstuffed inputs
Sprawling system prompts, too many retrieved chunks, and whole documents pasted in when a paragraph would do — sent again and again.
Over-powered models
A top-tier model answering "what's the PTO policy?" — like couriering a postcard by chartered jet. Also: no output limits, so answers ramble.
Loops & retries
Agentic workflows firing several model calls per request, repeated tool calls, and automatic retries on errors — all invisible to the user.
Dev & duplicate traffic
Testing, evaluation runs, and the same question asked a thousand times with no caching to catch the repeats.
No ownership
No monitoring by team, feature, or user — so nobody notices the spike until finance does, a month later.
The sneakiest of these is agentic amplification. To a user, "book me the cheapest flight and put it on the calendar" is one request. Under the hood it can become a small committee meeting: one call to plan the steps, another to retrieve options, another to validate them, one or more to run tools, another to revise after something fails, and a final call to write the answer. One visible question; six or eight billable model calls, each carrying its own stack of context. Agents are powerful precisely because they do a lot — and doing a lot is exactly what you're paying for.
5. Same Question, Two Architectures
Picture two teams building the same internal assistant, answering the same employee question: "Am I eligible to carry over unused leave into next year?" The user sees a similar, correct answer either way. The systems behind them could not be more different.
Architecture A — the Firehose. Every request bolts on the entire leave-policy manual, the full running chat history, and eight retrieved passages (most only loosely related), then sends the lot to a premium model with no cap on the response. It works. It's also carrying a filing cabinet to answer a yes/no question.
Architecture B — the Metered System. The same request first gets classified ("this is a simple policy lookup"), then routed to a small, inexpensive model. It retrieves only the three genuinely relevant passages, reuses cached policy text it has served before, replaces old chat turns with a short summary, and caps the answer length. It escalates to the premium model only on the rare hard question that actually needs it.
| Per request (illustrative) | A — Firehose | B — Metered |
|---|---|---|
| System prompt | 1,200 | 500 |
| Policy content sent | 18,000 (full manual) | 2,100 (3 passages) |
| Conversation history | 9,000 (verbatim) | 600 (summarised) |
| The question | 40 | 40 |
| Output | 1,500 (uncapped) | 500 (capped) |
| Model tier | Premium, always | Small model for ~80% |
| Input tokens processed | ≈ 28,240 | ≈ 3,240 |
Roughly a tenth of the input tokens — and most requests running on a cheaper model on top of that. Same answer, a fraction of the cost, and faster too, because there's less to read. Multiply that gap across two billion tokens a month and you understand why two companies with identical AI features can post wildly different invoices.
The honest caveat
RAG does not automatically make things cheaper. Retrieval is a tool, not a discount. Badly tuned — too many chunks, chunks too large, poor relevance — it can increase context, latency, and cost while lowering answer quality. Architecture B is cheaper not because it "uses RAG" but because it retrieves well: fewer, better passages.
6. The Two-Layer Explanation
For the Business Leader
You don't need to know what a token is to govern this well. You need the right dashboard. Stop staring at "cost per token" — it's a vanity metric — and start tracking the numbers that map to reality:
- Cost per successful task, not cost per token. A task that takes three retries isn't cheap just because each call was.
- Cost per active user, and how it moves as adoption grows.
- Cost broken down by team and use case, so you can see where the money actually goes.
- Business value produced against that cost — the only ratio that ultimately matters.
- Error and retry rates, which quietly inflate everything.
- Percentage of requests needing a premium model — a direct dial on spend.
- Unit economics as you scale: does cost per user fall with volume, or rise? One of those is a business; the other is a leak.
For the Architect & Engineer
The levers are concrete, and most are unglamorous plumbing:
- Prompt and context profiling — measure what's actually being sent before optimising anything.
- Model routing — classify the request and send it to the smallest model that can handle it.
- Semantic caching — recognise a question you've effectively answered before and serve it without a fresh call.
- Conversation summarisation — compress old turns instead of resending them verbatim forever.
- RAG relevance and reranking — retrieve fewer, better passages; rerank before you send.
- Chunk size and top-k tuning — the two dials that quietly govern retrieval cost and quality.
- Output limits — cap length where a shorter answer serves the user just as well.
- Batch processing for non-urgent jobs, where providers often price it more favourably.
- Rate limits and budgets per user, team, and feature — guardrails, not afterthoughts.
- Observability by user, feature, model, and workflow — you cannot cut what you cannot see.
- Loop limits for agents — a hard ceiling on calls per request so a stuck agent can't bill infinitely.
- Continuous evaluation of quality versus cost versus latency — the three-way trade-off, watched over time.
7. Ten Practical Ways to Reduce the Bill
8. Cheap AI Can Become Expensive AI
There's a tempting shortcut: always pick the cheapest model and call it cost control. It's a trap, and here's why. A weaker model that gets the answer wrong doesn't save money — it moves the cost somewhere less visible. Now you're paying in retries, in longer and more elaborate prompts to coax it into behaving, in human reviewers cleaning up after it, in the correction loops, and occasionally in the reputational cost of a confidently wrong answer reaching a customer.
The cheapest model and the cheapest system are different things. Sometimes a more capable model, used for a smaller number of well-chosen tokens, is dramatically cheaper end-to-end than a weak model flailing across three attempts. The reverse is also true: using a flagship model to answer "what time does the office open?" is money set on fire. The discipline is matching the tool to the task, both ways.
9. Every Useful Token Matters
Somewhere along the way, "every token matters" became a slogan, and like most slogans it's half true in a way that leads people astray. Taken literally, it pushes teams toward token starvation — stripping context until the model is guessing, clipping answers until they're useless, all to shave a number that was never the real target.
The sharper version is this:
A token that grounds an answer in the right policy is worth paying for. A token that resends the same instruction for the two-hundredth time this hour is waste. The skill — the entire skill — is telling those two apart, at scale, on every request. That's not penny-pinching. That's engineering.
The Executive Takeaway
Gen AI does not become expensive because people find it useful. Usefulness is the point; adoption is the win. It becomes expensive when the architecture is careless — when every request is allowed to carry yesterday's conversations, tomorrow's possibilities, and the entire company library on its back, then hand the whole load to the most powerful model available, and do it again a million times a day.
The fix isn't to use AI less. It's to build systems that ask the model to read only what the answer needs, remember only what's worth remembering, retrieve only what's relevant, and reach for the expensive model only when the cheap one genuinely can't cope. Do that, and the brilliant assistant stays brilliant — and the bill stops being a plot twist.
A note on the numbers: every token count, ratio, and table in this article is illustrative — chosen to show how the pieces relate, not to represent any specific provider's pricing. Tokenisation varies by model and language; pricing rules, caching discounts, reasoning charges, and multimodal accounting vary by provider and change over time. Before you budget, check the official pricing for your chosen model on the day you calculate.