RAG Cost Architecture: Where the Money Actually Goes in 2026

The four cost centers, and why three of them are decoys

A retrieval-augmented system has four places it spends money, and they sit at wildly different scales. Indexing embeddings is paid once when you ingest a corpus (and again, incrementally, as documents change). Per-query embeddings are paid every time you embed a user question to search the vector store. The generation step is the model call that reads your context and writes an answer. And reranking, if you use it, is an extra scoring pass over candidate chunks before they reach the generator.

Most teams instinctively reach for the embedding line first, because it is the part that feels most "RAG-specific." That instinct is almost always wrong. In a production system answering real questions, embeddings are rounding error and the generation call is the bill. But generation cost is not one number either — it splits into the tokens you send in and the tokens the model writes back, and the input side hides a quiet multiplier most dashboards never surface.

The thing nobody puts on a slide: every chunk you retrieve becomes input tokens on the generation call, charged again on every single request. Retrieval is not a one-time cost like the index. It rides along on each query, multiplied by your traffic. That is the cost center this guide is really about, because it is the one you can move the most and the one teams optimize last.

• Index embeddings — one-time-ish, paid at ingest and on document churn
• Per-query embeddings — cheap, paid once per question to search the store
• Generation input — retrieved context + instructions, paid on EVERY request (the silent giant)
• Generation output — what the model writes, the priciest per token (5-6x input)
• Reranking — optional scoring pass; can lower generation input if it lets you send fewer chunks

The core insight: retrieved context is input tokens, billed per request

Walk the arithmetic with a concrete shape. Say you run a support assistant on Claude Sonnet 4.6 ($3.00 input, $15.00 output per 1M tokens) at one million requests a month. Each request carries a 1,500-token instruction/system block, 1,500 tokens of retrieved context (roughly top-k of 8 medium chunks), and produces a 500-token answer.

Input side: the 1,500-token instruction block is 1,500 × 1,000,000 = 1.5B tokens/mo, which at $3.00/1M is $4,500. The 1,500 tokens of retrieved context are another 1.5B tokens — another $4,500. Output: 500 × 1M = 500M tokens at $15.00/1M is $7,500. Total: $16,500/mo, and the retrieved context alone accounts for 27% of it.

Now watch what happens when someone bumps top-k from 8 to 20 "to improve recall" — a one-line change in a retrieval config that no one reviews for cost. Retrieved context goes from 1,500 to roughly 3,750 tokens. That line jumps from $4,500 to $11,250/mo: an extra $6,750 every month for chunks the model mostly ignores. There was no model change, no traffic change, no alert. The bill just grew by 41% because retrieval is a per-request input multiplier and almost nobody treats it like one.

This is the mental model to keep: retrieved tokens are not free context, they are recurring input cost scaled by your QPS. Every fix below is, at bottom, a way to either stop paying full input price for the same tokens, or to send fewer tokens without losing the answer.

Lever 1: cache the stable prefix (and know what you can't cache)

The first lever costs almost nothing to pull. Your 1,500-token instruction/system block is identical on every request — same persona, same output rules, same few-shot examples. That is a textbook prompt-caching target. On Claude Sonnet 4.6, a cache read is $0.30/1M versus $3.00/1M standard input, a 10x reduction, after a one-time cache write at $3.75/1M. At a million warm requests a month, that one write is a rounding error and you essentially pay read rates all month.

Run the same workload with the prefix cached: the instruction block drops from $4,500 to 1,500 × 1M × $0.30/1M = $450/mo. Retrieved context and output are unchanged, so the total falls from $16,500 to $12,450/mo — $4,050 saved for what is essentially a configuration change. OpenAI and Gemini make this even simpler operationally: no write fee at all (GPT-5.4 cache read $0.25, Gemini 2.5 Flash cache read $0.03), though Anthropic's deeper read discount usually wins at volume. The full break-even math is in the Prompt Caching ROI guide.

Here is the catch that trips up RAG teams specifically: the retrieved chunks are usually NOT cacheable. Caching keys on an exact, stable token prefix. Your retrieved context changes with every query — different question, different chunks, different order — so it never produces a cache hit. Worse, if you place retrieved chunks before your stable instructions in the prompt, you break the cacheable prefix entirely, because the cache only extends up to the first token that differs.

The structural fix is ordering. Put everything stable first — system rules, persona, few-shot examples, any fixed boilerplate — mark the cache breakpoint there, and append the volatile retrieved chunks and user question after it. You cache what is constant and pay full freight only for what genuinely changes per request. Most teams that "tried caching and it didn't help" had the order inverted.

Lever 2: retrieve less, but better

Caching shrinks the price of the stable tokens. Lever 2 attacks the part you can't cache: the retrieved chunks themselves. Since those are full-price input on every call, the cheapest retrieved token is the one you never send. The goal is to feed the generator the smallest context that still contains the answer — not the largest context that might.

A high top-k is a comfort blanket. Pushing top-k from 8 to 20 rarely improves answer quality in proportion to the tokens it adds; past a point you are stuffing the prompt with near-duplicate or off-topic passages that dilute the signal and, on some models, actively hurt the answer ("lost in the middle"). Tighter, well-chunked retrieval frequently produces better answers and a smaller bill at the same time — one of the rare cases where the cheap option is also the good one.

This is where reranking earns its keep. A reranker is itself a cost, but it is a cheap scoring model whose job is to let you over-retrieve broadly and then send only the top few survivors to the expensive generator. Retrieve 30 candidates, rerank, pass the best 4–6. You pay a little for the rerank pass and a lot less on generation input, because you cut the per-request retrieved tokens hard. Reranking only pays off when generation input is your dominant cost — which, as the map above shows, it usually is.

Chunking strategy is the other half. Oversized chunks waste tokens by dragging irrelevant neighboring text into context; undersized chunks fragment answers across many retrieved pieces and force a higher top-k to reconstruct them. Tune chunk size and overlap against your own questions, then measure retrieved tokens per request — that number, multiplied by your monthly volume and your input rate, is a line item you can watch and drive down. Use the token counter to see exactly how many tokens your current chunks consume.

Lever 3: match the synthesis model to the question

The generation model is the largest cost center, so model choice is the largest single lever — but "pick the cheapest model" is the wrong framing. The right framing is routing: not every question needs your best model. A support RAG handles a long tail of simple lookups ("what are your hours," "how do I reset my password") alongside a smaller set of genuinely hard, multi-document reasoning questions.

Route the easy majority to a cheaper tier and reserve the flagship for the hard minority. Claude Haiku 4.5 ($1.00/$5.00, cache read $0.10) or Gemini 2.5 Flash ($0.30/$2.50) can answer a grounded, well-retrieved factual question perfectly well when the context already contains the answer — which is the whole point of RAG. You are not asking the model to know things; you are asking it to read what you handed it. That is a job a small model often does fine, and you should confirm it with your own eval rather than my assertion.

Long-context models change the architecture, not just the price. Gemini 2.5 Pro carries a 2M-token window; Claude Sonnet/Opus, Grok 4.3, and DeepSeek V4 all reach 1M. That headroom lets you make a deliberate tradeoff: for a small or slow-changing corpus, you can skip aggressive chunking and retrieval tuning and lean on a big window — but remember Lever 2's warning, because every token you pour into that window is still billed as input on every call. A big context window is permission to send more tokens, not a discount on them.

Two honest caveats. First, never assume quality equivalence across tiers from a price table — a cheaper synthesis model can fabricate or miss nuance on your specific distribution, so run a 200–500 prompt eval on real traffic before cutting over. Second, output is the expensive side (5-6x input), so capping verbose answers and trimming output format is its own lever; the output cost guide covers it, and tier comparisons like Haiku 4.5 vs GPT-5.4 mini show where the cheap tiers land.

Worked example: a 1M-request support RAG

Put the levers together on the workload above — Sonnet 4.6, 1M req/mo, 1,500-token stable prefix + 1,500-token retrieved context + 500-token output. Baseline with nothing optimized: $16,500/mo ($4,500 prefix + $4,500 retrieved + $7,500 output). This mirrors the scenario costed in the RAG startup: switch or stay? report, where the headline question is whether to change providers at all.

The instinct is to chase a cheaper provider. Switching the generation call to GPT-5.4 ($2.50/$15.00, cache read $0.25) with the prefix cached lands at $11,625/mo — a real 30% cut, but it costs you an SDK migration, a prompt rewrite, and a fresh round of evals. Now compare the in-place fixes. Lever 1 alone (cache the prefix on your existing Sonnet) drops you to $12,450/mo with zero model change. Add Lever 2 — rerank and trim retrieved context from 1,500 to ~800 tokens — and you reach $10,350/mo, a 37% reduction.

Read that result carefully: trimming context and caching on the model you already run ($10,350) beats the provider switch ($11,625) by about $1,275/mo, with none of the migration risk. The team that rewrites its stack to chase a lower sticker rate did more work for a worse result than the team that fixed its retrieval. This is the central lesson — the highest-leverage move is usually internal, not a vendor change.

Then stack Lever 3 on top. Route ~60% of traffic (the simple lookups) to Haiku 4.5 and keep the hard 40% on Sonnet, both with the trimmed context and cached prefix: total falls to roughly $6,210/mo, a 62% reduction from baseline — and still no provider migration, just routing inside your own stack. Model your own version of this in the migration ROI simulator with your real token counts before committing to any of it.

• Baseline (Sonnet 4.6, top-k=8, no cache): $16,500/mo
• Switch generation to GPT-5.4 + cache prefix: $11,625/mo (−30%, but full migration)
• Lever 1 — cache stable prefix on Sonnet: $12,450/mo (−25%, config only)
• Levers 1+2 — cache + rerank/trim to ~800 retrieved tokens: $10,350/mo (−37%, beats the switch)
• Levers 1+2+3 — add 60% routing to Haiku 4.5: ~$6,210/mo (−62%, no provider change)

Embeddings reality check: why the index isn't your problem

Embeddings get the most attention and deserve the least. Two distinct costs hide under the word. The index is paid once when you ingest a corpus, plus incrementally as documents change — genuinely one-time-ish unless your knowledge base churns constantly. Per-query embedding is paid each time you embed a user question to search the store, and a question is short, typically tens of tokens.

Scale it against the generation bill to feel the gap. Embedding a 5-million-document corpus at ~800 tokens each is 4 billion tokens — but at typical embedding rates (a small fraction of a dollar per million tokens), that whole one-time index lands in the low tens of dollars, not thousands. Per-query embeddings at, say, 40 tokens per question across a million requests is 40M tokens/mo — under a dollar at the same rates. Both numbers are illustrative (embedding SKUs vary by provider and aren't in this site's generation pricing), but the order of magnitude is the point: against $16,500/mo of generation, embeddings are noise.

"Cheap" is not "ignore," though. At very large scale or with high document churn — re-embedding a corpus nightly, embedding multi-thousand-token documents in bulk — the index line can climb into a real number, and using a batch tier (where available, typically 50% off) for ingest is worth it. But the triage order is clear: get generation input under control first. If you are tuning your embedding model to shave the index bill while a top-k of 20 quietly burns $6,750/mo on the generation call, you are optimizing the wrong cost center.

A triage order that respects the leverage

Because the cost centers differ by orders of magnitude, the order you tackle them in matters as much as the fixes themselves. Work from highest leverage and lowest risk toward lowest leverage and highest risk, and re-measure after each step so you stop when you've hit your target instead of over-engineering.

Measure first: instrument retrieved-tokens-per-request and cached-vs-uncached input on your live traffic. You cannot optimize a number you don't log, and most RAG bills are surprises precisely because no one tracked the per-request token shape. The cost monitoring guide covers what to capture.

• Cache the stable prefix and verify cache hits in the response usage metadata — biggest win for least effort
• Audit top-k and chunking; add reranking so you can over-retrieve then send only the best 4-6 chunks
• Route easy questions to a cheaper synthesis tier; keep the flagship for hard, multi-doc reasoning
• Cap and tighten output, since output tokens cost 5-6x input
• Only then consider a provider switch — and price the in-place fixes against it first; they often win
• Leave embeddings last unless your corpus churn is genuinely large

Frequently asked questions

Related reading