A RAG startup is paying $360k/yr on GPT-5.5. Switch or stay?

The reasoning behind the call

Start with where the $30,000/month actually goes, because it explains every move that follows. On GPT-5.5 at $5.00 input / $30.00 output per million tokens, each request bills 3,000 input tokens for $0.015 and 500 output tokens for another $0.015 — a clean 50/50 split. That symmetry is the tell: output costs 6x input per token here, so 500 output tokens weigh exactly as much as 3,000 input tokens. It means two independent levers are in play. The input half is mostly a fixed RAG scaffold — system prompt plus retrieved passages — that repeats across requests, which is the textbook case for caching. The output half is pure model rate, which only a cheaper model moves. Any plan that touches just one half leaves roughly $15,000/month on the table.

Caching attacks the input half first, and it is free to turn on. Of the 3,000 input tokens, 1,500 are cacheable; GPT-5.5's cache reads bill at $0.50/M instead of $5.00 — a 10x discount on that slice. That alone drops the bill to $23,250/month ($279,000/yr), saving $81,000 with no model change, no SDK change, and no eval. Note OpenAI charges no cache-write fee, so there is no break-even volume to clear; at 1M requests reusing the same prefix, the discount lands on essentially every call from day one. Do this regardless of what you decide about the model, and see the prompt-caching ROI guide for how reuse rate maps to savings.

The model swap attacks the expensive output half. GPT-5.4 is the same model family at $2.50/$15.00 — exactly half GPT-5.5's rate on both input and output, with cache reads at $0.25/M. Stack it with caching and every component of the per-request cost halves versus the cached-5.5 number, landing at $11,625/month ($139,500/yr). That is the $220,500/year, 61% headline, and it is the recommendation not because it is the cheapest row but because it is the cheapest row that carries almost no migration risk. The endpoint, auth, request shape, and SDK are identical; only the model string changes. Budget it at roughly one engineer-day, most of which is re-running evals rather than writing code — see why an in-family move is so much lighter than a cross-provider one.

The two genuinely cheaper-looking alternatives each carry a tax the price tag hides. Claude Sonnet 4.6 + caching comes in at $12,450/month — about $825/month more than GPT-5.4 here. Output rates are identical ($15.00/M), so the entire gap is on the input side: Sonnet's $3.00/M standard input versus GPT-5.4's $2.50/M adds about $750/month on the 1,500 non-cacheable tokens, and Sonnet's slightly higher cache-read rate ($0.30 vs $0.25/M) adds about $75/month on the cached half (the one-time cache write is negligible spread over a million reuses). So a provider switch costs you a prompt rewrite, SDK swap, and tool-schema translation to land at a higher bill; it only makes sense if you specifically need Sonnet's 1M-token context or its capability profile, not for cost. GPT-5.4 mini is the real outlier at $3,487/month — 88% off — and it keeps your SDK, but it is a smaller model being asked to do RAG answer synthesis, which is exactly the task where a weaker model quietly degrades faithfulness and citation accuracy. That is a quality bet, not a cost decision.

Which is the honest caveat: none of these numbers are quality numbers. This is list-price arithmetic on your stated workload, and it says nothing about whether GPT-5.4 — let alone the mini — answers your users' questions as well as GPT-5.5 does on your retrieval distribution. Before cutover, run 200–500 of your own production prompts through GPT-5.4 with caching enabled and score them the way you actually care about (faithfulness to retrieved context, refusal rate, format adherence), then do the same for the mini in parallel so you know what 88% would cost you in answer quality. Watch max_output_tokens too — the mini caps at 32,768, below GPT-5.5's 65,536. The defensible sequence is: enable caching on 5.5 today to bank $81,000 with zero risk, validate 5.4 against it this week, and treat the mini as a separate experiment to be earned, not assumed. You can re-run any of this on your exact numbers in the migration ROI simulator.