How a B2B SaaS Team Cut LLM API Spend by 65% with Model Routing, Prompt Caching, and Request Batching
A 120-engineer SaaS company slashed monthly LLM costs from $41k to $14k by routing cheap models for simple tasks, caching repeated prompts in Redis, and batching async jobs — without degrading user-facing quality.
AI & MLOps · intermediate · 12 min read · 2433 words · By Sandip Das · Published
Composite scenario: built from common industry patterns; the figures are illustrative.
What this covers
- The Situation
- The Problem
- Constraints
- Options Considered
- The Solution
- Implementation
- Results
- What Went Wrong Along the Way
- Lessons Learned
- Apply It Yourself
Key numbers
| Metric | Before | After |
|---|---|---|
| Monthly LLM API spend | $41,000 | $14,350 |
| Cost per 1,000 requests | $1.84 | $0.64 |
| Rate-limit errors per day | ~120 | ~4 |
| Prompt cache hit rate (clause-tagger) | 0% | 34% |
| Batch pipeline p95 delay | 45 min | 8 min |
Key components
- Model Router — Dispatches each inference request to the cheapest model capable of meeting the task's quality requirement, based on rules stored in Postgres.
- Complexity Classifier — Scores incoming requests by task type and token budget so the router can apply the correct routing rule without ML inference overhead.
- Prompt Cache (Redis) — Stores completed LLM responses keyed on a normalised prompt hash, serving repeated requests without an API call.
- Batch Queue + Worker — Accumulates latency-tolerant requests and submits them via the OpenAI Batch API to claim the 50% async discount.
- Cost Metrics + Spend Alerts — Tracks dollars-per-request and projected daily spend, alerting on-call engineers before budget thresholds are breached.
The Situation
This is a composite scenario assembled from patterns observed across multiple production AI deployments; all figures are illustrative and chosen for internal consistency, not to represent any single organisation.
The organisation in question is a 120-engineer B2B SaaS company selling a document-intelligence platform to mid-market legal and financial services firms. Their product lets analysts upload contracts, financial filings, and compliance documents, then ask natural-language questions, generate summaries, and extract structured data. The AI team — four ML engineers and one platform engineer — had shipped LLM-powered features across three product surfaces: a real-time Q&A chat, a batch document-summary pipeline, and an automated clause-tagging service that ran nightly.
The stack was Python-heavy: FastAPI services on Kubernetes (EKS), Celery workers backed by Redis and RabbitMQ, and PostgreSQL for structured data. All LLM calls went directly to the OpenAI API, exclusively using gpt-4-turbo at the time. Monthly spend had grown from $6k to $41k over nine months as the customer base tripled.
The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.
Topics: llm, cost-optimization, mlops, prompt-engineering, api-gateway, caching
Related case studies
- Air Canada's Chatbot Hallucination: How a RAG Support Bot Created Legal Liability in Production
- How a B2B SaaS Team Cut LLM Inference Costs by 71% by Moving to Self-Hosted vLLM
- How a B2B SaaS Team Cut Model Deployment Time from 3 Days to 4 Hours with MLflow and Kubeflow
- How a B2B SaaS Team Cut Prompt Regression Escapes by 94% with an Automated LLM Eval Pipeline in CI
- How a B2B SaaS Team Took a RAG Support Assistant from Demo to Production
- How a Feature Store Eliminated Training-Serving Skew and Cut Model Errors by 34%