How a B2B SaaS Team Cut LLM Inference Costs by 71% by Moving to Self-Hosted vLLM

A 120-engineer SaaS company replaced OpenAI API calls with self-hosted Mistral on vLLM, cutting per-query cost by 71% and p99 latency by 40% after a painful but instructive migration.

AI & MLOps · advanced · 12 min read · 2411 words · By Sandip Das · Published

Composite scenario: built from common industry patterns; the figures are illustrative.

What this covers

  1. The Situation
  2. The Problem
  3. Constraints
  4. Options Considered
  5. The Solution
  6. Implementation
  7. Results
  8. What Went Wrong Along the Way
  9. Lessons Learned
  10. Apply It Yourself

Key numbers

MetricBeforeAfter
Monthly inference cost$41,000$11,000
Cost per 1,000 queries$0.87$0.23
p99 latency11.4 s6.8 s
AI feature monthly uptime99.3%99.7%
Data residency complianceBlocked 2 EU dealsBoth deals unblocked

Key components

  • vLLM with continuous batching — Serves Mistral 7B Instruct with high GPU utilisation and OpenAI-compatible API, enabling a near-zero application code change.
  • Inference Router (FastAPI) — Handles prompt caching, feature-flag-based canary routing, pre-flight token counting, and circuit-breaking to the OpenAI fallback.
  • SQS FIFO Request Queue — Decouples the router from the vLLM workers, absorbing traffic spikes without dropping requests.
  • Prompt Cache (Redis) — Caches full responses keyed on a SHA-256 of the assembled prompt, eliminating 18% of GPU load at no model cost.
  • Online eval judge (DeBERTa classifier) — Scores a 2% sample of live responses during the canary ramp, providing a continuous quality signal without human annotation.
  • Karpenter GPU NodePool — Autoscales g5.12xlarge nodes for the vLLM cluster, with a minimum warm-node budget to prevent cold-start latency spikes.

The Situation

This is a composite scenario assembled from patterns observed across multiple real-world LLM migrations; all organisations, names, and figures are illustrative and invented for teaching purposes.

The organisation is a 120-engineer B2B SaaS company selling a document intelligence platform to mid-market legal and financial services firms. Their product lets analysts upload contracts, earnings reports, and regulatory filings and then ask natural-language questions against them. The AI feature launched 14 months ago as a thin wrapper around the OpenAI Chat Completions API (GPT-4 Turbo at the time). The ML team is five engineers: two who came from classical NLP, two who joined from backend, and a staff engineer who owns the architecture. Infrastructure runs on AWS, primarily EKS with Karpenter for node autoscaling. The data plane is PostgreSQL on RDS, S3 for object storage, and Redis for caching. Observability is Datadog with some OpenTelemetry instrumentation added over the past year.

At launch, the AI feature handled roughly 4,000 queries per day. Fourteen months later it was processing 47,000 queries per day, with peaks above 70,000 during earnings season. Each query involves a retrieval step (pgvector similarity search), prompt assembly (system prompt plus retrieved chunks), and a single Chat Completions call that returns a structured JSON response. Average prompt length had grown to 3,200 tokens; average completion was 420 tokens.

The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.

Topics: llm, vllm, mlops, inference, cost-optimization, self-hosting

Related case studies

Browse all case studies