How a B2B SaaS Team Took a RAG Support Assistant from Demo to Production
A 120-engineer SaaS company rebuilt their RAG support bot's retrieval pipeline, added an eval harness and guardrails, and cut hallucination-related escalations by 68%.
AI & MLOps · intermediate · 12 min read · 2121 words · By Sandip Das · Published
Composite scenario: built from common industry patterns; the figures are illustrative.
What this covers
- The Situation
- The Problem
- Constraints
- Options Considered
- The Solution
- Implementation
- Results
- What Went Wrong Along the Way
- Lessons Learned
- Apply It Yourself
Key numbers
| Metric | Before | After |
|---|---|---|
| Retrieval precision@5 | 54% | 81% |
| Faithfulness score (LLM-as-judge, 1–5) | 2.9 | 4.2 |
| Hallucination-driven escalations per week | ~20 | ~6.5 |
| Bot-session CSAT (out of 5) | 3.1 | 4.0 |
| Tier-1 deflection rate | 11% | 29% |
Key components
- Cross-Encoder Reranker — Scores top-40 ANN candidates against the raw query to return the 5 most relevant chunks, replacing single-stage top-k retrieval.
- OpenTelemetry Trace Store — Captures query, chunk IDs, reranker scores, and LLM response for every request, enabling debugging and offline eval.
- LLM-as-Judge Eval Harness — Nightly job that samples 200 traces and scores faithfulness using gpt-4o-mini, producing a continuous quality signal.
- Guardrails Layer — Validates inbound queries for prompt injection and off-topic intent, and checks outbound answers for unsupported claims before returning to the user.
- Metadata-Aware Chunking — Embeds product version and last-modified date in each chunk payload so retrieval can filter by version, eliminating cross-version hallucinations.
The Situation
This is a composite scenario drawn from patterns observed across multiple production RAG deployments; all figures are illustrative and internally consistent, not drawn from any single organisation.
The organisation is a 120-engineer B2B SaaS company selling a workflow-automation platform to mid-market operations teams. Their product generates a large, fast-changing knowledge base: release notes, API docs, integration guides, and an internal runbook wiki totalling roughly 14,000 documents updated weekly. The support team of 18 agents handles around 2,400 tickets per month, and leadership wanted to deflect at least 30% of tier-1 questions to a self-service chat assistant.
The ML platform sub-team — three engineers and one ML engineer — built a prototype in six weeks using LangChain, OpenAI gpt-4o, and a Pinecone index. It demoed well. Stakeholders approved a production rollout.
The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.
Topics: rag, llmops, retrieval, evals, guardrails, mlops
Related case studies
- Air Canada's Chatbot Hallucination: How a RAG Support Bot Created Legal Liability in Production
- How a B2B SaaS Team Cut LLM API Spend by 65% with Model Routing, Prompt Caching, and Request Batching
- How a B2B SaaS Team Cut LLM Inference Costs by 71% by Moving to Self-Hosted vLLM
- How a B2B SaaS Team Cut Model Deployment Time from 3 Days to 4 Hours with MLflow and Kubeflow
- How a B2B SaaS Team Cut Prompt Regression Escapes by 94% with an Automated LLM Eval Pipeline in CI
- How a Feature Store Eliminated Training-Serving Skew and Cut Model Errors by 34%