How a B2B SaaS Team Cut Prompt Regression Escapes by 94% with an Automated LLM Eval Pipeline in CI

A 120-engineer SaaS company wired LLM output evaluation into their CI pipeline using pytest, LangSmith and GitHub Actions, catching prompt regressions before they reached production.

AI & MLOps · intermediate · 12 min read · 2329 words · By Sandip Das · Published

Composite scenario: built from common industry patterns; the figures are illustrative.

What this covers

  1. The Situation
  2. The Problem
  3. Constraints
  4. Options Considered
  5. The Solution
  6. Implementation
  7. Results
  8. What Went Wrong Along the Way
  9. Lessons Learned
  10. Apply It Yourself

Key numbers

MetricBeforeAfter
Mean time to detect prompt regression3.2 days18 minutes
Peak customer-visible error rate during incident11.4%2.3% max (no incident exceeded baseline +1%)
Prompt-related production incidents per quarter3 per quarter0 over two quarters post-launch
Engineer hours on prompt regression post-mortems~40 hrs per incident~2 hrs per quarter
Eval suite cost per CI runN/A (no automated eval)$0.34 average

Key components

  • Golden Dataset (JSONL) — 147 synthetic contract snippets with expected outputs that serve as the ground truth for every eval run.
  • LLM-as-Judge Scorer (GPT-4o-mini) — Scores each model output against the expected answer using a rubric prompt, returning a 0–1 float and a rationale.
  • pytest-eval Runner — Orchestrates dataset loading, prompt invocation and score aggregation inside a standard pytest session so results integrate with existing CI tooling.
  • GitHub Actions Path Filter — Ensures the eval job only fires on PRs that touch prompt or chain files, keeping CI time unchanged for unrelated changes.
  • LiteLLM Proxy — Routes all eval-time LLM calls through a single gateway with spend limits and automatic Azure OpenAI fallback for resilience.
  • LangSmith Tracing — Captures full traces per PR branch so engineers can inspect the lowest-scoring cases and scorer rationales without leaving the PR.
  • Quality Gate (score threshold) — Blocks merges when the weighted mean score falls below 0.82 or any task category falls below 0.75, enforcing a hard regression check.

The Situation

This is a composite scenario assembled from patterns observed across multiple engineering organisations; all figures are illustrative and chosen for internal consistency, not to represent any single company.

The organisation is a 120-engineer B2B SaaS company that sells a document-intelligence platform to mid-market legal and financial services firms. Their core product extracts structured data—clause types, counterparty names, obligation dates—from uploaded contracts using a chain of GPT-4o calls orchestrated with LangChain. The AI team is six engineers embedded in a 30-person platform group. Deployments happen via GitHub Actions to Kubernetes on GKE, roughly twice a week. The LLM layer sits behind a LiteLLM proxy that routes to OpenAI as primary and Azure OpenAI as a hot standby.

At the point this story begins, the team had been shipping LLM features for about fourteen months. They had accumulated roughly forty distinct prompt templates across extraction, summarisation and classification tasks, all stored as Jinja2 files in the monorepo.

The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.

Topics: llmops, ci-cd, prompt-engineering, evaluation, github-actions, langchain

Related case studies

Browse all case studies