Case Studies

Page 2 of the LearnXops case study library: real-world cloud, DevOps, MLOps and AI engineering postmortems and production scenarios.

  1. How a B2B SaaS Team Cut Postgres P99 Latency by 73% with Pooling, Replicas and Partitioning

    An overloaded single Postgres instance was killing checkout latency. Connection pooling, read replicas, and time-based table partitioning brought P99 from 4.2 s to 1.1 s without a rewrite.

    Data & Storage · intermediate · 12 min read

  2. How a B2B SaaS Team Cut Prompt Regression Escapes by 94% with an Automated LLM Eval Pipeline in CI

    A 120-engineer SaaS company wired LLM output evaluation into their CI pipeline using pytest, LangSmith and GitHub Actions, catching prompt regressions before they reached production.

    AI & MLOps · intermediate · 12 min read

  3. How a B2B SaaS Team Cut Supply-Chain Risk by Enforcing SBOMs, Sigstore Signing, and OPA Admission Policies

    A 120-engineer SaaS company rebuilt its CI/CD pipeline around Syft SBOMs, Cosign signatures, and OPA Gatekeeper to block unsigned or unverified images from ever reaching production.

    Security · advanced · 12 min read

  4. How a B2B SaaS Team Cut Their AWS Bill by 40% with Rightsizing, Graviton, Savings Plans and S3 Lifecycle Policies

    A 120-engineer SaaS company slashed its AWS spend by 40% in 90 days by combining instance rightsizing, Graviton3 migration, Compute Savings Plans, and aggressive S3 lifecycle tiering — without touching a single line of product code.

    Infrastructure · intermediate · 12 min read

  5. How a B2B SaaS Team Cut Their Observability Bill by 60% with OpenTelemetry Sampling and Log Tiering

    A 120-engineer SaaS company slashed $28k/month in observability costs by combining head-based OTel sampling, tail-based error retention, and S3 log tiering—without losing production visibility.

    Observability & SRE · intermediate · 12 min read

  6. How a B2B SaaS Team Cut Weekly Alerts from 400 to 18 with SLO-Based Alerting

    A 120-engineer SaaS company replaced threshold spam with error-budget burn-rate alerts, dropping weekly pages from 400 to 18 and cutting MTTR by 60%.

    Observability & SRE · intermediate · 12 min read

  7. How a B2B SaaS Team Eliminated Duplicate Order Events by Adopting the Transactional Outbox Pattern

    A dual-write bug was silently duplicating and dropping order events in production. The team replaced it with a transactional outbox and a Debezium CDC relay, cutting event loss to zero.

    Messaging & Events · intermediate · 12 min read

  8. How a B2B SaaS Team Eliminated Plaintext Secrets and Cut Credential Exposure Risk by 94%

    A 120-engineer SaaS company replaced .env files and static DB passwords with HashiCorp Vault dynamic credentials, cutting secret sprawl from 340 hardcoded values to under 20 long-lived secrets.

    Security · intermediate · 12 min read

  9. How a B2B SaaS Team Migrated 8 TB from Self-Managed MySQL to Aurora with Zero Downtime

    A 120-engineer SaaS company moved 8 TB of MySQL data to Aurora MySQL with no service interruption, using DMS continuous replication, schema versioning, and a traffic-shifting cutover that took under 4 minutes.

    Data & Storage · advanced · 12 min read

  10. How a B2B SaaS Team Migrated a Rails Monolith to EKS With Zero Downtime

    A 120-engineer SaaS company moved a decade-old Rails app from hand-managed EC2 to Amazon EKS using a strangler-fig approach, cutting deploy time from 47 minutes to 4 and eliminating weekend outages.

    Containers & K8s · intermediate · 12 min read

  11. How a B2B SaaS Team Split a 4,000-Resource Terraform Monolith Into Modules Without Downtime

    A 120-engineer SaaS company decomposed a single 4,000-resource Terraform state file into scoped modules and isolated state backends, cutting plan times from 22 minutes to under 90 seconds with zero production incidents.

    Infrastructure · advanced · 12 min read

  12. How a B2B SaaS Team Took a RAG Support Assistant from Demo to Production

    A 120-engineer SaaS company rebuilt their RAG support bot's retrieval pipeline, added an eval harness and guardrails, and cut hallucination-related escalations by 68%.

    AI & MLOps · intermediate · 12 min read

  13. How a Config Push and a Maintenance Tool Severed Google Cloud Networking in June 2019

    A misconfigured ACL pushed to backbone routers, combined with a maintenance tool that withdrew routes, dropped Google Cloud network capacity across US regions for up to 4 hours 45 minutes.

    Networking & APIs · advanced · 12 min read

  14. How a Feature Store Eliminated Training-Serving Skew and Cut Model Errors by 34%

    A B2B SaaS company's recommendation model silently degraded because training and serving computed features differently. Adopting Feast on Redis fixed the skew and cut prediction errors by 34%.

    AI & MLOps · advanced · 12 min read

  15. How a Fintech Team Caught Silent Fraud-Model Drift Before It Cost Them $2M

    A fraud model degraded silently for six weeks before anyone noticed. The team built a drift-detection and auto-retraining pipeline that cut false-negative fraud losses by 61%.

    AI & MLOps · intermediate · 12 min read

  16. How a Five-Person SRE Team Cut Operational Toil by 50% with Automation and Observability

    A small SRE team drowning in manual alerts and runbooks automated their way to half the toil in 90 days using Terraform, PagerDuty, and a self-healing Kubernetes operator.

    Fundamentals · beginner · 12 min read

  17. How a New Telemetry Service Took Down OpenAI's Kubernetes API Servers and DNS in December 2024

    A new telemetry deployment in December 2024 overwhelmed OpenAI's Kubernetes API servers and DNS, causing a ~4-hour outage across ChatGPT, the API, and Sora.

    Containers & K8s · advanced · 12 min read

  18. How a Retail Platform Survived 10x Black Friday Traffic With Load Testing and Game Days

    A mid-size e-commerce platform used k6, chaos engineering, and capacity planning sprints to handle a 10x traffic spike on Black Friday without a single P1 incident.

    Fundamentals · intermediate · 12 min read

  19. How a Shared Kubernetes Cluster Raised GPU Utilisation from 25% to 70%

    A B2B SaaS company's ML platform team tripled effective GPU utilisation by combining time-slicing, bin-packing, and a fair-share queue — without buying a single new node.

    AI & MLOps · advanced · 12 min read

  20. How an Event-Driven Kafka Pipeline Cut Order Latency by 74% and Eliminated Cascade Failures

    A B2B SaaS company replaced a brittle chain of synchronous REST calls with a Kafka-backed order pipeline, cutting p99 latency from 4.2 s to 1.1 s and eliminating downstream cascade failures.

    Messaging & Events · advanced · 12 min read

  21. How Anomaly Detection Cut Alert Noise by 73% and Caught Incidents 18 Minutes Earlier

    A B2B SaaS platform drowning in threshold alerts rebuilt its observability pipeline around ML-based anomaly detection, slashing noise and catching real incidents before customers noticed.

    Observability & SRE · intermediate · 12 min read

  22. How Backoff, Jitter, and Circuit Breakers Stopped a Retry Storm from Killing a Production Database

    A slow Postgres instance triggered aggressive client retries that amplified load 40x, causing a full outage. Exponential backoff, jitter, and a circuit breaker pattern restored stability within one sprint.

    Distributed Systems · intermediate · 12 min read

  23. How Discord Migrated Trillions of Messages from Cassandra to ScyllaDB

    Discord's Cassandra cluster hit latency and hotspot limits at trillions of messages; a full migration to ScyllaDB cut p99 latency and reduced node count from 177 to 72.

    Data & Storage · advanced · 12 min read

  24. How Figma Scaled Postgres from One Overloaded Database to Vertical Partitioning and Horizontal Sharding

    Figma's single Postgres instance hit its vertical scaling ceiling as product growth exploded. The team escaped by vertically partitioning tables across logical databases, then horizontally sharding the largest ones.

    Data & Storage · advanced · 12 min read