Case Studies
Real-world engineering case studies for cloud, DevOps, MLOps and AI: public postmortems and production scenarios, with the architecture, the numbers and the lessons.
Air Canada's Chatbot Hallucination: How a RAG Support Bot Created Legal Liability in Production
Air Canada's support chatbot invented a bereavement fare policy that didn't exist; a tribunal held the airline liable. The lesson: LLM outputs need guardrails and legal accountability before going to production.
AI & MLOps · beginner · 12 min read
AWS Keys Leaked to GitHub: The First 60 Minutes and the Clean-Up
A developer accidentally committed long-lived AWS access keys to a public repo. This case walks through the detection, containment, and the permanent fix: short-lived credentials via IAM roles and automated secret scanning.
Security · beginner · 12 min read
Building an Incident Response Program from Scratch: On-Call, Severities, and Blameless Reviews
A 120-engineer B2B SaaS company had no formal incident process, causing multi-hour outages with no coordination. They built on-call rotations, severity tiers, and blameless postmortems—cutting MTTR from 4.2 hours to 38 minutes.
Fundamentals · beginner · 12 min read
Chasing 5-Second DNS Timeouts in Kubernetes: ndots, Conntrack Races, and NodeLocal DNSCache
Intermittent 5-second API latency spikes traced to Kubernetes DNS conntrack race conditions and ndots misconfiguration, resolved with NodeLocal DNSCache and search-domain tuning.
Networking & APIs · advanced · 12 min read
How 37signals Saved $10M by Leaving AWS for Owned Hardware
37signals moved HEY and Basecamp off AWS to owned servers, projecting over $10M in savings over five years. The lesson: cloud economics invert at scale.
Infrastructure · intermediate · 12 min read
How a 120-Engineer SaaS Company Cut Service Onboarding from 3 Days to 2 Hours with Backstage Golden Paths
A mid-size B2B SaaS company replaced a fragmented self-service mess with a Backstage IDP and opinionated golden paths, dropping new-service onboarding from 3 days to under 2 hours and halving platform support tickets.
Architecture · intermediate · 12 min read
How a 120-Engineer SaaS Company Migrated from One AWS Account to a Multi-Account Landing Zone
A B2B SaaS company collapsed security blast radius and achieved compliance by adopting AWS Organizations, Control Tower, and SCPs — learning hard lessons about OU design and SCP ordering along the way.
Infrastructure · intermediate · 12 min read
How a 120-Engineer SaaS Company Replaced Its Corporate VPN with Zero-Trust Access
A mid-size B2B SaaS company eliminated its legacy VPN bottleneck by rolling out Cloudflare Access and WARP, cutting engineer connection failures by 94% and slashing lateral-movement risk.
Security · intermediate · 12 min read
How a 300-Engineer Monorepo Cut CI Time by 78% with Remote Caching and Affected-Only Builds
A fast-growing SaaS platform tamed a sprawling monorepo by combining Nx remote caching with affected-graph builds, slashing median CI from 47 min to 10 min and saving $180k/year in compute.
CI/CD & Delivery · advanced · 12 min read
How a B2B SaaS Company Strangled Its Rails Monolith Into Microservices Without a Big Bang Rewrite
A 120-engineer SaaS team used the strangler fig pattern to extract five domains from a 9-year-old Rails monolith, cutting deploy time from 47 minutes to 6 and reducing p99 latency by 62%.
Architecture · intermediate · 12 min read
How a B2B SaaS Company Turned a Swampy S3 Data Lake into a Lakehouse with Apache Iceberg
A 120-engineer SaaS company replaced a brittle Hive-metastore S3 lake with Apache Iceberg, cutting query costs 68% and eliminating overnight partition repair jobs that blocked analysts for hours.
Data & Storage · advanced · 12 min read
How a B2B SaaS Platform Cut Observability Costs 41% by Migrating 150 Services to OpenTelemetry
A 120-engineer SaaS company replaced three competing vendor agents with OpenTelemetry collectors across 150 microservices, cutting observability spend by 41% and mean time to detect by 58%.
Observability & SRE · advanced · 12 min read
How a B2B SaaS Platform Halved Kubernetes Compute Costs with Karpenter and Right-Sized Workloads
A 120-engineer SaaS company cut EKS compute spend by 52% in 90 days by replacing Cluster Autoscaler with Karpenter, enforcing VPA-driven right-sizing, and enabling consolidation. The lesson: most Kubernetes waste is invisible until you measure it.
Containers & K8s · advanced · 12 min read
How a B2B SaaS Platform Migrated 200 Services to GitOps with Argo CD
A 120-engineer SaaS company replaced ad-hoc kubectl and CI-push deployments with Argo CD GitOps across 200 services, cutting deployment failures by 70% and mean time to recovery from 45 minutes to under 5.
CI/CD & Delivery · intermediate · 12 min read
How a B2B SaaS Platform Raised CDN Cache Hit Ratio from 40% to 92% and Halved Origin Load
A mid-size SaaS company fixed broken cache-control headers, split cacheable from dynamic traffic, and introduced edge-side cache warming to cut origin load by 52% and slash p99 latency.
Caching · intermediate · 12 min read
How a B2B SaaS Platform Upgraded 40 Kubernetes Clusters Across Three Versions Without an Outage
A 120-engineer SaaS company migrated 40 production Kubernetes clusters from 1.24 to 1.27 using a wave-based canary strategy, automated preflight checks, and zero customer-facing downtime.
Containers & K8s · advanced · 12 min read
How a B2B SaaS Team Built Multi-Region DR on AWS with a Tested 15-Minute RTO
A 120-engineer SaaS company replaced a single-region architecture with active-passive multi-region DR on AWS, cutting RTO from 4+ hours to under 15 minutes through automation, chaos testing, and disciplined runbook engineering.
Distributed Systems · advanced · 12 min read
How a B2B SaaS Team Contained Prompt Injection Across an AI Agent With Live Tool Access
An AI agent with database and API tool access was hijacked via prompt injection in user-supplied data, exfiltrating records. The team rebuilt the agent with layered defences and cut critical incidents to zero.
Security · advanced · 12 min read
How a B2B SaaS Team Cut CI and Batch Costs by 70% Using Spot Instances Without Losing Jobs
A 120-engineer SaaS team moved CI pipelines and nightly batch workloads to Spot/Preemptible instances, slashing compute bills by 70% through checkpointing, retries, and mixed instance-type pools.
Infrastructure · intermediate · 12 min read
How a B2B SaaS Team Cut CI Pipeline Time from 45 Minutes to 8
A 120-engineer SaaS team slashed CI from 45 min to 8 min using layer caching, parallel test sharding, and affected-test selection — without rewriting their test suite.
CI/CD & Delivery · beginner · 12 min read
How a B2B SaaS Team Cut Failed Deployments by 80% with Canary Releases and SLO-Driven Rollback
A 120-engineer SaaS company replaced big-bang deploys with canary releases gated by error-budget burn, cutting user-facing incidents from 14 to 3 per quarter and mean time to recovery from 47 to 8 minutes.
CI/CD & Delivery · intermediate · 12 min read
How a B2B SaaS Team Cut LLM API Spend by 65% with Model Routing, Prompt Caching, and Request Batching
A 120-engineer SaaS company slashed monthly LLM costs from $41k to $14k by routing cheap models for simple tasks, caching repeated prompts in Redis, and batching async jobs — without degrading user-facing quality.
AI & MLOps · intermediate · 12 min read
How a B2B SaaS Team Cut LLM Inference Costs by 71% by Moving to Self-Hosted vLLM
A 120-engineer SaaS company replaced OpenAI API calls with self-hosted Mistral on vLLM, cutting per-query cost by 71% and p99 latency by 40% after a painful but instructive migration.
AI & MLOps · advanced · 12 min read
How a B2B SaaS Team Cut Model Deployment Time from 3 Days to 4 Hours with MLflow and Kubeflow
A 120-engineer SaaS company replaced ad-hoc notebook deployments with MLflow tracking and Kubeflow Pipelines, cutting deployment time by 93% and eliminating silent model regressions.
AI & MLOps · intermediate · 12 min read