How a B2B SaaS Platform Upgraded 40 Kubernetes Clusters Across Three Versions Without an Outage
A 120-engineer SaaS company migrated 40 production Kubernetes clusters from 1.24 to 1.27 using a wave-based canary strategy, automated preflight checks, and zero customer-facing downtime.
Containers & K8s · advanced · 12 min read · 2516 words · By Sandip Das · Published
Composite scenario: built from common industry patterns; the figures are illustrative.
What this covers
- The Situation
- The Problem
- Constraints
- Options Considered
- The Solution
- Implementation
- Results
- What Went Wrong Along the Way
- Lessons Learned
- Apply It Yourself
Key numbers
| Metric | Before | After |
|---|---|---|
| Clusters on end-of-life Kubernetes (1.24) | 12 clusters | 0 clusters |
| Version spread across fleet | 3 versions (1.24–1.26) | 1 version (1.27) |
| Customer-facing downtime during upgrade | N/A (never attempted at scale) | 0 minutes |
| Engineer-hours for upgrade execution | ~200 hours (manual estimate) | 38 hours (actual) |
| Mean time to diagnose policy-drift incidents | 47 minutes | 8 minutes |
Key components
- Wave Orchestrator — Python script that groups clusters into upgrade waves and advances or blocks each wave based on Prometheus health signals.
- Preflight Job — CI step that validates PodDisruptionBudgets, deprecated API usage, and Argo CD sync status before any cluster is touched.
- Metric-gated Approval Gate — Decision point that queries live Prometheus metrics and blocks wave advancement if error rates or restart counts exceed thresholds.
- Argo CD Fleet Controller — Reads the cluster registry and identifies clusters whose desired Kubernetes version does not match their current running version.
- EKS Managed Node Groups — Provide rolling node replacement without requiring manual cordon-and-drain, enabling zero-downtime node upgrades.
- Terraform EKS Module — Single source of truth for cluster version and add-on versions; the wave orchestrator triggers applies against this module.
The Situation
This is a composite scenario assembled from upgrade patterns observed across multiple organisations; all figures are illustrative and chosen for internal consistency, not to represent any single company's experience.
A 120-engineer B2B SaaS company ran its multi-tenant analytics platform on 40 Amazon EKS clusters spread across four AWS regions. Clusters were segmented by customer tier: eight "platinum" clusters for enterprise customers with strict SLA commitments, twenty "standard" clusters for mid-market accounts, and twelve "dev/staging" clusters used by internal teams. The platform team — six engineers — owned cluster lifecycle, while roughly thirty application teams deployed workloads through Argo CD. The fleet had drifted: twelve clusters ran Kubernetes 1.24, nineteen ran 1.25, and nine ran 1.26. Kubernetes 1.24 reached end-of-life in July 2023, which meant no more CVE patches from upstream and a growing gap with the AWS-managed control plane support window.
The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.
Topics: kubernetes, containers, platform-engineering, upgrades, reliability, gitops
Related case studies
- How a B2B SaaS Platform Halved Kubernetes Compute Costs with Karpenter and Right-Sized Workloads
- How a B2B SaaS Team Migrated a Rails Monolith to EKS With Zero Downtime
- How a New Telemetry Service Took Down OpenAI's Kubernetes API Servers and DNS in December 2024
- How One Platform Team Tamed 30 Product Teams on a Single Kubernetes Cluster
- The Pi Day Outage: How a Renamed Kubernetes Node Label Took Reddit Down for Five Hours