ML Model inference Scaling on Kubernetes Using Horizontal Pod Autoscaling and Rolling Updates
Make ML Models running on Kubernetes survive a traffic spike, and ship a new model version without a single failed prediction
By Sandip Das · · 21 min read
💡
Hey — It's Sandip Das 👋
We use Kubernetes autoscaling and rolling updates to handle sudden inference traffic without downtime or failed predictions.
These techniques ensure ML services remain responsive, cost-efficient, and safely upgradable as demand and model versions change
We use Kubernetes autoscaling and rolling updates to handle sudden inference traffic without downtime or failed predictions.
These techniques ensure ML services remain responsive, cost-efficient, and safely upgradable as demand and model versions change
📚 Key Learnings
- Where we are and why this matters
- The three layers of Kubernetes scaling
- How the HPA control loop actually works
- Installing metrics-server
- The HPA algorithm, with worked numbers
- Writing the HPA manifest for iris-mlops
- Inspecting and debugging an HPA
- Tuning behavior: stabilization and policies
- Beyond CPU: custom and external metrics
- Rolling updates: the strategy field
- maxSurge and maxUnavailable, with math
- kubectl rollout: status, history, undo, pause
- PodDisruptionBudget and HPA together
- Troubleshooting table
🧠 Learn here
Read the full article on learnxops.com.
Topics: 100 Days of MLOps, MLOps, ML, Community, Kubernetes