How a Shared Kubernetes Cluster Raised GPU Utilisation from 25% to 70%
A B2B SaaS company's ML platform team tripled effective GPU utilisation by combining time-slicing, bin-packing, and a fair-share queue — without buying a single new node.
AI & MLOps · advanced · 12 min read · 2205 words · By Sandip Das · Published
Composite scenario: built from common industry patterns; the figures are illustrative.
What this covers
- The Situation
- The Problem
- Constraints
- Options Considered
- The Solution
- Implementation
- Results
- What Went Wrong Along the Way
- Lessons Learned
- Apply It Yourself
Key numbers
| Metric | Before | After |
|---|---|---|
| Average GPU utilisation | 25% | 71% |
| Median queue wait time | 47 minutes | 11 minutes |
| Monthly GPU cluster cost | $28,400 | $19,100 |
| Jobs completed per day | 38 | 94 |
| Requested-to-used GPU ratio | 2.3x | 1.1x |
Key components
- NVIDIA Time-Slicing (device plugin) — Exposes each physical GPU as multiple virtual GPUs, allowing concurrent small jobs to share a single device.
- Volcano Batch Scheduler — Provides fair-share queues, gang scheduling, and bin-packing to replace the default spread-based Kubernetes scheduler for ML workloads.
- DCGM Exporter — Scrapes per-GPU utilisation, memory, and power metrics from NVIDIA drivers and exposes them to Prometheus.
- Argo Workflows — Orchestrates multi-step ML pipelines as DAGs, including the new CPU-only prefetch step that decouples data loading from GPU allocation.
- CUDA MPS Server — Enables multiple CUDA processes to share a single GPU context with lower overhead, used as an opt-in mode for trusted CI batch jobs.
- Volcano Queues with weights — Enforce fair-share allocation across teams, eliminating the incentive to over-request GPUs as a hedge.
The Situation
This is a composite scenario built from patterns observed across multiple real ML platform teams; all figures are illustrative and chosen for internal consistency, not to represent any single organisation.
The organisation is a 140-engineer B2B SaaS company whose product includes real-time recommendation and fraud-detection models. The ML platform team — four engineers — runs a shared Kubernetes cluster on AWS for all model training, batch inference, and experimentation. The cluster holds 12 p3.8xlarge nodes (four NVIDIA V100s each, 48 GPUs total), plus a CPU node pool for preprocessing and serving. About 30 data scientists and ML engineers submit jobs daily, ranging from 10-minute hyperparameter sweeps to 18-hour full-model retrains.
The stack before the project: plain Kubernetes with the NVIDIA device plugin, Argo Workflows for DAG orchestration, S3 for model artefacts, and an NFS-backed dataset cache. Monitoring was ad-hoc — a few hand-rolled Prometheus queries and a single Grafana dashboard nobody trusted.
The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.
Topics: kubernetes, gpu, mlops, scheduling, resource-management, nvidia
Related case studies
- Air Canada's Chatbot Hallucination: How a RAG Support Bot Created Legal Liability in Production
- How a B2B SaaS Team Cut LLM API Spend by 65% with Model Routing, Prompt Caching, and Request Batching
- How a B2B SaaS Team Cut LLM Inference Costs by 71% by Moving to Self-Hosted vLLM
- How a B2B SaaS Team Cut Model Deployment Time from 3 Days to 4 Hours with MLflow and Kubeflow
- How a B2B SaaS Team Cut Prompt Regression Escapes by 94% with an Automated LLM Eval Pipeline in CI
- How a B2B SaaS Team Took a RAG Support Assistant from Demo to Production