How a Feature Store Eliminated Training-Serving Skew and Cut Model Errors by 34%
A B2B SaaS company's recommendation model silently degraded because training and serving computed features differently. Adopting Feast on Redis fixed the skew and cut prediction errors by 34%.
AI & MLOps · advanced · 12 min read · 2466 words · By Sandip Das · Published
Composite scenario: built from common industry patterns; the figures are illustrative.
What this covers
- The Situation
- The Problem
- Constraints
- Options Considered
- The Solution
- Implementation
- Results
- What Went Wrong Along the Way
- Lessons Learned
- Apply It Yourself
Key numbers
| Metric | Before | After |
|---|---|---|
| Seven-day activation lift vs. control | 1.1% | 6.8% |
| Model mean absolute error (held-out set) | 0.142 | 0.094 |
| P99 feature retrieval latency | 38ms | 6ms |
| Skew-related on-call incidents per week | 4.2 | 0.3 |
| Time to detect feature distribution shift | Days (manual) | 22 minutes (automated) |
Key components
- Feast Feature Store — Acts as the single source of truth for feature definitions, enforcing identical logic for both historical training retrieval and low-latency online serving.
- Point-in-time correct join — Ensures each training example uses only feature values that were available at the moment of that example's event timestamp, eliminating temporal leakage.
- ElastiCache Redis (Online Store) — Provides sub-10ms feature lookups on the critical serving path, materialised from the same pipeline that feeds the offline store.
- Shadow mode comparison — Runs both old and new feature computation paths in parallel before cutover, catching discrepancies without exposing users to risk.
- Evidently drift detector — Monitors Jensen-Shannon divergence of live feature distributions against the training baseline, alerting within minutes of distribution shift.
- MLflow Model Registry — Versions and stages trained models so the serving container always loads the artifact that corresponds to the Feast-generated training dataset.
The Situation
This is a composite scenario assembled from patterns observed across multiple production ML deployments; the organisation, team structure, and figures are illustrative but internally consistent.
The organisation is a 140-engineer B2B SaaS company selling a workflow-automation platform to mid-market and enterprise customers. Their data science team of eight engineers shipped a content-recommendation model in late 2022 that surfaces relevant workflow templates to users inside the product. The model runs on roughly 2.4 million prediction requests per day and directly influences a key activation metric: whether a new user completes their first automated workflow within seven days of signup.
The ML stack at the time of the incident was a mix of Spark on EMR for batch feature computation, a set of hand-rolled Python scripts that re-computed features at serving time, XGBoost models tracked in MLflow, and a FastAPI model-serving container deployed on ECS Fargate. There was no centralised feature store. Training data was produced by a scheduled Airflow DAG; serving features were recomputed live by a separate code path maintained by a different engineer.
The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.
Topics: mlops, feature-store, training-serving-skew, feast, redis, machine-learning
Related case studies
- Air Canada's Chatbot Hallucination: How a RAG Support Bot Created Legal Liability in Production
- How a B2B SaaS Team Cut LLM API Spend by 65% with Model Routing, Prompt Caching, and Request Batching
- How a B2B SaaS Team Cut LLM Inference Costs by 71% by Moving to Self-Hosted vLLM
- How a B2B SaaS Team Cut Model Deployment Time from 3 Days to 4 Hours with MLflow and Kubeflow
- How a B2B SaaS Team Cut Prompt Regression Escapes by 94% with an Automated LLM Eval Pipeline in CI
- How a B2B SaaS Team Took a RAG Support Assistant from Demo to Production