How a Fintech Team Caught Silent Fraud-Model Drift Before It Cost Them $2M

A fraud model degraded silently for six weeks before anyone noticed. The team built a drift-detection and auto-retraining pipeline that cut false-negative fraud losses by 61%.

AI & MLOps · intermediate · 12 min read · 2321 words · By Sandip Das · Published

Composite scenario: built from common industry patterns; the figures are illustrative.

What this covers

  1. The Situation
  2. The Problem
  3. Constraints
  4. Options Considered
  5. The Solution
  6. Implementation
  7. Results
  8. What Went Wrong Along the Way
  9. Lessons Learned
  10. Apply It Yourself

Key numbers

MetricBeforeAfter
False-negative fraud rate11.7%4.6%
Time to detect feature drift~6 weeks (manual)4 hours (automated)
Time to retrain and promote model3–4 weeks (ad hoc)6–9 hours (automated)
Estimated monthly fraud loss~$1.2M~$470K
Weekly chargeback volume vs baseline+34%+3%

Key components

  • Evidently AI drift reports — Computes PSI, KS-test, and JS divergence between reference and production feature windows every four hours without touching the inference hot path.
  • Label Joiner service — Matches delayed chargeback records to original feature vectors in S3, providing live ground-truth labels with explicit delay accounting.
  • Airflow drift-monitor DAG — Orchestrates the full detect-retrain-evaluate-promote loop, polling TimescaleDB for threshold breaches and launching SageMaker jobs on demand.
  • SageMaker Training with spot instances — Runs LightGBM retraining jobs on checkpointed spot instances, keeping per-run cost under $6 while supporting up to 37 triggered runs per month.
  • Champion/challenger traffic router — Splits live inference traffic 90/10 between the current champion and a new challenger, enabling safe shadow evaluation before full promotion.
  • MLflow Model Registry — Stores every trained artifact with metrics, parameters, and promotion status, giving the team a full audit trail required for model governance.
  • TimescaleDB metrics store — Persists time-series drift scores and model performance metrics, enabling threshold alerting and retrospective analysis of degradation patterns.

The Situation

This is a composite scenario built from patterns observed across multiple fintech and payments companies; all figures are illustrative and internally consistent, not drawn from any single organisation's incident report.

The organisation is a 160-engineer B2B payments platform that processes card-not-present transactions for roughly 4,000 small-to-medium merchants. The ML team is five people: two ML engineers, two data scientists, and one MLOps engineer who joined eight months before the incident. The fraud model — a LightGBM binary classifier — had been in production for fourteen months, trained once on a historical dataset and never retrained. It scored every transaction in under 40 ms via a FastAPI service backed by a Redis feature store. The model's features included merchant category codes, velocity counts over 1-, 7-, and 30-day windows, device fingerprint embeddings, and normalised transaction amounts.

At peak the platform handled 1,200 transactions per second. The fraud team reviewed flagged transactions manually; anything below a 0.35 score threshold passed automatically.

The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.

Topics: mlops, data-drift, fraud-detection, feature-store, model-monitoring, retraining

Related case studies

Browse all case studies