Data Validation with Great Expectations
Write Your Data Assumptions as Tests, Run Them on Every Batch, and Stop the Pipeline When They Break.
By Sandip Das · · 6 min read
💡
Hey — It's Sandip Das 👋
In Day 38 we served features from the Feast online store in milliseconds. But a feature store happily serves whatever data you feed it, good or bad.
Today we put a gate in front of the pipeline: write your assumptions about the iris data as tests with Great Expectations, run them on every batch, and stop training the moment they break.
In Day 38 we served features from the Feast online store in milliseconds. But a feature store happily serves whatever data you feed it, good or bad.
Today we put a gate in front of the pipeline: write your assumptions about the iris data as tests with Great Expectations, run them on every batch, and stop training the moment they break.
📚 Key Learnings
- Why data bugs are more dangerous than code bugs
- The GX vocabulary: Data Context, Expectations, Suites, Validation Definitions, Checkpoints
- Running a checkpoint and reading the Data Docs HTML report
- Failing DVC and CI pipelines automatically when data quality checks fail
- Hands-on: a full validation suite for iris-mlops, verified end to end
🧠 Learn here
Why validate data at all?
Code bugs crash lou
Read the full article on learnxops.com.
Topics: 100 Days of MLOps, MLOps, ML, GitOps