How a B2B SaaS Company Turned a Swampy S3 Data Lake into a Lakehouse with Apache Iceberg
A 120-engineer SaaS company replaced a brittle Hive-metastore S3 lake with Apache Iceberg, cutting query costs 68% and eliminating overnight partition repair jobs that blocked analysts for hours.
Data & Storage · advanced · 12 min read · 2449 words · By Sandip Das · Published
What this covers
- The Situation
- The Problem
- Constraints
- Options Considered
- The Solution
- Implementation
- Results
- What Went Wrong Along the Way
- Lessons Learned
- Apply It Yourself
Key numbers
| Metric | Before | After |
|---|---|---|
| Median analyst query latency | 97 seconds | 11 seconds |
| Monthly Athena cost | $38,000 | $12,200 |
| Average data scanned per query | 340 GB | 4.8 GB |
| Nightly partition repair job duration | 3.5–5 hours | Eliminated |
| Schema-mismatch incidents per quarter | 11 | 0 |
Key components
- Apache Iceberg — Provides ACID transactions, hidden partitioning, and file-level statistics that enable predicate pushdown and eliminate partition repair jobs on S3.
- AWS Glue Catalog (Iceberg REST) — Acts as the Iceberg catalog, storing table metadata and snapshot history without requiring a self-managed Hive metastore.
- Amazon Athena v3 — Query engine with native Iceberg support, used by analysts and dbt Core to query and transform lakehouse tables.
- EMR Spark with MERGE INTO — Handles incremental upserts for late-arriving data, replacing the nightly full-partition rewrite job.
- AWS Lake Formation — Enforces column-level masking on PII fields to satisfy SOC 2 compliance requirements.
- Airflow (MWAA) — Orchestrates compaction, dbt runs, and Iceberg maintenance procedures such as snapshot expiry and manifest rewriting.
The Situation
The subject is a 120-engineer B2B SaaS company that sells workflow-automation software to mid-market enterprises. The data platform team — four data engineers, two analytics engineers, and a staff engineer acting as tech lead — owned a data lake that had grown organically for four years. The stack was: Kafka for event streaming, custom Python Flink jobs writing Parquet to S3, a Hive metastore running on a single EC2 instance (r5.2xlarge), AWS Glue crawlers for schema discovery, and Amazon Athena as the primary query engine. dbt Core ran SQL transformations on top of Athena. About 140 TB of Parquet lived in S3 across roughly 2,200 tables. Daily ingest was 800 GB to 1.2 TB depending on customer activity.
Analysts numbered around 35, split between product analytics, customer success, and finance. They queried through Tableau and a thin internal Python API that wrapped Athena.
The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.
Topics: data engineering, apache iceberg, s3, lakehouse, aws, athena
Related case studies
- How a B2B SaaS Team Cut Postgres P99 Latency by 73% with Pooling, Replicas and Partitioning
- How a B2B SaaS Team Migrated 8 TB from Self-Managed MySQL to Aurora with Zero Downtime
- How Discord Migrated Trillions of Messages from Cassandra to ScyllaDB
- How Figma Scaled Postgres from One Overloaded Database to Vertical Partitioning and Horizontal Sharding
- The 2017 GitLab Database Deletion: When Five Backup Methods All Failed at Once
- The GitHub October 2018 Outage: How 43 Seconds of Network Partition Caused 24 Hours of Data Inconsistency