How a B2B SaaS Team Cut Their Observability Bill by 60% with OpenTelemetry Sampling and Log Tiering
A 120-engineer SaaS company slashed $28k/month in observability costs by combining head-based OTel sampling, tail-based error retention, and S3 log tiering—without losing production visibility.
Observability & SRE · intermediate · 12 min read · 2181 words · By Sandip Das · Published
Composite scenario: built from common industry patterns; the figures are illustrative.
What this covers
- The Situation
- The Problem
- Constraints
- Options Considered
- The Solution
- Implementation
- Results
- What Went Wrong Along the Way
- Lessons Learned
- Apply It Yourself
Key numbers
| Metric | Before | After |
|---|---|---|
| Monthly observability spend | $47,200 | $18,800 |
| Daily ingested trace spans | 4.2 billion | 310 million |
| Daily log volume to hot storage | 1.1 TB | 38 GB |
| P1 mean time to detect | 4.1 min | 3.8 min |
| Weekly alert noise (false positives) | 47 | 9 |
Key components
- OTel Collector tail sampling processor — Makes keep/drop decisions on complete traces after all spans arrive, ensuring 100% retention of error and slow traces while dropping healthy ones.
- Vector log router — Classifies log streams by severity and trace correlation, routing hot logs to Loki and cold logs to S3 in Parquet format.
- Grafana Tempo — Stores sampled distributed traces on Grafana Cloud's pay-as-you-go tier, replacing Datadog APM.
- Grafana Loki — Stores hot-path logs (ERROR, WARN, and trace-correlated) with 30-day retention for incident investigation.
- Amazon S3 plus Athena — Provides low-cost cold storage for DEBUG and INFO logs with on-demand SQL querying for deep-dive investigations.
- OTLP (OpenTelemetry Protocol) — Vendor-neutral wire protocol that decoupled application instrumentation from the backend, enabling backend replacement without touching service code.
The Situation
This is a composite scenario assembled from patterns observed across multiple engineering organisations; all figures are illustrative and chosen for internal consistency, not to represent any single company.
The organisation is a 120-engineer B2B SaaS company selling a workflow-automation platform to mid-market enterprises. Their backend runs roughly 40 microservices on Kubernetes (EKS, three regions), written primarily in Go and Python. The platform team—six engineers—owns the internal developer platform, including the observability stack. In early 2023 the stack was a fairly standard managed trio: Datadog APM for traces, Datadog Log Management for logs, and Datadog Metrics for infrastructure. Monthly spend had climbed to approximately $47,000, with 68% of that attributable to log ingestion and trace storage.
The platform team had no dedicated budget for tooling replacement, but they did have a quarterly cost-reduction OKR: reduce observability spend by at least 40% without degrading mean-time-to-detect (MTTD) or mean-time-to-resolve (MTTR) for P1 incidents.
The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.
Topics: observability, opentelemetry, cost-optimization, logging, tracing, platform-engineering