How a B2B SaaS Platform Cut Observability Costs 41% by Migrating 150 Services to OpenTelemetry
A 120-engineer SaaS company replaced three competing vendor agents with OpenTelemetry collectors across 150 microservices, cutting observability spend by 41% and mean time to detect by 58%.
Observability & SRE · advanced · 12 min read · 2496 words · By Sandip Das · Published
Composite scenario: built from common industry patterns; the figures are illustrative.
What this covers
- The Situation
- The Problem
- Constraints
- Options Considered
- The Solution
- Implementation
- Results
- What Went Wrong Along the Way
- Lessons Learned
- Apply It Yourself
Key numbers
| Metric | Before | After |
|---|---|---|
| Monthly observability spend | $47,000 | $7,100 |
| Mean time to detect (P1 incidents) | 14 minutes | 5.9 minutes |
| Services with distributed tracing | 112 of 150 (75%) | 150 of 150 (100%) |
| Trace coverage of error requests | ~10% (head sampling) | ~98% (tail sampling) |
| Vendor observability agents per node | 3 DaemonSets | 1 DaemonSet |
Key components
- OpenTelemetry Collector (gateway + DaemonSet) — Central signal aggregation, processing, and fan-out layer that decouples instrumentation from backend choice.
- Tail-based sampling processor — Holds spans in memory for 30 seconds and keeps traces based on error status or latency, replacing lossy head-based sampling.
- OTel Operator for Kubernetes — Injects auto-instrumentation agents into pods via annotations, enabling zero-code instrumentation for Java and Python services.
- Grafana Tempo — Self-hosted distributed trace backend with S3-backed block storage, replacing Honeycomb for trace storage and search.
- Prometheus + Alertmanager — Metrics storage and alerting layer that replaced Datadog monitors, integrating with the existing PagerDuty routing.
- Grafana Loki — Self-hosted log aggregation backend replacing Sumo Logic, with structured log correlation to traces via Grafana.
- Redaction processor — Scrubs PAN and IBAN patterns from log bodies in the collector pipeline before logs reach any storage backend.
The Situation
This is a composite scenario assembled from patterns observed across multiple real-world observability migrations; all figures are illustrative and chosen for internal consistency, not to represent any single company.
The organisation is a 120-engineer B2B SaaS company running a supply-chain visibility platform. Their product exposes REST and gRPC APIs consumed by logistics operators across Europe and North America. The platform runs 150 microservices on Kubernetes (EKS, three clusters across two AWS regions), processes roughly 18,000 requests per second at peak, and stores about 4 TB of time-series data per month. The engineering team is split into 14 product squads, a platform engineering team of 9, and a dedicated SRE team of 5.
Over four years of growth the company had accumulated three separate observability vendors: a legacy APM agent (Datadog) rolled out in the early monolith days, a distributed tracing tool (Honeycomb) adopted by two squads who needed high-cardinality queries, and a log aggregation SaaS (Sumo Logic) mandated by the security team for compliance reasons. Each vendor had its own SDK, its own agent DaemonSet, and its own billing model.
The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.
Topics: opentelemetry, observability, microservices, otel, tracing, cost-optimisation