How a B2B SaaS Team Eliminated Duplicate Order Events by Adopting the Transactional Outbox Pattern
A dual-write bug was silently duplicating and dropping order events in production. The team replaced it with a transactional outbox and a Debezium CDC relay, cutting event loss to zero.
Messaging & Events · intermediate · 12 min read · 2541 words · By Sandip Das · Published
What this covers
- The Situation
- The Problem
- Constraints
- Options Considered
- The Solution
- Implementation
- Results
- What Went Wrong Along the Way
- Lessons Learned
- Apply It Yourself
Key numbers
| Metric | Before | After |
|---|---|---|
| Lost events per day | ~310 (0.40% of orders) | 0 (0.00%) |
| Duplicate events per day | ~115 (0.15% of orders) | < 2 (< 0.003%) |
| Event delivery fidelity | 99.45% | 99.997% |
| Mean time to detect event loss | 4–6 hours (manual) | < 3 minutes (DLQ alert) |
| Order Service p99 write latency | 18 ms | 21 ms |
Key components
- Transactional Outbox Table — Stores event payloads in the same PostgreSQL transaction as the business record, making the DB commit the single unit of atomicity.
- Debezium PostgreSQL Connector — Tails the PostgreSQL WAL via logical replication and relays committed outbox rows to Kafka without any application-layer polling.
- EventRouter SMT — Debezium Single Message Transform that extracts routing metadata from outbox rows and directs events to the correct Kafka topic.
- Schema Registry — Enforces Avro schema compatibility on every event before it is written to Kafka, preventing malformed messages from reaching consumers.
- Dead-Letter Queue Topic — Captures events that consumers cannot process after retries, providing a recoverable buffer and a real-time alerting surface.
- Consumer Idempotency Store — Redis set keyed by event ID that prevents downstream services from acting on duplicate deliveries caused by consumer restarts.
The Situation
The organisation is a 120-engineer B2B SaaS company that operates a multi-tenant order management platform used by mid-market retailers. Their backend runs on AWS: roughly 40 microservices written in Go and Python, a PostgreSQL 15 cluster on RDS, and a self-managed Kafka 3.5 cluster on EC2 (three brokers, replication factor 3). Order volume averages 18,000 orders per hour on weekdays, spiking to 55,000 per hour during promotional events. The team responsible for the order pipeline is eight engineers — four backend, two platform, one data, one SRE.
The platform had been growing quickly. Eighteen months before the incident, the team had split a monolith into services and wired them together with Kafka. The integration was done under deadline pressure, and the pattern chosen for publishing events was a straightforward dual-write: the Order Service would INSERT a row into PostgreSQL and then immediately call the Kafka producer SDK to publish an order.created event. It worked fine in staging. It worked fine in early production. Then it didn't.
The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.
Topics: messaging, postgresql, kafka, cdc, outbox-pattern, reliability