The 2-Million-Message SQS Backlog: Beating Poison Messages with DLQs and Visibility Timeouts
A B2B SaaS team watched 2 million SQS messages pile up overnight because three malformed events kept recycling. Fixing visibility timeouts and wiring a dead-letter queue cleared the backlog in four hours.
Messaging & Events · beginner · 12 min read · 2213 words · By Sandip Das · Published
Composite scenario: built from common industry patterns; the figures are illustrative.
What this covers
- The Situation
- The Problem
- Constraints
- Options Considered
- The Solution
- Implementation
- Results
- What Went Wrong Along the Way
- Lessons Learned
- Apply It Yourself
Key numbers
| Metric | Before | After |
|---|---|---|
| SQS queue depth (steady state) | 2,100,000 messages | under 500 messages |
| Order confirmation latency p99 | 4.2 hours | 9 seconds |
| Worker CPU utilisation | 94% across all tasks | 18% across all tasks |
| Poison message time-to-isolation | indefinite (no DLQ) | under 3 minutes |
| DLQ alarm mean time to detect | N/A (no alarm) | 60 seconds |
Key components
- SQS Dead-Letter Queue — Captures messages that exceed maxReceiveCount, stopping poison messages from cycling indefinitely in the main queue.
- Visibility Timeout (300 s) — Gives workers enough time to complete processing without SQS re-delivering the message to a second worker.
- maxReceiveCount: 3 — Defines how many failed receive attempts trigger automatic routing to the DLQ, bounding the blast radius of any single bad message.
- ChangeMessageVisibility to zero — Allows a worker to fast-fail a known-bad message, burning through maxReceiveCount in seconds rather than waiting for the full visibility timeout.
- DLQ Inspector Lambda — Archives raw DLQ payloads to encrypted S3 and provides a human-gated replay path after the underlying bug is fixed.
- CloudWatch DLQ depth alarm — Alerts on-call engineers within 60 seconds of the first poison message arriving in the DLQ, replacing the previous hours-long detection gap.
The Situation
This is a composite scenario built from patterns the authors have observed across multiple production SQS deployments; the organisation, team structure and figures are illustrative but internally consistent.
The company is a 120-engineer B2B SaaS business that sells order-management software to mid-market retailers. Their platform processes purchase orders from roughly 400 retail clients, each of which can fire bursts of events whenever a warehouse system syncs. The backend runs on AWS: API Gateway fronts an Ingest Service (a Python Flask app on ECS Fargate), which drops every incoming order event onto a single standard SQS queue. A fleet of five Order Worker containers (also Fargate) polls that queue, validates the payload, and writes to an Aurora PostgreSQL database. At peak the queue handles about 80,000 messages per hour without complaint. There is no dead-letter queue. There is no CloudWatch alarm on queue depth. The visibility timeout is set to the SQS default of 30 seconds.
The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.
Topics: sqs, messaging, aws, dead-letter-queue, reliability, beginner