Chasing 5-Second DNS Timeouts in Kubernetes: ndots, Conntrack Races, and NodeLocal DNSCache
Intermittent 5-second API latency spikes traced to Kubernetes DNS conntrack race conditions and ndots misconfiguration, resolved with NodeLocal DNSCache and search-domain tuning.
Networking & APIs · advanced · 12 min read · 2556 words · By Sandip Das · Published
Composite scenario: built from common industry patterns; the figures are illustrative.
What this covers
- The Situation
- The Problem
- Constraints
- Options Considered
- The Solution
- Implementation
- Results
- What Went Wrong Along the Way
- Lessons Learned
- Apply It Yourself
Key numbers
| Metric | Before | After |
|---|---|---|
| API p99 latency (spike peak) | 5,042 ms | 44 ms |
| DNS SERVFAIL rate (per node, peak) | 18/min | 0/min |
| DNS p99 resolution time (external names) | 48 ms | 0.9 ms |
| Latency spike incidents per week | 3.1 avg | 0 |
| SLA credit events per month | 2 | 0 |
Key components
- NodeLocal DNSCache — DaemonSet that intercepts DNS queries at the node level, serving cached responses and forwarding cache misses to CoreDNS over TCP to eliminate UDP conntrack races.
- ndots setting — resolv.conf option controlling how many dots a hostname must contain before it is queried as-is; reducing from 5 to 2 cuts unnecessary search-domain expansions and concurrent UDP packets.
- Linux conntrack table — Kernel connection-tracking subsystem whose insert_failed counter is the definitive diagnostic signal for the UDP race condition underlying DNS timeouts.
- CoreDNS — Cluster DNS server that receives forwarded queries from NodeLocal DNSCache on cache misses, now reached via TCP rather than UDP.
- Mutating admission webhook — Kubernetes webhook that patches dnsConfig on every new pod to enforce the ndots:2 and single-request-reopen settings without application code changes.
- Jaeger distributed tracing — Provided the critical trace span showing getaddrinfo consuming 5,000 ms, pinpointing DNS as the root cause faster than metrics alone.
The Situation
This is a composite scenario assembled from patterns observed across multiple production Kubernetes environments; all organisation details and figures are illustrative.
The organisation is a 120-engineer B2B SaaS company running a multi-tenant workflow-automation platform. Their backend is a set of roughly 40 microservices deployed on three EKS clusters (us-east-1, eu-west-1, ap-southeast-1), each with 60–90 m5.2xlarge nodes. Services are written primarily in Go and Python, fronted by an Envoy-based service mesh (Istio 1.17), and backed by RDS PostgreSQL, ElastiCache Redis, and a handful of third-party APIs including Stripe. Traffic peaks at around 18,000 requests per second across the fleet.
The platform team of eight engineers owned cluster infrastructure, networking, and the observability stack (Prometheus, Grafana, Jaeger). Application teams owned their own services and were largely self-serve on deployment. Kubernetes version was 1.27 at the time of the investigation.
The rest of this case study — the trade-offs, the scaling numbers and the build prompt — is available to LearnXops members. Create an account or see membership.
Topics: kubernetes, networking, dns, observability, reliability, linux
Related case studies
- How a Config Push and a Maintenance Tool Severed Google Cloud Networking in June 2019
- Slack's January 2021 Outage: How AWS Transit Gateway Limits Took Down the First Workday of the Year
- The $30,000 NAT Gateway Bill: Finding the Traffic and Fixing It with VPC Endpoints
- The Cloudflare June 2022 Outage: How One BGP Change Took 19 Data Centres Offline
- The December 2021 AWS us-east-1 Outage: How an Automated Scaling Event Congested the Internal Network
- The Facebook October 2021 Outage: How One BGP Withdrawal Took Down 3.5 Billion Users