Nginx Stale DNS: 7 Proven Steps to Fix Kubernetes 404 Errors

Nginx stale DNS is one of those production problems that looks like a mystery until you have seen it once. Traffic that worked an hour ago suddenly returns 404 or 502 errors, the backend looks healthy, and a direct request from the very same server works perfectly. I recently helped untangle exactly this on a Kubernetes cluster, and the immediate fix was a single reload. The lasting fix needs a small change to how Nginx finds its backend.

In this guide I’ll explain why Nginx stale DNS happens, how to prove it quickly without guessing, and how to fix it properly. Everything here comes from a real incident, with names, hostnames and identifiers removed.

What is Nginx stale DNS and why does it happen?

When you write proxy_pass https://api.example.com; with a plain hostname, Nginx looks up that name once, while it loads its configuration. It then keeps the resulting IP address for as long as those worker processes live. Cloud load balancers and API gateways with custom domains can hand out different IP addresses over time, so the address Nginx remembered can quietly stop being the right one. That is Nginx stale DNS in a nutshell. The Nginx proxy_pass documentation describes this resolve-at-load behaviour.

You might wonder why the symptom is often a 404 rather than a connection error. My working theory is that the old address is now answered by a different service in the provider’s shared pool, which replies with a polite “no such service”. I could not prove that in this case because the upstream address was never logged, and that gap is a lesson in itself.

Symptoms of Nginx stale DNS

The pattern is what gives it away. The backend is fine, the failure sits in the middle, and it often disappears after a reload. Use this table as a quick triage sheet.

Step 1 to 3: prove Nginx stale DNS before you change anything

Step 1: compare the proxy path with the direct path

Run the same request twice from inside the affected pod: once through local Nginx, and once straight to the backend hostname. If the direct call works and the proxied call fails, the problem is in the proxy layer, not in the backend.

# Through local Nginx (set the Host header the public site would use)
kubectl exec -n <namespace> <frontend-pod> -- \
  curl -sS -D - -H 'Host: <public-hostname>' 'http://127.0.0.1/api/health'

# Directly to the backend hostname
kubectl exec -n <namespace> <frontend-pod> -- \
  curl -sS -D - 'https://<backend-hostname>/api/health'

Step 2: see what DNS returns right now

Resolve the backend name from the same pod and test each address that comes back. In my case both resolved addresses worked, which pointed away from the backend and towards the address Nginx had remembered. Capture the current values so you can compare them later.

Step 3: find where the Nginx config really lives

Before editing anything, check whether the configuration is baked into the container image or mounted from a ConfigMap. If it is in the image, a change made inside a running container will vanish at the next restart. The permanent change must go through Git and your normal release pipeline.

kubectl get deployment <deployment> -n <namespace> -o yaml
kubectl exec -n <namespace> <frontend-pod> -- nginx -T

Step 4: recover with a validated Nginx reload

A reload makes Nginx read the configuration and resolve the hostname again, without dropping existing connections. Always test the configuration first, and do one pod at a time so you can confirm each one recovers.

kubectl exec -n <namespace> <pod> -- nginx -t
kubectl exec -n <namespace> <pod> -- nginx -s reload
kubectl exec -n <namespace> <pod> -- \
  curl -sS -o /dev/null -w 'HTTP Status: %{http_code}\n' \
  -H 'Host: <public-hostname>' 'http://127.0.0.1/api/health'

In the incident, the local request changed from 404 to a normal backend response straight after the reload, and the same result repeated on all four replicas. That before-and-after evidence is why I call Nginx stale DNS the leading cause. Treat this as a temporary mitigation only. It does nothing to stop the same thing happening next month.

Step 5: fix Nginx stale DNS permanently with a resolver

The durable fix is to make Nginx look the name up while it serves traffic. Putting the hostname in a variable forces a runtime lookup, and the resolver directive tells Nginx which DNS server to ask and how long to trust the answer.

location ^~ /api/ {
    resolver <cluster-dns-ip> valid=30s ipv6=off;
    resolver_timeout 5s;

    set $backend_host backend.example.com;
    proxy_pass https://$backend_host$request_uri;

    proxy_ssl_server_name on;
    proxy_ssl_name backend.example.com;
    proxy_ssl_verify on;
    proxy_ssl_trusted_certificate /etc/ssl/certs/ca-certificates.crt;

    proxy_set_header Host backend.example.com;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_http_version 1.1;
    proxy_set_header Connection "";

    proxy_connect_timeout 10s;
    proxy_read_timeout 60s;
}

A few details matter here, and they are easy to get wrong:

• Use your cluster DNS address. In Kubernetes this is the ClusterIP of the DNS service, which you can read with kubectl get svc kube-dns -n kube-system. The Kubernetes DNS documentation explains how pods resolve names.
• valid=30s is not a timer. Nginx caches the answer for about 30 seconds and looks it up again only when a request needs it after that. It does not reload every 30 seconds.
• Keep the full URI. Using $request_uri preserves both the path and the query string, which matters for callbacks that carry parameters.
• Keep TLS honest. Send the server name and verify the certificate, so a wrong address fails loudly instead of quietly. If certificate errors appear, my guide on fixing Certbot not renewing covers the common certificate causes.
• Timeouts stack. A 60-second Nginx read timeout cannot stretch a 30-second timeout on an API gateway in front of your backend.

Recent Nginx releases can also re-resolve upstream servers with the resolve parameter. Check the documentation for your version before relying on it, because older open source builds do not support it.

Step 6 and 7: canary rollout, monitoring and rollback

Roll the new configuration out to one canary pod first. Validate it with nginx -t, then run the local smoke test and confirm you get the expected backend response. Have the application team test a genuine end-to-end request before you promote the change to the remaining replicas. Never send invented payloads to a production endpoint just to see what happens.

Record the current image digest and configuration before the rollout, so a rollback is a simple redeploy of a known-good version. Do not use periodic reload cron jobs or random pod restarts as the lasting fix. They hide the design flaw instead of removing it.

Finally, make the next incident cheaper. Log $upstream_addr, upstream status and upstream response time in the access log, without any secrets. Alert on sustained 404, 502, 503 and 504 rates. These signals pair well with resilience patterns such as the circuit breaker pattern, and with basic pod troubleshooting like my guide on fixing Kubernetes OOMKilled errors.

Do not confuse Nginx stale DNS with other failures

A reload that fixes one symptom can make everyone relax too early. During this incident, three other things were happening in the same window, and none of them was explained by the DNS fix.

• A WAF block. One public GET request was rejected with a 406 page from the web application firewall. That proves a rule matched that request, not that every callback was blocked. Inspect the matched rule, and avoid broad bypasses.
• A gateway timeout. A real POST reached the API gateway and ended in an integration timeout after 30 seconds. Check historical load balancer and gateway logs for that request instead of assuming the reload cured it.
• Scheduled jobs. Background jobs were hitting rate limits and not-found responses from third-party APIs around the same time. Those are separate operations and do not explain the callback failure.

Keep each failure on its own timeline. Nginx stale DNS is the leading explanation for the proxy 404, and it should be written up as exactly that.

One more thing from the post-incident review: logs contained sensitive values such as signing keys, tokens and signed payloads. Stop logging those, restrict access to old logs, and rotate anything that may have been exposed through your normal security process.

Nginx stale DNS prevention checklist

1. Search every config for proxy_pass lines that use a literal hostname.
2. Switch cloud-hosted backends to a variable plus a resolver with a short valid time.
3. Point the resolver at your cluster DNS service.
4. Keep SNI and certificate verification switched on.
5. Log the upstream address, status and latency for every proxied request.
6. Alert on sustained 404 and 5xx rates, not only on pod restarts.
7. Test changes on a canary, record the image digest, and keep a rollback ready.

Final thoughts

Nginx stale DNS is cheap to prevent and expensive to debug at the wrong moment. The fix is small: a variable, a resolver and a sensible cache time. The real value is in the habits around it, such as comparing the proxy path with the direct path, logging the upstream address, and refusing to treat one recovered symptom as the end of the investigation.

Have you ever chased a ghost 404 that a reload quietly fixed? That is usually your cue to go and check the DNS.

Frequently asked questions

What causes Nginx stale DNS?
Nginx resolves a literal hostname in proxy_pass only when it loads its configuration. If the backend IP changes later, Nginx keeps using the old one until a reload.

Does nginx -s reload fix Nginx stale DNS permanently?
No. A reload refreshes the address once, so the problem can come back when the backend IP changes again. Use a variable with a resolver for a lasting fix.

Which DNS server should the Nginx resolver use in Kubernetes?
Use the ClusterIP of your cluster DNS service, usually found with kubectl get svc kube-dns -n kube-system, and keep a short valid time such as 30 seconds.