Kubernetes OOMKilled: 7 Easy Steps to Fix Exit Code 137

A pod restarts again and again, and the status says OOMKilled. Nothing in the logs looks wrong. If you have met Kubernetes OOMKilled in production, you know how confusing it feels at first.

The good news is that this error is usually simple. A container used more memory than it was allowed. This guide shows how to confirm that, find the cause, and fix it calmly, one step at a time.

What Is Kubernetes OOMKilled?

Kubernetes OOMKilled means a container was stopped because it went over its memory limit. OOM stands for “out of memory”. The container exits with code 137, which is 128 plus signal 9 (SIGKILL). The process gets no chance to shut down cleanly.

Containers share the host kernel, so the kernel enforces the limit. If you want a refresher on that model, see Docker vs Virtual Machines. Kubernetes then restarts the container based on the pod’s restart policy. The Kubernetes resource management guide explains that the scheduler uses requests to place a pod, while the kubelet enforces limits.

In the Kubernetes documentation, a container that keeps allocating past its limit is terminated and shows OOMKilled, with exit code 137 in its last state. The restart count then climbs.

Why does it happen?

Most cases fall into three groups. The limit may simply be too low for the real workload. The application may have a memory leak, so usage creeps up until the limit is hit. Or a sudden burst of traffic or a large request may push memory up for a short time.

Runtimes add a fourth cause. A Java or Node.js process may be configured with a heap larger than the container limit. It looks fine at first and fails later, under load.

How to Fix Kubernetes OOMKilled: 7 Steps

Replace the values in angle brackets with your own pod and namespace names.

1. Confirm the reason. Describe the pod and look at the last state of the container.

   kubectl describe pod <pod-name> -n <namespace>
   # Look for:
   #   Last State: Terminated
   #   Reason:     OOMKilled
   #   Exit Code:  137

2. Read the reason in one line. This is handy in scripts and quick checks.

   kubectl get pod <pod-name> -n <namespace> \
     -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'

3. Check real memory use. Compare actual usage with the limit. This needs metrics-server. For trends over time, a dashboard helps, as in this Kubernetes observability guide.

   kubectl top pod <pod-name> -n <namespace> --containers

4. Look at the current requests and limits. Many OOM kills come from a limit that was never sized.

   kubectl get pod <pod-name> -n <namespace> \
     -o jsonpath='{.spec.containers[*].resources}'

5. Decide: leak, spike or low limit. Memory that climbs steadily points to a leak. A sudden jump points to a spike in load. A flat line close to the limit means the limit is too low. Only raise the limit when the usage is legitimate.
6. Raise the limit, or fix the app. If usage is genuine, set a request that matches normal use and a limit with some headroom.

   resources:
     requests:
       memory: "256Mi"
     limits:
       memory: "512Mi"

   Applications with their own heap, such as Java, should also be told to stay inside the limit. For the JVM, a setting like -XX:MaxRAMPercentage=75.0 lets the heap follow the container limit instead of the node’s memory.

7. Verify and keep watching. Roll out the change and watch the restart count.

   kubectl get pods -n <namespace> -w

   Then add an alert on memory use near the limit. If load varies, Kubernetes autoscaling can add replicas before any single pod runs hot.

A safe way to see it first

The Kubernetes tutorial uses a pod with a 100 MiB limit that tries to allocate 250 MiB. The pod ends up OOMKilled and cycles between OOMKilled and Running while the restart count rises. Trying this in a test namespace is a gentle way to learn the pattern before you meet it in production.

Common Mistakes

• Assuming exit code 137 always means memory. Any SIGKILL gives the same code. Always check that the reason says OOMKilled.
• Raising limits blindly. A bigger limit hides a leak and wastes money. Measure first.
• Setting no requests. Pods are then placed without regard to memory, and nodes become crowded.
• Ignoring the runtime heap. A heap larger than the container limit will always be killed under load.
• Treating the restarts as the problem. Repeated OOM kills often end in CrashLoopBackOff. Fix the memory cause, not just the symptom.

None of these mistakes are unusual. Most teams make at least one of them while they learn. What matters is that you notice early and measure before changing anything.

FAQ: Kubernetes OOMKilled

What does Kubernetes OOMKilled mean?

It means the container exceeded its memory limit and was terminated with exit code 137.

Can OOMKilled happen when the node has free memory?

Yes. The limit applies to the container, so a pod can be killed even if the node has memory to spare.

Is exit code 137 always an OOM kill?

No. It only means SIGKILL was sent. Confirm the reason field says OOMKilled before changing memory settings.

How much memory should I set as the limit?

There is no fixed number. Watch real usage over a normal week, note the peaks, and set the limit a little above them. Review it when the application changes.

Should I just remove the memory limit?

Usually not. Without a limit, one pod can starve others on the same node. Size the limit from measured usage.

Where do I start if the pod is also failing to pull its image?

Fix that first. This ImagePullBackOff guide walks through it.

Final Thoughts on Kubernetes OOMKilled

Kubernetes OOMKilled looks dramatic, but the path is short: confirm, measure, decide, then adjust. Take the time to measure before you change a number. Your future self, on a quiet on-call night, will be grateful.

Have you traced a stubborn OOMKilled to an unexpected cause? Share your story in the comments.

About the author: Nandan Sinha writes calm, practical guides on technology, wellbeing and spirituality at nandansinha.com.