☸️ Kubernetes Is Not Healing Your Application. It's Just Restarting It. Here's What You Need to Know.

Your Pod crashed 15 times. Kubernetes restarted it 15 times. And your application is still broken.

Welcome to Kubernetes. 😂

Imagine deploying an application:

kubectl apply -f deployment.yaml

You check the Pods:

kubectl get pods

And see:

NAME READY STATUS RESTARTS payment-api-7d8f9c6b-x2k9 0/1 CrashLoopBackOff 15

You delete the Pod.

Kubernetes creates another one.

It crashes again.

You restart the Deployment.

It crashes again.

At this point, you're wondering:

"Is Kubernetes actually intelligent, or is it just restarting things forever?" 💀

Here's the important lesson:

Kubernetes is designed to maintain the desired state of your infrastructure. It cannot automatically understand or fix every application failure.

Let's explore how Kubernetes actually works, why applications fail, and how to troubleshoot them like a real DevOps engineer.

🧠 1. Kubernetes Doesn't Manage Your Application the Way You Think

One of the biggest misunderstandings among Kubernetes beginners is believing that Kubernetes automatically makes applications healthy.

That's not how it works.

Kubernetes continuously observes the state of your cluster and reconciles the actual state toward the desired state.

For example, suppose you deploy an application with three replicas.

spec: replicas: 3

You're telling Kubernetes:

"I want three instances of this application running."

Kubernetes attempts to maintain that requirement.

If one Pod disappears, its controller attempts to create a replacement.

 Desired State 3 Replicas | v Kubernetes Controller | v Actual Cluster State | +--------+--------+ | | | v v v Pod 1 Pod 2 Pod 3

This is called the reconciliation loop.

The controller repeatedly observes the current state, compares it with the desired state, and takes corrective action.

But remember:

If your application has a bug, Kubernetes may repeatedly recreate the broken workload.

It doesn't magically repair your source code.

🔥 2. Understand the Difference Between Pod and Container

This is an important Kubernetes concept.

A Pod is the smallest deployable unit in Kubernetes.

A Pod can contain one or more containers that share networking and can share storage volumes.

For example:

Kubernetes Cluster | v Pod | +---+---+ | | v v App Sidecar Container Container

If a container crashes, its restart behavior depends on the Pod's restart policy and workload controller.

If a Pod is deleted, a Deployment-managed ReplicaSet can create another Pod to maintain the desired replica count.

Important: A replacement Pod is not necessarily the same Pod. It receives a new identity and may be scheduled on a different node.

This distinction becomes very important when troubleshooting production workloads.

💣 3. Five Kubernetes Problems Every DevOps Engineer Should Understand

Let's look at common problems you'll encounter in real clusters.

Problem 1: CrashLoopBackOff

Your application repeatedly crashes.

kubectl get pods

Possible causes:

Application startup error.
Incorrect environment variables.
Database connection failure.
Incorrect startup command.
Application exiting unexpectedly.

Start troubleshooting:

kubectl describe pod <pod-name>

Read the previous container logs:

kubectl logs <pod-name> --previous

Don't randomly delete Pods before understanding the failure.

Problem 2: ImagePullBackOff

Kubernetes cannot successfully pull the requested container image.

Example:

NAME READY STATUS my-app 0/1 ImagePullBackOff

Possible causes:

Incorrect image name.
Nonexistent image tag.
Private registry authentication problem.
Registry connectivity issue.

Inspect the Pod:

kubectl describe pod <pod-name>

Check the image reference:

kubectl get pod <pod-name> \ -o jsonpath='{.spec.containers[*].image}'

Verify that the image exists and the cluster can access the registry.

Problem 3: Pending Pods

Your Pod is not getting scheduled.

NAME READY STATUS my-app 0/1 Pending

Possible causes:

Insufficient CPU or memory.
Node selectors don't match available nodes.
Untolerated taints.
PersistentVolumeClaim problems.

Investigate:

kubectl describe pod <pod-name> kubectl get nodes kubectl get pvc

The Events section of describe often provides useful clues.

Problem 4: OOMKilled

Your container has been terminated because of an out-of-memory condition.

Check:

kubectl describe pod <pod-name>

You might find:

Last State: Terminated Reason: OOMKilled

Inspect resource usage:

kubectl top pods

Review resource requests and limits:

resources: requests: memory: "128Mi" cpu: "100m" limits: memory: "256Mi" cpu: "500m"

These are illustrative values. Set limits according to actual application requirements and observed usage.

Problem 5: Service Not Working

Your Pods are running, but your application isn't accessible.

Check your Services:

kubectl get svc

Inspect the Service:

kubectl describe svc my-service

Check its EndpointSlices:

kubectl get endpointslices

Common problems include:

Incorrect Service selector.
Incorrect target port.
Pods not Ready.
Network policies blocking traffic.
Application listening on the wrong interface.

A running Pod does not automatically mean your Service can route traffic to it.

❤️ 4. Kubernetes Probes: The Difference Between Running and Ready

This is one of the most important topics in Kubernetes.

Imagine your application starts successfully, but needs 30 seconds to connect to its database and load configuration.

Should Kubernetes send traffic to it immediately?

Probably not.

That's why Kubernetes provides three types of probes.

Probe	Purpose
Startup Probe	Checks whether the application has started
Readiness Probe	Determines whether the Pod should receive traffic
Liveness Probe	Determines whether the container should be restarted

Let's configure them.

apiVersion: apps/v1 kind: Deployment metadata: name: payment-api spec: replicas: 3 selector: matchLabels: app: payment-api template: metadata: labels: app: payment-api spec: containers: - name: payment-api image: nginx:1.27 ports: - containerPort: 80 startupProbe: httpGet: path: / port: 80 periodSeconds: 5 failureThreshold: 30 readinessProbe: httpGet: path: / port: 80 periodSeconds: 10 livenessProbe: httpGet: path: / port: 80 periodSeconds: 20

This example uses Nginx and its root HTTP endpoint to demonstrate probe configuration.

For a real application, implement meaningful health endpoints and choose thresholds based on startup time and recovery behavior.

Here's the difference:

Application Starts | v Startup Probe | v Startup Complete? | v Readiness Probe | v Ready for Traffic? | v Receive Traffic | v Liveness Probe | v Still Healthy?

A badly configured liveness probe can actually make an application less reliable by repeatedly restarting a process that could otherwise recover.

Choose probe behavior carefully.

🚀 5. The Real Power of Kubernetes: Self-Healing

Let's create a Deployment with three replicas.

Save this as deployment.yaml:

apiVersion: apps/v1 kind: Deployment metadata: name: web-app spec: replicas: 3 selector: matchLabels: app: web-app template: metadata: labels: app: web-app spec: containers: - name: nginx image: nginx:1.27 ports: - containerPort: 80

Deploy it:

kubectl apply -f deployment.yaml

Check the Pods:

kubectl get pods

You should see three Pods.

Now delete one:

kubectl delete pod <one-pod-name>

Check again:

kubectl get pods

Kubernetes will create a replacement Pod to maintain the Deployment's desired replica count.

That's self-healing in action.

But remember:

Self-healing infrastructure is not the same as automatically fixing application bugs.

📈 6. Scaling: What Happens When Traffic Increases?

Imagine your application normally receives 100 requests per second.

Suddenly, traffic increases to 10,000 requests per second.

Your three replicas may no longer be sufficient.

Kubernetes provides Horizontal Pod Autoscaling (HPA).

Here's a basic example:

kubectl autoscale deployment web-app \ --cpu-percent=60 \ --min=3 \ --max=10

Check the autoscaler:

kubectl get hpa

This configures a target average CPU utilization of 60% relative to requested CPU, with a minimum of three and maximum of ten replicas.

For CPU-based HPA to work, you generally need:

Metrics Server or another compatible metrics API.
CPU resource requests on your containers.
A workload that can actually scale horizontally.

For production workloads, CPU alone may not be enough. You might also scale using memory, request rate, queue length, or custom metrics.

And remember: adding Pods won't automatically fix a slow database or an application with shared bottlenecks.

🛠️ 7. The Kubernetes Troubleshooting Commands You Should Memorize

Here is a practical command cheat sheet.

Cluster information

kubectl cluster-info kubectl get nodes -o wide kubectl get namespaces

Pod troubleshooting

kubectl get pods -A kubectl describe pod <pod-name> kubectl logs <pod-name> kubectl logs <pod-name> --previous

Deployment troubleshooting

kubectl get deployments kubectl describe deployment <deployment-name> kubectl rollout status deployment/<deployment-name> kubectl rollout history deployment/<deployment-name>

Service troubleshooting

kubectl get svc kubectl describe svc <service-name> kubectl get endpointslices

Resource monitoring

kubectl top nodes kubectl top pods

Events

kubectl get events --sort-by=.metadata.creationTimestamp

Save these commands. You'll use them frequently when working with Kubernetes.

🧪 8. Build a Real Kubernetes Project for Your GitHub

Reading Kubernetes tutorials is useful.

But building and troubleshooting your own cluster is where the real learning happens.

Here's a project idea:

Project: Kubernetes Application Reliability Lab

Build a small application and deliberately simulate common infrastructure failures.

Technology stack:

Docker
Kubernetes / Minikube
Nginx
Prometheus
Grafana
YAML manifests
GitHub Actions

Project features:

Deploy an application with three replicas.
Configure startup, readiness, and liveness probes.
Configure CPU and memory requests and limits.
Expose the application using a Service.
Configure Horizontal Pod Autoscaling.
Simulate a failing container.
Simulate an unavailable dependency.
Investigate failures using logs and events.
Document root causes and solutions.

Architecture:

 Users | v Kubernetes Service | +-------+-------+ | | | v v v Pod 1 Pod 2 Pod 3 | | | +-------+-------+ | v Application | v Prometheus | v Grafana

What should you upload to GitHub?

kubernetes-reliability-lab/ │ ├── manifests/ │ ├── deployment.yaml │ ├── service.yaml │ └── hpa.yaml │ ├── monitoring/ │ └── prometheus.yaml │ ├── docs/ │ ├── crashloopbackoff.md │ ├── imagepullbackoff.md │ └── oomkilled.md │ └── README.md

Include screenshots, commands, failure scenarios, and troubleshooting explanations.

This project can help demonstrate practical Kubernetes knowledge during interviews.

🎯 Final Thoughts

Kubernetes is powerful, but it is not magic.

It can maintain replica counts, schedule workloads, restart containers, and reconcile desired state.

But it cannot automatically understand every application failure or make every architectural decision for you.

A strong Kubernetes engineer understands:

How Pods and Deployments work.
How Services route traffic.
How probes affect application availability.
How resource requests and limits work.
How autoscaling operates.
How to investigate failures using logs, events, and metrics.

Don't just memorize kubectl commands.

Understand what Kubernetes is doing behind those commands.

The difference between someone who knows Kubernetes and someone who can operate Kubernetes is the ability to troubleshoot when everything goes wrong.