Originally published on kuryzhev.cloud

The scenario

A deploy finishes and the checkout API starts returning 5xx. The on-call engineer types kubectl get pods -n shop, then kubectl describe, then kubectl logs --previous, each with a pod name copied by hand. Every command is fine. The copy-paste between them is where the minutes go. k9s incident debugging removes that loop. You get one terminal view where you filter, drill into the failing pod, read logs and events, and open a shell without retyping names.

Short answer: start k9s read-only against the right context and namespace, then filter to unhealthy pods with /!Running. Read previous logs with p on the pod. Finally, add two small plugins (pod events and an ephemeral debug container) in plugins.yaml. This applies when you debug interactively on a cluster you already have access to. It does not replace dashboards, alerts or an audit trail.

Shortcuts below are the documented defaults of recent k9s releases. Key bindings and config keys have shifted between versions, so press ? inside k9s and check the k9s documentation for your installed version.

Prerequisites

You need k9s and kubectl installed. For example, use brew install k9s on macOS and Linuxbrew, or the release binaries from the project's GitHub page. Run k9s version and note the result, because plugin fields vary slightly by version. The cluster can be any maintained Kubernetes release, including 1.37, since ephemeral containers have been stable for a long time.

RBAC decides what k9s can actually do. Reading pods, logs and events needs get/list/watch on pods, pods/log and events. The debug plugin needs two more permissions:

  • patch on pods/ephemeralcontainers, because kubectl debug adds the container with a patch.
  • create on pods/attach, so it can attach interactively.

Check them before an incident, not during one:

# all should print "yes" for the namespace you debug in kubectl auth can-i get pods/log -n shop kubectl auth can-i patch pods/ephemeralcontainers -n shop kubectl auth can-i create pods/attach -n shop 

Watch out for: k9s uses whatever your kubeconfig says. If your shell's current context is production, k9s opens production. Always pass --context explicitly during an incident.

Step 1: Launch read-only and triage with filters

Open straight into the pods view of the affected namespace. Use read-only mode so a stray ctrl-d cannot delete anything:

# --readonly disables edit/delete/shell actions; -c picks the starting resource view k9s --readonly --context prod-eu -n shop -c pods 

Inside the view, these keys do most of the triage work:

  • / starts a filter. /!Running is an inverse regex filter that hides healthy pods. That leaves the CrashLoopBackOff, Error and Pending rows.
  • /-l app=checkout filters by label. /-f is a fuzzy match.
  • : opens command mode. :deploy, :events, :node, :ns and :ctx switch resource types, namespaces and contexts.
  • :xray deploy shows a deployment's tree down to ReplicaSets and pods. :pulse gives a cluster-wide health overview.
  • Column sort shortcuts, for example by CPU or memory, are listed in the key hints. ctrl-w toggles wide mode, which adds extra columns such as readiness gates and nominated node.

Press d for describe on a selected pod. The Events section at the bottom usually names the cause: a failed image pull, a failing probe, or OOMKilled in the last state. Press y for the YAML if you need the exact probe or resource limits.

Step 2: Read logs for the failing container

Press l on a pod to open its logs. In a multi-container pod, press Enter on the pod to list its containers, then press l on the one you want.

A crash-looping container is the typical case where the current log is empty or only a few seconds old. Press p on the pod or container row instead of l. That opens the previous container instance's logs, which show the output from just before the crash.

Other log-view keys worth memorizing:

  • The number keys change the time window. In current releases, 0 is tail, 1 is head, and 2 to 6 cover 1m to 1h. Confirm the mapping in the hint bar.
  • t toggles timestamps and w toggles line wrap.
  • s toggles autoscroll and f goes fullscreen.
  • / filters log lines with a regex such as /error|panic|timeout.

The default log buffer can be too small for chatty services. Tune it in the k9s config.yaml. The path is often ~/.config/k9s/config.yaml; run k9s info to see the exact path on your machine:

k9s: logger: tail: 200 # lines loaded when the view opens buffer: 5000 # lines kept in memory sinceSeconds: 300 # -1 means "all available"; 300 keeps the first view fast textWrap: false showTime: true 

Watch out for: a larger buffer and a long time window mean more memory and more API server log traffic per open view. Keep sinceSeconds small and widen it with the number keys only when needed.

Step 3: Add plugins for events and a debug shell

Plugins bind a key to a shell command. They expose context as variables such as $CONTEXT, $NAMESPACE, $NAME, $POD and $KUBECONFIG. In the containers scope, $NAME is the container and $POD is its pod. Plugins live in plugins.yaml in the same config directory. This file adds a per-pod event list and an ephemeral debug container:

plugins: pod-events: shortCut: Shift-E # check for clashes with built-in keys via "?" description: Events for pod scopes: - pods command: bash background: false args: - -c - "kubectl --context $CONTEXT -n $NAMESPACE events --for pod/$NAME | less -R" debug-container: shortCut: Shift-D description: Debug with ephemeral container dangerous: true # hidden/blocked in --readonly mode confirm: true # asks before running scopes: - containers command: bash background: false args: - -c - "kubectl --context $CONTEXT -n $NAMESPACE debug -it $POD --target=$NAME --image=nicolaka/netshoot:latest -- bash" 

The debug plugin follows the documented kubectl debug workflow. It attaches an ephemeral container that targets the selected container's process namespace, where the container runtime supports it. The pod's network namespace is shared as well. That gives you dig, curl and ss next to a distroless container that has no shell of its own.

Costs and limits:

  • An ephemeral container cannot be removed from a pod. It disappears only when the pod is replaced.
  • The image is pulled from a public registry by default. That fails on air-gapped clusters or behind strict admission policies.
  • If either applies to you, mirror a debug image internally. Pin it by tag or digest rather than latest, and change --image.
  • Do not use this on a pod you must keep pristine for forensics.

Step 4: Save your triage views as hotkeys and aliases

The first minutes of an incident often go to the same few views. Put them on keys in hotkeys.yaml and give them short names in aliases.yaml, both in the k9s config directory:

# hotkeys.yaml hotKeys: shift-0: shortCut: Shift-0 description: Pods command: pods shift-1: shortCut: Shift-1 description: Events command: events # aliases.yaml aliases: dep: apps/v1/deployments # type ":dep" instead of ":deploy" cj: batch/v1/cronjobs 

A shell alias keeps the safe launch one word away. Add this to ~/.bashrc or ~/.zshrc:

# usage: k9sro prod-eu shop k9sro() { k9s --readonly --context "$1" -n "${2:-default}" -c pods; } 

Start read-only by default and drop the flag only for the specific action that needs it. Read-only protects against slips, but it is not an access control: anyone with the kubeconfig can still run kubectl directly. Real restrictions belong in RBAC. For more day-to-day tooling write-ups, see kuryzhev.cloud.

Verify and test

Run these checks in a non-production namespace before you need them:

  1. k9s info prints the config, plugin and log directories. Confirm that plugins.yaml, hotkeys.yaml and aliases.yaml sit in the listed config directory.
  2. Start k9s --context staging -n default -c pods, select a pod, press Enter to reach its containers, and look at the hint bar. Shift-D should appear there. Go back and select a pod row; Shift-E should appear as well.
  3. If a key is missing, the usual causes are a YAML indentation error, a wrong scopes value, or a clash with a built-in binding. k9s writes details to its log file, whose path k9s info also shows.
  4. Press Shift-E on a pod. You should get a table of events with columns such as LAST SEEN, TYPE, REASON and MESSAGE, in a pager you leave with q.
  5. Start k9s with --readonly and confirm the debug plugin is blocked. Then start without the flag and confirm it asks for confirmation before attaching.
  6. Deploy a throwaway pod with a bad image or a failing command, then practice the sequence: /!Running, d, l, then p for previous logs. The goal is to do it without touching the mouse or retyping a pod name.

If the debug shell fails with a forbidden error, the earlier kubectl auth can-i patch pods/ephemeralcontainers and create pods/attach checks would have said so. Fix the namespaced Role rather than widening it to cluster scope.

Where this leaves you

None of this is magic. k9s is a faster front end for the same API calls, and its value during an incident is fewer keystrokes and fewer wrong-pod mistakes. Keep three things:

  • the read-only launcher alias,
  • two or three plugins you actually trust,
  • an RBAC role that grants only what the on-call needs.

Skip plugins that mutate cluster state unless they carry dangerous and confirm. For repeatable evidence, post-incident review or anything that needs an audit trail, rely on your logging stack and API server audit logs. Treat k9s as the interactive tool that gets you to the cause sooner.

Related