Debugging pods with Ephemeral Containers

Sometimes a pod has no curl, no ping, no decent shell because it's built from a minimal image (distroless, scratch, a slim alpine). When something fails in production, you don't want to rebuild the image and redeploy just to add tools — that's exactly what Ephemeral Containers are for.

An ephemeral container is a container that gets added temporarily to a running pod to diagnose problems. It shares the pod's network and process namespaces, so you can see its traffic, its open files, its environment — but it doesn't need to be declared in the original manifest.

Create a debug container with kubectl debug

The most common way to use them is through kubectl debug:

kubectl debug buggypod --image=debian --attach

What this command does:

• Injects a temporary container with the debian image into the buggypod pod.
• --attach connects you immediately to its output.
• Because it's just another container in the pod, you can run curl, dig, tcpdump, etc., against the services the pod sees.

Useful options:

# Share the process namespace to see the PIDs of the other containers
kubectl debug buggypod --image=nicolaka/netshoot \
  --target=app --share-processes -it

# Create a copy of the pod with an extra container, without touching the original
kubectl debug buggypod -it --copy-to=buggypod-debug --image=busybox

To use --target the cluster must allow sharing the process namespace, and the Kubernetes version must be ≥ 1.25 for ephemeralContainers to be GA.

Why this often fails: certificates and network

When debugging pod-to-pod or pod-to-external communication, before blaming the code, check the transport layer:

• Expired SSL/TLS certificates or certificates not trusted by the base image.
• DNS names that don't resolve (CoreDNS, resolv.conf search domains).
• NetworkPolicies blocking outbound traffic.
• Ports closed by the node's firewall.

The nicolaka/netshoot ephemeral container comes with dig, nslookup, curl, openssl s_client, tcpdump and almost any networking tool you may need.

RBAC: what you need to run kubectl debug

To create ephemeral containers, the identity running kubectl debug needs permissions on the pods/ephemeralcontainers subresource. Typical RBAC combinations are:

• Role + RoleBinding → permissions limited to a namespace.
• ClusterRole + ClusterRoleBinding → cluster-wide permissions.
• ClusterRole + RoleBinding → reuse a ClusterRole but restrict its scope to a specific namespace.

Minimal example to allow debugging within a namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: app
  name: debugger
rules:
  - apiGroups: [""]
    resources: ["pods", "pods/ephemeralcontainers"]
    verbs: ["get", "list", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: app
  name: debugger
subjects:
  - kind: User
    name: oncall
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: debugger
  apiGroup: rbac.authorization.k8s.io

With this, user oncall can run kubectl debug against any pod in the app namespace without holding elevated permissions across the rest of the cluster.

Summary

Ephemeral containers are the modern replacement for "shelling into the pod to debug". They don't require rebuilds, they're not recorded in the manifest, and they disappear when the pod dies. Pair them with an image like netshoot or busybox and you'll have a toolbox ready for day-to-day incidents.