The service is still running. The process is there, the memory looks normal, and the CPU is almost idle. But every request hangs until it times out, and the logs stopped twenty minutes ago without a single error. Someone suggests restarting it. Before you do, spend one minute capturing what every thread inside it is doing right now. That snapshot is called a thread dump, and once you have read one, you will never want to restart a stuck service blind again.

On the Java virtual machine there are three common ways to take one. Run jstack followed by the process id. Or run jcmd with the process id and Thread.print. Or send the process a quit signal with kill -3, which makes the JVM write the dump to its own standard output, so in a container it appears in the logs. Take three dumps about ten seconds apart. A thread that is in the same place in all three is stuck, not busy.

Each thread appears with a name, a state and a stack trace. The state tells you a lot. RUNNABLE means it is doing work, or waiting on the network. BLOCKED means it is waiting to enter a synchronized section that another thread holds. WAITING and TIMED_WAITING mean it is parked until something wakes it.

Now look for patterns rather than reading every line. Are two hundred request threads all waiting at the same line, inside the method that borrows a database connection from the pool? Then the pool is empty, and the interesting question is who is holding the connections. Are several threads BLOCKED on the same lock, with one thread holding it while it waits on a slow external call? There is your problem. If two threads are each waiting for a lock the other holds, jstack will often say so plainly, with a line saying it found a Java level deadlock.

Other runtimes have the same idea. For Python, py-spy dump with a process id prints the stack of every thread without stopping the program.

Practise on a calm day. Take a dump of a healthy service, find your own code in it, and learn what normal looks like. Then, during the next incident, capture first and restart second. The restart fixes today. The dump tells you how to stop it happening again.

– Asael Shinder