Last updated: 15 November 2026. See theupdate logat the end.

A server feels slow, the monitoring graph shows a load average of 12, and someone asks whether it is "CPU". Sometimes it is. Often it is a disk, a hypervisor or a handful of processes stuck waiting, and the CPU is nearly idle. The fastest way through is to ask the same four questions in the same order: is the load real, is it CPU or waiting, which process, and which resource?

This checklist uses uptime, top, ps, vmstat, mpstat, pidstat and iostat. Every output below comes from a real run on Ubuntu 24.04 (kernel 6.18, 2 vCPUs, procps-ng 4.0.4, sysstat 12.6.1), where I created a CPU-bound load and a disk-bound load on purpose.

The checklist at a glance

  1. Read the load average against the CPU count. uptime, nproc.
  2. Split the CPU time. User, system, iowait, steal, idle: top header, mpstat.
  3. Check the queues. vmstat 1: r (runnable) and b (blocked).
  4. Find the processes. top, ps, pidstat, top -H for threads.
  5. Check the disks. iostat -xz and processes in D state.
  6. Check pressure and the odd causes. /proc/pressure, steal time, soft interrupts, limits.

1. What load average actually counts

uptime nproc cat /proc/loadavg 
 08:39:59 up 1 min, 0 user, load average: 1.98, 0.52, 0.17 2 1.98 0.52 0.17 5/105 316 

The three numbers are averages over roughly 1, 5 and 15 minutes. The kernel manual for /proc/loadavg describes them as the number of jobs in the run queue (state R) or waiting for disk I/O (stateD). That second part is why Linux load is not a pure CPU measure: processes blocked in uninterruptible sleep, usually on disk, raise it too. Brendan Gregg's history of the metric traces this to a 1993 kernel change.

Two practical consequences:

  • A common rule of thumb is to compare the load with the number of CPUs. A load of 2 on a 2-CPU machine means about one runnable task per CPU. Much higher than the core count means work is queuing. It is a rough guide, and the right threshold depends on the workload.
  • The numbers lag. In my CPU test I started four busy processes on two cores. After 40 seconds the 1-minute load was still only 1.98, although the run queue was already 4 (next section). The reverse happens after the problem ends. Right after my disk test the 1-minute load read 3.04 while the CPU was 99.5% idle. A high load by itself can be history, so look at what is happening now.

2. Is it CPU, I/O wait or something else?

The header of top splits CPU time. For a snapshot you can paste into a ticket:

top -b -n1 -w 160 | head -5 

With four sha256sum processes on two cores:

top - 08:39:59 up 1 min, 0 user, load average: 1.98, 0.52, 0.17 Tasks: 74 total, 5 running, 69 sleeping, 0 stopped, 0 zombie %Cpu(s): 95.0 us, 5.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st 

With six processes doing synchronous disk writes:

%Cpu(s): 0.0 us, 26.3 sy, 0.0 ni, 0.0 id, 47.4 wa, 0.0 hi, 21.1 si, 5.3 st 
Field Meaning What a high value suggests
us User-space code An application is burning CPU
sy Kernel time Many system calls, I/O, networking, context switching
wa Idle while I/O is outstanding Waiting on disk (or network storage)
si / hi Soft and hard interrupts Heavy network or block I/O handling
st Steal The hypervisor is giving your CPU time to someone else
id Idle Nothing to do. With high load, look for blocked tasks

The iostat manual defines iowait as the share of time the CPUs were idle while the system had an outstanding disk request. So iowait is not "CPU busy waiting for disk": it is idle time with I/O in flight, and it can look low on a busy machine even when disks are the bottleneck.

For per-CPU numbers, mpstat (part of the sysstat package) is clearer:

mpstat -P ALL 1 3 
Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle Average: all 96.77 0.00 1.99 0.00 0.00 0.00 1.24 0.00 0.00 0.00 Average: 0 97.52 0.00 1.98 0.00 0.00 0.00 0.50 0.00 0.00 0.00 Average: 1 96.00 0.00 2.00 0.00 0.00 0.00 2.00 0.00 0.00 0.00 

One CPU pegged and the rest idle points to a single-threaded process. All CPUs busy points to total demand.

3. Run queue and blocked tasks: vmstat

vmstat 1 5 

Ignore the first line: it shows averages since boot. The columns to read are r (runnable tasks, waiting for or using a CPU) and b (tasks in uninterruptible sleep).

# CPU-bound: four busy processes, two cores r b swpd free buff cache si so bi bo in cs us sy id wa st gu 4 0 0 7656268 17292 306300 0 0 0 0 506 537 98 2 0 0 0 0 # disk-bound: six synchronous writers r b swpd free buff cache si so bi bo in cs us sy id wa st gu 6 2 0 5989468 17400 1958900 0 0 0 144576 31704 100446 1 42 0 57 1 0 
  • r above the CPU count, us high, b near zero: a CPU problem.
  • b above zero, wa high, us low: processes are stuck waiting, usually for I/O.
  • si and so above zero over time: the machine is swapping, which also shows up as I/O. I did not generate swap in my tests (the machine had none).

Gregg's article also suggests the r column as a run-queue length measure, and notes that run-queue latency is a better saturation signal. Tools for that (runqlat, perf sched) need extra setup that I did not test.

4. Find the process

top, sorted by CPU

top -b -n2 -d1 -o %CPU -w 160 | awk '/^top -/ { n++ } n==2' | head -12 

Use two iterations and read the second. The two iterations can disagree (on an idle machine my first iteration read 95.2% idle with 4.8% steal and the second read 100% idle), so take the second, which covers the 1-second interval you asked for. Interactively, press 1 for per-CPU lines, H for threads and P to sort by CPU.

 PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 310 root 20 0 7936 3168 2620 R 54.5 0.0 0:19.90 sha256sum 309 root 20 0 7936 3168 2620 R 45.5 0.0 0:20.15 sha256sum 311 root 20 0 7936 3164 2620 R 45.5 0.0 0:19.93 sha256sum 312 root 20 0 7936 3164 2620 R 45.5 0.0 0:19.92 sha256sum 

Four processes sharing two cores, so each gets about half a CPU. In top, 100% means one full core, so a multi-threaded process can exceed 100%.

The ps %CPU trap

ps -eo pid,ppid,stat,pcpu,pmem,etime,comm --sort=-pcpu | head -7 

This is handy for a one-line list, but ps reports CPU use averaged over the whole life of the process, not what it is doing now. I tested it: a Python process burned a core for 15 seconds, then slept. Fifteen seconds after the burst:

ps: PID %CPU ELAPSED TIME STAT COMMAND 400 49.6 00:30 00:00:14 S python3 top: PID USER ... S %CPU ... COMMAND 400 root ... S 0.0 ... python3 

ps still showed 49.6% (14 seconds of CPU over 30 seconds of life) for a process that was idle, while top correctly showed 0.0. Use ps to list and inspect (stat, etime, wchan), and top or pidstat for current CPU use.

pidstat: a per-process view over time

pidstat -u 1 3 
Average: UID PID %usr %system %guest %wait %CPU CPU Command Average: 0 309 48.51 1.49 0.00 50.50 50.00 - sha256sum Average: 0 310 48.02 1.49 0.00 50.50 49.50 - sha256sum 

The %wait column is the useful one here: the share of time the process was runnable but waiting for a CPU. About 50% each, matching four processes on two cores. A process with high %wait and low %CPU is being starved, not misbehaving.

Threads

top -H -b -n1 -w 120 | head -10 ps -eLo pid,tid,pcpu,stat,comm --sort=-pcpu | head 

When one process of many threads is hot (a JVM, a PHP-FPM pool master, a database), the thread view tells you which thread. The same caveat applies to the ps percentage.

5. Check the disks

iostat -xz -y 1 3 

The iostat manual says the first report covers the time since boot unless -y is given. With -y that first report is omitted, so every report is a live interval. -z hides idle devices and -x shows extended statistics. During the six-writer test:

avg-cpu: %user %nice %system %iowait %steal %idle 0.54 0.00 36.02 62.90 0.54 0.00 Device r/s rkB/s ... w/s wkB/s ... w_await wareq-sz ... f/s f_await aqu-sz %util vda 0.00 0.00 ... 46364.00 136200.00 ... 0.10 2.94 ... 9734.00 0.09 5.30 98.80 

(Some columns trimmed. The exact set of columns varies between sysstat versions.)

How to read it:

  • r_await and w_await: average milliseconds for a request, including queue time. Rising values mean requests are waiting.
  • aqu-sz: average queue length for the device. Persistently above 1 means requests are queuing.
  • %util: share of time the device had requests in flight. The manual warns that saturation is signalled by a value near 100% only for devices serving requests serially; for RAID arrays and modern SSDs it does not reflect the performance limit. Here %util was 98.8% with a w_await of just 0.10 ms, so on this virtual disk it said "busy", not "slow". Judge by latency and queue length together.

Which processes are stuck?

ps -eo pid,stat,wchan:28,comm | awk 'NR==1 || $2 ~ /^D/' 
 PID STAT WCHAN COMMAND 347 D sync_dirty_buffer dd 348 D submit_bio_wait dd 350 D submit_bio_wait dd 

State D is uninterruptible sleep, and these are the processes pushing the load up while the CPU idles. The wchan column shows the kernel function the process is waiting in. Seeing block-layer functions like these points to disk; the names differ for NFS and other waits, and I did not test those. A few D processes that come and go are normal. Many of them staying put, with wa high, is the signal. To see which files are being written, pidstat -d 1 shows per-process read and write rates:

Average: UID PID kB_rd/s kB_wr/s kB_ccwr/s iodelay Command Average: 0 347 0.00 7750.00 0.00 0 dd Average: 0 348 2.00 7740.00 0.00 0 dd 

6. Pressure stall information, steal, interrupts and limits

If your kernel exposes pressure stall information, it answers "is the system actually stalling?" directly, and it is easier to read than load:

cat /proc/pressure/cpu /proc/pressure/io 
some avg10=59.42 avg60=29.56 avg300=18.23 total=93350611 full avg10=0.00 avg60=0.00 avg300=0.00 total=0 

(That is the CPU file, sampled while four busy processes ran on two cores.) The kernel documentation defines some as the share of time in which at least some tasks were stalled on the resource, and full as the share in which all non-idle tasks were stalled at once. The averages cover about 10, 60 and 300 seconds. For CPU, the full line is documented as undefined at system level and reported as zero, as above. Check that the files exist on your system, since availability depends on the kernel build.

Other causes to rule out:

  • Steal (st). On a VM, sustained steal means the host is oversubscribed. Your processes are not the problem; the answer lives with the hosting provider. I saw small steal values (1 to 5%) in my VM runs.
  • Soft interrupts (si). I saw 21% in the disk test. Heavy network traffic or block I/O raises it. cat /proc/softirqs shows which kind.
  • Container CPU limits. A container throttled by its CPU quota can look slow without high host CPU use, and load figures inside containers can reflect the host. The cgroup statistics file wasn't available in my environment, so I did not test this. Check your container runtime's documentation.
  • Swap. Non-zero si and so in vmstat, over time. Not tested here.
  • A runaway cron job or backup. Look at etime and ppid in the ps output to see what started it.

A triage script you can keep

This collects the checks above in one pass. It needs procps and sysstat (sudo apt install sysstat on Debian and Ubuntu):

#!/usr/bin/env bash # Quick first look at CPU / load / I/O. Needs procps and sysstat (iostat, mpstat). set -u cores=$(nproc) read -r l1 l5 l15 _ < /proc/loadavg echo "== Load average: $l1 $l5 $l15 (cores: $cores)" awk -v l="$l1" -v c="$cores" 'BEGIN { printf " 1-min load per core: %.2f\n", l / c }' echo "== CPU split over 3 seconds" mpstat 1 3 | awk '/^Average/ && $2=="all" { printf " user %s%% system %s%% iowait %s%% steal %s%% idle %s%%\n", $3, $5, $6, $9, $12 }' echo "== Run queue (r) and blocked (b), 3 samples" vmstat 1 4 | tail -3 | awk '{ printf " r=%s b=%s\n", $1, $2 }' echo "== Top 5 processes by CPU right now" top -b -n2 -d1 -o %CPU -w 160 | awk '/^top -/ { n++ } n==2' | sed -n '7,12p' echo "== Processes in uninterruptible sleep (D)" ps -eo pid,stat,wchan:28,comm | awk 'NR>1 && $2 ~ /^D/' | head -10 | sed 's/^/ /' echo "== Block devices with activity (iostat -x, one 1-second sample)" iostat -xz -y 1 1 | sed -n '/^Device/,$p' | grep -v '^$' | sed 's/^/ /' 

Output during the CPU test:

== Load average: 1.33 1.16 0.51 (cores: 2) 1-min load per core: 0.67 == CPU split over 3 seconds user 97.01% system 2.49% iowait 0.00% steal 0.50% idle 0.00% == Run queue (r) and blocked (b), 3 samples r=3 b=0 r=3 b=0 r=3 b=0 

And during the disk test:

== Load average: 2.33 1.40 0.61 (cores: 2) 1-min load per core: 1.17 == CPU split over 3 seconds user 0.36% system 27.03% iowait 60.00% steal 0.36% idle 0.36% == Run queue (r) and blocked (b), 3 samples r=2 b=5 == Processes in uninterruptible sleep (D) 459 D fsync_buffers_list dd 460 D sync_dirty_buffer dd 

Notice what the second case shows: the load per core is only 1.17, yet the machine is stalled, with 60% iowait and five blocked tasks. A load number alone would have hidden that.

Reproduce the two scenarios safely

# CPU-bound: four busy processes for 40 seconds for i in 1 2 3 4; do timeout 40 sha256sum /dev/zero > /dev/null & done # Disk-bound: six synchronous writers, capped at 30 seconds for i in 1 2 3 4 5 6; do timeout 30 dd if=/dev/zero of=/tmp/ddtest$i bs=4k count=200000 oflag=dsync status=none & done wait; rm -f /tmp/ddtest* 

Do this on a test machine only. The dd command writes up to 800 MB per process if not stopped, which is why timeout is there, and rm removes the files afterwards.

What next, by symptom

What you see Likely meaning Next step
High load, us high, r above cores, b ~0 CPU saturated Identify the process (top, pidstat), then profile or scale
High load, wa high, b above 0, D-state processes Disk or storage bound iostat -xz, pidstat -d, look at the slow queries or writers
High load, CPU mostly idle, nothing in D, r ~0 Load is history Check the 5- and 15-minute values and your monitoring timeline
High st Noisy host (VM) Check with the provider, or move instances
One CPU at 100%, rest idle Single-threaded bottleneck top -H to find the thread
High sy, high cs in vmstat Lots of syscalls or context switches Profile the process (for example strace -c or perf, not tested here)

Common mistakes

  • Treating a load of 8 as an emergency on a 16-core machine, or a load of 2 as fine on a single core.
  • Assuming high load means high CPU. On Linux it also counts tasks in D state.
  • Reading the first vmstat or iostat line, which are averages since boot.
  • Trusting ps %CPU for what a process is doing now.
  • Reading %util near 100% as "the disk is saturated" on SSDs and virtual disks.
  • Killing a process in D state. It cannot be interrupted while in that state, so fix the wait instead.
  • Ignoring steal time on virtual machines.

Related Dev Talk articles

Frequently asked questions

What is a good load average?

It depends on the CPU count and workload. As a rough rule of thumb, a load near or below the number of CPUs means the machine is keeping up. Use trends and the other metrics, not a fixed number.

Why is my load high but CPU usage low?

Because Linux counts processes in uninterruptible sleep (state D), usually waiting for disk, as well as runnable ones. Check iowait, vmstat's b column and iostat.

What does iowait mean?

The share of time the CPUs were idle while an I/O request was outstanding. It points to storage waits but can read low on a busy system.

Does a high load average mean the CPU is the bottleneck?

Not necessarily. Check whether us and r are high, or wa and b.

Do I need root for these tools?

Not to read system-wide numbers. Seeing other users' process details can depend on your system's settings; I ran everything as root and did not test the non-root differences.

update log

  • 15 November 2026: first version.
  • To recheck on each update: column names in the current sysstat and procps releases, and the PSI documentation.

Sources

Originally published on DEV Talk.