Load average is the oldest load signal in Unix, and still the one most alerts are built on. It is also the one most likely to send you to the wrong subsystem.
The number is not a utilization percentage. It is a count of tasks: runnable tasks plus tasks in uninterruptible sleep. On a four-core machine, load 8 can mean the CPU is oversubscribed, or it can mean eight processes are queued behind one slow disk while the CPU sits idle. The number looks identical. The fix is completely different.
Pressure Stall Information (PSI) answers the question load average cannot: how much wall-clock time did tasks actually lose, and to which resource? It has been in the kernel since 4.20, exports system-wide and per-cgroup numbers, and offers a pollable trigger you can wire into a service.
This is a practical tour: what the numbers mean, how to read them without fooling yourself, and how to make a process wake up when pressure crosses a threshold.
What load average counts, and what it cannot
The kernel's own documentation for /proc/loadavg is blunt: the first three fields are the number of jobs in the run queue (state R) or waiting for disk I/O (state D), averaged over 1, 5, and 15 minutes.
That definition creates three blind spots:
- It mixes resources. A runnable task waiting for CPU and a task blocked in the kernel waiting for a disk both add to the same number.
- It counts tasks, not time. Eight tasks that each waited 1 ms look the same as eight tasks that each waited 8 seconds.
- It cannot tell you who. The number is system-wide. It will not point at a container, a slice, or a service.
Load average is still a useful coarse signal. It is a terrible capacity-planning unit and an even worse alerting condition.
The PSI interface
PSI exports one file per resource under /proc/pressure/: cpu, memory, and io.
Here is a real Debian 13 host (kernel 6.12.117, 3 vCPU, 8 GB RAM), three days after boot:
cat /proc/pressure/cpu some avg10=0.00 avg60=0.17 avg300=0.13 total=511546971 full avg10=0.00 avg60=0.00 avg300=0.00 total=0 cat /proc/pressure/memory some avg10=0.00 avg60=0.00 avg300=0.00 total=817390 full avg10=0.00 avg60=0.00 avg300=0.00 total=780205 cat /proc/pressure/io some avg10=0.02 avg60=0.07 avg300=0.01 total=260148927 full avg10=0.02 avg60=0.07 avg300=0.01 total=253602477 Each line has the same shape:
-
avg10,avg60,avg300are percentages of wall-clock time over roughly the last 10, 60, and 300 seconds. -
totalis cumulative stall time in microseconds since boot. -
somemeans at least one task was stalled on that resource. -
fullmeans all non-idle tasks were stalled at the same time. No useful work was happening.
That full line is the one that matters for throughput. some is the one that matters for tail latency.
A four-worker example makes the distinction concrete. If one worker blocks on fsync() while the other three keep working, IO some rises for that period, but full stays at zero. If all four block at once, both rise. A workload that spends long stretches in full is thrashing: it is paying for the resource without doing work.
There is one deliberate asymmetry. At the system level, CPU full is undefined and is reported as zero for backward compatibility (the line has been emitted since Linux 5.13). Per-cgroup cpu.pressure files do carry a meaningful full value. Do not build a system-wide alert on CPU full; it will never fire.
Reading the numbers honestly
The three averages are convenient, but they are averages. A 10-second memory freeze inside a five-minute window barely moves avg300, and by the time avg60 looks bad, the incident is already a minute old.
The total counter is more honest. Sample it twice and divide by the elapsed time:
#!/usr/bin/env bash set -euo pipefail res=${1:-io} secs=${2:-10} file=/proc/pressure/$res read -r some1 full1 < <(awk '/^some/{s=$5} /^full/{f=$5} END{print s+0, f+0}' "$file") sleep "$secs" read -r some2 full2 < <(awk '/^some/{s=$5} /^full/{f=$5} END{print s+0, f+0}' "$file") awk -v s1="$some1" -v s2="$some2" -v f1="$full1" -v f2="$full2" \ -v t="$secs" -v r="$res" 'BEGIN { printf "%-6s over %ss: some %.2f%%", r, t, (s2-s1)/(t*1e6)*100 if (r == "cpu") print " (cpu has no meaningful full line)" else printf ", full %.2f%%\n", (f2-f1)/(t*1e6)*100 }' Run it against the three resources:
./psi-delta.sh cpu 3 cpu over 3s: some 0.00% (cpu has no meaningful full line) ./psi-delta.sh memory 3 memory over 3s: some 0.00%, full 0.00% ./psi-delta.sh io 3 io over 3s: some 0.00%, full 0.00% The arithmetic is simple: microseconds of stall divided by microseconds of wall time, times 100.
This is the shape you want in a monitoring check. Poll every 15 or 30 seconds, keep the last few deltas, and alert on sustained full time. Do not alert on a single sample, and do not alert on a moving average that has already absorbed the spike.
On the host above, the cumulative IO numbers tell a story the averages hide: some and full are almost equal (260 s versus 253 s of stall since boot). When this machine does stall on IO, nearly every non-idle task is stuck — the signature of a workload with very little parallelism at the moment of the stall. A busy database with many workers would show some far above full: some queries wait while others still make progress.
Per-cgroup pressure: find the noisy tenant
System-wide PSI tells you the machine is in trouble. Per-cgroup PSI tells you who.
Each cgroup in a unified (v2) hierarchy has cpu.pressure, memory.pressure, and io.pressure, in the same format. Find a service's cgroup and read it:
systemctl show -p ControlGroup -p MemoryAccounting postgresql.service cat /sys/fs/cgroup/system.slice/postgresql.service/memory.pressure ControlGroup=/system.slice/postgresql.service MemoryAccounting=yes some avg10=0.00 avg60=0.02 avg300=0.01 total=1842231 full avg10=0.00 avg60=0.00 avg300=0.00 total=41207 Two requirements:
- The system must use the unified cgroup v2 hierarchy. On a hybrid system, these files do not exist.
- Memory pressure accounting for a cgroup needs memory accounting enabled. That is what
MemoryAccounting=yesconfirms.
To rank services by current memory pressure:
for f in /sys/fs/cgroup/system.slice/*.service/memory.pressure; do awk -v f="$f" '/^some/{split($2,a,"="); print a[2], f}' "$f" done | sort -rn | head -5 Adjust the path for user.slice, container scopes, or your own slice layout. The numbers are percentages, so the sort is numeric.
PSI accounting is aggregated at every level of the hierarchy. A slice can show pressure even when its own processes look idle, because a descendant is stalled. That is useful for finding the loud tenant, but it also means a parent's number is not independent evidence.
If you run very deep cgroup trees, PSI's per-level accounting has a cost. The kernel exposes a non-hierarchical switch, cgroup.pressure:
cat /sys/fs/cgroup/system.slice/cgroup.pressure 1 Writing 0 disables PSI accounting for that cgroup only; it does not propagate to descendants and does not require enabling anything from the root. On a large host with thousands of intermediate cgroups, turning it off on non-leaf nodes you never query is a reasonable trade.
Triggers: wake up when pressure crosses a line
Averages make you poll. PSI also supports a trigger: open the pressure file, write a threshold, and poll() the file descriptor. The kernel wakes you when the stall budget is exceeded.
The write format is:
<some|full> <stall amount in us> <time window in us> For example, some 150000 1000000 means "150 ms of partial stall within any 1-second window." full 50000 1000000 means "50 ms in which everything was stalled, within a 1-second window."
Here is a complete watcher, based on the kernel's documented example:
#define _GNU_SOURCE #include <errno.h> #include <fcntl.h> #include <poll.h> #include <stdio.h> #include <stdlib.h> #include <string.h> #include <unistd.h> int main(int argc, char **argv) { const char *path = argc > 1 ? argv[1] : "/proc/pressure/memory"; const char *trig = argc > 2 ? argv[2] : "some 150000 2000000"; int fd = open(path, O_RDWR | O_NONBLOCK); if (fd < 0) { fprintf(stderr, "open %s: %s\n", path, strerror(errno)); return 1; } if (write(fd, trig, strlen(trig) + 1) < 0) { fprintf(stderr, "write trigger: %s\n", strerror(errno)); return 1; } struct pollfd pfd = { .fd = fd, .events = POLLPRI }; for (;;) { int n = poll(&pfd, 1, -1); if (n < 0) { if (errno == EINTR) continue; perror("poll"); return 1; } if (pfd.revents & POLLERR) { fprintf(stderr, "trigger source disappeared\n"); return 1; } if (pfd.revents & POLLPRI) { puts("pressure event"); fflush(stdout); } } } Build and run it:
gcc -O2 -Wall -Wextra -o psi-watch psi-watch.c ./psi-watch /proc/pressure/memory 'some 150000 2000000' The rules are worth knowing before you build on this:
- One trigger per file descriptor. A second write on the same fd fails with
EBUSY. Separate triggers need separateopen()calls, which is what lets you poll them independently. - The window is between 500 ms and 10 s. Unprivileged users must use a window that is a multiple of 2 seconds; anything else fails with
EINVAL. Root can use any window in range. - The trigger lives as long as the descriptor. Close the fd and the monitor is deregistered.
- Events are rate-limited to one per tracking window, and the monitor stays active for at least one full window to avoid flapping.
- Unprivileged triggers work where the pressure file is writable. On this Debian 13 host,
/proc/pressure/*is mode0666, so a normal user can register a 2-second-window trigger.
I tested the exact program above against /proc/pressure/io with a 1 µs threshold and a 2-second window while writing a file with fsync. It compiled cleanly with -Wall -Wextra, fired two POLLPRI events, and a second write on the same fd returned errno 16 (EBUSY), while a 1-second window as an unprivileged user returned errno 22 (EINVAL). Those are the failure modes you will hit first.
systemd already speaks PSI
systemd 254 added a protocol that hands pressure monitoring to services. Instead of hard-coding a path, a service receives two environment variables and watches the fd itself.
You can see the contract directly:
systemd-run --user --wait --pipe -p MemoryPressureWatch=yes env | grep PRESSURE MEMORY_PRESSURE_WATCH=/sys/fs/cgroup/user.slice/user-1000.slice/user@1000.service/app.slice/run-p25948-i26248.service/memory.pressure MEMORY_PRESSURE_WRITE=c29tZSAyMDAwMDAgMjAwMDAwMAA= MEMORY_PRESSURE_WATCH is the file to open. MEMORY_PRESSURE_WRITE is Base64-encoded data to write into it after opening — the trigger, configured by systemd. Decode it:
printf %s 'c29tZSAyMDAwMDAgMjAwMDAwMAA=' | base64 -d | tr '\0' '\n' some 200000 2000000 That is 200 ms of partial stall per 2-second window, the default threshold. The relevant settings:
-
MemoryPressureWatch=takesyes,no,auto, orskip. The default isauto: enabled when memory accounting is already on. -
MemoryPressureThresholdSec=defaults to 200 ms per 2-second window. - Setting
MemoryPressureWatch=yesalso enables memory accounting and relaxes the permissions on the service'smemory.pressurefile so the service user can write the trigger.
The service still has to cooperate. A process that ignores the variables gets no benefit. For sd-event programs, the integration is one line:
(void) sd_event_add_memory_pressure(event, NULL, NULL, NULL); That installs the default handler, which trims allocation caches. On systemd 261+, CPU and IO have their own callbacks because the right reaction is application-specific: reduce parallelism, defer batch work, or shed load.
Version trap: MemoryPressureWatch= and the memory threshold are from systemd 254. CPUPressureWatch= and IOPressureWatch= are from systemd 261. Debian 13 ships systemd 257.13, so those CPU/IO unit settings do not exist there even though the kernel supports the underlying files. Check systemctl --version before copying a drop-in from a newer distribution.
Where systemd-oomd fits
PSI measures. systemd-oomd acts.
systemd-oomd uses cgroup v2 and PSI to kill a cgroup before the kernel OOM killer has to. Its defaults are worth remembering:
-
DefaultMemoryPressureLimit=60%— the fraction of time in a 10-second window in which all tasks in the cgroup were delayed (PSIfull avg10). -
DefaultMemoryPressureDurationSec=30s— how long that must persist. -
SwapUsedLimit=90%— system memory and swap both above this fraction trigger swap-based action.
Only descendant cgroups are eligible, and only leaf cgroups or cgroups with memory.oom.group=1 can be killed. Monitoring is opt-in per unit: ManagedOOMMemoryPressure=kill on a parent slice, for example. The systemd documentation suggests a lower limit (around 40%) for user@$UID.service because interactive sessions are latency-sensitive, while 60% may be fine for system.slice.
This is a policy decision, not a monitoring tweak. If you run a database, a build farm, or anything where a kill is worse than a slow minute, understand the ruleset before enabling it. oomctl lists monitored cgroups and pressure information when the package is installed.
Putting it to work
A few patterns that hold up in production:
- Alert on deltas, not averages. Sample
totalevery 30 seconds and alert whenfullexceeds a few percent for three consecutive samples. This catches short, severe stalls thatavg300erases. - Use
fullfor throughput,somefor latency. A highsomewith lowfullmeans work is still progressing but some requests are slow. A highfullmeans the resource is setting your throughput ceiling. - Put per-cgroup files in your service dashboards.
systemctl show -p ControlGroupgives you the path; the file format is identical to/proc/pressure. - Trigger instead of polling in latency-sensitive daemons. Register a 2-second-window trigger with a threshold that matches your SLO, then release caches or reduce worker count in the handler.
- Disable accounting where you never read it.
cgroup.pressure=0on deep, non-leaf cgroups avoids paying for aggregation you do not use.
Traps worth memorizing
- No
/proc/pressuredirectory? PSI is missing. The kernel may predate 4.20, may be built withoutCONFIG_PSI, or may have been booted withpsi=0. Check the kernel command line and config before writing queries. - System-wide CPU
fullis always zero. It is defined as zero there. Usesome, or read a cgroup'scpu.pressure. -
someandfullare time shares, not task counts. 25%somedoes not mean a quarter of your threads are stuck; it means a quarter of the time at least one was. - Averages hide short events. A 5-second total stall in a 300-second window is easy to miss.
-
/proc/pressureis the host view. Inside a container, read the container cgroup's ownmemory.pressurerather than assuming/procis namespaced to you. - Per-cgroup PSI needs cgroup v2 and memory accounting. If the files are absent, check
stat -fc %T /sys/fs/cgroupandsystemctl show -p MemoryAccounting <unit>. - systemd-oomd is not a monitoring agent. It kills. Test the limits before you enable it on a host that matters.
Load average will keep appearing in dashboards because it is cheap and universal. Just stop asking it to identify a bottleneck. PSI gives you the missing dimension: not how many tasks were waiting, but how much time they lost, to which resource, and in which cgroup.
Sources
- Linux kernel documentation, Pressure Stall Information: https://docs.kernel.org/accounting/psi.html
- Linux kernel documentation, Control Group v2 (including
cgroup.pressure): https://docs.kernel.org/admin-guide/cgroup-v2.html - Linux kernel parameters,
psi=: https://docs.kernel.org/admin-guide/kernel-parameters.html -
proc_loadavg(5): https://man7.org/linux/man-pages/man5/proc_loadavg.5.html - systemd, Resource Pressure Handling: https://systemd.io/PRESSURE/
-
systemd.resource-control(5): https://www.freedesktop.org/software/systemd/man/latest/systemd.resource-control.html -
systemd-system.conf(5): https://www.freedesktop.org/software/systemd/man/latest/systemd-system.conf.html -
systemd-oomd.service(8): https://www.freedesktop.org/software/systemd/man/latest/systemd-oomd.service.html -
oomd.conf(5): https://www.freedesktop.org/software/systemd/man/latest/oomd.conf.html