How measuring Kubernetes node storage usage led to a simpler capacity decision in an Amazon EKS cluster. Engineering Build Notes is a series where I document engineering decisions, platform improvements, and infrastructure trade-offs from real production environments. Context In Engineering Build Notes #1, I investigated what initially looked like an opportunity to reduce worker-node capacity. The nodes appeared to have more capacity than the workloads needed. The metrics told a different story. One node was consistently running at approximately 98–100% CPU utilization, while memory remained around 12–15%. The bottleneck wasn't memory. It was CPU. So instead of downsizing the workers, I increased the node size: t3.medium ↓ t3.large That gave the workloads more compute headroom. Then Build Note #2 looked at another layer. The cluster had relatively low observed utilization, but some workloads were carrying resource requests significantly higher than their normal consumption. We right-sized those requests and the worker-node footprint was reduced: 5 worker nodes ↓ 4 worker nodes But there was another question. If we were already looking closely at node capacity, were the nodes themselves provisioned with more storage than they actually needed? That became the next investigation. The Problem The worker nodes were provisioned with 200Gi of root storage. On paper, that provided plenty of room. But provisioned capacity and actual requirement are two different things. Rather than immediately reducing the volume size, I wanted to understand what was actually consuming storage on the nodes. The question was simple: How much storage does the node actually need to operate reliably under its observed workload? That meant looking at the node itself. What Was Consuming the Storage? Node storage isn't consumed by application data alone. Kubernetes worker nodes use local storage for several purposes, including: Container images Writable container layers Container logs emptyDir volumes Kubelet-managed data Container runtime data System and temporary files These fall broadly under the area Kubernetes treats as local ephemeral storage. The kubelet also monitors filesystem and inode availability and can report disk pressure when storage becomes constrained. So simply looking at application volumes wouldn't give me the complete picture. I needed to understand the node's overall disk usage. Measure Before Reducing The first step was measurement. I inspected the main areas contributing to node storage usage, including: /var/lib/containerd /var/lib/kubelet /var/log The goal wasn't simply to ask: How much disk is free? It was: What is consuming the disk, and how does that compare with the provisioned capacity? That distinction matters. A node can have plenty of free storage today while still accumulating: Container images Logs Writable layers Runtime data Temporary files At the same time, provisioning significantly more storage than the workload requires doesn't automatically make the platform more reliable. The answer has to come from measurement. The Storage Picture The measurements showed that actual node disk consumption was substantially below the provisioned 200Gi capacity under the observed operating conditions. That made this investigation different from Build Note #1. In Build Note #1: Available compute ↓ Insufficient for workload ↓ Increase capacity Here: Provisioned storage ↓ Substantially above observed requirement ↓ Consider reducing capacity The finding wasn't that the nodes could never need 200Gi. It was that the existing allocation appeared larger than necessary for the workload and operating conditions we had observed. That distinction was important. Capacity Is Not the Same as Requirement A node having: Storage capacity ↓ 200Gi doesn't mean: Storage required ↓ 200Gi These are different measurements. Capacity is what the node has available. Requirement is what the workload and node services actually consume, plus the headroom needed for normal operation. The engineering decision came from comparing those two: Provisioned capacity ↓ 200Gi Observed consumption ↓ Significantly lower Operational headroom ↓ Required Decision ↓ Right-size capacity This is a simple idea, but it's easy for old infrastructure defaults to outlive the assumptions that created them. Why 80Gi? The next question was: Why 80Gi? The objective wasn't to find the smallest disk that could possibly run the cluster. That would simply replace over-provisioning with under-provisioning. The goal was to find a more appropriate balance between observed usage and operational headroom. The reduced capacity still needed to accommodate: Container images Container logs Runtime data Temporary workload storage Kubernetes and system overhead Normal operational growth So the decision was: 200Gi ↓ 80Gi More importantly, the decision can be represented simply: 200Gi → measure → validate → preserve headroom → 80Gi 80Gi wasn't chosen because it was the absolute minimum. It was chosen as a more appropriate capacity based on the storage usage we had observed and the operational margin we wanted to retain. Why Not Just Keep 200Gi? It can be tempting to treat extra storage as a harmless safety buffer. More disk means more room. But infrastructure defaults have a habit of becoming permanent. If every worker node receives significantly more storage than the workload normally requires, that capacity becomes part of the infrastructure footprint whether it is being used or not. The better question is: What amount of capacity gives the workload enough operational headroom without carrying unnecessary provisioned capacity? That is the same principle behind the earlier investigations. Not: How much can we remove? But: How much does the system actually need? The Right-Sizing Decision The node configuration was changed from: Before 200Gi to: After 80Gi The important part wasn't the number itself. It was the process: Measure ↓ Understand storage consumption ↓ Compare capacity with requirement ↓ Choose appropriate headroom ↓ Reduce provisioned capacity The storage reduction happened after measurement, not before it. That distinction is what made the change an engineering decision rather than simply a cost-cutting exercise. Validate the Change Reducing the volume size isn't successful simply because a node launches with a smaller disk. The real question is whether the reduced capacity is sufficient for normal cluster operation. The validation focused on the things that could turn a storage reduction into an operational problem: Node provisioning ↓ Node joins Kubernetes ↓ Node becomes Ready ↓ Workloads schedule successfully ↓ Monitor storage behavior ↓ Watch for disk pressure The new nodes were able to provision successfully and join the cluster with the reduced storage capacity. That's important because infrastructure changes should be validated through actual system behavior, not just configuration. And the validation doesn't end when the node becomes Ready. Storage is a resource that needs continued observation because image growth, logs, temporary files, and workload changes can alter consumption over time. Things worth watching include: Node conditions Disk-pressure events Container image accumulation Log growth Runtime storage growth Filesystem and inode availability A successful infrastructure change isn't just: The configuration applied successfully. It's: The system continued operating within acceptable conditions after the change. The Outcome The worker-node root storage allocation was reduced: 200Gi ↓ 80Gi That's a 60% reduction in provisioned root storage capacity per worker node. But the percentage isn't the most important result. The important result was the decision-making process: Measure node storage ↓ Understand what consumes it ↓ Compare capacity with requirement ↓ Choose appropriate headroom ↓ Right-size storage ↓ Validate node operation I would describe the outcome as reduced provisioned EBS capacity, rather than claiming a specific dollar saving from this change. The actual financial impact depends on factors such as the volume type, node count, region, and how long the volumes remain provisioned. The engineering result is straightforward: Under the observed operating conditions, the workloads and node services did not require the original 200Gi allocation. The Engineering Pattern Across the three investigations, the pattern became clearer. Build Note #1 — Compute We found an actual CPU constraint. CPU bottleneck ↓ Increase compute ↓ t3.medium → t3.large Build Note #2 — Workload Scheduling We found workloads carrying resource requests that were significantly higher than their observed consumption. Oversized requests ↓ Right-size workloads ↓ Better scheduling flexibility ↓ 5 nodes → 4 nodes Build Note #3 — Storage We found excess provisioned node storage. 200Gi storage ↓ Measure actual consumption ↓ Compare capacity vs. requirement ↓ 80Gi storage Three investigations. Three different engineering decisions. But the process remained the same: Measure ↓ Identify the problem ↓ Change the appropriate layer ↓ Validate ↓ Repeat What I Took Away 1. Capacity Is Not the Same as Requirement A node having 200Gi of storage doesn't mean the workload requires 200Gi. Provisioned capacity should be informed by observed consumption and operational requirements. 2. Measure Before Downsizing Reducing infrastructure without understanding what consumes it can simply move the problem somewhere else. Measurement comes first. 3. Right-Sizing Is Not Under-Sizing The goal isn't to operate as close to the limit as possible. It's to provide enough capacity for: Normal variation Growth Platform overhead Operational headroom The objective is appropriate capacity, not minimum capacity. 4. Infrastructure Defaults Should Be Reviewed A configuration that made sense when it was introduced may not remain appropriate as workloads and operating patterns change. Infrastructure defaults often survive much longer than the assumptions that created them. Periodic capacity reviews can expose opportunities to improve efficiency without reducing reliability. 5. Validate Infrastructure Changes A smaller configuration is only useful if the system continues to operate reliably after the change. Provisioning successfully is only the beginning. The platform still needs to be observed after the change. 6. Look at the Whole System CPU, memory, storage, scheduling, and autoscaling are connected. Optimizing one layer can expose another area worth investigating. That's what happened across these three investigations. The Bigger Lesson Build Note #1 started with a CPU constraint. Build Note #2 moved down to the workload and scheduling layer. Build Note #3 moved back to the node layer and examined storage capacity. The important lesson isn't: Always make your disks smaller. Or: Always reduce Kubernetes resource requests. It's: Don't optimize infrastructure based on assumptions. Measure the system, understand the requirement, and change the layer responsible for it. Sometimes the right decision is to increase capacity. Sometimes it's to reduce it. Sometimes the right answer is to change the workload configuration instead. The engineering work is figuring out which one the system actually needs. Final Thoughts Three investigations. Three different problems. CPU ↓ Increase compute Workload requests ↓ Right-size scheduling inputs Storage ↓ Right-size node capacity But the engineering principle remained the same: Measure ↓ Understand ↓ Change ↓ Validate ↓ Repeat Infrastructure optimization isn't about making everything smaller. It's about making infrastructure appropriate for the workload it actually runs. A 200Gi volume isn't inherently wrong. A smaller 80Gi volume isn't inherently better. The better configuration is the one supported by measurement, operational requirements, and appropriate headroom. Measure first. Right-size second. Validate always.