Every StatefulSet eventually hits the same wall: the 50 Gi volume you provisioned eighteen months ago for a PostgreSQL replica is at 92 %, the WAL is growing, and the on-call rotation is getting alerts at 03:00. The good news is that the CSI spec has supported online volume expansion for years. The bad news is that StatefulSet.spec.volumeClaimTemplates is immutable, so the obvious kubectl edit fails with a validation error and people start improvising — which is how data gets lost.
This article walks through the real procedure: what actually happens under the hood when you grow a PVC, how to work around the immutable volumeClaimTemplates field with an orchestrated recreation, and — the part most runbooks skip — how to prove that the filesystem inside the pod actually grew.
Context: why this is harder than it looks
A PersistentVolumeClaim created from a volumeClaimTemplates entry is not owned by the StatefulSet controller in the way Pods are. The controller creates the PVC named <templateName>-<statefulSetName>-<ordinal> if it does not exist, and then never touches it again. It does not reconcile size, labels, or annotations.
That gives us two independent problems:
- The live PVCs need to be expanded. That part is standard CSI work and can be done online on most drivers.
- The template needs to be updated too, otherwise the next replica you scale out (
postgres-3,postgres-4) will be created at the old size. You end up with a heterogeneous StatefulSet — the classic source of "replica 3 is full but 0, 1, 2 are fine" incidents.
KEP-0661 (in-place update of volumeClaimTemplates) is still not GA at the time of writing, so point 2 requires deleting and recreating the StatefulSet object without touching its Pods.
How CSI volume expansion actually works
Understanding the two-phase flow is what lets you debug a stuck resize instead of rebooting nodes at random.
Control-plane phase
When you increase spec.resources.requests.storage on a PVC:
- The API server validates that the new size is larger than the current one. Shrinking is rejected outright.
- The
external-resizersidecar (deployed alongside the CSI controller) watches PVCs. If the backingStorageClasshasallowVolumeExpansion: true, it callsControllerExpandVolumeon the driver. - The driver grows the backend device (EBS volume, Ceph RBD image, vSphere VMDK, NetApp LUN…).
- The resizer updates
PersistentVolume.spec.capacity.storageand, if the driver reportsnode_expansion_required, sets the PVC conditionFileSystemResizePending.
Node phase
The kubelet on the node where the Pod runs sees FileSystemResizePending, calls NodeExpandVolume, and the driver runs resize2fs or xfs_growfs against the mounted device. Only then does pvc.status.capacity.storage get updated to the new value.
This is the key invariant to remember:
spec.resources.requests.storageis your intent.status.capacity.storageis the reality the application sees. They are only equal once the filesystem has been grown.
Online expansion (the volume stays mounted and the Pod stays running) is GA in modern Kubernetes. Some drivers — and all volumeMode: Block volumes — still require the Pod to be stopped; we cover that case below.
Pre-flight checks
Never start without these four checks.
# 1. Which StorageClass backs the PVCs, and does it allow expansion? kubectl get pvc -l app=postgres \ -o custom-columns=NAME:.metadata.name,SC:.spec.storageClassName,REQ:.spec.resources.requests.storage,CAP:.status.capacity.storage kubectl get sc gp3-csi -o jsonpath='{.allowVolumeExpansion}{"\n"}' If that returns false or nothing, patch the StorageClass. The field is mutable:
kubectl patch sc gp3-csi --type merge -p '{"allowVolumeExpansion": true}' For reference, a correct StorageClass on OpenShift with the AWS EBS CSI driver:
apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: gp3-csi provisioner: ebs.csi.aws.com parameters: type: gp3 encrypted: "true" kmsKeyId: arn:aws:kms:eu-west-3:111122223333:key/REPLACE-ME allowVolumeExpansion: true reclaimPolicy: Retain volumeBindingMode: WaitForFirstConsumer # 2. Is the csi-resizer sidecar actually running? kubectl -n openshift-cluster-csi-drivers get pods -l app=aws-ebs-csi-driver-controller kubectl -n openshift-cluster-csi-drivers logs deploy/aws-ebs-csi-driver-controller -c csi-resizer --tail=50 # 3. What is the PVC retention policy? This decides whether an orphan delete is safe. kubectl get sts postgres -o jsonpath='{.spec.persistentVolumeClaimRetentionPolicy}{"\n"}' # 4. Take a backup. A snapshot is not a backup, but it is better than nothing. kubectl get volumesnapshotclass On point 3: if whenDeleted: Delete, the StatefulSet controller puts an ownerReference on the PVCs. Deleting the StatefulSet with the default cascade would garbage-collect your data. Set it to Retain before you go any further:
kubectl patch sts postgres --type merge -p \ '{"spec":{"persistentVolumeClaimRetentionPolicy":{"whenDeleted":"Retain","whenScaled":"Retain"}}}' And always take a snapshot of at least one replica:
apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: data-postgres-0-preresize namespace: databases spec: volumeSnapshotClassName: csi-aws-vsc source: persistentVolumeClaimName: data-postgres-0 Step 1: expand the live PVCs
Patch each PVC individually. Do it one ordinal at a time, highest ordinal first if your application has a primary on ordinal 0 — you want the least critical replica to be the guinea pig.
NS=databases NEW_SIZE=100Gi for i in 2 1 0; do kubectl -n "$NS" patch pvc "data-postgres-$i" --type merge \ -p "{\"spec\":{\"resources\":{\"requests\":{\"storage\":\"$NEW_SIZE\"}}}}" done Watch the conditions propagate:
kubectl -n databases get pvc data-postgres-0 \ -o jsonpath='{range .status.conditions[*]}{.type}{"="}{.status}{" "}{.message}{"\n"}{end}' Typical sequence:
Resizing=True FileSystemResizePending=True …and then both conditions disappear once the node phase completes. On a healthy EBS/Ceph setup this takes 10–60 seconds. If Resizing=True sticks for minutes, go read the csi-resizer logs — it is almost always a cloud API quota or an IAM permission (ec2:ModifyVolume).
Confirm the reality:
kubectl -n databases get pvc -l app=postgres \ -o custom-columns=NAME:.metadata.name,REQ:.spec.resources.requests.storage,STATUS:.status.capacity.storage NAME REQ STATUS data-postgres-0 100Gi 100Gi data-postgres-1 100Gi 100Gi data-postgres-2 100Gi 100Gi Step 2: the volumeClaimTemplates wall
Now try to update the template:
kubectl -n databases patch sts postgres --type json \ -p '[{"op":"replace","path":"/spec/volumeClaimTemplates/0/spec/resources/requests/storage","value":"100Gi"}]' The StatefulSet "postgres" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden Orchestrated recreation with --cascade=orphan
The trick is to delete the StatefulSet object while leaving its Pods and PVCs running, then recreate it with the new template. The controller adopts the existing Pods (match is by metadata.name and label selector) and leaves them alone because the Pod spec is unchanged.
# 1. Snapshot the current object — your rollback artifact. kubectl -n databases get sts postgres -o yaml > /tmp/postgres-sts-backup.yaml # 2. Delete the object only. Pods keep running, PVCs keep existing. kubectl -n databases delete sts postgres --cascade=orphan # 3. Verify nothing moved. kubectl -n databases get pods -l app=postgres -o wide --cascade=orphan sets the Orphan deletion propagation policy, which strips owner references from all dependents — Pods and PVCs alike. Your pods keep serving traffic throughout.
Now apply the updated manifest. Only the volumeClaimTemplates size changed:
apiVersion: apps/v1 kind: StatefulSet metadata: name: postgres namespace: databases spec: serviceName: postgres-headless replicas: 3 selector: matchLabels: app: postgres persistentVolumeClaimRetentionPolicy: whenDeleted: Retain whenScaled: Retain updateStrategy: type: RollingUpdate template: metadata: labels: app: postgres spec: securityContext: runAsNonRoot: true fsGroup: 26 seccompProfile: type: RuntimeDefault containers: - name: postgres image: registry.redhat.io/rhel9/postgresql-16:latest ports: - name: postgres containerPort: 5432 securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: ["ALL"] volumeMounts: - name: data mountPath: /var/lib/pgsql/data volumeClaimTemplates: - metadata: name: data labels: app: postgres spec: accessModes: ["ReadWriteOnce"] storageClassName: gp3-csi resources: requests: storage: 100Gi # was 50Gi kubectl -n databases apply -f statefulset.yaml kubectl -n databases rollout status sts/postgres --timeout=120s A critical detail: the existing PVCs must already be at 100Gi before you re-apply. If the template says 100Gi while data-postgres-0 is still 50Gi, the controller does nothing (it only creates missing PVCs), and you silently keep a mismatch that only surfaces the day you scale out.
Validate the template took effect by scaling out and back:
kubectl -n databases scale sts postgres --replicas=4 kubectl -n databases get pvc data-postgres-3 \ -o jsonpath='{.spec.resources.requests.storage}{"\n"}' # 100Gi kubectl -n databases scale sts postgres --replicas=3 Step 3: verify the filesystem actually followed
status.capacity says 100Gi. That is the kubelet's claim. Verify it from inside the container, because that is what your database sees:
kubectl -n databases exec postgres-0 -- df -h /var/lib/pgsql/data Filesystem Size Used Avail Use% Mounted on /dev/nvme2n1 98G 44G 54G 45% /var/lib/pgsql/data Cross-check the three layers — backend, PV, PVC:
PV=$(kubectl -n databases get pvc data-postgres-0 -o jsonpath='{.spec.volumeName}') kubectl get pv "$PV" -o jsonpath='{.spec.capacity.storage}{"\n"}' kubectl -n databases get pvc data-postgres-0 -o jsonpath='{.status.capacity.storage}{"\n"}' kubectl -n databases exec postgres-0 -- df -B1 --output=size /var/lib/pgsql/data | tail -1 If df still shows the old size while the PV shows the new one, the node phase did not run. Force it by rescheduling the Pod — the resize happens at mount time:
kubectl -n databases delete pod postgres-0 kubectl -n databases wait --for=condition=Ready pod/postgres-0 --timeout=300s For volumeMode: Block volumes there is no filesystem for Kubernetes to grow. The PV capacity changes, the block device gets bigger, and it is entirely your application's (or your init container's) job to extend the LVM PV / filesystem on top. Budget for that in your runbook.
When the expansion fails
If the backend rejects the new size (quota exceeded, max volume size), the PVC stays in Resizing with an event. On clusters where RecoverVolumeExpansionFailure is enabled, you can lower spec.resources.requests.storage back down — but never below status.capacity.storage. Check what the controller actually committed:
kubectl -n databases get pvc data-postgres-0 -o jsonpath='{.status.allocatedResourceStatuses}{"\n"}' kubectl -n databases describe pvc data-postgres-0 | sed -n '/Events/,$p' Without that feature gate, a failed expansion leaves the PVC stuck until the backend condition is fixed. This is precisely why you expand the least critical ordinal first.
Security best practices
Scope the RBAC. Resizing a PVC does not require cluster-admin. Create a dedicated role instead of handing out broad storage permissions:
apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: pvc-resizer namespace: databases rules: - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch", "patch", "update"] - apiGroups: ["apps"] resources: ["statefulsets"] verbs: ["get", "list", "watch"] Note that delete on statefulsets is deliberately absent — the orphan-delete step should go through a change-managed pipeline with four-eyes approval, not an ad-hoc shell.
Cap growth with a ResourceQuota. A typo turning 100Gi into 1000Gi is a four-figure cloud bill:
apiVersion: v1 kind: ResourceQuota metadata: name: storage-quota namespace: databases spec: hard: requests.storage: 500Gi gp3-csi.storageclass.storage.k8s.io/requests.storage: 500Gi persistentvolumeclaims: "12" Enforce a maximum per-PVC size with a ValidatingAdmissionPolicy (no webhook to operate, no single point of failure):
apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicy metadata: name: pvc-max-size spec: failurePolicy: Fail matchConstraints: resourceRules: - apiGroups: [""] apiVersions: ["v1"] operations: ["CREATE", "UPDATE"] resources: ["persistentvolumeclaims"] validations: - expression: "quantity(object.spec.resources.requests.storage).isLessThan(quantity('2Ti'))" message: "PVC requests above 2Ti require a platform-team exception" Keep reclaimPolicy: Retain on stateful workloads. The few orphaned PVs you clean up manually each quarter are cheap insurance against a cascading delete.
Encrypt at rest and keep it that way. An expansion never changes encryption parameters, but a migration (snapshot → restore into a new PVC) might silently drop kmsKeyId if your fallback StorageClass is not encrypted. Diff the StorageClass before restoring.
Audit the whole operation. delete statefulset --cascade=orphan and PVC patches should both land in your audit log with a named human. On OpenShift, check the API audit policy covers persistentvolumeclaims at RequestResponse level for the namespaces hosting production data.
Do not grant privileged pods to run resize2fs manually. If the node phase is broken, fix the CSI driver — do not spawn a privileged debug pod with nsenter on a production node. That is a lateral-movement vector you will not be able to justify in an audit.
Key takeaways
-
allowVolumeExpansion: trueon the StorageClass is a prerequisite, and it is mutable — you can turn it on for an existing class without touching existing volumes. - Expansion is a two-phase operation: controller (backend + PV capacity) then node (filesystem).
status.capacity.storageis the only field that tells you both phases completed. -
volumeClaimTemplatesis immutable. The only supported workaround iskubectl delete sts --cascade=orphanfollowed bykubectl applywith the updated template — Pods and PVCs survive untouched. - Set
persistentVolumeClaimRetentionPolicytoRetain/Retainbefore the orphan delete, and verify it. This single check is the difference between a routine maintenance and a restore from backup. - Expand the PVCs first, update the template second. The reverse order produces a silent size mismatch that only bites you on the next scale-out.
- Always verify with
dfinside the container, and prove the template is correct by scaling out one replica and inspecting the new PVC. - Shrinking is impossible. Size your initial claims with that in mind, and let a quota plus a
ValidatingAdmissionPolicyprotect you from the fat-finger in the other direction.


