Kubernetes CronJob pg_dump backup: is it on the same disk as the database?

There is no error message in this report. kubectl get jobs showed every backup Job as Complete 1/1, and kubectl get etcdsnapshotfile listed five snapshots, each with a LOCATION that starts with file://. If you arrived by searching kubernetes cronjob pg_dump backup, k3s etcd-snapshot, --etcd-s3, "does k3s etcd snapshot include persistent volume data", "k3s etcd snapshot stored on same node" or "local-path pvc backup node failure data loss", this is the state you are probably looking at. The cause in one line: five of our seven dump volumes and all five etcd snapshots were on the same single disk as the data they copy, and nothing left the cluster. This article gives you six checks to run on your own cluster with the value that means "same problem", the results of restoring six of the seven dumps (one needed four attempts and produced three different errors; the largest was only integrity-checked), and what we changed and did not change. Environment: a two-node k3s cluster, v1.33.5+k3s1, one server node with embedded etcd and one agent node, storage from the built-in local-path provisioner; all times are UTC.

What we saw

Seven backup CronJobs exist, six running pg_dump and one running mysqldump, each dumping one database server, scheduled daily between 02:50 and 03:50. All seven succeeded on 2026-11-11.

$ kubectl get jobs -A NAMESPACE NAME STATUS COMPLETIONS DURATION AGE prod-a app-a-postgres-backup-29906115 Complete 1/1 9m7s 22h prod-b app-b-postgres-backup-29906130 Complete 1/1 4s 21h prod-b app-b-postgres-3-backup-29906130 Complete 1/1 4s 21h prod-b app-b-postgres-2-backup-29906130 Complete 1/1 4s 21h prod-c app-c-postgres-backup-29906150 Complete 1/1 4s 21h prod-d app-d-mysql-backup-29906090 Complete 1/1 11s 22h prod-e app-e-postgres-backup-29906100 Complete 1/1 4s 22h (newest Job of each CronJob shown; the two older Jobs of each are also Complete 1/1; columns after AGE cut) 

The job logs showed a dump file written and no error:

$ kubectl logs -n prod-e job/app-e-postgres-backup-29906100 2026-11-11T03:00:00.896827010Z Starting pg_dump 20261111_030000 2026-11-11T03:00:01.689579227Z -rw-r--r-- 1 root root 61.3K Nov 11 03:00 /backup/app_e_20261111_030000.sql.gz 2026-11-11T03:00:01.691470394Z Backup completed 

Prometheus showed the same: at 01:27 on 2026-11-12 the seconds since the last successful run (time() - kube_cronjob_status_last_successful_time{cronjob=~".*backup.*"}) were between 77,854 and 81,447 for all seven, which is inside one day.

Every backup had succeeded within the last day. No alert rule read this value; of the two rules that covered the backup Jobs, one counted failed Jobs and one timed Jobs that were still running.

Kubernetes CronJob pg_dump backup: where the dumps are

Each CronJob writes <name>_<timestamp>.sql.gz to /backup, which is a PersistentVolumeClaim served by the local-path provisioner, and then deletes files older than 3 days (5 days for app-a-postgres, 7 for app-c-postgres). No command in any of the seven sends a file outside the cluster: no s3, rclone, restic, scp or rsync.

If your question is "how to find my persistent volume location", the volume list answers it: it shows which node each dump volume and each data volume is on, and on that node the directory is under /var/lib/rancher/k3s/storage:

$ kubectl get pv -o custom-columns=... (the command of check 2 below, with the class column named SC and a SIZE column added) NS PVC NODE RECLAIM SC SIZE prod-a app-a-postgres-backup-pvc agent-1 Delete local-path 5Gi prod-a data-app-a-postgres-0 server-1 Delete local-path-server-1 10Gi prod-b app-b-postgres-3-backup-pvc agent-1 Delete local-path-server-1 5Gi prod-b app-b-postgres-backup-pvc server-1 Delete local-path-server-1 5Gi prod-b app-b-postgres-3-data-app-b-postgres-3-0 server-1 Delete local-path-server-1 10Gi prod-b app-b-postgres-2-data-app-b-postgres-2-0 server-1 Delete local-path-server-1 10Gi prod-b app-b-postgres-data-app-b-postgres-0 server-1 Delete local-path-server-1 10Gi prod-b app-b-postgres-2-backup-pvc server-1 Delete local-path-server-1 5Gi prod-c app-c-postgres-data-app-c-postgres-0 server-1 Delete local-path-server-1 10Gi prod-c app-c-postgres-backup-pvc server-1 Delete local-path-server-1 5Gi prod-d mysql-data-app-d-mysql-0 server-1 Delete local-path-server-1 20Gi prod-d mysql-backup-app-d-mysql-0 server-1 Delete local-path-server-1 10Gi prod-e app-e-postgres-backup-pvc server-1 Delete local-path-server-1 10Gi prod-e app-e-postgres-data-app-e-postgres-0 server-1 Delete local-path-server-1 30Gi (14 of 32 rows: the backup volumes and the volumes of the databases they copy) 

Five of the seven dumps are on the same node as the database they copy. server-1 has one physical disk, a SATA SSD of 465.8 G with no RAID, and one LVM volume mounted at /. The etcd database, the etcd snapshots, the local-path volumes, /var/lib/kubelet, /var/log and /tmp are all on that one filesystem. So "same node" here means "same disk".

Sizes of the newest dumps on 2026-11-11: app-a-postgres 1,024,335,407 bytes (its pg_dump took 9 min 4 s), app-d-mysql 15,777,210, app-c-postgres 2,434,541, app-b-postgres 411,735, app-e-postgres 62,739, app-b-postgres-2 11,176 and app-b-postgres-3 8,287.

We found no copy outside the cluster. One limit of that statement: root crontabs and timers on the two hosts were not checked, because ssh to server-1 was refused on 2026-11-12. "Nothing leaves the node" is confirmed for everything Kubernetes runs and unverified for anything installed by hand on the hosts.

k3s etcd-snapshot: where the snapshots are, and --etcd-s3

$ kubectl get etcdsnapshotfiles -o wide NAME SNAPSHOTNAME NODE LOCATION SIZE CREATIONTIME local-etcd-snapshot-server-1-<unix time>-578428 etcd-snapshot-server-1-<unix time> server-1 file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-server-1-<unix time> 41775136 2d1h local-etcd-snapshot-server-1-<unix time>-f34892 etcd-snapshot-server-1-<unix time> server-1 file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-server-1-<unix time> 43311136 37h local-etcd-snapshot-server-1-<unix time>-6ee0dd etcd-snapshot-server-1-<unix time> server-1 file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-server-1-<unix time> 43311136 25h local-etcd-snapshot-server-1-<unix time>-3bcb04 etcd-snapshot-server-1-<unix time> server-1 file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-server-1-<unix time> 43311136 13h local-etcd-snapshot-server-1-<unix time>-4c0364 etcd-snapshot-server-1-<unix time> server-1 file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-server-1-<unix time> 45232160 85m (the unix time in each name replaced by <unix time>) 

The k3s unit on server-1 has no --etcd-s3 argument and there is no /etc/rancher/k3s/config.yaml (read on 2026-11-11 05:08).

Restore tests

On 2026-11-12 between 01:28 and 01:33 we restored the newest dump of six of the seven databases and integrity-checked the seventh. The method was one throwaway Job per dump in the dump's namespace: the dump volume mounted read-only, an empty database server initialised in an emptyDir, the dump piped into psql -v ON_ERROR_STOP=1 (or mysql), then table counts and exact row counts. The source counts were read with the database's own client inside the database pod.

The source rows were counted about a day after the dump was written, so the source figure is higher wherever the application wrote in between. The table counts are the comparison that has to match, and they do for all six.

Restore tests of the newest dump of each database, 2026-11-12.

app-a-postgres was not restored because its dump is 1.0 GB compressed and the node it sits on had 17.6 GB free; only its integrity was checked. "Restorable" is therefore confirmed for six of seven and unverified for the largest one.

Restoring produced three findings that no listing of Jobs, volumes or snapshots shows.

Finding 1: the dump is not the whole server. app-a-postgres holds four databases: app_a at 1,899 MB and three more at 123 MB, 67 MB and 8,428 kB. The CronJob ran pg_dump -d app_a only. The three other databases had no copy anywhere. The other six servers hold one application database each.

Finding 2: the app-c-postgres dump did not restore into an empty server. Four attempts, same dump file each time:

Attempt 1, image postgres:16-alpine (the image the backup itself uses):

-- Dumped from database version 16.4 (Debian 16.4-1.pgdg110+2) -- Dumped by pg_dump version 16.11 RESTORE FAILED ERROR: role "app_c" does not exist tables|0 

Attempt 2, same image, role created first:

CREATE ROLE RESTORE FAILED ERROR: extension "postgis" is not available DETAIL: Could not open extension control file "/usr/local/share/postgresql/extension/postgis.control": No such file or directory. HINT: The extension must first be installed on the system where PostgreSQL is running. tables|0 

Attempt 3, the database's own image postgis/postgis:16-3.4 (PostgreSQL 16.4), role created:

CREATE ROLE RESTORE FAILED invalid command \restrict tables|0 

Attempt 4, image postgis/postgis:16-3.5-alpine (psql 16.15), role created:

CREATE ROLE psql restore exit 0 restore seconds 10 tables|149 rows_exact|121040 

(Each block is cut to the lines that differ between attempts; the directory listing and the initdb warnings are removed.)

Finding 3: one script would exit 0 even if the dump failed. The app-d-mysql backup ran mysqldump … | gzip > file under /bin/sh -c with no set -e and no pipefail. No failed run was observed; this is read from the script. The six pg_dump scripts start with set -euo pipefail.

Why

The dump volumes are directories on a node. The k3s documentation says of its built-in storage: "By default, persistent volumes are created on the host filesystem under /var/lib/rancher/k3s/storage" (docs.k3s.io/add-ons/storage). The provisioner's README says the volumes it creates "use the kubernetes.io/hostname label for node affinity. This ensures the volume is only accessible on the node where it was provisioned" (local-path-provisioner README). A backup that ends with a file in such a volume ends on one node's disk.

The StorageClass name did not decide the node. We have two StorageClasses, local-path (default) and local-path-server-1. Both are rancher.io/local-path with volumeBindingMode: WaitForFirstConsumer. The second is named after a node, but nothing in it pins a volume to that node: it has no allowedTopologies and the same path parameter. Four of the eight volumes on agent-1 are of class local-path-server-1. The two exceptions in the table above have different causes:

• app-a-postgres-backup has a nodeSelector for agent-1. Its manifest carries a comment written on 2026-07-02 that says the other tenants' backups all sit on the database's node and this one is deliberately placed on the other machine.
• app-b-postgres-3-backup has no nodeSelector and uses the class local-path-server-1. Its volume is on agent-1. My inference from WaitForFirstConsumer is that the first pod to use the claim was scheduled there; the placement was not a decision.

Eight days after that comment was written, on 2026-07-10, app-c-postgres-backup was added with a nodeSelector for server-1, on the database's node again. The comment of 2026-07-02 recorded that the other backups share a node with their database, and the manifest of 2026-07-10 placed one more backup the same way. The app-d-mysql dump volume is a volumeClaimTemplate of the database StatefulSet itself, so it is always on the database's node.

The etcd snapshots are at the k3s defaults. From docs.k3s.io/cli/etcd-snapshot: "Scheduled snapshots are enabled by default, at 00:00 and 12:00 system time, with 5 snapshots retained per server node on local disk", in ${data-dir}/server/db/snapshots, and "K3s etcd snapshots are stored on the node file system, and may optionally be uploaded to an S3-compatible object store for disaster recovery scenarios." Without --etcd-s3, the optional part does not happen. A separate cleanup CronJob of ours also deletes all but the five newest files in that directory every day at 03:00.

An etcd snapshot does not contain volume data. The same page describes a snapshot as "a complete copy of the etcd datastore, along with cluster certificate authority (CA) certificates and private keys". The page does not contain the words "persistent volume" at all. The snapshot holds the API objects, which include the PersistentVolume and PersistentVolumeClaim objects. That it holds none of the bytes inside a volume is my inference from what etcd stores; third-party posts state it, the k3s documentation does not.

A snapshot restored on a new host needs the server token. From docs.k3s.io/datastore/backup-restore: "you must also back up the server token file at /var/lib/rancher/k3s/server/token", and "If you do not use the same token value when restoring, the snapshot will be unusable". The token file is on the same disk as the snapshots. The k3s unit passes --token explicitly, so the value existed somewhere else at install time; whether a copy exists off the node today is unverified.

What one disk failure on server-1 would remove. This is inferred from the facts above, not observed. It would take the etcd database, all five etcd snapshots and the server token; 24 of 32 volumes, which include every database server in the cluster except three on agent-1 (two PostgreSQL servers and a Qdrant); and the dumps of five of the seven databases that have a dump. The dumps of app-b-postgres-3 and app-a-postgres on agent-1 would survive, the second covering one of that server's four databases. Across both nodes, eighteen volumes are neither a dump target nor a dumped database. Among them are six production databases with no backup CronJob at all (three PostgreSQL servers, a Neo4j, an OpenSearch and a Qdrant; four of the six are on server-1), and on server-1 the container registry (50 Gi) and an object store (50 Gi).

What sits on each disk. Everything in the left box is lost together if that disk fails.

A deleted claim deletes the directory. All 32 PersistentVolumes had persistentVolumeReclaimPolicy: Delete at 01:25. The Kubernetes documentation: "Volumes that were dynamically provisioned inherit the reclaim policy of their StorageClass, which defaults to Delete" (Persistent Volumes). The provisioner's README: "The teardown script is run after the volume is deleted, to cleanup the volume directory on the node." Our production applications are Argo CD Applications with automated: {prune: true, selfHeal: true}. My inference: a manifest change that removes a claim would be pruned automatically, and with Delete the data directory would go with it, dump volumes included.

Why nothing reported it. A Job is Complete when its container exits 0. The alert rules that mention jobs were the chart defaults KubeJobFailed (kube_job_failed > 0) and KubeJobNotCompleted, and one application rule. No rule evaluated where a dump is written, the dump size, or the age of the last success.

Why the three restore errors happened.

• ERROR: role "app_c" does not exist: of the six pg_dump CronJobs only this one lacked --no-owner --no-privileges, so the dump contains owner and grant statements for a role that an empty server does not have.
• ERROR: extension "postgis" is not available: the database uses PostGIS and the plain postgres:16-alpine image does not ship it.
• invalid command \restrict: PostgreSQL 16.10, released 2025-08-14 for CVE-2025-8714, added a psql \restrict command "that prevents execution of further meta-commands" and changed pg_dump "to issue that before any data coming from the source server" (PostgreSQL 16.10 release notes). A psql older than 16.10 does not have the command. The dump was written by pg_dump 16.11 from the backup image and attempt 3 read it with psql 16.4 from the database server's own image.

The backup image tag postgres:16-alpine is mutable, and our two nodes hold different builds of it: dumps written on server-1 have the header line "Dumped by pg_dump version 16.11" and dumps written on agent-1 have 16.13 in that line.

Why that script would exit 0. My inference from the script: the exit status of the pipeline is gzip's, so a failed mysqldump would still leave a 20-byte gzip file, and the script would go on to prune old dumps and exit 0.

Restore time itself is small here: the largest restore we ran took 59 s. I infer from this that recovery time for these databases is dominated by finding the dump and a compatible client, not by the restore.

What to check in your cluster now

All of these are read-only except the last, which creates one throwaway Job.

1. Where are the etcd snapshots? (k3s with embedded etcd)

kubectl get etcdsnapshotfile 

Same problem if: every LOCATION starts with file:// and none with s3://. Those snapshots are on the server node's own disk, next to the etcd database they copy. If there is no such resource, the cluster is not k3s or RKE2 with embedded etcd, or is older than the releases that added it.

2. Which node holds each volume, and what happens when its claim is deleted?

kubectl get pv --sort-by=.spec.claimRef.namespace -o custom-columns='NS:.spec.claimRef.namespace,PVC:.spec.claimRef.name,NODE:.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0],RECLAIM:.spec.persistentVolumeReclaimPolicy,CLASS:.spec.storageClassName' 

Same problem if: the row for a backup PVC and the row for the database's data PVC show the same NODE, and that node has one disk. Read NODE, not CLASS: a StorageClass name does not decide the node. RECLAIM Delete means deleting the PVC deletes the directory on the node.

3. Does any backup job send anything out of the cluster?

kubectl get cronjob -A -o json | jq -r '.items[] | select(.metadata.name | test("backup|dump")) | [.metadata.namespace, .metadata.name, ([.spec.jobTemplate.spec.template.spec.containers[] | ((.command // []) + (.args // [])) | join(" ")] | join(" ") | test("s3|rclone|restic|gsutil|azcopy|mc cp|scp|rsync") | tostring)] | @tsv' 

Same problem if: the third column is false for a job whose dump PVC is node-local. It is a text match on the job's command, so true only means "look closer". Ours was false for all seven.

4. Does the dump contain every database on the server?

kubectl exec -n <ns> <postgres-pod> -- sh -c 'psql -U "$POSTGRES_USER" -d postgres -Atc "select datname, pg_size_pretty(pg_database_size(datname)) from pg_database where not datistemplate"' 

Same problem if: the list is longer than the -d arguments in the backup script. pg_dump -d one_db dumps one database.

5. Does the script exit non-zero when the dump fails?

kubectl get cronjob -A -o json | jq -r '.items[] | select(.metadata.name | test("backup|dump")) | [.metadata.namespace, .metadata.name, ([.spec.jobTemplate.spec.template.spec.containers[] | ((.command // []) + (.args // [])) | join(" ")] | join(" ") | test("pipefail") | tostring)] | @tsv' 

Same problem if: false and the script is dump | gzip > file. Without pipefail the pipeline's exit status is gzip's, so a failed dump still ends in a Complete Job.

6. Does the newest dump restore into an empty server?

Create a Job in the dump's namespace that mounts the dump PVC read-only, starts an empty server in an emptyDir, restores with ON_ERROR_STOP and counts tables. This is the PostgreSQL version; use the image the backup uses (uid 70 is postgres in the alpine image).

apiVersion: batch/v1 kind: Job metadata: {name: restore-test, namespace: <ns>} spec: backoffLimit: 0 ttlSecondsAfterFinished: 3600 activeDeadlineSeconds: 900 template: spec: restartPolicy: Never securityContext: {runAsUser: 70, runAsGroup: 70, fsGroup: 70} containers: - name: restore image: postgres:16-alpine command: ["/bin/sh", "-c"] args: - | set -eu F=$(ls -t /backup/*.sql.gz | head -1); echo "latest $F" gzip -t "$F" initdb -D /work/pg -U postgres >/dev/null pg_ctl -D /work/pg -o "-c listen_addresses='' -c unix_socket_directories=/work" -w start >/dev/null createdb -h /work -U postgres restored zcat "$F" | psql -h /work -U postgres -d restored -v ON_ERROR_STOP=1 -q psql -h /work -U postgres -d restored -Atc "select count(*) from pg_tables where schemaname not in ('pg_catalog','information_schema')" volumeMounts: - {name: backup, mountPath: /backup, readOnly: true} - {name: work, mountPath: /work} volumes: - {name: backup, persistentVolumeClaim: {claimName: <dump-pvc>, readOnly: true}} - {name: work, emptyDir: {}} 

Same problem if: the Job fails, or the table count differs from the source. Read the source count in the database pod with the same select count(*) from pg_tables … query, run through kubectl exec as in check 4. The three errors we met and what each means:

Checks 1, 2 and 3 are what kubegap automates. We had run kubegap 0.2.0 on this cluster on 2026-11-11 at 05:08, the day before the restore tests. Its report stated that 5 of 7 backups write to the source database's node, that the etcd snapshots are all file:// with no --etcd-s3, that 6 production databases have no backup CronJob, and that 24 of 24 production volumes were Delete. It also listed "whether a dump was ever restored" under what it had not seen. All of those statements matched what we found by hand the next day. It did not report the three restore findings: the databases missing from a dump, the dump that needs a role, an extension and a newer client, and the script that exits 0 when the dump fails. kubegap does not open dumps or connect to databases, so those came only from checks 4, 5 and 6.

What we changed and what we have not

Changed on 2026-11-12

Change reclaim policy to Retain. At 01:35 we patched persistentVolumeReclaimPolicy to Retain on 30 PersistentVolumes: every application namespace, the registry and default. The logging and monitoring volumes were left at Delete. No pod restarted.

# 2026-11-12T01:35:27Z before 32 Delete # 2026-11-12T01:35:39Z after 2 Delete 30 Retain 

All 32 stayed Bound. This protects against a deleted claim, not against a failed disk. New volumes will still be created with Delete, because the StorageClass field is immutable.

Backup scripts. One manifests commit, pushed at 01:36, changed three CronJobs:

• the app-a-postgres backup now loops over every non-template database on the server instead of dumping one;
• the app-d-mysql backup now runs under bash with set -euo pipefail;
• the app-c-postgres backup now passes --no-owner --no-privileges, and a comment in the manifest states the restore conditions: an image with PostGIS and a psql of 16.10 or newer.

Alert. A second commit added a PrometheusRule named BackupCronJobStale:

expr: time() - kube_cronjob_status_last_successful_time{cronjob=~".*backup.*"} > 26h for: 10m labels: severity: warning 

The new rule compares the time of the last success with the current time; the two rules that existed before evaluate Job objects.

Verification. After Argo CD synced, at 01:39 the three CronJob specs contained the changes, and we started one Job from each. All three completed:

$ kubectl logs -n prod-c job/verify-app-c-postgres-backup 2026-11-12T01:39:49.620160742Z Starting pg_dump 20261112_013949 2026-11-12T01:39:52.198996704Z Backup done: /backup/app_c_20261112_013949.sql.gz $ kubectl logs -n prod-d job/verify-app-d-mysql-backup 2026-11-12T01:39:58.703582767Z -rw-r--r-- 1 root root 16M Nov 12 01:39 app-d-mysql-20261112-013950.sql.gz (the warning line and four older files cut) 

The app-a-postgres run with the new loop wrote four files where the old script wrote one (timestamps converted to UTC, database names replaced):

$ kubectl logs -n prod-a job/verify-app-a-postgres-backup 01:39:50 Starting pg_dump app_a 20261112_013950 01:49:32 -rw-r--r-- 1 root root 976.9M Nov 12 01:49 /backup/app_a_20261112_013950.sql.gz 01:49:32 Starting pg_dump events 20261112_013950 01:49:33 -rw-r--r-- 1 root root 3.0K Nov 12 01:49 /backup/events_20261112_013950.sql.gz 01:49:33 Starting pg_dump traces 20261112_013950 01:49:38 -rw-r--r-- 1 root root 4.4M Nov 12 01:49 /backup/traces_20261112_013950.sql.gz 01:49:38 Starting pg_dump analytics 20261112_013950 01:49:45 -rw-r--r-- 1 root root 7.5M Nov 12 01:49 /backup/analytics_20261112_013950.sql.gz 01:49:45 Backup completed 

The new app-c-postgres dump was then restored into an empty postgis/postgis:16-3.5-alpine server with no role created first: exit 0 in 8 s, 149 tables, 122,884 rows, the same counts as the source. At 01:49 the Prometheus rules API reported BackupCronJobStale with health ok and state inactive.

Not changed

No dump leaves the cluster yet. This is the main gap and it is still open. Sending dumps out needs a destination outside both nodes, which means object storage, and credentials for it. That has not been decided.

--etcd-s3 is not configured. It needs the same destination, a change to the k3s server arguments on server-1, and a restart of the only API server in the cluster.

The six databases without a backup CronJob still have no dump.

The server token has not been confirmed to exist off the node.

The app-a-postgres dump has not been restored end to end.

Where else this happened

These three are deletions by automation, which is what our Delete reclaim policy and Argo CD pruning allowed until 2026-11-12. In each, the outcome depended on whether a copy existed somewhere else and had already been restored once.

• InfluxData, 2022: a merged change made Argo CD delete a whole production environment. Volumes had been set to be retained and the data was also in object storage, so data loss was zero.
• Skyscanner, 2021: a templating change made Argo CD delete all 478 services in every region. Their written lesson was that a backup only counts once it has been restored.
• Spotify, 2019: Terraform deleted two production GKE clusters in one evening. Hourly cluster backups whose restore had already been tested were essential to the recovery.

More: 13 public cases where resources were deleted in the Incident Library.

Timeline

FAQ

kubernetes cronjob pg_dump backup: is a Complete Job enough?

No. A Complete Job only means the script exited 0. Check which node holds the dump volume with kubectl get pv, whether any command sends the file out of the cluster, and whether the newest dump restores into an empty server. Five of our seven dumps were on the same disk as their database.

does k3s etcd snapshot include persistent volume data

No, by my reading. The k3s documentation describes a snapshot as a complete copy of the etcd datastore plus the cluster CA certificates and private keys, and never mentions persistent volumes. etcd stores the PersistentVolume and PersistentVolumeClaim objects, not the files inside a volume. Volume contents need their own backup.

k3s etcd snapshot stored on same node

That is the default. k3s takes snapshots at 00:00 and 12:00 and keeps five per server node on local disk under ${data-dir}/server/db/snapshots. Run kubectl get etcdsnapshotfile: if every LOCATION starts with file://, the snapshots share a disk with etcd. Uploading elsewhere requires the --etcd-s3 options.

local-path pvc backup node failure data loss

A local-path volume is a directory on one node, bound to that node by node affinity, so it is lost when that node's disk fails. If the backup PVC is local-path on the same node as the database, both are lost together. List each volume's NODE with kubectl get pv and compare the backup PVC with the data PVC.

psql: error: invalid command \restrict

The dump was written by pg_dump 17.6, 16.10, 15.14, 14.19, 13.22 or newer, which emits a \restrict line, and an older psql is reading it. We hit it restoring a pg_dump 16.11 dump with psql 16.4 from the database's own image. Restore with a psql at least as new as the pg_dump.

ERROR: role "…" does not exist pg_dump restore

The dump contains owner and grant statements for a role that the target server does not have. Either create the role before restoring, or write the dump with --no-owner --no-privileges. In our cluster one of the six pg_dump CronJobs lacked those flags, and only an actual restore into an empty server showed it.

ERROR: extension "postgis" is not available

The image you are restoring into does not ship an extension that the source database uses. A plain postgres:16-alpine image has no PostGIS. Restore with an image that includes the extension, such as a postgis/postgis tag, and check its psql version too, because ours was too old for the dump.

Originally published at benchlet.com.