Every cluster operator remembers Kubernetes 1.16 and 1.22: whole API groups disappeared overnight, Helm charts exploded, and kubectl apply started returning no matches for kind. Since then, teams built reflexes around deprecated API scanning, and that is good. The problem with Kubernetes 1.33 is that those reflexes are no longer sufficient.

1.33 removes no major GA API. If your upgrade plan is "run pluto, see zero removed APIs, click upgrade", you will pass the audit and still break the cluster — because in 1.33 the breakage moved to three other places: feature gates that no longer exist, kubelet default behaviour changes, and aggregated APIService endpoints that silently rot. This article builds a practical upgrade plan covering all four surfaces, with pluto and kubent for the API audit and a reproducible pre-production validation loop.

Why the failure modes shifted

Kubernetes has a predictable lifecycle policy: a feature gate that reaches GA stays recognized for two or three more minor releases, then the flag is removed from the binary. A removed gate is not ignored — the component refuses to start:

E0415 10:12:04.118312 1 run.go:74] "command failed" err="unrecognized feature gate: MinDomainsInPodTopologySpread" 

That is a crash loop on kube-apiserver or kubelet, on every node where the stale flag is baked into /var/lib/kubelet/config.yaml, a kubeadm ClusterConfiguration, or a Terraform/Ansible variable nobody has touched since 1.28. kubeadm upgrade plan does not audit your feature gates, and neither does pluto or kubent. This is the single most common way a 1.32 → 1.33 upgrade turns into an incident.

The second shift: the kubelet changed defaults. gitRepo volumes are disabled by default in 1.33 (behind the GitRepoVolumeDriver gate, off), kube-proxy nftables mode is GA, user namespaces (hostUsers: false) is beta and on by default, in-place pod resize is beta, and sidecar containers (initContainers with restartPolicy: Always) are GA. None of these are "removals" — they are behaviour changes that your workloads or your node images may depend on.

The third shift: v1.Endpoints is now formally deprecated in favour of EndpointSlice. Nothing breaks today, but any controller or service mesh still writing Endpoints objects is on borrowed time, and 1.33 is the right release to inventory them.

Step 1 — Audit the API surface with pluto and kubent

Both tools answer "which manifests reference an API version that is deprecated or removed in my target version", but they look in different places, so run both.

pluto: manifests, Helm releases, live objects

# install (Linux amd64) curl -sSL https://github.com/FairwindsOps/pluto/releases/download/v5.21.6/pluto_5.21.6_linux_amd64.tar.gz \ | tar -xz -C /usr/local/bin pluto # 1. static manifests in Git (the source of truth you can actually fix) pluto detect-files -d ./deploy --target-versions k8s=v1.33.0 -o wide # 2. Helm 3 releases stored as Secrets in the cluster pluto detect-helm --target-versions k8s=v1.33.0 -o markdown # 3. everything the API server still serves, including CRDs pluto detect-all-in-cluster --target-versions k8s=v1.33.0 -o wide 

pluto exits non-zero when it finds something (2 for deprecated, 3 for removed), which makes it a one-line CI gate:

pluto detect-files -d ./deploy --target-versions k8s=v1.33.0 --ignore-deprecations # fail the pipeline only on APIs already removed in the target version 

kubent: live cluster plus Helm, with a target version

sh -c "$(curl -sSL https://git.io/install-kubent)" kubent --target-version 1.33.0 --exit-error --output json | tee kubent-1.33.json # scan rendered manifests before they ever reach the cluster helm template my-app ./chart | kubent --cluster=false --filename - --target-version 1.33.0 

A key detail both tools share: they read the kubectl.kubernetes.io/last-applied-configuration annotation to recover the API version a client used, not the one the API server stores. Objects created by controllers or by kubectl create without that annotation will be invisible. That is why the next step matters more than the scanners.

Step 2 — Audit real traffic, not just stored objects

The API server exposes exactly which deprecated APIs are being requested, and in which release they disappear:

kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis # apiserver_requested_deprecated_apis{group="",removed_release="1.40",resource="endpoints",subresource="",version="v1"} 1 

Joined with request counts in Prometheus, you get the offending clients:

sum by (group, version, resource, removed_release, user_agent) ( increase(apiserver_request_total[7d]) * on (group, version, resource, subresource) group_left(removed_release) apiserver_requested_deprecated_apis ) 

If you do not have Prometheus on the control plane, add a narrow audit policy for the duration of the upgrade window:

apiVersion: audit.k8s.io/v1 kind: Policy omitStages: - RequestReceived rules: - level: Metadata resources: - group: "" resources: ["endpoints"] - group: "flowcontrol.apiserver.k8s.io" - group: "resource.k8s.io" - level: None 

On OpenShift, you get this for free and historically — use it:

oc get apirequestcounts -o custom-columns=\ NAME:.metadata.name,REMOVED:.status.removedInRelease,LASTHOUR:.status.requestCount \ | awk '$2 != "<none>"' oc get apirequestcount endpoints.v1 \ -o jsonpath='{range .status.currentHour.byNode[*]}{.byUser[*].username}{"\n"}{end}' | sort -u 

Step 3 — Audit feature gates, flags and APIServices

This is the part no scanner does for you.

Compare your configured gates against the 1.33 binary

# list every gate the 1.33 API server still knows about docker run --rm registry.k8s.io/kube-apiserver:v1.33.1 \ kube-apiserver --help 2>&1 | sed -n '/--feature-gates/,/^ *--/p' > gates-1.33.txt # what your control plane is actually configured with kubectl -n kube-system get cm kubeadm-config -o yaml | grep -A20 featureGates grep -o 'feature-gates=[^ ]*' /etc/kubernetes/manifests/kube-apiserver.yaml # and on every node grep -i featureGates -A10 /var/lib/kubelet/config.yaml 

Any gate present in your config and absent from gates-1.33.txt is a guaranteed crash loop. In practice, look hard at gates that reached GA around 1.30 (MinDomainsInPodTopologySpread, NodeOutOfServiceVolumeDetach, NewVolumeManagerReconstruction, CloudDualStackNodeIPs, StableLoadBalancerNodeSet, LegacyServiceAccountTokenCleanUp, KubeProxyDrainingTerminatingNodes…) — the GA + 3 releases rule lands precisely on 1.33.

You can also ask a running component what it has enabled:

kubectl get --raw /metrics | grep '^kubernetes_feature_enabled' | grep 'stage="ALPHA"' 

Clean up the kubeadm configuration first

Fix the configuration before running the upgrade, using the v1beta4 API (v1beta3 is deprecated):

apiVersion: kubeadm.k8s.io/v1beta4 kind: ClusterConfiguration kubernetesVersion: v1.33.1 featureGates: ControlPlaneKubeletLocalMode: true apiServer: extraArgs: - name: audit-log-path value: /var/log/kubernetes/audit.log - name: audit-policy-file value: /etc/kubernetes/audit-policy.yaml extraVolumes: - name: audit-policy hostPath: /etc/kubernetes/audit-policy.yaml mountPath: /etc/kubernetes/audit-policy.yaml readOnly: true pathType: File --- apiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration cgroupDriver: systemd # no stale featureGates block here 

Note ControlPlaneKubeletLocalMode: in 1.33 it is beta and on by default, making a joining control-plane node's kubelet talk to its local API server instead of the load balancer. If your LB health checks or firewall rules assumed otherwise, test it.

Do not ignore aggregated APIServices

An APIService whose backing Deployment is gone or whose CA bundle expired makes discovery fail. kubectl get all starts printing errors, controllers that enumerate resources stall, and kubeadm upgrade preflight can hang. Check before and after:

kubectl get apiservices -o custom-columns=\ NAME:.metadata.name,AVAILABLE:'.status.conditions[?(@.type=="Available")].status',\ REASON:'.status.conditions[?(@.type=="Available")].reason' | grep -v True 

A healthy aggregated API server declaration looks like this — explicit CA bundle, TLS verification on:

apiVersion: apiregistration.k8s.io/v1 kind: APIService metadata: name: v1beta1.metrics.k8s.io spec: group: metrics.k8s.io version: v1beta1 groupPriorityMinimum: 100 versionPriority: 100 insecureSkipTLSVerify: false caBundle: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0t... # base64 PEM, rotate with your CA  service: name: metrics-server namespace: kube-system port: 443 

Metrics-server, Prometheus adapter, cert-manager's cainjector outputs, Kyverno's reporting APIs and KEDA all register APIService objects. Bump them to versions that declare support for 1.33 before touching the control plane.

Step 4 — Validate in pre-production

Respect the skew rules

kubeadm upgrades one minor version at a time: 1.31 → 1.32 → 1.33. kubelet may be up to three minor versions older than the API server, kubectl one in either direction, and kube-proxy must never be newer than the API server. Upgrade control plane first, then nodes.

Rehearse on a disposable cluster

# snapshot etcd before anything kubectl -n kube-system exec etcd-cp-1 -- etcdctl \ --endpoints=https://127.0.0.1:2379 \ --cacert=/etc/kubernetes/pki/etcd/ca.crt \ --cert=/etc/kubernetes/pki/etcd/server.crt \ --key=/etc/kubernetes/pki/etcd/server.key \ snapshot save /var/lib/etcd/pre-1.33.db kubeadm certs check-expiration kubeadm config images list --kubernetes-version v1.33.1 kubeadm upgrade plan v1.33.1 kubeadm upgrade apply v1.33.1 --dry-run kubeadm upgrade apply v1.33.1 

Then the nodes, one at a time:

kubectl drain node-01 --ignore-daemonsets --delete-emptydir-data --timeout=300s # upgrade kubeadm/kubelet/kubectl packages, then: kubeadm upgrade node systemctl daemon-reload && systemctl restart kubelet kubectl uncordon node-01 kubectl get nodes -o wide 

Post-upgrade smoke tests that catch 1.33-specific regressions:

# kubeProxyVersion is no longer populated in 1.33 — fix any tooling that parses it kubectl get nodes -o jsonpath='{.items[*].status.nodeInfo.kubeProxyVersion}' # gitRepo volumes are off by default kubectl get pods -A -o json | jq -r '.items[] | select(.spec.volumes[]?.gitRepo) | "\(.metadata.namespace)/\(.metadata.name)"' # cgroup v1 nodes are in maintenance mode: plan the migration kubectl get nodes -o json | jq -r '.items[].status.nodeInfo.osImage' | sort -u stat -fc %T /sys/fs/cgroup/ # cgroup2fs expected # in-place resize is beta in 1.33 kubectl patch pod demo --subresource=resize \ --patch '{"spec":{"containers":[{"name":"app","resources":{"requests":{"cpu":"200m"}}}]}}' 

Make the audit continuous rather than a one-off ritual:

apiVersion: batch/v1 kind: CronJob metadata: name: api-deprecation-scan namespace: platform-tools spec: schedule: "0 6 * * 1" concurrencyPolicy: Forbid jobTemplate: spec: backoffLimit: 1 template: spec: serviceAccountName: deprecation-scanner restartPolicy: Never securityContext: runAsNonRoot: true runAsUser: 65532 seccompProfile: type: RuntimeDefault containers: - name: pluto image: us-docker.pkg.dev/fairwinds-ops/oss/pluto:v5.21.6 args: ["detect-all-in-cluster", "--target-versions", "k8s=v1.33.0", "-o", "json"] securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true capabilities: drop: ["ALL"] 

Security best practices

Give the scanners read-only, scoped access. pluto detect-helm and kubent read Helm release data stored in Secrets. A ClusterRole with get/list on secrets cluster-wide is a serious privilege. Scope it where you can, and never pipe scanner output to an external SaaS without reviewing it:

apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: deprecation-scanner rules: - apiGroups: ["*"] resources: ["*"] verbs: ["get", "list"] - apiGroups: [""] resources: ["secrets"] verbs: [] # grant per-namespace only if you need detect-helm 

Pin control plane images by digest and verify signatures with cosign verify --certificate-identity-regexp '.*' registry.k8s.io/kube-apiserver:v1.33.1 before promoting them into your internal registry. An upgrade window is a perfect time for a supply-chain mistake.

Keep audit logging on during the whole upgrade. The deprecation audit policy above is also your forensic trail if something goes wrong.

Rotate and verify certificates. kubeadm certs check-expiration before every upgrade; kubeadm upgrade apply renews control plane certs by default, which is a feature, not a reason to skip the check.

Use admission policy to stop the drift from coming back. Once you have eliminated Endpoints writes, enforce it:

apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingAdmissionPolicy metadata: name: no-manual-endpoints spec: failurePolicy: Fail matchConstraints: resourceRules: - apiGroups: [""] apiVersions: ["v1"] operations: ["CREATE", "UPDATE"] resources: ["endpoints"] validations: - expression: "request.userInfo.username.startsWith('system:')" message: "Endpoints is deprecated since 1.33; use EndpointSlice." 

Treat 1.33 defaults as hardening opportunities. User namespaces are beta and enabled by default: adding hostUsers: false to a pod spec maps container UID 0 to an unprivileged host UID, which neutralises a whole class of container escapes on kernels ≥ 6.3. Test it on non-critical workloads during the same window.

Key takeaways

  • 1.33 removes no major GA API, so a clean pluto/kubent report proves almost nothing on its own. Run them anyway — on Git manifests, Helm releases and the live cluster — and wire the exit codes into CI.
  • The real 1.33 killers are removed feature gates in kube-apiserver and kubelet configuration. Diff your configured gates against the 1.33 binary's --help output before you touch anything.
  • Use apiserver_requested_deprecated_apis (or apirequestcounts on OpenShift) to find clients, not just stored objects. Deprecated Endpoints usage is your 1.33 inventory task.
  • Check APIService availability before and after; a broken aggregation layer breaks discovery for every controller in the cluster.
  • Behaviour changes to validate explicitly: gitRepo off by default, kubeProxyVersion no longer populated, nftables proxy GA, sidecar containers GA, in-place resize and user namespaces beta, cgroup v1 in maintenance mode.
  • Rehearse the full kubeadm upgrade plan → --dry-run → apply → upgrade node sequence in pre-production, with an etcd snapshot and a documented rollback, one minor version at a time.

An upgrade plan built on these four audit surfaces — APIs, gates, kubelet behaviour, aggregation layer — turns a 1.33 migration from a gamble into a maintenance task.