What moving Kubernetes workloads from GKE to DigitalOcean taught me about ingress, DNS and data.

Moving an app from one Kubernetes cluster to another sounds easy. Build the new cluster, deploy, switch DNS, switch off the old one.

On paper, yes. In practice, that's where the trouble starts.

I've worked with Kubernetes on AWS, Azure and Google Cloud. One migration moved workloads from Google Kubernetes Engine (GKE) to DigitalOcean Kubernetes (DOKS). The manifests were never the hard part. The hard part was making sure the new environment could carry the same workloads, networking and storage before any traffic moved.

Here's why that matters. Every pod can report healthy while users still can't reach the app. A service can exist with no working ingress. A load balancer can be ready while DNS still points at the old cluster. And a database app can look fine right up until you find the storage never moved.

None of that shows up when you run kubectl apply against a new cluster.

Kubernetes is portable. Your cluster isn't.

The Kubernetes API is the same everywhere. What sits under it is not.

GKE and DOKS both run Kubernetes, but their networking, storage, load balancing and identity are different. A deployment may point at a StorageClass that doesn't exist on the new side. An ingress may rely on cloud-specific load balancer annotations. A workload may use Google Workload Identity, which has no direct match in the destination.

So before moving anything, I take stock of what the app depends on:

# Workloads, networking and storage kubectl get deployments,statefulsets,daemonsets -A kubectl get svc,ingress -A kubectl get pvc,pv -A kubectl get hpa,pdb -A # Storage classes and Helm releases kubectl get storageclass helm list -A # Versions and APIs (deprecated APIs bite on newer clusters) kubectl version kubectl api-resources 

Helm matters here because a chart can hide a lot. One release may create a Service, a ConfigMap, a ServiceAccount and several workloads together.

One thing I won't do is export every resource and apply it to the new cluster. Those exports carry API-generated fields like UIDs and resource versions. I'd rather use the original Helm charts, Kustomize overlays or version-controlled manifests.

I also list the rest: Secrets, NetworkPolicies, service accounts, external databases, queues, DNS records and TLS certificates. The goal isn't to copy the old cluster. It's to reproduce what the app needs.

Build the new cluster in code, and check it's ready

I provision the destination with Terraform. It's repeatable, and I can review changes before they run.

resource "digitalocean_kubernetes_cluster" "platform" { name = "platform-cluster" region = "nyc1" version = var.kubernetes_version node_pool { name = "application-pool" size = "s-4vcpu-8gb" node_count = 3 } } 

That's an illustrative config, not a production copy. Node size should come from what the workload really needs.

But a cluster that provisions successfully isn't an app-ready cluster. Before deploying, I ask a few plain questions. Is there enough CPU and memory? Can pods pull their images? Do the StorageClasses exist? Is the ingress controller installed? Can the cluster reach its external dependencies?

Helm helps too, with one catch. It makes a deployment repeatable. It doesn't make it portable. Values like ingress class, storage and external endpoints often differ between clouds.

# values.yaml (simplified) replicaCount: 3 image: repository: registry.example.com/application tag: "1.4.2" service: type: ClusterIP port: 8080 ingress: enabled: true className: nginx hosts: - host: app.example.com paths: - path: / pathType: Prefix 

The image tag and port can stay. The ingress, storage and secrets often can't. So I keep a values file per cloud instead of editing the chart by hand:

helm/application/ Chart.yaml templates/ values.yaml values-gke.yaml values-doks.yaml 

Then I render the manifests and check them before installing anything:

helm template application ./helm/application -f values-doks.yaml # Compare against what's really deployed today helm get values application -n production -a 

That last command often shows config that never made it into the repo.

Test ingress before you touch DNS

This is the step I watch most closely.

Ingress sits between the internet and your services. Get it wrong, and a perfect deployment is still unusable. Annotations that work on one controller can be ignored on another. Even two NGINX setups can behave differently if the versions differ.

Before cutover, I check that the controller is ready, its Service has an external endpoint, TLS certificates are valid, and host and path routing work.

kubectl get ingress -A kubectl get svc -n ingress-nginx kubectl get endpointslice -A kubectl logs -n ingress-nginx \ -l app.kubernetes.io/component=controller --tail=200 

Namespace and labels depend on how you installed the controller.

My favourite trick is testing the new endpoint with the real hostname before any DNS change:

curl --resolve app.example.com:443:203.0.113.10 \ https://app.example.com/health 

That IP is a documentation example. The --resolve flag makes curl connect to a specific address while still using the real hostname for TLS and routing. Many ingress rules depend on the hostname, so hitting the bare load balancer IP won't prove much.

And don't stop at /health. A 200 there doesn't tell you that login, database queries or file uploads work. Test what users actually do.

DNS is slower than you think

Once the new side is ready, DNS usually moves the traffic. But DNS changes aren't instant. Resolvers and clients cache records, and some hold on longer than they should.

So I lower the TTL ahead of time, say to 60 seconds. That shortens the caching window, though it doesn't guarantee every client switches at once. Because of this, I never shut down the old cluster the moment DNS changes. It stays up while the new one proves itself.

My sequence looks like this:

  1. Provision the destination cluster
  2. Deploy applications and validate internal services
  3. Validate ingress and TLS
  4. Run smoke tests
  5. Prepare data sync
  6. Update DNS
  7. Monitor both environments
  8. Confirm the destination is stable
  9. Only then, decommission the source cluster

Stateless is easy. Stateful is where the pain lives.

A stateless API can often run in both clusters at once, as long as both reach the same dependencies. A database-backed StatefulSet is a different story.

A PVC in GKE doesn't turn into a usable PVC in DOKS. The data has to be migrated or replaced. Sometimes the database is external and stays put. Then the job is mostly network access and connection settings. If the data itself moves, you're looking at replication, backup and restore, or a write freeze. The right choice depends on your recovery point and recovery time targets.

A backup-and-restore move may need a maintenance window. Replication can shorten the cutover, but it brings lag, consistency and promotion to worry about.

I wouldn't promise zero downtime on a stateful migration until I'd worked out how the data stays consistent. Zero downtime is a goal you prove with testing and monitoring. Kubernetes doesn't hand it to you.

Green pods are not a finish line

Pod health and app health are different things. A readiness probe can pass while a dependency is failing. A Service can have endpoints while the ingress is misconfigured.

So I check in layers:

# Nodes kubectl get nodes -o wide kubectl top nodes # Workloads kubectl get pods -A kubectl get deployments -A # Services and endpoints kubectl get svc -A kubectl get endpointslice -A 

Then comes the layer that matters most, which is real user flows. That means login, API calls, database operations and background jobs. I also compare source and destination on request throughput, error rate, P95 latency, pod restarts, database connection errors and ingress errors.

If the pods are green but errors are climbing, the migration isn't done.

Plan the rollback before the cutover

"Point DNS back" is not a rollback plan.

Say users create records in the new environment after cutover. Point DNS back to the old one, and those records aren't there. Rollback has to cover data, and not only routing.

So before the migration, I agree on the triggers and write them down:

Rollback if: - Critical application functionality fails - Error rate exceeds the agreed threshold - P95 latency exceeds the agreed limit - Data consistency checks fail - Required dependencies become unavailable 

The thresholds should come from the app's normal behaviour and its service targets. I also decide who can call the rollback. During a production cutover, those decisions shouldn't be made on the fly.

I'd automate the evidence. Terraform for the cluster, Helm or GitOps for the app, CI checks on the manifests, automated smoke tests and monitoring checks. For a high-risk migration, though, I'd keep the final cutover as an explicit human approval.

What I'd do again, and what I'd change

  • Build the destination early. Find out about a missing StorageClass or a bad annotation weeks before the cutover, and not during it.
  • Change one thing at a time. Same app version on both sides. New app version, new schema, new ingress and new cloud all at once makes troubleshooting a nightmare.
  • Test with realistic traffic. Health checks catch the obvious. Real requests catch the rest.
  • Budget for the overlap. Running both clusters for a while costs money, and it's worth it.
  • Treat data as its own workstream. Don't bury it in a deployment checklist.
  • Keep the old cluster alive until the new one is proven and the rollback window has passed.

Final thoughts

Kubernetes gives you a consistent way to deploy apps. It doesn't make every cloud the same. The API may be portable. Your networking, storage, identity and load balancing assumptions may not be.

If the first time you test ingress is after the DNS change, you've left too much to chance. If you find out mid-cutover that your storage isn't portable, the plan wasn't finished.

A good migration isn't about getting pods running somewhere else. It's about moving the service without losing the behaviour, reliability and data users already count on.

That's the gap between a successful deployment and a successful migration.