6000 tokens from a vLLM server on a private GKE GPU node reached my Mac in 76.7 seconds through a plain kubectl port-forward, with nothing in between but Google's DNS-based endpoint for the cluster. The model is for me, so I didn't want an Ingress or a bastion VM for it. I tried port-forward through both the DNS-based endpoint and the Connect gateway.

The cluster is a Standard GKE cluster in us-east4, on 1.36.4-gke.2046003 when I ran these tests. Its IP endpoints are disabled and its DNS-based endpoint allows external traffic, so the only way in from outside is the DNS name, gated by IAM. The model runs on a spot a3-highgpu-1g node (one H100) with GPU time-sharing, behind a ClusterIP Service on port 8000.

Getting a kubeconfig for the DNS endpoint is one command:

gcloud container clusters get-credentials CLUSTER --location LOCATION --dns-endpoint

The kubeconfig it writes has no certificate-authority-data: the DNS endpoint serves a certificate from Google's public CA. Then:

kubectl -n llm port-forward svc/MODEL_SERVICE 8000:8000

and in a second terminal a streamed chat completion against http://127.0.0.1:8000/v1/chat/completions.

What a long stream does through the forward

The 6000 tokens came as 6002 chunks. The first chunk arrived after 0.50 s, and the longest gap between two chunks was 0.22 s. Then I left the forward alone for 90 seconds and sent a second request through it: 64 tokens in 1.3 s, first chunk after 0.46 s. A 3000-token run took 38.4 s. A plain HTTP probe through the same kind of endpoint later stayed up for 50 minutes.

kubectl 1.35 runs this forward over a WebSocket (portforward.k8s.io). If the WebSocket is refused, it falls back to SPDY without saying so, and I only noticed when a client of my own that speaks just the WebSocket got 403s that kubectl never showed.

RBAC for the forward, and what kubectl auth can-i really checks

If the person calling the model isn't a cluster admin, they need RBAC for the forward. On 1.36 the WebSocket forward needs both get and create on pods/portforward:

  • create only: 403 naming get
  • get only: 403 naming create
  • both: HTTP/1.1 200 OK

kubectl 1.35 works with create alone because it tries the WebSocket (a GET), gets the 403, and falls back to SPDY (a POST, 101 Switching Protocols).

kubectl auth can-i create pods/portforward answers no for an identity that can forward, because it reads pods/portforward as TYPE/NAME and checks create on a pod named portforward. Pass the subresource separately:

kubectl auth can-i create pods --subresource=portforward -n llm kubectl auth can-i get pods --subresource=portforward -n llm

Connect gateway: the same forward through the fleet

The Connect gateway needs the cluster registered to a fleet, and then the kubeconfig points at a Google API instead of the cluster:

gcloud container clusters update CLUSTER --location LOCATION --enable-fleet gcloud container fleet memberships get-credentials CLUSTER --location MEMBERSHIP_LOCATION

On my test cluster (same GKE version, europe-central2) that wrote the regional host https://europe-central2-connectgateway.googleapis.com/v1/projects/PROJECT_NUMBER/locations/europe-central2/gkeMemberships/CLUSTER.

The first kubectl call on my Mac failed:

getting credentials: exec: executable gke-gcloud-auth-plugin not found It looks like you are trying to use a client-go credential plugin that is not installed.

The plugin was installed. clusters get-credentials writes the plugin's absolute path into the kubeconfig, and fleet memberships get-credentials writes the bare name gke-gcloud-auth-plugin. With the Homebrew cask of the Google Cloud SDK, the plugin lives in the SDK's own bin directory, which isn't on PATH. Put it there:

export PATH="$(gcloud info --format='value(installation.sdk_root)')/bin:$PATH"

After that kubectl get ns listed the namespaces. Port-forward through the gateway is supported from kubectl 1.31 and cluster version 1.31, and both kubectl's forward and a raw v4.channel.k8s.io WebSocket got HTTP/1.1 200 OK.

How long the gateway keeps a forward

Google's Connect gateway page says long streams end after about 20 to 30 minutes ("it is an expected limitation due to security reasons"), and that there is "a quota limit of 10 active streams per fleet host project". A port-forward is one of those streams.

I kept one kubectl port-forward through the gateway to a Service open and sent a request through it every 10 seconds. 178 requests in a row got 200. At 30 minutes the forward died, and kubectl exited with status 1 after printing:

error: lost connection to pod

If you see it after exactly half an hour on a fleet cluster, look at the gateway before you look at the pod.

The same probe through the DNS endpoint of a second cluster (Autopilot, same GKE version) ran the full 50 minutes I gave it: 298 requests, all 200, and the forward was still up when I stopped it.

For a chat with a model that is fine. For a forward you leave open all afternoon, it isn't: kubectl doesn't reconnect, the process exits, and whatever points at localhost:8000 starts getting refused connections. Wrap it in a loop (while true; do kubectl port-forward …; done) if you need it to come back.

When the gateway says Forbidden and nothing else

A second account of mine with roles/container.clusterViewer but no gateway role runs the same kubectl and gets:

Error from server (Forbidden): unknown (get pods)

That message names no permission. The raw response from the gateway does:

Permission 'gkehub.gateway.get' denied on resource 'projects/…/gkeMemberships/CLUSTER' (or it may not exist)

The fix is a gateway role on the fleet host project plus the RBAC above inside the cluster.

roles/gkehub.gatewayReader is not enough for port-forward, and the error you get with it is worse than the one above. I gave a test service account gatewayReader and the RBAC for the forward. kubectl get pods worked. kubectl port-forward printed error: error upgrading connection: unable to upgrade connection: followed by a Google HTML error page, Error 400 (Bad Request)!!1, and nothing else. With -v=6 the real refusal shows up one line earlier, on the WebSocket attempt:

websocket: bad handshake (403 Forbidden): Permission 'gkehub.gateway.stream' denied on resource 'projects/…/gkeMemberships/CLUSTER'

kubectl then falls back to SPDY, the gateway answers that with a 400, and the 400 is what you see. Google's page says the Connect Gateway Admin role (roles/gkehub.gatewayAdmin) "is the only predefined role that contains the gkehub.gateway.stream permission, which is required to run kubectl CLI commands like attach, exec, port-forward, and cp." You don't need the whole admin role. A custom role with the reader's three gkehub permissions plus that one was enough, and the forward answered 200:

gcloud iam roles create gatewayPortForward --project PROJECT \ --title "Connect gateway port-forward" \ --permissions=gkehub.gateway.generateCredentials,gkehub.gateway.get,gkehub.memberships.get,gkehub.gateway.stream

Keep in mind that the same permission also allows exec, attach and cp through the gateway, so RBAC is what keeps a model user out of a shell in the pod.

Which one I use

  • DNS endpoint: one flag on the cluster, IAM decides who gets in, no fleet. The forward held a long stream, a 90-second pause, and 50 minutes of requests. Turn it on only if you're comfortable with the control plane answering on a public DNS name for anyone holding valid Google credentials and IAM.
  • Connect gateway: works for any fleet member, including clusters in other projects, with no network exposure of the control plane. You pay with a fleet registration, the plugin PATH trap on a Mac, a two-layer permission error, and forwards that end at 30 minutes.

If neither is allowed in your organization, the private endpoint through an IAP bastion still works. kubectl then talks to a local forward at 127.0.0.1, which the control plane's certificate does not list, so the kubeconfig needs tls-server-name: kubernetes.

What I built for this

Ostgate 0.0.8 adds GKE. Use with kubectl… adds a cluster to ~/.kube/ostgate, and kubectl, k9s and other tools then sign in through Ostgate, without gcloud or gke-gcloud-auth-plugin. Private clusters work through the DNS endpoint, the Connect gateway, a VPN or a bastion VM over IAP; Ostgate picks the path and shows it. Forward Port… on a Service keeps listening while pods are replaced, and can answer on the Service's own name, such as checkout.shop.svc.cluster.local. GPU nodes show how busy their GPUs are, a vLLM server shows its model, port and the Secret that holds its key, and a forward to it offers Copy curl, which reads the key from OPENAI_API_KEY in your shell. It runs on Apple Silicon with macOS 15 or later, with a 7-day trial. Setup: https://ostgate.app/docs/gke

I'm the developer of Ostgate (a paid macOS app; not affiliated with Google).