I used EKS Pod Identity for cert-manager, the EBS CSI driver and Vault, then ended up with a plain IAM user for Elasticsearch snapshots. Here is what failed, how to tell which identity your pod really used, and what has changed upstream since.

TL;DR: EKS Pod Identity is simpler than IRSA, and I use it wherever the software supports it. Elasticsearch’s bundled repository-s3 plugin was the exception. On Elasticsearch 8.15.0, IRSA silently fell back to the node role and Pod Identity was rejected by the bundled AWS SDK, so snapshots run on a narrowly scoped IAM user stored in the Elasticsearch keystore. Newer Elasticsearch versions behave differently, so test yours before copying my choice.

Figure 1. The three credential paths an Elasticsearch pod can use for S3 snapshots, and what happened on each.

Why I Chose Pod Identity First

I built a centralized logging platform on EKS (Elasticsearch through ECK, Filebeat, Vault, cert-manager, ingress-nginx). Every component that talks to AWS needs credentials without static keys, which on EKS means IRSA or Pod Identity.

Pod Identity removes most of the ceremony from IRSA:

The last row is where this post gets interesting. Pod Identity is only simpler if the software inside your pod can actually use it.

Where It Worked: cert-manager, EBS CSI and Vault

For cert-manager (Route 53 DNS-01 challenges), the EBS CSI driver and Vault’s AWS access, Pod Identity was a clear improvement. The trust policy trusts the EKS service principal, and the association ties the role to a namespace and ServiceAccount:

# Simplified pattern, not the full module resource "aws_eks_pod_identity_association" "cert_manager" { cluster_name = var.cluster_name namespace = "cert-manager" service_account = "cert-manager" role_arn = aws_iam_role.cert_manager.arn } 

{ "Effect": "Allow", "Principal": { "Service": "pods.eks.amazonaws.com" }, "Action": ["sts:AssumeRole", "sts:TagSession"] } 

There is no OIDC provider to manage, no annotations to forget, and the same role definition works across clusters. These components use modern AWS SDKs that natively support the Pod Identity credential provider.

Where It Broke: Elasticsearch Snapshots

For Elasticsearch, I wanted the same pattern: snapshots to S3 with no static credentials. I tried both federated options in order, and both attempts ran on Elasticsearch 8.15.0.

Attempt 1: IRSA

On 8.15.0, snapshot repository registration failed with AccessDenied. The detail that mattered was the identity named in the error: it was the EC2 node’s instance role, not the IRSA role. The pod had never assumed the role at all. Three problems stacked on top of each other:

1. No dedicated ServiceAccount by default. ECK does not create a dedicated ServiceAccount per Elasticsearch resource unless you set one in podTemplate. Without one carrying the IRSA annotation, the EKS mutating webhook has nothing to inject credentials into, and the SDK falls back to the node instance role.
2. A sandboxed token path. Elastic documents that the plugin reads the web identity token from config/repository-s3/aws-web-identity-token-file, because a repository cannot read files outside its config directory. Having AWS_ROLE_ARN injected is not enough on its own.
3. Token refresh failures. Reports against Elasticsearch describe the plugin failing to pick up refreshed tokens after the first one expires: snapshots work, then fail hours later until the node restarts (elastic/elasticsearch#115186).

Attempt 2: Pod Identity

Pod Identity is the newer mechanism, so I tried it next. The snapshot repository check failed with an error that points straight at the bundled SDK:

The full URI (http://169.254.170.23/v1/credentials) contained within environment variable AWS_CONTAINER_CREDENTIALS_FULL_URI has an invalid host. Host can only be one of [localhost, 127.0.0.1] 

When you create a Pod Identity association, EKS injects AWS_CONTAINER_CREDENTIALS_FULL_URI pointing at the link-local address 169.254.170.23. The older AWS Java SDK v1 bundled with Elasticsearch 8.15.0 only accepts localhost or 127.0.0.1 there, so it rejects the endpoint. This is tracked in elastic/cloud-on-k8s#8320, where an ECK maintainer explains that ECK is not setting the variable and the limitation sits in Elasticsearch itself.

What I Deployed: A Scoped IAM User in the Keystore

Because both federated paths were dead ends on 8.15.0, I used Elastic’s documented fallback: an IAM user with access to nothing except the snapshot bucket, loaded into the Elasticsearch keystore through spec.secureSettings:

# Simplified fragment of the Elasticsearch resource spec: secureSettings: - secretName: es-s3-snapshot-credentials # holds s3.client.default.access_key and secret_key 

It is a deliberate, documented exception to "no static credentials", not an oversight. The trade-off is a long-lived key that you must scope tightly and rotate yourself.

How to Tell Which Identity Your Pod Really Used

The most transferable lesson from this build is how to debug pod credential chains. When an AWS call from a pod fails with AccessDenied, read the caller ARN in the error message before touching IAM policies. If it names the node instance role, federation never kicked in, and no policy edit on your workload role will fix it.

Next, inspect what the platform actually injected into the pod:

# 1. Check for IRSA injection kubectl exec -n <ns> <pod> -- env | grep -E 'AWS_ROLE_ARN|AWS_WEB_IDENTITY_TOKEN_FILE' # 2. Check for Pod Identity injection kubectl exec -n <ns> <pod> -- env | grep -E 'AWS_CONTAINER_CREDENTIALS_FULL_URI|AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE' # 3. Verify the Pod Identity association exists in the EKS control plane aws eks list-pod-identity-associations --cluster-name <cluster> 

Those three commands separate "the platform did not inject credentials" (missing ServiceAccount, missing association, or a pod created before the association existed) from "the application ignored or rejected them" (outdated SDK or a custom credential path).

One hardening tip: setting the IMDSv2 hop limit to 1 on your nodes stops regular pods from reaching the instance metadata service, so a failed federation produces a clear error instead of a silent fall back to the node role. Pods on the host network are the exception, and any workload that legitimately needs the metadata service will need a closer look first.

What Has Changed Upstream Since

I ran 8.15.0, and the picture has moved on. Check these before copying my workaround:

• Token refresh: in the IRSA token refresh issue, an Elasticsearch maintainer says the underlying "connection pool shut down" failure is fixed from 8.18.0 (#115186).
• AWS SDK v2: Elastic’s breaking-changes notes list the repository-s3 plugin moving from AWS SDK v1 to v2 in 9.1.0 (#126843, release notes). AWS SDK v2 is the line that understands Pod Identity credentials, which is the root cause of my attempt 2 failure.
• Pod Identity on 9.1.0: a commenter in elastic/cloud-on-k8s#8320 reports that Pod Identity works on 9.1.0 if you symlink the token into the plugin’s config directory and set the two container credential variables yourself. I have not tested this workaround:

# Reported in the issue thread, untested by me containers: - name: elasticsearch env: - name: AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE value: /usr/share/elasticsearch/config/repository-s3/eks-pod-identity-token - name: AWS_CONTAINER_CREDENTIALS_FULL_URI value: http://169.254.170.23/v1/credentials initContainers: - name: symlink-token command: - sh - -c - mkdir -p /usr/share/elasticsearch/config/repository-s3 && ln -s /var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token /usr/share/elasticsearch/config/repository-s3/eks-pod-identity-token 

If you upgrade across 9.1.0, read the SDK v2 notes first. They say you must set the signing region (s3.client.default.region) and that SDK v2 does not support the older IMDSv1 protocol.

My IAM user was the right call for 8.15.0. On a current release, testing federated credentials first is worth it, because removing a static key is a real security gain.

A Practical Decision Guide

Takeaways

• Pod Identity is the better default, provided the container’s SDK supports it. Verify SDK compatibility before migrating stateful or Java-heavy workloads.
• Read the failing caller ARN first. An AccessDenied naming the EC2 node role is a credential delivery problem, not an IAM policy problem.
• Document deliberate exceptions. My repository’s troubleshooting guide explains why snapshots use an IAM user, so a future maintainer does not "fix" it back to a broken association without testing.
• Pin advice to software versions. Workarounds expire, so write down the exact version that forced the compromise.

Project:

terraform/infrastructure/environments/<env> -> EKS and AWS resources terraform/platform/environments/<env> -> Kubernetes and Helm resources 

cp terraform/infrastructure/environments/dev/backend.hcl \ terraform/backend-dev-infrastructure.hcl cp terraform/platform/environments/dev/backend.hcl \ terraform/backend-dev-platform.hcl

Troubleshooting guide: [docs/troubleshooting.md]

Have you made the leap to EKS Pod Identity yet, or are you still sticking with IRSA? Let me know what tripped you up in the comments!