I built a clean private-ingress edge on AWS: CloudFront out front for TLS, WAF, and Shield; a private Network Load Balancer as the origin; no public load balancer anywhere. Everything provisioned without error. Then the first request came back 504 Gateway Timeout.

The fix was a single security-group rule. The reason is a mental model about CloudFront VPC origins that's easy to get wrong, so it's worth writing down.

The architecture

I used CloudFront VPC origins, the feature that lets CloudFront connect to a private resource in your VPC, so you don't expose a public ALB/NLB at all:

The NLB is pure L4 passthrough, so the origin TLS rides straight through to the application. WAF runs at the edge. No public entry point.

The symptom

Every request through CloudFront returned 504 Gateway Timeout. Not 502, not 403, a timeout, which means CloudFront opened a connection to the origin and got nothing back.

The usual 504 suspects all checked out:

  • NLB target group was healthy.
  • The app was up and reachable from inside the VPC.
  • CloudFront's origin config (domain, port, protocol, SNI) was correct.
  • Security group existed and allowed HTTPS.

That last one was the trap.

The root cause

The NLB's security group allowed inbound 443 from the VPC CIDR:

# WRONG, the intuitive-but-broken rule ingress { from_port = 443 to_port = 443 protocol = "tcp" cidr_blocks = [var.vpc_cidr] # "it's a VPC origin, so traffic is in-VPC… right?" }

That reasoning is exactly backwards. A CloudFront VPC origin does not reach your NLB from inside the VPC. CloudFront connects to the origin from CloudFront's public, origin-facing IP ranges, and because an NLB preserves the client source IP (it doesn't SNAT), the security group sees those public CloudFront addresses, never your VPC CIDR.

So the SG, scoped to the VPC CIDR, silently dropped every CloudFront connection. CloudFront waited, got nothing, and returned 504. "VPC origin" had quietly convinced us the source was internal; it isn't.

The fix

AWS publishes a managed prefix list for exactly this, "_com.amazonaws.global.cloudfront.origin-facing_" containing CloudFront's origin-facing ranges. Allow that on the origin's security group instead of the VPC CIDR:

data "aws_ec2_managed_prefix_list" "cloudfront_origin_facing" { name = "com.amazonaws.global.cloudfront.origin-facing" } resource "aws_security_group" "origin_nlb" { name = "edge-origin-nlb" vpc_id = var.vpc_id ingress { description = "HTTPS from CloudFront origin-facing ranges (VPC origin presents a PUBLIC source, not the VPC CIDR)" from_port = 443 to_port = 443 protocol = "tcp" prefix_list_ids = [data.aws_ec2_managed_prefix_list.cloudfront_origin_facing.id] } egress { description = "To the backend targets" from_port = 443 to_port = 443 protocol = "tcp" cidr_blocks = [var.vpc_cidr] # egress to your app IS in-VPC, this part is fine } }

Using the managed prefix list (rather than hardcoding IPs) means AWS keeps the ranges current for you, CloudFront's IP space changes, and you don't want to chase it.

After applying this, the same request returned 200. End to end, nothing else changed.

How to diagnose this yourself

If a CloudFront VPC origin times out, run this check in order:

  1. Confirm it's a connectivity drop, not the app. Target group healthy + app reachable from a bastion inside the VPC, but CloudFront still 504s? Suspect the SG.
  2. Look at the origin SG's inbound rule. If it references your VPC CIDR (or a subnet), that's almost certainly it.
  3. Remember NLBs preserve source IP. Whatever is talking to your NLB, the SG sees its real address, for a VPC origin, that's CloudFront's public origin-facing range.
  4. Replace the CIDR rule with the com.amazonaws.global.cloudfront.origin-facing prefix list. Re-test.

The one-line mental model: CloudFront VPC origin ≠ in-VPC source. Treat your origin NLB's security group as if it faces the public internet, because from the SG's point of view, it does.

Source