Direct Connect is the right answer when you need deterministic latency, committed bandwidth and a contractual SLA. It is also a 3-month lead time, a cross-connect fee, a port charge starting around $220/month for a 1 Gbps hosted connection, and usually a second one for redundancy. Plenty of teams need a reliable hybrid link, not a dedicated one, and they need it next Tuesday.

This article walks through a production-grade alternative: two redundant WireGuard tunnels between an on-premise datacenter and an AWS Transit Gateway, with BGP via FRRouting for dynamic routing and sub-second failover. We will cover sizing, MTU arithmetic (the part that silently destroys your throughput), the AWS-side plumbing, how OpenShift clusters on both ends have to be configured, and how to actually test the failover instead of hoping.

First, be honest about the alternatives

Before building anything custom, check AWS Site-to-Site VPN. It costs ~$0.05/hour per connection (~$36/month), gives you two tunnels, BGP, and AWS operates half of it. It is the correct default.

You build a self-managed WireGuard interconnect when at least one of these is true:

  • Throughput per flow. An AWS VPN tunnel caps at ~1.25 Gbps per tunnel and single flows do not ECMP across tunnels. WireGuard on a c6gn.xlarge sustains 3–5 Gbps aggregate with multiple flows and is cheaper per Gbps.
  • Crypto and key control. You want ChaCha20-Poly1305, 1-RTT handshakes, and keys you rotate yourself — not IKEv2 with a provider-managed lifecycle.
  • Multi-cloud symmetry. The same WireGuard + FRR appliance pattern works against Azure, GCP and a colo, with one operational model.
  • Overlay for Kubernetes. You already run WireGuard for pod-level encryption and want one toolchain.

If none of those apply, stop reading and run aws ec2 create-vpn-connection.

Target architecture

Two WireGuard appliances in AWS, in two Availability Zones, in a dedicated "transit" VPC. Two gateways on-premise, in different racks and ideally behind different ISPs. Four tunnels in a full mesh is tempting; two tunnels (DC-GW-A ↔ AWS-GW-A, DC-GW-B ↔ AWS-GW-B) plus BGP is simpler to reason about and sufficient.

The non-obvious part is how the AWS appliances inject routes into the Transit Gateway. A plain EC2 instance cannot BGP-peer with a TGW. Transit Gateway Connect can: it exposes a GRE transport over an existing VPC attachment and runs BGP inside it, specifically for third-party appliances. That is the piece that turns a pair of EC2 boxes into a real, dynamically-routed hybrid edge.

 on-prem 10.10.0.0/16 Internet transit VPC 10.42.0.0/16 ┌──────────────┐ ┌──────────────┐ │ dc-gw-a │══ WireGuard UDP/51820 ═════════════│ aws-gw-a │─ GRE+BGP ─┐ │ FRR + BFD │ │ FRR + BFD │ │ └──────────────┘ └──────────────┘ ┌──┴──┐ ┌──────────────┐ ┌──────────────┐ │ TGW │ │ dc-gw-b │══ WireGuard UDP/51820 ═════════════│ aws-gw-b │─ GRE+BGP ─┘ └──────────────┘ └──────────────┘ │ spoke VPCs / OpenShift

Sizing

WireGuard is multi-threaded across peers and flows, but a single flow is handled by a limited set of kernel workers. Rules of thumb from real deployments:

  • Budget ~1 Gbps per modern vCPU for ChaCha20-Poly1305 at 1440-byte packets. AES-NI does not help; WireGuard does not use AES.
  • Prefer instances with high packet-per-second ceilings and ENA Express: c6gn.large (~3 Gbps network baseline, Graviton2) is the sweet spot for 1–2 Gbps links; c6in.xlarge if you need x86 for other tooling.
  • Small packets kill you. 1 Gbps of 64-byte packets is ~1.9 Mpps and will saturate a c6gn.large long before the bandwidth budget. Measure pps, not just bps.
  • Enable multi-queue and pin IRQs: ethtool -L ens5 combined 4 and keep irqbalance running.

Data transfer out to the internet is $0.09/GB — identical to a VPN, and roughly 4× the cost of Direct Connect egress. At sustained >50 TB/month, Direct Connect wins on pure economics. Do that calculation before committing.

MTU: the arithmetic that must be exact

WireGuard overhead on IPv4 is 60 bytes: 20 (outer IPv4) + 8 (UDP) + 32 (WireGuard header, counter and Poly1305 tag). On a 1500-byte internet path, your tunnel MTU is 1440. If the outer path is IPv6, the overhead is 80 and the MTU is 1420. If the on-prem uplink is PPPoE (1492), it drops to 1432.

Pick 1420 and stop thinking about it. The 20 bytes you lose cost ~1.4% of throughput; a black-holed PMTUD path costs you a Friday night.

# Verify the real path MTU to the AWS public IP before choosing ping -M do -s 1472 -c 3 203.0.113.10 # 1472 + 28 = 1500 ping -M do -s 1412 -c 3 203.0.113.10 # falls back here if the path is 1440

Two more traps:

  1. VPC instances use MTU 9001. A 9001-byte packet from an EC2 spoke to on-prem hits the WireGuard appliance and must be fragmented or rejected. Either set the appliance's VPC-facing route MTU, or — better — clamp TCP MSS and make sure ICMP fragmentation-needed is not filtered by your security groups or NACLs.
  2. PMTUD is unreliable over the internet. Clamp MSS on the appliance for everything crossing the tunnel. 
# nftables on both ends: clamp MSS and allow ICMP PTB nft add table inet fw nft add chain inet fw forward '{ type filter hook forward priority 0; }' nft add rule inet fw forward tcp flags syn tcp option maxseg size set rt mtu nft add rule inet fw forward icmp type destination-unreachable accept

For OpenShift clusters whose pods must reach the other side: pod traffic is decapsulated on the node before it leaves, so the relevant constraint is pod MTU <= 1420. With OVN-Kubernetes and Geneve (100 bytes of overhead), a node at 1500 gives a cluster MTU of 1400 — which fits. The failure case is an OpenShift cluster in AWS with 9001-byte nodes and a cluster MTU of 8901: those pods cannot talk to on-prem. Run an MTU migration:

apiVersion: operator.openshift.io/v1 kind: Network metadata: name: cluster spec: migration: mtu: network: from: 8901 to: 1400 machine: to: 1500

Then apply the final value once all nodes have rebooted, and remove the migration block. Check progress with oc get mcp and oc get nodes -o jsonpath='{.items[*].status.nodeInfo}'.

WireGuard configuration

Keys live in a secret store, never in git. Generate per-host:

umask 077 wg genkey | tee /etc/wireguard/private.key | wg pubkey > /etc/wireguard/public.key wg genpsk > /etc/wireguard/psk-aws-gw-a.key # optional pre-shared key, see security section

/etc/wireguard/wg0.conf on dc-gw-a:

[Interface] Address = 169.254.21.1/30 ListenPort = 51820 MTU = 1420 PrivateKey = <dc-gw-a private key> Table = off # FRR owns the routing table, not wg-quick [Peer] PublicKey = <aws-gw-a public key> PresharedKey = <psk> Endpoint = 203.0.113.10:51820 AllowedIPs = 169.254.21.0/30, 10.42.0.0/16, 10.0.0.0/8 PersistentKeepalive = 15

Table = off is critical: without it, wg-quick installs routes derived from AllowedIPs and fights with FRR. AllowedIPs remains the cryptographic ACL — a packet whose source is not in that list is dropped on decrypt, regardless of what BGP advertises. Keep it as tight as the aggregates you actually expect.

PersistentKeepalive = 15 keeps NAT and stateful firewall state alive and gives BFD a stable transport.

systemctl enable --now wg-quick@wg0 wg show wg0 latest-handshakes

BGP and BFD with FRRouting

WireGuard has no link-state signal: if the remote endpoint dies, the interface stays UP and packets vanish. BFD is not optional.

/etc/frr/frr.conf on dc-gw-a (AS 65010, AWS side AS 64512):

frr version 9.1 log syslog informational ! router bgp 65010 bgp router-id 10.10.0.1 bgp bestpath as-path multipath-relax neighbor 169.254.21.2 remote-as 64512 neighbor 169.254.21.2 description aws-gw-a neighbor 169.254.21.2 bfd neighbor 169.254.21.2 password <bgp-auth-secret> neighbor 169.254.21.2 timers 10 30 ! address-family ipv4 unicast network 10.10.0.0/16 neighbor 169.254.21.2 soft-reconfiguration inbound neighbor 169.254.21.2 route-map FROM-AWS in neighbor 169.254.21.2 route-map TO-AWS out neighbor 169.254.21.2 maximum-prefix 100 restart 5 exit-address-family ! bfd peer 169.254.21.2 interface wg0 detect-multiplier 3 receive-interval 300 transmit-interval 300 exit ! ip prefix-list AWS-SPACE seq 5 permit 10.42.0.0/16 le 24 ip prefix-list AWS-SPACE seq 10 permit 10.60.0.0/14 le 24 ip prefix-list DC-SPACE seq 5 permit 10.10.0.0/16 ! route-map FROM-AWS permit 10 match ip address prefix-list AWS-SPACE route-map TO-AWS permit 10 match ip address prefix-list DC-SPACE

On dc-gw-b, the same config with set as-path prepend 65010 65010 in TO-AWS and set local-preference 90 in FROM-AWS if you want a deterministic active/standby. For active/active, leave them symmetric and enable maximum-paths 2.

With 300 ms × 3, convergence is ~900 ms plus BGP withdrawal. That is the number to quote to application teams for TCP session survivability.

Wiring the AWS side: Transit Gateway Connect

The appliances sit in a VPC attached to the TGW. The Connect attachment rides on top of that attachment.

# 1. Connect attachment on top of the existing VPC attachment aws ec2 create-transit-gateway-connect \ --transport-transit-gateway-attachment-id tgw-attach-0a1b2c3d4e5f \ --options Protocol=gre # 2. One Connect peer per appliance (gives ECMP and redundancy) aws ec2 create-transit-gateway-connect-peer \ --transit-gateway-attachment-id tgw-attach-0connect1111 \ --peer-address 10.42.1.10 \ --inside-cidr-blocks 169.254.100.0/29 \ --bgp-options PeerAsn=65010 # 3. Retrieve the TGW-side GRE endpoint and inside addresses aws ec2 describe-transit-gateway-connect-peers \ --query 'TransitGatewayConnectPeers[].ConnectPeerConfiguration'

On the appliance:

ip link add gre-tgw type gre local 10.42.1.10 remote 10.42.0.200 ttl 64 ip link set gre-tgw mtu 8977 up ip addr add 169.254.100.2/29 dev gre-tgw # BGP peers with the two TGW inside addresses (169.254.100.3 and .4)

Each Connect peer carries up to 5 Gbps; up to four peers per attachment ECMP to 20 Gbps. TGW supports BFD on Connect peers, so the AWS leg converges as fast as the on-prem leg.

Three items that break this setup in practice:

  • Disable source/destination check on every appliance ENI: aws ec2 modify-instance-attribute --instance-id i-0... --no-source-dest-check.
  • VPC route tables in the spokes must send 10.10.0.0/16 to the TGW. Route propagation from the TGW attachment handles this if the attachment is associated with a propagating route table; otherwise add static entries.
  • Security groups on the appliances must allow UDP/51820 from the on-prem public IPs, and GRE (protocol 47) is handled by the TGW attachment — no SG rule needed for the Connect transport.

Enable forwarding and make it survive reboots:

cat >/etc/sysctl.d/99-router.conf <<'EOF' net.ipv4.ip_forward=1 net.ipv4.conf.all.rp_filter=2 net.ipv6.conf.all.forwarding=1 EOF sysctl --system

Running the gateway on OpenShift

If your on-prem edge is already an OpenShift cluster, run the WireGuard/FRR gateway as a DaemonSet pinned to two dedicated nodes rather than maintaining pet VMs.

oc create namespace hybrid-edge oc create sa wg-gateway -n hybrid-edge oc adm policy add-scc-to-user privileged -z wg-gateway -n hybrid-edge oc label node edge-01 edge-02 node-role.kubernetes.io/hybrid-edge= 
apiVersion: apps/v1 kind: DaemonSet metadata: name: wg-gateway namespace: hybrid-edge spec: selector: matchLabels: {app: wg-gateway} template: metadata: labels: {app: wg-gateway} spec: serviceAccountName: wg-gateway hostNetwork: true hostPID: false dnsPolicy: ClusterFirstWithHostNet nodeSelector: node-role.kubernetes.io/hybrid-edge: "" tolerations: - key: node-role.kubernetes.io/hybrid-edge operator: Exists effect: NoSchedule containers: - name: frr-wireguard image: quay.io/frrouting/frr:9.1.0 command: ["/usr/lib/frr/docker-start"] securityContext: privileged: true capabilities: add: ["NET_ADMIN", "SYS_MODULE", "NET_RAW"] volumeMounts: - {name: frr-config, mountPath: /etc/frr} - {name: wg-config, mountPath: /etc/wireguard, readOnly: true} - {name: modules, mountPath: /lib/modules, readOnly: true} resources: requests: {cpu: "2", memory: 512Mi} limits: {cpu: "4", memory: 1Gi} volumes: - name: frr-config configMap: {name: frr-config} - name: wg-config secret: {secretName: wg-keys, defaultMode: 0400} - name: modules hostPath: {path: /lib/modules, type: Directory}

The wg-keys secret should be produced by External Secrets from Vault or AWS Secrets Manager, not committed:

apiVersion: external-secrets.io/v1beta1 kind: ExternalSecret metadata: name: wg-keys namespace: hybrid-edge spec: refreshInterval: 1h secretStoreRef: {name: vault-backend, kind: ClusterSecretStore} target: {name: wg-keys, creationPolicy: Owner} dataFrom: - extract: {key: hybrid-edge/wireguard}

Testing failover for real

A failover you have not triggered does not exist. Three scenarios, in increasing nastiness:

# 1. Clean shutdown of one appliance sudo systemctl stop frr # BGP sends NOTIFICATION, convergence <1s # 2. Black hole: interface stays up, packets vanish (the realistic failure) sudo nft add rule inet fw input udp dport 51820 drop # BFD must detect it in ~900ms. Measure with a flood ping from a client. # 3. Brownout: 2% loss and 80ms added latency sudo tc qdisc add dev ens5 root netem loss 2% delay 80ms

Measure during the event, from a host behind the tunnel:

# Packet loss window during failover sudo ping -i 0.1 -D 10.42.5.20 | ts # Watch route withdrawal in real time sudo vtysh -c 'show bgp ipv4 unicast summary' sudo vtysh -c 'show bfd peers brief' ip route get 10.42.5.20

Acceptance criteria worth writing into a runbook: < 1.5 s of loss on link failure, no asymmetric routing afterwards (conntrack -L | grep ASSURED on the standby should stay empty), and automatic re-preemption when the primary returns.

Throughput validation, always with multiple streams:

iperf3 -c 10.42.5.20 -P 8 -t 60 -O 5 iperf3 -c 10.42.5.20 -u -b 900M -l 1372 # UDP at tunnel MSS, check for fragmentation

Observability

Export WireGuard and FRR metrics and alert on the two signals that matter: handshake age and BGP session state.

apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: hybrid-edge-rules namespace: hybrid-edge spec: groups: - name: wireguard rules: - alert: WireGuardHandshakeStale expr: time() - wireguard_latest_handshake_seconds > 240 for: 2m labels: {severity: critical} annotations: summary: "No WireGuard handshake with {{ $labels.peer }} for 4 minutes" - alert: HybridLinkDegraded expr: count(frr_bgp_peer_state{state="established"}) < 2 for: 1m labels: {severity: warning} annotations: summary: "Only one BGP session up — hybrid link has no redundancy" - alert: TunnelThroughputSaturated expr: rate(node_network_transmit_bytes_total{device="wg0"}[5m]) * 8 > 2.5e9 for: 10m labels: {severity: warning}

A stale handshake older than 180 s means WireGuard considers the peer dead; 240 s is a safe alert threshold with PersistentKeepalive=15.

Security best practices

  • Pre-shared keys. Add PresharedKey on every peer. It costs nothing, mixes an extra symmetric secret into the handshake and provides hedging against future quantum attacks on Curve25519. Rotate it with the same cadence as your TLS certs.
  • Key rotation without downtime. WireGuard accepts multiple peers; add the new public key as a second peer with the same AllowedIPs, shift the endpoint, then remove the old peer. Automate it, and keep private keys only in Vault/Secrets Manager with 0400 file mode.
  • Treat AllowedIPs as a firewall. It is the only cryptographic source-address check in the stack. Never set 0.0.0.0/0 on a site-to-site peer — BGP will happily advertise anything and your appliance will accept spoofed traffic.
  • Authenticate and bound BGP. MD5 passwords (neighbor ... password), GTSM (neighbor ... ttl-security hops 1) where the peer is adjacent, strict inbound prefix-lists, and maximum-prefix so a misconfigured peer cannot blow up your RIB.
  • Harden the appliances. IMDSv2 required, no SSH keys (use SSM Session Manager), minimal AMI, unattended security upgrades for the kernel since WireGuard is in-kernel, and an SG that only exposes UDP/51820 to the known on-prem public IPs.
  • Separate blast radius. Put the appliances in a dedicated transit VPC with its own route tables and a TGW route table that only permits the prefixes the datacenter is supposed to reach. Do not attach the gateway subnet to a NAT gateway shared with workloads.
  • Log and audit the control plane. Ship FRR logs to your SIEM; a BGP session flapping at 03:00 from an unknown ASN is an incident, not a networking quirk.
  • Encrypt what crosses even inside the tunnel. WireGuard protects the transit; it does not authenticate the workloads. Keep mTLS between services.

Key takeaways

  • WireGuard + FRRouting + Transit Gateway Connect gives you a dynamically-routed, redundant hybrid link for the price of two small EC2 instances. It is a credible answer when Direct Connect is unaffordable and AWS Site-to-Site VPN does not meet your throughput or key-control requirements.
  • Fix MTU at 1420 and clamp TCP MSS. Most "the hybrid link is slow" tickets are MTU tickets, especially against 9001-byte VPC instances.
  • BFD is mandatory. WireGuard never reports a down link; without BFD at 300 ms × 3 your failover takes 90–180 seconds instead of under a second.
  • Transit Gateway Connect is what makes it dynamic. GRE plus BGP to the TGW removes every static route and every Lambda-driven route table rewrite from your design.
  • Test the black hole, not the clean shutdown. systemctl stop frr converges instantly and proves nothing; a silently dropped UDP port is the failure you will actually experience.
  • Own the keys properly. Pre-shared keys, External Secrets, tight AllowedIPs, authenticated BGP with prefix limits. The tunnel is cheap; a route leak is not.