This is an English version of my original Japanese article on Zenn, translated with the help of AI.
TL;DR
- On k3s with the flannel VXLAN backend running over a VPN interface (Tailscale here), restarting or upgrading
tailscaledleft one node unable to reach Pods on any other node. The node stayedReadyand nothing alerted. - flannel recreated
flannel.1, but the kernel had already dropped the routes, ARP and FDB entries that went through it, and flannel only rewrites those when a remote lease changes. - The fix replays the known remote leases after the device is recreated: flannel-io/flannel#2618, released in v0.28.11.
- Until your k3s ships it, restart
k3s/k3s-agentafter a VPN restart and alert on the number offlannel.1routes.
Issue: vxlan: routes/ARP/FDB not restored after the external interface is recreated (#2617)
Who this affects: anyone running the flannel VXLAN backend (including k3s's embedded flannel) with --iface (flannel-iface in k3s) set to a VPN interface such as Tailscale, NetBird or WireGuard, on flannel v0.28.10 or earlier.
Environment
- k3s v1.35.10+k3s1 (embedded flannel v0.28.10, VXLAN backend, IPv4 only)
- 4 nodes, Oracle Linux 9.7 (aarch64)
- Node-to-node traffic over Tailscale 1.104.1 (
flannel-iface: tailscale0)
k3s also has a Tailscale integration (--vpn-auth, experimental); flannel runs on tailscale0 in that case as well.
Symptoms
After restarting tailscaled (or a Tailscale package upgrade), the k3s log showed:
Interface flannel.1 deleted vxlan device missing, attempting to recreate... external interface tailscale0 not found, retrying in 1s VXLAN device flannel.1 recreated successfully flannel.1 came back, so it looked fine. But on that node:
-
ip route | grep -c 'dev flannel.1'dropped from 3 (the number of other nodes) to 0 -
ip neigh show dev flannel.1andbridge fdb show dev flannel.1were empty - Traffic to Pods on other nodes timed out (the ingress proxy on the node logged
upstream connect error ... connection timeout) - The node stayed
Ready, and nothing more was logged afterrecreated successfully
It did not recover after about 2.5 minutes. Restarting k3s-agent brought the routes back within about a second.
Root cause
Following pkg/backend/vxlan/vxlan_network.go, two things combine:
- When the external interface (
tailscale0) disappears, the kernel removesflannel.1together with the routes, ARP and FDB entries that use it. - flannel's
reCreateVxlan()recreates the device itself, but routes, ARP and FDB entries are written only byhandleSubnetEvents(), which runs only when a lease (another node's subnet) changes.
Recreating the interface does not change any remote lease, so the entries are never written again and the node stays isolated until flannel restarts.
Older releases (v0.28.4, for example) panicked on this path in IPv4-only setups (#2433), which restarted k3s and, as a side effect, restored the routes. Once that panic was fixed, the recreate path completed "successfully" and the node quietly stayed isolated instead.
The fix
The PR makes three changes:
- Keep the remote leases passed to
handleSubnetEvents()in alease.LeaseWatcherowned by the Run loop. - Make
reCreateVxlan()return the new device instead of overwritingnw.dev/nw.v6Devfrom a goroutine. - In the Run loop, take the new device and replay the known leases into
handleSubnetEvents()asEventAdded, which rewrites routes, ARP and FDB entries.
// knownLeaseEvents returns an EventAdded for every known remote lease. func (nw *network) knownLeaseEvents() []lease.Event { batch := make([]lease.Event, 0, len(nw.knownLeases.Leases)) for _, l := range nw.knownLeases.Leases { batch = append(batch, lease.Event{Type: lease.EventAdded, Lease: l}) } return batch } The second change also removes an unsynchronized write to nw.dev from another goroutine.
Test
Using ns.SetUpNetlinkTest, as pkg/backend/route_network_test.go does, the new test TestRecreatedDeviceGetsKnownLeases creates a dummy interface in a network namespace and:
- creates the VXLAN device and processes one remote lease,
- deletes and recreates the external interface,
- checks that the route to the remote subnet comes back via
flannel.1.
Against the old code it fails with no route to 198.18.2.0/24 via flannel.1 after flannel.1 was recreated; with the fix it passes, both shown in CI.
pkg/backend/vxlan was not in make test's TEST_PACKAGES at the time, so following review feedback I added it in the same PR.
Detecting it until the fix reaches you
The node stays Ready, so you need another signal. node_exporter's network_route collector (listed under "Disabled by default"; enable it with --collector.network_route) exports the number of routes per interface as node_network_routes{device="..."}.
With the VXLAN backend, a healthy node has as many flannel.1 routes as there are other nodes. When all routes disappear the series disappears too, so the rule also catches a missing series:
- alert: FlannelVxlanRoutesMissing expr: | ( ( max by (node) (node_network_routes{job="node-exporter", device="flannel.1"}) < scalar(count(kube_node_info) - 1) ) or ( max by (node) (node_uname_info{job="node-exporter"}) unless on (node) node_network_routes{job="node-exporter", device="flannel.1"} ) ) and on () (count(kube_node_info) > 1) for: 3m - This assumes a
nodelabel on node-exporter series (commonly added by relabeling in kube-prometheus-stack) andkube_node_infofrom kube-state-metrics. - The same collector also exports one
node_network_route_infoseries per route. With Tailscale that grows with the number of peers, so I dropped it withmetricRelabelingsand kept only the counts.
What to do
- Upgrade to flannel v0.28.11 or later. k3s embeds flannel, so you need a k3s release that includes v0.28.11 (as of 2026-11-20, k3s
mastergo.modstill pins flannel v0.28.10). - Until then, restart
k3s/k3s-agentafter restarting or upgradingtailscaled, and check that the number offlannel.1routes equals the number of other nodes. - Consider pausing Tailscale auto-updates until the fix is in.
Notes on sending it upstream
- An issue with the same symptom (reproduced with NetBird, #2617) already existed, so instead of opening a new one I added reproduction steps and logs from a different setup (Tailscale) as a comment, and linked the related k3s issue (k3s-io/k3s#12436).
- The PR's e2e tests failed, but the failure was unrelated: it also happened on
masterwith the newubuntu-latestrunner. I wrote up the cause (the test CA certificate was not accepted as a CA) in a separate issue (#2620), sent the fix as a separate PR (#2621), got that merged first, and then rebased the main PR. - I showed in CI that the new test fails on the old code, so reviewers could see it really catches the bug.
How I used generative AI
I used generative AI (Claude Code and Codex) for the investigation and the implementation. Roughly:
- What I did myself: noticed the symptom in my own cluster, reproduced it by restarting
tailscaled, checked the logs, routes and FDB, matched it to the existing issue, judged whether the fix approach (replaying known leases) was sound, and followed the CI results and the conversation with the maintainers. - What I delegated to AI: reading the flannel source and locating the relevant code, drafting the fix and the test, and writing the English issue comment and PR description.
I did not ship the AI's code or explanations as they were: I confirmed in CI that the test fails before the fix and passes after it before opening the PR. Checking a root-cause hypothesis against tests and the kernel's actual state, rather than against an AI's explanation, is an order I intend to keep.
Summary
- Up to v0.28.10, when flannel VXLAN's external interface was recreated, the device came back but routes, ARP and FDB entries did not.
- The node stays
Ready, so you won't notice unless you watch the route count. - v0.28.11 replays the known leases onto the recreated device.
- I used generative AI for reading code and drafting, and verified correctness with tests and the real environment.
Reproducing a problem from your own environment, finding the cause, and sending it upstream with a test helps everyone running the same setup, and lets you remove the workaround from your own operations. I plan to keep sending what I hit upstream.