On Friday, November 13, every departing flight in the United States was held on the ground for about four hours. More than 8,000 flights were cancelled that evening, according to Cirium data cited by TechCrunch, and thousands more on Saturday.
The explanation officials gave on Saturday is the part that made me put my coffee down. It wasn't a cyberattack, and they say it wasn't the new AI tool people were speculating about. It was routine maintenance.
What officials say happened
A long-time FAA contractor, L3Harris, was replacing old copper lines with fiber optic cable at a remote location. The work happened around 3pm on Friday. Afterwards, according to the FAA's acting CTO, two locations were "pinging our network at a very high rate", about "a thousand connections over every four to five seconds." That flood overloaded a critical communication system. The FAA disabled both sources and the network stabilized.
The FAA is still doing a root cause analysis. Officials say they don't yet know whether it was user error, installer error or a faulty router. So treat the details as preliminary.
One more detail stuck with me. NPR notes that the previous nationwide ground stop, in 2023, was also caused by contract workers, who deleted critical files in the NOTAM system. That's two national shutdowns in a few years, and both started with somebody doing ordinary work.
The dangerous word is "routine"
I've been writing software for 15+ years, first as a developer, then a tech lead, now a team lead. I've run small startups where I was also the entire ops team. Here's my honest pattern: the big, scary releases rarely took me down.
Big releases get respect. There's a plan, a rollback, a quiet hour picked on purpose, three people watching dashboards. The pain always came from changes that didn't feel like changes. A config value. A dependency bump. Swapping one piece of hardware for an "identical" one.
Here's the everyday version. A big kitchen renovation gets planned for months. Replacing a single tap is something you do on a Sunday evening, and that's exactly when you flood the flat downstairs.
The FAA story is the tap, at national scale. One device started knocking on the door about 1,000 times every few seconds. Older systems on the other side had no way to say "slow down", so everyone behind that door stopped hearing anything else.
Is "never deploy on Friday" the lesson?
Partly, and this is where I'll push back on the internet a bit.
Officials said big-airport switchovers usually happen late at night, while remote-site work happens during the day, and that "there will be reevaluation of when we do this work." Fair. But a blanket Friday freeze isn't free either. Work done in daylight means the people who can fix things are awake. Work pushed to 2am means tired people and thin coverage.
What I believe is narrower. The timing rule matters less than whether the change was treated as a change at all. A Tuesday change with no rollback plan is worse than a Friday change with one.
The three questions I ask before any "small" change
These work for code, servers, office Wi-Fi and, yes, the router at home:
- What's the worst thing this can touch? Not the likely thing, the worst one. If the answer is "everything downstream", it's not small.
- Who notices first if it goes wrong, and are they around? If the first person to notice is a customer, or a few thousand passengers, the monitoring is in the wrong place.
- How fast can we undo it? Unplugging something counts as a rollback plan. "We'll figure it out" doesn't.
On the technical side, if you own systems that accept connections, it's worth checking that they limit how fast a single client can connect (rate limiting). It's also worth checking that one noisy device can't starve everyone else. Old systems often assumed every neighbour would behave. That assumption ages badly.
What I'm taking back to my team
I'm not adding a Friday ban. I'm adding one line to how we talk about work: "routine" isn't a risk level. Every change gets the three questions, even the boring ones. Especially the boring ones.
Somewhere in the US there's a router that will never live this down. I hope the person who plugged it in has a kind manager.
What's your team's rule: Friday changes allowed, or Friday frozen? And why?
