FAA ground stop: how a network upgrade went wrong

New to this?Ping explains the hardest part in plain words.

Open all explanations

A router swapped in during a routine upgrade at one remote site helped shake the radar and radio links behind US air traffic control badly enough that the FAA stopped every departure in the country for more than four hours. The FAA ground stop is a textbook case of how a small network change turns into a national outage, and of why network teams save this kind of work for the quietest hours they can find.

#What happened in the FAA ground stop

On Friday afternoon, November 13, an equipment outage disrupted radar feeds and communications with aircraft, and the FAA halted departures nationwide. It lifted the ground stop at 8:15 p.m. ET, and by Saturday morning FAA chief Bryan Bedford said the air traffic control system was stable, though airlines still had a long recovery ahead.

The network involved is run by contractor L3Harris Technologies, which was switching lines at a remote FAA site from copper wire to fiber because the copper line was so old it was being discontinued. Controllers felt it directly: Nick Daniels, president of the National Air Traffic Controllers Association, said some planes' communications were disrupted and controllers moved flights to working frequencies.

#How a network upgrade went wrong

Transportation Secretary Sean Duffy said a router involved in the copper-to-fiber switch "started flooding other routers in our air traffic control system." Officials put a number on it: more than 1,000 connection attempts every 4 to 5 seconds. Investigators are still working out whether that came from human error, an installation mistake or a problem with the equipment itself.

The older equipment around it matters too. Senior government sources told CBS News the problem was isolated to certain older routers, Cisco 600 series units made between 2001 and 2006. That's the part that turns a flood into an outage:

• The router's brain gets swampedRouters forward normal traffic in fast dedicated hardware, but setting up connections and processing routing updates runs on the router's own processor (the control plane), which is far smaller. A storm of connection attempts lands there, and a busy processor starts missing the routing updates and keepalive messages that tell neighbors a link is healthy.
• Healthy links look deadWhen those messages go missing, neighboring routers decide links have failed and reroute traffic, then reroute again when the links reappear. Every reroute is more work for processors that are already behind, so the instability feeds itself and spreads.
• Old hardware has less headroomA flood that a modern router absorbs can max out equipment that's two decades old, which may be how a problem at one remote site rippled through older routers elsewhere.
• Unreliable is as bad as downBedford said no facilities went dark and nothing was fully lost. For air traffic control that isn't comfort: radar data that arrives late or out of order can't be trusted, and a feed you can't trust is grounds to stop flights.

ELI5: explain it simply▾

Ping explainsNo experience needed.

THE WORDS

Router

A device that decides which path each piece of network traffic takes to reach its destination.

Keepalive

A small "I'm still here" message routers send each other so they know a link is working.

THE IDEA

Picture a dispatcher who routes trucks and also takes calls from other dispatchers saying "my road is open." If one caller phones a thousand times, the dispatcher misses the real calls, assumes roads are closed and starts rerouting trucks for no reason. Now every dispatcher is confused at once.

Why the backups didn't save it

Backup channels exist, Bedford said, but they all run over the same network: "We have one federal telephony infrastructure. We don't have two, but one." A backup that shares the network with the main path shares its failures too. Bedford said the flooding ran for 45 minutes without the FAA finding the cause, and spread to 17 facilities before the ground stop was called. On a network where symptoms show up far from the device causing them, that much time can pass before anyone is looking at the right router.

#It happened before, in 2009

A Transportation Department inspector general review found that in 2009, a technician for Harris Corp., L3Harris' predecessor, misconfigured a router while moving the network to fiber, disrupting 129 air traffic facilities. Same kind of project, same kind of device, 17 years apart. An analyst told CNN that the inspector general's report faulted the company for not having systems in place to test changes before putting them live.

#Changing a network you can't test

On paper, this change looked small: one site, swapping copper for fiber. The catch is the equipment around it. With routers built in the early 2000s, building a faithful copy in a lab is often impossible, because the exact hardware, firmware and traffic patterns are hard to reproduce, so a change that looks low-risk only gets its real test once it's live.

Duffy defended the upgrade work on Friday: "By doing nothing, you inject way more risk into the system." Both sides of that are true. Leave an old network alone until a full replacement arrives and it keeps aging while the people who know its quirks move on. Upgrade it piece by piece and every cutover becomes a live test on a network nobody can fully simulate. Congress has already approved $12.5 billion for the overhaul, which the FAA has used to replace thousands of copper lines and upgrade hundreds of radio systems, according to the AP as reported by Israel National News.

That's why the checklist below leans on containment (rate limits, isolation, a separate backup path) and not only on testing. When you can't prove a change is safe beforehand, you limit how far it can spread if it isn't.

#The 3 p.m. Friday problem

The switch was done mid-afternoon on a Friday, not overnight. Asked about it, Duffy said, "There'll be a reevaluation of when we do this work," and acknowledged, "It shouldn't have that kind of impact, which is what we saw yesterday." Most network teams would recognize what a safer version of this change looks like:

Routing, redundancy and change management are core Network+ topics. CertPrep+ has study material for Network+ among its 12 certs from CompTIA, AWS, Microsoft and ISC2, if you'd like to understand networks like this one: see CertPrep+.

opnF.sh

News that matters to the tools we teach.