It wasn’t a hurricane. It wasn’t a hacker. It was one piece of equipment being swapped out at a quiet radio site in south Arkansas.
On Friday afternoon, the Federal Aviation Administration says, new network gear installed at its radio site in El Dorado, Arkansas, helped set off a flood of data that knocked the country’s air traffic system off its feet. Within hours, every flight in the United States was held on the ground. Thousands of planes already in the sky were brought down. Airports from coast to coast filled with stranded travelers, and the cancellations kept coming through Saturday and into Sunday.
El Dorado is about 225 miles north of Abbeville as the crow flies. That’s roughly a five-hour drive up U.S. 167, just over the Louisiana line.
How one box took down the grid
The FAA’s contractor, L3Harris, was at the El Dorado site Friday installing a new router, the box that moves data between the radio site and the rest of the FAA’s network. Transportation Secretary Sean Duffy said on CBS’s “Face the Nation” Sunday that the phone company was retiring the site’s old copper line, which forced a switch to fiber. “We brought in our routers and did the work, and that’s when we saw the flooding of the network,” he said.
At about 2:30 p.m. Central, the FAA’s network was hit with a thousand connection requests every few seconds, according to the agency. A second site, a radar facility in Amarillo, Texas, is also under investigation. The aging equipment on the other end couldn’t take it. Radio and radar feeds began failing at 17 of the 22 big control centers that guide planes between airports.
The FAA called a nationwide ground stop. Roughly 3,500 planes in the air at the time landed safely, the agency says. Flights didn’t start moving again until about 7:15 p.m. Central, nearly five hours later. The Philadelphia Inquirer reported it was the longest stoppage of its kind in 25 years.
Two days of chaos
Airlines canceled about 12,462 flights on Friday alone, the FAA says. With planes and crews stuck in the wrong cities, the mess rolled into the weekend. In Philadelphia, more than 150 flights were canceled by Saturday afternoon, and dozens more were still being scrubbed on Sunday. FAA Administrator Bryan Bedford told reporters that flying should recover fully by Sunday night, though he warned of more delays first.
Duffy said the same kind of switch has been made “thousands of times” without trouble, which is why crews at a remote site like El Dorado did the work in daylight. “We’re going to have to reevaluate now when we do this work,” he said.
What’s next
The FAA says there is no evidence of a cyberattack. As a precaution, it shut off the same function on about 60 similar routers across the country. The FAA and L3Harris are still hunting for the root cause, and Duffy said the FBI is helping.
Duffy called the system a 1980s network running on copper wire. He is asking Congress for $50 billion to rebuild it.
Are south Louisiana’s sites safe now? We asked.
The FAA hasn’t said where those 60 routers are, or whether any of them serve south Louisiana, where Houston Center handles most of the high-altitude traffic and Lafayette’s controllers handle the approaches.
So The Vermilion Times asked. On Sunday we sent questions to the FAA press office and to L3Harris, the contractor that runs the network. We asked whether any Louisiana radio or radar sites have the same kind of router, whether shutting off that function is a permanent fix, and whether this kind of work has been paused or moved to overnight hours. We’ll report what they tell us, and when they don’t answer.
Update, Nov. 16: L3Harris answered Monday with a written statement attributed to a company spokesperson. The company said it, its telephone partners and the FAA have finished more than 70% of the planned switch of airspace communications from old copper lines to fiber, and that it is “supporting the FAA in its investigation to understand the conditions that led to the Nov. 13 ground stop.” L3Harris “takes the nationwide ground stop and its impact on travelers, airlines and air traffic operations seriously,” the statement said.
The statement did not say whether any sites serving Louisiana have the same router, or whether the company has changed when it does this kind of work. The FAA did not answer by our Tuesday, Nov. 17, deadline. [Our follow-up is here.](/news/faa-no-answer-louisiana-sites-ground-stop-router/)
Were you stuck at an airport this weekend? Tell us your story in the comments.
