A buddy was asking me if this story "holds water" earlier today (because I'm more into networking). I eventually decided it was plausible, but only through at least one, and probably multiple "rookie mistakes".
As described it was in the course of switching out copper TDM lines with fiber. On it's face that could be straightforward, but it's easy to imagine the router(s) terminating these lines didn't support the required line cards, so router upgrade(s) were required. So theoretically straightforward could have become more involved.
However, when doing far less critical updates, it's common to set up a test environment, do a dry run of the work, test the roll-back, test the monitoring and alerting, that sort of thing. Doesn't guarantee no outages, and I'm playing armchair QB here, but it seems like a botched job.
It's also telling the Transportation Secretary said things like "we aren't sure they if they aren't doing a great work for us". He also said "this is something businesses have been doing for decades and the FAA is the last to do it"; you'd assume that this is a process that someone (maybe not those doing the work) have developed deep experience with.
I'm not familiar with this L3HARRIS, I gather they are a defense contractor with billion dollar revenue.
Lots of things in this definitely raised my eyebrows.
As another with a network engineering background from the customer side as lead architect for a 1+ million port healthcare network, principal solution architect for a VAR, and now wired PLM for a major networking vendor (not the one in the article), I can't say the details given really say enough to come to any conclusion or doubts whatsoever.
I've seen simple copper->fiber migrations like this take down a large network and I've seen them be blamed when they turned out to be unrelated. I could come up with a scenario for most any view anyone wanted to speculate on because the details are so light and the possibilities are so many, but I won't because I don't have enough details to say which it might have been without just guessing myself. If they say they identified a broadcasting knob they disabled on the other ~60 routers already in service then that's what I'd leave such a conversation at for now - but I wouldn't necessarily trust that or what it really means without more info.
In the thread the other day I saw people blaming anything from AI to attacks to Starlink but few willing to admit they just needed more information to have any certainty about what to think beyond "it sucks this is down and I wonder what the reason will be". The only thing I'd raise my eyebrows at at this point would be if we don't get further details in reasonable time.
I was the Sr. Internet Mail Administrator at AOL at the time, and this article was what The Register wrote based on what I had submitted to them. They made a few changes. I know they were trying to protect my identity, but I think they also wanted to make it a better story.
For us, everything started with a planned downtime to upgrade the main core router for the entire service. We had the best network engineers in the business, but even they can make a mistake. And they did. And then there was a whole domino effect of events that followed.
Ok, you have to make a Tell HN and share your war stories from AOL from that time! Must be so many fascinating firefighting stories to general being at the forefront of the Internet revolution. Wow!!
I don't have the full story because it happened in another business unit, but one of the worst outages I ever worked the rumour was it was caused by someone doing prep work for a later cutover to new hardware. Whatever they were doing to prep for the upcoming cutover, the router rebooted. They didn't know it rebooted, so they did the same thing to the secondary and tertiary a few minutes later. Also happened to be on a Friday.
So sometimes it's easy to rehearse all the critical stuff, and get a bit complacent with the stuff that is just expected to work, plugging something in that's not configured and getting it ready for the big migration. And it's hard to get a 100% perfect replica in a lab. So I think odds are you're right, but I think there is also a longer tail of possibilities as well. And in my experience, there is always tension between chasing the 9's of availability, and getting some "safe" work done when convenient.
This is a fairly common pattern. I’ve also seen a few cases where a failover environment doubles as smoke test for prod rollouts. Hey, it’s a mirror of prod, and sits idle almost all the time, so it’s simpler and cheaper to test that way. Until prod goes down and a failover is triggered mid-deployment to the backup environment.
The explanation could hold water, but that only demonstrates plausibility, not truth.
Seems that a router that previously terminated copper interfaces was switched to terminate fiber. That alone doesn't really tell us much without knowing whether this is really all that happened or if it's shorthand for a bigger move at a higher level in the protocol stack. For example you might "just" have changed copper Cat6 ports to fiber/SFP but still be doing GigEthernet and I rather doubt that a move that simple could make a router misbehave.
Moving up the stack this get more interesting. For example they might have moved from a copper star topology to a fiber ring, and rings need care or else chaos ensues. Similarly Ethernet gets pissy if there are loops but the spanning tree protocols need very careful usage. I assume since the word router was used that Ethernet was fine and we are in layer 3. An interface switch could have triggered recalculation or reannouncement of routes, for example by erroneously configured routing protocols. eg the Arkansas router announces it is the best hop for traffic to reach Chicago, gets swamped with traffic from Dallas, NY, etc, and falls over. (Yes, the tools provide guardrails. Who knows if they were enabled?)
However, and esp because Duffy is pointing at L3Harris, I suspect they installed (or perhaps removed) a Juniper TDM to Ethernet converter module (eg DS3 SmartSFP from that vendor) and something went bad. These modules are supposed to encap TDM into Ethernet but they don't (iirc) have any semantic awareness of a telephone call, so if they were to be hit by, say, a storm of TDM circuit messages (call setups, say) they might just pass them on, not rate limit them like a true telco packet switch would.
It might be pure coincidence that an AI system was hooked up to the FAA system the week before.
Bottom line: could be bad luck, could be basic screw up, we won't know for years.
This reads very human but "that only demonstrates plausibility, not truth." reads AI ;) Are you imitating ChatGPT? :-) But I agree. This could be a routing issue or some other interaction. Could also just be overload of traffic between these networks of different capabilities. At you say, probably not rate limited in the interface between the systems.
It's not hard to screw up networks. I remember our IT person introducing a loop into a simple office network that took like two hours to troubleshoot.
You'd think they would have a) noticed sooner b) made the connection between the last change and the issues. But easy to be a backseat quarterback as they say.
> However, when doing far less critical updates, it's common to set up a test environment, do a dry run of the work, test the roll-back, test the monitoring and alerting, that sort of thing. Doesn't guarantee no outages, and I'm playing armchair QB here, but it seems like a botched job.
Quite possible. The DOGE antics from the early days of the administration applied an indiscriminate flame thrower to govt agencies gutting them without thought or planning. It's not surprising we are seeing critical govt systems start to fail. What's surprising is how long it took for the true extent of the damage to be revealed.
Hopefully, the midterm results will mean greater scrutiny and transparency into DOGE's antics.
Plausible I guess. Essentially a denial-of-service attack from a new device connected to the network. The article doesn't give more details but a device in the network can ... send traffic into the network ... and a device connected to a high speed link in a network with many old legacy/slow devices ... could cause problems.
The router was apparently a Juniper router. L3Harris I guess buys them and install/configures/whatnot. They don't make them.
The ground radio and radar stations for enroute operations are often not near where the controllers are. Voice and radar data have to be transmitted around.
Airport radio and radar can be primarily local, but enroute cannot.
Seems like a bad design though to put both on the same network. Voice is relatively low bandwidth but more important. Also voice is typically more localized, a controller in SFO isn't talking to an airplane in NYC.
Do we know that voice and radar were lost though? How did the flights in the air land? I thought the thing that broke was the handover between different regions/spaces/controllers?
The ATC operating at airports, like ground, tower, departure, etc - are all local and simply transmit voice over AM frequencies. They were not affected so flights could land.
Regional/sector ATC (managing flights en route that left the departing airport airspace and have not yet arrived to the arrival airport airspace) are also transmitting in AM but the actual operations are not where the transmitters are (as a lot of them are in the middle of nowhere). The voice/data transmits back and forth from the actual op centers to the transmitters/receivers over the network. Which is what went down.
Actually some short flights (like flight schools or private pilots on short flights) were allowed to continue to operate, as they never left the airport or arrived at an airport so close that they never needed to talk to ATC managing a sector.
Sounds like a bad default Juniper setup for RTSP all or spanning tree max on. Which is not the factory default.
“FAA officials identified a feature on the router that appeared to be resulting in the data overload, according to government and industry officials. A government official described it as a broadcasting feature that sent data more broadly than necessary, and said the agency has since disabled it on all the Juniper routers.”
The engineers that write the guides for deploying routers and the technicians that field them could have an old or local version of install notes leading to this.
I’ve seen multiple versions of install guides floating around in emails with no documentation control leading to bad installs.
It reminds me of a time, over 20 years ago, when I was the main part of upgrading large plant's phone system to my company's on-premise VOIP offering.
It was more than a normal factory; it was a giant place that designed and built complex machines that were sent out to make other complex machines at much higher volume.
Anyway, the cutover was badly-planned but it went smooth-enough. I had issues with their PRI and the T1, and my fucking manufacturer rep who was supposed to be present on-site was completely fucking AWOL, but I got lasting help from some dude from Sprint who was exquisitely patient and we walked through switch configuration options at all levels, including for circuits that did not belong to Sprint.
It worked. It worked great, in fact. We met the deadline. There were some details to sort, but they weren't significant and everyone agreed that they could wait a day or three.
I dusted myself off after a 38-hour continuous shift, drove home, and collapsed on the sofa in the front room with a huge, bright south-facing window.
20 minutes later, my phone rings. It's my boss.
"Hey, uh. They say their network is completely hosed. For the entire plant. They want you to call them right now."
I know that all I did was replace a phone system and put in some VOIP desksets -- all I did was add some network endpoints -- but whatever. I call them.
It's positively fire and brimstone over there. "WHAT DID YOU DO TO OUR NETWORK! NOTHING WORKS! THE WHOLE THING IS JUST FULL OF PACKETS, AND NONE OF THEM FUCKING MEAN ANYTHING! PRODUCTION, OFFICE, EVERYTHING!"
yawn
I got my boots on, found some coffee, and started driving back.
When I walked back in I expected to be drawn, quartered, and gutted, but it was instead weirder than that: "WHY DID YOU PLUG BOTH PORTS OF THIS VOIP PHONE INTO THE WALL?"
I hadn't. None of us had. It was all working quite sensibly when we left.
"BOTH PORTS WERE PLUGGED INTO THE WALL! THIS CRASHED OUR ENTIRE NETWORK! WE LOOKED EVERYWHERE! AND WHEN WE UNPLUGGED YOUR PHONE, WITH BOTH OF ITS CABLES, THINGS GOT BETTER! WE'RE SO PISSED AT YOU RIGHT NOW!"
I was several layers beyond being too tired to care much about these strong words. And it was working again, so I left. I made it back home somehow.
---
What happened:
This plant had a switched network that they were very proud of. It used HP Procurve switches, in a special mesh mode. They were very pleased to explain, quite early on, exactly how robust this was: It was very fast when the mesh was intact, and it would adjust immediately to failures between switches and still be Fast Enough when the mesh was not intact.
But! This version of that topology was incompatible with STP, so STP was not turned on.
So when someone (not us-the-installers, nor them-the-plant-IT folks, but someone) plugged both dumb ports of the 10/100 switch in one lowly IP phone into that network: Their entire network shit the bed. Bad. It shit up the entire bed very completely.
(I understand that this HP Procurve mesh/STP incompatibility problem was subsequently resolved with firmware but that was well after this plant's network crash.)
One of the highlights of my childhood was, in the late 90's or early 2000's, touring a museum that showed off an operating wire-wrapped computer, a chain printer, and vector displays with light pens, except it wasn't a museum it was a PAVEPAWS radar installation that Space Command was operating at a nearby military base. They didn't upgrade it until 2012.
But I recall years ago the ATC system made heavy use of multicast in ways that scared me. I dunno if that stuff is still out there but it was some real I wouldn't do that shit if I were you stuff. And yeah mcast can sure fuck a network fast.
As for in general there's plenty of ways a single box can melt a network. Routing protocols let you do bad things fast and pull in more traffic than you can eat.
I mean, if rookies are in charge of the plan then sure. It’s the network engineering equivalent of a pilot ignoring all standard safety checks and a flight plan.
TL;DR of my top-level comment: Yes, but only if the work was done by clowns. ;-)
Do these routers not have rate limiting on them, and if not why in the world not? Sounds like a route loop that then spread out to the rest of the network.
all this ip-based shit is incredibly complex. the fact that facebook, google, etc have all had huge outages related to bgp shows that even the highest paid and most talented network engineers will make mistakes. now i think L3Harris is still incompetent with their design and safeguards, and thats probably why we dont even have a proper postmortem here. but the amount of complexity and failure modes is unarguably higher after these upgrades.
But “L3Harris takes the nationwide ground stop and its impact on travelers, airlines and air traffic operations seriously,” the company said in a statement.
So serious that they didn't take it serious and did it over day. Can you sue them for lying?
So why didn't they replace the cables and router SW overnight? There's not much air traffic at night, it's really trivial compared to all other cases. Amateurs. And then they are asking for $50b.
2. it was a "newer system overloading the legacy system" (Duffy): check
3. it was "old copper wires" (Duffy): check
4. it was not related whatsoever to the SMART AI system: check
5. the process of upgrading the legacy systems is part of the problem: check
6. the combination of a "newer" system with an "older" system, something that likely should have been planned for if the contractors weren't being rushed, was involved [1]: check
A buddy was asking me if this story "holds water" earlier today (because I'm more into networking). I eventually decided it was plausible, but only through at least one, and probably multiple "rookie mistakes".
As described it was in the course of switching out copper TDM lines with fiber. On it's face that could be straightforward, but it's easy to imagine the router(s) terminating these lines didn't support the required line cards, so router upgrade(s) were required. So theoretically straightforward could have become more involved.
However, when doing far less critical updates, it's common to set up a test environment, do a dry run of the work, test the roll-back, test the monitoring and alerting, that sort of thing. Doesn't guarantee no outages, and I'm playing armchair QB here, but it seems like a botched job.
It's also telling the Transportation Secretary said things like "we aren't sure they if they aren't doing a great work for us". He also said "this is something businesses have been doing for decades and the FAA is the last to do it"; you'd assume that this is a process that someone (maybe not those doing the work) have developed deep experience with.
I'm not familiar with this L3HARRIS, I gather they are a defense contractor with billion dollar revenue.
Lots of things in this definitely raised my eyebrows.
As another with a network engineering background from the customer side as lead architect for a 1+ million port healthcare network, principal solution architect for a VAR, and now wired PLM for a major networking vendor (not the one in the article), I can't say the details given really say enough to come to any conclusion or doubts whatsoever.
I've seen simple copper->fiber migrations like this take down a large network and I've seen them be blamed when they turned out to be unrelated. I could come up with a scenario for most any view anyone wanted to speculate on because the details are so light and the possibilities are so many, but I won't because I don't have enough details to say which it might have been without just guessing myself. If they say they identified a broadcasting knob they disabled on the other ~60 routers already in service then that's what I'd leave such a conversation at for now - but I wouldn't necessarily trust that or what it really means without more info.
In the thread the other day I saw people blaming anything from AI to attacks to Starlink but few willing to admit they just needed more information to have any certainty about what to think beyond "it sucks this is down and I wonder what the reason will be". The only thing I'd raise my eyebrows at at this point would be if we don't get further details in reasonable time.
The irony here is that this is basically a mirror of what happened at AOL on August 7th, 1996. See: https://www.theregister.com/on-prem/2018/04/16/i-crashed-aol...
I was the Sr. Internet Mail Administrator at AOL at the time, and this article was what The Register wrote based on what I had submitted to them. They made a few changes. I know they were trying to protect my identity, but I think they also wanted to make it a better story.
For us, everything started with a planned downtime to upgrade the main core router for the entire service. We had the best network engineers in the business, but even they can make a mistake. And they did. And then there was a whole domino effect of events that followed.
Ok, you have to make a Tell HN and share your war stories from AOL from that time! Must be so many fascinating firefighting stories to general being at the forefront of the Internet revolution. Wow!!
Also, being 30 years ago, it matches the timeline for public sector adopting technology.
I don't have the full story because it happened in another business unit, but one of the worst outages I ever worked the rumour was it was caused by someone doing prep work for a later cutover to new hardware. Whatever they were doing to prep for the upcoming cutover, the router rebooted. They didn't know it rebooted, so they did the same thing to the secondary and tertiary a few minutes later. Also happened to be on a Friday.
So sometimes it's easy to rehearse all the critical stuff, and get a bit complacent with the stuff that is just expected to work, plugging something in that's not configured and getting it ready for the big migration. And it's hard to get a 100% perfect replica in a lab. So I think odds are you're right, but I think there is also a longer tail of possibilities as well. And in my experience, there is always tension between chasing the 9's of availability, and getting some "safe" work done when convenient.
This is a fairly common pattern. I’ve also seen a few cases where a failover environment doubles as smoke test for prod rollouts. Hey, it’s a mirror of prod, and sits idle almost all the time, so it’s simpler and cheaper to test that way. Until prod goes down and a failover is triggered mid-deployment to the backup environment.
The explanation could hold water, but that only demonstrates plausibility, not truth.
Seems that a router that previously terminated copper interfaces was switched to terminate fiber. That alone doesn't really tell us much without knowing whether this is really all that happened or if it's shorthand for a bigger move at a higher level in the protocol stack. For example you might "just" have changed copper Cat6 ports to fiber/SFP but still be doing GigEthernet and I rather doubt that a move that simple could make a router misbehave.
Moving up the stack this get more interesting. For example they might have moved from a copper star topology to a fiber ring, and rings need care or else chaos ensues. Similarly Ethernet gets pissy if there are loops but the spanning tree protocols need very careful usage. I assume since the word router was used that Ethernet was fine and we are in layer 3. An interface switch could have triggered recalculation or reannouncement of routes, for example by erroneously configured routing protocols. eg the Arkansas router announces it is the best hop for traffic to reach Chicago, gets swamped with traffic from Dallas, NY, etc, and falls over. (Yes, the tools provide guardrails. Who knows if they were enabled?)
However, and esp because Duffy is pointing at L3Harris, I suspect they installed (or perhaps removed) a Juniper TDM to Ethernet converter module (eg DS3 SmartSFP from that vendor) and something went bad. These modules are supposed to encap TDM into Ethernet but they don't (iirc) have any semantic awareness of a telephone call, so if they were to be hit by, say, a storm of TDM circuit messages (call setups, say) they might just pass them on, not rate limit them like a true telco packet switch would.
It might be pure coincidence that an AI system was hooked up to the FAA system the week before.
Bottom line: could be bad luck, could be basic screw up, we won't know for years.
This reads very human but "that only demonstrates plausibility, not truth." reads AI ;) Are you imitating ChatGPT? :-) But I agree. This could be a routing issue or some other interaction. Could also just be overload of traffic between these networks of different capabilities. At you say, probably not rate limited in the interface between the systems.
It's not hard to screw up networks. I remember our IT person introducing a loop into a simple office network that took like two hours to troubleshoot.
You'd think they would have a) noticed sooner b) made the connection between the last change and the issues. But easy to be a backseat quarterback as they say.
> However, when doing far less critical updates, it's common to set up a test environment, do a dry run of the work, test the roll-back, test the monitoring and alerting, that sort of thing. Doesn't guarantee no outages, and I'm playing armchair QB here, but it seems like a botched job.
Quite possible. The DOGE antics from the early days of the administration applied an indiscriminate flame thrower to govt agencies gutting them without thought or planning. It's not surprising we are seeing critical govt systems start to fail. What's surprising is how long it took for the true extent of the damage to be revealed.
Hopefully, the midterm results will mean greater scrutiny and transparency into DOGE's antics.
Plausible I guess. Essentially a denial-of-service attack from a new device connected to the network. The article doesn't give more details but a device in the network can ... send traffic into the network ... and a device connected to a high speed link in a network with many old legacy/slow devices ... could cause problems.
The router was apparently a Juniper router. L3Harris I guess buys them and install/configures/whatnot. They don't make them.
I’m not clear what scenario you are trying to describe and if by into networking you are a principal level IC or home hobbyist
Did this router also jam radio and and radar?
or was that a coincidence?
The ground radio and radar stations for enroute operations are often not near where the controllers are. Voice and radar data have to be transmitted around. Airport radio and radar can be primarily local, but enroute cannot.
Seems like a bad design though to put both on the same network. Voice is relatively low bandwidth but more important. Also voice is typically more localized, a controller in SFO isn't talking to an airplane in NYC.
Do we know that voice and radar were lost though? How did the flights in the air land? I thought the thing that broke was the handover between different regions/spaces/controllers?
The ATC operating at airports, like ground, tower, departure, etc - are all local and simply transmit voice over AM frequencies. They were not affected so flights could land.
Regional/sector ATC (managing flights en route that left the departing airport airspace and have not yet arrived to the arrival airport airspace) are also transmitting in AM but the actual operations are not where the transmitters are (as a lot of them are in the middle of nowhere). The voice/data transmits back and forth from the actual op centers to the transmitters/receivers over the network. Which is what went down.
Actually some short flights (like flight schools or private pilots on short flights) were allowed to continue to operate, as they never left the airport or arrived at an airport so close that they never needed to talk to ATC managing a sector.
They weren't jammed.
Sounds like a bad default Juniper setup for RTSP all or spanning tree max on. Which is not the factory default.
“FAA officials identified a feature on the router that appeared to be resulting in the data overload, according to government and industry officials. A government official described it as a broadcasting feature that sent data more broadly than necessary, and said the agency has since disabled it on all the Juniper routers.”
The engineers that write the guides for deploying routers and the technicians that field them could have an old or local version of install notes leading to this.
I’ve seen multiple versions of install guides floating around in emails with no documentation control leading to bad installs.
It reminds me of a time, over 20 years ago, when I was the main part of upgrading large plant's phone system to my company's on-premise VOIP offering.
It was more than a normal factory; it was a giant place that designed and built complex machines that were sent out to make other complex machines at much higher volume.
Anyway, the cutover was badly-planned but it went smooth-enough. I had issues with their PRI and the T1, and my fucking manufacturer rep who was supposed to be present on-site was completely fucking AWOL, but I got lasting help from some dude from Sprint who was exquisitely patient and we walked through switch configuration options at all levels, including for circuits that did not belong to Sprint.
It worked. It worked great, in fact. We met the deadline. There were some details to sort, but they weren't significant and everyone agreed that they could wait a day or three.
I dusted myself off after a 38-hour continuous shift, drove home, and collapsed on the sofa in the front room with a huge, bright south-facing window.
20 minutes later, my phone rings. It's my boss.
"Hey, uh. They say their network is completely hosed. For the entire plant. They want you to call them right now."
I know that all I did was replace a phone system and put in some VOIP desksets -- all I did was add some network endpoints -- but whatever. I call them.
It's positively fire and brimstone over there. "WHAT DID YOU DO TO OUR NETWORK! NOTHING WORKS! THE WHOLE THING IS JUST FULL OF PACKETS, AND NONE OF THEM FUCKING MEAN ANYTHING! PRODUCTION, OFFICE, EVERYTHING!"
yawn
I got my boots on, found some coffee, and started driving back.
When I walked back in I expected to be drawn, quartered, and gutted, but it was instead weirder than that: "WHY DID YOU PLUG BOTH PORTS OF THIS VOIP PHONE INTO THE WALL?"
I hadn't. None of us had. It was all working quite sensibly when we left.
"BOTH PORTS WERE PLUGGED INTO THE WALL! THIS CRASHED OUR ENTIRE NETWORK! WE LOOKED EVERYWHERE! AND WHEN WE UNPLUGGED YOUR PHONE, WITH BOTH OF ITS CABLES, THINGS GOT BETTER! WE'RE SO PISSED AT YOU RIGHT NOW!"
I was several layers beyond being too tired to care much about these strong words. And it was working again, so I left. I made it back home somehow.
---
What happened:
This plant had a switched network that they were very proud of. It used HP Procurve switches, in a special mesh mode. They were very pleased to explain, quite early on, exactly how robust this was: It was very fast when the mesh was intact, and it would adjust immediately to failures between switches and still be Fast Enough when the mesh was not intact.
But! This version of that topology was incompatible with STP, so STP was not turned on.
So when someone (not us-the-installers, nor them-the-plant-IT folks, but someone) plugged both dumb ports of the 10/100 switch in one lowly IP phone into that network: Their entire network shit the bed. Bad. It shit up the entire bed very completely.
(I understand that this HP Procurve mesh/STP incompatibility problem was subsequently resolved with firmware but that was well after this plant's network crash.)
https://wallhop.io/https://www.wsj.com/business/airlines/the...
This router should be on display in the Smithsonian air and space museum in dc. It belongs in a museum!
One of the highlights of my childhood was, in the late 90's or early 2000's, touring a museum that showed off an operating wire-wrapped computer, a chain printer, and vector displays with light pens, except it wasn't a museum it was a PAVEPAWS radar installation that Space Command was operating at a nearby military base. They didn't upgrade it until 2012.
It would be nice to know what was disabled on the juniper routers.
For the network engineers on here: does this explanation make sense to you?
Not enough information.
But I recall years ago the ATC system made heavy use of multicast in ways that scared me. I dunno if that stuff is still out there but it was some real I wouldn't do that shit if I were you stuff. And yeah mcast can sure fuck a network fast.
As for in general there's plenty of ways a single box can melt a network. Routing protocols let you do bad things fast and pull in more traffic than you can eat.
I mean, if rookies are in charge of the plan then sure. It’s the network engineering equivalent of a pilot ignoring all standard safety checks and a flight plan.
TL;DR of my top-level comment: Yes, but only if the work was done by clowns. ;-)
Do these routers not have rate limiting on them, and if not why in the world not? Sounds like a route loop that then spread out to the rest of the network.
all this ip-based shit is incredibly complex. the fact that facebook, google, etc have all had huge outages related to bgp shows that even the highest paid and most talented network engineers will make mistakes. now i think L3Harris is still incompetent with their design and safeguards, and thats probably why we dont even have a proper postmortem here. but the amount of complexity and failure modes is unarguably higher after these upgrades.
But “L3Harris takes the nationwide ground stop and its impact on travelers, airlines and air traffic operations seriously,” the company said in a statement. So serious that they didn't take it serious and did it over day. Can you sue them for lying?
So why didn't they replace the cables and router SW overnight? There's not much air traffic at night, it's really trivial compared to all other cases. Amateurs. And then they are asking for $50b.
So... It was BGP and not DNS this time?
Also, "the flood of messages" sounds suspiciously like a routing loop somewhere.
"See this router here? If I switch it off, the ENTIRE US AIRSPACE GOES DOWN."
"Sure thing buddy, I believe you."
Related:
L3Harris statement on the Nov. 13 ground stop
https://news.ycombinator.com/item?id=50501844
The irony of not being able to read the article because archive.is giving a 500 error for the page
so everything we were told was correct:
1. it was L3Harris (CBS news): check
2. it was a "newer system overloading the legacy system" (Duffy): check
3. it was "old copper wires" (Duffy): check
4. it was not related whatsoever to the SMART AI system: check
5. the process of upgrading the legacy systems is part of the problem: check
6. the combination of a "newer" system with an "older" system, something that likely should have been planned for if the contractors weren't being rushed, was involved [1]: check
[1] https://www.flyingpenguin.com/faa-outage-why-duffy-is-spelle... - this is a much better article and no paywall
You take the Y out, you replace it with a E. And most people don't know that 'Dummbe' has a B, so we throw that out too.
Requires $50b to fix: check