I wonder if it'll be a more useful course than the one we had when Crowdstrike broke everything, or the many times when AWS US-East-1 has fallen over taking down many popular services.
The lesson most people learn from this sort of outage is that things bounce back afterwards, and the risk is minimal enough that a day out of every few years isn't enough of a problem to pay more.
Microsoft says "while this took longer..." [1] but they should have said "while this is taking longer" because the outage isn't fixed yet. What kind of outage takes 2 days to fix?
"The incident was introduced by an issue during an earlier network capacity upgrade that surfaced following a recent network change. We have addressed the underlying issue and continue to restore service to remaining customers and regions.
Given the scale of demand, we restored services gradually, validating stability at each stage. While this took longer, it helped ensure a stable restoration."
The postmortem on this will be interesting. A few days prior Microsoft said several versions of on premise Exchange are permanently vulnerable.
Maybe AI scanning found these Exchange exploits and after laying off so many developers, they vibe coded an exploit fix and deployed it. But behind the scenes it is all just still Exchange which if anyone has experience managing, its a complex nightmare of code going back decades. With all sorts of Active Directory requirements and everything else. The mail database system it uses, and probably still used in Office 365 (Office 363 today?) is also a nightmare.
> But behind the scenes it is all just still Exchange which if anyone has experience managing, its a complex nightmare of code going back decades. With all sorts of Active Directory requirements and everything else. The mail database system it uses, and probably still used in Office 365 (Office 363 today?) is also a nightmare.
I hear this so much, but this hasn't been my experience. My experience goes back to Exchange 4.0. I've run every version of Exchange in production. Am I the only person who has a favorable impression of the product?
Exchange and Active Directory are easily my favorite MSFT products. I've always wished they would port them to Linux, even though I know that OS dependencies make that a non-starter (though they surprised me w/ SQL Server for Linux).
I'll concede that the the biggest Exchange environment I ever ran was around 1,000 mailboxes. I've never seen how it fails in really large environments. I was also fortunate that I only ran the earlier versions (pre-2013) in sub-500 mailbox environments. In those settings, though, I found it worked well and wasn't bad to administer.
It was very reasonable to use in small business environments, too (sub-100 mailboxes). I put a lot of little Exchange 2003 boxes out in small offices. It plugged-away very effectively.
Yes-- there were updates here-and-there that had issues, for sure. By-and-large it was a stable and predictable product.
I always tried to be mindful of maintaining free disk space, following MSFT's recommendations for speeds-and-feeds for I/O, and running on solid hardware (ECC memory, redundant power, UPS). I feel like I would have had more problems if I'd scrimped on that stuff, for sure, but I was unwilling to let Customers cheap-out.
With Exchange 2019 and high-availability (database availability groups) I was easily able to do zero-downtime maintenance. That "big for me" Exchange Customer I worked with had more downtime in the last 24 hours than they did the entire time they used Exchange 2019 in production.
I haven't seen the code, so I can't speak to the "nightmare of code". Large parts of it were ported to the .NET Framework in between the 2010 and 2013 releases. The 2013, 2016, 2019, and current Subscription Edition releases appear to be an evolution of the codebase with no signs of major rewrites. I question how much pre-2013 code is even present in the codebase anymore.
I dunno. My experience just hasn't been that bad. I don't want to be an apologist for MSFT. I've never fully gotten over the suspicion I have for their motives going all the way back to the Halloween documents. I don't like how they're forcing everyone to hosted offerings and killing their on-prem products. I've been pleased with the value my Customers realized from Exchange.
> I hear this so much, but this hasn't been my experience. My experience goes back to Exchange 4.0. I've run every version of Exchange in production. Am I the only person who has a favorable impression of the product?
yup. You're the only one.
When we ran our own mail servers, "some fuckery with client's Exchange server config" was good 90% of mail problems we had.
And my experience with the cloud version management decided we will migrate to has been entirely awful too, simple stuff like "stop spamming the mailing list I get constantly" is entirely impossible to set up, despise both global and per mailbox rules to let it thru, it still lands in spam and the useless support we apparently pay for can't figure out why
Managing it in small and large corporate enviornments for over a decade led me to believe that when it was working, Exchange was very stable and reliable.. But when things would break, they would break in a way that required extensive knowledge of all the areas to fix. Which was a form of job security so Im not complaining.
I guess past reliability might be irrelevent here if an AI slop patch for AI discovered exploits caused the outage. Just speculation though. Since exchange has so many moving parts and so much legacy code, with even more bolted on to make it a cloud service, I could see how an AI exploit fix would not have all the context needed.
There is no information that it was caused by Exchange either, guess we have to wait on those answers. It is just a reasonable assumption.
naa .. it's totally fine. The only thing that will move consciences will be data loss/leak at scale. But even in that case I have some doubts ...
And, if eventually a big bunch of CTOs starts considering going back so self-hosting/managing people will pop up everywhere saying "even if <big corp> sucks, you cannot do better" or "it will cost you more" or "what about the peak usage".
It's incredible how we ended up paying much more, giving voluntarily all our data, and getting a shitty service.
I wonder if it'll be a more useful course than the one we had when Crowdstrike broke everything, or the many times when AWS US-East-1 has fallen over taking down many popular services.
The lesson most people learn from this sort of outage is that things bounce back afterwards, and the risk is minimal enough that a day out of every few years isn't enough of a problem to pay more.
especially if everyone else is also down with you
at least when us-east-1 goes down, Amazon stock prices go up (outages => stockholders remember that aws exists).
Microsoft says "while this took longer..." [1] but they should have said "while this is taking longer" because the outage isn't fixed yet. What kind of outage takes 2 days to fix?
[1] https://x.com/Microsoft/status/2121282013600907325
"The incident was introduced by an issue during an earlier network capacity upgrade that surfaced following a recent network change. We have addressed the underlying issue and continue to restore service to remaining customers and regions.
Given the scale of demand, we restored services gradually, validating stability at each stage. While this took longer, it helped ensure a stable restoration."
The postmortem on this will be interesting. A few days prior Microsoft said several versions of on premise Exchange are permanently vulnerable.
Maybe AI scanning found these Exchange exploits and after laying off so many developers, they vibe coded an exploit fix and deployed it. But behind the scenes it is all just still Exchange which if anyone has experience managing, its a complex nightmare of code going back decades. With all sorts of Active Directory requirements and everything else. The mail database system it uses, and probably still used in Office 365 (Office 363 today?) is also a nightmare.
Just speculation though
> But behind the scenes it is all just still Exchange which if anyone has experience managing, its a complex nightmare of code going back decades. With all sorts of Active Directory requirements and everything else. The mail database system it uses, and probably still used in Office 365 (Office 363 today?) is also a nightmare.
I hear this so much, but this hasn't been my experience. My experience goes back to Exchange 4.0. I've run every version of Exchange in production. Am I the only person who has a favorable impression of the product?
Exchange and Active Directory are easily my favorite MSFT products. I've always wished they would port them to Linux, even though I know that OS dependencies make that a non-starter (though they surprised me w/ SQL Server for Linux).
I'll concede that the the biggest Exchange environment I ever ran was around 1,000 mailboxes. I've never seen how it fails in really large environments. I was also fortunate that I only ran the earlier versions (pre-2013) in sub-500 mailbox environments. In those settings, though, I found it worked well and wasn't bad to administer.
It was very reasonable to use in small business environments, too (sub-100 mailboxes). I put a lot of little Exchange 2003 boxes out in small offices. It plugged-away very effectively.
Yes-- there were updates here-and-there that had issues, for sure. By-and-large it was a stable and predictable product.
I always tried to be mindful of maintaining free disk space, following MSFT's recommendations for speeds-and-feeds for I/O, and running on solid hardware (ECC memory, redundant power, UPS). I feel like I would have had more problems if I'd scrimped on that stuff, for sure, but I was unwilling to let Customers cheap-out.
With Exchange 2019 and high-availability (database availability groups) I was easily able to do zero-downtime maintenance. That "big for me" Exchange Customer I worked with had more downtime in the last 24 hours than they did the entire time they used Exchange 2019 in production.
I haven't seen the code, so I can't speak to the "nightmare of code". Large parts of it were ported to the .NET Framework in between the 2010 and 2013 releases. The 2013, 2016, 2019, and current Subscription Edition releases appear to be an evolution of the codebase with no signs of major rewrites. I question how much pre-2013 code is even present in the codebase anymore.
I dunno. My experience just hasn't been that bad. I don't want to be an apologist for MSFT. I've never fully gotten over the suspicion I have for their motives going all the way back to the Halloween documents. I don't like how they're forcing everyone to hosted offerings and killing their on-prem products. I've been pleased with the value my Customers realized from Exchange.
> I hear this so much, but this hasn't been my experience. My experience goes back to Exchange 4.0. I've run every version of Exchange in production. Am I the only person who has a favorable impression of the product?
yup. You're the only one.
When we ran our own mail servers, "some fuckery with client's Exchange server config" was good 90% of mail problems we had.
And my experience with the cloud version management decided we will migrate to has been entirely awful too, simple stuff like "stop spamming the mailing list I get constantly" is entirely impossible to set up, despise both global and per mailbox rules to let it thru, it still lands in spam and the useless support we apparently pay for can't figure out why
Managing it in small and large corporate enviornments for over a decade led me to believe that when it was working, Exchange was very stable and reliable.. But when things would break, they would break in a way that required extensive knowledge of all the areas to fix. Which was a form of job security so Im not complaining.
I guess past reliability might be irrelevent here if an AI slop patch for AI discovered exploits caused the outage. Just speculation though. Since exchange has so many moving parts and so much legacy code, with even more bolted on to make it a cloud service, I could see how an AI exploit fix would not have all the context needed.
There is no information that it was caused by Exchange either, guess we have to wait on those answers. It is just a reasonable assumption.
> The postmortem on this will be interesting.
And bets on whether it's DNS?
my bet is on some copilot integration
Well, my home computer with ddns lost network connection at the exact same time and I had to reboot it for it to work so, maybe?
It's never DNS
...given it is windows environment... maybe
DNS, IPV6 or BGP. It’s always those guys
Certificates expiring
naa .. it's totally fine. The only thing that will move consciences will be data loss/leak at scale. But even in that case I have some doubts ...
And, if eventually a big bunch of CTOs starts considering going back so self-hosting/managing people will pop up everywhere saying "even if <big corp> sucks, you cannot do better" or "it will cost you more" or "what about the peak usage".
It's incredible how we ended up paying much more, giving voluntarily all our data, and getting a shitty service.
It’s pretty crazy how much Microsoft can get away with. Almost a government level of getawayability.