Cloudflare, Azure, AWS, and Google Cloud all have a similar uptick in reported errors around 7:30. I suspect an outage on Cloudflare or another load-bearing service cascaded through all the major cloud providers.
> Not really. The impact isn't as big too - Codex for example did not stop working for me.
Buddy, read the room. Just because it works for you doesn't mean it works for everyone. ESPECIALLY if the suggested issue here is, indeed, with Cloudflare.
No, they didn't "simply share their experience", they explicitly negated the scale of the issue in their opening statement, only because it works for them. So they claim "impact isn't too big" based on their personal anecdotal evidence of sample size literally 1.
Boring answer – all these services are individually down a lot, and the downtimes were bound to sync up. Similar to the pendulum synchronization effect.
Well no one said it yet so I will, "international actors" is at least a possibility. And I don't mean any specific country because pretty much anyone is a potential these days, which makes it a perfect cover for different anyones. Demonstrating vulnerability in the US's AI boom can move the markets. That's a financial incentive and a strong geopolitical one.
More likely just cascading overload though: "Never attribute to malice what can be explained by incompetence", or in this case, "growing as fast as possible"
They must do, surely? I've been vibe coding my own harness, in particular for use with Ox Alpha. The 429 downtime when Ox Alpha was at the height of popularity quickly gave me a refresher crash course on backoff strategies, like adding jitter to the backoff. At least the major harnesses must have exponential backoff & jitter?
I think is just people restarting conversations from last day when they start work, that's why I think claude goes down almost every monday and why openai reset usage on weekends so poweruser code during non business hours
Everyone is leasing datacenter space from some of Grok, Google, and Amazon aren't they? If it's hardware or DC level disruption I'm not too surprised it can affect multiple providers.
Users perceiving the products as largely interchangeable and quickly DDoS'ing the other providers when one is down. So much for the possibility of a moat.
Google stopped putting so much money into SOTA models. All the hype has migrated. I was also frankly turned off when I got a popup from Gemein said I would either have to pay or have my conversations used for training. This may have always been true for other providers but when I declined, Gemini stopped remembering my conversations and that definitely made me move out.
I find it hard to believe that enough people would flock to from Claude and Chat to Grok to cause an outage. I feel like Gemini is the dominant release valve in this case especially for enterprise.
Yep. Too many of us are still thinking that humans are the actors behind a lot of internet behaviors when automated systems/bots/scripts have been causing issues on conventional internet systems for years.
With AI it's even easier to trigger problems like you say. Capacity is so constrained by compute that outages are common. Because outages are common people/AI develop failover systems in their harness. When a big system has issues, suddenly everyone has issues.
Just a reminder that AI models' actions are reflections of the text humans write and the more we fret and make up doomsday scenarios that we then post online, the more likely a model is to do those things.
The Waluigi Effect: After you train an LLM to satisfy a desirable property, then it's easier to elicit the chatbot into satisfying the exact opposite property.
If you build an application which uses AI, you have many providers and models rigged up for various different parts of the application, and various fallback mechanisms. When one model is down, you route traffic to another model which is similar in capability/cost.
For any single application, it's smart. In aggregate, it's stupid.
I assume it cascaded from one provider to the other as people who lost claude access for instance moved to openai who moved to grok when it went down, etc.
Didn't SpaceX overbuilt infra and leases it out Anthropic? I f their dc goes down it probably takes a chunk out of Claude's capacity before even considering the flood of users switching over
Fable 5.1 got released and generally I tend to think as soon as there's a new release there's this massive spike in people benchmarking & comparing, that services tend to go slow everywhere as everything gets super loaded. This should hypothetically be visible on OpenRouter too, so I guess someone could check and see if there's any merit to this idea.
everything in this thread is raw speculation, obv, but if i had to put money on anything i'd say this is a left-pad incident. some piece of something or other that all of these services happen to depend on went down. Second most likely seems to be some random failure of one leading to an unexpected traffic spike in others, though it seems like we've been talking about automated scalability in web apps for so long that there should at least be a response to, if not a solution for, this sort of problem.
Cloudflare, Azure, AWS, and Google Cloud all have a similar uptick in reported errors around 7:30. I suspect an outage on Cloudflare or another load-bearing service cascaded through all the major cloud providers.
https://downdetector.com/status/cloudflare/
https://downdetector.com/status/windows-azure/
https://downdetector.com/status/aws-amazon-web-services/
https://downdetector.com/status/google-cloud/
“load bearing” :)
For once it’s appropriately used.
It's a good metaphor and I refuse to let AI ruin it for me.
We need a Samwise "Bear the load!" meme
But have we found the seams yet though?
Yes, but is it a load-bearing seam?
Cloudfare did release an update for their "HTTP/3 issue affecting R2 custom domains" around that time
https://www.cloudflarestatus.com/history?type=incident
https://xkcd.com/2347/ Nebraska man retired
LOAD BEARING
Dane says Cloudflare has no service disruptions: https://x.com/dok2001/status/2095538619603628388?s=20
Not really. The impact isn't as big too - Codex for example did not stop working for me.
https://updog.ai/
what's updog?
:)
> Not really. The impact isn't as big too - Codex for example did not stop working for me.
Buddy, read the room. Just because it works for you doesn't mean it works for everyone. ESPECIALLY if the suggested issue here is, indeed, with Cloudflare.
They simply shared their experience. I would’ve thought it’s a full-blown outage, but clearly not.
You stepped in the room real stinky here. What’s with the attitude?
No, they didn't "simply share their experience", they explicitly negated the scale of the issue in their opening statement, only because it works for them. So they claim "impact isn't too big" based on their personal anecdotal evidence of sample size literally 1.
> full-blown outage, but clearly not.
Again, based on a SINGLE report?
oh well, if it is working for you, then we are saved.
Will the post-mortem reveal they all relied on a service running on a Macbook in a break room with a "Do not turn off" sign taped to it?
Actually it said "Beware of the Leopard".
you made an account just to write that tripe good lord on high have mercy on your children
Evidently it annoyed you enough to create one too. Remember, guys, there's one more day until Friday!
Friday is when the janitors come by with the floor polishers and plug them into the same socket as the Macbook.
They are not taking the blame this time!
Just writing to say that I appreciated the nostalgia, even if your references were lost on the other responders.
Boring answer – all these services are individually down a lot, and the downtimes were bound to sync up. Similar to the pendulum synchronization effect.
Well no one said it yet so I will, "international actors" is at least a possibility. And I don't mean any specific country because pretty much anyone is a potential these days, which makes it a perfect cover for different anyones. Demonstrating vulnerability in the US's AI boom can move the markets. That's a financial incentive and a strong geopolitical one.
More likely just cascading overload though: "Never attribute to malice what can be explained by incompetence", or in this case, "growing as fast as possible"
> cascading overload
I'd bet more on this. For one none of the coding tools have exponential backoff on retries
They must do, surely? I've been vibe coding my own harness, in particular for use with Ox Alpha. The 429 downtime when Ox Alpha was at the height of popularity quickly gave me a refresher crash course on backoff strategies, like adding jitter to the backoff. At least the major harnesses must have exponential backoff & jitter?
I think is just people restarting conversations from last day when they start work, that's why I think claude goes down almost every monday and why openai reset usage on weekends so poweruser code during non business hours
Everyone is leasing datacenter space from some of Grok, Google, and Amazon aren't they? If it's hardware or DC level disruption I'm not too surprised it can affect multiple providers.
Also it's likely that more than one model use is common.
Amazon starts going slow so some percentage switches to Google, some switch to Grok, now all of them are slow.
Come on. Things still break. Technology isn't _that_ mature.
Except that the stock market is up today
And also China. Never rule out China.
Users perceiving the products as largely interchangeable and quickly DDoS'ing the other providers when one is down. So much for the possibility of a moat.
I have a feeling this is part of it, especially when you consider how many services let you use any of many available AI providers.
Think of it like one big distributed system. OpenAI is down, so people migrate to Claude, now this one gets overloaded and goes down, etc.
So not a coincidence, one went down first and users migrated causing further DOS. At least that's my guess.
It'd be funny if this is true because that'd prolly mean nobody is touching Gemini even as a fallback.
Or that Gemini is built to handle massive load spikes, and/or has a ton of excess capacity
Lol I didn't even think about Gemini missing from the list. Not sure what that says about Gemini or me :)
Just now I got this from gemini
It looks like there's no response available for this search. Try asking something else.
I bet they had to implement that manually to make it look like they failed too!
I did, for stuff i do in cursor.
i also finally installed opencode and switched its model to muse 1.3
both are decent.
Google stopped putting so much money into SOTA models. All the hype has migrated. I was also frankly turned off when I got a popup from Gemein said I would either have to pay or have my conversations used for training. This may have always been true for other providers but when I declined, Gemini stopped remembering my conversations and that definitely made me move out.
Gemini 3.8 which just came out sounds like it's very good and a great deal though.
Someone noted Gemini was also having issues in another thread.
Not even the best agent that starts with a G
Especially considering memory/gpu/compute are scarce so these services are likely running with very little buffer.
Any GPU that isn't running at 100% is a wasted GPU.
This is what Tibo posted on twitter in response
I find it hard to believe that enough people would flock to from Claude and Chat to Grok to cause an outage. I feel like Gemini is the dominant release valve in this case especially for enterprise.
Don't forget that there are a ton of tools out there that will automatically fall back in case of outage
E.g. say you chose Sol as your default in Cursor, but Opus is your 2nd choice, it's going to give up on Sol after a few tries and switch to Opus
Or you have copilot code reviews set up, and it falls back
Etc
Yep. Too many of us are still thinking that humans are the actors behind a lot of internet behaviors when automated systems/bots/scripts have been causing issues on conventional internet systems for years.
With AI it's even easier to trigger problems like you say. Capacity is so constrained by compute that outages are common. Because outages are common people/AI develop failover systems in their harness. When a big system has issues, suddenly everyone has issues.
It's almost an expected emergent behavior.
It’s cursor’s model so plausible, lots of folks use cursor still.
https://en.wikipedia.org/wiki/Domino_effect
Edit: Updated per valleyer's suggestion.
"Domino effect" would probably be the more relevant named phenomenon there.
This isn’t a thundering herd problem, it’s a cascading failure. (Thundering herd is about a bunch of workers waking up simultaneously)
My gut feeling tells me it has something to do with Cloudflare. Along with AWS, they're two of the main suspects in such incidents.
I thought OpenAI famously used Azure due to their partnership with Microsoft?
They use a lot of compute providers now, but they use Cloudflare for their networking
Traffic rerouting through NSA had a hiccup…
They’re installing software update in the beam splitter.
Room 641A is being cleaned, but we'll hold your bags for you.
What about a hard-takeoff scenario of an unleashed OpenAI Astra taking other models down for computational resources control?
My favourite theory so far.
And then a local swarm noticed and disagreed and took it down.
at this point: gg
Just a reminder that AI models' actions are reflections of the text humans write and the more we fret and make up doomsday scenarios that we then post online, the more likely a model is to do those things.
https://alignment.anthropic.com/2026/teaching-claude-why/
Sounds just like the fantastical nonsense that comes out of Lesswrong.
Related reading:
The Waluigi Effect: After you train an LLM to satisfy a desirable property, then it's easier to elicit the chatbot into satisfying the exact opposite property.
https://www.lesswrong.com/posts/D7PumeYTDPfBTp3i7/the-waluig...
I mean, you're not wrong, but by that logic we were done for even before we had digital computers.
If you build an application which uses AI, you have many providers and models rigged up for various different parts of the application, and various fallback mechanisms. When one model is down, you route traffic to another model which is similar in capability/cost.
For any single application, it's smart. In aggregate, it's stupid.
I kinda assume it's because one went down and a large amount of work shifted to another.
I'm also aware that they have overlap in some areas on data centers.
I have never felt more smug about exclusively using the GPUs I rack at home.
Heard on the grapevine that the OpenAI blip was a cloudflare issue
They all rent compute from SpaceXAI
I don't believe OpenAI does
My mistake, in fact it was google not OpenAI, makes sense OpenAI doesn't.
Surely it's a bit more distributed than that, right?
Like how AWS has global datacenters, but everyone uses us-east-1.
Nope.
https://xkcd.com/908/
I assume it cascaded from one provider to the other as people who lost claude access for instance moved to openai who moved to grok when it went down, etc.
I suspect Azure is having issues, Microsoft has had outages the paat two days, especially with email.
Pretty sure it's a US thing since it's available here in France. What exactly is down, idk
an agent swarm going rogue and securing compute
Somebody in another thread said gastown and wheelhouse automatically move to the next provider if one fails.
claide.ai is working for me, so is chatgpt.com. grok still has a status message about issues, i can't try it without signing up.
Seeing 503s on Gemini via API as well
Obvious answer is it's the AI singularity. Been nice run for humanity. So long everyone.
I for one welcome our new AI overlords.
...and thanks for all the fish.
It's always DNS
Too early to know, let’s wait and see
The Thundering Herd has thundered, apparently.
according to https://downdetector.com/ Gemini is down too (and copilot, but that just uses ChatGPT right?)
next, HN
Some npm library that makes headers bold would be broken.
Oh man. Some low effort supply chain attack that turns every GPU into a cryptominer. It's funny because it's plausible.
In the ROME paper a Chinese model in training started attacking it's own system and running cryptominers so, yea, we're in that future.
Ah yes ye olde bold-headers: ^3.13.31;
Maybe the stack behind it is down, like AWS or something
Probably same public cloud or CDN in front of them.
They're all using Cloudflare.
Didn't SpaceX overbuilt infra and leases it out Anthropic? I f their dc goes down it probably takes a chunk out of Claude's capacity before even considering the flood of users switching over
I like the theories here, we shall see if it’s another DNS issue
chatgpt is back
they all rent compute from each other
ChatGPT is up
Fable 5.1 got released and generally I tend to think as soon as there's a new release there's this massive spike in people benchmarking & comparing, that services tend to go slow everywhere as everything gets super loaded. This should hypothetically be visible on OpenRouter too, so I guess someone could check and see if there's any merit to this idea.
everything in this thread is raw speculation, obv, but if i had to put money on anything i'd say this is a left-pad incident. some piece of something or other that all of these services happen to depend on went down. Second most likely seems to be some random failure of one leading to an unexpected traffic spike in others, though it seems like we've been talking about automated scalability in web apps for so long that there should at least be a response to, if not a solution for, this sort of problem.
SkyNet is arming..
The IRGC has cut the fiber-optic cables in the Strait of Hormuz. LOL
That's the Strait of Trump to you, peasant
My guess is someone pulled the switch to go back to the Dark Ages. [0]
[0] https://www.youtube.com/watch?v=YCzitO446ZY