“We believe we have found the cause of the problem and are working on mitigation.” Wow - they believe they’ve found it! Comforting.
Having worked at Microsoft in the recent past I’m not surprised everything is starting to fall over a lot more.
While there are tons of good people at Microsoft, the overall culture and number of people who stand in the way of getting stuff done well is immense. People view you or your team doing work as an existential threat to their team, and react accordingly.
I don’t think the recent uptick in failures is related to vibe coding or AI and has everything to do with the corporate culture. Probably some poor product manager was tasked with closing 10k automated security flags/tickets and in doing so broke some load bearing microservice that depended on a kind of service principal that’s now not allowed.
I have great compassion for the people who sort these things out. It takes a lot of skill and experience to pick up the pieces of something like Github and get it working properly again. I'm sure it's exhausting having problem after problem, endlessly chasing down sev-0 bugs in production.
But I have nothing but disdain for the management that won't listen to those people.
That's only part of the problem really. The other thing is how many downstream things rely on GH being up. Things like Go packages, Helm charts, blah blah...
i wonder what sorts of creative "haha copilot blah blah" and "haha microsoft blah blah" and "but they said AI would do xyz!" jokes we will get to see today.
(please github, you need more uptime so that people can come up with some new jokes)
Copilot has improved GitHub's uptime from five nines to nine fives, a four magnitude improvement!
“We believe we have found the cause of the problem and are working on mitigation.” Wow - they believe they’ve found it! Comforting.
Having worked at Microsoft in the recent past I’m not surprised everything is starting to fall over a lot more.
While there are tons of good people at Microsoft, the overall culture and number of people who stand in the way of getting stuff done well is immense. People view you or your team doing work as an existential threat to their team, and react accordingly.
I don’t think the recent uptick in failures is related to vibe coding or AI and has everything to do with the corporate culture. Probably some poor product manager was tasked with closing 10k automated security flags/tickets and in doing so broke some load bearing microservice that depended on a kind of service principal that’s now not allowed.
I have great compassion for the people who sort these things out. It takes a lot of skill and experience to pick up the pieces of something like Github and get it working properly again. I'm sure it's exhausting having problem after problem, endlessly chasing down sev-0 bugs in production.
But I have nothing but disdain for the management that won't listen to those people.
It's time for some anonymous account or blog post try to share the insights on what the heck is happening, it's not normal, it can't be normal.
They also had to remove GPT-5.3-Codex "as we focus on our platform reliability" https://x.com/github/status/2021040916451164412
Ah the daily HN get together for devs waiting for Github to resolve its issues.
They're really struggling to keep their last 9 of uptime.
I thought AI was supposed to replace Software engineers by now?
It's already, that's why Microsoft abandoned GitHub
If you can't deploy to production when some GitHub services hiccup maybe the problem is not on GitHub.
For folks still on GitHub for $JOB, are you looking at alternatives yet?
That's only part of the problem really. The other thing is how many downstream things rely on GH being up. Things like Go packages, Helm charts, blah blah...
Yep. Daily. I can't bear GitLab though, so currently I'm thinking self-hosted Gitea or OneDev.
Again... Unicorns when trying to view files or PRs, errors trying to leave comments or review things if they do load.
As is tradition
i wonder what sorts of creative "haha copilot blah blah" and "haha microsoft blah blah" and "but they said AI would do xyz!" jokes we will get to see today.
(please github, you need more uptime so that people can come up with some new jokes)
Nice to see you all again!
If you want to kill some (down)time, checkout https://www.mr-spankys-meatballs.com - a FPS built with Svelte and ThreeJS.
Moving GitHub to Azure has made GitHub as reliable as everything else that runs on Azure.
DJ Khaled: Another One
(Sorry for low effort comment).
Who needs GitHub anyway?
Another day, another downtime. Lovely.