The August 17 outage
github.blog
The August 17 outage
1–10 of 804 posts
Re: The August 17 outage
#2[dead]
Re: The August 17 outage
#3"We are committed to fixing these problems, as long as it doesn't involve buying things other than AI computers, hiring humans, or using non-Microsoft products."
Calling Azure the solution to this problem when it is in fact the source of most of these problems is just fantastic doublespeak.
Github is ripe for disruption and I hope it is disrupted soon.
Re: The August 17 outage
#4 "Since April, monthly commits have grown from 1.4 billion to 2.9 billion. "
Re: The August 17 outage
#5"We are committed to fixing these problems, as long as it doesn't involve buying things other than AI computers, hiring humans, or using non-Microsoft products." Calling Azure the solution to this problem when it is in fact the source of most of these problems is just fantastic doublespeak. Github is ripe for disruption and I hope it is disrupted soon.
If you're a big company, you can afford having one engineer spend one or two days per year to maintain your self-hosted GitLab or Forgejo. On top of better reliability than GitHub, you'll get the additional bonus that your source code won't accidentally leak through being in Copilot's training set.
If you're a hobbyist, Codeberg is great, has a nice community and automatically shields you from slop contributions.
Re: The August 17 outage
#6"We are committed to fixing these problems, as long as it doesn't involve buying things other than AI computers, hiring humans, or using non-Microsoft products." Calling Azure the solution to this problem when it is in fact the source of most of these problems is just fantastic doublespeak. Github is ripe for disruption and I hope it is disrupted soon.
> We installed as much hardware as available power allowed in our existing data centers while accelerating our migration to Azure.
And from the RCA [1]:
> The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic.
Re: The August 17 outage
#7> Errors in those services triggered a client-side retry loop that increased traffic during recovery.
The worst outages I've been part of always have some version of this :(
Re: The August 17 outage
#8Exponential growth. No company could handle that without some issues. Good luck to them. And for those who cannot tolerate this, there are many self hosted options.
Re: The August 17 outage
#9Almost 8 hours of downtime across all core workflows, and the word "sorry" or "apologize" appears nowhere in this post.
"If you were trying to ship software that day, we let you down" is classic corporate non-apology speak.
I’m done.
Re: The August 17 outage
#10And another outage. [0] Looking forward to the subsequent post-mortem on that one.
You might want to not go all in on GitHub anymore since it is very unstable to use. A self-hosted instance would have a far better uptime than GitHub over the years.
6 years ahead [1] on not going all in an centralizing everything on GitHub.