AWS CloudWatch has an option to show the trend and what it will be like after x-period.
Doesn't Azure have such options so that engineers can predict to scale better? Seems like engineers are not ready for this per postmortem
The August 17 outage
21–30 of 804 posts
Re: The August 17 outage
#22Github down, no hard drives available, no memory available, thanks AI! Seems like we are headed for Tech Gridlock.
This stuff is good! This is what a booming economy looks like. There are people out there competing with you for resources because they have cool ideas they want to implement.
Re: The August 17 outage
#23[deleted]
Re: The August 17 outage
#24Re: The August 17 outage
#25We need to have a package of FLOSsoftware that you could run on the cloud of your choice that offers most of what GitHub does (niceties on top of Git) without the centralization. GitLab was close last I remember but there was some sort of enterprise tier when I tried hosting stuff on a local server years ago. I want true FLOSS, not another SaaS equivalent of the coke dealer giving clients the good uncut stuff when th…
Have you looked into Forgejo?
Re: The August 17 outage
#26"We are committed to fixing these problems, as long as it doesn't involve buying things other than AI computers, hiring humans, or using non-Microsoft products." Calling Azure the solution to this problem when it is in fact the source of most of these problems is just fantastic doublespeak. Github is ripe for disruption and I hope it is disrupted soon.
> Github is ripe for disruption and I hope it is disrupted soon. It's an expensive, low revenue generating site. There are, and have always been, competitors, including "host it all yourself" solutions, but nothing has really stuck. How is it "ripe" for disruption?
They had $1b revenue in 2023 and now probably more than $2b in revenue... do you have cost figures showing what their expenses are?
Re: The August 17 outage
#27Github down, no hard drives available, no memory available, thanks AI! Seems like we are headed for Tech Gridlock.
What they can implement is to slowdown the commit rate, rate limt or just queue-up messages not to overburden their downstream service.
I don't think GH has any of those, but just keep scaling, but that scaling failed.
Just bad architectural decisions from the postmortem.
--
It will only get worse due to AIs spawning massive commits, and they don't have unlimited cloud resource.
They can scale but not scalable in terms of effort, resources, and $
Re: The August 17 outage
#28Earlier quoted context omitted.
If you're a big company, you can afford having one engineer spend one or two days per year to maintain your self-hosted GitLab or Forgejo. On top of better reliability than GitHub, you'll get the additional bonus that your source code won't accidentally leak through being in Copilot's training set. If you're a hobbyist, Codeberg is great, has a nice community and automatically shields you from slop contributions.
The issue with these systems is that they lack Github's sophistication for issue tracking, knowledge transfer, and automation. I think Gitlab is a mature product in its own space and unlikey to change, for instance, at this point. Codeberg also has the issue of having a political stance which means they will not accept just anyone's use of the platform. That is absolutely their right and I have no issue with it, but…
I never found GitHub's systems for issue tracking to be all that great. Cross-repository issues and development plans are hard to track within a git host. I've always used an external panning and issue tracking tool, mostly Linear, and it works really well. GitLab's Linear integration is excellent, FWIW.
I've actually worked with a couple of companies who do use GitHub for their code, and they all use Linear in addition to GitHub.
I understand the concern you're talking about wrt. Codeberg, but I wouldn't view it as a significantly bigger risk than anything else. Any platform can suddenly decide that your project is against ToS (GitHub will absolutely not accept just anyone's use of their platform either) and Codeberg introducing some rules recently doesn't, in my mind, drastically increase the risk of a dramatic ToS change in the future. But we all have to make our own risk evaluations and I won't judge yours. Luckily, moving between Git hosts isn't that difficult; setting up CI again and losing merge request history does suck but it's not the end of the world, unlike something like, say, losing your AWS/GCP/whatever account.
Re: The August 17 outage
#29> Errors in those services triggered a client-side retry loop that increased traffic during recovery. The worst outages I've been part of always have some version of this :(
the 'ol thundering herd problem...
Re: The August 17 outage
#30Earlier quoted context omitted.
[flagged]
Baseless accusation made from a position of zero information.
Opinion based on stated facts.
Please share the information you have which contradicts the conclusions I have drawn from Github's statement.
(And we know they're liars. They report very few of the actual incidents they have; see for example https://mrshu.github.io/github-statuses/)