well written
The August 17 outage
71–80 of 804 posts
Re: The August 17 outage
#72AWS CloudWatch has an option to show the trend and what it will be like after x-period. Doesn't Azure have such options so that engineers can predict to scale better? Seems like engineers are not ready for this per postmortem
[deleted]
Re: The August 17 outage
#73Earlier quoted context omitted.
It really is. I know I've gone from tens a month to thousands a month. They have to be projecting >100B/month in the next year or two.
wow. they should really institute a maximum amount of individual pushes per-month per-user.
Which will just increase the cries of "enshittification" and hasten the mass migration to the next free platform that surely, this time, won't ever go down.
Re: The August 17 outage
#74Almost 8 hours of downtime across all core workflows, and the word "sorry" or "apologize" appears nowhere in this post. "If you were trying to ship software that day, we let you down" is classic corporate non-apology speak. I’m done.
thats what i liked about it. its fact and action oriented. what does a "sorry" buy you that the "we let you down" doesn't.
Re: The August 17 outage
#75Re: The August 17 outage
#76Reading this port-mortem / plan shocks me, this doesn't look like a service that has been serving high-throughput services for more than a decade. In fact it is almost like they've barely started. It seems the solution has been capacity, capacity rather than architectural or data changes. > Our next milestone is an architecture that scales read capacity linearly with the number of readers, enabling unlimited read ope…
> this doesn't look like a service that has been serving high-throughput services for more than a decade. In fact it is almost like they've barely started.
Well that's because in comparison to the absolute flood of traffic brought on by AI, they really haven't been operating on this scale before.
Re: The August 17 outage
#77"Since April, monthly commits have grown from 1.4 billion to 2.9 billion. " Wow, that is some incredible growth in a really short time.
Not that impressive when you realize it's mostly due to AI slop
edit: AI actually writes 99.9% of my code these days. I'm just saying of course the number of commits to github is going to climb astronomically due to AI.
Re: The August 17 outage
#78Everyone suggesting that they simply charge users for commits to drive off AI-heavy users forgets that Github is owned by Microsoft, who has a big incentive to keep having developers use AI.
I suspect that Microsoft would even prefer to have Github operate at a loss, if that loss were because all its users were using their models and paying for OpenAI subscriptions to generate the code.
Re: The August 17 outage
#79Earlier quoted context omitted.
The issue with these systems is that they lack Github's sophistication for issue tracking, knowledge transfer, and automation. I think Gitlab is a mature product in its own space and unlikey to change, for instance, at this point. Codeberg also has the issue of having a political stance which means they will not accept just anyone's use of the platform. That is absolutely their right and I have no issue with it, but…
Gitlab has the benefit of having very little traffic, both free and paid. Their limits are still way above the current usage so less likely to be an issue
Worth mentioning GitLab's paid enterprise offering are more expensive than GitHub's, on a per-seat basis.
Lots of companies moved because it was cheap, but it's not anymore. Ironic that companies might choose to migrate to them now for stability, rather than price.
Re: The August 17 outage
#80Earlier quoted context omitted.
A server box now has around 256 CPU cores. So that's about 12000 servers. If each one is $10k that's $120 million. Not a lot compared to Github's income.
Does that $10k server not need RAM?
If parent commenter is off by an order of magnitude in their costs, their point still stands.