I don't understand why Github hasn't solved this problem with pricing updates. My understanding is they are getting hammered with LLM generated code growing their traffic by over an order of magnitude. So why not rate limit non-paying users and charge for whatever scarce resource is being consumed that is causing them to constantly fall-over? This seems like a basic economics problem.
Incident with Github.com [resolved]
171–180 of 1001 posts
Re: Incident with Github.com [resolved]
#172Can anyone shed some light why GitHub is having so many problems lately?
Microsoft bought it, https://damrnelson.github.io/github-historical-uptime/
Re: Incident with Github.com [resolved]
#173Earlier quoted context omitted.
That's fine, but rate limiting is a solved problem at this point. They need to rate-limit free accounts and be realistic about their limits instead of letting the system fail.
Probably can't do that easily due to shitty engineering decisions in the past
Or inept managers.
Manager: Why is the site down!?
Engineer: Because it can’t handle this level of traffic. We can rate limit to the level we can handle to keep the service running.
Manager: Okay, let’s do that!
Engineer: Okay. Rate limiting at N req/min.
Manager: I’m getting complaints we’re denying service to customers!? We can’t rate limit our users if they want to use it. Increase the rate limits.
I’ve had this conversation more than once in my career.
Re: Incident with Github.com [resolved]
#174I recall reading years ago that cloud services were expected to run with a reliability of 3 or 4 '9's and that if they didn't competing services would quickly overtake them in adoption. The industry was supposed to be that cut throat. Has big tech reached a similar status like banks in that they are "too big to fail" i.e. when they do fail we all just look the other way and say: "well everyone else is out too". Didn'…
Competing services can't afford the data storage to compete with Big tech
Re: Incident with Github.com [resolved]
#175Maybe they should charge a little more per action, and the onslaught would stop.
Seems like a simple solution.
Re: Incident with Github.com [resolved]
#176To everyone who is angry: calm down. Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code Unless you have a emergency hotfix (you don’t), go hit the gym or walk outside. If a work tool going down triggers you enough mentally to start angrily ranting online, it’s a sign you need to chill out and focus more on your health
What if you, you know, have a job?
Re: Incident with Github.com [resolved]
#177I recall reading years ago that cloud services were expected to run with a reliability of 3 or 4 '9's and that if they didn't competing services would quickly overtake them in adoption. The industry was supposed to be that cut throat. Has big tech reached a similar status like banks in that they are "too big to fail" i.e. when they do fail we all just look the other way and say: "well everyone else is out too". Didn'…
That canot happen if the market becomes monopolistic, with every bigtech out there buying every startup worth a penny. The erosion of anti trust in the US created this monstrosity.
Re: Incident with Github.com [resolved]
#178To everyone who is angry: calm down. Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code Unless you have a emergency hotfix (you don’t), go hit the gym or walk outside. If a work tool going down triggers you enough mentally to start angrily ranting online, it’s a sign you need to chill out and focus more on your health
I understand the need to level-head the discussion and calming everyone.
However, your "(you don't)" comment is not going to calm down all the people, who, you know, DO have a hotfix to push now, and DO have an angry customer that could not care less for which part of our infrastructure is breaking _their_ workflow.
The only things would calm everyone down is guarantee that Microsoft would be paying for _our_ SLA breach compensation. But they don't. And I don't think anyone is paying their GH bill with a prorata of the number of time the platform was actually available.
(I'm also aware that the wording of the contract probably clearly says that you should not use GitHub for anything critical, that Microsoft is only a small startup in their garage, that you can't credibly expect 90% uptime anyway, and that it's all the fault of LLM slop ! Bad LLM slop ! Also, please buy our LLMs to generate more slop, please.)
Re: Incident with Github.com [resolved]
#179This is starting to feel like Twitter. For a variety of reasons, we had a centralized place where everyone of a particular set of persuasions could connect, and this had outsize benefits for the community as a whole. That place is becoming untenable, and with the loss of goodwill and stability, the community’s started to dissipate. But there isn’t one obvious transition candidate, so the diaspora is finding itself sp…
Twitter is unstable? I can't remember the last time it had an outage.
Re: Incident with Github.com [resolved]
#180Earlier quoted context omitted.
That's an overly optimistic view of things. For a lot of us GitHub is critical infrastructure, which if it goes down loses us and our customers money.
Boomer opinion but trusting third parties to be critical infrastructure, especially with no SLA in sight, will always end in tears. "The cloud" is very convenient, but its providers will never care about your infrastructure or your customers as much as you will.
Truly a boomer opinion. Respect!