Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident with Github.com [resolved]

githubstatus.com

Post reply on HN
Print view View original

251–260 of 1001 posts

Re: Incident with Github.com [resolved]

#251

Post by stetrain »

ieie3366 wrote: post #100

To everyone who is angry: calm down. Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code Unless you have a emergency hotfix (you don’t), go hit the gym or walk outside. If a work tool going down triggers you enough mentally to start angrily ranting online, it’s a sign you need to chill out and focus more on your health

> as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code

Github and their parent company are active participants in pushing LLM-driven coding.

Re: Incident with Github.com [resolved]

#252

Post by DharmaPolice »

ieie3366 wrote: post #100

To everyone who is angry: calm down. Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code Unless you have a emergency hotfix (you don’t), go hit the gym or walk outside. If a work tool going down triggers you enough mentally to start angrily ranting online, it’s a sign you need to chill out and focus more on your health

This might be naïve but wouldn't the appropriate response be to reduce access/rate limit free/new accounts in order for service to be maintained for everyone else?

Re: Incident with Github.com [resolved]

#253

Post by john_strinlai »

TZubiri wrote: post #221
ieie3366 wrote: post #100

To everyone who is angry: calm down. Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code Unless you have a emergency hotfix (you don’t), go hit the gym or walk outside. If a work tool going down triggers you enough mentally to start angrily ranting online, it’s a sign you need to chill out and focus more on your health

is the infrastructure of the free tier shared with that of paid users? That might be the issue. Otherwise if usage of paid users scales, then it would be fine.

>is the infrastructure of the free tier shared with that of paid users?

github enterprise is fully operational at the moment

Re: Incident with Github.com [resolved]

#255

Post by OtomotO »

duxup wrote: post #220
Arubis wrote: post #99

This is starting to feel like Twitter. For a variety of reasons, we had a centralized place where everyone of a particular set of persuasions could connect, and this had outsize benefits for the community as a whole. That place is becoming untenable, and with the loss of goodwill and stability, the community’s started to dissipate. But there isn’t one obvious transition candidate, so the diaspora is finding itself sp…

And like Twitter most people will keep using it anyway?

Just in the US. I have ONE European client... ONE of dozens who use GitHub.

EVERY other client I've ever worked with used their own gitforge or at least bitbucket...

Re: Incident with Github.com [resolved]

#256

Post by aenis »

AlexandrB wrote: post #167
askonomm wrote: post #139

Earlier quoted context omitted.

That's an overly optimistic view of things. For a lot of us GitHub is critical infrastructure, which if it goes down loses us and our customers money.

Boomer opinion but trusting third parties to be critical infrastructure, especially with no SLA in sight, will always end in tears. "The cloud" is very convenient, but its providers will never care about your infrastructure or your customers as much as you will.

Boomer here as well, but I'd add that trusting your own org for critical infra usually also ends in tears. Most everything in IT involves failure, including in well designed systems designed by great engineers.

I worked for a few years in an exceedingly well capitalised place which ran everything in their own data centers, money no object, with a truck parked somewhere, ready to go, with a smaller version of our critical infra. We had a serious business-stopping outage once every 18 months or so, every time for fringe reasons one only learns about when trying to run a large data center. Its convenient to blame the cloud and pretend that self-hosting in private sector was so, so great with six nines.

Re: Incident with Github.com [resolved]

#257

Post by Waterluvian »

leishman wrote: post #171

I don't understand why Github hasn't solved this problem with pricing updates. My understanding is they are getting hammered with LLM generated code growing their traffic by over an order of magnitude. So why not rate limit non-paying users and charge for whatever scarce resource is being consumed that is causing them to constantly fall-over? This seems like a basic economics problem.

The people with the spreadsheets have determined the value of this loss leader to be too enormous to shut it off.

Re: Incident with Github.com [resolved]

#258

Post by datsci_est_2015 »

askonomm wrote: post #139
ieie3366 wrote: post #100

To everyone who is angry: calm down. Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code Unless you have a emergency hotfix (you don’t), go hit the gym or walk outside. If a work tool going down triggers you enough mentally to start angrily ranting online, it’s a sign you need to chill out and focus more on your health

That's an overly optimistic view of things. For a lot of us GitHub is critical infrastructure, which if it goes down loses us and our customers money.

It is interesting, how much money is being lost during this outage?

My significant other was just let go from their job as a scapegoat for an organizational error: 3 layers of failure - IC, manager, director, and the IC was let go. The error caused a 7 figure loss for the company that has 10 figures of revenue per year. The manager and director may not see any consequences, though the director will probably be forced out by end-of-year due to incompetence. The new executive has taken to firing employees much more eagerly than their predecessor, like some sort of Jack Welch acolyte.

Their firing has put a lot of things into perspective for me. Mostly, fuck "at-will" employment and its negative effect on the American social contract.

But also this "angry ranting" online that the original poster was referencing. Not everyone has the privilege to calmly respond to things that directly impact their livelihood.

Re: Incident with Github.com [resolved]

#259

Post by twistedpair »

jjice wrote: post #164
ieie3366 wrote: post #100

To everyone who is angry: calm down. Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code Unless you have a emergency hotfix (you don’t), go hit the gym or walk outside. If a work tool going down triggers you enough mentally to start angrily ranting online, it’s a sign you need to chill out and focus more on your health

> Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code I sympathized with them when they said this a handful of months ago, but then I saw this [0] page that shows how it's been shot for years prior (which tracks with my memory). I feel for the GH engineers that have to deal with this, especially the SREs. I also don't hate the dow…

They _are_ the LLM coding agent vendor, and are _owned_ by MS, the ~majority~ biggest shareholder of OpenAI. How can you NOT consider that 20x+ scaling in your capacity roadmap projections, where you are trying to get everyone to use these agents as part of your core OKRs?

Or, maybe your #1 IT priority was moving everything to Azure instead ;)

Re: Incident with Github.com [resolved]

#260

Post by normie3000 »

anonymousab wrote: post #213
gottagocode wrote: post #49

Earlier quoted context omitted.

20%, do we just make numbers up?

It seems vastly higher on every region and VPN I've tested, so I think this is a case of the "technicalies". It is technically 20%, because a bunch of the requests that happen on page load do succeed. Just not the few crucial ones that are required for the page to load correctly - those return a 500 the vast supermajority of the time.

20% of requests on a GUI page also makes the whole thing unusable.
Post reply on HN