Live data from Hacker News

GitHub Is Down

githubstatus.com

31–40 of 61 posts

Re: GitHub Is Down

#31
post #22

Earlier quoted context omitted.

Microsoft management/engineering practices+AI slop.

I get that, but all of that was in there for quite some time though, no?

Digital systems don't necessarily deteriorate immediately after the causal factors. Like technical debt, issues grow unnoticed and become visible gradually.

Re: GitHub Is Down

#33

The Google SRE book offers the following as one of the reasons to not gun for 100% reliability (emphasis added): > users typically don’t notice the difference between high reliability and extreme reliability in a service, because the user experience is dominated by less reliable components like the cellular network or the device they are working with . Put simply, a user on a 99% reliable smartphone cannot tell the d…

Apparently Github is experiencing a huge increase in usage due to LLMs and this is the cause for a lot of their instability as of late.

Re: GitHub Is Down

#35
post #19

This has gotten to a point where it doesn't really matter anymore. When a service crosses a certain reliability threshold it's like a phase change. The customer base eventually adapts to the situation. Anyone who still genuinely cares has moved to self hosted enterprise or something else by now. It was most tenuous for me when they almost met the SLA. Now that they've blown so far beyond it, the stress is mostly gone…

I care but I can't move out because it's Orders from Up High which migrated us to Github. We haven't been here a year and it's not worth my neck to mirror our code in some kind of skunkworks forge instance. Woe. Woe is me.

We have so many automated workflows and pipelines moving through Github Actions + other Github integrations it would be a giant headache to migrate. Not clear where we would go either. Gitlab??

Re: GitHub Is Down

#36

The Google SRE book offers the following as one of the reasons to not gun for 100% reliability (emphasis added): > users typically don’t notice the difference between high reliability and extreme reliability in a service, because the user experience is dominated by less reliable components like the cellular network or the device they are working with . Put simply, a user on a 99% reliable smartphone cannot tell the d…

> Put simply, a user on a 99% reliable smartphone cannot tell the difference between 99.99% and 99.999% service reliability!

Sure they can. If Google loads and Github doesn't, then it's clearly Github being down, not the mobile network.

Also not everyone uses a phone. My desktop & fibre internet has way better than 99% reliability.

Re: GitHub Is Down

#37

The Google SRE book offers the following as one of the reasons to not gun for 100% reliability (emphasis added): > users typically don’t notice the difference between high reliability and extreme reliability in a service, because the user experience is dominated by less reliable components like the cellular network or the device they are working with . Put simply, a user on a 99% reliable smartphone cannot tell the d…

Github isn't having a debate over how many 9s they have, they're having a zero 9s problem.

I think there's 3 big themes with this, thought not

1. LLM tools have added considerable load.

2. LLM used by developers to increase velocity seem to be leading more outages. This calls into question the increased velocity.

3. Roadmaps focused on pushing features that aren't reliability problems. i.e. github moving to azure, or adding AI features.

All these same problems happen to orgs with other fads that aren't AI. Following fads is not good engineering.

Re: GitHub Is Down

#38

The Google SRE book offers the following as one of the reasons to not gun for 100% reliability (emphasis added): > users typically don’t notice the difference between high reliability and extreme reliability in a service, because the user experience is dominated by less reliable components like the cellular network or the device they are working with . Put simply, a user on a 99% reliable smartphone cannot tell the d…

"unauthorized" is a bit different than "unauthenticated". The former suggests trying to access something you don't have permission for while the latter suggests you're just not logged in.

At a guess, I could imagine some sort of failure of cached pages, which can be cached for signed out users but probably not for signed in users (as the rendered HTML would need to have user context like their avatar etc)

Re: GitHub Is Down

#39
post #22

What's going on with GitHub lately? I've never seen them having so much issues over the years as they do have now.

Microsoft management/engineering practices+AI slop.

That, but also AI tools have made github req/res go up by 100x. There is simply too muvh AI generated traffic.

I k ow for a fact that ANY other platform would fail faster than github if they had the same volume of http requests.

Re: GitHub Is Down

#40
People seem to miss entirely that this is not (only) some slop code that makes github go down, but its the fact that they get 100x the number of requests since AI tools came to the devs daily workflow.
Post reply on HN