Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Is Down

githubstatus.com

Post reply on HN
Print view View original

51–60 of 61 posts

Re: GitHub Is Down

#51

Post by SOLAR_FIELDS »

kelseydh wrote: post #35
JsonDemWitOster wrote: post #26

Earlier quoted context omitted.

I care but I can't move out because it's Orders from Up High which migrated us to Github. We haven't been here a year and it's not worth my neck to mirror our code in some kind of skunkworks forge instance. Woe. Woe is me.

We have so many automated workflows and pipelines moving through Github Actions + other Github integrations it would be a giant headache to migrate. Not clear where we would go either. Gitlab??

I’ve been running an SMB trending mid market out of gitlab for the greater part of a year with no issues. One of my favorite benefits is that ci runners are colocated with the self hosted instance on k8s so suddenly a whole huge slew of shit that you had to care about with GitHub, security, provenance and supply chain is just… not an issue.

Getting off the GitHub actions dependency is a feature, not a bug

Re: GitHub Is Down

#52

Post by bigstrat2003 »

m-schuetz wrote: post #15
rvz wrote: post #5

Once again, GitHub is down. Last time an incident happened was 2 days ago [0]. Could not have happened on a worse day (Monday) and you can see how unreliable GitHub has been. Better of self-hosting. [0] https://news.ycombinator.com/item?id=48418183

Anything I'd self-host would be down way more often than github.

I self host my own VCS repos. It's never down. It's not rocket surgery to self host, even for business stuff rather than personal.

Re: GitHub Is Down

#53

Post by PeterStuer »

kelseydh wrote: post #35
JsonDemWitOster wrote: post #26

Earlier quoted context omitted.

I care but I can't move out because it's Orders from Up High which migrated us to Github. We haven't been here a year and it's not worth my neck to mirror our code in some kind of skunkworks forge instance. Woe. Woe is me.

We have so many automated workflows and pipelines moving through Github Actions + other Github integrations it would be a giant headache to migrate. Not clear where we would go either. Gitlab??

Of course. Every non trivial migration is a 'headache'. But so is being down too many times.

At the very least explore and prepare for alternatives. Map out dependencies that are not trivial to replace. There's probably fewer of those than you think.

Re: GitHub Is Down

#54

Post by PeterStuer »

ownagefool wrote: post #37
JsonDemWitOster wrote: post #24

The Google SRE book offers the following as one of the reasons to not gun for 100% reliability (emphasis added): > users typically don’t notice the difference between high reliability and extreme reliability in a service, because the user experience is dominated by less reliable components like the cellular network or the device they are working with . Put simply, a user on a 99% reliable smartphone cannot tell the d…

Github isn't having a debate over how many 9s they have, they're having a zero 9s problem. I think there's 3 big themes with this, thought not 1. LLM tools have added considerable load. 2. LLM used by developers to increase velocity seem to be leading more outages. This calls into question the increased velocity. 3. Roadmaps focused on pushing features that aren't reliability problems. i.e. github moving to azure, or…

So they failed to manage growth. That is a business management problem first, and only a technical problem second. Yet Github management seems to constantly deflect to operations.

If you take on load (this is 100% by choice) beyond capacity, then obviously the system collapses.

Re: GitHub Is Down

#55

Post by PeterStuer »

kelseydh wrote: post #33
JsonDemWitOster wrote: post #24

The Google SRE book offers the following as one of the reasons to not gun for 100% reliability (emphasis added): > users typically don’t notice the difference between high reliability and extreme reliability in a service, because the user experience is dominated by less reliable components like the cellular network or the device they are working with . Put simply, a user on a 99% reliable smartphone cannot tell the d…

Apparently Github is experiencing a huge increase in usage due to LLMs and this is the cause for a lot of their instability as of late.

'Experiencing' makes it sound like they have no deliberate choice. No, they let this happen, by choice. They could have prevented this, contractually, by pricing, by governance, but chose not to.

Re: GitHub Is Down

#56

Post by PeterStuer »

m-schuetz wrote: post #15
rvz wrote: post #5

Once again, GitHub is down. Last time an incident happened was 2 days ago [0]. Could not have happened on a worse day (Monday) and you can see how unreliable GitHub has been. Better of self-hosting. [0] https://news.ycombinator.com/item?id=48418183

Anything I'd self-host would be down way more often than github.

My on prem forgejo is way less down than github. I use both. You don't have to chose.

Re: GitHub Is Down

#57

Post by aweiher »

tosti wrote: post #46
craigmart wrote: post #25

Earlier quoted context omitted.

I had checked that page not long ago, and as far as I remember there were many "red" or "orange" days in the past 3 months. Now it's all green. That's concerning

If they've actually improved the service, it would be fine. Is there reasonable plausability, reasonable doubt, or reasonable suspicion?

it should not change any historical records

Re: GitHub Is Down

#58

Post by rurban »

ownagefool wrote: post #37
JsonDemWitOster wrote: post #24

The Google SRE book offers the following as one of the reasons to not gun for 100% reliability (emphasis added): > users typically don’t notice the difference between high reliability and extreme reliability in a service, because the user experience is dominated by less reliable components like the cellular network or the device they are working with . Put simply, a user on a 99% reliable smartphone cannot tell the d…

Github isn't having a debate over how many 9s they have, they're having a zero 9s problem. I think there's 3 big themes with this, thought not 1. LLM tools have added considerable load. 2. LLM used by developers to increase velocity seem to be leading more outages. This calls into question the increased velocity. 3. Roadmaps focused on pushing features that aren't reliability problems. i.e. github moving to azure, or…

Nope. It's entirely azure management fault. https://isolveproblems.substack.com/p/how-microsoft-vaporize...

Re: GitHub Is Down

#59

Post by ownagefool »

rurban wrote: post #58
ownagefool wrote: post #37

Earlier quoted context omitted.

Github isn't having a debate over how many 9s they have, they're having a zero 9s problem. I think there's 3 big themes with this, thought not 1. LLM tools have added considerable load. 2. LLM used by developers to increase velocity seem to be leading more outages. This calls into question the increased velocity. 3. Roadmaps focused on pushing features that aren't reliability problems. i.e. github moving to azure, or…

Nope. It's entirely azure management fault. https://isolveproblems.substack.com/p/how-microsoft-vaporize...

Whilst that's crazy, github hasn't migrated to azure yet, so it's probably not exclusively an azure problem and we've seen the same problems with amazon too ( and several other large orgs ).

Re: GitHub Is Down

#60

Post by rurban »

ownagefool wrote: post #59
rurban wrote: post #58

Earlier quoted context omitted.

Nope. It's entirely azure management fault. https://isolveproblems.substack.com/p/how-microsoft-vaporize...

Whilst that's crazy, github hasn't migrated to azure yet, so it's probably not exclusively an azure problem and we've seen the same problems with amazon too ( and several other large orgs ).

Github thanksfully not, but GH actions are running entirely on Azure. They've setup actions as a testbed for Azure afterall.
Post reply on HN