Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident with Github.com [resolved]

githubstatus.com

Post reply on HN
Print view View original

741–750 of 1001 posts

Re: Incident with Github.com [resolved]

#741

Post by spockz »

sznio wrote: post #204
benrutter wrote: post #66

Earlier quoted context omitted.

I'm really curious how other codeforges are faring. I only use codeberg and github - codeberg seems to have less outages, especially for a much smaller organisation, but they've recently removed support for LLM projects which could be linked. Anyone here have data on whether the likes of gitlab have been impacted in the same was as github?

my personal forgejo instance has had 100% availability for the last two years.

So it has run on a server that hasn’t had updates in 2 years? And it itself hasn’t been updated in 2 years?

Re: Incident with Github.com [resolved]

#742

Post by stephenway »

prdonahue wrote: post #472

What are some good SaaS alternatives to GitHub for CI (that aren't GitLab)? Is anyone doing anything particularly novel in this space?

I build Fjord, managed Forgejo with runners. The idea is to keep the GitHub-like workflow without having to operate the forge and CI infrastructure yourself.

Re: Incident with Github.com [resolved]

#743

Post by stephenway »

jubilanti wrote: post #637

Almost 3 hours in and still "We are still working to identify the root cause" Can't even view diffs in the web interface! I'm out! I am willing to pay ~$5-10/mo for a reliable host I can swap as painlessly as possible. Mostly just building little static sites and little PWAs, no more than 500mb a repo and 10gb total. Need a repo with PRs and issues, where people can register free account and contribute or raise issue…

Forgejo is a great fit for what you're describing. I run Fjord, which is managed Forgejo. Our dedicated instances are above your $5-10 budget, but we also run a free public Forgejo instance if you want to try it without operating anything yourself.

Re: Incident with Github.com [resolved]

#744

Post by datsci_est_2015 »

pkilgore wrote: post #735
datsci_est_2015 wrote: post #258

Earlier quoted context omitted.

It is interesting, how much money is being lost during this outage? My significant other was just let go from their job as a scapegoat for an organizational error: 3 layers of failure - IC, manager, director, and the IC was let go. The error caused a 7 figure loss for the company that has 10 figures of revenue per year. The manager and director may not see any consequences, though the director will probably be forced…

Name names.

Local (to us) firm. Two main reasons I didn't:

  - Have never seen it mentioned on this forum, in any context.
  - We're potentially pursuing legal action.

Re: Incident with Github.com [resolved]

#745

Post by winrid »

perks_12 wrote: post #163

The LLM code influx is not the problem. Microsoft mismanagement is: https://damrnelson.github.io/github-historical-uptime/

Pretty sure this is caused by trying to move to Azure (and I'd bet like $1000 partially due to trying to move DBs to network backed disk instances)

Re: Incident with Github.com [resolved]

#746

Post by jscd »

beardedetim wrote: post #15

We're starting to have the conversation of "do we need to replace GitHub as our CI/CD pipeline?" No idea if/when we decide to pull that bandaid but I think once we can't merge a hot fix into main because of GH being down, it'll be a big org push to replace it. What's everyone else using? Self hosted Gitlab? Gitea?

It's not the most elegant thing ever, but I've been using (self-hosted) WoodpeckerCI and it's been dead simple. It doesn't have many bells and whistles, but honestly I think that's good. Forced me to put complex behavior in better systems as opposed to doing bash scripts in YAML.

Re: Incident with Github.com [resolved]

#747

Post by rsanheim »

chilipepperhott wrote: post #532

What I want: A self hosted version of GitHub that is a precise mirror of an actual GitHub repository. That is, PRs from GitHub appear on the selfhosted instance and my reviews from the self hosted instance appear on GitHub. That way I can work on my private instance when GitHub is down (or slow) and I can be assured my work still appears to everyone online. Does this exist?

AFAIK there is nothing like this out there, because there is not one standard for a "pull reauest" in the sense of how a web based forge does it. and it gets complicated quickly due to complexity of git and forks.

the closest thing i've found are tooks like https://github.com/raylabshq/gitea-mirror

Re: Incident with Github.com [resolved]

#748

Post by winrid »

philippta wrote: post #711

The most shocking thing to me about the entire GitHub fiasco is: No matter how bad the uptime is, no matter how bad the UI is. We will keep using the software / service we depend on. Why do we as software engineers have set fault tolerance and high availability as the primary goal of any software / infrastructure architecture? This shows it's completely irrelevant.

Personally, Fastcomments and all my projects can easily be switched to a self hosted provider, keeping GitHub as a read only mirror. It's probably a couple days work with today's tools.

Re: Incident with Github.com [resolved]

#749

Post by esseph »

hombre_fatal wrote: post #714
unknownfuture wrote: post #685

Earlier quoted context omitted.

Weird, I've been told you can just throw AI at all your logs and metrics and it can magically solve all the things...

Well, it would be a mistake on your part to assume that everything in an organization, especially a large one like github, is a reflection of the capabilities of the state of the art in anything (AI, architecture, programming language, organizational management, operations, etc.). It's clear that updating the status page as quickly as possible isn't even something github cares about.

> It's clear that updating the status page as quickly as possible isn't even something github cares about.

Status pages for large companies are social, political and legal tools, and not technical indicators. It has always been this way, and likely always will unless some laws were passed.

Re: Incident with Github.com [resolved]

#750

Post by ajasmin »

edgyquant wrote: post #669
Scubabear68 wrote: post #661

Earlier quoted context omitted.

The thing is Github’s core is git’s core plus extras, and thet core is fundamentally text wrangling. Youtube and Insta and Facebook and many other social media platforms handle very resource intensive media for billions of people plus bots. Increased influx from AI is not plausible when viewed from that lens.

It is though. Maybe you think they should be doing a better job but 12-24xing scale when you were already handling billions of commits is definitely understandable to me. It just does feel like they should have some way of ensuring customers, paying ones, don’t have to suffer so much.

Lots of paying customers also depend on the open source part of GitHub to run their business.
Post reply on HN