Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident with Github.com [resolved]

githubstatus.com

Post reply on HN
Print view View original

751–760 of 1001 posts

Re: Incident with Github.com [resolved]

#751

Post by bryanmikaelian »

stephenway wrote: post #743
jubilanti wrote: post #637

Almost 3 hours in and still "We are still working to identify the root cause" Can't even view diffs in the web interface! I'm out! I am willing to pay ~$5-10/mo for a reliable host I can swap as painlessly as possible. Mostly just building little static sites and little PWAs, no more than 500mb a repo and 10gb total. Need a repo with PRs and issues, where people can register free account and contribute or raise issue…

Forgejo is a great fit for what you're describing. I run Fjord, which is managed Forgejo. Our dedicated instances are above your $5-10 budget, but we also run a free public Forgejo instance if you want to try it without operating anything yourself.

Disclaimer: I work at exe.dev

Forgejo has been great for me too. I liked it so much, I added a feature to make it relatively easy to get up and running on exe.dev. Support for Gitea and Tangle.org is also available.

Re: Incident with Github.com [resolved]

#753

Post by ALLTaken »

adamddev1 wrote: post #697
jubilanti wrote: post #637

Almost 3 hours in and still "We are still working to identify the root cause" Can't even view diffs in the web interface! I'm out! I am willing to pay ~$5-10/mo for a reliable host I can swap as painlessly as possible. Mostly just building little static sites and little PWAs, no more than 500mb a repo and 10gb total. Need a repo with PRs and issues, where people can register free account and contribute or raise issue…

I was in a very similar situation and this is what I did: - put Forgejo on a little $5 Vultr VPS (I share the VPS with other things, Forgejo doesn't need much resources. I also added some block storage to host my Docker containers in the Forgejo Docker registry) - got a 8GB ram cheaper Hetzner VPS (you have to wait for their availability) for around 8 EUR and used that to install the CI runner - Use Netlify for hosti…

Great setup, two questions, do you have 2 VPNs now or all on one Hetzner VPS? And how do you setup security?

Just recommend swapping vercel/netlify with something else, we had quite a few issues and high costs with them. Not opiniated here, currently using Cloudflare and BunnyCDN (which is great)

Re: Incident with Github.com [resolved]

#754

Post by vaylian »

askonomm wrote: post #139
ieie3366 wrote: post #100

To everyone who is angry: calm down. Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code Unless you have a emergency hotfix (you don’t), go hit the gym or walk outside. If a work tool going down triggers you enough mentally to start angrily ranting online, it’s a sign you need to chill out and focus more on your health

That's an overly optimistic view of things. For a lot of us GitHub is critical infrastructure, which if it goes down loses us and our customers money.

> For a lot of us GitHub is critical infrastructure

Please read https://berthub.eu/articles/posts/cyber-security-pre-war-rea...

Re: Incident with Github.com [resolved]

#755

Post by blitzar »

leishman wrote: post #171

I don't understand why Github hasn't solved this problem with pricing updates. My understanding is they are getting hammered with LLM generated code growing their traffic by over an order of magnitude. So why not rate limit non-paying users and charge for whatever scarce resource is being consumed that is causing them to constantly fall-over? This seems like a basic economics problem.

I don't understand why Github hasn't solved this problem with vibe coded updates.

Re: Incident with Github.com [resolved]

#756

Post by whateveracct »

evilfred wrote: post #415
perrygeo wrote: post #62

Earlier quoted context omitted.

Github's, COO Kyle Daigle, claimed that the number of commits has increased 14x since last year. So shear scale has something to do with it. I don't know of too much software already operating a planetary scale that could easily absorb a 14x bump in a year.

14x is barely an order of magnitude

seriously. if you are ready to scale, you are ready for that much. if that kills you, you had architecture issues that a company as big and established as GH shouldn't have.

Re: Incident with Github.com [resolved]

#757

Post by ALLTaken »

rurban wrote: post #700
unknownfuture wrote: post #685

Earlier quoted context omitted.

Weird, I've been told you can just throw AI at all your logs and metrics and it can magically solve all the things...

My blackest Monday today. Deepseek is off/too expensive, Claude tokens out for 3 days, kimi refuses, and Github down for the whole afternoon. Maybe I'll take the week off

hmm very interesting combination of services being down. I've been using DeepSeek v4 Pro and Qwen3.8 now daily for close to 2 weeks. Damn good and reliable. Anytime I need to verify math I check with Kimi k3 tho, it still feels like being superior to these both, not sure if actually true.

Re: Incident with Github.com [resolved]

#758

Post by ajmurmann »

rurban wrote: post #700
unknownfuture wrote: post #685

Earlier quoted context omitted.

Weird, I've been told you can just throw AI at all your logs and metrics and it can magically solve all the things...

My blackest Monday today. Deepseek is off/too expensive, Claude tokens out for 3 days, kimi refuses, and Github down for the whole afternoon. Maybe I'll take the week off

It's crazy that it wasn't that long ago that all we worked with was a compiler on the local machine and maybe a SVN server you'd interact with every few hours

Re: Incident with Github.com [resolved]

#759

Post by cyberax »

spockz wrote: post #741
sznio wrote: post #204

Earlier quoted context omitted.

my personal forgejo instance has had 100% availability for the last two years.

So it has run on a server that hasn’t had updates in 2 years? And it itself hasn’t been updated in 2 years?

You can run it in an HA configuration with multiple servers and live updates.

Re: Incident with Github.com [resolved]

#760

Post by ALLTaken »

stingraycharles wrote: post #52
khvn26 wrote: post #29

I had a lot of goodwill for GitHub but I think today is the tipping point. Looking at a unicorn page, I feel this lingering hope that it's transient (like it usually was in the old days) but my mind reassures me it's probably going to be a long full outage again. The hope is dead.

I’m willing to give them a break as I’m assuming they have a lot of scaling problems due to the influx of LLM assisted coding. But maybe I’m wrong?

No, that's almost exactly right, except that scaling problem is inverse!

MSFT is giving away most of their Azure Datacenters resources to AI, to make money and doesn't care if Github suffers along the way and drains Github's available resources to bare minimum. Leading to outages like this, but you're 100% spot on right too. All those Claude/OpenClaw instances produce so much Code, people setup more and more repositories.

Post reply on HN