Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident with Github.com [resolved]

githubstatus.com

Post reply on HN
Print view View original

991–1000 of 1001 posts

Re: Incident with Github.com [resolved]

#991

Post by peterzat »

askonomm wrote: post #139
ieie3366 wrote: post #100

To everyone who is angry: calm down. Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code Unless you have a emergency hotfix (you don’t), go hit the gym or walk outside. If a work tool going down triggers you enough mentally to start angrily ranting online, it’s a sign you need to chill out and focus more on your health

That's an overly optimistic view of things. For a lot of us GitHub is critical infrastructure, which if it goes down loses us and our customers money.

A lot of casual tech folks I know don't understand GitHub's role in CI/CD, and think it's "just" revision control storage. It's natural, since many people have no reason to know about systems like ghcr.io and npm registry if they're storing vibe-coded personal projects.

Re: Incident with Github.com [resolved]

#992

Post by baconslices »

beardedetim wrote: post #15

We're starting to have the conversation of "do we need to replace GitHub as our CI/CD pipeline?" No idea if/when we decide to pull that bandaid but I think once we can't merge a hot fix into main because of GH being down, it'll be a big org push to replace it. What's everyone else using? Self hosted Gitlab? Gitea?

>What's everyone else using? Self hosted Gitlab? Gitea?

Self hosted github enterprise and runners.

Re: Incident with Github.com [resolved]

#993

Post by itemize123 »

TwelveEyes wrote: post #777
datsci_est_2015 wrote: post #258

Earlier quoted context omitted.

It is interesting, how much money is being lost during this outage? My significant other was just let go from their job as a scapegoat for an organizational error: 3 layers of failure - IC, manager, director, and the IC was let go. The error caused a 7 figure loss for the company that has 10 figures of revenue per year. The manager and director may not see any consequences, though the director will probably be forced…

Is responding calmly a privilege to have, or a kindness to yourself?

how is it mutually exclusive

Re: Incident with Github.com [resolved]

#994

Post by AlexeyBelov »

aurareturn wrote: post #649
duped wrote: post #624

Earlier quoted context omitted.

How much more value did those commits create than in 2025? But that discards my point, which is that you should probably have been prevented from pushing all those to github unless you were paying them for it, but you weren't because the executives running github can't admit that AI is bad for their platform.

How much more value did those commits create than in 2025? We have 2x fewer developers now than in 2025. Yet, the amount of features and bug fixes we're pushing is something like 5x. So probably 10x more value is my guess for my entire team. If you mean translating that into revenue, I'm not sure. Our product is better than ever with features a much bigger team couldn't dream of building before. Stability has also be…

So.. you fired your devs and replaced them with AI?

Re: Incident with Github.com [resolved]

#995

Post by aurareturn »

AlexeyBelov wrote: post #994
aurareturn wrote: post #649

Earlier quoted context omitted.

How much more value did those commits create than in 2025? We have 2x fewer developers now than in 2025. Yet, the amount of features and bug fixes we're pushing is something like 5x. So probably 10x more value is my guess for my entire team. If you mean translating that into revenue, I'm not sure. Our product is better than ever with features a much bigger team couldn't dream of building before. Stability has also be…

So.. you fired your devs and replaced them with AI?

Some of them left. Some of them moved to different teams. We just never replaced them on the team.

So more accurately, we placed developers who left the team with humans using more tokens.

Re: Incident with Github.com [resolved]

#996

Post by aurareturn »

slowin wrote: post #730
dannyw wrote: post #671

Earlier quoted context omitted.

Services like Facebook aggressively block automated activity, including AI agents. GitHub sells this as a service.

Bots or not, Facebook has orders of magnitude more traffic.

HTTP traffic is different from run this CPU intensive pipeline service for 10 minutes and now increase the volume by 100x in a few months.

I used to manage a popular app/website that had tens of billions of real user page views per month. It wasn't easy but it also wasn't hard. It was surprisingly easy with a single large Postgres DB, a lot of caching, optimized tables and queries.

Re: Incident with Github.com [resolved]

#997

Post by jen20 »

ProAm wrote: post #945
throw-the-towel wrote: post #869

Earlier quoted context omitted.

In my experience, we were typically affected just like everyone else. Maybe a public company with its market cap in the billions isn't big enough.

Self host an infrastructure like real billion dollar companies.

Having worked at places with GitHub Enterprise, that isn’t much of an improvement. You could host something else, but almost everything in this space isn’t great at scale.

Re: Incident with Github.com [resolved]

#998

Post by yurishimo »

frollogaston wrote: post #843
s_dev wrote: post #145

I recall reading years ago that cloud services were expected to run with a reliability of 3 or 4 '9's and that if they didn't competing services would quickly overtake them in adoption. The industry was supposed to be that cut throat. Has big tech reached a similar status like banks in that they are "too big to fail" i.e. when they do fail we all just look the other way and say: "well everyone else is out too". Didn'…

That number of 9s might be a thing for the lower layer cloud services like on AWS or Azure or GCP. Even then, that us-east-1 outage was an accidental power move. It established that many are best off putting all their stuff on us-east-1 cause then their outages only happen at the best possible times, whereas multi-region complexity might cause its own kind of outage when nobody else is down.

Except you cannot run a business in Asia with datacenters in the US. The latency is unacceptable.

Re: Incident with Github.com [resolved]

#999

Post by _kblcuk_ »

elric wrote: post #30
beardedetim wrote: post #15

We're starting to have the conversation of "do we need to replace GitHub as our CI/CD pipeline?" No idea if/when we decide to pull that bandaid but I think once we can't merge a hot fix into main because of GH being down, it'll be a big org push to replace it. What's everyone else using? Self hosted Gitlab? Gitea?

> We're starting to have the conversation of "do we need to replace GitHub as our CI/CD pipeline?" Did you have the conversation "do we need to move from $whatever CI/CD to GitHub Actions?" beforehand? What were the arguments in favour?

Jenkins was (and still is) a PITA to maintain. Moved two different companies off from it to gh actions (self-managed runners in case of a second company), never had any regrets.

Re: Incident with Github.com [resolved]

#1000

Post by ALLTaken »

adamddev1 wrote: post #771
ALLTaken wrote: post #753

Earlier quoted context omitted.

Great setup, two questions, do you have 2 VPNs now or all on one Hetzner VPS? And how do you setup security? Just recommend swapping vercel/netlify with something else, we had quite a few issues and high costs with them. Not opiniated here, currently using Cloudflare and BunnyCDN (which is great)

I just SSH into the servers when I need to. I don't have any VPNs with these servers. I just try to follow normal server security setup practices, disabling password authentication etc.

I worried being behind in server-security, since all those 0days now being so easy to pull from claude/chatgpt, that maybe you guys found a risk-minimizing front-end to reduce the attack-surface.
Post reply on HN