Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident with multple GitHub services

githubstatus.com

Post reply on HN
Print view View original

131–136 of 136 posts

Re: Incident with multple GitHub services

#131

Post by bloppe »

CodesInChaos wrote: post #126
eddyg wrote: post #123

Earlier quoted context omitted.

The scale of growth they’re dealing with is insane . “There were 1 billion commits in 2025. Now, it's 275 million per week, on pace for 14 billion this year if growth remains linear (spoiler: it won't.) GitHub Actions has grown from 500M minutes/week in 2023 to 1B minutes/week in 2025, and now 2.1B minutes so far this week.” Source: GitHub COO on April 3, 2026. https://x.com/kdaigle/status/2040164759836778878

What's causing that growth? AI bots spamming commits?

Yep

Re: Incident with multple GitHub services

#132

Post by CoastalCoder »

robertwt7 wrote: post #101
agartner wrote: post #82

There was another really bad incident today: https://www.githubstatus.com/incidents/zsg1lk7w13cf > We have resolved a regression present when using merge queue with either squash merges or rebases. If you use merge queue in this configuration, some pull requests may have been merged incorrectly between 2026-04-23 16:05-20:43 UTC. We had ~8 commits get entirely reverted on our default branch during this time. I've nev…

yeah this is crazy we had many PRs reverted as well on many repos. downtime is 1 thing, but reverting PRs is failure on another level

Really makes one appreciate that concept of A.C.I.D. database transactions.

Re: Incident with multple GitHub services

#133

Post by rvz »

If any company had so many incidents like GitHub, they would be out of business by now.

Another outage at GitHub thanks to the AI agents like (Tay.ai, GitHub Copilot) at the helm with no employees maintaining it as I said before [0]

"Centralizing" everything to GitHub in 2026 is starting to look quite unthinkable.

[0] https://news.ycombinator.com/item?id=47592294

Re: Incident with multple GitHub services

#134

Post by estimator7292 »

embedding-shape wrote: post #17

I've been on a somewhat binge to move a bunch of stuff to self-hosting at home. Yesterday I finally completed my self-hosted Forgejo instance at home, together with Linux, Windows (via VM) and macOS (via Mac Mini) runners/workers for CI/CD, so everything finally lives in-house (literally), instead of all source code + Actions being on GitHub but the infrastructure actually living locally. This is probably the first t…

My GitLab server at home is so nice. My last job ran a self-hosted GitLab on AWS with CI runners on-prem, which was a delight to work with.

New job runs on GitHub. I frequently have to stop work and wait for GitHub to recover before I can checkout some dependency or push a commit. It's outrageous.

Re: Incident with multple GitHub services

#135

Post by jeffrallen »

frakkingcylons wrote: post #100
jpleger wrote: post #96

Earlier quoted context omitted.

I wonder if there is any correlation between them moving towards Azure. https://thenewstack.io/github-will-prioritize-migrating-to-a...

They are dealing with vastly more activity as a result of AI usage. It's that simple.

They have not (re-)engineered their system, even via temporary hacks, in the last 12 months to defend itself with soft failures, circuit breakers, hard quotas, etc.

It's that simple.

Re: Incident with multple GitHub services

#136

Post by koyote »

oefrha wrote: post #121
koyote wrote: post #85

Earlier quoted context omitted.

And once you start self-hosting, you realise how slow the 'modern' web actually is. I host forgejo on a single NUC with a bunch of other stuff in Proxmox, the page loads in 6ms! Immich is not quite as fast but still a ton faster than Google photos.

That’s not a given. Self-hosted GitLab on my pretty good hardware is still slow. I just opened a very small repo with ~20 files and ~5 commits. The page spun for 5s+ before showing me the directory listing and readme. Subsequent loads are faster (~1s) but still not instant.

GitLab is what drove me to forgejo, it just became unsustainably slow.
Post reply on HN