Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident with multple GitHub services

githubstatus.com

Post reply on HN
Print view View original

101–110 of 136 posts

Re: Incident with multple GitHub services

#101

Post by robertwt7 »

agartner wrote: post #82

There was another really bad incident today: https://www.githubstatus.com/incidents/zsg1lk7w13cf > We have resolved a regression present when using merge queue with either squash merges or rebases. If you use merge queue in this configuration, some pull requests may have been merged incorrectly between 2026-04-23 16:05-20:43 UTC. We had ~8 commits get entirely reverted on our default branch during this time. I've nev…

yeah this is crazy we had many PRs reverted as well on many repos. downtime is 1 thing, but reverting PRs is failure on another level

Re: Incident with multple GitHub services

#103

Post by TranquilMarmot »

yakattak wrote: post #28
embedding-shape wrote: post #17

I've been on a somewhat binge to move a bunch of stuff to self-hosting at home. Yesterday I finally completed my self-hosted Forgejo instance at home, together with Linux, Windows (via VM) and macOS (via Mac Mini) runners/workers for CI/CD, so everything finally lives in-house (literally), instead of all source code + Actions being on GitHub but the infrastructure actually living locally. This is probably the first t…

I moved my forge to my home, outside of a little stress getting all the containers wrangled it was pretty effortless to setup Forgejo. I do need a good backup solution though, that’s one thing I’m missing.

I use https://github.com/garethgeorge/backrest to manage nightly encrypted backups of my Forgejo instance to a Hetzner Storage Box. Immich automatically dumps its DB every day, for Forgejo I have a little script that runs as part of the Backrest backup that does a pgdumb of the database before doing the backup.

It works great, I even had to do disaster recovery on it once and it went smooth.

Re: Incident with multple GitHub services

#104

Post by TranquilMarmot »

nextaccountic wrote: post #64
radlad wrote: post #42

Earlier quoted context omitted.

There is a fairly straightforward feature in Forgejo to sync your repos to Github, if that's what you want to do. It's not perfect, of course, but should help to advertise your projects and keep your activity heatmap green. I mostly use Forgejo for my private repos, which are free at Github, but with many limitations. One month I burned all my private CI tokens on the 1st due to a hung Mac runner. Love not having to…

or you can just have two remotes and push to both sites and enjoy git's distributed nature

I do this, but beware if you have LFS files. You can easily get into weird states with LFS pushing up to two different remotes and it's really not fun to fix.

Re: Incident with multple GitHub services

#105

Post by lone-cloud »

yakattak wrote: post #28
embedding-shape wrote: post #17

I've been on a somewhat binge to move a bunch of stuff to self-hosting at home. Yesterday I finally completed my self-hosted Forgejo instance at home, together with Linux, Windows (via VM) and macOS (via Mac Mini) runners/workers for CI/CD, so everything finally lives in-house (literally), instead of all source code + Actions being on GitHub but the infrastructure actually living locally. This is probably the first t…

I moved my forge to my home, outside of a little stress getting all the containers wrangled it was pretty effortless to setup Forgejo. I do need a good backup solution though, that’s one thing I’m missing.

I use rclone + backblaze. You get 10GB for free which is more than enough for self-hosted stuff.

Re: Incident with multple GitHub services

#106

Post by lone-cloud »

embedding-shape wrote: post #17

I've been on a somewhat binge to move a bunch of stuff to self-hosting at home. Yesterday I finally completed my self-hosted Forgejo instance at home, together with Linux, Windows (via VM) and macOS (via Mac Mini) runners/workers for CI/CD, so everything finally lives in-house (literally), instead of all source code + Actions being on GitHub but the infrastructure actually living locally. This is probably the first t…

Are you me? Somebody was talking about gitea on here yesterday and I also ended up self-hosting and moving all of my private projects to Ferjero yesterday after a bit of research. I can't bring myself to move public projects due to job prospects + GitHub network effect. Otherwise I'm role playing as a system admin now with 20 local services for various things I need. I think the most important thing is to have regular backups as you're now in charge of keeping your data from getting lost.

Re: Incident with multple GitHub services

#107

Post by ozarkerD »

braiamp wrote: post #92
hedayet wrote: post #87

Is Github losing any significant business from all these outages? Curious because for a long time we as an industry maintained that reliability and brand value are business critical; but seems like they are cared very little now a days. Happy to be corrected about my perception too.

They are entrenched enough that it's wrote off as cost of doing business. Big business have their internal instances so they are "insulated", everyone else isn't as critical and have the resources to do an internal solution or move.

Org instances have been affected by these outages too. Including the one today

Re: Incident with multple GitHub services

#108

Post by datadrivenangel »

braiamp wrote: post #92
hedayet wrote: post #87

Is Github losing any significant business from all these outages? Curious because for a long time we as an industry maintained that reliability and brand value are business critical; but seems like they are cared very little now a days. Happy to be corrected about my perception too.

They are entrenched enough that it's wrote off as cost of doing business. Big business have their internal instances so they are "insulated", everyone else isn't as critical and have the resources to do an internal solution or move.

They have on-prem / dedicated instances? I thought that microsoft only offered that through their Azure DevOps git offering.

Re: Incident with multple GitHub services

#109

Post by cjonas »

femiagbabiaka wrote: post #30
cjonas wrote: post #24

Earlier quoted context omitted.

also I never had considered that breaking your up-time into a bunch of different components is just a strategy to make your SRE look better than it actually is. The combined up-time tells the real story (88%!). Thanks for the link

The number of nines assigned to a suite of services is not indicative of the quality of SRE at any given company, but rather a reflection of the tradeoffs a business has decided to make. Guaranteed there's a dashboard somewhere at Github looking at platform stickiness vs. reliability and deciding how hard to let teams push on various initiatives.

this is fair. I should have just said "Site Reliability", as it's almost certainly out of the engineers control.

Re: Incident with multple GitHub services

#110

Post by jamesfinlayson »

datadrivenangel wrote: post #108
braiamp wrote: post #92

Earlier quoted context omitted.

They are entrenched enough that it's wrote off as cost of doing business. Big business have their internal instances so they are "insulated", everyone else isn't as critical and have the resources to do an internal solution or move.

They have on-prem / dedicated instances? I thought that microsoft only offered that through their Azure DevOps git offering.

GitHub Enterprise has existed for a while: https://docs.github.com/en/enterprise-server@3.20/admin/over...

I'm pretty sure it still does - I used it at a previous job and at somewhere that I interviewed recently they said they used GitHub (given their size and being a somewhat regulated industry I can't imagine they rely on github.com).

Post reply on HN