Live data from Hacker News

GitHub Git Operations Are Down

githubstatus.com

241–250 of 271 posts

Re: GitHub Git Operations Are Down

#242
post #160

Yet another reason to self host your VCS. Only thing hosted to “the cloud” these days are my backups - split between S3 and GCS. ~$25/mo

That’s expensive, I get it for free with GitHub. And a loooot more functionality.

Re: GitHub Git Operations Are Down

#243
post #133

Earlier quoted context omitted.

I was imagining something a bit more disastrous than that. A big enough solar flare could take parts of the planet offline for months. Years if they can't source enough replacement transformers. There are also political reasons that countries go offline. Then it'll be up to the nerds who manage to cobble together their own distributed version of everything--even if it's a significantly reduced definition of everythin…

ok fine, ill thank the nerds in that case.

I doubt it, I don't think the distributed stuff is anywhere near ready. Instead it'll be time to kiss the ring of whoever manages to grab control during the gap.

Re: GitHub Git Operations Are Down

#244
post #74
post #48

Not sure if I am just noticing GitHub's issues more often as I am using their tools pretty much every day but their availability is kinda not great. Be it Actions failing or something "not core business" (read: git operations) but I can't remember a month in the past where I was not annoyed by any outage on GitHub

Agreed, I wonder what their downtime percentage is. If guess it's down on the order of one hour per month, so 1/1000. Update: They promise >99.9% on a quarterly basis for enterprise customers - https://github.com/github/docs/blob/main/content%2Fsite-poli...

99.9% uptime corresponds to about 2 hours downtime per quarter, if my maths is correct. If that is indeed the guarantee, based on the experience at my company, GitHub has failed its promise recently (or is getting damn close). I recall 2 decent outages in the past few weeks alone. It's making me begin to doubt if GitHub's reliability is appropriate for an enterprise service.

Re: GitHub Git Operations Are Down

#245
post #192

Earlier quoted context omitted.

Things start to go sideways when you have tens of thousands of users.

Most self hosted instances would not have tens of thousands of users.

Agreed, that’s why products of that nature start to break when you do.

Re: GitHub Git Operations Are Down

#246

Earlier quoted context omitted.

The whole product feels like it’s getting g progressively jankier. The front-end is glacial nowadays and frequently has issues actually loading the page, actions frequently has some kind of panic attack and breaks or just grinds along a glacial speeds. The UX has gotten worse (why no merge-queue button?).

I wouldn't say glacial. If you want glacial try Jira. Mind-bogglingly slow.

Oh I know, I have to use both at work. God forbid you mis-click in Jira.

Navigating around takes so much time, it should probably have its own timesheet code.

Re: GitHub Git Operations Are Down

#247
post #211

Earlier quoted context omitted.

> Things start to go sideways when you have tens of thousands of users. If that’s really the case, run another GitHub instance then. Not all tens of thousands of users need access to the same codebases. In the kind of environment described someone would want identity boundaries established around each project anyway…

It’s fairly stable, but with a large codebase I’ve seen it take a day + to rebuild the search index, not to mention GHES relies on GitHub.com for the allowed actions list functionality which is a huge PITA. It should not rely on the cloud hosted version for any functionality. That having been said, I don’t think there’s much of an alternative and I quite like it.

you don't have to manage access to Actions that way.

on GHES you can use https://github.com/actions/actions-sync/ to pull the actions you want down to your local GHES instance, turn off the ability to automatically use actions from github.com via GitHub Connect, and use the list of actions you sync locally as your whitelist.

My employer did this for years. It worked very well. Once a day, pull each action that we had whitelisted into GHES and the runners would use those instead of the actions on github.com.

Re: GitHub Git Operations Are Down

#248
post #192

Earlier quoted context omitted.

We self-host GitHub using GitHub Enterprise Server. It is a mature product that requires next-to-no maintenance and is remarkably stable. (We did have a period of downtime caused by running it on an underprovisioned VM for our needs, but since resolving that it hasn't had problems.) Of course we have a small and mostly unchanging number of users, don't have to deal with DDoS attacks, and can schedule the fairly-infre…

Things start to go sideways when you have tens of thousands of users.

> Things start to go sideways when you have tens of thousands of users.

Hm not really. I manage the GHES instance at my employer and we have 15k active users. We haven't needed to scale horizontally, yet.

GHES is amazingly reliable. Every outage we have ever had has been self-inflicted; either we were too cheap to give it the resources it needed to handle the amount of users who were using it, or we tried to outsmart the recommended and supported procedures by doing things in a non-supported way.

Along the way we have learned to never deviate from the supported ways to do things, and to keep user API quota as small as possible (the team which managed this service prior to my team would increase quota per user anytime anyone asked, which was a capital-M Mistake.)

Re: GitHub Git Operations Are Down

#249

Earlier quoted context omitted.

I use a fork of Gitea called Forgejo. It works nicely as well. https://forgejo.org/

What drove you to the fork?

First I was using Gogs. Someone forked Gogs and made Gitea, because Gogs was under control of a single person and some other people found that frustrating.

https://blog.gitea.com/welcome-to-gitea/

I was using Gitea for a long time, and then someone forked Gitea to create Forgejo. At this time, my installation of Gitea was already out of date a bit because I had previously been manually building and installing Gitea from source. Soon after Forgejo was created, it landed in FreeBSD ports and then it became available in the FreeBSD package manager.

So at this point, and having read a bit about Forgejo and seeing that Forgejo was maintained by people with connection to Codeberg, I thought “hey I need to migrate my current Gitea setup anyway. Either to Gitea installed from FreeBSD packages, or to something else. I might as well try Forgejo.”

And that’s how I ended up installing Forgejo and I’ve stuck with it since.

Re: GitHub Git Operations Are Down

#250

Earlier quoted context omitted.

What drove you to the fork?

First I was using Gogs. Someone forked Gogs and made Gitea, because Gogs was under control of a single person and some other people found that frustrating. https://blog.gitea.com/welcome-to-gitea/ I was using Gitea for a long time, and then someone forked Gitea to create Forgejo. At this time, my installation of Gitea was already out of date a bit because I had previously been manually building and installing Gitea f…

Aha, makes sense. Thanks for explaining. I see that Forgejo was created at least partially due to concerns that Gitea was trending towards freemium as well. Good to know.
Post reply on HN