Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Git Operations Are Down

githubstatus.com

Post reply on HN
Print view View original

211–220 of 271 posts

Re: GitHub Git Operations Are Down

#211

Post by lukevp »

randmeerkat wrote: post #201
jcims wrote: post #192

Earlier quoted context omitted.

Things start to go sideways when you have tens of thousands of users.

> Things start to go sideways when you have tens of thousands of users. If that’s really the case, run another GitHub instance then. Not all tens of thousands of users need access to the same codebases. In the kind of environment described someone would want identity boundaries established around each project anyway…

It’s fairly stable, but with a large codebase I’ve seen it take a day + to rebuild the search index, not to mention GHES relies on GitHub.com for the allowed actions list functionality which is a huge PITA. It should not rely on the cloud hosted version for any functionality. That having been said, I don’t think there’s much of an alternative and I quite like it.

Re: GitHub Git Operations Are Down

#212

Post by jamesfinlayson »

deathanatos wrote: post #148
macintux wrote: post #118

Earlier quoted context omitted.

The nice thing about git, from my perspective, is that if your entire hosted service vanishes, you can still reconstruct what you need from your users’ working directories. All of the important branches should be there. Somewhere. And any important integration branches that aren’t cached can be reconstructed. Of all the many dependencies on cloud services, git is by far the last I’d worry overly much about.

The source code, perhaps, but a good many of the orgs I've worked at also use Github for PRs, Actions (CI), and for triggering deployments. Those things take time to setup, and across a whole org, I wouldn't want to have to have an unplanned change to another vendor. In particular, I've moved a CI for a large repository between different CI systems. It was anything but trivial: you want to believe "it's just a YAML t…

Ugh, agreed - I remember doing a GitHub+TeamCity to GitLab migration and I had specifically designed all of my jobs to just directly call out to a bash script. Artifacts and GitLab's infinite hooks made that much harder than it seemed like it would be.

Re: GitHub Git Operations Are Down

#213

Post by shortrounddev2 »

chrisandchris wrote: post #209
shortrounddev2 wrote: post #207

Earlier quoted context omitted.

Could have a self hosted git or ci/cd pipeline for deployments and code access during outages. They don't need to be up constantly, just need to be able to keep a backup of the code somewhere and have some way to run critical deployment scripts if github actions are unavailable

Sounds like a backup one never tested, nobody verified and absolutely will not work if needed. Either you go full in, or you'll better don't do it.

Well these things should be tested and verified periodically, just like ANY backup. You shouldn't just tick the "enable backups" box in AWS and then never actually test them

Re: GitHub Git Operations Are Down

#214

Post by __MatrixMan__ »

arrowsmith wrote: post #122
__MatrixMan__ wrote: post #119

Earlier quoted context omitted.

I was imagining something a bit more disastrous than that. A big enough solar flare could take parts of the planet offline for months. Years if they can't source enough replacement transformers. There are also political reasons that countries go offline. Then it'll be up to the nerds who manage to cobble together their own distributed version of everything--even if it's a significantly reduced definition of everythin…

If large parts of the planet lose their digital infrastructure for months, I really don’t think that “finding a good platform to host my code” is going to be one of my biggest problems.

I think that how big those problems get is going to depend on how much critical infrastructure we can hack back into a working state despite the fact that it can't phone home, which is going to be a problem if nobody knows how to work offline anymore.

Or, if it's a political scenario, it may depend on how well we can coordinate en masse without the cut connection. If we can exceed a certain threshold then we'll have removed the incentive to cut it in the first place.

Re: GitHub Git Operations Are Down

#216

Post by bigiain »

jcims wrote: post #192
bgrainger wrote: post #190

Earlier quoted context omitted.

We self-host GitHub using GitHub Enterprise Server. It is a mature product that requires next-to-no maintenance and is remarkably stable. (We did have a period of downtime caused by running it on an underprovisioned VM for our needs, but since resolving that it hasn't had problems.) Of course we have a small and mostly unchanging number of users, don't have to deal with DDoS attacks, and can schedule the fairly-infre…

Things start to go sideways when you have tens of thousands of users.

I would have thought if you had tens of thousands of developers all needing access to the same git repos, then you'd probably have a follow-the-sun team of maybe 50 or 100 engineers working on your git infra.

Re: GitHub Git Operations Are Down

#217

Post by physicsguy »

deathanatos wrote: post #148
macintux wrote: post #118

Earlier quoted context omitted.

The nice thing about git, from my perspective, is that if your entire hosted service vanishes, you can still reconstruct what you need from your users’ working directories. All of the important branches should be there. Somewhere. And any important integration branches that aren’t cached can be reconstructed. Of all the many dependencies on cloud services, git is by far the last I’d worry overly much about.

The source code, perhaps, but a good many of the orgs I've worked at also use Github for PRs, Actions (CI), and for triggering deployments. Those things take time to setup, and across a whole org, I wouldn't want to have to have an unplanned change to another vendor. In particular, I've moved a CI for a large repository between different CI systems. It was anything but trivial: you want to believe "it's just a YAML t…

This is a good reason for keeping your build steps in scripts or a Makefile rather than jumping head first into the ecosystem.

I think unless you’ve been burned by having to move CI provider before it’s easy to lean in. I had to change from Travis many years ago because of pricing changes.

Re: GitHub Git Operations Are Down

#218

Post by saagarjha »

kspacewalk2 wrote: post #195
vrosas wrote: post #132

Earlier quoted context omitted.

Once in a blue moon? It’s like, monthly at best.

When was the last time you noticed? Not saw a HN post, but it actually stopped you from pushing commits or commenting on an issue or whatever?

Today ;)

Re: GitHub Git Operations Are Down

#219

Post by paulddraper »

jcims wrote: post #192
bgrainger wrote: post #190

Earlier quoted context omitted.

We self-host GitHub using GitHub Enterprise Server. It is a mature product that requires next-to-no maintenance and is remarkably stable. (We did have a period of downtime caused by running it on an underprovisioned VM for our needs, but since resolving that it hasn't had problems.) Of course we have a small and mostly unchanging number of users, don't have to deal with DDoS attacks, and can schedule the fairly-infre…

Things start to go sideways when you have tens of thousands of users.

Most self hosted instances would not have tens of thousands of users.

Re: GitHub Git Operations Are Down

#220

Post by paulddraper »

shortrounddev2 wrote: post #207
thayne wrote: post #189

Earlier quoted context omitted.

Let's say you self-host Github. Now you are responsible for maintaining uptime, and you have less expertise with the service and less resources to dedicate to keeping it up, so it's going to be hard to match, much less exceed the uptime of Github cloud. And it doesn't protect you from a "forced exit" either. Github could terminate your contract, and change the terms of the license in a way that you found unacceptable…

Could have a self hosted git or ci/cd pipeline for deployments and code access during outages. They don't need to be up constantly, just need to be able to keep a backup of the code somewhere and have some way to run critical deployment scripts if github actions are unavailable

What self hosted CI do you recommend?
Post reply on HN