Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitHub Git Operations Are Down

githubstatus.com

Post reply on HN
Print view View original

201–210 of 271 posts

Re: GitHub Git Operations Are Down

#201

Post by randmeerkat »

jcims wrote: post #192
bgrainger wrote: post #190

Earlier quoted context omitted.

We self-host GitHub using GitHub Enterprise Server. It is a mature product that requires next-to-no maintenance and is remarkably stable. (We did have a period of downtime caused by running it on an underprovisioned VM for our needs, but since resolving that it hasn't had problems.) Of course we have a small and mostly unchanging number of users, don't have to deal with DDoS attacks, and can schedule the fairly-infre…

Things start to go sideways when you have tens of thousands of users.

> Things start to go sideways when you have tens of thousands of users.

If that’s really the case, run another GitHub instance then. Not all tens of thousands of users need access to the same codebases. In the kind of environment described someone would want identity boundaries established around each project anyway…

Re: GitHub Git Operations Are Down

#202

Post by Over2Chars »

ikiris wrote: post #197
Over2Chars wrote: post #161

Earlier quoted context omitted.

If Jeff is using DataCenter XYZ, and then letting me use his API to manage my VMs on his "cloud", and I decide to rent a cage at the exact same DataCenter XYZ on my own servers, I'm now "self-hosting". Same Datacenter. Same reliability infrastructure wise - power, earthquakes, tsunami, typhoons, black plague, monkey pox, etc.

You seem to equate the some straw man version of the cloud to simple rack hosting in a single location. They are nothing alike. A cloud service is almost always geographically diverse and highly available in a way that is beyond most people to build out.

Am I? Cloud providers usually quote price estimates for their least redundant single region services to give the impression of cost competitiveness.

But their virtual offerings are much less reliable than a standalone system by a lot (they guarantee to refund you the 25cents for your instance if it goes down, not the value of the service interruption or its cost. lol! Read that TOS)

What's the solution to their inherent unreliability? Redundancy at more cost.

Well, hey, you can rent two colocation facilities if you really need redundancy across geographic regions. And maybe you can just use your colo as a source for a CDN that is geographically diverse (for latency, not hurricanes).

Geographic diversity and HA is beyond most people? If you're the kind of business that needs that, you can hire the exact same people that Amazon Pip'd and fired because they didn't hit some arbitrary ticket metrics, to scale your business.

e.g. https://www.forbes.com/sites/lucianapaulise/2022/10/27/amazo...

Re: GitHub Git Operations Are Down

#203

Post by doublepg23 »

vrosas wrote: post #132
kspacewalk2 wrote: post #65

Earlier quoted context omitted.

Counterpoint: it's a 45 minute outage once in a blue moon. Very small price to pay for the convenience of a centralized VCS with many features that aren't easy to reliably set up in standalone installations.

Once in a blue moon? It’s like, monthly at best.

Once a full moon maybe? There's one tonight.

Re: GitHub Git Operations Are Down

#206

Post by [deleted] »

arrowsmith wrote: post #122
__MatrixMan__ wrote: post #119

Earlier quoted context omitted.

I was imagining something a bit more disastrous than that. A big enough solar flare could take parts of the planet offline for months. Years if they can't source enough replacement transformers. There are also political reasons that countries go offline. Then it'll be up to the nerds who manage to cobble together their own distributed version of everything--even if it's a significantly reduced definition of everythin…

If large parts of the planet lose their digital infrastructure for months, I really don’t think that “finding a good platform to host my code” is going to be one of my biggest problems.

[deleted]

Re: GitHub Git Operations Are Down

#207

Post by shortrounddev2 »

thayne wrote: post #189
jcims wrote: post #88

Tangential at best but I work in a regulated industry and we're starting to get some heavy scrutiny from external auditors and regulators on the way we plan to address extended outages and forced exits from third party vendors. This goes beyond basic DR/BCP and plans are reviewed with at least a high level scenario exercise. https://www.federalregister.gov/documents/2023/06/09/2023-12... On the surface a product like…

Let's say you self-host Github. Now you are responsible for maintaining uptime, and you have less expertise with the service and less resources to dedicate to keeping it up, so it's going to be hard to match, much less exceed the uptime of Github cloud. And it doesn't protect you from a "forced exit" either. Github could terminate your contract, and change the terms of the license in a way that you found unacceptable…

Could have a self hosted git or ci/cd pipeline for deployments and code access during outages. They don't need to be up constantly, just need to be able to keep a backup of the code somewhere and have some way to run critical deployment scripts if github actions are unavailable

Re: GitHub Git Operations Are Down

#208

Post by wongogue »

Over2Chars wrote: post #158
gerdesj wrote: post #89

Earlier quoted context omitted.

A bloke called Linus Thorvalds created git for Linux development when the commercial service used for that ceased to be useful, for one non technical reason or another. github basically shoves a webby frontend and workflows on top of someone else's work. That's all fine and good but github is not git. As a professional IT consultant, I want tools, I use lots of other's and I also create my own and I also generally in…

I believe he found the open source tools being used (CVS?) weren't good enough, and he started using a commercial closed source tool called "bitkeeper", which rankled the ire of the FOSS community who wanted to eat their own dogfood. So Torvald's opted to "clone" the features of bitkeeper into an open source version he named 'git'. That's the story I heard, no idea if it's true.

They were using BitKeeper for years but git was created when BitKeeper pulled the rug licensing-wise.

Source: A Git Story from https://blog.brachiosoft.com/en/posts/git/

Re: GitHub Git Operations Are Down

#209

Post by chrisandchris »

shortrounddev2 wrote: post #207
thayne wrote: post #189

Earlier quoted context omitted.

Let's say you self-host Github. Now you are responsible for maintaining uptime, and you have less expertise with the service and less resources to dedicate to keeping it up, so it's going to be hard to match, much less exceed the uptime of Github cloud. And it doesn't protect you from a "forced exit" either. Github could terminate your contract, and change the terms of the license in a way that you found unacceptable…

Could have a self hosted git or ci/cd pipeline for deployments and code access during outages. They don't need to be up constantly, just need to be able to keep a backup of the code somewhere and have some way to run critical deployment scripts if github actions are unavailable

Sounds like a backup one never tested, nobody verified and absolutely will not work if needed.

Either you go full in, or you'll better don't do it.

Re: GitHub Git Operations Are Down

#210

Post by supriyo-biswas »

dangoldin wrote: post #139
macintux wrote: post #118

Earlier quoted context omitted.

The nice thing about git, from my perspective, is that if your entire hosted service vanishes, you can still reconstruct what you need from your users’ working directories. All of the important branches should be there. Somewhere. And any important integration branches that aren’t cached can be reconstructed. Of all the many dependencies on cloud services, git is by far the last I’d worry overly much about.

Problem is that often you also end up relying on GitHub for CI/CD so not as easy of a change. Imagine GH being down and you need to deploy a hotfix. How do you handle that? Especially, if you followed best practices and set up a system where all PRs need to go through code review.

This is why I personally like to use none of the CI features, and mostly use it like a shell script executor. Images? Stick to OS images only so that you can easily spin them up with `docker run` locally. Artifacts? Read and write them into S3 buckets and avoid the native artifact features.

This is obviously more difficult in the Github actions ecosystem, but I have mostly used Gitlab CI so far. My CI pipelines mostly look like this:

    image: ubuntu:24.04
    before_script:
        - apt-get install ...
    script:
        - ./ci/build-project.sh
    after_script:
        - ./ci/upload-build-artifacts.sh
Post reply on HN