Live data from Hacker News

GitHub was down

status.github.com

81–90 of 197 posts

Re: GitHub was down

#81
post #70

Earlier quoted context omitted.

Is it still considered a runtime dependency if all you do is merge code and deploy while relying on GitHub? I assume most apps will not go down if GitHub goes down, but their ability to move code to production gets stalled.

Git is inherently distributed. There's really no reason that Github has to stand between development and production.

[deleted]

Re: GitHub was down

#82

I would love to see the outage postmortem. To be honest designing a completely redundant service today is just not that hard are compared to 10 years ago. The ability to load balance, route, us VMs/containers and move loads makes is fairly simple. In 1998 when I was building an backbone and ISP it was much harder. You had a ton of single points of failure by the nature of the hardware and software at the time. We pur…

Kind of? How many servers did the average app depend on in 1998? 1? Get dual HD and you were in decent shape. Compare to a modern microservice app, that maybe depends on 100 internal services and 4-5 external services. A lot of things need to go right or mostly right for things to function.

I think you both have a point. The tools are mostly all here now (some in primitive forms), but no one person on earth knows how to use them all. From netfilter to kubernetes to sharded redis instances to nginx RTMP to embeded golang structs and git commit hooks.

It takes an amazing team to cover all the bases.

Re: GitHub was down

#83
post #60

Earlier quoted context omitted.

They offer a local instance of GitHub actually.

Which I can't imagine to be any more reliable than hosted GitHub. But hey, I guess there's the comfort in know that when it goes down it's our fault!

Reliability can always be improved... It just costs more.

Re: GitHub was down

#84
post #70

Earlier quoted context omitted.

Git is inherently distributed. There's really no reason that Github has to stand between development and production.

There's the convenience reason. Teams making that trade-off (not having a local mirror) need to understand they made GitHub itself a runtime dependency.

But in theory in a pinch you could spin up a git server somewhere publicly accessible and switch your site to pull from there, and push to there yourself.

Re: GitHub was down

#85
post #51
post #34

Earlier quoted context omitted.

Does anyone know an alternative to githost.io? We don't want to use gitlab.org as it's painfully slow but also don't want to manage our own gitlab instance. edit: We don't like the recent changes made in githost. ( Replaced the $35 option with $149 ) We're a team of 15 people. edit2: should have clarified; I'm hooked to GitLab, just looking for another service that does managed hosting. Not looking for GitLab alterna…

Gitea[0] is extremely simple to host, if you want a self-hosted solution that's not as heavy as gitlab. [0] https://gitea.io

Gitea sadly has some issues (for example, the "Public Activity" tab on user profiles 500's atm: https://try.gitea.io/crackedmind?tab=activity; it throws 500's everywhere once you throw on a modern Xeon more than 30 users/second at it, because it doesn't cache database accesses, at all, etc).

Re: GitHub was down

#86
post #10

What do people do to get around this? Run my own git server like the good 'ol days? Github has become a central source of failure for us now...

You need to either ensure you control the resources you depend on, or plan for outages.

At my gig, we use GH Enterprise (on-prem) as well as some smaller plain-vanilla git repos (mainly for CM and other critical services). Any third-party code and all other dependencies live in an artifact manager or private mirror repos.

Personally, I don't use any cloud provider to store my data. Cliche, but true: if you don't control the hardware, you don't control the data on it.

Re: GitHub was down

#87
post #60

Earlier quoted context omitted.

They offer a local instance of GitHub actually.

Which I can't imagine to be any more reliable than hosted GitHub. But hey, I guess there's the comfort in know that when it goes down it's our fault!

The argument is with a self-hosted solution you can choose your "danger times".

If you know you have a major deliverable coming up, you can choose not to touch your git server until it's at a somewhat safe moment to do so.

Re: GitHub was down

#88
post #40

Earlier quoted context omitted.

Exactly this. It's painful to realize that Git was invented to solve this very problem, but so many use GitHub as their source control server just like CVS and all the other old source control systems.

Then what's the alternative? Push and pull directly from other users who have access to the repository? What if that user is currently offline? There are huge benefits to having a main centralized repo. If you need redundancy for some reason, it's fairly trivial to mirror the repository somewhere else.

If your workflow depends on centralization, run your own centralized service. If you can't live without the GUI, Github will even sell you Github Enterprise. Or just run Gitolite or one of the other tools.

Re: GitHub was down

#89
post #10

What do people do to get around this? Run my own git server like the good 'ol days? Github has become a central source of failure for us now...

just set up a server via webhook that pulls the repo everytime someone pushes to it. When GitHub goes down you can use that server.

Re: GitHub was down

#90

I would love to see the outage postmortem. To be honest designing a completely redundant service today is just not that hard are compared to 10 years ago. The ability to load balance, route, us VMs/containers and move loads makes is fairly simple. In 1998 when I was building an backbone and ISP it was much harder. You had a ton of single points of failure by the nature of the hardware and software at the time. We pur…

In 1998

There's a big difference between what a high-capacity/availability site had to handle in 1998 and 20 years later.

There are also lots of easy criticisms one can level at github, given their uptime and what they've published about their architecture. 'What they're doing is fairly simple' is probably not among them.

Post reply on HN