Status.github.com: “We're failing over a data storage system” • Hacker News Board

Re: Status.github.com: “We're failing over a data storage system”

#91

Post by r_singh » Mon, Oct 22, 2018, 7:37 AM UTC

Just a few weeks ago my organization was in the position of choosing a version control platform for our repos. I'm so glad we went ahead with self hosted gitlab. We installed it on a CentOS server at our premise, SSL'd via Let's encrypt, I've even set up a dedicated gitlab runner to use Gitlab CI for continuous delivery and so far the testing is progressing pretty smoothly.

All this for $0.

Update:

I agree, gitlab and github will cost the same in the long run. Time costs money and self hosted has a lot of downsides as well.

However, gitlab being self hosted (and not costing anything in terms of separate invoice) makes us feel like we're more in control and are somehow saving some money (as we add more developers as well). This comment here was just sparked by a sigh of relief because this week is an important release for us and had we been on github this monday wouldn't have started too well.

Overall, all the comments listing the trade-offs are true. Time costs $$$ and a paid hosted solution is worth the minimal cost, time and expertise required.

Re: Status.github.com: “We're failing over a data storage system”

#92

Post by manigandham » Mon, Oct 22, 2018, 7:42 AM UTC

arthurcolle wrote: post #64

CodeM0nkey wrote: post #63

Earlier quoted context omitted.

GitHub has not yet been acquired by Microsoft, they're completely separate.

I wonder if this will vaporize the deal.

That's not how these deals work. A little infrastructure downtime is not going to stop a 7.5B platform acquisition.

Re: Status.github.com: “We're failing over a data storage system”

#93

Post by OJFord » Mon, Oct 22, 2018, 7:42 AM UTC

mehrdadn wrote: post #55

Fuzzwah wrote: post #32

Earlier quoted context omitted.

It is to ensure that the updates get mirrored onto twitter, where exact duplicates can't be posted.

They can't alternate?

For several hours it was alternating.

Re: Status.github.com: “We're failing over a data storage system”

#94

Post by rurban » Mon, Oct 22, 2018, 7:44 AM UTC

sciurus wrote: post #22

igni wrote: post #12

Before all the trolling about Microsoft starts up, does anyone have current information on what these systems are? In the enterprise space, a 'data storage system' could be an Array or a SAN or a lightpath etc, with usually quite long failover times. For an org like GitHub I'd think more like an object store (an Array-of-Hosts, if you will) or whatever storage mechanism holds their database files. Do they self host t…

For storing things other than git repos, GitHub is heavily invested in MySQL. AFAIK all of GitHub is hosted on their own hardware. https://githubengineering.com/mysql-high-availability-at-git...

This blog post describes exactly the scenario we were experiencing here. A master (single writer) failure, with missing fail over. You can only guess what went wrong with this plan. Looks good on paper, but some unexpected network or HW or routing problem could have caused the problem to identify the single writer.

Re: Status.github.com: “We're failing over a data storage system”

#95

Post by woolvalley » Mon, Oct 22, 2018, 7:49 AM UTC

r_singh wrote: post #91

Just a few weeks ago my organization was in the position of choosing a version control platform for our repos. I'm so glad we went ahead with self hosted gitlab. We installed it on a CentOS server at our premise, SSL'd via Let's encrypt, I've even set up a dedicated gitlab runner to use Gitlab CI for continuous delivery and so far the testing is progressing pretty smoothly. All this for $0. Update: I agree, gitlab an…

+ cost of additional labor

Re: Status.github.com: “We're failing over a data storage system”

#96

Post by akerl_ » Mon, Oct 22, 2018, 7:49 AM UTC

r_singh wrote: post #91

Just a few weeks ago my organization was in the position of choosing a version control platform for our repos. I'm so glad we went ahead with self hosted gitlab. We installed it on a CentOS server at our premise, SSL'd via Let's encrypt, I've even set up a dedicated gitlab runner to use Gitlab CI for continuous delivery and so far the testing is progressing pretty smoothly. All this for $0. Update: I agree, gitlab an…

I feel like this is the first chapter of a story whose ending is probably happy, but whose plot twist is "and then the on-prem server crashed and it took a week to figure out how to fix it".

Self-hosting vs 3rd party hosting has a ton of tradeoffs, but systems can fall over no matter who's hosting them.

Re: Status.github.com: “We're failing over a data storage system”

#97

Post by yread » Mon, Oct 22, 2018, 7:50 AM UTC

r_singh wrote: post #91

Just a few weeks ago my organization was in the position of choosing a version control platform for our repos. I'm so glad we went ahead with self hosted gitlab. We installed it on a CentOS server at our premise, SSL'd via Let's encrypt, I've even set up a dedicated gitlab runner to use Gitlab CI for continuous delivery and so far the testing is progressing pretty smoothly. All this for $0. Update: I agree, gitlab an…

You value your time at 0$ ?

Re: Status.github.com: “We're failing over a data storage system”

#98

Post by ToFab123 » Mon, Oct 22, 2018, 7:51 AM UTC

woolvalley wrote: post #95

r_singh wrote: post #91

Just a few weeks ago my organization was in the position of choosing a version control platform for our repos. I'm so glad we went ahead with self hosted gitlab. We installed it on a CentOS server at our premise, SSL'd via Let's encrypt, I've even set up a dedicated gitlab runner to use Gitlab CI for continuous delivery and so far the testing is progressing pretty smoothly. All this for $0. Update: I agree, gitlab an…

+ cost of additional labor

Plus cost of nobody can work because their single server crashed

Re: Status.github.com: “We're failing over a data storage system”

#99

Post by danpalmer » Mon, Oct 22, 2018, 7:51 AM UTC

r_singh wrote: post #91

Just a few weeks ago my organization was in the position of choosing a version control platform for our repos. I'm so glad we went ahead with self hosted gitlab. We installed it on a CentOS server at our premise, SSL'd via Let's encrypt, I've even set up a dedicated gitlab runner to use Gitlab CI for continuous delivery and so far the testing is progressing pretty smoothly. All this for $0. Update: I agree, gitlab an…

Your time isn’t free.

GitLab may well be the right choice for many teams, but I’ve not yet seen price be the winning factor over the long term. Between general sysadmin tasks, scaling with the size of your team, doing upgrades, and performing database migrations, the cost of running GitLab will probably be the same ballpark as GitHub.

Re: Status.github.com: “We're failing over a data storage system”

#100

Post by ObsoleteNerd » Mon, Oct 22, 2018, 7:53 AM UTC

Memosyne wrote: post #70

ObsoleteNerd wrote: post #42

Earlier quoted context omitted.

Targeted attacks on specific aspects/assets of a major websites infrastructure? Eg target a specific service that both sites have in common in their back end? Not even sure if that's feasible but it's an idea.

I figured it was just a coincidence, but the HN network is so well informed I thought I'd just ask the question on the off-chance it wasn't. I personally don't find a relation likely.

Oh I find the relation very unlikely. I was just trying to think up a potential connection for fun/curiosity, as the coincidence is pretty crazy that two huge sites known for very few/little outages had major outages in the same week.