Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Status.github.com: “We're failing over a data storage system”

news.ycombinator.com

Post reply on HN
Print view View original

121–130 of 173 posts

Re: Status.github.com: “We're failing over a data storage system”

#121

Post by r_singh »

akerl_ wrote: post #96
r_singh wrote: post #91

Just a few weeks ago my organization was in the position of choosing a version control platform for our repos. I'm so glad we went ahead with self hosted gitlab. We installed it on a CentOS server at our premise, SSL'd via Let's encrypt, I've even set up a dedicated gitlab runner to use Gitlab CI for continuous delivery and so far the testing is progressing pretty smoothly. All this for $0. Update: I agree, gitlab an…

I feel like this is the first chapter of a story whose ending is probably happy, but whose plot twist is "and then the on-prem server crashed and it took a week to figure out how to fix it". Self-hosting vs 3rd party hosting has a ton of tradeoffs, but systems can fall over no matter who's hosting them.

This would definitely be true if we didn't have experience with self hosting services that are critical to the businesses of our customers.

Luckily for us, we do have experience with figuring out servers and are usually required to fix crashes within minutes as the unit measured rather than days.

Re: Status.github.com: “We're failing over a data storage system”

#122

Post by r_singh »

yread wrote: post #97
r_singh wrote: post #91

Just a few weeks ago my organization was in the position of choosing a version control platform for our repos. I'm so glad we went ahead with self hosted gitlab. We installed it on a CentOS server at our premise, SSL'd via Let's encrypt, I've even set up a dedicated gitlab runner to use Gitlab CI for continuous delivery and so far the testing is progressing pretty smoothly. All this for $0. Update: I agree, gitlab an…

You value your time at 0$ ?

Definitely not, that was just a hyperbole sparked by a sigh of relief that our release won't be affected because of this outage.

Re: Status.github.com: “We're failing over a data storage system”

#123

Post by r_singh »

dis-sys wrote: post #110
r_singh wrote: post #91

Just a few weeks ago my organization was in the position of choosing a version control platform for our repos. I'm so glad we went ahead with self hosted gitlab. We installed it on a CentOS server at our premise, SSL'd via Let's encrypt, I've even set up a dedicated gitlab runner to use Gitlab CI for continuous delivery and so far the testing is progressing pretty smoothly. All this for $0. Update: I agree, gitlab an…

you are basically assuming that your deployed version of GitLab is bug free and your hardware will be able to serve you 24/7 without downtime in the next few years.

I'm confident about the downtime not being an issue for us because of experience with all kinds of server issues.

I'm interested to see what our experience with bugs and updates is in the future, my happiness could very well be premature.

Re: Status.github.com: “We're failing over a data storage system”

#124

Post by OJFord »

samontar wrote: post #85
steventhedev wrote: post #54

Earlier quoted context omitted.

Are you a paying customer of GitHub Enterprise? If not, then you're getting your money's worth. Snark aside, this is a great time to reassess your deployment strategies and look into things like local apt and pypi proxies. I'm confident you can find similar projects that will transparently cache your dependencies.

If you’re paying for GHE you don’t have any trouble. That’s delivered as an appliance you host.

No it isn't: https://enterprise.github.com/features#pricing

Re: Status.github.com: “We're failing over a data storage system”

#125

Post by dvfjsdhgfv »

jononor wrote: post #78
dvfjsdhgfv wrote: post #75

Earlier quoted context omitted.

> Before all the trolling about Microsoft starts up, does anyone have current information on what these systems are? To be fair, you can't call it trolling without knowing for sure that the problem hasn't been caused by Azure-related part of the infrastructure. Looking forward to reading the post-mortem.

It is incredibly unlikely that there is any Azure in the production infrastructure. The acquisition is too recent for big changes like that.

I admire your confidence but will withhold my judgement until the postmortem is published.

Re: Status.github.com: “We're failing over a data storage system”

#126

Post by majewsky »

ajb257 wrote: post #81
keyle wrote: post #20

I don't know if they keep changing the text updates with a slightly different version to - prove it's a human that typed it - there is code the prevents repeating twice the same message either way it's entertaining... But it's Monday morning in Australia and we need to release! (yep we do this via pr/tagging etc.)

Given the latest update is "We are currently in the later stages of a restore operation, with the aim of serving fully consistent data within the next 2 hours", I can only assume someone at GH read your comment!

Hacker News is, surprisingly, not the center of the universe.

Re: Status.github.com: “We're failing over a data storage system”

#127

Post by Grue3 »

ToFab123 wrote: post #98
woolvalley wrote: post #95

Earlier quoted context omitted.

+ cost of additional labor

Plus cost of nobody can work because their single server crashed

Git is a distributed system. People can work if they want to, but it's certainly a convenient excuse to not be working.

Re: Status.github.com: “We're failing over a data storage system”

#128

Post by dustinmoris »

r_singh wrote: post #91

Just a few weeks ago my organization was in the position of choosing a version control platform for our repos. I'm so glad we went ahead with self hosted gitlab. We installed it on a CentOS server at our premise, SSL'd via Let's encrypt, I've even set up a dedicated gitlab runner to use Gitlab CI for continuous delivery and so far the testing is progressing pretty smoothly. All this for $0. Update: I agree, gitlab an…

> makes us feel like we're more in control and are somehow saving some money

This is precisely the problem with developers. You decided to go with a complex self built solution because it "feels" cheaper. If only I would have received a $ every time someone made a wrong decision based on a feeling I'd be a rich man today.

Re: Status.github.com: “We're failing over a data storage system”

#129

Post by lowry »

dustinmoris wrote: post #128
r_singh wrote: post #91

Just a few weeks ago my organization was in the position of choosing a version control platform for our repos. I'm so glad we went ahead with self hosted gitlab. We installed it on a CentOS server at our premise, SSL'd via Let's encrypt, I've even set up a dedicated gitlab runner to use Gitlab CI for continuous delivery and so far the testing is progressing pretty smoothly. All this for $0. Update: I agree, gitlab an…

> makes us feel like we're more in control and are somehow saving some money This is precisely the problem with developers. You decided to go with a complex self built solution because it "feels" cheaper. If only I would have received a $ every time someone made a wrong decision based on a feeling I'd be a rich man today.

There's no cloud. Just someone else's computer.
Post reply on HN