Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Status.github.com: “We're failing over a data storage system”

news.ycombinator.com

Post reply on HN
Print view View original

101–110 of 173 posts

Re: Status.github.com: “We're failing over a data storage system”

#101

Post by Cthulhu_ »

r_singh wrote: post #91

Just a few weeks ago my organization was in the position of choosing a version control platform for our repos. I'm so glad we went ahead with self hosted gitlab. We installed it on a CentOS server at our premise, SSL'd via Let's encrypt, I've even set up a dedicated gitlab runner to use Gitlab CI for continuous delivery and so far the testing is progressing pretty smoothly. All this for $0. Update: I agree, gitlab an…

What did the server cost? How many hours did you spend on it? What's your SLA? Who will be woken up at night when there's an outage? What's your backup and recovery procedure?

I mean kudos to you for setting it up but it's a bit naive to believe it's better than a hosted solution right away.

Re: Status.github.com: “We're failing over a data storage system”

#102

Post by Cthulhu_ »

olingern wrote: post #82
jypepin wrote: post #67

It's been over 6hrs without update, except the hourly message which states the same. I'm really looking forward this post mortem, but having been in the situation where I've had to deal with large scale outages like this one, I guarantee some engineers are having a bad time right now, and I feel for them. Github engineers, if you are reading me (probably not), KEEP IT UP, it happens to the best of us! <3

Posted at 15:51 Japan Standard Time "...with the aim of serving fully consistent data within the next 2 hours." That's somewhat significant

Yeah, sounds like a backup restore or a RAID synchronization is in progress.

Re: Status.github.com: “We're failing over a data storage system”

#103

Post by reidrac »

Someone posted a comment on an open issue, I got the mail, but there's nothing on the web interface.

What I don't understand is why they don't set it all to "read only" until the problem is sorted. Looks like any update goes to /dev/null, just let your users know!

(unless they plan to replay those updates, somehow)

Re: Status.github.com: “We're failing over a data storage system”

#104

Post by Skinney »

jononor wrote: post #77
avip wrote: post #68

Earlier quoted context omitted.

Transition to BitBucket or gilab is one click away. Companies surely will move if incentives are there.

Not when you use third party integrations. Like when using Travis for CI, and Travis doing your deploy.

Travis supports both Bitbucket and Gitlab

Re: Status.github.com: “We're failing over a data storage system”

#105

Post by majidazimi »

rodorgas wrote: post #58
ComputerGuru wrote: post #39

I can't add comments to pull requests("you can't do that right now") and any commits pushed to branches are not updating updating the visible status in the web interface, no are newly created branches showing up. However, if you navigate to a new commit directly with its SHA (so you can share it with someone if you really want), it'll show up (so they're just not being indexed). EDIT: Obligatory "that's what happens…

I’m receiving emails (a lot of duplicates) from comments in PR, but they won’t show up in the browser. I guess people are trying to submit multiple times, the email is sent but comment isn’t posted.

Welcome to eventual consistency.

Re: Status.github.com: “We're failing over a data storage system”

#106

Post by k_ »

Cthulhu_ wrote: post #101
r_singh wrote: post #91

Just a few weeks ago my organization was in the position of choosing a version control platform for our repos. I'm so glad we went ahead with self hosted gitlab. We installed it on a CentOS server at our premise, SSL'd via Let's encrypt, I've even set up a dedicated gitlab runner to use Gitlab CI for continuous delivery and so far the testing is progressing pretty smoothly. All this for $0. Update: I agree, gitlab an…

What did the server cost? How many hours did you spend on it? What's your SLA? Who will be woken up at night when there's an outage? What's your backup and recovery procedure? I mean kudos to you for setting it up but it's a bit naive to believe it's better than a hosted solution right away.

I agree that most of these are good points, although they can be worth it depending on your priorities.

As for this:

> Who will be woken up at night when there's an outage?

Unless they are working with teams around the world, does it matter if their server is down the whole night, as long as they can get it back up first thing in the morning?

Re: Status.github.com: “We're failing over a data storage system”

#107

Post by r_singh »

Cthulhu_ wrote: post #101
r_singh wrote: post #91

Just a few weeks ago my organization was in the position of choosing a version control platform for our repos. I'm so glad we went ahead with self hosted gitlab. We installed it on a CentOS server at our premise, SSL'd via Let's encrypt, I've even set up a dedicated gitlab runner to use Gitlab CI for continuous delivery and so far the testing is progressing pretty smoothly. All this for $0. Update: I agree, gitlab an…

What did the server cost? How many hours did you spend on it? What's your SLA? Who will be woken up at night when there's an outage? What's your backup and recovery procedure? I mean kudos to you for setting it up but it's a bit naive to believe it's better than a hosted solution right away.

Fair questions.

We're a company making WISP/ISP software here in India and our cloud (hosted) offering is also hosted on servers at our office itself[1]. Our infra manages hundreds of thousands of internet subscribers in India.

We're a small tech company with 25-30 people or so, with linux kernel experts who have experience of running software on questionable hardware so that service providers can take internet to new places in India, that experience helps a lot in keeping all our services up with minimum downtime.

Our office runs 24x7 with support. If anything goes down our business will be in jeopardy so we have protocol to be back up in minimum amount of time.

How many hours we spent on it? Less than a couple hours on setting up the self hosted gitlab instance. More time on getting people to get used to it and setting up the runner for CI, but I would have to spend that time on other paid solutions too.

As for time costing $0, definitely not, but people in India don't cost nearly as much as they do in the USA. Also because of experience with self hosting it is less of a deal for us to spend time on this.

[1] we have multiple bandwidth backup, electricity backup, etc needed to self host and we've always stuck with self hosting.

Re: Status.github.com: “We're failing over a data storage system”

#108

Post by steventhedev »

guywhocodes wrote: post #56
steventhedev wrote: post #54

Earlier quoted context omitted.

Are you a paying customer of GitHub Enterprise? If not, then you're getting your money's worth. Snark aside, this is a great time to reassess your deployment strategies and look into things like local apt and pypi proxies. I'm confident you can find similar projects that will transparently cache your dependencies.

I don't know how much my org is paying but it's no small amount, Enterprise no. Are we getting what we pay for? I don't think so, I think effective SLA of 99% isn't good enough for any SasS. But of course you understand that moving away from github is no small decision. Absolutely is this a perfect time to assess deployment strategies and challenge all the advice of how big a company has to be before it's worth to do…

It sounds like you're using GH for ticketing also, which means that your R&D org's productivity is tightly coupled with an external companies uptime. Self-hosting Gitlab is easy, as is running Jira, redmine, and a dozen other tools.

If you can't do your job without a tool, then you need to have a plan B on hand for when that tool fails. Both in the micro sense of the tools you use to code (editor, browser, laptop, mouse, coffee mug, etc) and in the macro sense of tools your organization uses (ticketing systems, chat, bathrooms). Show some initiative, figure out some mirrors for your dependencies, and try standing up a local caching proxy for your team. It'll probably take a lot less time than you think.

Re: Status.github.com: “We're failing over a data storage system”

#109

Post by r_singh »

danpalmer wrote: post #99
r_singh wrote: post #91

Just a few weeks ago my organization was in the position of choosing a version control platform for our repos. I'm so glad we went ahead with self hosted gitlab. We installed it on a CentOS server at our premise, SSL'd via Let's encrypt, I've even set up a dedicated gitlab runner to use Gitlab CI for continuous delivery and so far the testing is progressing pretty smoothly. All this for $0. Update: I agree, gitlab an…

Your time isn’t free. GitLab may well be the right choice for many teams, but I’ve not yet seen price be the winning factor over the long term. Between general sysadmin tasks, scaling with the size of your team, doing upgrades, and performing database migrations, the cost of running GitLab will probably be the same ballpark as GitHub.

This is true, we've always had troubles with amortizing costs for time spent on doing something.

We're a profitable software company that intends to remain in business forever, we're well staffed though and people do have time to try out things and even do R&D on the side. Frugality is built in to our DNA and having spare time on top means failure at amortizing time costs.

Re: Status.github.com: “We're failing over a data storage system”

#110

Post by dis-sys »

r_singh wrote: post #91

Just a few weeks ago my organization was in the position of choosing a version control platform for our repos. I'm so glad we went ahead with self hosted gitlab. We installed it on a CentOS server at our premise, SSL'd via Let's encrypt, I've even set up a dedicated gitlab runner to use Gitlab CI for continuous delivery and so far the testing is progressing pretty smoothly. All this for $0. Update: I agree, gitlab an…

you are basically assuming that your deployed version of GitLab is bug free and your hardware will be able to serve you 24/7 without downtime in the next few years.
Post reply on HN