Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Status.github.com: “We're failing over a data storage system”

news.ycombinator.com

Post reply on HN
Print view View original

161–170 of 173 posts

Re: Status.github.com: “We're failing over a data storage system”

#161

Post by deadbunny »

jammygit wrote: post #49
hactually wrote: post #47

Earlier quoted context omitted.

That's a silly statement. By saying 'nobody' - a single point of data invalidates your assertion. I use gmail, I'm quite happy to trust that contract.

You're saying that you trust that contract today, or are you saying that you have always trusted that contract? Its only recently that gmail's contract involved keeping out of your data. I think they also only say they abstain from using your data for targeted advertising, not that they don't use it for other purposes. I haven't read the terms in quite a while though and I could be mistaken. Great products though. I…

There is a whole world of difference between paid for and not paid for Google services.

Re: Status.github.com: “We're failing over a data storage system”

#162

Post by deadbunny »

sajithdilshan wrote: post #155
sajithdilshan wrote: post #149

I think it is obvious what happened. They must have moved their backend to Windows 10.

It's a joke people. Why downvote?

Because it's a boring, obvious joke which wasn't funny even in the 90s?

Re: Status.github.com: “We're failing over a data storage system”

#163

Post by dsumenkovic »

r_singh wrote: post #91

Just a few weeks ago my organization was in the position of choosing a version control platform for our repos. I'm so glad we went ahead with self hosted gitlab. We installed it on a CentOS server at our premise, SSL'd via Let's encrypt, I've even set up a dedicated gitlab runner to use Gitlab CI for continuous delivery and so far the testing is progressing pretty smoothly. All this for $0. Update: I agree, gitlab an…

Hello, Community Advocate from GitLab here.

Thank you for sharing your feedback and comments with everyone, we really appreciate that. It's great to hear how you are handling your self-hosted GitLab instance, that's awesome!

Re: Status.github.com: “We're failing over a data storage system”

#164

Post by benmmurphy »

sudhirj wrote: post #143
benmmurphy wrote: post #137

My guess at what happened: They had a split brain when multiple masters were running. Then they were not able to choose a master to keep because the data in both masters was 'corrupted' so they are now restoring from a backup. So how do they get data corruption from multiple masters running: 1) Performing reads from slaves during an update operation. If you perform a read from a slave then you might get data from the…

Split brain can run for a while if the all identifiers are UUIDs and tables are used as append only. Restoration is complex, though.

depending on your application logic you can get external inconsistencies. like your typical CRUD app might dump the state of an entity in an edit form, then a user might edit one field, and the whole form is sent to the backend and written over whatever exists there. if you have two different masters then you can get a series of updates where you can't tell if the update is because the user meant to make the change or if it was because the change was incorrectly propagated from the other master. like even if you knew the user read from the other master you can't tell if the user intended the change or not. maybe they saw the value was the value they wanted so they didn't change it.

i guess this is a 'bug' because when you do stuff like this you should include some kind of version identifier to catch a concurrent update in the normal case. this scenario is also 'storing stuff outside the DB' kind of similar to the user_id in the cookie getting out of sync.

Re: Status.github.com: “We're failing over a data storage system”

#165

Post by OnlyRepliesToBS »

Ma8ee wrote: post #150
tjoff wrote: post #148

Earlier quoted context omitted.

I appreciate that they try. As a user I would never rely on those estimates, but it at least gives you a sense of what's going on and what they are doing. I take that over the more common silence or "we are working on it" any day.

Exactly. We all know not to trust that those estimates are exact, but they are much better than nothing. When they say nothing we don't know if it's half an hour, half a day or half a week.

Ping? Pong!

It's still useless to us.

Re: Status.github.com: “We're failing over a data storage system”

#166

Post by r_singh »

simonjgreen wrote: post #133
r_singh wrote: post #107

Earlier quoted context omitted.

Fair questions. We're a company making WISP/ISP software here in India and our cloud (hosted) offering is also hosted on servers at our office itself[1]. Our infra manages hundreds of thousands of internet subscribers in India. We're a small tech company with 25-30 people or so, with linux kernel experts who have experience of running software on questionable hardware so that service providers can take internet to ne…

What do you make out of curiosity? I run an ISP in the UK.

Awesome!

We have SaaS offering for provisioning, subscriber management (AAA, Radius), billing, OSS, CRM, customer portals (including mobile apps) and more!

Please let me know if you'd like to know more, also I would really love to know what software you use to manage your users, etc. and what WISP software is popular in the UK. Thanks!

Re: Status.github.com: “We're failing over a data storage system”

#167

Post by ksec »

r_singh wrote: post #91

Just a few weeks ago my organization was in the position of choosing a version control platform for our repos. I'm so glad we went ahead with self hosted gitlab. We installed it on a CentOS server at our premise, SSL'd via Let's encrypt, I've even set up a dedicated gitlab runner to use Gitlab CI for continuous delivery and so far the testing is progressing pretty smoothly. All this for $0. Update: I agree, gitlab an…

The Only reason for sticking with GitHub is UX. gitlab is expanding in all directions may be they will catch up.

Re: Status.github.com: “We're failing over a data storage system”

#168

Post by softawre »

vbezhenar wrote: post #134
Cthulhu_ wrote: post #101

Earlier quoted context omitted.

What did the server cost? How many hours did you spend on it? What's your SLA? Who will be woken up at night when there's an outage? What's your backup and recovery procedure? I mean kudos to you for setting it up but it's a bit naive to believe it's better than a hosted solution right away.

Git is tool for developers. If developer can't fix broken server, he's doing something wrong. No need to outsource trivial tasks.

Might as well have your developers do the support and the testing and answering the phones and cleaning the bathrooms too, eh?

Re: Status.github.com: “We're failing over a data storage system”

#169

Post by ryanmccullagh »

r_singh wrote: post #91

Just a few weeks ago my organization was in the position of choosing a version control platform for our repos. I'm so glad we went ahead with self hosted gitlab. We installed it on a CentOS server at our premise, SSL'd via Let's encrypt, I've even set up a dedicated gitlab runner to use Gitlab CI for continuous delivery and so far the testing is progressing pretty smoothly. All this for $0. Update: I agree, gitlab an…

It's definitely not zero cost. Your time and maintenance efforts will cost you time.

Re: Status.github.com: “We're failing over a data storage system”

#170

Post by beatgammit »

keyle wrote: post #20

I don't know if they keep changing the text updates with a slightly different version to - prove it's a human that typed it - there is code the prevents repeating twice the same message either way it's entertaining... But it's Monday morning in Australia and we need to release! (yep we do this via pr/tagging etc.)

I've seen some that are spaced at regular intervals with very subtle changes between messages, none of which actually say anything. I really like Github's messages since they seem like they're actually being updated by a human.
Post reply on HN