Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Loss at GitLab

blog.2ndquadrant.com

Post reply on HN
Print view View original

71–80 of 236 posts

Re: Data Loss at GitLab

#71

Post by balls187 »

BHSPitMonkey wrote: post #67
balls187 wrote: post #59

Earlier quoted context omitted.

Why? Everyone has their own copy of the gitlab remote.

Not everyone has their own copy of the project's issues/pull requests/settings/webhooks/CI config/etc., which the incident affected (and 6 hours worth were lost). Git repositories themselves do have the happy side effect that at least someone on your team has a full latest clone on their machine, but that's only because the source code is what you're there to use every day. Even if GitLab had a great "checkout your i…

Right, there was data loss, and it was not trivial.

Does that mean you should only use gitlab for toy projects? I don't think so.

I think they'll quickly learn from this.

Re: Data Loss at GitLab

#72

Post by leesalminen »

testUser69 wrote: post #37
aamederen wrote: post #29

With this incident, they once again showed that they are dedicated to transparency, even in the worst days. This increased their popularity on me and I believe among other developers. However, this may not be the case with the business people. I hope they can survive that and also publish a guide for getting better at the "ops" side of the things.

I think the "business people" are people who hide their technological inadequacy behind supposed business expertise. A businesses success is not determined by whether or not they use tools that adhere to business micro-cultural "values". If "too much transparency" is a turn off for you, you're probably just an authoritarian trying to scheme and scam your way into profit, and you probably lack the confidence required…

At my company we always heir on the side of transparency and liberally post to our status. subdomain whenever an issue is identified.

Recently, our Redis cluster failed and both master & slave host machines rebooted.

This caused a latency spike in our app, from roughly a 70ms to 400ms response time for less than 10 minutes.

We posted to status within 60 seconds and posted 3 updates within those 10 minutes.

The next day, a new customer (who hadn't gone live with the app yet) cancelled their subscription because the app was "not reliable".

I guess my point is that there is a balance to strike. Our customers are not tech-savvy in any way and treat any small issue as the end of the world. Maybe there's no need to freak people out for a minutes-long latency spike.

Re: Data Loss at GitLab

#73

Post by YorickPeterse »

neals wrote: post #6

I don't get to work on databases this size and today has been an incredible lesson and a journey. I've been reading all the comments and blogs, watching the stream and Googling what I didn't know or understand. I feel like the next step for me is scaling my business so that we have an actual usage for my newly found interests :)

It's both fun and incredibly frustrating at times.

Re: Data Loss at GitLab

#74

Post by kakwa_ »

activatedgeek wrote: post #22

While this is disastrous, I still think Gitlab is the best thing happened to OSS. This could be taken as a rhetoric, but on a more actionable side, we must all learn from Gitlab's experience. Almost everybody experiences this issue, but very few come out clean.

Well, even if their transparency in failure is admirable, I'm not completely surprised.

Just running an apt-get install gitlab gives me around ~350 dependencies. In those dependencies, I see python, ruby, nodejs, redis, postgres.

With a little Java, and a little Go, plus some admin scripts written in Perl, the picture would be mostly complete...

I may be a little harsh, but when I see a piece of software with so much complexity in it that it requires 3 stacks, I'm not completely convinced it's a well conceived piece of software.

Having 5 backup systems, with none of them working properly kind of falls in the same category.

Re: Data Loss at GitLab

#75

Post by YorickPeterse »

tshannon wrote: post #53
intsunny wrote: post #30

The gitlab situation and Uber's article speak to the level of immaturity of PGSQL's native replication feature, and more importantly: how not widely google-able nor documented/adopted the replication strategies are.

I believe gitlab used slony, not the native replication. I'm not well versed in postgres, but that's what I gleaned from reading their event log.

As mentioned below we only used Slony to upgrade from 9.2.something to 9.6.1. For regular replication we use PostgreSQL's streaming replication.

Re: Data Loss at GitLab

#76

Post by jasonm23 »

leesalminen wrote: post #72
testUser69 wrote: post #37

Earlier quoted context omitted.

I think the "business people" are people who hide their technological inadequacy behind supposed business expertise. A businesses success is not determined by whether or not they use tools that adhere to business micro-cultural "values". If "too much transparency" is a turn off for you, you're probably just an authoritarian trying to scheme and scam your way into profit, and you probably lack the confidence required…

At my company we always heir on the side of transparency and liberally post to our status. subdomain whenever an issue is identified. Recently, our Redis cluster failed and both master & slave host machines rebooted. This caused a latency spike in our app, from roughly a 70ms to 400ms response time for less than 10 minutes. We posted to status within 60 seconds and posted 3 updates within those 10 minutes. The next d…

FYI

https://en.oxforddictionaries.com/definition/err_on_the_side...

Just in case you ever use that phrase in critical correspondence. Better to err on the side of caution.

Re: Data Loss at GitLab

#77

Post by dman »

kakwa_ wrote: post #74
activatedgeek wrote: post #22

While this is disastrous, I still think Gitlab is the best thing happened to OSS. This could be taken as a rhetoric, but on a more actionable side, we must all learn from Gitlab's experience. Almost everybody experiences this issue, but very few come out clean.

Well, even if their transparency in failure is admirable, I'm not completely surprised. Just running an apt-get install gitlab gives me around ~350 dependencies. In those dependencies, I see python, ruby, nodejs, redis, postgres. With a little Java, and a little Go, plus some admin scripts written in Perl, the picture would be mostly complete... I may be a little harsh, but when I see a piece of software with so much…

I would be surprised if most startups arent doing something similar behind the scenes.

Re: Data Loss at GitLab

#78

Post by nicpottier »

drvdevd wrote: post #64
nicpottier wrote: post #58

Earlier quoted context omitted.

I think you are confused. The best thing to happen to OSS is GitHub, not GitLab. GitLab is just a fast follower and likely wouldn't even exist without the former. I for one am happy to throw money GitHub's way for their role in so dramatically changing how we code.

I pay for GitHub and can't deny their positive role in OSS, but I also use GitLab and will happily continue (and love that they have an OSS edition). I don't think it's a binary choice. Personally I will often adopt multiple pieces of software with redundant features because nuance often dictates the right tool for the job, and I enjoy being prepared for that.

Absolutely, but the parent was saying GitLab changed OSS, and though that may be true to some extent, it is a pittance compared to the imapct GitHub has made.

Re: Data Loss at GitLab

#79

Post by nicpottier »

flukus wrote: post #68
nicpottier wrote: post #58

Earlier quoted context omitted.

I think you are confused. The best thing to happen to OSS is GitHub, not GitLab. GitLab is just a fast follower and likely wouldn't even exist without the former. I for one am happy to throw money GitHub's way for their role in so dramatically changing how we code.

Github was just a follower with a new SCM. The original would be source forge.

SourceForge pretty much failed where GitHub strived though. For all practical purposes they were just a hosted CVS/SVN and nothing more. GitHub really nailed the idea of community and that made all the difference. Git is incidental, though lightweight branching was probably an important contributor to their approach.

Re: Data Loss at GitLab

#80

Post by neurotech1 »

FlorianRappl wrote: post #28
activatedgeek wrote: post #22

While this is disastrous, I still think Gitlab is the best thing happened to OSS. This could be taken as a rhetoric, but on a more actionable side, we must all learn from Gitlab's experience. Almost everybody experiences this issue, but very few come out clean.

Couldn't agree more. Also transparency is key here. Everyone makes mistakes, but being open about it does not only let customers (and employees!) know what's going on, but also gives other professionals the ability to learn from it.

There are two types of web application providers. Those who have already had database mishaps causing data loss, and those that WILL. No matter how good, or our how large and 'professional' the company is, mistakes can happen.

Being open about it inspires confidence that they will improve in the future, instead of being quiet and having repeat incidents.

Post reply on HN