Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Loss at GitLab

blog.2ndquadrant.com

Post reply on HN
Print view View original

131–140 of 236 posts

Re: Data Loss at GitLab

#131

Post by rimantas »

kakwa_ wrote: post #74
activatedgeek wrote: post #22

While this is disastrous, I still think Gitlab is the best thing happened to OSS. This could be taken as a rhetoric, but on a more actionable side, we must all learn from Gitlab's experience. Almost everybody experiences this issue, but very few come out clean.

Well, even if their transparency in failure is admirable, I'm not completely surprised. Just running an apt-get install gitlab gives me around ~350 dependencies. In those dependencies, I see python, ruby, nodejs, redis, postgres. With a little Java, and a little Go, plus some admin scripts written in Perl, the picture would be mostly complete... I may be a little harsh, but when I see a piece of software with so much…

There not many project these days that would not require similar stacks. At least one core language: be it Ruby, or Java or whatever. Then you will have a DB. Then you will have some fancy frontend which will sure depend on node js. Then you might have some KV store/caching so you get Redis.

Re: Data Loss at GitLab

#132

Post by tigershark »

activatedgeek wrote: post #22

While this is disastrous, I still think Gitlab is the best thing happened to OSS. This could be taken as a rhetoric, but on a more actionable side, we must all learn from Gitlab's experience. Almost everybody experiences this issue, but very few come out clean.

seriously? almost everybody has FIVE different backups system and not one of them works? I don't know where you work, but in my job monitoring everything is essential. And a couple of times per year we do a disaster recovery test. Something like this is completely and totally unacceptable. It means that they never tested any backup.

Re: Data Loss at GitLab

#133

Post by manquer »

nicpottier wrote: post #78
drvdevd wrote: post #64

Earlier quoted context omitted.

I pay for GitHub and can't deny their positive role in OSS, but I also use GitLab and will happily continue (and love that they have an OSS edition). I don't think it's a binary choice. Personally I will often adopt multiple pieces of software with redundant features because nuance often dictates the right tool for the job, and I enjoy being prepared for that.

Absolutely, but the parent was saying GitLab changed OSS, and though that may be true to some extent, it is a pittance compared to the imapct GitHub has made.

git itself is the real force behind the OSS impact. Github model while successful was merely replacing sourceforge and google code. People always had ways to host and share open source projects.

Github enables centralization of code which is not really inline with philosophy of git being fully distributed . Gitlab with its OSS model and the ability to host your own git service makes is bit closer to that vision than Github is.

Re: Data Loss at GitLab

#135

Post by tigershark »

rimantas wrote: post #131
kakwa_ wrote: post #74

Earlier quoted context omitted.

Well, even if their transparency in failure is admirable, I'm not completely surprised. Just running an apt-get install gitlab gives me around ~350 dependencies. In those dependencies, I see python, ruby, nodejs, redis, postgres. With a little Java, and a little Go, plus some admin scripts written in Perl, the picture would be mostly complete... I may be a little harsh, but when I see a piece of software with so much…

There not many project these days that would not require similar stacks. At least one core language: be it Ruby, or Java or whatever. Then you will have a DB. Then you will have some fancy frontend which will sure depend on node js. Then you might have some KV store/caching so you get Redis.

I completely disagree. Normally you choose ONE core language and you use only that in a monolith application. I can admit different languages for the unit tests, like groovy on java side or F# on .net. Or using two languages on the same platform, like Scala and Java on the JVM and C# + F# on .net. Or one language server side with the web side written in javascript or in something else. If you are not breaking your application into micro services using several languages is just a very very bad smell. And in this case, apparently, they are using python, ruby, javascript, java, go and perl. I would say that from my point of view this is unacceptable and it is just a symptom of scarce planning, as we have seen in this incident where apparently nobody thought to plan for a backup test.

Re: Data Loss at GitLab

#136

Post by Kiro »

aamederen wrote: post #29

With this incident, they once again showed that they are dedicated to transparency, even in the worst days. This increased their popularity on me and I believe among other developers. However, this may not be the case with the business people. I hope they can survive that and also publish a guide for getting better at the "ops" side of the things.

> I believe among other developers

I don't know. I'm very hesitant trying out GitLab now whereas I was interested before.

Re: Data Loss at GitLab

#137

Post by developer2 »

synicalx wrote: post #104

Honestly, I'm completely flabbergasted by this. Five backups, and NONE worked properly? Who made this? The S3 bucket was EMPTY? Has no one ever tested any of these backups? It's not just the impact, which is fairly sizeable in it's own right, but it's the HUGE oversight on their part and the fact they tried to pin part of this on PostreSQL? Credit where it's due; their report/transparency were good if a little unprof…

I agree. Their catastrophe is not the kind of thing that represents mild hiccups in operations that will be quickly resolved by the same employees who allowed this scenario to occur the first time around. This isn't the result of a single oversight. It screams of a systemic problem with the way the business operates - period. Maybe every project is rushed out with the deadline being the only metric that counts, quality be damned. Or maybe they're missing talent in areas that matter - like experienced postgres DBAs managing the database rather than having the developers and/or someone who has only ever used MySQL trying to wrangle it.

What they went through is what you'd expect from a first-time pet project, not a professional business out to make money. I suspect they prioritized trying to scale with enough features to catch up with the competition above all else, to the point where sustainability is an afterthought. When desperately trying to gain market share is more important than the quality of the product, this is what you can expect.

Re: Data Loss at GitLab

#138

Post by marklgr »

praneshp wrote: post #93
aamederen wrote: post #29

With this incident, they once again showed that they are dedicated to transparency, even in the worst days. This increased their popularity on me and I believe among other developers. However, this may not be the case with the business people. I hope they can survive that and also publish a guide for getting better at the "ops" side of the things.

HN can be funny sometimes. GitHub got a lot of hate about a year ago just for not releasing new features. GitLab cost everyone a day because their backup/ops practices were silly, and everyone loves them more. I've screwed up before, and I sympathize/empathize with their ops folks, but this should make us think about plan B in case something like this happens again.

That's Internet culture for you: being lovable trumps pretty much anything else.

Re: Data Loss at GitLab

#139

Post by jjirsa »

pzh wrote: post #125

Does nobody else find the report cringeworthy? Apparently, there are some junior engineers fumbling around and committing serious errors, but where are the senior ones and the process/failsafes to prevent all this?

Yes. It makes me angry. It's not that backups failed, it's more like broad incompetence at all the wrong places.

Re: Data Loss at GitLab

#140

Post by jjirsa »

neals wrote: post #6

I don't get to work on databases this size and today has been an incredible lesson and a journey. I've been reading all the comments and blogs, watching the stream and Googling what I didn't know or understand. I feel like the next step for me is scaling my business so that we have an actual usage for my newly found interests :)

Running databases at scale is great - it's what I do for a living, and I wouldn't want to do anything else

But please realize that this is a horrible example. Almost everything done was wrong, technical choices, processes, everything - please don't use it as a positive example.

Post reply on HN