Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Loss at GitLab

blog.2ndquadrant.com

Post reply on HN
Print view View original

51–60 of 236 posts

Re: Data Loss at GitLab

#51

Post by djsumdog »

Vinnl wrote: post #46
cptskippy wrote: post #23

Earlier quoted context omitted.

I would counter that they're still in an awful spot because this announcement reeks of incompetence and isn't something you want to hear from the guys you're entrusting with keeping your code safe. It would be like Boeing or Airbus announcing all the safety features on their airliners were non functioning.

> the guys you're entrusting with keeping your code safe. To add to the other replies: I'm not trusting them with keeping my code safe (everybody has copies on their own computers). I'm trusting them with facilitating my workflow, helping me collaborate, and also to keep my issues etc. safe. So yes, this can have a relatively large impact in terms of not being able to work as efficiently for a day and potentially los…

Exactly, that's the whole point of git. Your entire code base is distributed (or at least everything you push out to a remote). Unlike CVS/SVN, you have that whole copy and collective history of everything that was pushed in that repo.

If you want to move it, in most cases you can simply create a new bare repo on another remote and push yours to it. It's probably the easiest system out there to simply pick up your data and go, unlike the walled gardens of social networking, video (YouTube isn't video, it's video + annotations + another ecosystem of tools that's not easy to export), and other services.

Even blog software isn't as resilient. You still have to export your Wordpress or Ghost blog when you want to move it. With git, when you work on it, you already have a full copy (with a few exceptions of course, like remote branches people prune without merging or local branches people never push).

Re: Data Loss at GitLab

#52

Post by tyre »

testUser69 wrote: post #37
aamederen wrote: post #29

With this incident, they once again showed that they are dedicated to transparency, even in the worst days. This increased their popularity on me and I believe among other developers. However, this may not be the case with the business people. I hope they can survive that and also publish a guide for getting better at the "ops" side of the things.

I think the "business people" are people who hide their technological inadequacy behind supposed business expertise. A businesses success is not determined by whether or not they use tools that adhere to business micro-cultural "values". If "too much transparency" is a turn off for you, you're probably just an authoritarian trying to scheme and scam your way into profit, and you probably lack the confidence required…

This demonstrates a remarkable lack of empathy. Try understanding why people on the business side do what they do—it's almost never out of malice or desired authoritarianism.

Working with people is hard, specifically because you have to get outside of your head and think about how they see the world.

Re: Data Loss at GitLab

#53

Post by tshannon »

intsunny wrote: post #30

The gitlab situation and Uber's article speak to the level of immaturity of PGSQL's native replication feature, and more importantly: how not widely google-able nor documented/adopted the replication strategies are.

I believe gitlab used slony, not the native replication. I'm not well versed in postgres, but that's what I gleaned from reading their event log.

Re: Data Loss at GitLab

#54

Post by gukov »

cptskippy wrote: post #23
marricks wrote: post #20

From their blog post, > So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place. We ended up restoring a 6 hours old backup. That must be _terrifying_ to realize. I mean, thank goodness they had a 6 hour old back up or they'd be in such an awful spot.

I would counter that they're still in an awful spot because this announcement reeks of incompetence and isn't something you want to hear from the guys you're entrusting with keeping your code safe. It would be like Boeing or Airbus announcing all the safety features on their airliners were non functioning.

They are in an awful spot.

It can be argued that once a company messes up this bad it will make sure nothing similar happens ever again. However, it can also be argued that if a tech company has all five of its back up procedures fail it's borderline criminally negligent.

Re: Data Loss at GitLab

#55

Post by iurisilvio »

activatedgeek wrote: post #22

While this is disastrous, I still think Gitlab is the best thing happened to OSS. This could be taken as a rhetoric, but on a more actionable side, we must all learn from Gitlab's experience. Almost everybody experiences this issue, but very few come out clean.

Exactly. today we're able to see a bad disaster recovery happening, but it happens all the time. I revised my backups and added some redundancy to it.

Everyone knows how much backups are important and talk about it all the time, but I bet a lot of companies don't do it right. It is expensive, don't add real value (except when it does), etc.

Today I added automated replication of backups from AWS to another cloud provider. Just in case...

I did this becauase recently, a local brazilian cloud provider (ServerLoft) didn't paid his server (Equinix) and went offline forever. 16k companies went offline without time to recover anything there.

Re: Data Loss at GitLab

#56

Post by Klathmon »

iurisilvio wrote: post #55
activatedgeek wrote: post #22

While this is disastrous, I still think Gitlab is the best thing happened to OSS. This could be taken as a rhetoric, but on a more actionable side, we must all learn from Gitlab's experience. Almost everybody experiences this issue, but very few come out clean.

Exactly. today we're able to see a bad disaster recovery happening, but it happens all the time. I revised my backups and added some redundancy to it. Everyone knows how much backups are important and talk about it all the time, but I bet a lot of companies don't do it right. It is expensive, don't add real value (except when it does), etc. Today I added automated replication of backups from AWS to another cloud prov…

As terrible as it is for GitLab, it's doing some good here as well.

I'm using this as a great example of a reason to bump up some of the fixes to some of our backups and replication issues up the priority list. And it's much easier to sell to some of the "higher ups" when you can point at a concrete example of how badly a misstep here can hurt.

I'm floored with their honesty and openness, I can honestly say I wouldn't be able to put this out there like they have... But i'm really glad they are doing it, and I'm really happy at the outpouring of support they are getting for it from people like 2ndQuadrant.

Re: Data Loss at GitLab

#57

Post by mynameisvlad »

andyana wrote: post #48
mynameisvlad wrote: post #25

Earlier quoted context omitted.

I don't think those two scenarios are comparable. The safety features are in use on an airplane constantly. Backups are only needed when a disaster happens. It'd be more akin to B/A announcing that some emergency system like the air masks isn't functioning. Still incredibly troubling, especially if there was a cabin depressurization scenario, but not to the level of all safety features being broken.

Not all the safety features are in use all the time. Like, you don't use the inflatable slide unless your evacuating.

Sure, and I never said all of them are used all the time, but parent comment specifically did say "all the safety features". I wanted to make the distinction clear that safety features in planes can and are used actively during flights, whereas backups are only used when there's a catastrophe.

Re: Data Loss at GitLab

#58

Post by nicpottier »

activatedgeek wrote: post #22

While this is disastrous, I still think Gitlab is the best thing happened to OSS. This could be taken as a rhetoric, but on a more actionable side, we must all learn from Gitlab's experience. Almost everybody experiences this issue, but very few come out clean.

I think you are confused.

The best thing to happen to OSS is GitHub, not GitLab. GitLab is just a fast follower and likely wouldn't even exist without the former.

I for one am happy to throw money GitHub's way for their role in so dramatically changing how we code.

Re: Data Loss at GitLab

#59

Post by balls187 »

empath75 wrote: post #41
aamederen wrote: post #29

With this incident, they once again showed that they are dedicated to transparency, even in the worst days. This increased their popularity on me and I believe among other developers. However, this may not be the case with the business people. I hope they can survive that and also publish a guide for getting better at the "ops" side of the things.

Yeah it made me not consider using them for anything other than toy projects.

Why?

Everyone has their own copy of the gitlab remote.

Re: Data Loss at GitLab

#60

Post by jessaustin »

tshannon wrote: post #53
intsunny wrote: post #30

The gitlab situation and Uber's article speak to the level of immaturity of PGSQL's native replication feature, and more importantly: how not widely google-able nor documented/adopted the replication strategies are.

I believe gitlab used slony, not the native replication. I'm not well versed in postgres, but that's what I gleaned from reading their event log.

The decision not to use native replication might be attributed to its purported immaturity?
Post reply on HN