Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Loss at GitLab

blog.2ndquadrant.com

Post reply on HN
Print view View original

81–90 of 236 posts

Re: Data Loss at GitLab

#81

Post by leesalminen »

jasonm23 wrote: post #76
leesalminen wrote: post #72

Earlier quoted context omitted.

At my company we always heir on the side of transparency and liberally post to our status. subdomain whenever an issue is identified. Recently, our Redis cluster failed and both master & slave host machines rebooted. This caused a latency spike in our app, from roughly a 70ms to 400ms response time for less than 10 minutes. We posted to status within 60 seconds and posted 3 updates within those 10 minutes. The next d…

FYI https://en.oxforddictionaries.com/definition/err_on_the_side... Just in case you ever use that phrase in critical correspondence. Better to err on the side of caution.

Ah, thanks! It didn't feel right but was too lazy to check. Will definitely remember this for next time.

Re: Data Loss at GitLab

#82

Post by debaserab2 »

testUser69 wrote: post #37
aamederen wrote: post #29

With this incident, they once again showed that they are dedicated to transparency, even in the worst days. This increased their popularity on me and I believe among other developers. However, this may not be the case with the business people. I hope they can survive that and also publish a guide for getting better at the "ops" side of the things.

I think the "business people" are people who hide their technological inadequacy behind supposed business expertise. A businesses success is not determined by whether or not they use tools that adhere to business micro-cultural "values". If "too much transparency" is a turn off for you, you're probably just an authoritarian trying to scheme and scam your way into profit, and you probably lack the confidence required…

It really wouldn't have been that hard to write your opinion in a way that doesn't disparage the character of the person (that you don't really know) that you're responding to.

There's no reason to hide behind a new account other than to be an asshole.

Re: Data Loss at GitLab

#83

Post by dkural »

nicpottier wrote: post #58
activatedgeek wrote: post #22

While this is disastrous, I still think Gitlab is the best thing happened to OSS. This could be taken as a rhetoric, but on a more actionable side, we must all learn from Gitlab's experience. Almost everybody experiences this issue, but very few come out clean.

I think you are confused. The best thing to happen to OSS is GitHub, not GitLab. GitLab is just a fast follower and likely wouldn't even exist without the former. I for one am happy to throw money GitHub's way for their role in so dramatically changing how we code.

Github itself is not open source. The irony.

Re: Data Loss at GitLab

#84

Post by grzm »

dkural wrote: post #83
nicpottier wrote: post #58

Earlier quoted context omitted.

I think you are confused. The best thing to happen to OSS is GitHub, not GitLab. GitLab is just a fast follower and likely wouldn't even exist without the former. I for one am happy to throw money GitHub's way for their role in so dramatically changing how we code.

Github itself is not open source. The irony.

There are plenty of useful closed source services and tools that can be used to support open source projects, or just software development in general. Indeed, they facilitate open source projects with an easy to use service. Is there anything they're doing that prevents or blocks open source development? They do have open source projects that they've made available.

Re: Data Loss at GitLab

#85

Post by sgs1370 »

bfrog wrote: post #63

Testing backups or at least monitoring them for correctness is a huge deal and it's problem I myself fudged up on one occasion, right around the holiday season which was terrible. I've since setup wal-e with daily base backups deleting things older than a week along with the nightly pg_dump's along with a hot stanby. Maybe thats overkill, but after having lost data once. Never again! The nice part about doing wal arc…

One thing I could never figure out, and it is probably because I didn't study the docs enough, is when I could delete the archive files. What is the best reference on this. OR, maybe I was trying to reinvent the wheel, I didn't really look at 3rd party libs - only postgresql options (plus my glue on top) - is the only right way to use the archive (file shipping) solution to combine it with a 3rd party?

Re: Data Loss at GitLab

#86

Post by beefsack »

balls187 wrote: post #59
empath75 wrote: post #41

Earlier quoted context omitted.

Yeah it made me not consider using them for anything other than toy projects.

Why? Everyone has their own copy of the gitlab remote.

We use gitlab.com for code reviews and CI, and we basically lost a day of integration productivity because of it.

Yes, our process is too tightly tied to a single service, but it happens because we don't have the resources to self-host our own solution. We love GitLab, but this has absolutely got us looking elsewhere.

Re: Data Loss at GitLab

#87

Post by freehunter »

kakwa_ wrote: post #74
activatedgeek wrote: post #22

While this is disastrous, I still think Gitlab is the best thing happened to OSS. This could be taken as a rhetoric, but on a more actionable side, we must all learn from Gitlab's experience. Almost everybody experiences this issue, but very few come out clean.

Well, even if their transparency in failure is admirable, I'm not completely surprised. Just running an apt-get install gitlab gives me around ~350 dependencies. In those dependencies, I see python, ruby, nodejs, redis, postgres. With a little Java, and a little Go, plus some admin scripts written in Perl, the picture would be mostly complete... I may be a little harsh, but when I see a piece of software with so much…

That's what I thought when I read the report earlier today. A system with that much complexity is doomed to failure eventually.

I can understand "right tool for the job" but at some point it should all come together. An MVP can be hacked together from bits and bobs, but when it becomes a business it should be refactored to reduce complexity where possible.

Trying to maintain five seperate backup solutions, let alone trying to restore from all five of them, sounds like my worst nightmare. Trying to restore from one backup is often hard enough by itself.

Re: Data Loss at GitLab

#88

Post by jasonwatkinspdx »

Thaxll wrote: post #66
nicpottier wrote: post #58

Earlier quoted context omitted.

I think you are confused. The best thing to happen to OSS is GitHub, not GitLab. GitLab is just a fast follower and likely wouldn't even exist without the former. I for one am happy to throw money GitHub's way for their role in so dramatically changing how we code.

You code differently using Git vs SVN? That's an interesting concept. It doesn't change the way you code it just changes the way we share code.

Git makes multi-tasking on multiple branches far easier than SVN. In the SVN days I always had multiple checkouts and a pile of scripts for changing/updating to branches for handling a few regular tasks in that context. Once git won I was able to delete them and have a far simpler workflow.

Re: Data Loss at GitLab

#89

Post by synicalx »

aamederen wrote: post #29

With this incident, they once again showed that they are dedicated to transparency, even in the worst days. This increased their popularity on me and I believe among other developers. However, this may not be the case with the business people. I hope they can survive that and also publish a guide for getting better at the "ops" side of the things.

Getting better at ops? Checking backups regularly to make sure they work is not a new thing.

Re: Data Loss at GitLab

#90

Post by flukus »

nicpottier wrote: post #79
flukus wrote: post #68

Earlier quoted context omitted.

Github was just a follower with a new SCM. The original would be source forge.

SourceForge pretty much failed where GitHub strived though. For all practical purposes they were just a hosted CVS/SVN and nothing more. GitHub really nailed the idea of community and that made all the difference. Git is incidental, though lightweight branching was probably an important contributor to their approach.

I know the execution was a lot better with github, but what did it really add that source forge, google code, etc didn't do? They all had bug trackers, mailing lists, etc.
Post reply on HN