Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Loss at GitLab

blog.2ndquadrant.com

Post reply on HN
Print view View original

21–30 of 236 posts

Re: Data Loss at GitLab

#21

Post by johannes1234321 »

stock_toaster wrote: post #12
aexaey wrote: post #11

Earlier quoted context omitted.

> don't have a sense of scale From [1], complete db is ~300GB and from some iffy pixel measurement of the graph at the very bottom of that page, copying speed between otherwise idle db hosts was about 22.8 GB/hour (in-production replication is probably slower than that). From that, 4GB of replication lag would represent 1.3% of db by size, or 10+ minutes of lag (as measured by time required to catch up under ideal ci…

Thanks for the context, in my personal experience at jobs dealing with large databases, we always used "minutes behind" to determine how well our replication was keeping up. This was the first I had heard of someone using data size for that same metric.

Both have different uses. Seconds behind is useful to see how critical this is to the user (they see outdated data), data behind tells you about load and let's you guess how much time is required for catching up

Re: Data Loss at GitLab

#22

Post by activatedgeek »

While this is disastrous, I still think Gitlab is the best thing happened to OSS. This could be taken as a rhetoric, but on a more actionable side, we must all learn from Gitlab's experience. Almost everybody experiences this issue, but very few come out clean.

Re: Data Loss at GitLab

#23

Post by cptskippy »

marricks wrote: post #20

From their blog post, > So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place. We ended up restoring a 6 hours old backup. That must be _terrifying_ to realize. I mean, thank goodness they had a 6 hour old back up or they'd be in such an awful spot.

I would counter that they're still in an awful spot because this announcement reeks of incompetence and isn't something you want to hear from the guys you're entrusting with keeping your code safe.

It would be like Boeing or Airbus announcing all the safety features on their airliners were non functioning.

Re: Data Loss at GitLab

#24

Post by anw »

neals wrote: post #6

I don't get to work on databases this size and today has been an incredible lesson and a journey. I've been reading all the comments and blogs, watching the stream and Googling what I didn't know or understand. I feel like the next step for me is scaling my business so that we have an actual usage for my newly found interests :)

For those wondering about the live Gitlab stream, you can watch them work here:

https://www.youtube.com/watch?v=nc0hPGerSd4

Re: Data Loss at GitLab

#25

Post by mynameisvlad »

cptskippy wrote: post #23
marricks wrote: post #20

From their blog post, > So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place. We ended up restoring a 6 hours old backup. That must be _terrifying_ to realize. I mean, thank goodness they had a 6 hour old back up or they'd be in such an awful spot.

I would counter that they're still in an awful spot because this announcement reeks of incompetence and isn't something you want to hear from the guys you're entrusting with keeping your code safe. It would be like Boeing or Airbus announcing all the safety features on their airliners were non functioning.

I don't think those two scenarios are comparable. The safety features are in use on an airplane constantly. Backups are only needed when a disaster happens. It'd be more akin to B/A announcing that some emergency system like the air masks isn't functioning. Still incredibly troubling, especially if there was a cabin depressurization scenario, but not to the level of all safety features being broken.

Re: Data Loss at GitLab

#26

Post by jobvandervoort »

mandlar wrote: post #18
cornedor wrote: post #7

Earlier quoted context omitted.

I totally agree, the live stream [1] is amazing, discussing steps in the open like that and if there is some time left answer questions from the YouTube chat. [1] https://www.youtube.com/c/Gitlab/live

Is there an archive of the stream?

Part of the stream is here: https://www.youtube.com/watch?v=nc0hPGerSd4

Not sure if we'll be able to get the full 8+ hours up.

Re: Data Loss at GitLab

#27

Post by joking »

hyperpape wrote: post #19
aexaey wrote: post #11

Earlier quoted context omitted.

> don't have a sense of scale From [1], complete db is ~300GB and from some iffy pixel measurement of the graph at the very bottom of that page, copying speed between otherwise idle db hosts was about 22.8 GB/hour (in-production replication is probably slower than that). From that, 4GB of replication lag would represent 1.3% of db by size, or 10+ minutes of lag (as measured by time required to catch up under ideal ci…

I didn't think to eyeball the graph to guesstimate how long the 4GB translated to, so thanks. However, scale was the wrong word for what I was wondering about. My question should've been whether 1% of your total DB/10 minutes of replication lag seems reasonable/nothing to worry about, like the article suggested.

It was an issue, part of the reason that a tired person was working trying to reduce it.

Re: Data Loss at GitLab

#28

Post by FlorianRappl »

activatedgeek wrote: post #22

While this is disastrous, I still think Gitlab is the best thing happened to OSS. This could be taken as a rhetoric, but on a more actionable side, we must all learn from Gitlab's experience. Almost everybody experiences this issue, but very few come out clean.

Couldn't agree more. Also transparency is key here. Everyone makes mistakes, but being open about it does not only let customers (and employees!) know what's going on, but also gives other professionals the ability to learn from it.

Re: Data Loss at GitLab

#29

Post by aamederen »

With this incident, they once again showed that they are dedicated to transparency, even in the worst days. This increased their popularity on me and I believe among other developers. However, this may not be the case with the business people. I hope they can survive that and also publish a guide for getting better at the "ops" side of the things.

Re: Data Loss at GitLab

#30

Post by intsunny »

The gitlab situation and Uber's article speak to the level of immaturity of PGSQL's native replication feature, and more importantly: how not widely google-able nor documented/adopted the replication strategies are.
Post reply on HN