Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data Loss at GitLab

blog.2ndquadrant.com

Post reply on HN
Print view View original

11–20 of 236 posts

Re: Data Loss at GitLab

#11

Post by aexaey »

hyperpape wrote: post #8

I'm surprised by the statement that 4 gigs of replication lag is normal. However, I don't manage backups for anything larger than personal pet projects, so I don't have a sense of scale.

> don't have a sense of scale

From [1], complete db is ~300GB and from some iffy pixel measurement of the graph at the very bottom of that page, copying speed between otherwise idle db hosts was about 22.8 GB/hour (in-production replication is probably slower than that).

From that, 4GB of replication lag would represent 1.3% of db by size, or 10+ minutes of lag (as measured by time required to catch up under ideal circumstances).

[1] https://about.gitlab.com/2017/02/01/gitlab-dot-com-database-...

Re: Data Loss at GitLab

#12

Post by stock_toaster »

aexaey wrote: post #11
hyperpape wrote: post #8

I'm surprised by the statement that 4 gigs of replication lag is normal. However, I don't manage backups for anything larger than personal pet projects, so I don't have a sense of scale.

> don't have a sense of scale From [1], complete db is ~300GB and from some iffy pixel measurement of the graph at the very bottom of that page, copying speed between otherwise idle db hosts was about 22.8 GB/hour (in-production replication is probably slower than that). From that, 4GB of replication lag would represent 1.3% of db by size, or 10+ minutes of lag (as measured by time required to catch up under ideal ci…

Thanks for the context, in my personal experience at jobs dealing with large databases, we always used "minutes behind" to determine how well our replication was keeping up. This was the first I had heard of someone using data size for that same metric.

Re: Data Loss at GitLab

#13

Post by jobvandervoort »

cornedor wrote: post #7
wheelerwj wrote: post #2

i wish everything was discussed/handled as publicly and transparently as this whole scenario. i really hope this becomes a thing.

I totally agree, the live stream [1] is amazing, discussing steps in the open like that and if there is some time left answer questions from the YouTube chat. [1] https://www.youtube.com/c/Gitlab/live

Glad to hear this was appreciated. It was an experiment -one we only hope to repeat in other scenarios.

Re: Data Loss at GitLab

#14

Post by skywhopper »

sulam wrote: post #10
neals wrote: post #6

I don't get to work on databases this size and today has been an incredible lesson and a journey. I've been reading all the comments and blogs, watching the stream and Googling what I didn't know or understand. I feel like the next step for me is scaling my business so that we have an actual usage for my newly found interests :)

Then you end up having deep expertise in a topic that's only important for larger companies. Sometimes this works out well, sometimes it leaves you a little stuck. :)

Oh, there's plenty of demand for skilled DBAs with deep expertise with handling large volumes of data, even at small-to-medium-sized companies. GitLab being one of those companies! In fact, demand for this sort of thing is likely to increase. If it interests you, dig in. You'll never stop learning.

Re: Data Loss at GitLab

#16

Post by stusmall »

sulam wrote: post #10
neals wrote: post #6

I don't get to work on databases this size and today has been an incredible lesson and a journey. I've been reading all the comments and blogs, watching the stream and Googling what I didn't know or understand. I feel like the next step for me is scaling my business so that we have an actual usage for my newly found interests :)

Then you end up having deep expertise in a topic that's only important for larger companies. Sometimes this works out well, sometimes it leaves you a little stuck. :)

Meh. Just learn, learn, learn. If it isn't completely applicable to your life that's okay. Sometimes there are pearls of wisdom in best practices in field completely unrelated to your own. Learning something new is always a good thing.

Re: Data Loss at GitLab

#17

Post by hyperpape »

sulam wrote: post #10
neals wrote: post #6

I don't get to work on databases this size and today has been an incredible lesson and a journey. I've been reading all the comments and blogs, watching the stream and Googling what I didn't know or understand. I feel like the next step for me is scaling my business so that we have an actual usage for my newly found interests :)

Then you end up having deep expertise in a topic that's only important for larger companies. Sometimes this works out well, sometimes it leaves you a little stuck. :)

I'm not sure if it's really company size, though there's probably a correlation. My company is only marginally bigger than GitLab in terms of employees (just recently 200+), and we have several dozen databases bigger than this.

Re: Data Loss at GitLab

#18

Post by mandlar »

cornedor wrote: post #7
wheelerwj wrote: post #2

i wish everything was discussed/handled as publicly and transparently as this whole scenario. i really hope this becomes a thing.

I totally agree, the live stream [1] is amazing, discussing steps in the open like that and if there is some time left answer questions from the YouTube chat. [1] https://www.youtube.com/c/Gitlab/live

Is there an archive of the stream?

Re: Data Loss at GitLab

#19

Post by hyperpape »

aexaey wrote: post #11
hyperpape wrote: post #8

I'm surprised by the statement that 4 gigs of replication lag is normal. However, I don't manage backups for anything larger than personal pet projects, so I don't have a sense of scale.

> don't have a sense of scale From [1], complete db is ~300GB and from some iffy pixel measurement of the graph at the very bottom of that page, copying speed between otherwise idle db hosts was about 22.8 GB/hour (in-production replication is probably slower than that). From that, 4GB of replication lag would represent 1.3% of db by size, or 10+ minutes of lag (as measured by time required to catch up under ideal ci…

I didn't think to eyeball the graph to guesstimate how long the 4GB translated to, so thanks.

However, scale was the wrong word for what I was wondering about. My question should've been whether 1% of your total DB/10 minutes of replication lag seems reasonable/nothing to worry about, like the article suggested.

Re: Data Loss at GitLab

#20

Post by marricks »

From their blog post,

> So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place. We ended up restoring a 6 hours old backup.

That must be _terrifying_ to realize. I mean, thank goodness they had a 6 hour old back up or they'd be in such an awful spot.

Post reply on HN