Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitLab Database Incident – Live Report

docs.google.com

Post reply on HN
Print view View original

51–60 of 621 posts

Re: GitLab Database Incident – Live Report

#53

Post by tetrep »

Walkman wrote: post #25

Seems like very basic mistakes were made, not at the event but way long before. If you don't test to restore your backups, you don't have a backup. How does it go unnoticed that S3 backups don't work for so long?

Yeah, the "You don't have backups unless you can restore them" stikes again.

Virtually the only way to lose data is to not have backups. We live in such fancy times that there's no reason to ever lose data that you care about.

Re: GitLab Database Incident – Live Report

#54

Post by mschuster91 »

Walkman wrote: post #25

Seems like very basic mistakes were made, not at the event but way long before. If you don't test to restore your backups, you don't have a backup. How does it go unnoticed that S3 backups don't work for so long?

> How does it go unnoticed that S3 backups don't work for so long?

My uneducated guess (this one hit a friend of mine): expired/revoked AWS credentials combined with a backup script that doesn't exit(1) on failure and just writes the exception trace to stderr.

Re: GitLab Database Incident – Live Report

#55

Post by sytse »

sbuttgereit wrote: post #8
CameronBanga wrote: post #2

Not sure if the doc here is refreshing or scary. But Godspeed GitLab team. I've loved the product for about two years now, so curious to see how this plays out.

It's both. I very much appreciate their forthrightness and the way they conduct their company generally. Having said that, I have the code I work on, related content, and a number of clients on the service. [edit for additional point] They need the infrastructure guy they've been looking for sooner than later. I hope there's good progress on that front.

We've hired some great new people recently but as you can see there is still a lot work to do. https://about.gitlab.com/jobs/production-engineer/

Re: GitLab Database Incident – Live Report

#56

Post by gnufied »

echelon wrote: post #44
eric_h wrote: post #14

Earlier quoted context omitted.

Good lesson on making command prompts on machines always tell you exactly what machine you're working on.

This doesn't really help if there are multiple production databases. It could be sharded, replicated, multi-tenant, etc.

Why would it matter? In my last job we had user home directories synced via puppet (I am overly simplifying this) which enabled any ops guy to have same set of shell and vim configuration settings on production machines too.

I daresay - having hostname as part of prompt saves lot of trouble.

Re: GitLab Database Incident – Live Report

#57

Post by connorshea »

gizmo wrote: post #19

This is painful to read. It's easy to say that they they should have tested their backups better, and so on, but there is another lesson here, one that's far more important and easily missed. When doing something really critical (such as playing with the master database late at night) ALWAYS work with a checklist. Write down WHAT you are going to do, and if possible, talk to a coworker about it so you can vocalize th…

Yup, it's never the fault of a person, always of the system. Once we get this resolved we'll definitely look at ways to prevent anything like it in the future.

Re: GitLab Database Incident – Live Report

#60

Post by kofejnik »

Perihelion wrote: post #37
m_fam_wa_k wrote: post #21

Earlier quoted context omitted.

I like to color code my terminal. Production systems are always red. Dev are blue/green. Staging is yellow.

All of my non-production machines have emojis in PS1 somewhere. It sounds ridiculous, but I know that if I see a cheeseburger or a burrito I'm not about to completely mess everything up. Silly terminal = silly data that I can obliterate.

I've been color-coding my PS1 for years, but this is seriously brilliant, thanks!
Post reply on HN