Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitLab Database Incident – Live Report

docs.google.com

Post reply on HN
Print view View original

21–30 of 621 posts

Re: GitLab Database Incident – Live Report

#21

Post by m_fam_wa_k »

eric_h wrote: post #14
ams6110 wrote: post #9

23:00-ish YP thinks that perhaps pg_basebackup is being super pedantic about there being an empty data directory, decides to remove the directory. After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com Good lesson on the risks of working on a live production system late at night when you're tired and/or frustrated.

Good lesson on making command prompts on machines always tell you exactly what machine you're working on.

I like to color code my terminal. Production systems are always red. Dev are blue/green. Staging is yellow.

Re: GitLab Database Incident – Live Report

#22

Post by bigiain »

YorickPeterse wrote: post #10
CameronBanga wrote: post #5

Earlier quoted context omitted.

I think it's a staff member. Can't remember first name, Yuri maybe, who is fairly active with the project.

Nope, that would be me.

Tough night dude. I'll buy you a drink or three if you're ever in Sydney...

Re: GitLab Database Incident – Live Report

#23

Post by Perihelion »

StavrosK wrote: post #13
Perihelion wrote: post #4

Earlier quoted context omitted.

Yes, those are the initials of an employee here. Sorry for the confusion!

Is your username a Spin reference?

Haha, it wasn't intentional. I'm just a space nerd. That book ranks pretty highly on my list of things every space nerd should read though.

Re: GitLab Database Incident – Live Report

#24

Post by StavrosK »

Perihelion wrote: post #23
StavrosK wrote: post #13

Earlier quoted context omitted.

Is your username a Spin reference?

Haha, it wasn't intentional. I'm just a space nerd. That book ranks pretty highly on my list of things every space nerd should read though.

I quite enjoyed it! Also, +1 for space. And Greek.

Re: GitLab Database Incident – Live Report

#26

Post by sbuttgereit »

ams6110 wrote: post #9

23:00-ish YP thinks that perhaps pg_basebackup is being super pedantic about there being an empty data directory, decides to remove the directory. After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com Good lesson on the risks of working on a live production system late at night when you're tired and/or frustrated.

Also a good lesson for testing your availability and disaster recovery measures for effectiveness.

Far, far too many companies get production going and then just check to see that certain things "completed successfully" or didn't throw an overt alert in terms of their safety nets.

Just because things seem to be working doesn't mean they are or that they are working in a way that is recoverable.

Re: GitLab Database Incident – Live Report

#28

Post by joshaidan »

m_fam_wa_k wrote: post #21
eric_h wrote: post #14

Earlier quoted context omitted.

Good lesson on making command prompts on machines always tell you exactly what machine you're working on.

I like to color code my terminal. Production systems are always red. Dev are blue/green. Staging is yellow.

How do you go about colour coding your terminal?

Re: GitLab Database Incident – Live Report

#29

Post by TimWolla »

m_fam_wa_k wrote: post #21
eric_h wrote: post #14

Earlier quoted context omitted.

Good lesson on making command prompts on machines always tell you exactly what machine you're working on.

I like to color code my terminal. Production systems are always red. Dev are blue/green. Staging is yellow.

In this case it looks like it has been a confusion between two different replicated Production databases. So this would not have helped.
Post reply on HN