GitLab Database Incident – Live Report • Hacker News Board

Re: GitLab Database Incident – Live Report

#91

Post by gadders » Wed, Feb 01, 2017, 1:50 AM UTC

YorickPeterse wrote: post #10

CameronBanga wrote: post #5

Earlier quoted context omitted.

I think it's a staff member. Can't remember first name, Yuri maybe, who is fairly active with the project.

Nope, that would be me.

Unlucky mate. Even monkeys fall out of trees. Good luck with the fixing.

Re: GitLab Database Incident – Live Report

#92

Post by Washuu » Wed, Feb 01, 2017, 1:51 AM UTC

ckdarby wrote: post #78

"Our backups to S3 apparently don’t work either: the bucket is empty" 6/6 failed backup procedures. Looks like they are going to be hiring a new sysadmin/devops person...

The best system administrator is the one that has learned from their catastrophic fuck up.

To that effect, I still have the same job as I did before I ran "yum update" without knowing it attempts to do in place kernel upgrades. Which resulted in a corrupted RedHat installation on a server we could not turn off.

Re: GitLab Database Incident – Live Report

#93

Post by leesalminen » Wed, Feb 01, 2017, 1:52 AM UTC

overcast wrote: post #87

leesalminen wrote: post #64

Earlier quoted context omitted.

I am :/ ... currently maintaining service for 500 high-volume businesses 24x7x365 in 9 timezones. Luckily the product and infrastructure is pretty stable and problems occur maybe once a quarter. But the constant nagging in the back of your head that shit can go wrong at any second is draining and has been the biggest stressor in my life for a long time now. My S.O. still gets mildly upset when I pack up the laptop on…

I hope you are being SERIOUSLY compensated.

Cash salary today is well under market for my skill set. But, I do own 1/3 of the company so it's not all bad :).

Re: GitLab Database Incident – Live Report

#94

Post by kevindong » Wed, Feb 01, 2017, 1:53 AM UTC

m_fam_wa_k wrote: post #21

eric_h wrote: post #14

Earlier quoted context omitted.

Good lesson on making command prompts on machines always tell you exactly what machine you're working on.

I like to color code my terminal. Production systems are always red. Dev are blue/green. Staging is yellow.

It seems Gitlab has noticed your comment.

Recovery item 3f currently says:

> Create issue to change terminal PS1 format/colours to make it clear whether you’re using production or staging (red production, yellow staging)

Re: GitLab Database Incident – Live Report

#95

Post by jeffmcjunkin » Wed, Feb 01, 2017, 1:55 AM UTC

For item 3h under recovery, consider:

  chattr +i /var/opt/gitlab/postgresql/data

Yes, it doesn't completely stop foot-guns, but it means you have to shoot twice [0].

[0]:

  chattr -i /whatever
  rm /whatever

Re: GitLab Database Incident – Live Report

#96

Post by cesarb » Wed, Feb 01, 2017, 1:56 AM UTC

ams6110 wrote: post #9

23:00-ish YP thinks that perhaps pg_basebackup is being super pedantic about there being an empty data directory, decides to remove the directory. After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com Good lesson on the risks of working on a live production system late at night when you're tired and/or frustrated.

> Good lesson on the risks of working on a live production system late at night when you're tired and/or frustrated.

In these situations, I always keep the following xkcd in mind: https://xkcd.com/349/

Re: GitLab Database Incident – Live Report

#97

Post by Shelnutt2 » Wed, Feb 01, 2017, 1:56 AM UTC

TimWolla wrote: post #80

btgeekboy wrote: post #62

> Our backups to S3 apparently don’t work either: the bucket is empty followed by > So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place. is no way to be running a public service with customer data. Did the person who set up that S3 job simply write a script or something and just go "yep, it's done" and walk away? Seriously?

> is no way to be running a public service with paying customers Is it even possible to pay for the hosted GitLab.com instance?

Gitlab.com is free, not payed. https://about.gitlab.com/products/

The only for pay versions are self hosted EE and independent cloud hosting.

Re: GitLab Database Incident – Live Report

#98

Post by overcast » Wed, Feb 01, 2017, 1:57 AM UTC

leesalminen wrote: post #93

overcast wrote: post #87

Earlier quoted context omitted.

I hope you are being SERIOUSLY compensated.

Cash salary today is well under market for my skill set. But, I do own 1/3 of the company so it's not all bad :).

With all of that clout, why aren't you hiring?

Re: GitLab Database Incident – Live Report

#99

Post by btgeekboy » Wed, Feb 01, 2017, 1:57 AM UTC

Shelnutt2 wrote: post #97

TimWolla wrote: post #80

Earlier quoted context omitted.

> is no way to be running a public service with paying customers Is it even possible to pay for the hosted GitLab.com instance?

Gitlab.com is free, not payed. https://about.gitlab.com/products/ The only for pay versions are self hosted EE and independent cloud hosting.

Very well; my apologies. I've updated my comment.

Re: GitLab Database Incident – Live Report

#100

Post by ocdtrekkie » Wed, Feb 01, 2017, 2:00 AM UTC

Walkman wrote: post #25

Seems like very basic mistakes were made, not at the event but way long before. If you don't test to restore your backups, you don't have a backup. How does it go unnoticed that S3 backups don't work for so long?

Helpful hint: Have a employee who regularly accidentally deletes folders. I have a couple, it's why I know my backups work. :D