Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitLab Database Incident – Live Report

docs.google.com

Post reply on HN
Print view View original

391–400 of 621 posts

Re: GitLab Database Incident – Live Report

#391

Post by grhmc »

sytse wrote: post #368
grhmc wrote: post #102

Earlier quoted context omitted.

It seems to me that, as a customer, it is blame-shifting away from the company to a particular person. Blameless post-mortems are great, but when speaking to people outside the company I think it is important to own it collectively, "after a second or two we notice we ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com." I believe this isn't your intention, but that is how I interpreted it.

We are all responsible for this, starting with me. We don't blame any person. References to people are just to determine what happend. Also see https://twitter.com/sytses/status/826598260831842308

Yeah, I know, I read your reply. I think it is wrong.

Re: GitLab Database Incident – Live Report

#392

Post by Traubenfuchs »

So tl;dr: Gitlab is experiencing heavy DOS attacks that created so much data that replication stopped working. In the process of getting replication to work again, "YP" wanted to delete the empty data directory of the slave DB server, but accidentally deleted it on the master DB server. Out of 5 backup/replication techniques they use not one is working reliably. YP manually created the backup they could use 6 hours ago by chance.

Tell me if I misunderstood something. I hope the customer I met last week does not remember I ever recommended GitLab to him.

Re: GitLab Database Incident – Live Report

#393

Post by devdas »

danielsamuels wrote: post #283
kelnos wrote: post #227

Earlier quoted context omitted.

You shouldn't let yourself get to that point. Your alerting system should alert you when disk is at 70% or something with a ton of margin. If it's not set up that way, go stop what you're doing and fix that. (Seriously.) If you're running your systems so they usually run at 90% disk usage, go give them more disk (or rotate logs sooner). And even assuming all that fails and I'm in that situation where I have seconds u…

> You shouldn't let yourself get to that point. Your alerting system should alert you when disk is at 70% or something with a ton of margin. What would your recommendation be?

Monitor the rate at which the disk is filling up, and extrapolate that to when it will hit 90%. If that time is outside business hours, alert early. If current time is not in business hours, alert later if possible.

Re: GitLab Database Incident – Live Report

#395

Post by throwaway8124 »

atmosx wrote: post #321
foota wrote: post #288

Earlier quoted context omitted.

Perhaps, Do you really want to delete the dir XYZ with 300000000000 bytes in it?

The problem in this case was that YP confused the host. YP thought he was operating in db2 (the host that went out of sync) and not db1 (the host that held the data), a message that doesn't display the current host wouldn't help in this case.

Indeed. That's happened on our systems as well. Someone issued a reboot on what they thought was a non-critical host, and instead they did it on some very essential host. The host came up in a bad state, and critical services did not start (kerberos....).

It lead to a "Very Bad Day". I found out about it after reading the post mortem.

Re: GitLab Database Incident – Live Report

#396

Post by z3t4 »

There's really no point doing intensive congnitive work for more then 8 hours straight. After that you go by instinct and muscle memory. Surprisingly a lot of tasks can be done, but you shouldn't do anything critical.

Re: GitLab Database Incident – Live Report

#397

Post by aubreykilian »

This happened to me one night late a few years back, with Oracle on a CentOS server. rm -rf /data/oradata/ on the wrong machine.

I managed to get the data back though, as Oracle was still running and had the files open. "lsof | grep '(deleted)'" and /proc//fd/* saved my life. I managed to stop all connections to the database, copy all the (deleted) files into a temp directory, stop Oracle, copy the files to their rightful place, and start up Oracle, with no data lost.

Re: GitLab Database Incident – Live Report

#398

Post by throwaway8124 »

chrissnell wrote: post #143
MattRogish wrote: post #109

Earlier quoted context omitted.

We have 2 people on rotation and anything like this you pair with someone else and talk through it. As we like to say "It's only a mistake if we both make it."

Absolutely. Same goes for any ops response. You need two people: one to triage the issue and another one to communicate with external stakeholders and to help the one doing the triage. The military does a very similar thing. An Army company commander usually has a RTO (radiotelephone operator) to handle taking on the radio. This frees the commander to make real-time decisions and response quickly to the situation on…

Indeed, and a very good way to proceed.

Another thing to note, is that the RTO will occasionally state someone and "ACTUAL". That means that whomever they are speaking for is actually speaking and not the RTO on behalf of the CO.

Post reply on HN