Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postmortem of database outage of January 31

about.gitlab.com

Post reply on HN
Print view View original

191–200 of 269 posts

Re: Postmortem of database outage of January 31

#191

Post by sytse »

atmosx wrote: post #173

Great to have a full-featured, professional post-mortem. Incidentally I work at a company that suffered data loss because of this outage and we're looking for ways to move out of GL. My 2 cents... I might be the only one, but I don't like the way GL handled this case. I understand transparency as a core value and all, but they've gotten a bit too far. IMHO this level of exposure has far-reaching, privacy implications…

Hi Panagiotis. I'm glad to hear you like the postmortem. I'm very sorry your company suffered data loss. If you want to move from GitLab.com please know that you can easily export projects and import them on a self-hosted instance https://gitlab.com/help/user/project/settings/import_export.... (and if in the future we regain your trust you can also go the other way).

It is not our intent to have one of our team members implicated by the transparency. That is why we redacted their name to team-member-1 and in any future incidents we'll do the same. It should be their choice to be identified or not. We are very aware of the stress that such a mistake might cause and the rest of the team has been very supportive.

I agree that we don't look good because of the broken backup policy. The way to fix that is to improve our processes. We recognize the risk to the company of being transparent, but your values are defined by what you do when it is hard.

Re: Postmortem of database outage of January 31

#192

Post by pgaddict »

RhodesianHunter wrote: post #99
memracom wrote: post #93

Earlier quoted context omitted.

RDS PostgreSQL is like the Hotel California, you can check in any time you want, but you can never leave. Maybe it is OK as a simple data store for a single app, but not for a real database. I gained a lot of my knowledge of PostgreSQL internals by helping my company get off of RDS and onto a dedicated EC2 instance solution. RDS imposes too many limitations. Also, your snapshot backup solution is trivial to implement…

"Maybe it is OK as a simple data store for a single app, but not for a real database." Currently working at company number 2 with large (many terabytes) databases on RDS and can safely say this is horse shit. The amount of time and energy it allows our engineers to spend on our actual products instead of database management is worth all of the extra cost and lock in and then some. Edit: I just realized that you were…

We're in the business of PostgreSQL support, and some of our customers use RDS for various reasons. Not having to care about the deployment is one of the usual goals of using a managed environment, but considering they subsequently go and buy support from a third party might be a sign of something.

Of course, my view is biased because we only hear about the issues - there might be a 100x more people using RDS without any issues, and we never hear about them.

In general, the pattern we see is that people start using RDS, and they're fairly happy because it allows them to build their product and RDS more or less works. Then they grow a bit over time, and something breaks.

Which brings me to two main RDS issues that we run into - lack of insight, and difficulty when migrating from RDS with minimum downtime. Once in a while we run into an issue where we wish we could connect over SSH and get some data from the OS, attach gdb to the backend, or something like that. Or using auto_explain. Not even mentioning custom C extensions that we use from time to time ...

Re: Postmortem of database outage of January 31

#193

Post by sperglord »

_Marak_ wrote: post #67

I've noticed a lot of other positive activity and press for Gitlab for in the past month. It's unfortunate they had this technical issue, but it's good to see others ( besides Github ) operating in this space. I should give Gitlab a try sometime.

We just switched from Github to Gitlab for our private repos. The choice (based upon cost alone) was between them and Bitbucket, and the professional way that this was handled and the transparent communication was really nice to see.

Re: Postmortem of database outage of January 31

#194

Post by nhumrich »

illumin8 wrote: post #13

I have to say - if they were using a managed relational database service, like Amazon's RDS Postgres, this likely would have never happened. RDS fully automates nightly database snapshots, and ships archive logs to S3 every 5 minutes, which gives you the ability to restore your database to any point in time within the last 35 days, down to the second. Also, RDS gives you a synchronously replicated standby database, a…

I know about the daily snapshots, but didn't know about the archive logs. Is this something I have to enable? How do I get the logs and how do I restore using them?

Re: Postmortem of database outage of January 31

#195

Post by NPegasus »

milesrout wrote: post #161
developer2 wrote: post #112

Earlier quoted context omitted.

>> the root cause is you have no senior engineers who have been through this before They openly publish their database hostnames in this postmortem (db1.cluster.gitlab.com and db2.cluster.gitlab.com). These actually have public DNS that resolves. The last straw: port 22 on each server is running an open sshd server (the fact that password auth is disabled is of little consolation). A production database server should…

I can't tell if this is some kind of joke or not. There's absolutely nothing wrong, whatsoever, with having a public IP address on a production database server.

  > There's absolutely nothing wrong, whatsoever, with having a public IP address on a production database server.
Umm, yes there is. Explain why you would ever need your database server to be publicly reachable. The only situations I can think of are if you're running everything off a single server, which of course is not relevant to this thread, or if you don't have a suitable gateway, which is rare and also not relevant to this thread.

Re: Postmortem of database outage of January 31

#196

Post by AsyncAwait »

manojlds wrote: post #125
angry_octet wrote: post #121

Earlier quoted context omitted.

The worst part of the whole episode, even worse than 'deleted the active database by accident', was '(backups) were no one's responsibility'. This is not an oversight by an individual engineer, but an aspect of the management and company culture. It shows they lack processes derived from requirements. Lots on introspection required from gitlab at this point.

That's why I was critical of the other article on HN recently that evangelized their remote work.

This has nothing to do with remote work, it has to do with having dedicated devops people on staff.

Re: Postmortem of database outage of January 31

#197

Post by YorickPeterse »

AsyncAwait wrote: post #190
ky738 wrote: post #25

RIP the engineer

Actually, I'll be surprised if he hasn't received any offers by now. I would perhaps specifically hire him to deal with databases as I am pretty sure he's never going to make this mistake again.

I haven't received any offers so far. I don't intend on leaving GitLab any time soon either.

Re: Postmortem of database outage of January 31

#198

Post by gl- »

ancarda wrote: post #182

>Unfortunately DMARC was not enabled for the cronjob emails, resulting in them being rejected by the receiver. This means we were never aware of the backups failing, until it was too late. At my dayjob, we gradually stopped using email for almost all alerts, instead we have several Slack channels like #database-log where errors to MySQL go. Any cron jobs that fail post in #general-log. Uptime monitoring tools post in…

This is a step in the right direction but still misses a big part of it IMO: push versus pull notifications. If the agent stops functioning correctly or someone makes a config change, the alerts just stop and no one notices.

At the very least you want some kinda dead-man's switch that gets pissed if it's seen no events in the last x amount of time. Ideally you want to be polling the box in a stateful way; although with ephemeral nodes & flexible infra being all the rage that's fallen to the side a bit lately.

Re: Postmortem of database outage of January 31

#199

Post by mschuster91 »

cookiecaper wrote: post #175

I really hate to pile on, but after reading through this whole thread and the whole post-mortem, there are a few basic things that are troubling besides the widely-acknowledged backup methodology. I don't see issues directly related to addressing these things. 1. notifications go through regular email. Email should be only one channel used to dispatch notifications of infrastructure events. Tools like VictorOps or Pa…

> Not sure how fast your disks were, but 300GB gone in "a few seconds" sounds like a stretch.

That only depends on the # of files. If it's even a thousand files, any modern Linux rm -rf will remove them in less time than a blink.

> The theoretically best thing to do in such a situation would probably be to unplug the machine ASAP after ^C (without going through formal shutdown processes that may try to "clean up" unfinished disk work), remove the disk, attach it to a machine with a write blocker, and take a full-disk image for forensics purposes. This would maximize the ability to extract any data that the system was unable to eat/destroy.

Their infrastructure is cloud based. No way to get a physical disk - if there is a "disk" at all and not a couple of huge fat NetApp filers providing the storage to the CPU nodes. (This is how a couple web-hosters operate)

Re: Postmortem of database outage of January 31

#200

Post by AsyncAwait »

sytse wrote: post #191
atmosx wrote: post #173

Great to have a full-featured, professional post-mortem. Incidentally I work at a company that suffered data loss because of this outage and we're looking for ways to move out of GL. My 2 cents... I might be the only one, but I don't like the way GL handled this case. I understand transparency as a core value and all, but they've gotten a bit too far. IMHO this level of exposure has far-reaching, privacy implications…

Hi Panagiotis. I'm glad to hear you like the postmortem. I'm very sorry your company suffered data loss. If you want to move from GitLab.com please know that you can easily export projects and import them on a self-hosted instance https://gitlab.com/help/user/project/settings/import_export.... (and if in the future we regain your trust you can also go the other way). It is not our intent to have one of our team membe…

> your values are defined by what you do when it is hard.

Precisely.

Most companies would stay as quiet about this as possible, you guys remained transparent and this is why I'll remain a customer.

Post reply on HN