Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postmortem of database outage of January 31

about.gitlab.com

Post reply on HN
Print view View original

31–40 of 269 posts

Re: Postmortem of database outage of January 31

#31

Post by yarper »

jdavis703 wrote: post #19
yarper wrote: post #14

It's amazing how quickly it descends into "one of the engineers" did x or y. Who was steering this ship exactly? It's really simple to point the finger and try to find a single cause of failure - but it's a fools errand - comparable to finding the single source behind a great success.

Do you expect management to be staring over your shoulder every time you do some kind of `rm` on a production server? With great power comes great responsibility.

People get tired, sick, frustrated, panic - part of being a responsible engineer is accepting you're as fallible as the next person and building in protection against your own errors.

However, if "the engineer" that caused this happens to read this, the above is not a sign that you should quit the profession and become a hermit. A chain of events caused this, you just happened to be the one without a chair to sit in when the music stopped.

Re: Postmortem of database outage of January 31

#33

Post by haldean »

a_t48 wrote: post #18
meowface wrote: post #10

>Trying to restore the replication process, an engineer proceeds to wipe the PostgreSQL database directory, errantly thinking they were doing so on the secondary. Unfortunately this process was executed on the primary instead. The engineer terminated the process a second or two after noticing their mistake, but at this point around 300 GB of data had already been removed. I could feel the sweat drops just from readin…

Misread that as "The engineer was terminated" at first. Poor guy.

Potential outage prevention plan: put an alias on all production machines that emails HR to schedule a disciplinary meeting every time you run `rm -rf`.

Re: Postmortem of database outage of January 31

#34

Post by meowface »

rectang wrote: post #20
meowface wrote: post #10

>Trying to restore the replication process, an engineer proceeds to wipe the PostgreSQL database directory, errantly thinking they were doing so on the secondary. Unfortunately this process was executed on the primary instead. The engineer terminated the process a second or two after noticing their mistake, but at this point around 300 GB of data had already been removed. I could feel the sweat drops just from readin…

For me, it was when I meant to execute this... rm -rf ~/foo ... but executed this instead: rm -rf ~ /foo

For me it was a simple `rm -rf .`. Thought I was in ~/somefolder/, was actually in ~...

I stopped it after about 3 seconds, but that was enough to do critical damage.

Re: Postmortem of database outage of January 31

#35

Post by teej »

tedunangst wrote: post #27
bitshepherd wrote: post #17

Earlier quoted context omitted.

A great number of issues can be attributed to the selection of Azure as the platform of choice. That said, a little bird told me that the decision was largely a cost factor. "You get what you pay for" never rang more true.

But none of the issues were azure/cost related except for the slow recovery? I mean, neither AWS nor GCE can make you notice youre not getting cron mail.

Setup a cloudwatch alarm and have cron send a keepalive email to it..? Without looking, I'm pretty sure that does exactly what you want.

Re: Postmortem of database outage of January 31

#36

Post by tedunangst »

teej wrote: post #35
tedunangst wrote: post #27

Earlier quoted context omitted.

But none of the issues were azure/cost related except for the slow recovery? I mean, neither AWS nor GCE can make you notice youre not getting cron mail.

Setup a cloudwatch alarm and have cron send a keepalive email to it..? Without looking, I'm pretty sure that does exactly what you want.

And also entirely possible to do with azure. AWS doesn't make you do that.

Also, depending on precise nature of the dmarc fuckup, wouldn't help anyway. Cloud watch receiving mail doesn't guarantee that you will receive mail.

Re: Postmortem of database outage of January 31

#38

Post by overcast »

illumin8 wrote: post #13

I have to say - if they were using a managed relational database service, like Amazon's RDS Postgres, this likely would have never happened. RDS fully automates nightly database snapshots, and ships archive logs to S3 every 5 minutes, which gives you the ability to restore your database to any point in time within the last 35 days, down to the second. Also, RDS gives you a synchronously replicated standby database, a…

This likely would have never happened, if one of their one hundred and sixty employees, just took the time to make sure backups were setup at all. You also need to be a sufficiently large enough organization to warrant the prices that the "cloud" services demand. As stated below, cloud computing is just someone else's server somewhere, and they are making lots of money doing it. Unless you need that level of scalability, and processing, then it's not worth it. I think Gitlab stated their entire PostgreSQL database was only a few hundred gigabytes. That's not exactly huge.

Re: Postmortem of database outage of January 31

#39

Post by YorickPeterse »

illumin8 wrote: post #13

I have to say - if they were using a managed relational database service, like Amazon's RDS Postgres, this likely would have never happened. RDS fully automates nightly database snapshots, and ships archive logs to S3 every 5 minutes, which gives you the ability to restore your database to any point in time within the last 35 days, down to the second. Also, RDS gives you a synchronously replicated standby database, a…

RDS, or any hosted database solution, is not some kind of silver bullet that solves all problems. While it's true it takes care of backups automatically, it does also restrict you in terms of what you can do.

For example, you can't load custom extensions into RDS. Also, to the best of my knowledge RDS does not support a hot standby replica you can use for read-only queries, and replication between RDS and non RDS is also not supported. This means you can't balance load between multiple hosts, unless you're OK with running a multi-master setup (of which I'm not sure how well this would play out on RDS).

Most important of all, we ship PostgreSQL as part of our Omnibus package. As a result the best way of testing this over time is to use it ourselves, something we strive to do with everything we sihp. This means we need to actually run our own things. Using a hosted database would mean we wouldn't be using a part of what we ship, thus not being able to test it over time.

Re: Postmortem of database outage of January 31

#40

Post by [deleted] »

YorickPeterse wrote: post #39
illumin8 wrote: post #13

I have to say - if they were using a managed relational database service, like Amazon's RDS Postgres, this likely would have never happened. RDS fully automates nightly database snapshots, and ships archive logs to S3 every 5 minutes, which gives you the ability to restore your database to any point in time within the last 35 days, down to the second. Also, RDS gives you a synchronously replicated standby database, a…

RDS, or any hosted database solution, is not some kind of silver bullet that solves all problems. While it's true it takes care of backups automatically, it does also restrict you in terms of what you can do. For example, you can't load custom extensions into RDS. Also, to the best of my knowledge RDS does not support a hot standby replica you can use for read-only queries, and replication between RDS and non RDS is…

[deleted]
Post reply on HN