Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postmortem of database outage of January 31

about.gitlab.com

Post reply on HN
Print view View original

241–250 of 269 posts

Re: Postmortem of database outage of January 31

#241

Post by jjirsa »

KayEss wrote: post #132

The engineers still seem to have a physical server mindset rather than a cloud mindset. Deleting data is always extremely dangerous and there was no need for it in this situation. They should have spun up a new server to act as secondary the moment replication failed. This new server is the one you run all of these commands on, and if you make a mistake you spin up a new one. Only when the replication is back in good…

The fact that they're using a single master software is already an antiquated concept in 2017

Re: Postmortem of database outage of January 31

#242

Post by mkopinsky »

ancarda wrote: post #182

>Unfortunately DMARC was not enabled for the cronjob emails, resulting in them being rejected by the receiver. This means we were never aware of the backups failing, until it was too late. At my dayjob, we gradually stopped using email for almost all alerts, instead we have several Slack channels like #database-log where errors to MySQL go. Any cron jobs that fail post in #general-log. Uptime monitoring tools post in…

How do you set cron to post to Slack?

Re: Postmortem of database outage of January 31

#243

Post by cookiecaper »

bshanks wrote: post #235
cookiecaper wrote: post #209

Earlier quoted context omitted.

> Hot standby is also supported just fine out of the box, you just need something third party for the actual load balancing. I'm aware that hot standby is supported, though it's not the default configuration for the standby server (default and safest is a standby mode that you can't query at all; hot standby introduces possible conflicts between hot read queries and write transactions coming in from the WAL, so if fa…

Regarding the spam user deletions, I am not involved but my interpretation of what GitLab wrote ( https://gitlab.com/gitlab-org/gitlab-ce/issues/27581 https://gitlab.com/MrChrisW/gitlab-anti-spam-toolkit/issues/... ) is that, currently, there is no auto-delete running, and a human must approve each deletion.

Thanks for pointing that out. It doesn't appear that this was clarified until a couple of hours after I posted my comment, but it's definitely a relief and the wise course of action.

Re: Postmortem of database outage of January 31

#244

Post by milesrout »

user5994461 wrote: post #234
cookiecaper wrote: post #167

Earlier quoted context omitted.

I wouldn't say there's "absolutely nothing wrong with it". Someone may have a valid use case to leave a database server exposed to the public internet, but they probably don't. The only things that should be public facing are things that clients need to access directly. In most cases, that's just an HTTP server. In GitLab's case, it's an HTTP server and a git server. One of the most important principles of infrastruc…

Little correction: HTTP servers should not be publicly accessible. Only a couple of load balancers need to be accessible.

Firstly, what do you think HTTP load balancers are, if not HTTP servers?

Secondly, why would anyone use a load balancer? Almost every website in existence doesn't need a load balancer.

Re: Postmortem of database outage of January 31

#245

Post by bartvk »

dsavinkov wrote: post #53
rectang wrote: post #20

Earlier quoted context omitted.

For me, it was when I meant to execute this... rm -rf ~/foo ... but executed this instead: rm -rf ~ /foo

this is the reason why I always use quotes before specifying folders with rf -rf :)

Great tip, thanks!

Re: Postmortem of database outage of January 31

#246

Post by IA21 »

jjirsa wrote: post #240
matt_wulfeck wrote: post #157

Earlier quoted context omitted.

This type of thing always sounds good and all, but the reality is people get desperate and emotional when their website is down and everyone wants it up ASAP.

Desperate and emotional is no way to run a business

I agree but engineers are people, not machines. They're bound to make mistakes under pressure.

Re: Postmortem of database outage of January 31

#247

Post by Achshar »

cmatija wrote: post #223
Achshar wrote: post #178

Does anyone have a link to the YouTube stream they's talking about? Can't seem to find it on their channel. And the link in the doc is redirecting to the live link [1] which doesn't list the stream. [1] - https://www.youtube.com/c/Gitlab/live

Yup, Thanks for taking the interest to check it out. It's an unlisted YT video, so that's why it might be hard to find. Here it is: https://www.youtube.com/watch?v=nc0hPGerSd4

Thanks! I'll check it out.

Re: Postmortem of database outage of January 31

#248

Post by ancarda »

mkopinsky wrote: post #242
ancarda wrote: post #182

>Unfortunately DMARC was not enabled for the cronjob emails, resulting in them being rejected by the receiver. This means we were never aware of the backups failing, until it was too late. At my dayjob, we gradually stopped using email for almost all alerts, instead we have several Slack channels like #database-log where errors to MySQL go. Any cron jobs that fail post in #general-log. Uptime monitoring tools post in…

How do you set cron to post to Slack?

We use Monolog (PHP library) that is able to post to Slack.

Messages in Monolog, like syslog have a level attached, so DEBUG, INFO, NOTICE and WARNING will only be written to a log file on disk. Anything higher, so ERROR, CRITICAL, ALERT or EMERGENCY will write to Slack (as well as log to disk). This means we only get notified of things failing and we can go on the server and see everything from DEBUG upwards which lets us mentally step through the cron job's run.

It's a very cool library. https://github.com/Seldaek/monolog

You can see the handlers here: https://github.com/Seldaek/monolog/tree/master/src/Monolog/H... which includes Slack, HipChat, IFTTT, Pushover, etc...

Re: Postmortem of database outage of January 31

#249

Post by atmosx »

memracom wrote: post #93
illumin8 wrote: post #13

I have to say - if they were using a managed relational database service, like Amazon's RDS Postgres, this likely would have never happened. RDS fully automates nightly database snapshots, and ships archive logs to S3 every 5 minutes, which gives you the ability to restore your database to any point in time within the last 35 days, down to the second. Also, RDS gives you a synchronously replicated standby database, a…

RDS PostgreSQL is like the Hotel California, you can check in any time you want, but you can never leave. Maybe it is OK as a simple data store for a single app, but not for a real database. I gained a lot of my knowledge of PostgreSQL internals by helping my company get off of RDS and onto a dedicated EC2 instance solution. RDS imposes too many limitations. Also, your snapshot backup solution is trivial to implement…

Could you describe the limitations you encounter in the RDS PSQL setup vs a self-managed?

Re: Postmortem of database outage of January 31

#250

Post by ancarda »

gl- wrote: post #198
ancarda wrote: post #182

>Unfortunately DMARC was not enabled for the cronjob emails, resulting in them being rejected by the receiver. This means we were never aware of the backups failing, until it was too late. At my dayjob, we gradually stopped using email for almost all alerts, instead we have several Slack channels like #database-log where errors to MySQL go. Any cron jobs that fail post in #general-log. Uptime monitoring tools post in…

This is a step in the right direction but still misses a big part of it IMO: push versus pull notifications. If the agent stops functioning correctly or someone makes a config change, the alerts just stop and no one notices. At the very least you want some kinda dead-man's switch that gets pissed if it's seen no events in the last x amount of time. Ideally you want to be polling the box in a stateful way; although wi…

Absolutely, that's a great idea!

You could also check for evidence a run has been successful, although that does depend on what you're doing exactly.

For our backup system, we're going to build an audit cron job on our main server that checks all our Azure containers to see if each server has pushed a file lately. It'll alert us if a file hasn't been uploaded in a few days or if it's smaller than a few MB (which is suspiciously small; we'd expect a few hundred MB for mysqldump+files).

Post reply on HN