Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postmortem of database outage of January 31

about.gitlab.com

Post reply on HN
Print view View original

91–100 of 269 posts

Re: Postmortem of database outage of January 31

#91

Post by Cyph0n »

gech wrote: post #89
NPegasus wrote: post #63

> Root Cause Analysis > [...] > [List of technical problems] No, the root cause is you have no senior engineers who have been through this before. A collection of distributed remote employees, none of whom has enough experience to know any of the list of "Basic Knowledge Needed to Run a Website at Scale" that you list as the root causes. $30 million in funding and still running the company like a hobby project among…

Didn't Samsung have some phones catch on fire? Didn't Delta go down recently?

Nah, let's just all board the anti-GitLab train. I honestly don't understand why the HN crowd has their panties in a knot. HN wasn't this bad even during the VW fiasco.

Re: Postmortem of database outage of January 31

#93

Post by memracom »

illumin8 wrote: post #13

I have to say - if they were using a managed relational database service, like Amazon's RDS Postgres, this likely would have never happened. RDS fully automates nightly database snapshots, and ships archive logs to S3 every 5 minutes, which gives you the ability to restore your database to any point in time within the last 35 days, down to the second. Also, RDS gives you a synchronously replicated standby database, a…

RDS PostgreSQL is like the Hotel California, you can check in any time you want, but you can never leave. Maybe it is OK as a simple data store for a single app, but not for a real database. I gained a lot of my knowledge of PostgreSQL internals by helping my company get off of RDS and onto a dedicated EC2 instance solution. RDS imposes too many limitations.

Also, your snapshot backup solution is trivial to implement on EC2 or anywhere else for that matter. But it is not easy to do it right in some scenarios. Read https://www.postgresql.org/docs/9.6/static/backup-file.html for details. LVM or ZFS are likely needed under the db layer.

Re: Postmortem of database outage of January 31

#94

Post by bogomipz »

renaudg wrote: post #77
illumin8 wrote: post #13

I have to say - if they were using a managed relational database service, like Amazon's RDS Postgres, this likely would have never happened. RDS fully automates nightly database snapshots, and ships archive logs to S3 every 5 minutes, which gives you the ability to restore your database to any point in time within the last 35 days, down to the second. Also, RDS gives you a synchronously replicated standby database, a…

This is usually true, except when it's not : I have personally experienced a near-catastrophic situation 3 years ago, where 13 out of 15 days' worth of nightly RDS MySQL snapshots were corrupt and would not restore properly. The root cause was a silent EBS data corruption bug (RDS is EBS-based), that Amazon support eventually admitted to us had slipped through and affected a "small" number of customers. Unlucky us. W…

This is certainly true of all backups to an extent though not just the cloud. Back in the day of backing up to external tape storage it was important to test restores in case heads weren't calibrated or were calibrated differently between different tape machines etc.

I am curious did you manage to automate an restore smoke test after going through this?

Re: Postmortem of database outage of January 31

#95

Post by nodesocket »

My main question is still:

>> Why did replication stop? - A spike in database load caused the database replication process to stop. This was due to the primary removing WAL segments before the secondary could replicate them.

Is this a bug/defect in PostgreSQL then? Incorrect PostgreSQL configuration? Insufficient hardware? What was the root cause of Postgres primary removing the WAL segments?

Re: Postmortem of database outage of January 31

#96

Post by memracom »

dsavinkov wrote: post #53
rectang wrote: post #20

Earlier quoted context omitted.

For me, it was when I meant to execute this... rm -rf ~/foo ... but executed this instead: rm -rf ~ /foo

this is the reason why I always use quotes before specifying folders with rf -rf :)

I always type

ls /stuff/wherever/*

Then examine the output to see if that is the stuff I really want to delete and if it is,

up-arrow Ctrl-A right right backspace backspace rm -rf enter

Never deleted the wrong stuff again in 30 years of doing that

Re: Postmortem of database outage of January 31

#97

Post by saganus »

koolba wrote: post #79
renaudg wrote: post #77

Earlier quoted context omitted.

This is usually true, except when it's not : I have personally experienced a near-catastrophic situation 3 years ago, where 13 out of 15 days' worth of nightly RDS MySQL snapshots were corrupt and would not restore properly. The root cause was a silent EBS data corruption bug (RDS is EBS-based), that Amazon support eventually admitted to us had slipped through and affected a "small" number of customers. Unlucky us. W…

> We got a shitload of AWS credits for our trouble, but the company obviously went through a very near-death experience, and to this day I don't 100% trust cloud backups unless we also have a local copy created regularly. Cloud backups, and more generally all backups should be treated like nuclear profliferation treaties: Trust, but verify! If your periodically restore your backups you'll catch this kind of crap when…

Are there any guidelines on how often you should be doing restore tests?

It probably depends on the criticality of the data, but if you test say every 2 week, you can still fall in the OPs case, right?

At what size/criticality should you have a daily restore test? maybe even a rolling restore test? so you check today's backup, but then check it again every month or something?

Re: Postmortem of database outage of January 31

#98

Post by p4lindromica »

bogomipz wrote: post #90
elliotec wrote: post #85

Earlier quoted context omitted.

They also openly pay well below market rate, so they should expect to get what they pay for.

Can you elaborate on that? For instance I just looked at their jobs listings and they advertise a range for annual compensation, the high end of that range looks about right for the locations I checked SF, and London in the UK for PE positions. https://about.gitlab.com/jobs/production-engineer/

Compensation is more than just salary

Re: Postmortem of database outage of January 31

#99

Post by RhodesianHunter »

memracom wrote: post #93
illumin8 wrote: post #13

I have to say - if they were using a managed relational database service, like Amazon's RDS Postgres, this likely would have never happened. RDS fully automates nightly database snapshots, and ships archive logs to S3 every 5 minutes, which gives you the ability to restore your database to any point in time within the last 35 days, down to the second. Also, RDS gives you a synchronously replicated standby database, a…

RDS PostgreSQL is like the Hotel California, you can check in any time you want, but you can never leave. Maybe it is OK as a simple data store for a single app, but not for a real database. I gained a lot of my knowledge of PostgreSQL internals by helping my company get off of RDS and onto a dedicated EC2 instance solution. RDS imposes too many limitations. Also, your snapshot backup solution is trivial to implement…

"Maybe it is OK as a simple data store for a single app, but not for a real database."

Currently working at company number 2 with large (many terabytes) databases on RDS and can safely say this is horse shit.

The amount of time and energy it allows our engineers to spend on our actual products instead of database management is worth all of the extra cost and lock in and then some.

Edit: I just realized that you were talking about Postgres on RDS in particular. I don't have experience with Postgres so you may well be right.

Re: Postmortem of database outage of January 31

#100

Post by elliotec »

bogomipz wrote: post #90
elliotec wrote: post #85

Earlier quoted context omitted.

They also openly pay well below market rate, so they should expect to get what they pay for.

Can you elaborate on that? For instance I just looked at their jobs listings and they advertise a range for annual compensation, the high end of that range looks about right for the locations I checked SF, and London in the UK for PE positions. https://about.gitlab.com/jobs/production-engineer/

Using that same job calculator that you used (for the developer position) near the bottom of this page: https://about.gitlab.com/jobs/developer/, the rates are way way off from my area.

They are out of their minds if they think the top rate for a Senior engineer in Salt Lake City with above average experience is $78k. Most other areas also seem pretty low from what I've looked into but I suppose a few areas could be outliers in their calculator.

Their docs on the calculator are here: https://about.gitlab.com/handbook/people-operations/global-c...

It's a bit strange that they cut or raise your pay by that calculator when you move cities.

Post reply on HN