Live data from Hacker News

Postmortem of database outage of January 31

about.gitlab.com

41–50 of 269 posts

Re: Postmortem of database outage of January 31

#41
post #5

Am I missing something or didn't they mention 'test recovery, not backups'?

Two of the issues linked from the article deal with testing the backups:

- Automated testing of recovering PostgreSQL database backups https://gitlab.com/gitlab-com/infrastructure/issues/1102

- Build Streaming Database Restore https://gitlab.com/gitlab-com/infrastructure/issues/1152

Re: Postmortem of database outage of January 31

#42

I have to say - if they were using a managed relational database service, like Amazon's RDS Postgres, this likely would have never happened. RDS fully automates nightly database snapshots, and ships archive logs to S3 every 5 minutes, which gives you the ability to restore your database to any point in time within the last 35 days, down to the second. Also, RDS gives you a synchronously replicated standby database, a…

RDS, or any hosted database solution, is not some kind of silver bullet that solves all problems. While it's true it takes care of backups automatically, it does also restrict you in terms of what you can do. For example, you can't load custom extensions into RDS. Also, to the best of my knowledge RDS does not support a hot standby replica you can use for read-only queries, and replication between RDS and non RDS is…

You can use the failover standby replica for reads with Aurora at least. And you can manually via MySQL set up replication with non RDS, just not via AWS APIs.

Re: Postmortem of database outage of January 31

#43

I have to say - if they were using a managed relational database service, like Amazon's RDS Postgres, this likely would have never happened. RDS fully automates nightly database snapshots, and ships archive logs to S3 every 5 minutes, which gives you the ability to restore your database to any point in time within the last 35 days, down to the second. Also, RDS gives you a synchronously replicated standby database, a…

RDS, or any hosted database solution, is not some kind of silver bullet that solves all problems. While it's true it takes care of backups automatically, it does also restrict you in terms of what you can do. For example, you can't load custom extensions into RDS. Also, to the best of my knowledge RDS does not support a hot standby replica you can use for read-only queries, and replication between RDS and non RDS is…

> Also, to the best of my knowledge RDS does not support a hot standby replica you can use for read-only queries

RDS has very nice Read Replicas.

http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_R...

For HA you can use High Availability (Multi-AZ).

http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concep...

Re: Postmortem of database outage of January 31

#44
post #20

Earlier quoted context omitted.

For me, it was when I meant to execute this... rm -rf ~/foo ... but executed this instead: rm -rf ~ /foo

I did the Windows equivalent once a long time ago (I think it's deltree?) and I did it on a university computer system. It cleared out a TON of files and the computer itself pretty much stopped working. I had to hard turn it off. Fortunately the University was using some tool that can re-image a computer each time it boots before hitting Windows so starting it back up and all the deleted system and application files…

Probably Ghost server, that's why schools do this.

Re: Postmortem of database outage of January 31

#45
Shouldn't the conclusion of this post mortem be a move to a managed database service like RDS? The database doesn't sound huge, RDS is affordable enough, sounds to me that you spend less money and have better uptime and sleep by moving away from this in-house solution.

Re: Postmortem of database outage of January 31

#46
Shouldn't the conclusion of this post mortem be a move to a managed database service like RDS? The database doesn't sound huge, RDS is affordable enough, sounds to me that you spend less money and have better uptime and sleep by moving away from this in-house solution.

Re: Postmortem of database outage of January 31

#47
post #43

Earlier quoted context omitted.

RDS, or any hosted database solution, is not some kind of silver bullet that solves all problems. While it's true it takes care of backups automatically, it does also restrict you in terms of what you can do. For example, you can't load custom extensions into RDS. Also, to the best of my knowledge RDS does not support a hot standby replica you can use for read-only queries, and replication between RDS and non RDS is…

> Also, to the best of my knowledge RDS does not support a hot standby replica you can use for read-only queries RDS has very nice Read Replicas. http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_R... For HA you can use High Availability (Multi-AZ). http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concep...

[deleted]

Re: Postmortem of database outage of January 31

#48
post #43

Earlier quoted context omitted.

RDS, or any hosted database solution, is not some kind of silver bullet that solves all problems. While it's true it takes care of backups automatically, it does also restrict you in terms of what you can do. For example, you can't load custom extensions into RDS. Also, to the best of my knowledge RDS does not support a hot standby replica you can use for read-only queries, and replication between RDS and non RDS is…

> Also, to the best of my knowledge RDS does not support a hot standby replica you can use for read-only queries RDS has very nice Read Replicas. http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_R... For HA you can use High Availability (Multi-AZ). http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concep...

That said, it did take AWS a disturbingly long time to add support for Postgres read replicas.

Re: Postmortem of database outage of January 31

#49
post #43

Earlier quoted context omitted.

RDS, or any hosted database solution, is not some kind of silver bullet that solves all problems. While it's true it takes care of backups automatically, it does also restrict you in terms of what you can do. For example, you can't load custom extensions into RDS. Also, to the best of my knowledge RDS does not support a hot standby replica you can use for read-only queries, and replication between RDS and non RDS is…

> Also, to the best of my knowledge RDS does not support a hot standby replica you can use for read-only queries RDS has very nice Read Replicas. http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_R... For HA you can use High Availability (Multi-AZ). http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concep...

Aha, I probably overlooked read replicas (haven't used RDS in about 1.5 years).

Re: Postmortem of database outage of January 31

#50
post #43

Earlier quoted context omitted.

> Also, to the best of my knowledge RDS does not support a hot standby replica you can use for read-only queries RDS has very nice Read Replicas. http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_R... For HA you can use High Availability (Multi-AZ). http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concep...

That said, it did take AWS a disturbingly long time to add support for Postgres read replicas.

Why disturbingly long? As opposed to frustratingly long? Or something else? What does the time it took Amazon to add such support imply to you?
Post reply on HN