Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitLab.com Downtime Postmortem

docs.google.com

Post reply on HN
Print view View original

11–20 of 50 posts

Re: GitLab.com Downtime Postmortem

#11

Post by pilif »

I wonder what their drbd is using as its backing store and why it's needed at all.

Personally, I honestly wouldn't trust to run a database on top of drbd and I'm unsure whether using a drbd volume as a database directory will even cause the other replica to have correct and usable data in case of an error.

Personally, I would use a real disk or maybe LVM as the store for the database in order to reduce points of failures and remove less tested components from the picture.

Then of course you need a database slave, but it looks like you had that anyways which, again, leads me to question why drbd was even involved.

Re: GitLab.com Downtime Postmortem

#12

Post by sytse »

pilif wrote: post #11

I wonder what their drbd is using as its backing store and why it's needed at all. Personally, I honestly wouldn't trust to run a database on top of drbd and I'm unsure whether using a drbd volume as a database directory will even cause the other replica to have correct and usable data in case of an error. Personally, I would use a real disk or maybe LVM as the store for the database in order to reduce points of fail…

We use EXT4 on DRBD on LVM on RAID10. We already have DRBD for the git repo's and prefer to reuse our experience for the database. There are other reasons too some of which are articulated in http://wiki.postgresql.org/images/0/07/Ha_postgres.pdf

Re: GitLab.com Downtime Postmortem

#13

Post by sytse »

sytse wrote: post #12
pilif wrote: post #11

I wonder what their drbd is using as its backing store and why it's needed at all. Personally, I honestly wouldn't trust to run a database on top of drbd and I'm unsure whether using a drbd volume as a database directory will even cause the other replica to have correct and usable data in case of an error. Personally, I would use a real disk or maybe LVM as the store for the database in order to reduce points of fail…

We use EXT4 on DRBD on LVM on RAID10. We already have DRBD for the git repo's and prefer to reuse our experience for the database. There are other reasons too some of which are articulated in http://wiki.postgresql.org/images/0/07/Ha_postgres.pdf

I gave a presentation yesterday about our HA setup https://docs.google.com/presentation/d/1gIzmp-d5X86jJMQz7Ixs...

Re: GitLab.com Downtime Postmortem

#14

Post by Rapzid »

pilif wrote: post #11

I wonder what their drbd is using as its backing store and why it's needed at all. Personally, I honestly wouldn't trust to run a database on top of drbd and I'm unsure whether using a drbd volume as a database directory will even cause the other replica to have correct and usable data in case of an error. Personally, I would use a real disk or maybe LVM as the store for the database in order to reduce points of fail…

Why would you not trust a database server on DRBD? It's a very solid product and using it for database replication/failover is a common pattern. Word on the street is that Amazon RDS uses DRBD for multi-az failover.

Re: GitLab.com Downtime Postmortem

#15

Post by caw »

pilif wrote: post #11

I wonder what their drbd is using as its backing store and why it's needed at all. Personally, I honestly wouldn't trust to run a database on top of drbd and I'm unsure whether using a drbd volume as a database directory will even cause the other replica to have correct and usable data in case of an error. Personally, I would use a real disk or maybe LVM as the store for the database in order to reduce points of fail…

I used DRBD before it was cool (before it was added into the kernel), and it works great for highly available directories like home dirs, which was my use case. Even then it was pretty solid, and it's very similar to using a high end NAS product like NetApp or EMC with automatic failover, except you don't have to shell out $100K+++ for it.

If you used it for a database then everything committed to disk is immediately available on the partner, which you can spin up instantly with some scripts. And look, your transaction log is exactly where you left it! With a database and built-in replication you could go either way, but there are some applications with persistency that you need to make highly available, and that's another area DRBD would shine.

Re: GitLab.com Downtime Postmortem

#17

Post by fsiefken »

Could this be caused by DRBD not reading the changes from the Postgresql database quickly enough? Or is it some issue with the interaction between the software RAID10, DRBD under high postgresql I/O load? What are the respective versions of the kernel, database and DRBD? Are there disk I/O, network I/O and cpu logs available of the time leading up to the crash?

Re: GitLab.com Downtime Postmortem

#18

Post by sytse »

fsiefken wrote: post #17

Could this be caused by DRBD not reading the changes from the Postgresql database quickly enough? Or is it some issue with the interaction between the software RAID10, DRBD under high postgresql I/O load? What are the respective versions of the kernel, database and DRBD? Are there disk I/O, network I/O and cpu logs available of the time leading up to the crash?

We're not sure. There was a lot of disk locking going on at the time.

Re: GitLab.com Downtime Postmortem

#19

Post by np422 »

pilif wrote: post #11

I wonder what their drbd is using as its backing store and why it's needed at all. Personally, I honestly wouldn't trust to run a database on top of drbd and I'm unsure whether using a drbd volume as a database directory will even cause the other replica to have correct and usable data in case of an error. Personally, I would use a real disk or maybe LVM as the store for the database in order to reduce points of fail…

I used to ran a production database on top of drdb without any problem, active-passive fail-over HA-setup.

The major advantage is that no other specialized components (NAS/SAN/etc) is needed and you'll still have decent shared storage between the nodes.

Three nodes in such a cluster is highly recommended resolve/avoid any insane-in-the-split-brain scenarios, drdb-storage needs to be protected from split and can not help you to resolve it in the same way as a traditional NAS/SAN-lun.

We had to do some tuning to get the performance we needed, but this was a few years ago and the situation has probably improved since. Databases usually likes vm-dirty-* to be very low or 0.

Re: GitLab.com Downtime Postmortem

#20

Post by ismail »

sytse wrote: post #2

GitLab B.V. CEO here, please let us know if you have any questions (you can also leave a comment or suggestion in the doc if you want). This whole real-time postmortem is something we thought of to contribute back after having downtime. Feel free to let us know what you think of it.

have you tried HackPad instead of docs?
Post reply on HN