Nice move from DO to give everyone $500 credit. As I remember, they don't guarantee data safety (you still need backups even if they did). Double disk failure is a rare thing, but it happens.
DigitalOcean lost our data and gave us $500
41–50 of 69 posts
Re: DigitalOcean lost our data and gave us $500
#42Earlier quoted context omitted.
As long as you issue the pg_start_backup/pg_stop_backup pair and keep the WAL logs. If you don't, then you've got a corrupt backup. At least you would catch that problem in your first test restore.
... what? The point of consistency is that if the power to your server is cut, then you can reboot and pick up precisely where you left off. That means the database on disk must have consistency. Meaning you should be able to copy it at any point in time without any problems. If you can't, then that's not consistency, and if postgres really works that way, then it's failing one of the basic tenants of being a databas…
Unless tarsnap does something like LVM snapshotting, then it's not going to get a consistent snapshot. You can't just copy the directory of an active server.
See: http://www.postgresql.org/docs/9.0/static/continuous-archivi... section 24.3.2. Making a Base Backup.
IF you have something like LVM of ZFS doing snapshots, then you can just tar the data directory.
Re: DigitalOcean lost our data and gave us $500
#43The author is sweet, his conclusion was "always backup your data" if it was me I would probably say "I'm moving away, will never trust them again on my data" ..
Is there a provider that credibly offers high availability Linux servers? Disks fail, capacitors fail, power fails, network equipment fails (a lot). I'm sure it's possible to build an ultra-reliable server that mitigates all that but I doubt it would be worth the money.
Re: DigitalOcean lost our data and gave us $500
#44Earlier quoted context omitted.
As long as you issue the pg_start_backup/pg_stop_backup pair and keep the WAL logs. If you don't, then you've got a corrupt backup. At least you would catch that problem in your first test restore.
... what? The point of consistency is that if the power to your server is cut, then you can reboot and pick up precisely where you left off. That means the database on disk must have consistency. Meaning you should be able to copy it at any point in time without any problems. If you can't, then that's not consistency, and if postgres really works that way, then it's failing one of the basic tenants of being a databas…
The WAL is used for recovering from power loss. You need both the db files and your WAL to get a backup.
Re: DigitalOcean lost our data and gave us $500
#45Earlier quoted context omitted.
As long as you issue the pg_start_backup/pg_stop_backup pair and keep the WAL logs. If you don't, then you've got a corrupt backup. At least you would catch that problem in your first test restore.
... what? The point of consistency is that if the power to your server is cut, then you can reboot and pick up precisely where you left off. That means the database on disk must have consistency. Meaning you should be able to copy it at any point in time without any problems. If you can't, then that's not consistency, and if postgres really works that way, then it's failing one of the basic tenants of being a databas…
If you're taking an instantaneous snapshot of the system then yes. A standard copy/rsync/etc. isn't going to give you that. If the copy takes a long time at what point do you grab the pg_xlog directory? and are all the files there that you need/ed?
Re: DigitalOcean lost our data and gave us $500
#46This is the reason why I moved all data away from my server instances. My images are hosted by cloudinary(with s3 bucket backup) and my databases are Amazon RDS instances. I don't care if a server goes down, I can launch a new one in a matter of minutes (with ansible) without any data loss.
Which of those things you named is protecting you from losing your database? I paid the uber-high fees for RDS with Multi-AZ failover and... well... it failed, then didn't fail over to another AZ. The instance ended up down for hours before they recovered it. That's when I jumped ship from AWS, wrote off the reserved instance payments, moved the database to some rented servers at SoftLayer, and handle nightly off-site backups myself. Not only do I have working backups and failover, but 4-8x the capacity per dollar.
Re: DigitalOcean lost our data and gave us $500
#47This might sound a bit glib, but raid 5 shouldn't really be used in modern storage.
If you ignore the performance issues (which can vary by device) its just not safe. Depending on the size of drive can take anywhere up 30hours+ to rebuild.
bear in mind that you tend to use disks that are all the same batch, it leaves you in the danger zone for far too long.
Your options are: somesort of clever RAID (ZFS type thing) Another type of clever RAID (Like the LSI chunk thingy in the DCS37000) RAID 10
Re: DigitalOcean lost our data and gave us $500
#48Earlier quoted context omitted.
As long as you issue the pg_start_backup/pg_stop_backup pair and keep the WAL logs. If you don't, then you've got a corrupt backup. At least you would catch that problem in your first test restore.
... what? The point of consistency is that if the power to your server is cut, then you can reboot and pick up precisely where you left off. That means the database on disk must have consistency. Meaning you should be able to copy it at any point in time without any problems. If you can't, then that's not consistency, and if postgres really works that way, then it's failing one of the basic tenants of being a databas…
ACID doesn't apply since you can't copy a large file in an instant. The copy takes time, in which time the files on disk can change. This isn't the same thing as the server losing power.
VM snapshots, zfs snapshots, etc are the way to go.
Re: DigitalOcean lost our data and gave us $500
#49This is 2013. Why are we still talking about backups as a lesson learned? Is it because startups are skimping on Sys Admins?
Yes, of course. A full fledged sysadmin is expensive, and startups will typically make several costly mistakes before going to that expense.
This is not surprising, and is not even regrettable. If the business can't support the overhead of someone who doesn't directly bring in revenue, then it can't. And if there's a large investment that makes good infrastructure engineering possible, first-time entrepreneurs might not realize that they need that function.
The key to long term success is in realizing what you will need before it's too late to get it.
Re: DigitalOcean lost our data and gave us $500
#50The abrasive headline is kind of unfortunate, as the actual moral of the story given at the end is exactly the right takeaway: Never assume your hardware is infallible, so always have backups that you know you can use when your server experiences a wildly improbable catastrophe. Also, very impressed by Digital Ocean's response here. Given their reputation as a budget host, they really do put a lot of effort into serv…
> wildly improbable catastrophe Or an extremely probable one like a hard disk failure. They only last a few years; most data centers see an annual replacement rate in the 2-13% range. The failure rate is a known quantity, and their limited 1-3 year warranties that reflect that expectation. There isn't a host I've used more than a few years where I haven't seen hard drives (and power supplies) fail. I don't know if my…
Well it's just a bit unlucky to have both drives fail in a RAID (although not impossible).