"we had backups". Do you mean you had backups on digital ocean (using their backup service) or something else?
I'm wondering the same thing
DigitalOcean lost our data and gave us $500
11–20 of 69 posts
Re: DigitalOcean lost our data and gave us $500
#12The abrasive headline is kind of unfortunate, as the actual moral of the story given at the end is exactly the right takeaway: Never assume your hardware is infallible, so always have backups that you know you can use when your server experiences a wildly improbable catastrophe.
Also, very impressed by Digital Ocean's response here. Given their reputation as a budget host, they really do put a lot of effort into service.
Re: DigitalOcean lost our data and gave us $500
#13It seems that is very nice from DO. I would not expect them to be responsible of data loss in case of hardware failure.
Re: DigitalOcean lost our data and gave us $500
#14So this is a technical problem I am having right now that's preventing me from backing up a Postgres database completely (hope someone here can help). I have a master Postgres database that is receiving a TON of transactions per second (I'm talking about thousand concurrent transactions). We tried running pg_dump on this database, but the DB is just too huge, and it took more than 4 days to completely dump out everyt…
What filesystem are you running on? Can you snapshot it outside of the postgres environment? The database may be mid-transaction at that point, but it's still better if it does log replay at startup, than losing all the data.
Also if your filesystem snapshots can be exposed as files / block devs, you can rsync them to another host lowering the amount of transferred data (keep the previous copy so rsync will only copy the blocks that differ).
Re: DigitalOcean lost our data and gave us $500
#15So this is a technical problem I am having right now that's preventing me from backing up a Postgres database completely (hope someone here can help). I have a master Postgres database that is receiving a TON of transactions per second (I'm talking about thousand concurrent transactions). We tried running pg_dump on this database, but the DB is just too huge, and it took more than 4 days to completely dump out everyt…
There's a setting in postgresql.conf that will let you up the limit on long running queries on the standby from 30 seconds to ~ unlimited.
http://www.postgresql.org/docs/9.0/static/runtime-config-wal... See max_standby_archive_delay and max_standby_streaming_delay, -1 lets them wait forever.
Alternately, you can issue pg_start_backup('label'), backup the filesystem, then issue pg_stop_backup() and keep all the WAL logs from that time. That'll get you a base backup similar to the slave.
What I'm doing is this:
I've got a primary/hot spare pair, and a tertiary db on lesser equipment that's my second copy for cases where I have one of the main machines down or I have to rebuild the secondary from the primary.
The tertiary db ships logs to s3, after gpging them. Every $timeframe, I take a base backup and throw it up as well. I keep a couple, and delete the older ones. Every few months, I test a restore on ec2. There's a balance between the WAL logs that you need to keep, the time to restore, and the frequency of base backups.
[edit - parameter names. Further edit - strategy.]
Re: DigitalOcean lost our data and gave us $500
#16So this is a technical problem I am having right now that's preventing me from backing up a Postgres database completely (hope someone here can help). I have a master Postgres database that is receiving a TON of transactions per second (I'm talking about thousand concurrent transactions). We tried running pg_dump on this database, but the DB is just too huge, and it took more than 4 days to completely dump out everyt…
Was the database designed using transactions to achieve consistency? If so, then you can just instruct Tarsnap to back up the folder containing your database every day, and you're done.
If the DB uses transactions for consistency, you can copy it at any time without any problems.
Re: DigitalOcean lost our data and gave us $500
#17That's way more compensation than I would have expected. AWS usually won't even notify you until after the node has gone down. Hardware failures happen; an application needs to be tolerant of it.
And with S3 storage so cheap, they should be backing up directly to S3, across multiple regions.
Re: DigitalOcean lost our data and gave us $500
#18This is 2013. Why are we still talking about backups as a lesson learned? Is it because startups are skimping on Sys Admins?
Re: DigitalOcean lost our data and gave us $500
#19Was this really a dual drive failure, or was this the rather common single drive failure plus undetected errors on a backup drive, that show up when trying to rebuild?
Because that happens a lot, and it's very important to do a full read of every drive in the array at least weekly! You have two options for doing that:
If you are using linux md raid then run the "check" command, which automatically does the test using background I/O (but does still impact things). On debian, and perhaps other distros too the mdadm command will do it every month by default. Make sure to set a minimum speed or it might never finish if you have a busy system.
You can also use the built in SMART on the disk to do a long self test. This also uses background I/O and I think it has a bit less impact on existing operations. (But you have to have some idle time on the disk or it will never finish.) If you install smartmontools you can set smartd to do this test for you every week, and keep an eye on the results.
I personally do both, plus a short self test of the disk every night.
Re: DigitalOcean lost our data and gave us $500
#20What is the best/cost effectiv way to backup a windows server?