Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DigitalOcean lost our data and gave us $500

dfernandez.me

Post reply on HN
Print view View original

31–40 of 69 posts

Re: DigitalOcean lost our data and gave us $500

#31

Post by sillysaurus2 »

wiredfool wrote: post #21
sillysaurus2 wrote: post #16

Earlier quoted context omitted.

Was the database designed using transactions to achieve consistency? If so, then you can just instruct Tarsnap to back up the folder containing your database every day, and you're done. If the DB uses transactions for consistency, you can copy it at any time without any problems.

As long as you issue the pg_start_backup/pg_stop_backup pair and keep the WAL logs. If you don't, then you've got a corrupt backup. At least you would catch that problem in your first test restore.

... what? The point of consistency is that if the power to your server is cut, then you can reboot and pick up precisely where you left off. That means the database on disk must have consistency. Meaning you should be able to copy it at any point in time without any problems. If you can't, then that's not consistency, and if postgres really works that way, then it's failing one of the basic tenants of being a database. http://en.wikipedia.org/wiki/ACID

Any database that purports to have consistency must be able to withstand cutting the power to the server at any time. And if it can do that, then it must be true that you can copy the database folder at any time, too, without any special commands. (pg_start_backup is not issued before every power loss, so why would it need to be issued before a copy?)

On the other hand, if postgres doesn't support consistency, then that'd be a major reason not to use it.

EDIT: I'd run the server in a VM and backup VM snapshots. VMware makes this painless (and the snapshotting process is designed to have minimal impact disk I/O performance for precisely the scenario the OP described). VirtualBox probably has something similar. These replies seem crazily overcomplicated in comparison.

Re: DigitalOcean lost our data and gave us $500

#32

Post by khaless »

AznHisoka wrote: post #9

So this is a technical problem I am having right now that's preventing me from backing up a Postgres database completely (hope someone here can help). I have a master Postgres database that is receiving a TON of transactions per second (I'm talking about thousand concurrent transactions). We tried running pg_dump on this database, but the DB is just too huge, and it took more than 4 days to completely dump out everyt…

Just a thought... If your storage layer has support for taking a consistent snapshot of your file system then you might be able to use this to get a backup.

You would get a copy of your database that you would need to run log-replay recovery on but after that it should be all good.

Re: DigitalOcean lost our data and gave us $500

#33

Post by cbsmith »

> And if you just launched and have a single instance running, let your alpha users know that there will probably be some downtime.

How about instead "alpha users should know that there will probably be some downtime". Multiple instances don't really fix that.

Re: DigitalOcean lost our data and gave us $500

#36

Post by sebslomski »

This is the reason why I moved all data away from my server instances. My images are hosted by cloudinary(with s3 bucket backup) and my databases are Amazon RDS instances. I don't care if a server goes down, I can launch a new one in a matter of minutes (with ansible) without any data loss.

Re: DigitalOcean lost our data and gave us $500

#37

Post by astrodust »

gregd wrote: post #18

This is 2013. Why are we still talking about backups as a lesson learned? Is it because startups are skimping on Sys Admins?

It's because some startups have developers that open w3schools, start typing examples, and somehow ship a quasi-working proof-of-concept that goes into production.

There's a bit of "if it ain't broke don't fix it" here, but a whole lot of "get with the program" still required.

Re: DigitalOcean lost our data and gave us $500

#39

Post by arjie »

level09 wrote: post #29

The author is sweet, his conclusion was "always backup your data" if it was me I would probably say "I'm moving away, will never trust them again on my data" ..

To whom would you migrate? It seems to me that you wouldn't get better service without a managed server.

Re: DigitalOcean lost our data and gave us $500

#40

Post by dangrossman »

chc wrote: post #12

The abrasive headline is kind of unfortunate, as the actual moral of the story given at the end is exactly the right takeaway: Never assume your hardware is infallible, so always have backups that you know you can use when your server experiences a wildly improbable catastrophe. Also, very impressed by Digital Ocean's response here. Given their reputation as a budget host, they really do put a lot of effort into serv…

> wildly improbable catastrophe

Or an extremely probable one like a hard disk failure. They only last a few years; most data centers see an annual replacement rate in the 2-13% range. The failure rate is a known quantity, and their limited 1-3 year warranties that reflect that expectation.

There isn't a host I've used more than a few years where I haven't seen hard drives (and power supplies) fail. I don't know if my experience is typical, but hardware RAID controllers seem to go bad on me not-infrequently too, losing the whole array at once. They don't pay you when it happens, they just replace it. DO was extremely generous here.

Post reply on HN