DigitalOcean lost our data and gave us $500
dfernandez.me
DigitalOcean lost our data and gave us $500
1–10 of 69 posts
Re: DigitalOcean lost our data and gave us $500
#2That's always a risk with servers. They can die and everything can do with them. But they had backups so they didn't lose everything.
Re: DigitalOcean lost our data and gave us $500
#3That's way more compensation than I would have expected. AWS usually won't even notify you until after the node has gone down.
Hardware failures happen; an application needs to be tolerant of it.
Re: DigitalOcean lost our data and gave us $500
#4So if you were backing up your data to Tarsnap, then you'd be up and running as quickly as you could launch a new instance and redownload everything. And $500 credit is enough to power a micro droplet for 100 months, or a small droplet for 50 months. DO handled this well.
EDIT: s/years/months/g. Thanks.
Re: DigitalOcean lost our data and gave us $500
#5Good thing you had backups.
With that being said, these days it's a good idea to use a deployment tool or configuration management system like puppet/salt/ansible/chef/etc, especially in a virtualized environment. This will help with scalability as well as situations such as these.
Re: DigitalOcean lost our data and gave us $500
#6It's great you had backups, but why a write-up. Is it an attempt to smear DO's otherwise good name? It's an un-managed VPS so it's your responsibility to keep backups of your box, not theirs. And hardware fails all the time, so you can expect this to happen anywhere.
Re: DigitalOcean lost our data and gave us $500
#7So if you were backing up your data to Tarsnap, then you'd be up and running as quickly as you could launch a new instance and redownload everything. And $500 credit is enough to power a micro droplet for 100 months, or a small droplet for 50 months. DO handled this well. http://www.tarsnap.com EDIT: s/years/months/g. Thanks.
You mean months.
Re: DigitalOcean lost our data and gave us $500
#8"we had backups".
Do you mean you had backups on digital ocean (using their backup service) or something else?
Re: DigitalOcean lost our data and gave us $500
#9So this is a technical problem I am having right now that's preventing me from backing up a Postgres database completely (hope someone here can help).
I have a master Postgres database that is receiving a TON of transactions per second (I'm talking about thousand concurrent transactions). We tried running pg_dump on this database, but the DB is just too huge, and it took more than 4 days to completely dump out everything. Not only that but it impacted performance to the point where backing it up was just not feasible.
No problem.. just create a slave-DB and run pg_dump on that, right? We did just that, but the problem is that you can't run long running queries on a hot standby (queries that take more than a minute).
What would you do in my scenario? With the hot standby, I technically am backing up my data, but I would have 100% piece of mind if I could daily backups in case someone accidentally ran a "DROP DATABASE X", which would also delete the hot standby/slave db as well.
Re: DigitalOcean lost our data and gave us $500
#10DO is affordable enough that the minimum you should run are 2 droplets. Having said that, I'm actually fairly impressed with the 500 credit and now you have no excuses to run 2 vms. Consider it a lesson learnt.