DigitalOcean lost our server • Hacker News Board

Re: DigitalOcean lost our server

#41

Post by toomuchtodo » Thu, Feb 04, 2016, 8:53 PM UTC

codazoda wrote: post #37

Although I have backups of my critical files, I don't want to try to rebuild from those files if I can help it. After learning that Digital Ocean does no backups of their own (even for critical hardware failure on their own side) I've enabled weekly backups for an additional $1 per month. Glad you posted this so that I know that option is available and really necessary. And who wouldn't pay $1? Edit: It costs 20% but…

If you're running in "the cloud", you should _always_ be able to destroy your instances and have them auto-rebuild from configuration management and a persistent object storage system (git repo + S3 or Google Nearline).

Re: DigitalOcean lost our server

#42

Post by cortesoft » Thu, Feb 04, 2016, 8:54 PM UTC

phamilton wrote: post #2

I'm glad the author didn't turn this into a negative review of DO. "What will you do when your server is lost?" is exactly the right question to learn from a scenario like this.

I think it was an ad for some of their own services instead.

Re: DigitalOcean lost our server

#43

Post by chrismartin » Thu, Feb 04, 2016, 8:55 PM UTC

trustfundbaby wrote: post #35

The weekly backup deal is something I really do not like with Digital Ocean, I think Linode does it better, I've been planning to move for a while now, I've just been super lazy, but this might be the push I need. 7 days of lost data is unacceptable to me, especially when I'm paying for backups. > Three backup slots are executed and rotated automatically: a daily backup, a 2-7 day old backup, and an 8-14 day old back…

I see DO's weekly backup as a convenience feature for if you need to restore your server, not as your primary backup method. Use the DO snapshot to restore most of the way, then run an up-to-date differential restore from your other (real) backup solution (that is hosted somewhere else).

Re: DigitalOcean lost our server

#44

Post by rascul » Thu, Feb 04, 2016, 8:57 PM UTC

[deleted] wrote: post #29

[deleted]

"Virtual private server" would be more accurate. How is it wrong, though?

Re: DigitalOcean lost our server

#45

Post by Johnny555 » Thu, Feb 04, 2016, 8:58 PM UTC

[deleted] wrote: post #29

[deleted]

So what is the correct interpretation? At a high level, it seems like a reasonable simplification.

Re: DigitalOcean lost our server

#46

Post by andrewthetechie » Thu, Feb 04, 2016, 9:00 PM UTC

fweespeech wrote: post #30

brayton wrote: post #14

They had to make space for the YC portfolio companies getting $250k of credits ;)

I'd still not host production on DO unless your setup allows you to failover to a backup DC with a different provider.

Why?

Re: DigitalOcean lost our server

#47

Post by watty » Thu, Feb 04, 2016, 9:02 PM UTC

gdgtfiend wrote: post #4

This seems like a non-issue to me. If you're using an IaaS provider you should be treating the network as volatile from the get-go. This is the reason AWS has things like auto-scaling groups. You should be designing for failure in "the cloud"

Isn't "non-issue" a big of an exaggeration? If the dry cleaners lost my clothes, the bank lost my money, a valet lost my car, or gmail lost my inbox I'd be angry.

Re: DigitalOcean lost our server

#48

Post by andrewthetechie » Thu, Feb 04, 2016, 9:03 PM UTC

protomyth wrote: post #34

Glyptodon wrote: post #20

Between the lines takeaway: hardware raid is dangerous.

I'd be curious what their RAID is setup. I agree and run ZFS for our data storage.

Check out https://www.digitalocean.com/community/questions/how-did-do-... for your answer. It's a bit old, but still true.

Re: DigitalOcean lost our server

#49

Post by fweespeech » Thu, Feb 04, 2016, 9:07 PM UTC

andrewthetechie wrote: post #46

fweespeech wrote: post #30

Earlier quoted context omitted.

I'd still not host production on DO unless your setup allows you to failover to a backup DC with a different provider.

Why?

They regularly are unable to create new droplets [or other control issues where you can't perform normal functions with normal latency] and/or have full DC outages.

Pretty much any use case where I'd use something like DO being unable to create new VMs, etc. is the same as an outage.

Re: DigitalOcean lost our server

#50

Post by notwhereyouare » Thu, Feb 04, 2016, 9:08 PM UTC

toomuchtodo wrote: post #41

codazoda wrote: post #37

Although I have backups of my critical files, I don't want to try to rebuild from those files if I can help it. After learning that Digital Ocean does no backups of their own (even for critical hardware failure on their own side) I've enabled weekly backups for an additional $1 per month. Glad you posted this so that I know that option is available and really necessary. And who wouldn't pay $1? Edit: It costs 20% but…

If you're running in "the cloud", you should _always_ be able to destroy your instances and have them auto-rebuild from configuration management and a persistent object storage system (git repo + S3 or Google Nearline).

do you have any guides or tutorials I could possibly follow to have a setup like that? I'm working on configuring a site and would like the ability to 1) scale as quickly as possibly or 2) rebuild in event of total failure.

I just don't even know where to begin. Is this something I could stand up locally and push to like s3, spin up a new server and install 1 thing and have it pull the configs and software installs down?

Any suggestions?