Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DigitalOcean lost our server

murze.be

Post reply on HN
Print view View original

71–80 of 219 posts

Re: DigitalOcean lost our server

#71

Post by brianwawok »

notwhereyouare wrote: post #50
toomuchtodo wrote: post #41

Earlier quoted context omitted.

If you're running in "the cloud", you should _always_ be able to destroy your instances and have them auto-rebuild from configuration management and a persistent object storage system (git repo + S3 or Google Nearline).

do you have any guides or tutorials I could possibly follow to have a setup like that? I'm working on configuring a site and would like the ability to 1) scale as quickly as possibly or 2) rebuild in event of total failure. I just don't even know where to begin. Is this something I could stand up locally and push to like s3, spin up a new server and install 1 thing and have it pull the configs and software installs d…

Its a semi-religious topic, so you will get varied and loud opinions.

My opinion is ansible. Basically python + yml, you list out steps to set up your server. Takes a few minutes longer round 1, but you can remake the server in 60 seconds in 6 months. Lots of ansible tutorials around, try it out..

Re: DigitalOcean lost our server

#72

Post by vardump »

twunde wrote: post #40

DO just gave a talk (~2 weeks ago) where they mentioned that they were moving to software RAID.

Sounds great, for example ZFS has proven to be more reliable than PCI-e card based hardware RAID solutions.

EMC and Netapp are of course another matter, they're great. If you can afford them.

Re: DigitalOcean lost our server

#73

Post by tshtf »

kentonv wrote: post #63

I think Google Compute Engine does the right thing here: by default it uses "persistent disks" (network-attached redundant/highly-available block devices) for all disks. The only case I've heard of where persistent disk data was lost was a few acknowledged writes occurring just before an unusual lightning-induced power outage: https://status.cloud.google.com/incident/compute/15056 For added protection, you can take r…

Elastic storage on AWS has a fairly low AFR (https://aws.amazon.com/ebs/details/):

Amazon EBS volumes are designed for an annual failure rate (AFR) of between 0.1% - 0.2%, where failure refers to a complete or partial loss of the volume, depending on the size and performance of the volume. This makes EBS volumes 20 times more reliable than typical commodity disk drives, which fail with an AFR of around 4%. EBS also supports a snapshot feature, which is a good way to take point-in-time backups of your data.

Re: DigitalOcean lost our server

#74

Post by gedrap »

trustfundbaby wrote: post #35

The weekly backup deal is something I really do not like with Digital Ocean, I think Linode does it better, I've been planning to move for a while now, I've just been super lazy, but this might be the push I need. 7 days of lost data is unacceptable to me, especially when I'm paying for backups. > Three backup slots are executed and rotated automatically: a daily backup, a 2-7 day old backup, and an 8-14 day old back…

Agreed. When you actually need to restore something from backup, often you need a fairly recent state. Hours or days ago (and not 7 days ago).

Having something 7 days old might be useful in some cases (especially e.g. DB dump) but it is far from 'good enough'.

I am not sure what's the reason why they do not provide more recent backups (for additional fee, obviously).

Would it be so complicated that the engineering effort required wouldn't make it worth it? Or such a service wouldn't be popular enough among DO users?

Re: DigitalOcean lost our server

#75

Post by moondev »

notwhereyouare wrote: post #50
toomuchtodo wrote: post #41

Earlier quoted context omitted.

If you're running in "the cloud", you should _always_ be able to destroy your instances and have them auto-rebuild from configuration management and a persistent object storage system (git repo + S3 or Google Nearline).

do you have any guides or tutorials I could possibly follow to have a setup like that? I'm working on configuring a site and would like the ability to 1) scale as quickly as possibly or 2) rebuild in event of total failure. I just don't even know where to begin. Is this something I could stand up locally and push to like s3, spin up a new server and install 1 thing and have it pull the configs and software installs d…

Source code lives in a repo

user data lives in redundant databases / storage service

server setup lives in a repo via bash scripts or ansible etc (and bake that into an ami or container)

setup your launch-configuration for your servers to setup all of the above and you can destroy and spin up servers on a whim

Re: DigitalOcean lost our server

#76

Post by nailer »

gdgtfiend wrote: post #4

This seems like a non-issue to me. If you're using an IaaS provider you should be treating the network as volatile from the get-go. This is the reason AWS has things like auto-scaling groups. You should be designing for failure in "the cloud"

> You should be designing for failure in "the cloud"

DO should be designing for failure: VM storage should be on a SAN. A single physical server failing should not cause data loss. This is basic stuff.

Re: DigitalOcean lost our server

#77

Post by gedrap »

chrismartin wrote: post #43
trustfundbaby wrote: post #35

The weekly backup deal is something I really do not like with Digital Ocean, I think Linode does it better, I've been planning to move for a while now, I've just been super lazy, but this might be the push I need. 7 days of lost data is unacceptable to me, especially when I'm paying for backups. > Three backup slots are executed and rotated automatically: a daily backup, a 2-7 day old backup, and an 8-14 day old back…

I see DO's weekly backup as a convenience feature for if you need to restore your server, not as your primary backup method. Use the DO snapshot to restore most of the way, then run an up-to-date differential restore from your other (real) backup solution (that is hosted somewhere else).

But wouldn't it be much more convenient (largely because of the complexity) to have the complete backup solution at the same place (i.e. DO)?

Obviously, you want some offsite backups rather than completely relying on DO but that should be for disaster recovery (massive failures in DO's DCs, etc) rather than a more routine one.

Re: DigitalOcean lost our server

#78

Post by nickbauman »

gdgtfiend wrote: post #4

This seems like a non-issue to me. If you're using an IaaS provider you should be treating the network as volatile from the get-go. This is the reason AWS has things like auto-scaling groups. You should be designing for failure in "the cloud"

CAP theorem applies everywhere, but most software practices still do not assume that you must code for your app to be able to handle performance variation and partial failure state both of which are common scenarios of cloud infra.

Re: DigitalOcean lost our server

#79

Post by fweespeech »

andrewthetechie wrote: post #67
fweespeech wrote: post #49

Earlier quoted context omitted.

They regularly are unable to create new droplets [or other control issues where you can't perform normal functions with normal latency] and/or have full DC outages. Pretty much any use case where I'd use something like DO being unable to create new VMs, etc. is the same as an outage.

Huh, that's pretty odd. I asked why as I work for DO's operations team and I wanted to know what your concerns were. Do you have a specific instance of a problem? What you're describing would be considered a MAJOR outage for us and we have not had one in quite some time.

https://status.digitalocean.com/

* Jan 6th

* Dec 11th

* Dec 4th

* Nov 27th

* Oct 10th

* Oct 2nd

That is annoying enough I'd want to be able to failover if it exceeded 15 minutes. I get it might not be an issue for anyone else but failing over is less complex than mitigating latency issues with droplet creation or whatever.

I get "regularly" to me might mean something different to you but if the 2 DO DCs I was using are hit literally every month with a problem of some kind...it seems regular to me.

Re: DigitalOcean lost our server

#80

Post by aesthetics1 »

mikeash wrote: post #38
chrismartin wrote: post #26

Probably due to optimization of the conversion funnel, it's VERY easy to build things on DigitalOcean without understanding that - unlike many other *-as-a-service - droplets do not include backups and that is your responsibility. As a sysadmin, this is fine since I don't trust a single provider for both prod and backups anyhow, but I fear for those who are just getting started and don't realize this. I agree that DO…

I would really hope that the presence of a "Backups" checkbox on the droplet creation page is enough to tell the (presumably tech-savvy) user that backups don't happen unless you check that box.

That's fine, but they didn't introduce this functionality until recently. One day, I logged into my DO console (I don't have a reason to do this very often), and saw a new backup tab. Older customers might not even be aware.

I don't host anything I couldn't live without though, so I'm not really worried.

Post reply on HN