Live data from Hacker News

DigitalOcean lost our server

murze.be

101–110 of 219 posts

Re: DigitalOcean lost our server

#101
I hadn't heard of the backup providers mentioned in the article. Does anyone have experience with those or with other recommended solutions?

I can re-implement the infrastructure of my VMs fairly easily (thank you Ansible), but backing up content outside of the provider's built-in options is something I haven't played with yet, and would obviously be the crucial piece.

Re: DigitalOcean lost our server

#102
post #91
post #82

Earlier quoted context omitted.

That's a very unreasonable expectation at cloud price levels. Does any cloud provider have SAN backed VMs? If any do, at what price?

That's what AWS's "Elastic Block Storage" is. You can turn it off and just use instance storage (and I personally prefer to, for truly ephemeral nodes), but it increases spawn time since your disk image actually has to get copied over to the VM host machine in that case, rather than just "attached" over EBS.

On AWS, an EBS volume is only usable from one availability zone. You still need to use application-level replication to get geographic redundancy for important data, and when you have that, EBS just lets you be lazy rather than eager about copying a snapshot to local instances.

Re: DigitalOcean lost our server

#103
post #82

Earlier quoted context omitted.

That's a very unreasonable expectation at cloud price levels. Does any cloud provider have SAN backed VMs? If any do, at what price?

Not a huge company like DO, but iwStack [1] provides a SAN backed cloud, with selectable KVM/XEN instances, custom ISO and virtual network support. The prices are similar to DO. [1] http://iwstack.com/

That's pretty damn interesting for more traditional hosting type usage.

Seems like they sell 50 GB of real SAN backed storage for just 3.6€ per month.

How can they afford to be so cheap?

Re: DigitalOcean lost our server

#104
post #99

Earlier quoted context omitted.

I think the other point of contention was your term "database". There are lots and lots of systems out there that rely on data that is not stored in a database. The data is the same kinds of things you would store in a databases but they exist as files on a filesystem instead (or in another data storage devices that is usually not referred to as a database). That said you could easily use the same advice for that dat…

"Files on a file system", on a server dedicated to holding files, could be termed an object store. S3 is a database.

Agreed. But that is a increasingly rarer understanding of the term database.

Re: DigitalOcean lost our server

#105
post #91

Earlier quoted context omitted.

That's what AWS's "Elastic Block Storage" is. You can turn it off and just use instance storage (and I personally prefer to, for truly ephemeral nodes), but it increases spawn time since your disk image actually has to get copied over to the VM host machine in that case, rather than just "attached" over EBS.

Then why EBS failure rate is several orders of magnitude higher than in SAN deployments? A SAN provider would be quickly out of business with 0.1-0.5% annual failure rate. SAN reliability ratings start at 99.999%.

Just because it's a SAN doesn't mean a given abstract block device from it is backed by RAID. It's literally just a multiplexed and QoSed network-attached storage cluster.

I actually prefer the lower-level abstraction: if you want a lower failure rate (or higher speed), you can RAID together attached EBS volumes yourself on the client side and work with the resultant logical volume.

Re: DigitalOcean lost our server

#106
Two things to learn from this:

1. Never have a single point of failure. Relying on DO for Server+Backups is putting your eggs on one basket.

2. Your server state should be programmable. This is not quite easy for complex configurations. But today, DO has an API, we have Docker, and quite modern deployment tools.

Here is setup:

1. Github for the server state. Basically, a repository to configure and deploy my infrastructure.

2. Enable DO backups in case I mess up something and want a quick come-back.

3. File backups through Tarsnap. Since I use Docker, I have a volume container. Backup the volume container with Tarsnap.

Re: DigitalOcean lost our server

#108
post #38

Earlier quoted context omitted.

I would really hope that the presence of a "Backups" checkbox on the droplet creation page is enough to tell the (presumably tech-savvy) user that backups don't happen unless you check that box.

That's fine, but they didn't introduce this functionality until recently. One day, I logged into my DO console (I don't have a reason to do this very often), and saw a new backup tab. Older customers might not even be aware. I don't host anything I couldn't live without though, so I'm not really worried.

The functionality has been there for a long time. It's possible the checkbox on droplet creation is new, and it used to require activating backups afterwards, I don't recall.

Re: DigitalOcean lost our server

#109
post #58

Earlier quoted context omitted.

It's so insanely easy to get started with puppet or ansible I wouldn't start rolling out a product without using them from the get-go at this point. At the very least properly package your software (read: DEB or RPM, not docker containers) so it's easy to reinstall quickly if need be.

Personally, I like platform libraries rather than OS packages. I have several Ruby projects set up as gems. What's neat about that is it allows code reuse across projects.

So go ahead and vendor them in, outside of inclusion in distribution software packages (with exceptions) there's nothing stopping you from including your own dependencies. Packaging your application properly makes deployment a hell of a lot easier and less error-prone.

Re: DigitalOcean lost our server

#110

This seems like a non-issue to me. If you're using an IaaS provider you should be treating the network as volatile from the get-go. This is the reason AWS has things like auto-scaling groups. You should be designing for failure in "the cloud"

It is no different if you are running your own hardware - your server hardware can fail no matter who owns the machine.

This is so true, I wish there was a way to make this more noticeable (flashing red lights on the order forms, regular email warnings about the single point of failure,etc.). This simple fact is taught in entry level system administration courses (RAID, Disaster Recovery Plans, Replication, High Availability, Fault Tolerance, etc.) but in reality some seem to not plan for it when moving to cloud services. I always suggest people set their apps up to be able to run out of multiple availability zones in case one goes down if it is very important to them. For those that are not able to I will suggest they at least setup replication to another server within the same data center and an offsite location in case their server has hardware failure, an account gets locked or other possible situation to help take of the just in case scenarios that occur to everyone.
Post reply on HN