Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DigitalOcean lost our server

murze.be

Post reply on HN
Print view View original

81–90 of 219 posts

Re: DigitalOcean lost our server

#81

Post by [deleted] »

dubcanada wrote: post #54
pdabbadabba wrote: post #51

Earlier quoted context omitted.

Agreed. There may also be a customer service lesson in that refund email. Typically, your customer is not happy with you even after they receive a refund, so exclamation points and language like "Booyah!" is really not a good idea for that sort of message. It's always going to sound a little bit tone deaf -- and very tone deaf when the refund was for a major incident like the loss of an entire server. Keep it profess…

It's the standard DO credit email, which typically is from referring friends or doing something else to acquire credit. Not getting a refund because DO accidentally deleted your droplet. Though their should be another avenue for such things.

[deleted]

Re: DigitalOcean lost our server

#82

Post by vardump »

nailer wrote: post #76
gdgtfiend wrote: post #4

This seems like a non-issue to me. If you're using an IaaS provider you should be treating the network as volatile from the get-go. This is the reason AWS has things like auto-scaling groups. You should be designing for failure in "the cloud"

> You should be designing for failure in "the cloud" DO should be designing for failure: VM storage should be on a SAN. A single physical server failing should not cause data loss. This is basic stuff.

That's a very unreasonable expectation at cloud price levels.

Does any cloud provider have SAN backed VMs? If any do, at what price?

Re: DigitalOcean lost our server

#83

Post by random_upvoter »

nailer wrote: post #76
gdgtfiend wrote: post #4

This seems like a non-issue to me. If you're using an IaaS provider you should be treating the network as volatile from the get-go. This is the reason AWS has things like auto-scaling groups. You should be designing for failure in "the cloud"

> You should be designing for failure in "the cloud" DO should be designing for failure: VM storage should be on a SAN. A single physical server failing should not cause data loss. This is basic stuff.

DO should do what's written in their terms of service, and the customer should read them carefully.

Re: DigitalOcean lost our server

#84

Post by pdabbadabba »

dubcanada wrote: post #54
pdabbadabba wrote: post #51

Earlier quoted context omitted.

Agreed. There may also be a customer service lesson in that refund email. Typically, your customer is not happy with you even after they receive a refund, so exclamation points and language like "Booyah!" is really not a good idea for that sort of message. It's always going to sound a little bit tone deaf -- and very tone deaf when the refund was for a major incident like the loss of an entire server. Keep it profess…

It's the standard DO credit email, which typically is from referring friends or doing something else to acquire credit. Not getting a refund because DO accidentally deleted your droplet. Though their should be another avenue for such things.

Hmm, maybe that's what they're thinking. But I wonder if it's really true that a refer-a-friend or similar, happy scenario is the "typical" cause for a refund. Certainly in my life I've received far more refunds due to company mistakes than than credits due to the happier reasons you mention.

And when in doubt, if you're refund system is not capable of discriminating between the two, it seems to me wiser to err on the side of a less jubilant message.

Re: DigitalOcean lost our server

#85

Post by justizin »

"Luckily, we made the decision at Spatie to host every site on it’s own droplet, so only one site was affected."

I think that's a poor lesson learned here. Were this me, I would have said:

"Luckily, all of our sites run on several servers, access data in a shared, replicated cluster, and a small shell script I wrote kept me from writing this entire blog post."

IaaS has only surfaced what has always been true: your data lives on little physical things that are screwed into a thing and goes through a controller that could fuck up due to cosmic rays.

Do better for your customers.

Re: DigitalOcean lost our server

#86

Post by chrismartin »

apocalyptic0n3 wrote: post #65

Does anyone have any good backup solutions to mitigate this? My agency uses a script we wrote in-house to more-or-less rsync our data to an AWS instance, but it's always seemed a poor way to handle it. I'm using DO's backup service for my personal site which was always supposed to be a temporary solution (the timing of the backups is inconvenient as mentioned by the article). A better solution would be wonderful.

This isn't exactly turnkey but for my personal servers, I use rsync.net (ask for the HN discount) and attic [0]. It's easy to script and call with cron. (Of course, this is still somewhat DIY, and any backup solution is only as good as your monitoring + restore testing.)

[0] https://attic-backup.org/

Re: DigitalOcean lost our server

#87

Post by toomuchtodo »

notwhereyouare wrote: post #50
toomuchtodo wrote: post #41

Earlier quoted context omitted.

If you're running in "the cloud", you should _always_ be able to destroy your instances and have them auto-rebuild from configuration management and a persistent object storage system (git repo + S3 or Google Nearline).

do you have any guides or tutorials I could possibly follow to have a setup like that? I'm working on configuring a site and would like the ability to 1) scale as quickly as possibly or 2) rebuild in event of total failure. I just don't even know where to begin. Is this something I could stand up locally and push to like s3, spin up a new server and install 1 thing and have it pull the configs and software installs d…

> I just don't even know where to begin. Is this something I could stand up locally and push to like s3, spin up a new server and install 1 thing and have it pull the configs and software installs down?

Yes. You could use shell scripts, Ansible, Salt, Puppet, whatever. I'm fond of Ansible or shell scripts, depending on complexity.

Start here: http://www.opsschool.org/en/latest/config_management.html

Re: DigitalOcean lost our server

#88

Post by nailer »

random_upvoter wrote: post #83
nailer wrote: post #76

Earlier quoted context omitted.

> You should be designing for failure in "the cloud" DO should be designing for failure: VM storage should be on a SAN. A single physical server failing should not cause data loss. This is basic stuff.

DO should do what's written in their terms of service, and the customer should read them carefully.

> DO should do what's written in their terms of service

Agreed.

> the customer should read them carefully.

Sure, but they shouldn't need to, as Digital Ocean should set expectations clearly.

Keep in mind Digital Ocean's : Simple Cloud Infrastructure. Being surprised because there's an unsafe default hidden in a document somewhere didn't work out for MongoDB and it won't work out for DO.

Re: DigitalOcean lost our server

#89

Post by kasey_junk »

vinceguidry wrote: post #70
lqdc13 wrote: post #59

Earlier quoted context omitted.

What? Why is it a bad thing? I guess it depends on someone's definition of a database.

Because state left on servers is inevitably unmanaged and will get lost eventually. At my job, I have cron jobs running on the production system from years ago that I have no clue what they do and no time to try to figure it out. I find out when they fail and someone, usually customer service, complains. If the server goes away, then even once I redeploy, I've lost all those cron jobs. Who knows what will happen then…

I think the other point of contention was your term "database". There are lots and lots of systems out there that rely on data that is not stored in a database. The data is the same kinds of things you would store in a databases but they exist as files on a filesystem instead (or in another data storage devices that is usually not referred to as a database).

That said you could easily use the same advice for that data. But in some cases the application itself needs to be on the same server as the database. Think if you were building postgres for instance...

Post reply on HN