Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DigitalOcean lost our server

murze.be

Post reply on HN
Print view View original

91–100 of 219 posts

Re: DigitalOcean lost our server

#91

Post by derefr »

vardump wrote: post #82
nailer wrote: post #76

Earlier quoted context omitted.

> You should be designing for failure in "the cloud" DO should be designing for failure: VM storage should be on a SAN. A single physical server failing should not cause data loss. This is basic stuff.

That's a very unreasonable expectation at cloud price levels. Does any cloud provider have SAN backed VMs? If any do, at what price?

That's what AWS's "Elastic Block Storage" is. You can turn it off and just use instance storage (and I personally prefer to, for truly ephemeral nodes), but it increases spawn time since your disk image actually has to get copied over to the VM host machine in that case, rather than just "attached" over EBS.

Re: DigitalOcean lost our server

#92

Post by therealmarv »

No surprise for me. It also happened on DO for me one time. It seemed I've had my machine on a wrong rack/server there. At the end you get what you pay for I would say... you can have luck on your DO server but there is a reason why Rackspace and Amazon still exist nowadays! If you can afford one of the bigger ones go for it.

Re: DigitalOcean lost our server

#95

Post by vinceguidry »

kasey_junk wrote: post #89
vinceguidry wrote: post #70

Earlier quoted context omitted.

Because state left on servers is inevitably unmanaged and will get lost eventually. At my job, I have cron jobs running on the production system from years ago that I have no clue what they do and no time to try to figure it out. I find out when they fail and someone, usually customer service, complains. If the server goes away, then even once I redeploy, I've lost all those cron jobs. Who knows what will happen then…

I think the other point of contention was your term "database". There are lots and lots of systems out there that rely on data that is not stored in a database. The data is the same kinds of things you would store in a databases but they exist as files on a filesystem instead (or in another data storage devices that is usually not referred to as a database). That said you could easily use the same advice for that dat…

> There are lots and lots of systems out there that rely on data that is not stored in a database. The data is the same kinds of things you would store in a databases but they exist as files on a filesystem instead (or in another data storage devices that is usually not referred to as a database).

If you can't afford to lose it, then it needs to be appropriately managed so it can be recreated in the case that it's lost. If you can afford to lose it, then it's not really state and you can forget about it.

Re: DigitalOcean lost our server

#96

Post by lotyrin »

cortesoft wrote: post #39
gdgtfiend wrote: post #4

This seems like a non-issue to me. If you're using an IaaS provider you should be treating the network as volatile from the get-go. This is the reason AWS has things like auto-scaling groups. You should be designing for failure in "the cloud"

It is no different if you are running your own hardware - your server hardware can fail no matter who owns the machine.

Yep, somehow cloud customers have decided that by being in the cloud everything is redundant and durable and has several other magical properties.

At one of my positions we had stupidly put too many eggs in the basket of a single physical machine. Its disk controller failed in a way that it trashed the data volumes. I was unable to convince anyone that "move to Amazon" was not a one-step solution to "how do we make sure this never happens again".

Re: DigitalOcean lost our server

#97

Post by hyperknot »

vardump wrote: post #82
nailer wrote: post #76

Earlier quoted context omitted.

> You should be designing for failure in "the cloud" DO should be designing for failure: VM storage should be on a SAN. A single physical server failing should not cause data loss. This is basic stuff.

That's a very unreasonable expectation at cloud price levels. Does any cloud provider have SAN backed VMs? If any do, at what price?

Not a huge company like DO, but iwStack [1] provides a SAN backed cloud, with selectable KVM/XEN instances, custom ISO and virtual network support. The prices are similar to DO. [1] http://iwstack.com/

Re: DigitalOcean lost our server

#98

Post by vinceguidry »

snuxoll wrote: post #58
vinceguidry wrote: post #53

If you're relying on backups for servers other than your database then you're keeping state on your servers and that's a Bad Thing. You should regularly destroy your own servers and recreate them using your configuration / deployment scripts if the prospect of this happening worries you. Do it before your business starts to rely on it. For database servers you need to have procedures in place to quickly switch the pr…

It's so insanely easy to get started with puppet or ansible I wouldn't start rolling out a product without using them from the get-go at this point. At the very least properly package your software (read: DEB or RPM, not docker containers) so it's easy to reinstall quickly if need be.

Personally, I like platform libraries rather than OS packages. I have several Ruby projects set up as gems. What's neat about that is it allows code reuse across projects.

Re: DigitalOcean lost our server

#99

Post by derefr »

kasey_junk wrote: post #89
vinceguidry wrote: post #70

Earlier quoted context omitted.

Because state left on servers is inevitably unmanaged and will get lost eventually. At my job, I have cron jobs running on the production system from years ago that I have no clue what they do and no time to try to figure it out. I find out when they fail and someone, usually customer service, complains. If the server goes away, then even once I redeploy, I've lost all those cron jobs. Who knows what will happen then…

I think the other point of contention was your term "database". There are lots and lots of systems out there that rely on data that is not stored in a database. The data is the same kinds of things you would store in a databases but they exist as files on a filesystem instead (or in another data storage devices that is usually not referred to as a database). That said you could easily use the same advice for that dat…

"Files on a file system", on a server dedicated to holding files, could be termed an object store. S3 is a database.

Re: DigitalOcean lost our server

#100

Post by vardump »

derefr wrote: post #91
vardump wrote: post #82

Earlier quoted context omitted.

That's a very unreasonable expectation at cloud price levels. Does any cloud provider have SAN backed VMs? If any do, at what price?

That's what AWS's "Elastic Block Storage" is. You can turn it off and just use instance storage (and I personally prefer to, for truly ephemeral nodes), but it increases spawn time since your disk image actually has to get copied over to the VM host machine in that case, rather than just "attached" over EBS.

Then why EBS failure rate is several orders of magnitude higher than in SAN deployments? A SAN provider would be quickly out of business with 0.1-0.5% annual failure rate.

SAN reliability ratings start at 99.999%.

Post reply on HN