Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DigitalOcean lost our server

murze.be

Post reply on HN
Print view View original

131–140 of 219 posts

Re: DigitalOcean lost our server

#131

Post by zandor »

notwhereyouare wrote: post #50
toomuchtodo wrote: post #41

Earlier quoted context omitted.

If you're running in "the cloud", you should _always_ be able to destroy your instances and have them auto-rebuild from configuration management and a persistent object storage system (git repo + S3 or Google Nearline).

do you have any guides or tutorials I could possibly follow to have a setup like that? I'm working on configuring a site and would like the ability to 1) scale as quickly as possibly or 2) rebuild in event of total failure. I just don't even know where to begin. Is this something I could stand up locally and push to like s3, spin up a new server and install 1 thing and have it pull the configs and software installs d…

As a couple of the other users have mentioned, Ansible is a nice starting point. Check out https://sysadmincasts.com/episodes/43-19-minutes-with-ansibl... for a quick intro.

Re: DigitalOcean lost our server

#132

Post by mwcampbell »

kentonv wrote: post #63

I think Google Compute Engine does the right thing here: by default it uses "persistent disks" (network-attached redundant/highly-available block devices) for all disks. The only case I've heard of where persistent disk data was lost was a few acknowledged writes occurring just before an unusual lightning-induced power outage: https://status.cloud.google.com/incident/compute/15056 For added protection, you can take r…

I've been suspicious of network-attached block storage ever since the Amazon EBS cascading failure of April 2011 and these three blog posts that Joyent did in response:

https://www.joyent.com/blog/on-cascading-failures-and-amazon...

https://www.joyent.com/blog/magical-block-store-when-abstrac...

https://www.joyent.com/blog/network-storage-in-the-cloud-del...

Besides Joyent, I think DigitalOcean and Vultr do it right, with RAID-backed local storage and a backup option.

Re: DigitalOcean lost our server

#133

Post by nikdow »

Tell me if I'm doing something wrong: my two EC2 servers both use EBS including for the root device, so I'm not using any device-attached storage. If Amazon "lost" my instance, it would appear to me to be just a reboot.

And yes, I do a nightly backup of databases, webroot, /etc and some other directories.

Re: DigitalOcean lost our server

#134

Post by tomphoolery »

That really stinks, but this should be a warning to you that in the future, you should have some kind of redundancy plan so data does not get lost. Even if Amazon decided to terminate all of my app instances right now, I would just need to rebuild them. No data is actually lost, and furthermore backups are made frequently so that if a server decides to explode one day, we're still covered. DigitalOcean droplets are really not meant to be holding your database and application server...you shouldn't have experienced "data loss" by losing a droplet because your database should not have been hosted on a droplet.

Re: DigitalOcean lost our server

#135

Post by hinkley »

vinceguidry wrote: post #53

If you're relying on backups for servers other than your database then you're keeping state on your servers and that's a Bad Thing. You should regularly destroy your own servers and recreate them using your configuration / deployment scripts if the prospect of this happening worries you. Do it before your business starts to rely on it. For database servers you need to have procedures in place to quickly switch the pr…

And the nice thing about DO and some of their cohort is that they charge by the hour for servers. Fire up new droplets and install your upgrades on them, smoke test the system, swap your routing over to the new machines, back up the old droplets (just in case you missed something) and and then destroy them.

If that takes 4 hours you've hardly spent any money at all, and you're running your cluster the way a lot of people already do.

Re: DigitalOcean lost our server

#136

Post by tzakrajs »

tomphoolery wrote: post #134

That really stinks, but this should be a warning to you that in the future, you should have some kind of redundancy plan so data does not get lost. Even if Amazon decided to terminate all of my app instances right now, I would just need to rebuild them. No data is actually lost, and furthermore backups are made frequently so that if a server decides to explode one day, we're still covered. DigitalOcean droplets are r…

Which is the irony of the entire post because the tldr was: "our DO instance was lost, fortunately we keep multiple backups of the data so we were able to restore without any data loss". It seems more like a passive aggressive note against Digital Ocean.

Re: DigitalOcean lost our server

#137

Post by hinkley »

vinceguidry wrote: post #70
lqdc13 wrote: post #59

Earlier quoted context omitted.

What? Why is it a bad thing? I guess it depends on someone's definition of a database.

Because state left on servers is inevitably unmanaged and will get lost eventually. At my job, I have cron jobs running on the production system from years ago that I have no clue what they do and no time to try to figure it out. I find out when they fail and someone, usually customer service, complains. If the server goes away, then even once I redeploy, I've lost all those cron jobs. Who knows what will happen then…

The other reason to keep the DB separate is that they have different IO patterns that are not always complementary, and it's easier to gather diagnostic metrics if the data isn't munged. In the days of physical servers you'd also choose different hardware for the DB machine.

Configuration capture is a big thing, and it separates the adults from the kids. I've had hardware failure of my dev box twice in my career, and when I tell certain people that I have to rebuild my machine you can see their faces blanche. These are the people I know I haven't converted yet.

Everything important, except for the handful of things I'm actively investigating, is always stored on shared machines. Version control. Wiki. Some sort of artifact repository. The instructions for getting an environment set up with version X of our software are also stored in one of those places, and have been vetted by every new developer and some of the QA team. I can have my machine back up and running in a couple of hours, and most of that is waiting for downloads, if we didn't have the presence to store copies locally.

Why do I care about this stuff? Seems sort of OCD on the face of it, and maybe you're right. But sooner or later you're going to have a high severity issue in production that should be an all-hands-on-deck affair, and if you haven't done this work, losing a hard drive on a dev laptop will be the least of your worries. Everyone busy doing work on X+1 needs to be able to get back to version X and all of its dependencies in under an hour, and by themselves, because the people who could help them are most likely on the front lines of fixing the bug as fast as possible.

What's more, someone probably needs to get versions X-1 and X-2 running to figure out if you need to warn people using older versions. So that's getting people running 3 or 4 versions of your software autonomously, so that you can identify repro steps, long and short term mitigation strategies, verify that they work, formulate a bulletin and provide patches for people. Not only do you need to get your configuration captured/documented, you need it captured 2-3 versions of your software ago, which means you need to start thinking about this stuff pretty early in your project.

Re: DigitalOcean lost our server

#138

Post by RubyPinch »

Fizzadar wrote: post #123
gdgtfiend wrote: post #4

This seems like a non-issue to me. If you're using an IaaS provider you should be treating the network as volatile from the get-go. This is the reason AWS has things like auto-scaling groups. You should be designing for failure in "the cloud"

I have to disagree - DO's "cloud servers" are equivalent to virtual private servers, which you would never expect to loose in this manner from other providers. The lack of explanation is what worries me most - it leads me to think this might have been a case of "we forgot to replace a bad drive, then the second in the pair failed".

I'd expect to lose them.

Fire happens, floods happen, electrical faults happen, mistakes happen. I can't really expect any human or humans to be perfect.

Re: DigitalOcean lost our server

#139

Post by kentonv »

mwcampbell wrote: post #132
kentonv wrote: post #63

I think Google Compute Engine does the right thing here: by default it uses "persistent disks" (network-attached redundant/highly-available block devices) for all disks. The only case I've heard of where persistent disk data was lost was a few acknowledged writes occurring just before an unusual lightning-induced power outage: https://status.cloud.google.com/incident/compute/15056 For added protection, you can take r…

I've been suspicious of network-attached block storage ever since the Amazon EBS cascading failure of April 2011 and these three blog posts that Joyent did in response: https://www.joyent.com/blog/on-cascading-failures-and-amazon... https://www.joyent.com/blog/magical-block-store-when-abstrac... https://www.joyent.com/blog/network-storage-in-the-cloud-del... Besides Joyent, I think DigitalOcean and Vultr do it right,…

I'm having trouble unpacking useful information out of these rambly blog posts. It sounds like the basic complaint is that abstractions add complexity, which is more trouble when it goes wrong.

I agree that if you have a large-scale sophisticated operation you will probably want to choose local disks and handle availability your own way (and GCE provides those options). But for small-scale operations that can't justify as much engineering, abstractions like persistent disks and automatic migration will save a ton of time and avoid data loss. Ironically, Digital Ocean seems to target this smaller scale, yet they've set the wrong defaults for their target market.

Re: DigitalOcean lost our server

#140

Post by hinkley »

justizin wrote: post #85

"Luckily, we made the decision at Spatie to host every site on it’s own droplet, so only one site was affected." I think that's a poor lesson learned here. Were this me, I would have said: "Luckily, all of our sites run on several servers, access data in a shared, replicated cluster, and a small shell script I wrote kept me from writing this entire blog post." IaaS has only surfaced what has always been true: your da…

My sentiments exactly. If they lost their db replica that would get the blood pumping but the customers would never see an actual outage.
Post reply on HN