Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

DigitalOcean lost our server

murze.be

Post reply on HN
Print view View original

51–60 of 219 posts

Re: DigitalOcean lost our server

#51

Post by pdabbadabba »

phamilton wrote: post #2

I'm glad the author didn't turn this into a negative review of DO. "What will you do when your server is lost?" is exactly the right question to learn from a scenario like this.

Agreed. There may also be a customer service lesson in that refund email. Typically, your customer is not happy with you even after they receive a refund, so exclamation points and language like "Booyah!" is really not a good idea for that sort of message. It's always going to sound a little bit tone deaf -- and very tone deaf when the refund was for a major incident like the loss of an entire server.

Keep it professional.

Re: DigitalOcean lost our server

#52

Post by protomyth »

andrewthetechie wrote: post #48
protomyth wrote: post #34

Earlier quoted context omitted.

I'd be curious what their RAID is setup. I agree and run ZFS for our data storage.

Check out https://www.digitalocean.com/community/questions/how-did-do-... for your answer. It's a bit old, but still true.

Is it single RAID controller or is there a redundant setup?

Re: DigitalOcean lost our server

#53

Post by vinceguidry »

If you're relying on backups for servers other than your database then you're keeping state on your servers and that's a Bad Thing. You should regularly destroy your own servers and recreate them using your configuration / deployment scripts if the prospect of this happening worries you. Do it before your business starts to rely on it.

For database servers you need to have procedures in place to quickly switch the production application over to a database you just spun up and migrated the data from a recent backup to in order to test your data backups. You want this to happen as smoothly as possible in the case of a failure. Keep data backups in three different places and test your procedure on all of them.

On a production system, do not keep the database on the same server as the application.

Re: DigitalOcean lost our server

#54

Post by dubcanada »

pdabbadabba wrote: post #51
phamilton wrote: post #2

I'm glad the author didn't turn this into a negative review of DO. "What will you do when your server is lost?" is exactly the right question to learn from a scenario like this.

Agreed. There may also be a customer service lesson in that refund email. Typically, your customer is not happy with you even after they receive a refund, so exclamation points and language like "Booyah!" is really not a good idea for that sort of message. It's always going to sound a little bit tone deaf -- and very tone deaf when the refund was for a major incident like the loss of an entire server. Keep it profess…

It's the standard DO credit email, which typically is from referring friends or doing something else to acquire credit. Not getting a refund because DO accidentally deleted your droplet.

Though their should be another avenue for such things.

Re: DigitalOcean lost our server

#56

Post by chc »

watty wrote: post #47
gdgtfiend wrote: post #4

This seems like a non-issue to me. If you're using an IaaS provider you should be treating the network as volatile from the get-go. This is the reason AWS has things like auto-scaling groups. You should be designing for failure in "the cloud"

Isn't "non-issue" a big of an exaggeration? If the dry cleaners lost my clothes, the bank lost my money, a valet lost my car, or gmail lost my inbox I'd be angry.

Yes, but those aren't expected outcomes when using those services. That's the difference gdgtfiend was pointing out — you should expect cloud servers to sometimes go away.

Re: DigitalOcean lost our server

#57

Post by endymi0n »

gdgtfiend wrote: post #4

This seems like a non-issue to me. If you're using an IaaS provider you should be treating the network as volatile from the get-go. This is the reason AWS has things like auto-scaling groups. You should be designing for failure in "the cloud"

While my first intuition was to agree with you, there's certainly an upcoming generation of developers who have never operated their own root servers and the abstraction level in the cloud nowadays is so high it makes you easily forget that "Droplets" are just VMs are just servers running software. On the other hand, hardware reliability has increased in recent years due to RAID, fully redundant networking & power adaptors, you name it.

These facts combined certainly make it easy for the younger generation to forget that redundancy and backup still don't replace each other (never did).

Re: DigitalOcean lost our server

#58

Post by snuxoll »

vinceguidry wrote: post #53

If you're relying on backups for servers other than your database then you're keeping state on your servers and that's a Bad Thing. You should regularly destroy your own servers and recreate them using your configuration / deployment scripts if the prospect of this happening worries you. Do it before your business starts to rely on it. For database servers you need to have procedures in place to quickly switch the pr…

It's so insanely easy to get started with puppet or ansible I wouldn't start rolling out a product without using them from the get-go at this point. At the very least properly package your software (read: DEB or RPM, not docker containers) so it's easy to reinstall quickly if need be.

Re: DigitalOcean lost our server

#59

Post by lqdc13 »

vinceguidry wrote: post #53

If you're relying on backups for servers other than your database then you're keeping state on your servers and that's a Bad Thing. You should regularly destroy your own servers and recreate them using your configuration / deployment scripts if the prospect of this happening worries you. Do it before your business starts to rely on it. For database servers you need to have procedures in place to quickly switch the pr…

What? Why is it a bad thing? I guess it depends on someone's definition of a database.

Re: DigitalOcean lost our server

#60

Post by z3t4 »

helper wrote: post #27
z3t4 wrote: post #22

I don't know how to write this without upsetting quite a lot of people, but RAID!???! I run Redundant disk controllers and hard drives on ZFS. And it's cheaper than a decent RAID system. First thing I do with a new server is disabling RAID.

How many servers? 1, 100, 1000,10,000? Where are you hosting them, in your home or in a datacenter? How much are you paying for bandwidth and power? What SLAs do you have in place for network and power? How much do you pay for your remote hands? I hope you see what I'm getting at. While I love ZFS and think self-hosting can make a lot of economic sense, without knowing all the variables you can't really make a fair c…

I understand there are huge benefits of cloud hosting. But why are they running RAID? Why can't my setup be used at scale!?
Post reply on HN