Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why the “Digital Ocean killed my company” incident scares the hell out of me

blog.checklyhq.com

Post reply on HN
Print view View original

81–90 of 185 posts

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#81

Post by neovive »

freehunter wrote: post #12

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…

If you don't mind me asking, what is the preferred method to backup to S3? Is it possible to scp or rsync a mysqldump to S3 or do you install aws tools on DO and run aws s3 cp as a scheduled job?

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#82

Post by brianwawok »

olau wrote: post #44
Someone1234 wrote: post #31

Earlier quoted context omitted.

Because hypothetically your instance should be hardware agnostic. If the physical hardware dies, it should automatically migrate to another physical server at their data center without your intervention. It will only look like an unexpected restart from your perspective. That's something worth paying for. It would be more comparable to two servers at Hetzner with rapid failover. But even that is more involved since y…

That's wishful thinking. If there's state in that virtual machine, it's probably either stored on the physical host or in a SAN. If it's in the physical host, it has to be fished out of that machine or restored from a backup. If it's a SAN, you can lose your virtual machine if the SAN goes down. I've seen both happen. Actually, a single machine with RAID is surprisingly stable. A provider like Hetzner can switch out…

Thats not wishful thinking. I did it from day one with my startup with GKE. You can blow up any of my servers, and a new one gets spun up and dockers deployed while the glitch is load balanced around. All for only a few hours of work.

GKE has given me the infrastructure of a 10 million dollar company, with me as the IT guy doing 1h or less a week of IT work.

I would not be able to do this on a $35 hertzer machine.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#83

Post by siffland »

If you are running a business then always have a good disaster recovery strategy. Split services between cloud providers and make sure your backups are good and offsite. You can do this all with opensource software if you dont want to pay for anything too (and if you have the staff that can do it).

If your business can be "killed" by your provider losing your data you have set yourself up to fail. I am not saying the provider is not to blame, but the burden should also be on the customer for not protecting themselves, what if DO burnt down?

We have yearly DR exercises, we have to bring up about 40 of our 120 production instances, it is a difficult task, but we know we can do it.

Never depend on your provider to save you, always make sure you have planed this out.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#84

Post by thaumaturgy »

olau wrote: post #44
Someone1234 wrote: post #31

Earlier quoted context omitted.

Because hypothetically your instance should be hardware agnostic. If the physical hardware dies, it should automatically migrate to another physical server at their data center without your intervention. It will only look like an unexpected restart from your perspective. That's something worth paying for. It would be more comparable to two servers at Hetzner with rapid failover. But even that is more involved since y…

That's wishful thinking. If there's state in that virtual machine, it's probably either stored on the physical host or in a SAN. If it's in the physical host, it has to be fished out of that machine or restored from a backup. If it's a SAN, you can lose your virtual machine if the SAN goes down. I've seen both happen. Actually, a single machine with RAID is surprisingly stable. A provider like Hetzner can switch out…

I receive on average about one notification from DigitalOcean each month that there's a problem with the host for one of my VMs and there may be some downtime while they migrate it to a new host. Usually the downtime is about the same as a reboot, sometimes there's no downtime at all.

This works in part because both Xen and KVM hypervisors (and possibly others) support live migrations, so it's altogether false that virtualization does not increase stability. Both DigitalOcean and Linode use KVM behind the scenes.

So, at lower cost than rented hardware, customers get staff whose job it is to constantly monitor systems for hardware failures and deal with them proactively in a way that minimizes downtime.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#86

Post by true_religion »

hadlock wrote: post #67
jwr wrote: post #22

Earlier quoted context omitted.

> But I still do business with DO because it costs ~half the price of a comparable EC2 instance I'm curious why people who don't have massive scaling and variability issues choose DO or AWS for their hosting. For 34€/month, Hetzner will rent you a physical server (i7-6700, 64GB RAM, 2x512GB SSD, 1Gbit/s networking). That's a monster of a machine and can run most sites out there. And if you need more oomph and better…

Most $10MM ARR startups are running on two medium T class aws instances behind a load balancer at $35/mo plus another small server for accessories (redis, message queue, etc) and then a sql backend. That's something like $200/mo for infrastructure, you never need to maintain the hardware and you have the option to autoscale at any time to support $100MM ARR. Plus you get access to backups and all the other tools and…

You’re telling me that 10 million dollar “startup” companies can’t run 1 server without incurring “years” of tech debt? Sorry, that just sounds unbelievable.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#88

Post by social_quotient »

hadlock wrote: post #67
jwr wrote: post #22

Earlier quoted context omitted.

> But I still do business with DO because it costs ~half the price of a comparable EC2 instance I'm curious why people who don't have massive scaling and variability issues choose DO or AWS for their hosting. For 34€/month, Hetzner will rent you a physical server (i7-6700, 64GB RAM, 2x512GB SSD, 1Gbit/s networking). That's a monster of a machine and can run most sites out there. And if you need more oomph and better…

Most $10MM ARR startups are running on two medium T class aws instances behind a load balancer at $35/mo plus another small server for accessories (redis, message queue, etc) and then a sql backend. That's something like $200/mo for infrastructure, you never need to maintain the hardware and you have the option to autoscale at any time to support $100MM ARR. Plus you get access to backups and all the other tools and…

$200/mm for a 10MM ARR company. Can you add more info here. What sort of app/company are we taking about here? Genuinely curious

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#89

Post by wiredfool »

noir_lord wrote: post #63
freehunter wrote: post #12

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…

FWIW I do the same I have backups of everything. All that said in close to a decade (actually might be a decade) I've still yet to have any issue with linode (tempting the computer gods here) or their backups. I remember when DO was first starting out and they used to knock their own stuff offline updating router tables. Ever since I have a jaundiced view of their capabilities.

N=1

Linode backups start to fail when you have more then 3 million files on a drive.

When debugging this, they told me the reason, I did a snapshot that worked, and then they turned off backups because they were failing. They didn’t highlight that backups were now off.

Restores on the larger machines can take 5+ hours, and they will often report that the restore failed if you are restoring a drive that contains docker’s pipe files.

Ymmv, but I’m trying to get off linode.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#90

Post by metildaa »

phonon wrote: post #74
barkingcat wrote: post #30

Earlier quoted context omitted.

Because DO has their $5 and $10 / month plan, and that's really enough for me.

https://aws.amazon.com/lightsail/pricing/?opdp1=pricing ?

The CPU allocations on Lightsail are anemic compared to DO.
Post reply on HN