Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why the “Digital Ocean killed my company” incident scares the hell out of me

blog.checklyhq.com

Post reply on HN
Print view View original

41–50 of 185 posts

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#41

Post by vorticalbox »

jwr wrote: post #22
freehunter wrote: post #12

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…

> But I still do business with DO because it costs ~half the price of a comparable EC2 instance I'm curious why people who don't have massive scaling and variability issues choose DO or AWS for their hosting. For 34€/month, Hetzner will rent you a physical server (i7-6700, 64GB RAM, 2x512GB SSD, 1Gbit/s networking). That's a monster of a machine and can run most sites out there. And if you need more oomph and better…

If you use a lambda on aws you get 1 million requests for free each month and after that factions of a penny per invoke.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#42

Post by scarface74 »

EdwardDiego wrote: post #21
scarface74 wrote: post #19

Earlier quoted context omitted.

You have 24 hours to respond and why would you have a business and not be on at least their business support plan?

That's a very good question for their CTO. Maybe we should ask him on Twitter.

I’m referring to AWS’s policy compared to DO’s.

If he were using AWS or Azure and he had the same set of issues, I wouldn’t blame him at all. But he is using a third rate cloud provider because it was cheaper and then he acts surprise that they don’t have the same level of competence and support as AWS or Azure.

Yeah I purposefully left out GCP. I wouldn’t trust their support anymore than DO.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#43

Post by TokyoKid »

I have the opposite problem: DO seems to keep my stuff "up" and keeps charging me. I go and shut stuff down, delete droplets, etc, then months later they warn me about my invoices adding up at a rate of $80/mo. At this point I am suspended, but apparently still being charged. So I owe them hundreds of dollars despite repeatedly shutting down and deleting all droplets.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#44

Post by olau »

Someone1234 wrote: post #31
jwr wrote: post #22

Earlier quoted context omitted.

> But I still do business with DO because it costs ~half the price of a comparable EC2 instance I'm curious why people who don't have massive scaling and variability issues choose DO or AWS for their hosting. For 34€/month, Hetzner will rent you a physical server (i7-6700, 64GB RAM, 2x512GB SSD, 1Gbit/s networking). That's a monster of a machine and can run most sites out there. And if you need more oomph and better…

Because hypothetically your instance should be hardware agnostic. If the physical hardware dies, it should automatically migrate to another physical server at their data center without your intervention. It will only look like an unexpected restart from your perspective. That's something worth paying for. It would be more comparable to two servers at Hetzner with rapid failover. But even that is more involved since y…

That's wishful thinking.

If there's state in that virtual machine, it's probably either stored on the physical host or in a SAN. If it's in the physical host, it has to be fished out of that machine or restored from a backup. If it's a SAN, you can lose your virtual machine if the SAN goes down.

I've seen both happen.

Actually, a single machine with RAID is surprisingly stable. A provider like Hetzner can switch out a faulty disk in less than five minutes, or switch a faulty motherboard/power supply in less than half an hour.

Virtualization on top of this does not increase stability, in my experience.

Now, some cloud providers do have more sophisticated distributed systems that do not have single points of failure, and that's a completely different story.

Of course, the software itself is a source of correlated failures, so even there you should never rely on a single cloud vendor.

There's a poster here on this site who commented some months ago that he had for years rented three servers, each on a different continent, each from a different provider, and never had downtime. That's engineering.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#45

Post by dmix »

zamu wrote: post #20
dmix wrote: post #17

Earlier quoted context omitted.

The article addresses this in the "Why not just do x?" section. > - Why were you only hosting one JUST ONE cloud provider? > - Why didn't you have backups outside of JUST ONE cloud provider? Telling them to just use two cloud hosting providers sounds easy on paper but when you're a cash-strapped startup it's a significant ask. Especially if you need to have the same architecture replicated across both, in order not t…

Then save it locally as well? There really isn't an excuse to having DO or any provider being your sole backup.

Why is everyone talking about backups and not the actual infrastructure?

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#46

Post by aquaticsunset »

the_common_man wrote: post #29

Was there a post mortem by DO?

This was linked above

https://blog.digitalocean.com/an-update-on-last-weeks-custom...

It doesn't explain any detail behind why there was radio silence until social media support stepped in, which is what I'm very curious about, but it does have a timeline of events and an apology.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#47

Post by Operyl »

When I joined Digital Ocean, back when it was 2-3 months old, I hit a KVM bug in their stack with my network traffic that caused them to rate limit my droplet to 1Kbit/s because it kept taking the host machine down I guess(?), and then them ultimately terminating it without a ticket asking me to even investigate. That was enough to pretty much never consider using Digital Ocean for anything “production” ever again. They didn’t terminate my account, and apologized with credit, but wow.

Another thought: That was also back when support responded in a reasonable time frame, and they actually spoke on IRC and everything too.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#48

Post by sieabahlpark »

crazygringo wrote: post #24
cm2187 wrote: post #3

Earlier quoted context omitted.

And have a separate DNS provider.

No reason to. If your cloud provider does your DNS and you switch to a different cloud provider... ...then just set your registrar to point to the new DNS provider. You've still got total control.

More about the record tables probably

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#49

Post by eyeinthepyramid »

dmix wrote: post #45
zamu wrote: post #20

Earlier quoted context omitted.

Then save it locally as well? There really isn't an excuse to having DO or any provider being your sole backup.

Why is everyone talking about backups and not the actual infrastructure?

because that was the biggest issue in the original article, they didn't have backups of their data, so the company was dead. If they had had backups, they could have recreated the infrastructure.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#50

Post by kijin »

crazygringo wrote: post #24
cm2187 wrote: post #3

Earlier quoted context omitted.

And have a separate DNS provider.

No reason to. If your cloud provider does your DNS and you switch to a different cloud provider... ...then just set your registrar to point to the new DNS provider. You've still got total control.

Changing nameservers can take up to 48 hours. Things usually settle down after the first few hours, but a few odd users will continue to be directed to the old nameservers well into the next day. This is exactly the kind of intermittent, hard-to-diagnose issue that you don't want to have to worry about in the middle of a crisis such as "Digital Ocean killed my company."

Changing A records, on the other hand, can take as little time as you want depending on the TTL value.

Post reply on HN