Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why the “Digital Ocean killed my company” incident scares the hell out of me

blog.checklyhq.com

Post reply on HN
Print view View original

111–120 of 185 posts

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#111

Post by dijit »

brianwawok wrote: post #82
olau wrote: post #44

Earlier quoted context omitted.

That's wishful thinking. If there's state in that virtual machine, it's probably either stored on the physical host or in a SAN. If it's in the physical host, it has to be fished out of that machine or restored from a backup. If it's a SAN, you can lose your virtual machine if the SAN goes down. I've seen both happen. Actually, a single machine with RAID is surprisingly stable. A provider like Hetzner can switch out…

Thats not wishful thinking. I did it from day one with my startup with GKE. You can blow up any of my servers, and a new one gets spun up and dockers deployed while the glitch is load balanced around. All for only a few hours of work. GKE has given me the infrastructure of a 10 million dollar company, with me as the IT guy doing 1h or less a week of IT work. I would not be able to do this on a $35 hertzer machine.

GKE (and GCP in general) is a special beast when it comes to this. They support online migrations of instances and it's nearly seamless.

In fact, it's very commonly happening to instances, probably ones you have.

But if you're running on AWS with their EC2 instances for your application, then it's not more reliable than a single chunk of hardware in my experience (there are special cases, some hardware is just crappy)

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#112

Post by freehunter »

jotm wrote: post #73
freehunter wrote: post #12

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…

Do you not have a mirror copy of everything somewhere else? Or is that not possible in your case?

I only run one server, because like the dead company in question, I don't have the money or manpower to do a full high-availability solution. Just a database backup and my site's code in Gitlab, so DR failover is a matter of 'git clone' and 'pg_restore'.

It might take me a few hours to get back online and piece my docker containers back together but my customers can absorb a few hours of downtime as long as it's a very rare occasion. If they couldn't, I'd charge them enough to afford a real HA solution.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#113

Post by tpetry »

pkaye wrote: post #94
luckylion wrote: post #66

Earlier quoted context omitted.

> Also, why would they want to? One of these days, they'll fuck up big time for many customers and get sued. They'll survive, but it'll cost a lot of money. Especially for Google (and other life-or-death services for many people), the solution seems kind of simple: charge for support. Google terminated your GMail-Account because you logged in from Turkey? Pay $50 to get somebody to listen to your story and work with…

There is Google Apps which is a paid service but most people gravitate to the free stuff.

The problem is if google is shutting down access for you how will you use the support provided by google apps. its a support chat you have to be logged in...

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#114

Post by opportune »

vorticalbox wrote: post #41
jwr wrote: post #22

Earlier quoted context omitted.

> But I still do business with DO because it costs ~half the price of a comparable EC2 instance I'm curious why people who don't have massive scaling and variability issues choose DO or AWS for their hosting. For 34€/month, Hetzner will rent you a physical server (i7-6700, 64GB RAM, 2x512GB SSD, 1Gbit/s networking). That's a monster of a machine and can run most sites out there. And if you need more oomph and better…

If you use a lambda on aws you get 1 million requests for free each month and after that factions of a penny per invoke.

The issue with lambda is that at high loads it's a lot more expensive than using a server. I looked into doing something with lambda that would take about ~10b invocations/day and it was so expensive compared to just standing up a microservice.

So I've never deployed a lambda before and I'm sure at low loads it's fine, but I would be afraid of taking a dependency on lambda/functions in an application architecture.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#115

Post by phonon »

metildaa wrote: post #90
phonon wrote: post #74

Earlier quoted context omitted.

https://aws.amazon.com/lightsail/pricing/?opdp1=pricing ?

The CPU allocations on Lightsail are anemic compared to DO.

Are they? They come out fairly comparably here. I think Lightsail is a T3 instance "under the hood".

https://joshtronic.com/2019/06/03/vps-showdown-digitalocean-...

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#116

Post by cm2187 »

bradyd wrote: post #103
cm2187 wrote: post #97

Earlier quoted context omitted.

That transition requires authorisation from the original provider. If they locked your account and don't pick up the phone, you won't get the move approved and won't be able to repoint your urls to a different cloud.

No it doesn't. Name servers are controlled by the registrar not the DNS provider.

By provider I mean the registrar (which typically provides both services). Moving to another registrar requires an authorization code, and good luck getting that on a short notice if your provider doesn't talk to you.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#117

Post by ksec »

jwr wrote: post #22
freehunter wrote: post #12

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…

> But I still do business with DO because it costs ~half the price of a comparable EC2 instance I'm curious why people who don't have massive scaling and variability issues choose DO or AWS for their hosting. For 34€/month, Hetzner will rent you a physical server (i7-6700, 64GB RAM, 2x512GB SSD, 1Gbit/s networking). That's a monster of a machine and can run most sites out there. And if you need more oomph and better…

>For 34€/month, Hetzner ...

Hetzner is only available in EU and specifically Germany. And my bet most of these people want it within US.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#118

Post by proyb2 »

chocolatebunny wrote: post #40
freehunter wrote: post #12

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…

Cloud or not, you have to trust your vendor to deliver the service you paid them for. That's the whole reason for doing business with them.

I trust UpCloud more than other VPS provider because the technical support (Regional) has been very approachable and you know "who" they are. Beside, they promise 100% uptime SLA when your business is critical.

https://upcloud.com/blog/

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#119

Post by PhasmaFelis »

freehunter wrote: post #12

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…

> I don't see "I've never run a business before" as a valid excuse, nor do I see "the cloud vendor is better equipped to handle backups" as a valid excuse.

It's not unreasonable to be annoyed when a company you are paying specifically to do the things you're not familiar with, catastrophically fails to do them correctly.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#120

Post by PhasmaFelis »

freehunter wrote: post #12

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…

> I don't see "I've never run a business before" as a valid excuse, nor do I see "the cloud vendor is better equipped to handle backups" as a valid excuse.

If you're paying someone specifically to make backups for you, you should be able to trust that they've taken every reasonable measure to ensure that backups are actually being made and preserved.

Post reply on HN