Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why the “Digital Ocean killed my company” incident scares the hell out of me

blog.checklyhq.com

Post reply on HN
Print view View original

131–140 of 185 posts

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#131

Post by jwr »

Someone1234 wrote: post #31
jwr wrote: post #22

Earlier quoted context omitted.

> But I still do business with DO because it costs ~half the price of a comparable EC2 instance I'm curious why people who don't have massive scaling and variability issues choose DO or AWS for their hosting. For 34€/month, Hetzner will rent you a physical server (i7-6700, 64GB RAM, 2x512GB SSD, 1Gbit/s networking). That's a monster of a machine and can run most sites out there. And if you need more oomph and better…

Because hypothetically your instance should be hardware agnostic. If the physical hardware dies, it should automatically migrate to another physical server at their data center without your intervention. It will only look like an unexpected restart from your perspective. That's something worth paying for. It would be more comparable to two servers at Hetzner with rapid failover. But even that is more involved since y…

Right. I'm using stock Ubuntu LTS with Docker and my software is in docker images or otherwise easily deployable. Changes to the host system configuration are documented, so re-creating a host from scratch is not a problem.

Things like automatic failover and high availability in general are cool, but your SLOs should be set according to your business model, e.g. they need to make business sense.

I do understand how people like VPSs at $5 or $10/month (I've used one like that myself for years), and I do understand why companies that need massive scalability use AWS. But I do not understand why people in the middle (which I think includes a significant fraction of this audience) assume that a provider like AWS is the best solution.

Anyway, those are just my thoughts. I've been running a business on Hetzner servers for years now. And this is not necessarily a recommendation to use Hetzner: there are many providers which will offer low-cost physical servers. I think in spite of the cloud hype, these make sense for many applications.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#132

Post by cyberferret »

What scares me the most these days is that a lot of vendors have a policy of "we're shutting you down, but we won't tell you why or how you can fix things so that it doesn't happen again".

I have had friends who have had their Twitter or Youtube accounts shut down with no explanation from the companies involved. Just recently, MailChimp shut down a brand new account for my SaaS (we had only 4 subscribers added, and hadn't sent out any email campaigns yet!) with a refusal to give us an explanation why. We set up another account and did everything exactly the same way (including tediously replicating lots of email templates manually) without the same thing happening. Why was the first account shut down? We will never know, it seems.

I am all for using AI to detect dark patterns and raising a flag, but I think these companies need to bring a more human aspect into the after effects. Surely a legitimate operator will always email in with a "WTF?" whereas a nefarious actor will simply automatically move on to the next thing?

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#133

Post by unixhero »

cm2187 wrote: post #97
crazygringo wrote: post #24

Earlier quoted context omitted.

No reason to. If your cloud provider does your DNS and you switch to a different cloud provider... ...then just set your registrar to point to the new DNS provider. You've still got total control.

That transition requires authorisation from the original provider. If they locked your account and don't pick up the phone, you won't get the move approved and won't be able to repoint your urls to a different cloud.

That's wrong.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#134

Post by ptype »

neovive wrote: post #81
freehunter wrote: post #12

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…

If you don't mind me asking, what is the preferred method to backup to S3? Is it possible to scp or rsync a mysqldump to S3 or do you install aws tools on DO and run aws s3 cp as a scheduled job?

check out restic which does dedup backups to s3, very easy to use

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#135

Post by luckylion »

pkaye wrote: post #94
luckylion wrote: post #66

Earlier quoted context omitted.

> Also, why would they want to? One of these days, they'll fuck up big time for many customers and get sued. They'll survive, but it'll cost a lot of money. Especially for Google (and other life-or-death services for many people), the solution seems kind of simple: charge for support. Google terminated your GMail-Account because you logged in from Turkey? Pay $50 to get somebody to listen to your story and work with…

There is Google Apps which is a paid service but most people gravitate to the free stuff.

Paying for the product isn't the issue, the company in the DO debacle was happy to pay - support is the issue. I pay Amazon to send me stuff, but I'd honestly be happy to pay them $5 to have my emails to support read & replied to by a person that doesn't have to rely on a low tier auto translate.

I understand that support is expensive, and especially for ad financed or cheap products, having an agent look into something can quickly cost more than you'll ever make off that customer. If the customer pays for support, support becomes a product and the company doesn't have to treat support as a profit-killer (that is, automate it, make it annoying for the customer so he avoids it, and staff it with the cheapest available labor creating a high fluctuation because of bad working conditions).

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#136

Post by turrini »

lohszvu wrote: post #106
turrini wrote: post #70

I've got a stable setup: --------------------------------------- Vultr ========================== Linode --------------------------------------- Los Angeles ==================== Dallas Mastercard 1234 ============= Visa 4567 --------------------------------------- VMs with custom Linux (ROOT/ZFS Debian) Replicating snapshots every 5 minutes CloudFlare + HAProxy Load Balancing Total cost: 6 VMS at $20 each: $120, Clou…

Are you using a managed host for HAProxy? What if it goes down.

I have a failover setup that changes Cloudflare DNS when necessary.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#137

Post by ksec »

hathawsh wrote: post #128
ksec wrote: post #117

Earlier quoted context omitted.

>For 34€/month, Hetzner ... Hetzner is only available in EU and specifically Germany. And my bet most of these people want it within US.

I googled for "Hetzner USA" and found some similar offerings: https://www.wholesaleinternet.net/dedicated/ https://www.nocix.net/dedicated/

I think most people goes with DO not because it was just "affordable", it has also been around long enough and large enough to be trusted. Both OVH and Hetzner is the same as DO in this regards.

The two listed aren't really well known.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#138

Post by marcosdumay »

luckylion wrote: post #66
ehnto wrote: post #59

Earlier quoted context omitted.

I would support your theory. Support staff don't scale as well or as fast as the technology they're supporting does. I have often wondered about that for Google. Sure, they have bucket loads of cash, but could they even feasibly stand up a large enough support staff to handle all their platforms? They have so many services, across dozens of languages, and serve hundreds of millions of customers in different timezones…

> Also, why would they want to? One of these days, they'll fuck up big time for many customers and get sued. They'll survive, but it'll cost a lot of money. Especially for Google (and other life-or-death services for many people), the solution seems kind of simple: charge for support. Google terminated your GMail-Account because you logged in from Turkey? Pay $50 to get somebody to listen to your story and work with…

> Pay $50 to get somebody to listen to your story and work with you on proving your identity.

Personally, I would understand this as extortion.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#139

Post by marcosdumay »

cm2187 wrote: post #97
crazygringo wrote: post #24

Earlier quoted context omitted.

No reason to. If your cloud provider does your DNS and you switch to a different cloud provider... ...then just set your registrar to point to the new DNS provider. You've still got total control.

That transition requires authorisation from the original provider. If they locked your account and don't pick up the phone, you won't get the move approved and won't be able to repoint your urls to a different cloud.

The GPs cloud provider is the registar.

It's not common to use those two words as synonyms. Although it is common to use the registar as a cloud provider, it may not be a good practice either (depends on the registar).

Post reply on HN