Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why the “Digital Ocean killed my company” incident scares the hell out of me

blog.checklyhq.com

Post reply on HN
Print view View original

141–150 of 185 posts

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#141

Post by philjohn »

true_religion wrote: post #86
hadlock wrote: post #67

Earlier quoted context omitted.

Most $10MM ARR startups are running on two medium T class aws instances behind a load balancer at $35/mo plus another small server for accessories (redis, message queue, etc) and then a sql backend. That's something like $200/mo for infrastructure, you never need to maintain the hardware and you have the option to autoscale at any time to support $100MM ARR. Plus you get access to backups and all the other tools and…

You’re telling me that 10 million dollar “startup” companies can’t run 1 server without incurring “years” of tech debt? Sorry, that just sounds unbelievable.

They can, but let's be honest, people will be doing multiple roles when in the very early stage, so your lead programmer will be configuring the server ... probably won't be using a configuration management solution and whomever takes it over is going to have to comb through the various configuration files to work out what it's meant to be doing.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#142

Post by mort96 »

z3t4 wrote: post #71
eyeinthepyramid wrote: post #49

Earlier quoted context omitted.

because that was the biggest issue in the original article, they didn't have backups of their data, so the company was dead. If they had had backups, they could have recreated the infrastructure.

They did have backups, only problem backups was also on DO. Backing up petabyte of data even on a Gbit connection is slow, and also cost more then a pennies if you for example back it up on AWS. Linus tech tips have a video on youtube explaining the problem.

It's not really a "backup" if it's just on the same server/data center though. Moving data from one drive to another in the same machine, or from one machine to another in the same building, provides fault tolerance should the main drive/machine die, that's about it.

That's true whether the computer(s) is your own or someone else's.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#143

Post by xiaomai »

freehunter wrote: post #12

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…

I remember getting the e-mail about their backups not having been working for months. I was shocked at how negligent that was. I run my own backups now.

(Also their backup offering is so much more cumbersome to use than borg backup that it wasn't a great loss to manage these myself).

Still amazed that they didn't notice no one's backups had been running for so long.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#144

Post by floatboth »

ksec wrote: post #117
jwr wrote: post #22

Earlier quoted context omitted.

> But I still do business with DO because it costs ~half the price of a comparable EC2 instance I'm curious why people who don't have massive scaling and variability issues choose DO or AWS for their hosting. For 34€/month, Hetzner will rent you a physical server (i7-6700, 64GB RAM, 2x512GB SSD, 1Gbit/s networking). That's a monster of a machine and can run most sites out there. And if you need more oomph and better…

>For 34€/month, Hetzner ... Hetzner is only available in EU and specifically Germany. And my bet most of these people want it within US.

Hetzner is also a company that really wants you to send them a scan of your government ID / passport. Meanwhile e.g. prgmr takes bitcoin.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#145

Post by bcooks »

weaksauce wrote: post #130

I made this comment in an older thread but haven't gotten an answer on... the circumstances seem off to me: The thing that has me scratching my head is how this chain of events unfolded. I get that your fraud algorithm flagged it because of lack of established payment. how is this possible if what the tweet referred to as "locking us out of all of our backups and work"? surely an account history of any significance w…

Hi weaksauce... sorry I must have missed your earlier question.

The account had been live for some time and in that sense had history but because of credits it didn't have payment history. As some others have commented lots of startups use credits to get their business going and depending on your usage they can last you for quite a while. Payment history indicates a willingness and capability to make payments.

Part of the issue here was what triggers the algorithm used when looking at remaining credits, payment history (none), workload deltas (the new spin ups), and effective run rate (think of that as the amount of money they would be charged for the workload they were spinning up). The bug in this case was both simple and super impactful. Raisup did nothing wrong, everything right in fact. We just blew it.

Thanks for the comment on request for download of backups or snapshot. That is a great idea, I guess we just never expected to actually go shoot a real customer and the fraudsters don't ask for their data.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#146

Post by qes »

fhood wrote: post #51

Arguably Raisup should have been better prepared for this eventuality. But , I suspect that many of the people here talking about how their system is infallible made it that way because they, had time and/or money to spare working on it, have a very portable system, or have had something like this happen before. I feel like HN comments are often sanctimonious to the point of ignorance. A solo dev, scrambling to get h…

They've been in business for at least five years. They also decided to make a lot of noise about their Fortune 500 clients (multiple). If they're not well past "scrambling to succeed" then they failed on the business side as much as they did the technical side.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#147

Post by cdumler »

fhood wrote: post #51

Arguably Raisup should have been better prepared for this eventuality. But , I suspect that many of the people here talking about how their system is infallible made it that way because they, had time and/or money to spare working on it, have a very portable system, or have had something like this happen before. I feel like HN comments are often sanctimonious to the point of ignorance. A solo dev, scrambling to get h…

> I feel like HN comments are often sanctimonious to the point of ignorance. A solo dev, scrambling to get his project to the point where there is even the tiniest chance it will succeed, is likely to have a complex network of hardcoded filepaths, hostnames, and other magic numbers, strings, and config files that would make it very difficult to make portable without significant time and effort.

No, no, no. That's the "well, it's never happen before so why should I have planned for it?" attitude. If you are in a solo business it is the _most_ important time to pull out your SWOT chart be honest about threats and weaknesses. A solo company is literally the worse time to not consider weaknesses as you have the least amount of assets to cover calamity. His primary concern right now is to be able to reproduce his work on AWS or Azure. Period.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#148

Post by __d »

turrini wrote: post #70

I've got a stable setup: --------------------------------------- Vultr ========================== Linode --------------------------------------- Los Angeles ==================== Dallas Mastercard 1234 ============= Visa 4567 --------------------------------------- VMs with custom Linux (ROOT/ZFS Debian) Replicating snapshots every 5 minutes CloudFlare + HAProxy Load Balancing Total cost: 6 VMS at $20 each: $120, Clou…

Where's your DNS?

And good work on the credit cards -- too many people forget that point of failure.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#149

Post by sonnyblarney »

jwr wrote: post #22
freehunter wrote: post #12

I've had a DO mistake take down my site before, and when it was brought back up it had been reverted to several months prior. DO support was at a loss as to why this would have happened. I tried to restore from DO's backup service, but their backups had apparently stopped running several months prior as well. This was a major issue and could have easily been the death of my company, all because of a DO glitch. But it…

> But I still do business with DO because it costs ~half the price of a comparable EC2 instance I'm curious why people who don't have massive scaling and variability issues choose DO or AWS for their hosting. For 34€/month, Hetzner will rent you a physical server (i7-6700, 64GB RAM, 2x512GB SSD, 1Gbit/s networking). That's a monster of a machine and can run most sites out there. And if you need more oomph and better…

Because AWS et. al. are considerably more than just instances?

There's a whole host of services that make dev and deployment at lot easier.

Though instance costs are more expensive ... that's not the issue, the issue is 'TCO' (total cost of ownership) not 'server cost'.

If your team can move at 2x the speed on AWS, well that's worth a lot.

Cloud services are very useful, so the article raises a very legit concern/paradox.

Re: Why the “Digital Ocean killed my company” incident scares the hell out of me

#150

Post by dmix »

eyeinthepyramid wrote: post #49
dmix wrote: post #45

Earlier quoted context omitted.

Why is everyone talking about backups and not the actual infrastructure?

because that was the biggest issue in the original article, they didn't have backups of their data, so the company was dead. If they had had backups, they could have recreated the infrastructure.

Okay, but say you do get shutdown and you have your offsite backups, like most people. It's still a significant amount of work starting from scratch to rebuild and link all of your servers, dbs, services, etc. Plus all of the basic OS tweaking.

Losing your backups because your on one site is always dangerous. Getting completely kicked off your VPS systems with no warning or help is what's new and scary about this DO story.

Post reply on HN