Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

"DigitalOcean Killed Our Company"

twitter.com

Post reply on HN
Print view View original

171–180 of 620 posts

Re: "DigitalOcean Killed Our Company"

#171

Post by nurettin »

znpy wrote: post #135
thaumaturgy wrote: post #114

Some people on HN hate Linode because of their past security screwups (which is valid), but having used both DO and Linode quite a lot, the support on Linode is way, way, way better than DO's. DO's tier 1 support is almost useless. I set up a new account with them recently for a droplet that needed to be well separated from the rest of my infrastructure, and ran into a confusing error message that was preventing it f…

I have interviewed with DO and they tried diverting me towards a support position. They told me that on a single day a support engineer was supposed to help/advice customers on pretty much whatever the customer was having issue with and also handle something between 80-120 tickets per day. It's nice to see that DO is willing to help on pretty much anything they (read: their team) has knowledge about, but with 80-120…

7 tireless hours of work (with lunch break) 15 minutes to Listen, understand and resolve an issue, assuming perfect knowledge, a lot of luck and normal human speed, that would still amount to less than 30 resolutions a day.

Re: "DigitalOcean Killed Our Company"

#172

Post by prophesi »

Dylan16807 wrote: post #168
prophesi wrote: post #152

Earlier quoted context omitted.

The difference is that when AWS goes down, Netflix/Spotify still have backups and could adapt infrastructure if the outage involved permanent data-loss. You're talking about the people who built https://github.com/Netflix/chaosmonkey I'd argue that it should be _easier_ for a 2-man company to adapt to cloud service outages, as they likely don't have to keep up with nearly as many backups or moving parts.

So pretend they had offsite backups. That's a separate issue from an entire contingency plan. The ability to adapt is not the same thing. This company could certainly adapt to a new host if they had an extra backup.

The ability to adapt is the definition of a contingency plan. It's essentially, "If this person/service/database/customer/etc vanished off the face of the earth, what do?"

Re: "DigitalOcean Killed Our Company"

#173

Post by walrus01 »

thaumaturgy wrote: post #166
walrus01 wrote: post #151

Earlier quoted context omitted.

ISP here: The margins in bulk hosting services are incredibly thin, and companies have resorted to automation tools. If somebody asked me to run backend infrastructure for something like DigitalOcean or Linode, I would run away screaming. It would literally be my own personal hell. I would rather run any other sort of ISP services on the planet than a bulkhosting service where anybody with a pulse and $10 to $20/mont…

This race to the bottom has reached a point that it's harming customers. It's okay to be more expensive than the competition if you provide a better service.

Personal opinion, it's really important in the ISP/hosting world to identify what market categories are a race to the bottom, and if at all possible, refuse to participate in them.

I look at companies selling $5 to $15/month VPS services and try to figure out how many customers they need to be set up for monthly recurring services, in order to pay for reasonably reliable and redundant infrastructure, and the math just doesn't pencil out without:

a) massive oversubscription

b) near full automation of support, neglect of actual customer issues, callous indifference caused by overworked first tier support

Conversely, as a customer, you should be suspicious when some company is offering a ridiculous amount of RAM, disk space and "unlimited 1000 Mbps!" at a cheap price. You should expect that there will be basically no support, it might have two nines of uptime, you're responsible for doing all your own offsite backups, etc.

If you use such a service for anything that you would consider "production", you need to design your entire configuration of the OS and daemons/software on the VM with one thing in mind: The VM might disappear completely at any time, arbitrarily, and not come back, and any efforts to resolve a situation through customer support will be futile.

Re: "DigitalOcean Killed Our Company"

#174

Post by ehutch79 »

Soooo... an automated script flagged them. they get themselves unflagged, and proceed to do the same thing immediately. without any real confirmation that they wouldn't be flagged again.

They've basically been flagged as abusing the system multiple times, and they're surprised they had to kick up a storm to get themselves reactivated again?

Not to mention, that process they need to run every couple of months, that takes 1s, but they still need to parallelize over a bunch of vms, that's weird and sounds like something that needs to be rearchitected at the very least.

Re: "DigitalOcean Killed Our Company"

#175

Post by ziddoap »

GordonS wrote: post #80
ziddoap wrote: post #30

Earlier quoted context omitted.

In all practicality, DO freezing your account has the same effect of DO being down (or closing, etc.), or your account being compromised and you being locked out of it. A contingency plan should ideally have been in place for a scenario where, regardless of root cause, you have lost access to your DO account.

> In all practicality, DO freezing your account has the same effect of DO being down (or closing, etc.) Given their size, that is extremely unlikely to happen without warning.

Sure, but them closing combined with the chance of them freezing your account (feasible, considering the topic here) and the chance of account compromise, and the chance they go down for extended maintenance... It is inexcusable not to have a disaster recovery plan for the scenario where you cannot access your DO account.

Imagine you are a customer of this company. Would you be rallying to their defense, "backups aren't needed because the scenarios are unlikely", or would you be angry that the company had zero contingency planning and lost all of your data (or the data you rely upon)?

If you can honestly say, as a (hypothetical) customer of the company in the thread, that you wouldn't care if a company you relied upon has no disaster recovery planning, more power to you. I, however, like to make sure that the companies I'm relying on have some sort of contingency that protects me as a customer.

Re: "DigitalOcean Killed Our Company"

#176

Post by Dylan16807 »

prophesi wrote: post #172
Dylan16807 wrote: post #168

Earlier quoted context omitted.

So pretend they had offsite backups. That's a separate issue from an entire contingency plan. The ability to adapt is not the same thing. This company could certainly adapt to a new host if they had an extra backup.

The ability to adapt is the definition of a contingency plan. It's essentially, "If this person/service/database/customer/etc vanished off the face of the earth, what do?"

Okay, then that means they did have a contingency plan, except for a single rsync.

Which would mean you disagree strongly with coldtea?

Re: "DigitalOcean Killed Our Company"

#177

Post by jdietrich »

stingraycharles wrote: post #122
_bxg1 wrote: post #101

Earlier quoted context omitted.

It could literally be a cron job that dumps your DB to a desktop computer once a week. Not exactly CIA-level stuff.

More realistically they would have done backups inside DO and would still be locked out. Not many people actually do complete offsite backups to a completely different hosting provider, getting locked out of your account is usually just not a consideration. It’s unrealistic to expect this of a tiny startup.

>getting locked out of your account is usually just not a consideration

How many horror stories need to reach the front page of HN before people stop believing this? Getting locked out of your cloud provider is a very common failure mode, with catastrophic effects if you haven't planned for it. To my mind, it should be the first scenario in your disaster recovery plan.

Dumping everything to B2 is trivially easy, trivially cheap and gives you substantial protection against total data loss. It also gives you a workable plan for scenarios that might cause a major outage like "we got cut off because of a billing snafu" or "the CTO lost his YubiKey".

Re: "DigitalOcean Killed Our Company"

#178

Post by ksec »

thaumaturgy wrote: post #114

Some people on HN hate Linode because of their past security screwups (which is valid), but having used both DO and Linode quite a lot, the support on Linode is way, way, way better than DO's. DO's tier 1 support is almost useless. I set up a new account with them recently for a droplet that needed to be well separated from the rest of my infrastructure, and ran into a confusing error message that was preventing it f…

And Linode has always had faster CPU, I/O, Network. And lots of small things like that DO only catches up in the recent years, like pooled bandwidth.

Although these days I tend to go with UpCloud, it is very similar to Linode and DO, except you can do custom instances like 20x vCPU with 1GB Memory, spin it up for $0.23 an hour. Compared to standard plan on Linode and DO, 20 vCPU with 96GB Memory would be $0.72/Hr.

Re: "DigitalOcean Killed Our Company"

#179

Post by keypusher »

Given that the author was quite vague about the nature of this “pipeline” and that their product is an “AI-powered Startup Selection engine”, I have a suspicion they were probably crawling and scraping a whole bunch of pages for new startups. It’s possible that this was totally legit and it just looked like a ddos attack, or that it was something else entirely, but everyone here seems to have taken him at his word that what they were doing was actually above board.

Re: "DigitalOcean Killed Our Company"

#180

Post by z3t4 »

h3throw wrote: post #155
z3t4 wrote: post #124

Earlier quoted context omitted.

Having backups in two places could easily triple the hosting costs. The question is what costs more. Eg. Losing data vs backup costs.

That seems like an odd cost increase. How do you figure it would lead to a tripling of op costs?

Lets say you have 100 TB of data, plus two backups, you are now paying for 300 TB of data.
Post reply on HN