"DigitalOcean Killed Our Company" • Hacker News Board

Re: "DigitalOcean Killed Our Company"

#281

Post by tgsovlerkhgsel » Fri, May 31, 2019, 10:47 PM UTC

The timeline is interesting in itself: This story was posted 4 hours ago on Twitter, 3 hours ago on HN.

The cofounder picked it up 3h ago. DO responded and apologized from the official account 2 hours ago, claiming it was fixed, and is actively responding to people tweeting at them, doing damage control, since about 1 hour, promising a public postmortem.

While it's sad that a social media escalation was needed (and it confirms that getting attention on social media is the only effective way to resolve hard issues like this), the response after that was quite fast. Let's see how well and how quickly they deliver on the postmortem.

Re: "DigitalOcean Killed Our Company"

#282

Post by jdsully » Fri, May 31, 2019, 10:48 PM UTC

klank wrote: post #273

windowsworkstoo wrote: post #252

Earlier quoted context omitted.

Yep, this is spot on - I used to work on a webhosting help desk and could bang out about 100 tickets a shift, because so many were small queries that required no depth work.

Old MSFT rule of thumbs was 2 bugs per day during bug crunch mode. Sounds crazy, but when you consider the number of "this text is wrong" and "that text box is too short" bugs that existed after a year of furious development, it wasn't too hard to achieve. Gotta hit that ZBB!

Brought back memories. I think it might be a little Stockholm syndrome but there was just something about the pressure of getting a release out when you know it only happens once every few years. Bug triage definitely improved my persuasion technique.

Now its just "meh, we'll fix it in next months release".

Re: "DigitalOcean Killed Our Company"

#283

Post by rhacker » Fri, May 31, 2019, 10:48 PM UTC

I am a software developer and yes software is eating the world. One of the side effects of software eating the world is out of control software.

   * Autobans in facebook
   * Cheated instacart drivers
   * $10000 stolen from thousands of bank accounts (and returned hopefully) on Etsy
   * Tesla cars literally killing people (it now feels like it's once a month right?)

Now the software that runs software is running amok.

The interesting thing about software is that it runs very quickly and it acts as a giant lever that affects the entire world.

You can think of it like a giant airport suddenly being installed in your back yard and just start having planes take off, changing your $300k investment into a $120k valued house overnight. That's how quickly software is changing the real world.

I know there is at least one HN browser writing a book on it. But I would love to see more books on how the internet, and software, is messing our world up.

Re: "DigitalOcean Killed Our Company"

#284

Post by ufmace » Fri, May 31, 2019, 10:52 PM UTC

stingraycharles wrote: post #122

_bxg1 wrote: post #101

Earlier quoted context omitted.

It could literally be a cron job that dumps your DB to a desktop computer once a week. Not exactly CIA-level stuff.

More realistically they would have done backups inside DO and would still be locked out. Not many people actually do complete offsite backups to a completely different hosting provider, getting locked out of your account is usually just not a consideration. It’s unrealistic to expect this of a tiny startup.

I don't know, it seems simple enough to me. I have a server on DO hosting some toy-level projects, and IIRC it took me 15-30 min to set up a daily Cron job to dump the DB, tar it, and send it to S3, with a minimum-privilege account created for the purpose, so that any hacker that got in couldn't corrupt the backups. I'm not a CLI or Linux automation whiz, others could probably do it faster.

Re: "DigitalOcean Killed Our Company"

#285

Post by Axsuul » Fri, May 31, 2019, 10:52 PM UTC

_7siz wrote: post #250

vbezhenar wrote: post #88

Earlier quoted context omitted.

Probably because of publicity. How many of those companies went bankrupt silently, because their case did not cause much attention in news?

Accidents can happen. Don't really blame Digital Ocean for the accidental locking but this response is insane: https://pbs.twimg.com/media/D76ocofXoAY_xB5.png

That response is ice cold. Reminds me of suspension emails Amazon sends out to their FBA sellers.

Re: "DigitalOcean Killed Our Company"

#286

Post by jwr » Fri, May 31, 2019, 10:52 PM UTC

dahfizz wrote: post #272

jrochkind1 wrote: post #254

Earlier quoted context omitted.

Uretsky: > Account should be re-activated - need to look deeper into the way this was handled. It shouldn't have taken this long to get the account back up, and also for it not be flagged a second time. So... he doesn't address what is the scariest part to me, the message that just says "Nope, we've decided never to give your account back, it's gone, the end."

How would you like that to have been addressed? I think it's entirely reasonable for companies to have that option. "You are doing something malicious and against the rules, you have been permanently removed". In this case, that option was misused, but I don't think the existence of that possiblity is inheritly surprising.

Access to your data should never be denied. Ever. It was not DigitalOcean's data. If you are a hosting provider, you can't ever hold customer data hostage or deny them access to it in any way.

Re: "DigitalOcean Killed Our Company"

#287

Post by toss1 » Fri, May 31, 2019, 10:53 PM UTC

dahfizz wrote: post #272

jrochkind1 wrote: post #254

Earlier quoted context omitted.

Uretsky: > Account should be re-activated - need to look deeper into the way this was handled. It shouldn't have taken this long to get the account back up, and also for it not be flagged a second time. So... he doesn't address what is the scariest part to me, the message that just says "Nope, we've decided never to give your account back, it's gone, the end."

How would you like that to have been addressed? I think it's entirely reasonable for companies to have that option. "You are doing something malicious and against the rules, you have been permanently removed". In this case, that option was misused, but I don't think the existence of that possiblity is inheritly surprising.

Reasonable to have the shutdown part of the option, yes.

At the very least, they should also provide ALL, as in every last byte, of data, schemas, code, setup etc. to the defenestrated customers. As in: "sorry, we cannot restart your account, but you can download a full backup of your system as of it's last running configuration here: -location xyz-, and all previous backups are available here: -location pdq-".

Anything less is simply malicious destruction of a customer's property.

If you violate a lease and get evicted, they don't keep your furniture & equipment unless you abandon it.

Re: "DigitalOcean Killed Our Company"

#288

Post by rootusrootus » Fri, May 31, 2019, 10:55 PM UTC

_7siz wrote: post #250

vbezhenar wrote: post #88

Earlier quoted context omitted.

Probably because of publicity. How many of those companies went bankrupt silently, because their case did not cause much attention in news?

Accidents can happen. Don't really blame Digital Ocean for the accidental locking but this response is insane: https://pbs.twimg.com/media/D76ocofXoAY_xB5.png

That's the kind of response you only send when you're convinced the customer is actually nefarious and you don't care about losing them. I wonder if there is any missing backstory here or if it really is just a case of mistaken analysis.

Re: "DigitalOcean Killed Our Company"

#289

Post by xorcist » Fri, May 31, 2019, 10:55 PM UTC

CharlesColeman wrote: post #126

badrequest wrote: post #109

Earlier quoted context omitted.

"2-man teams generally don't prioritize backups" isn't an excuse for not prioritizing backups.

> "2-man teams generally don't prioritize backups" isn't an excuse for not prioritizing backups. They had backups, but being arbitrarily cut-off from their hosting provider wasn't part of their threat model. Isn't a big part of cloud marketing the idea that they're so good at redundancy, etc. that you don't need to attempt that stuff on your own? The idea that you have to spread your infrastructure across multiple cl…

> being arbitrarily cut-off from their hosting provider wasn't part of their threat model

Let's be fair: The threat model here is "lose access to our data".

This can happen in a number of ways, lost (or worse, leaked) password to the cloud provider, provider goes bankrupt, developer gets hacked, and a thousand other things.

Even if you trust your provider to have good uptime, there's really no excuse for not having any backups. Especially not if you're doing business with Fortune 500's.

Re: "DigitalOcean Killed Our Company"

#290

Post by jdietrich » Fri, May 31, 2019, 10:58 PM UTC

williamdclt wrote: post #234

jdietrich wrote: post #177

Earlier quoted context omitted.

>getting locked out of your account is usually just not a consideration How many horror stories need to reach the front page of HN before people stop believing this? Getting locked out of your cloud provider is a very common failure mode, with catastrophic effects if you haven't planned for it. To my mind, it should be the first scenario in your disaster recovery plan. Dumping everything to B2 is trivially easy, triv…

> How many horror stories need to reach the front page of HN before people stop believing this Sounds like the opposite of the survivor bias. I don't believe it's any sort of common (though it does happen), even less that "it should be the first scenario in your disaster recovery plan"

If you plan for the "all of our cloud infrastructure has failed simultaneously and irreparably" scenario, you get a whole bunch of other disaster scenarios bundled in for free.