Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stop Using Digital Ocean Now: The Aftermath

serdardogruyol.com

Post reply on HN
Print view View original

71–80 of 102 posts

Re: Stop Using Digital Ocean Now: The Aftermath

#71

Post by ehutch79 »

sdogruyol wrote: post #53
veemjeem wrote: post #50

Earlier quoted context omitted.

I think it's a bit sad that you have to go to such great lengths to protect your company's image. This was a very obvious case of customer fault. He admitted that he doesn't know if his server was hacked. The customer probably should have looked into the issue before making a inflammatory blog post. Maybe if you guys charged more you wouldn't get these ultra-cheap customers who think they should get 24/7 support for…

Being attacked or even getting compromised is customer fault ? Okay i get that. But what about closing the account instantly, not notifying the customer, accusing that customer of being a cheap liar and treating them in a bad way?

Sounds like they gave you plenty of chance to check it out;

""" We informed the customer that it may be a good idea to check through the virtual server to see if there were any signs of a compromise just in case """

In fact from the dates listed, it looks like you had over two weeks to check for compromise. Even then, it seems like they tried to talk to you before just closing your account;

""" A second UDP pattern was detected on 2013-09-24 12:27:09 and a ticket was opened 2013-09-24 12:27:14 to request more information from the customer. Because this was already a second occurrence we had to do a more thorough follow up. Discussing the matter with the customer, he informed us that it was a mysql db dump script that was pushing data to dropbox. """

Re: Stop Using Digital Ocean Now: The Aftermath

#72

Post by orclev »

zagi wrote: post #39

Hi, this is Ben, CEO and Co-Founder of DigitalOcean, we have received the document and will discuss the matter publicly. ----- All times are UTC. Our monitoring picked up a malicious UDP traffic pattern on 2013-09-08 00:58:23. A ticket was then opened with the customer at : 2013-09-08 01:05:55 roughly 7 minutes later. The customer informed us that it was a script that was crawling in the background. We informed the c…

A far more thorough response than I expected to see. It does look like, best case scenario his instance was compromised and being used for malicious purposes, worst case scenario he was actively doing something malicious himself. In either case taking the compromised/malicious instance offline is the appropriate first response in the case of an active attack underway.

I think I have to agree with a comment someone else made in the previous thread about this, with the rise of cheap VPS services we're seeing an influx of people unqualified and unprepared to run their own internet routable servers and things like this are the outcome of that. When you choose to stand up a VPS with a service like DO you take on the responsibility of keeping it secure and preventing it from engaging in malicious activity. If you fail at that task, the consequence is your servers will be shutdown for the good of the internet as a whole. If you are either unprepared or incapable of dealing with that responsibility you should be paying for a hosting service that's prepared to offer those services for you.

I say this as someone that currently has accounts with a number of VPS providers where I do take the responsibility of managing my servers seriously, as well as previously helping to administer a server that was compromised and taken offline until such a time as we had performed a full audit and verified our code on a new instance.

You should be prepared to treat a compromise of your servers the same as any other form of disaster. Treat it the same as if a flood happened and took out the facility you were hosted out of. You should have a backup plan in place so you can roll over to your backup until such time as you can fix the "broken" server, or else accept the downtime.

Re: Stop Using Digital Ocean Now: The Aftermath

#73

Post by dragonwriter »

sdogruyol wrote: post #53
veemjeem wrote: post #50

Earlier quoted context omitted.

I think it's a bit sad that you have to go to such great lengths to protect your company's image. This was a very obvious case of customer fault. He admitted that he doesn't know if his server was hacked. The customer probably should have looked into the issue before making a inflammatory blog post. Maybe if you guys charged more you wouldn't get these ultra-cheap customers who think they should get 24/7 support for…

Being attacked or even getting compromised is customer fault ? Okay i get that. But what about closing the account instantly, not notifying the customer, accusing that customer of being a cheap liar and treating them in a bad way?

Suspending -- what was done in the first instance -- a system which is engaging in an apparent DDOS is a perfectly reasonable action.

Locking the account when the explanation given is inconsistent with the observed behavior it was supposed to explain and the system is again engaing in an apparent DDOS is also a perfectly reasonable action.

It also seems from your posts that both times you were notified of the action by DO, so "not notifying the customer" is not an issue.

I don't see any evidence you were accused of being a "cheap liar", either.

The fact that services now make it cheap and easy to set up servers doesn't mean that you have no responsibility for what the servers you set up do. If you are really running a service with 25K active users, you probably ought to be able to respond to your VPS hosts questions about unusual UDP activity with either an explanation that holds water, or an up-front admission that you don't know where it is coming from and will take action to prevent it, rather than claim it comes from a database dump script that doesn't use UDP.

Re: Stop Using Digital Ocean Now: The Aftermath

#74

Post by arthulia »

csomar wrote: post #45

And now after nearly 10 hours or so i still haven’t heart from DO. There is a problem on the Internet is that people demand things NOW. Really? How long will it take for stuff to happen in real life (especially if you are dealing with government). Some stuff does happen immediately (like registering or purchasing something), but stuff which requires human intervention is obviously slow. And it requires time. Yes, you…

10 hours is unreasonably long for someone who is running an application that other users need to access. People are not going to be happy if Farmville is suddenly not remembering all the cows they milked last time they were logged in. This is about keeping customers happy down the chain.

That said, I actually really like DO and I've only had prompt, helpful responses from their customer service. I don't build apps but it is an excellent personal vps, and I still recommend them to my friends when they are looking for one.

Re: Stop Using Digital Ocean Now: The Aftermath

#75

Post by veemjeem »

sdogruyol wrote: post #66
sdogruyol wrote: post #53

Earlier quoted context omitted.

Being attacked or even getting compromised is customer fault ? Okay i get that. But what about closing the account instantly, not notifying the customer, accusing that customer of being a cheap liar and treating them in a bad way?

You know what they stopped answering my ticket after first response. If it wasn't HN post gaining this much traction i'm pretty sure that they won't respond to me.

They responded to you 7 minutes after discovering the irregular behavior. It's up to you to figure out the cause. It's not their responsibility to tail log files.

Had you chosen a VPS like amazon, you probably wouldn't even get notified. The first time you'd notice a problem is when you get your $1000 bandwidth bill from amazon.

You're like the customer that shows up at a restaurant and complains publicly about the food and demands a refund. When you make it a public issue, the company will move mountains to help you, but you're still an asshole for doing it.

Re: Stop Using Digital Ocean Now: The Aftermath

#76

Post by chc »

sdogruyol wrote: post #58
sdogruyol wrote: post #46

Earlier quoted context omitted.

Hello Ben, thanks for the response. Fırst of all at first ticket i told that the only possibility of having an UDP outgoing is that script that i wrote. Other than that i've no other activity or script that can generate that much traffic. Haven't you even considered that my droplet may be compromised or being attacked ? Instead of letting me know what exactly happened or which processes were running at that time you…

I was also tremendously happy with DO and their service. But what if you get your production apps down without even any notification and proper reasoning ? That's the thing which makes you feel insecure.

I'm sympathetic to your frustration — but when your box is owned and actively participating in illegal activity, you kind of have to expect it to get shut down.

Re: Stop Using Digital Ocean Now: The Aftermath

#77

Post by sdogruyol »

sdogruyol wrote: post #66
sdogruyol wrote: post #53

Earlier quoted context omitted.

Being attacked or even getting compromised is customer fault ? Okay i get that. But what about closing the account instantly, not notifying the customer, accusing that customer of being a cheap liar and treating them in a bad way?

You know what they stopped answering my ticket after first response. If it wasn't HN post gaining this much traction i'm pretty sure that they won't respond to me.

They notified me what? They closed my account first and then mailed me after? It's like killing a man first and then saying the reason why.

Re: Stop Using Digital Ocean Now: The Aftermath

#78

Post by anotherbadlogin »

zagi wrote: post #39

Hi, this is Ben, CEO and Co-Founder of DigitalOcean, we have received the document and will discuss the matter publicly. ----- All times are UTC. Our monitoring picked up a malicious UDP traffic pattern on 2013-09-08 00:58:23. A ticket was then opened with the customer at : 2013-09-08 01:05:55 roughly 7 minutes later. The customer informed us that it was a script that was crawling in the background. We informed the c…

That is a good and thorough response, but why wasn't this communicated to the customer in the INITIAL notice (assuming that the doc he posted is accurate)? And if that notice predated your investigation, you should at least have stated that strange UDP traffic was the reason for the takedown.

It's totally unacceptable to take a customer's account down without providing a specific reason IMMEDIATELY.

Also unacceptable are those 1-4 items you allegedly demanded for identity verification. That list is unprofessional at best.

Re: Stop Using Digital Ocean Now: The Aftermath

#79

Post by j_baker »

> And now after nearly 10 hours or so i still haven’t heart from DO. Also now that the HN topic is not on the front pages no one is getting updated about the situation. That’s why i wanted to write this post.

I won't speak for everyone, but I can personally live without having updates on the OP's Digital Ocean drama on my HN homepage at all times.

Re: Stop Using Digital Ocean Now: The Aftermath

#80

Post by toomuchtodo »

csomar wrote: post #62
opendais wrote: post #51

Earlier quoted context omitted.

So you shutdown the VM due to what appears to be a compromise. That is typical host behavior at least. Locking the user out of their account seems...odd, tho. I've never had that happen, personally, at any host.

Maybe Ben can explain what "locking" an account is actually is. If the customer did, in fact, break the TOS, I think it's logical to lock his account.

Locking the account should put it in a read-only mode. No changes to your DNS, no ability to start the droplet back up, firewall rules put in place at the hypervisor/network layer that block outbound traffic, but still allows you inbound access.
Post reply on HN