Stop Using Digital Ocean Now: The Aftermath • Hacker News Board

Re: Stop Using Digital Ocean Now: The Aftermath

#51

Post by opendais » Wed, Sep 25, 2013, 10:13 PM UTC

zagi wrote: post #39

Hi, this is Ben, CEO and Co-Founder of DigitalOcean, we have received the document and will discuss the matter publicly. ----- All times are UTC. Our monitoring picked up a malicious UDP traffic pattern on 2013-09-08 00:58:23. A ticket was then opened with the customer at : 2013-09-08 01:05:55 roughly 7 minutes later. The customer informed us that it was a script that was crawling in the background. We informed the c…

So you shutdown the VM due to what appears to be a compromise. That is typical host behavior at least.

Locking the user out of their account seems...odd, tho. I've never had that happen, personally, at any host.

Re: Stop Using Digital Ocean Now: The Aftermath

#52

Post by jrmiii » Wed, Sep 25, 2013, 10:14 PM UTC

sdogruyol wrote: post #46

zagi wrote: post #39

Hi, this is Ben, CEO and Co-Founder of DigitalOcean, we have received the document and will discuss the matter publicly. ----- All times are UTC. Our monitoring picked up a malicious UDP traffic pattern on 2013-09-08 00:58:23. A ticket was then opened with the customer at : 2013-09-08 01:05:55 roughly 7 minutes later. The customer informed us that it was a script that was crawling in the background. We informed the c…

Hello Ben, thanks for the response. Fırst of all at first ticket i told that the only possibility of having an UDP outgoing is that script that i wrote. Other than that i've no other activity or script that can generate that much traffic. Haven't you even considered that my droplet may be compromised or being attacked ? Instead of letting me know what exactly happened or which processes were running at that time you…

Looks like they looked at the code and determined it could not be UDP from your script.

I think they did consider your droplets to be compromised.

I think at issue here is that they have to assume that the droplet owner is the malicious party, if they don't lock your account, they can't stop you from creating more droplets.

I think you may have a point that they did not clearly explain to you why your account was locked.

However, this incident makes me more likely to continue using Digital Ocean. With the new private networking they have in NYC2, I for one am thrilled that they do this kind of proactive monitoring.

Re: Stop Using Digital Ocean Now: The Aftermath

#53

Post by sdogruyol » Wed, Sep 25, 2013, 10:15 PM UTC

veemjeem wrote: post #50

zagi wrote: post #39

Hi, this is Ben, CEO and Co-Founder of DigitalOcean, we have received the document and will discuss the matter publicly. ----- All times are UTC. Our monitoring picked up a malicious UDP traffic pattern on 2013-09-08 00:58:23. A ticket was then opened with the customer at : 2013-09-08 01:05:55 roughly 7 minutes later. The customer informed us that it was a script that was crawling in the background. We informed the c…

I think it's a bit sad that you have to go to such great lengths to protect your company's image. This was a very obvious case of customer fault. He admitted that he doesn't know if his server was hacked. The customer probably should have looked into the issue before making a inflammatory blog post. Maybe if you guys charged more you wouldn't get these ultra-cheap customers who think they should get 24/7 support for…

Being attacked or even getting compromised is customer fault ? Okay i get that. But what about closing the account instantly, not notifying the customer, accusing that customer of being a cheap liar and treating them in a bad way?

Re: Stop Using Digital Ocean Now: The Aftermath

#54

Post by veemjeem » Wed, Sep 25, 2013, 10:15 PM UTC

sdogruyol wrote: post #46

zagi wrote: post #39

Hi, this is Ben, CEO and Co-Founder of DigitalOcean, we have received the document and will discuss the matter publicly. ----- All times are UTC. Our monitoring picked up a malicious UDP traffic pattern on 2013-09-08 00:58:23. A ticket was then opened with the customer at : 2013-09-08 01:05:55 roughly 7 minutes later. The customer informed us that it was a script that was crawling in the background. We informed the c…

Hello Ben, thanks for the response. Fırst of all at first ticket i told that the only possibility of having an UDP outgoing is that script that i wrote. Other than that i've no other activity or script that can generate that much traffic. Haven't you even considered that my droplet may be compromised or being attacked ? Instead of letting me know what exactly happened or which processes were running at that time you…

Most cheap VPS providers would not go do that much work to decipher the root cause of the problem. When you have a server sending 1Gbps traffic, it's easier to shut it down than tail log files to figure out the cause. Digital Ocean can't do that for every customer that pays $5 a month. If you want a high level support, you should have picked a mid-level VPS like rackspace.

Re: Stop Using Digital Ocean Now: The Aftermath

#55

Post by gozzoo » Wed, Sep 25, 2013, 10:17 PM UTC

cortesoft wrote: post #49

zagi wrote: post #39

Hi, this is Ben, CEO and Co-Founder of DigitalOcean, we have received the document and will discuss the matter publicly. ----- All times are UTC. Our monitoring picked up a malicious UDP traffic pattern on 2013-09-08 00:58:23. A ticket was then opened with the customer at : 2013-09-08 01:05:55 roughly 7 minutes later. The customer informed us that it was a script that was crawling in the background. We informed the c…

I am curious to see the blog poster's response to this, as your course of action seems completely reasonable. May I ask what about the UDP traffic made it appear abusive?

how about 'totaling 1Gbps'?

Re: Stop Using Digital Ocean Now: The Aftermath

#56

Post by csomar » Wed, Sep 25, 2013, 10:17 PM UTC

zagi wrote: post #39

Hi, this is Ben, CEO and Co-Founder of DigitalOcean, we have received the document and will discuss the matter publicly. ----- All times are UTC. Our monitoring picked up a malicious UDP traffic pattern on 2013-09-08 00:58:23. A ticket was then opened with the customer at : 2013-09-08 01:05:55 roughly 7 minutes later. The customer informed us that it was a script that was crawling in the background. We informed the c…

After 1 day or so i tried to reach my server but couldn’t even ping or ssh to my server. I thought that DO was down but the truth was that they’ve locked my account without any notification. Can you imagine this?

The customer is lying in this case?

Re: Stop Using Digital Ocean Now: The Aftermath

#57

Post by chc » Wed, Sep 25, 2013, 10:17 PM UTC

sdogruyol wrote: post #46

zagi wrote: post #39

Hi, this is Ben, CEO and Co-Founder of DigitalOcean, we have received the document and will discuss the matter publicly. ----- All times are UTC. Our monitoring picked up a malicious UDP traffic pattern on 2013-09-08 00:58:23. A ticket was then opened with the customer at : 2013-09-08 01:05:55 roughly 7 minutes later. The customer informed us that it was a script that was crawling in the background. We informed the c…

Hello Ben, thanks for the response. Fırst of all at first ticket i told that the only possibility of having an UDP outgoing is that script that i wrote. Other than that i've no other activity or script that can generate that much traffic. Haven't you even considered that my droplet may be compromised or being attacked ? Instead of letting me know what exactly happened or which processes were running at that time you…

According to Ben's account, they did suggest to you that the droplet might be compromised. That is what they believed to be the case.

Re: Stop Using Digital Ocean Now: The Aftermath

#58

Post by sdogruyol » Wed, Sep 25, 2013, 10:17 PM UTC

sdogruyol wrote: post #46

zagi wrote: post #39

Hi, this is Ben, CEO and Co-Founder of DigitalOcean, we have received the document and will discuss the matter publicly. ----- All times are UTC. Our monitoring picked up a malicious UDP traffic pattern on 2013-09-08 00:58:23. A ticket was then opened with the customer at : 2013-09-08 01:05:55 roughly 7 minutes later. The customer informed us that it was a script that was crawling in the background. We informed the c…

Hello Ben, thanks for the response. Fırst of all at first ticket i told that the only possibility of having an UDP outgoing is that script that i wrote. Other than that i've no other activity or script that can generate that much traffic. Haven't you even considered that my droplet may be compromised or being attacked ? Instead of letting me know what exactly happened or which processes were running at that time you…

I was also tremendously happy with DO and their service. But what if you get your production apps down without even any notification and proper reasoning ? That's the thing which makes you feel insecure.

Re: Stop Using Digital Ocean Now: The Aftermath

#59

Post by skizm » Wed, Sep 25, 2013, 10:18 PM UTC

I use DO for a messing around on a small web app I'm developing. Within 24 hours of having my droplet up the root password was guessed and my machine was used for some DDoS. Granted I was an idiot for not changing the password immediately but I definitely felt like DO should just use ssh key validation like AWS does right off the bat. That deters attackers from even trying brute force attacks in the first place.

Anyway, I checked the logs and pretty much the minute my machine was deployed a script was guessing my password (lots of failed login attempts for "root" and "oracle"). This probably means someone knows DO's IP addresses and their automatically generated password scheme (all lower case alpha characters of a fixed length).

I reported the incident and destroyed my droplet since there was nothing important on it. When I heard back from DO I basically got (paraphrasing here) "you should install fail2ban next time". Case closed. I'm not a big customer or anything so I don't expect premium support or anything but I feel like someone should have looked into the attack a bit more. Seems like a lot of people are experiencing the same thing.

I guess what I am saying is you get what you pay for (it is only 5 bucks after all).

EDIT: Still using DO. I was just a bit more careful next time I deployed a droplet.

Re: Stop Using Digital Ocean Now: The Aftermath

#60

Post by csomar » Wed, Sep 25, 2013, 10:18 PM UTC

sdogruyol wrote: post #53

veemjeem wrote: post #50

Earlier quoted context omitted.

I think it's a bit sad that you have to go to such great lengths to protect your company's image. This was a very obvious case of customer fault. He admitted that he doesn't know if his server was hacked. The customer probably should have looked into the issue before making a inflammatory blog post. Maybe if you guys charged more you wouldn't get these ultra-cheap customers who think they should get 24/7 support for…

Being attacked or even getting compromised is customer fault ? Okay i get that. But what about closing the account instantly, not notifying the customer, accusing that customer of being a cheap liar and treating them in a bad way?

According to Ben, they did notify and talk to you about the issue.