Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Stop Using Digital Ocean Now: The Aftermath

serdardogruyol.com

Post reply on HN
Print view View original

91–100 of 102 posts

Re: Stop Using Digital Ocean Now: The Aftermath

#91

Post by veemjeem »

sdogruyol wrote: post #83
veemjeem wrote: post #43

Earlier quoted context omitted.

I'm guessing that most people who run a server may not even realize when they get hacked. These people (you included) probably should not run their own servers and stick to PaaS solutions like Heroku or Google App Engine. It happens all the time to guys who think they can install & maintain Wordpress themselves. You probably should have analyzed the issue before making a blog post about it. I have had servers hacked…

You miss the point here. It's not about getting hacked or so. It's their way of handling it. Like i said they kill it first and then tell you the reason why. What's the point in it ?

My guess is that Digital Ocean is the first VPS you've ever used. I don't know of any provider that will wait 10 hours for you to respond while the server sends out 1Gbps traffic. If they let the server continue running, the bandwidth costs would probably be more than 100x your monthly subscription cost. The VPS providers that don't shutdown your server will usually just bill you the bandwidth costs and you won't notice your server got hacked until you see your $800 bandwidth bill.

Killing it first is the proper course of action here. I understand you're new, so there's always a first time. I just disagree that you have to make an inflammatory blog post about it.

Re: Stop Using Digital Ocean Now: The Aftermath

#92

Post by orclev »

sdogruyol wrote: post #77
sdogruyol wrote: post #66

Earlier quoted context omitted.

You know what they stopped answering my ticket after first response. If it wasn't HN post gaining this much traction i'm pretty sure that they won't respond to me.

They notified me what? They closed my account first and then mailed me after? It's like killing a man first and then saying the reason why.

Well, we've now heard two different stories. According to Ben, there were two events. The first time they took your server offline and contacted you to tell you about it, they did not however lock your account. When the second event occurred with you being unable to provide a reasonable explanation and apparently being unable to deal with whatever compromise occurred, they took the system down again and this time also locked your account.

Your explanation of the events only mentions a single occurrence, at which time your account was locked in addition to the server being shutdown.

In either case, a server engaging in malicious activity, is normally taken offline as soon as the malicious activity is discovered to prevent further damage from occurring. You'd get a similar response from just about any other hosting provider you care to name. If you're lucky, and they're feeling generous, they might work with you to find the problem prior to taking the system offline, but normally standard procedure is to take the system offline immediately. The fact that you seem surprised about this shows you don't have much experience administering your own servers.

The standard response usually goes something like:

1) Server is discovered doing something malicious

2) Server is taken offline/shutdown

3) Administrator is notified

3a) Read only copy of the old server HD is brought online on a new server to allow administrator to perform forensic and backup work*

4) Administrator must bring up new server to replace old compromised one

*Sometimes the provider will provide you the old HD image, sometimes not, really depends on the provider.

Re: Stop Using Digital Ocean Now: The Aftermath

#93

Post by debian3 »

sdogruyol wrote: post #77
sdogruyol wrote: post #66

Earlier quoted context omitted.

You know what they stopped answering my ticket after first response. If it wasn't HN post gaining this much traction i'm pretty sure that they won't respond to me.

They notified me what? They closed my account first and then mailed me after? It's like killing a man first and then saying the reason why.

They basically killed the killer, 1gbit udp probably killed the website or servers of someone else.

Re: Stop Using Digital Ocean Now: The Aftermath

#94

Post by debian3 »

sdogruyol wrote: post #58
sdogruyol wrote: post #46

Earlier quoted context omitted.

Hello Ben, thanks for the response. Fırst of all at first ticket i told that the only possibility of having an UDP outgoing is that script that i wrote. Other than that i've no other activity or script that can generate that much traffic. Haven't you even considered that my droplet may be compromised or being attacked ? Instead of letting me know what exactly happened or which processes were running at that time you…

I was also tremendously happy with DO and their service. But what if you get your production apps down without even any notification and proper reasoning ? That's the thing which makes you feel insecure.

The only thing who feel insecure here is your vps.

Re: Stop Using Digital Ocean Now: The Aftermath

#95

Post by debian3 »

dgdkfjghkdfhkhg wrote: post #85

Never heard on DO but will use them for a project soon - looks great

Indeed, never though a $5 provider would go that far for a noob on their network. Looks like their monitoring system is good too, took them only a couple of minute to take him down, which is good.

Re: Stop Using Digital Ocean Now: The Aftermath

#96

Post by erichocean »

adamlj wrote: post #31

About a week ago I got to experience Digital Oceans very tight suspension policy first hand. What happened was that one of the accounts I manage was suspended. I had to go through a very long and detailed validation process before they understood that they had done wrong (they admitted to doing an error and apologized). But when my account was activated again my droplet had been destroyed.. After some more time they…

These kinds of seemingly random suspensions at DO are starting to concern me. I've got a service with ~10K active users I'm migrating to DO as we speak, but man! I'm starting to feel like it might be professionally negligent to do so if they're trigger-happy with shutting down instances.

It'd be nice if DO had some way to communicate "hey, this VM matters to my business, PLEASE don't do anything stupid/automated without contacting me first", but that's probably too much to ask for the cost.

Gah. Back to AWS. This sucks. :/

Re: Stop Using Digital Ocean Now: The Aftermath

#97

Post by opendais »

csomar wrote: post #62
opendais wrote: post #51

Earlier quoted context omitted.

So you shutdown the VM due to what appears to be a compromise. That is typical host behavior at least. Locking the user out of their account seems...odd, tho. I've never had that happen, personally, at any host.

Maybe Ben can explain what "locking" an account is actually is. If the customer did, in fact, break the TOS, I think it's logical to lock his account.

That would be ideal. Its sounds like it was more than preventing changes to the account information tho.

Re: Stop Using Digital Ocean Now: The Aftermath

#98

Post by waps »

dragonwriter wrote: post #73
sdogruyol wrote: post #53

Earlier quoted context omitted.

Being attacked or even getting compromised is customer fault ? Okay i get that. But what about closing the account instantly, not notifying the customer, accusing that customer of being a cheap liar and treating them in a bad way?

Suspending -- what was done in the first instance -- a system which is engaging in an apparent DDOS is a perfectly reasonable action. Locking the account when the explanation given is inconsistent with the observed behavior it was supposed to explain and the system is again engaing in an apparent DDOS is also a perfectly reasonable action. It also seems from your posts that both times you were notified of the action…

Sadly, suspending a system which is the target of a ddos is the only action an ISP can take. Even if you're not ddossing anyone but merely the target you'll get suspended, assuming you're not paying enough to make it worth it for the ISP

Re: Stop Using Digital Ocean Now: The Aftermath

#99

Post by quicksilver03 »

zagi wrote: post #39

Hi, this is Ben, CEO and Co-Founder of DigitalOcean, we have received the document and will discuss the matter publicly. ----- All times are UTC. Our monitoring picked up a malicious UDP traffic pattern on 2013-09-08 00:58:23. A ticket was then opened with the customer at : 2013-09-08 01:05:55 roughly 7 minutes later. The customer informed us that it was a script that was crawling in the background. We informed the c…

Instead of locking the account, couldn't you just have blocked all outgoing UDP packets from the IP address of the customer's droplet?

Blocking UDP would have stopped the attack, it would have given you a time window to contact the customer (allowing for time zone differences) and would have given both parties a chance to resolve the issue in private and with much less drama.

(speaking as a current customer of DigitalOcean, using UDP (thru collectd) to monitor my droplet and starting to have uneasy feelings).

Re: Stop Using Digital Ocean Now: The Aftermath

#100

Post by dclausen »

sdogruyol wrote: post #83
veemjeem wrote: post #43

Earlier quoted context omitted.

I'm guessing that most people who run a server may not even realize when they get hacked. These people (you included) probably should not run their own servers and stick to PaaS solutions like Heroku or Google App Engine. It happens all the time to guys who think they can install & maintain Wordpress themselves. You probably should have analyzed the issue before making a blog post about it. I have had servers hacked…

You miss the point here. It's not about getting hacked or so. It's their way of handling it. Like i said they kill it first and then tell you the reason why. What's the point in it ?

As a customer of DO, I'm glad they turned you off. Your ignorance was negatively affecting my service.
Post reply on HN