Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Pastie.org host pulls hosting after DDoS attack

pastie.org

Post reply on HN
Print view View original

31–40 of 90 posts

Re: Pastie.org host pulls hosting after DDoS attack

#31

Post by postfuturist »

jsprinkles wrote: post #18
postfuturist wrote: post #5

DDOS attacks are a fact of life, nice to know that Rails Machine will throw you under the bus when one happens. Doesn't seem to fit their homepage description: "You write Rails apps. We deploy, manage, support, monitor, and scale them. Done."

So if someone throws multiple tens of gigabits at your customer, and your upstream threatens to turn off your entire hosting company, you would respond "no way, we're going the extra mile for our customer"? Rails Machine was, in all likelihood, compelled to act to either (a) preserve its relationship with is upstream or (b) preserve its relationship with its other paying customers that do not attract DoS attacks. You…

Any given site might not experience a DDoS attack, but if you run a hosting company, it will happen. The frequency depends on how many customers you have, how popular they are, etc. If a DDoS attack catches you by surprise, then you are very ill-prepared. One possible strategy is to throw the targeted customer under the bus and call it a day. For hosts of a certain size, that may be reasonable, as long as you clearly communicate that to customers.

I've seen these attacks mitigated single-handedly by an experienced fellow with no access to fancy equipment. I believe it went something like this, change the DNS to point at some EC2 instances to do front-end load balancing with some scripts that detect and drop connections from attacking IP addresses and do severe rate limiting. You can proxy legitimate requests back to the original servers.

Granted, not everyone knows how to do that in a pinch, but it shouldn't be a big deal for a hosting company. You only have to figure out how to do it once and when you detect a DDoS attack, flip the switch for that customer.

Re: Pastie.org host pulls hosting after DDoS attack

#32

Post by saurik »

seanp2k2 wrote: post #26
pestaa wrote: post #6

I can not understand these attacks. Why block a service that is free of charge, useful and did no harm? Unless of course this DDoS was not targeted, which makes even less sense to me. Also, why did Rails Machine throw out the site so quickly? If I choose to sponsor someone out of my free will, I'd do so without distinction from paying customers.

@everyone who doesn't work in hosting.... >"why did Rails Machine throw out the site so quickly?" If you run a datacenter, you pay for an uplink. That uplink has limited capacity. 4gbit, 10gbit...whatever. A big attack can saturate that link completely, so even with the biggest most expensive "mitigation device" on the market (some of this gear can get into the hundreds-of-thousands-of-dollars for /one/ device, mind…

As someone who does not work in hosting, I am somewhat surprised that "upstream bandwidth providers" don't have mediation strategies for DDoS attacks. It is my understanding that most of the asshats out there aren't sitting on Stuxnet: they have well under a hundred machines that they are able to flood traffic at you with (which is certainly more than enough). It seems like there are network-level mechanisms you can use to just block such an attack at the ingress connections. Is anyone willing to explain more about the issues here? (I, at least, would find it fascinating.)

Re: Pastie.org host pulls hosting after DDoS attack

#33

Post by petercooper »

bryanl wrote: post #4

This is a good chance to move over to http://gist.github.com for all your pasting needs.

I use both Gist and pastie.org. While I prefer Gist and wish it would just add support for disposable gists (without making me log out!), pastie.org is awesome for the disposable stuff on IRC, etc.

Re: Pastie.org host pulls hosting after DDoS attack

#34

Post by seanp2k2 »

heliostatic wrote: post #25
dknecht wrote: post #17

Earlier quoted context omitted.

We would love to help, and have offered.

Awesome; I hope Josh is able to get things back online. I'm curious how much support or assistance Rails Machine offered prior to pulling hosting.

>"I'm curious how much support or assistance Rails Machine offered prior to pulling hosting."

Me too, just out of curiosity, but if you have an attack that is large enough to be affecting other clients, you can't usually wait around too long for the targeted client to respond (and even if they did, there isn't typically much you can do for a large attack.) How long? I'd say about 5 minutes. The typical attack we see rises up over the course of a few minutes and lasts 6-48 hours.

Re: Pastie.org host pulls hosting after DDoS attack

#35

Post by jsprinkles »

postfuturist wrote: post #31
jsprinkles wrote: post #18

Earlier quoted context omitted.

So if someone throws multiple tens of gigabits at your customer, and your upstream threatens to turn off your entire hosting company, you would respond "no way, we're going the extra mile for our customer"? Rails Machine was, in all likelihood, compelled to act to either (a) preserve its relationship with is upstream or (b) preserve its relationship with its other paying customers that do not attract DoS attacks. You…

Any given site might not experience a DDoS attack, but if you run a hosting company, it will happen. The frequency depends on how many customers you have, how popular they are, etc. If a DDoS attack catches you by surprise, then you are very ill-prepared. One possible strategy is to throw the targeted customer under the bus and call it a day. For hosts of a certain size, that may be reasonable, as long as you clearly…

> I've seen these attacks mitigated single-handedly by an experienced fellow with no access to fancy equipment. I believe it went something like this, change the DNS to point at some EC2 instances to do front-end load balancing with some scripts that detect and drop connections from attacking IP addresses and do severe rate limiting. You can proxy legitimate requests back to the original servers.

That isn't how it works, but I appreciate you trying to explain it to me (I have personal experience as a former employee of a hosting company, in this exact position). A typical DoS attack is designed to flood the pipe, usually with UDP. UDP spam attacks are far simpler than layer 7 attacks that you describe, and layer 7 attacks (if small enough) are easier to handle.

As a point of reference, I don't even have access to a botnet these days but I could knock your home cable connection offline in about five minutes of my time.

In your scenario, if you were to update DNS to point the victim at some EC2 instances, you'd accomplish DoS attacking EC2 instead as they only offer you gigabit connectivity on most EC2 instances. If the DoS attack is multiple gigabit, that isn't going to help you unless Amazon steps in and mitigates about as best they can. A DoS mitigation strategy generally has nothing to do with your application. If an engineer's first answer to a huge DoS attack -- as in, larger than the uplink -- is iptables or "scripts to detect and drop connections", that engineer is uninformed. I'm sorry.

I have personally witnessed a 40 gigabit/sec attack. I'm sure the smart engineers at CloudFlare have seen bigger. Upstreams don't give a shit what's flowing across the wire, and a small TCP SYN flood directed at your server won't gain any attention from your host or their upstream. A multiple-gigabit attack that takes down the entire uplink will.

Re: Pastie.org host pulls hosting after DDoS attack

#36

Post by jsprinkles »

saurik wrote: post #32
seanp2k2 wrote: post #26

Earlier quoted context omitted.

@everyone who doesn't work in hosting.... >"why did Rails Machine throw out the site so quickly?" If you run a datacenter, you pay for an uplink. That uplink has limited capacity. 4gbit, 10gbit...whatever. A big attack can saturate that link completely, so even with the biggest most expensive "mitigation device" on the market (some of this gear can get into the hundreds-of-thousands-of-dollars for /one/ device, mind…

As someone who does not work in hosting, I am somewhat surprised that "upstream bandwidth providers" don't have mediation strategies for DDoS attacks. It is my understanding that most of the asshats out there aren't sitting on Stuxnet: they have well under a hundred machines that they are able to flood traffic at you with (which is certainly more than enough). It seems like there are network-level mechanisms you can…

Dropping UDP is a start, because most of these attacks randomize the UDP source address. The real problem is AS operators who let forged UDP addresses escape their network. If your outgoing edge ACL is not dropping source addresses that do not belong to you, you are doing it wrong. Period. Full stop.

If the packets are random UDP sources, then there's not really much you can do on the receiving end except strategies that involve the destination address. There's really only one effective one: nulling it.

Re: Pastie.org host pulls hosting after DDoS attack

#37

Post by guelo »

jsprinkles wrote: post #35
postfuturist wrote: post #31

Earlier quoted context omitted.

Any given site might not experience a DDoS attack, but if you run a hosting company, it will happen. The frequency depends on how many customers you have, how popular they are, etc. If a DDoS attack catches you by surprise, then you are very ill-prepared. One possible strategy is to throw the targeted customer under the bus and call it a day. For hosts of a certain size, that may be reasonable, as long as you clearly…

> I've seen these attacks mitigated single-handedly by an experienced fellow with no access to fancy equipment. I believe it went something like this, change the DNS to point at some EC2 instances to do front-end load balancing with some scripts that detect and drop connections from attacking IP addresses and do severe rate limiting. You can proxy legitimate requests back to the original servers. That isn't how it wo…

Would have appreciated if you did explain what an effective mitigation strategy is.

Re: Pastie.org host pulls hosting after DDoS attack

#38

Post by jsprinkles »

guelo wrote: post #37
jsprinkles wrote: post #35

Earlier quoted context omitted.

> I've seen these attacks mitigated single-handedly by an experienced fellow with no access to fancy equipment. I believe it went something like this, change the DNS to point at some EC2 instances to do front-end load balancing with some scripts that detect and drop connections from attacking IP addresses and do severe rate limiting. You can proxy legitimate requests back to the original servers. That isn't how it wo…

Would have appreciated if you did explain what an effective mitigation strategy is.

Depending on your position, there really isn't one. The hosting company can implement one, which they might be able to insert in your path if you are at the receiving end of an attack. There are Cisco products and a bunch of up-and-comers that can do this.

If you're a "typical startup" with an Amazon footprint, you have no mitigation strategy for flooding attacks aside from not attracting them. If someone points multiple gigabit at you, there is just about nothing you can do except hope Amazon can do something.

Re: Pastie.org host pulls hosting after DDoS attack

#39

Post by saurik »

jsprinkles wrote: post #36
saurik wrote: post #32

Earlier quoted context omitted.

As someone who does not work in hosting, I am somewhat surprised that "upstream bandwidth providers" don't have mediation strategies for DDoS attacks. It is my understanding that most of the asshats out there aren't sitting on Stuxnet: they have well under a hundred machines that they are able to flood traffic at you with (which is certainly more than enough). It seems like there are network-level mechanisms you can…

Dropping UDP is a start, because most of these attacks randomize the UDP source address. The real problem is AS operators who let forged UDP addresses escape their network. If your outgoing edge ACL is not dropping source addresses that do not belong to you, you are doing it wrong . Period. Full stop. If the packets are random UDP sources, then there's not really much you can do on the receiving end except strategies…

Okay, but then why null route the entire machine, as opposed to dropping just the UDP packets going towards it? If you already have routing infrastructure that can disseminate "attempts to access this IP address will fail" it does not seem a stretch to disseminate "attempts to access this IP address over UDP will fail; over TCP there is no issue". Are "upstream bandwidth providers" really that impotent against that kind of issue? :(

I, personally, have absolutely no need to have incoming UDP packets of any kind at all entering my network, and can not come up with a reason why any web hosting company would: it seems like it should almost be a question on your bandwidth contract "will you need UDP (recommendation: no)". (Given the simplicity of the UDP problem, I personally assumed that the complexity would come from state-ful TCP filtering issues.)

Re: Pastie.org host pulls hosting after DDoS attack

#40

Post by jsprinkles »

saurik wrote: post #39
jsprinkles wrote: post #36

Earlier quoted context omitted.

Dropping UDP is a start, because most of these attacks randomize the UDP source address. The real problem is AS operators who let forged UDP addresses escape their network. If your outgoing edge ACL is not dropping source addresses that do not belong to you, you are doing it wrong . Period. Full stop. If the packets are random UDP sources, then there's not really much you can do on the receiving end except strategies…

Okay, but then why null route the entire machine, as opposed to dropping just the UDP packets going towards it? If you already have routing infrastructure that can disseminate "attempts to access this IP address will fail" it does not seem a stretch to disseminate "attempts to access this IP address over UDP will fail; over TCP there is no issue". Are "upstream bandwidth providers" really that impotent against that k…

That's what most providers (certainly the one I've worked at) do, in fact, do, is null route the entire machine. Rather than leave you nulled in the router for weeks waiting for the attack to subside, eventually, they'll just cut you loose. That's the typical form these things take.

It would be tempting to blackhole UDP, but it's just as easy to flood a pipe with TCP. You don't need an established connection to get packets in the pipe, and there are many mismanaged networks on planet Earth (as I alluded to) that let just about anything exit. A small botnet of five or ten machines with gigabit uplinks on poorly-managed networks is enough to be a force to reckon with. When I was an administrator on IRC, Cisco routers themselves were common targets to exploit and use for this purpose; often, they're directly plugged into gigabit or even ten gigabit connectivity, and there are really easy ways in IOS to perform a DoS attack.

I can't speak to the decision Pastie's host made in this case, but it sounds like since Rails Machine was donating resources (and Pastie wasn't paying them), Rails Machine must act as a smart company -- certainly, any entrepreneur who enjoys Hacker News would sympathize -- and protect its income flow. Which means, you mess with my customers, I fire you.

Post reply on HN