Ask HN: How do you handle DDoS attacks? • Hacker News Board

Re: Ask HN: How do you handle DDoS attacks?

#61

Post by tumdum_ » Sun, Aug 28, 2016, 7:37 PM UTC

DyslexicAtheist wrote: post #34

tumdum_ wrote: post #2

https://www.cloudflare.com

You mean the biggest MiTM on the web?[0] The only reason why they're not constantly called out by serious infosec folk for their scam is because they hire guys also involved in DefCon/BlackHat planning (try to sneak a hostile talk against Cloudflare past REDACTED[2] who btw is also advising Mr. Robot). It's lobbying at its finest. [0] https://scotthelme.co.uk/tls-conundrum-and-leaving-cloudflar... [1] https://blog.to…

Yes, I'm well aware that cloudflare is mitm, yet for my needs I've decided that this is not a problem.

I can see that you are not happy with what they provide. Luckily theirs service is not forced on you. Neither do you have to use it, nor visit server that use it.

Re: Ask HN: How do you handle DDoS attacks?

#62

Post by simon_acca » Sun, Aug 28, 2016, 7:45 PM UTC

dogma1138 wrote: post #59

Post was too long http://pastebin.com/48J9Ufdd : Random "wisdom", not in any particular order more like do's and dont's that I picked up with dealing with and executing DoS/DDoS attacks. Testing, testing, testing, regardless of how you choose and what you implement your mitigation test it and test it well because there are a lot of things you need to know. Know and understand exact effect that the DDOS/DoS mitigation…

Why would you turn off compression? A given amount of requests is more burdensome on the CPU when decompression is to be performed.

Re: Ask HN: How do you handle DDoS attacks?

#63

Post by dogma1138 » Sun, Aug 28, 2016, 7:53 PM UTC

simon_acca wrote: post #62

dogma1138 wrote: post #59

Post was too long http://pastebin.com/48J9Ufdd : Random "wisdom", not in any particular order more like do's and dont's that I picked up with dealing with and executing DoS/DDoS attacks. Testing, testing, testing, regardless of how you choose and what you implement your mitigation test it and test it well because there are a lot of things you need to know. Know and understand exact effect that the DDOS/DoS mitigation…

Why would you turn off compression? A given amount of requests is more burdensome on the CPU when decompression is to be performed.

Because you want to test it with it on and off and understand the impact it has. Text (Javascript, HTML) compresses rather well (on average 3 to 1 or better) it's considerably more likely that you are going to hit a networking limit than than a CPU limit when not using GZIP or any other compression method. So when you do stress or load testing you really want to issue the same requests with and without the Accept-Encoding: compress, gzip or any similar headers.

Also in some cases you want to disable accepting GZIP on the server side completely because if you accept GZIP encoded requests an attacker can send very large requests that compress very well forcing you to decompress them eating up a lot of memory and CPU cycles on your side only to discard them. In principle you want to accept only non-compressed requests but send compressed responses to save bandwidth, but in any case you want to know how your service/application scaling works in with all cases and combinations.

Re: Ask HN: How do you handle DDoS attacks?

#64

Post by bch » Sun, Aug 28, 2016, 8:03 PM UTC

simon_acca wrote: post #62

dogma1138 wrote: post #59

Post was too long http://pastebin.com/48J9Ufdd : Random "wisdom", not in any particular order more like do's and dont's that I picked up with dealing with and executing DoS/DDoS attacks. Testing, testing, testing, regardless of how you choose and what you implement your mitigation test it and test it well because there are a lot of things you need to know. Know and understand exact effect that the DDOS/DoS mitigation…

Why would you turn off compression? A given amount of requests is more burdensome on the CPU when decompression is to be performed.

Assuming the remote is actually decompressing, and the origin isn't dynamically compressing for each request.

Re: Ask HN: How do you handle DDoS attacks?

#65

Post by dogma1138 » Sun, Aug 28, 2016, 8:06 PM UTC

bch wrote: post #64

simon_acca wrote: post #62

Earlier quoted context omitted.

Why would you turn off compression? A given amount of requests is more burdensome on the CPU when decompression is to be performed.

Assuming the remote is actually decompressing, and the origin isn't dynamically compressing for each request.

Even if the remote is decompressing it doesn't matter a botnet owner or some one who compromised a few AWS accounts doesn't care it won't really slow them down. CPU's are pretty fast at compression and when a site is easily 3-4 times the size when not served compressed you are going to hit your network cap limit faster than exhausting the CPU even with dynamic compression and no caching under most circumstances.

Re: Ask HN: How do you handle DDoS attacks?

#66

Post by jaypaulynice » Sun, Aug 28, 2016, 8:14 PM UTC

I work at a CDN/Security engineering company, but this is just my view.

First off you need to determine where the attack is coming from. You could redirect based on IP/request headers in a .htaccess file or apache rules.

Your next bet is to distribute/auto-scale your application if possible.

You need to setup a web application firewall that sits in front of your web servers and analyzes the requests/responses that hit the web servers. A lot of the ddos campaigns are easy to identify based on the request headers/IP/Geo and requests/second.

It's not hard to write a small web server/proxy to do this, but it would be best left to someone who knows what they're doing because you don't want to block real user requests. You can use ModSecurity's open source WAF for apache/nginx, but again you have to know what you're doing.

When I faced this issue, I wrote a small web server/proxy here that you can start on port 80:

https://github.com/julesbond007/java-nio-web-server

Here I wrote some rules to drop the request if it's malicious:

https://github.com/julesbond007/java-nio-web-server/blob/mas...

Re: Ask HN: How do you handle DDoS attacks?

#67

Post by DivineTraube » Sun, Aug 28, 2016, 8:24 PM UTC

We (Baqend) use an approach that is somewhat different from what has been proposed here so far:

- Every one of our servers rate limits critical resources, i.e. the ones that cannot be cached. The servers autoscale when neccessary.

- As rate limiting is expensive (you have to remember every IP/resource pair across all servers) we keep that state in a locally approximated representation using a ring buffer of Bloom filters.

- Every cacheable resource is cached in our CDN (Fastly) with TTLs estimated via an exponential decay model over past reads and writes.

- When a user exceeds his rate limit the IP is temporarily banned at the CDN-level. This is achieved through custom Varnish VCLs deployed in Fastly. Essentially the logic relies on the bakend returning a 429 Too Many Requests for a particular URL that is then cached using the requester's ID as a hash key. Using the restart mechanism of Varnish's state machine, this can be done without any performance penalty for normal requests. The duration of the ban simply is the TTL.

TL;DR: Every abusive request is detected at the backend servers using approximations via Bloom filters and then a temporary ban is cached in the CDN for that IP.

Re: Ask HN: How do you handle DDoS attacks?

#68

Post by ryanlol » Sun, Aug 28, 2016, 8:33 PM UTC

DivineTraube wrote: post #67

We (Baqend) use an approach that is somewhat different from what has been proposed here so far: - Every one of our servers rate limits critical resources, i.e. the ones that cannot be cached. The servers autoscale when neccessary. - As rate limiting is expensive (you have to remember every IP/resource pair across all servers) we keep that state in a locally approximated representation using a ring buffer of Bloom fil…

Have you guys been DDoSd before? All that sounds very nice until someone UDP floods you and you get nullrouted.

Looks like you're hosting at least some stuff at Hetzner, they're not going to do any filtering for you.

Re: Ask HN: How do you handle DDoS attacks?

#69

Post by jaypaulynice » Sun, Aug 28, 2016, 8:45 PM UTC

buro9 wrote: post #38

I've faced DoS attacks for years as I run internet forums. The simple advice for layer 7 (application) attacks: 1. Design your web app to be incredibly cacheable 2. Use your CDN to cache everything 3. When under attack seek to identify the site (if you host more than one) and page that is being attacked. Force cache it via your CDN of choice. 4. If you cannot cache the page then move it. 5. If you cannot cache or mov…

Have you heard about cache busting? Someone just needs to request a page that's not cached and the request will always hit your web servers.

Re: Ask HN: How do you handle DDoS attacks?

#70

Post by rootlocus » Sun, Aug 28, 2016, 8:49 PM UTC

DivineTraube wrote: post #67

We (Baqend) use an approach that is somewhat different from what has been proposed here so far: - Every one of our servers rate limits critical resources, i.e. the ones that cannot be cached. The servers autoscale when neccessary. - As rate limiting is expensive (you have to remember every IP/resource pair across all servers) we keep that state in a locally approximated representation using a ring buffer of Bloom fil…

I'm sorry, I know it's irrelevant, offtopic and I'm a horrible person but... "Baqend"? Who came up with this name? Was there some brainstorming involved and that was the best candidate? What does the branding say about your business? How is it pronounced?