Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

February 28th DDoS Incident Report

githubengineering.com

Post reply on HN
Print view View original

71–80 of 114 posts

Re: February 28th DDoS Incident Report

#71

Post by mirashii »

scurvy wrote: post #69
corndoge wrote: post #15

Very cool that they are able to change BGP advertisements from ChatOps, achieve convergence and mitigate the attack in all of 4 minutes, that is some insane engineering.

Meh, or just block UDP to your networks that have no reason to run UDP. Every carrier will do upstream ACL's these days. 5 years ago that wasn't the case. These days, they all do. Some free, some charge. re: chat ops vs a web page. It's just a single BGP advertisement -- big whoop. Chatops is just hipster famous right now.

A snarky reply like this comes up every time there's discussion of a DDOS, but it ignores the fact that there is some point that has to filter that UDP traffic, and if that point is saturated, the DDOS still worked. Mitigating attacks of this size isn't a firewall rule or a support ticket with your ISP.

Re: February 28th DDoS Incident Report

#72

Post by megous »

lol768 wrote: post #64
feelin_googley wrote: post #47

Earlier quoted context omitted.

"... you need to have a stupidly complicated JavaScript engine enabled and accessible to a webpage..." Does anyone have an example of this webpage? Unless I am engaging in e-commerce, I do not run a browser JavaScript engine. I rarely if ever encounter a webpage that truly "requires" one. GitHub certainly does not require JavaScript for me to use it via www.

> Does anyone have an example of this webpage? It's a requirement when a CloudFlare'd site is in "I'm under attack" mode.

Yes that's it.

https://blog.cloudflare.com/introducing-im-under-attack-mode...

Re: February 28th DDoS Incident Report

#73

Post by eridius »

arkadiyt wrote: post #65
r1ch wrote: post #13

This is a great example of why it's important to pick secure defaults when writing software, especially software that is often deployed on high bandwidth servers or cloud instances. If no listening interfaces are specified then the default should be to exit with an error, not listen on everything! I also wonder if you can store something in a memcached cache that looks like a valid request, then reflect that with the…

Shortly after Cloudflare's blog post, memcached pushed a commit that disabled UDP by default: https://github.com/memcached/memcached/commit/dbb7a8af90054b...

That's good, but they should go a step further and stop listening on all addresses by default.

Re: February 28th DDoS Incident Report

#74

Post by scurvy »

mirashii wrote: post #71
scurvy wrote: post #69

Earlier quoted context omitted.

Meh, or just block UDP to your networks that have no reason to run UDP. Every carrier will do upstream ACL's these days. 5 years ago that wasn't the case. These days, they all do. Some free, some charge. re: chat ops vs a web page. It's just a single BGP advertisement -- big whoop. Chatops is just hipster famous right now.

A snarky reply like this comes up every time there's discussion of a DDOS, but it ignores the fact that there is some point that has to filter that UDP traffic, and if that point is saturated, the DDOS still worked. Mitigating attacks of this size isn't a firewall rule or a support ticket with your ISP.

Snark or not, the traffic is filtered upstream before your handoff. If you pick your carriers well, there's not a problem. Many carriers have turned upstream filters into a product. NTT's DPS Lite springs to mind.

This just comes down to experience and knowing how to build a network. I'd think that Github would have people knowing how to architect this. They've been through a few DDoS before.

Edit: It looks like Github uses NTT for traffic. Hello Github Netops person, you need to call your sales rep and turn on DPS Lite. It's like $100 per 10gig port and you get full ACLs. Telia, another one of your carriers, will do this too. At least they have for me. Level3 though? Lol kick that sorry network to the curb

Also, get another /22 allocation so you can at least separate out your DC-origin traffic from your customer traffic.

Re: February 28th DDoS Incident Report

#75

Post by vannevar »

These attacks are often described as denial of service attacks, but I wonder if many of them aren't employed as cover for an intrusion attempt. Is it possible that intrusive traffic could be mixed in with such an attack?

Re: February 28th DDoS Incident Report

#76

Post by Benjammer »

vannevar wrote: post #75

These attacks are often described as denial of service attacks, but I wonder if many of them aren't employed as cover for an intrusion attempt. Is it possible that intrusive traffic could be mixed in with such an attack?

A DoS attack is, by literal definition, an attempt to overwhelm a host until it is forced to _deny service_ to valid user requests. Are there intrusion techniques that both bring down the server and break into it at the same time? I'm not a security expert, but that doesn't seem like it makes a whole lot of sense to me.

Re: February 28th DDoS Incident Report

#77

Post by lbotos »

scurvy wrote: post #69
corndoge wrote: post #15

Very cool that they are able to change BGP advertisements from ChatOps, achieve convergence and mitigate the attack in all of 4 minutes, that is some insane engineering.

Meh, or just block UDP to your networks that have no reason to run UDP. Every carrier will do upstream ACL's these days. 5 years ago that wasn't the case. These days, they all do. Some free, some charge. re: chat ops vs a web page. It's just a single BGP advertisement -- big whoop. Chatops is just hipster famous right now.

AS much as I hate chatops as a fad, it makes it really easy for others to also see what happened and follow along.

"Hey Jim, Did you go adjust that thing?"

vs.

"10:15am [JIM] /chatbot adjust x to y"

Re: February 28th DDoS Incident Report

#78

Post by paulie_a »

jgrahamc wrote: post #23
hyperpower wrote: post #2

Wow, 1.35Tbps? That's a lot for a DoS attack, right?

Yes and there are a lot of attacks of very, very large sizes going on. Over the last few days we've mitigated some huge attacks. Luckily, everyone is working together to rate limit and clean up this problem.

This is honestly why I have no problem with things like brickerbot.

I consider it community service.

Re: February 28th DDoS Incident Report

#79

Post by feelin_googley »

megous wrote: post #72
lol768 wrote: post #64

Earlier quoted context omitted.

> Does anyone have an example of this webpage? It's a requirement when a CloudFlare'd site is in "I'm under attack" mode.

Yes that's it. https://blog.cloudflare.com/introducing-im-under-attack-mode...

"We've also designed the new checks to not block search engine crawlers, your existing whitelists, and other pre-vetted traffic. As a result, enabling I'm Under Attack Mode will not negatively impact your SEO or known legitimate visitors."

"What's also cool is that data on attack traffic that doesn't pass the automatic checks is fed back into CloudFlare's system to further enhance our traditional protections."

"[P]re-vetted traffic"?

Does this mean they are whitelisting certain IP addresses?

GoogleBot can make hundreds of requests and double digit parallel connections, as frequently as they like, but a single user making one request and one connection is blocked because they are not enabling Javascript?

This does not sound like an intelligent filter.

"[K]nown legitimate visitors"?

What exactly does this mean? How do they "know" a visitor is "legitimate"?

"[A]ttack traffic that doesn't pass the automatic checks..."

Is it possible that non-attack traffic could fail the checks?

What about a single request from a single IP that does not pass the checks because the user does not have JavaScript enabled?

Does the IP address end up on some blacklist?

I have seen Cloudflare reject connections based on certain user agent strings, a header that everyone knows is user-configurable, arbitrary and not a reliable indicator of anything meaningful.

This despite volumes of "legitimate" traffic from same source preceding it. Pick wrong user agent string and suddenly the source becomes "illegitimate".

It would be interesting to know what "checks" the Javascript in question is performing.

Re: February 28th DDoS Incident Report

#80

Post by vannevar »

Benjammer wrote: post #76
vannevar wrote: post #75

These attacks are often described as denial of service attacks, but I wonder if many of them aren't employed as cover for an intrusion attempt. Is it possible that intrusive traffic could be mixed in with such an attack?

A DoS attack is, by literal definition, an attempt to overwhelm a host until it is forced to _deny service_ to valid user requests. Are there intrusion techniques that both bring down the server and break into it at the same time? I'm not a security expert, but that doesn't seem like it makes a whole lot of sense to me.

Maybe in the milliseconds between packet swarms, or immediately before or after? Just seems like a lot of resources to pour into an attack that was defeated in a few minutes. To what end?
Post reply on HN