February 28th DDoS Incident Report • Hacker News Board

Re: February 28th DDoS Incident Report

#81

Post by w0rd-driven » Fri, Mar 02, 2018, 12:25 AM UTC

vannevar wrote: post #80

Benjammer wrote: post #76

Earlier quoted context omitted.

A DoS attack is, by literal definition, an attempt to overwhelm a host until it is forced to _deny service_ to valid user requests. Are there intrusion techniques that both bring down the server and break into it at the same time? I'm not a security expert, but that doesn't seem like it makes a whole lot of sense to me.

Maybe in the milliseconds between packet swarms, or immediately before or after? Just seems like a lot of resources to pour into an attack that was defeated in a few minutes. To what end?

There's validity to the approach of sending packet swarms to cover intrusion attempts but the traffic levels were more than a small amount of cover. It is possible that it was designed as a smokescreen and someone's calculations were wildly incorrect. No one's safe from off by one errors :>

Re: February 28th DDoS Incident Report

#82

Post by scurvy » Fri, Mar 02, 2018, 12:26 AM UTC

lbotos wrote: post #77

scurvy wrote: post #69

Earlier quoted context omitted.

Meh, or just block UDP to your networks that have no reason to run UDP. Every carrier will do upstream ACL's these days. 5 years ago that wasn't the case. These days, they all do. Some free, some charge. re: chat ops vs a web page. It's just a single BGP advertisement -- big whoop. Chatops is just hipster famous right now.

AS much as I hate chatops as a fad, it makes it really easy for others to also see what happened and follow along. "Hey Jim, Did you go adjust that thing?" vs. "10:15am [JIM] /chatbot adjust x to y"

Fair enough, I guess visibility, in the one thing everyone's looking at in a crisis, is good.

Re: February 28th DDoS Incident Report

#83

Post by jtl999 » Fri, Mar 02, 2018, 12:26 AM UTC

eridius wrote: post #73

arkadiyt wrote: post #65

Earlier quoted context omitted.

Shortly after Cloudflare's blog post, memcached pushed a commit that disabled UDP by default: https://github.com/memcached/memcached/commit/dbb7a8af90054b...

That's good, but they should go a step further and stop listening on all addresses by default.

And it's going to take a while for the new version to propagate to a released version, then to distributions, then to customer images and scripts, etc.

Re: February 28th DDoS Incident Report

#84

Post by dingo_bat » Fri, Mar 02, 2018, 12:47 AM UTC

feelin_googley wrote: post #79

megous wrote: post #72

Earlier quoted context omitted.

Yes that's it. https://blog.cloudflare.com/introducing-im-under-attack-mode...

"We've also designed the new checks to not block search engine crawlers, your existing whitelists, and other pre-vetted traffic. As a result, enabling I'm Under Attack Mode will not negatively impact your SEO or known legitimate visitors." "What's also cool is that data on attack traffic that doesn't pass the automatic checks is fed back into CloudFlare's system to further enhance our traditional protections." "[P]re…

I think cloudfare just needs to reject some percentage of all connections to reduce load on the website. The algorithm to decide which to accept/reject is meaningless as long as they hit the required reject percentage.

Re: February 28th DDoS Incident Report

#85

Post by vlan0 » Fri, Mar 02, 2018, 1:13 AM UTC

vannevar wrote: post #80

Benjammer wrote: post #76

Earlier quoted context omitted.

A DoS attack is, by literal definition, an attempt to overwhelm a host until it is forced to _deny service_ to valid user requests. Are there intrusion techniques that both bring down the server and break into it at the same time? I'm not a security expert, but that doesn't seem like it makes a whole lot of sense to me.

Maybe in the milliseconds between packet swarms, or immediately before or after? Just seems like a lot of resources to pour into an attack that was defeated in a few minutes. To what end?

I’d be willing to bet it was a “test”. Like what we saw with the mirai botnet against Kreb’s blog.

Re: February 28th DDoS Incident Report

#86

Post by scaryclam » Fri, Mar 02, 2018, 2:25 AM UTC

bhauer wrote: post #60

Am I old-fashioned to raise an eyebrow when I discover that Memcached servers are running visible to the public Internet? This strikes me as approximately as bizarre as having a database server that accepts connections from the public Internet. In my day, such back-end services were either simply not connected to the Internet (connected via a private network to the application services), firewalled, or at the very le…

No, it's not out of favor. There are a lot of unqualified people out there pushing buttons on cloud providers dashboards and not caring about security (or not even understanding that it's an issue) though.

When it's easier to just open up a server to the wide world than it is to learn how to connect safely, you'll always get a lot of people doing it.

Re: February 28th DDoS Incident Report

#87

Post by hueving » Fri, Mar 02, 2018, 2:38 AM UTC

toomuchtodo wrote: post #24

toast0 wrote: post #19

Earlier quoted context omitted.

> Who is not filtering outbound UDP traffic from their memcached instances? This is of course the wrong way to do it -- you need to filter inbound UDP to your memcached instances so you don't waste your resources generating the responses, and also so you don't accidentally fragment the responses and only drop the first fragment outbound.

I disagree, due to seperation of responsibilities. Having run both an ISP and a hosting company, you have to filter traffic at your edge that can impact external resources (just as ISPs block outbound NetBios and SMTP traffic on port 25/tcp). Yes, the server or instance customer should be doing this. But they’re not, because poor security practices are an externality, not a cost they sustain. Security is more importa…

Your confused if you think it's the clouds that are misconfigured here. The issue is the ISPs allowing the spoofed traffic going towards the memcached servers.

Re: February 28th DDoS Incident Report

#88

Post by hueving » Fri, Mar 02, 2018, 2:41 AM UTC

dec0dedab0de wrote: post #57

lima wrote: post #48

Earlier quoted context omitted.

Yes, absolutely - there are plenty of scenarios like, ironically, DDoS mitigation where you use source IP spoofing/asymmetric routing. It's still possible to restrict it, but simple RPF checks don't always cut it.

How could source IP spoofing help with DDoS mitigation?

This particular case relied on botnets being able to spoof their their source IP to match github's so the memcached responses would go to github.

Google 'DRDoS attacks' to learn more. They are responsible for most of the largest volume attacks IIUC.

Re: February 28th DDoS Incident Report

#89

Post by dc_gregory » Fri, Mar 02, 2018, 2:56 AM UTC

shuntress wrote: post #40

flafla2 wrote: post #14

Earlier quoted context omitted.

Ah ok. This makes much more sense -- leaving a port open seems like an amateur mistake for a firm like Github. Thanks for the link.

If Github had no open ports they wouldn't have much of a website.

I think it was more likely meant re. the original (incorrect) interpretation that github had left public access to memcache instances.

Re: February 28th DDoS Incident Report

#90

Post by unethical_ban » Fri, Mar 02, 2018, 2:57 AM UTC

lima wrote: post #58

unethical_ban wrote: post #56

Earlier quoted context omitted.

Let's rephrase the question - Is there any reason consumer ISP's don't follow BCP38? There is almost no reason whatsoever for clients to spoof their public IP address. Obviously, there are reasons to SNAT at the carrier level for load balance or routing purposes.

No good reason except "it costs money".

No good reason except it's for the health of the Internet.

And it doesn't cost any significant amount of money except initial configuration and automation. The "CPU power" to add an ACL on interfaces is negligible.