Live data from Hacker News

February 28th DDoS Incident Report

githubengineering.com

21–30 of 114 posts

Re: February 28th DDoS Incident Report

#21
post #3

Wow, 1.35Tbps? That's a lot for a DoS attack, right?

According to [0] that is around 1/400th of total internet traffic per second. This begs the question: who has that kind of botnet at their disposal and why are they targeting Github? Edit: The attacker didn't need nearly that kind of bandwidth to execute this attack. See [1] Edit: 1/50th -> 1/400th (bits vs bytes) [0] http://www.internetlivestats.com/one-second/#traffic-band [1] https://news.ycombinator.com/item?id=1…

[deleted]

Re: February 28th DDoS Incident Report

#22
post #20

Earlier quoted context omitted.

It's the largest recorded attack: https://www.wired.com/story/github-ddos-memcached

The GitHub post says the amplification factor is 51000, while the Wired article says it's 50. So which is it?

We've seen 51,000x. I've personally looked at the pcaps of that.

Re: February 28th DDoS Incident Report

#23

Wow, 1.35Tbps? That's a lot for a DoS attack, right?

Yes and there are a lot of attacks of very, very large sizes going on. Over the last few days we've mitigated some huge attacks. Luckily, everyone is working together to rate limit and clean up this problem.

Re: February 28th DDoS Incident Report

#24
post #19

Earlier quoted context omitted.

Who is exposing their memcached instances on the public internet? Who is not filtering outbound UDP traffic from their memcached instances? Rhetorical of course. Akamai should have logs of their offenders. Off to scan for offenders and notify their providers!

> Who is not filtering outbound UDP traffic from their memcached instances? This is of course the wrong way to do it -- you need to filter inbound UDP to your memcached instances so you don't waste your resources generating the responses, and also so you don't accidentally fragment the responses and only drop the first fragment outbound.

I disagree, due to seperation of responsibilities. Having run both an ISP and a hosting company, you have to filter traffic at your edge that can impact external resources (just as ISPs block outbound NetBios and SMTP traffic on port 25/tcp).

Yes, the server or instance customer should be doing this. But they’re not, because poor security practices are an externality, not a cost they sustain.

Security is more important than developer velocity, but users pay the bills.

Re: February 28th DDoS Incident Report

#25

Earlier quoted context omitted.

Who is exposing their memcached instances on the public internet? Who is not filtering outbound UDP traffic from their memcached instances? Rhetorical of course. Akamai should have logs of their offenders. Off to scan for offenders and notify their providers!

During some analysis we did notice that at least some cloud providers default to having instances with public IPs (with no network-level ACLs) by default, and some Linux distributions default to having memcached listening for UDP traffic and binding to `0.0.0.0` by default as soon as it's installed. The unfortunate combination of these result in the machine being vulnerable to being used as an amplification vector in…

Did you receive any cooperation from those cloud providers in using ACLs at their network edge to drop that traffic (that they should’ve been blocking in the first place)?

Re: February 28th DDoS Incident Report

#27

Earlier quoted context omitted.

Who is exposing their memcached instances on the public internet? Who is not filtering outbound UDP traffic from their memcached instances? Rhetorical of course. Akamai should have logs of their offenders. Off to scan for offenders and notify their providers!

During some analysis we did notice that at least some cloud providers default to having instances with public IPs (with no network-level ACLs) by default, and some Linux distributions default to having memcached listening for UDP traffic and binding to `0.0.0.0` by default as soon as it's installed. The unfortunate combination of these result in the machine being vulnerable to being used as an amplification vector in…

This is one area where I really disagree with Debian and derivatives default behavior of starting a service immediately after installation and not having a firewall enabled by default.

If I install a service on CentOS/RHEL/Fedora it is disabled by default, if I start the service firewalld will block traffic until I have explicitly enabled a rule to allow it (or explicitly stopped and disabled the firewalld service).

Does this prevent people from making poor decisions, like just blindly starting the service without reading the configuration file, or disabling firewalld/enabling a rule without checking the configuration first? No, it doesn't - but that small hurdle at least prevents people from inadvertently turning on a service and opening it up to the world just by installing a package.

Re: February 28th DDoS Incident Report

#28
post #3

Wow, 1.35Tbps? That's a lot for a DoS attack, right?

According to [0] that is around 1/400th of total internet traffic per second. This begs the question: who has that kind of botnet at their disposal and why are they targeting Github? Edit: The attacker didn't need nearly that kind of bandwidth to execute this attack. See [1] Edit: 1/50th -> 1/400th (bits vs bytes) [0] http://www.internetlivestats.com/one-second/#traffic-band [1] https://news.ycombinator.com/item?id=1…

https://begthequestion.info

Re: February 28th DDoS Incident Report

#29

Is there any legitimate reason to spoof a source IP? I don't think there is, why don't ISPs block any traffic with a source IP that isn't in their network. And then the rest of us block any ISPs that don't do that.

Tragedy of the commons situation. It is advantageous for an individual ISP to wait out for the other ISPs to block source IP spoofing.

Re: February 28th DDoS Incident Report

#30
post #27

Earlier quoted context omitted.

During some analysis we did notice that at least some cloud providers default to having instances with public IPs (with no network-level ACLs) by default, and some Linux distributions default to having memcached listening for UDP traffic and binding to `0.0.0.0` by default as soon as it's installed. The unfortunate combination of these result in the machine being vulnerable to being used as an amplification vector in…

This is one area where I really disagree with Debian and derivatives default behavior of starting a service immediately after installation and not having a firewall enabled by default. If I install a service on CentOS/RHEL/Fedora it is disabled by default, if I start the service firewalld will block traffic until I have explicitly enabled a rule to allow it (or explicitly stopped and disabled the firewalld service).…

As a compromise, Debian also comes with a secure-by-default configuration. In the case of memcached, the service is configured to listen on 127.0.0.1.
Post reply on HN