Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

February 28th DDoS Incident Report

githubengineering.com

Post reply on HN
Print view View original

11–20 of 114 posts

Re: February 28th DDoS Incident Report

#11

Post by tlunter »

flafla2 wrote: post #9
voltagex_ wrote: post #4

Earlier quoted context omitted.

memcached reflection: https://blogs.akamai.com/2018/03/memcached-fueled-13-tbps-at...

I see. For anyone else who doesn't have any background in this attack: memcached is an open source general purpose cache that uses sockets to cache data. From what I gather, the attack here was possible because Github engineers accidentally left the memcached port open. So the attackers were able to spam memcached with large requests, and memcached responds immediately with the full contents of the cached memory (ass…

I don't think it was GitHub's memcached instances. It was other public instances that with spoofed network requests ended up sending traffic back towards GitHub's network.

Re: February 28th DDoS Incident Report

#12

Post by kevinconaway »

flafla2 wrote: post #9
voltagex_ wrote: post #4

Earlier quoted context omitted.

memcached reflection: https://blogs.akamai.com/2018/03/memcached-fueled-13-tbps-at...

I see. For anyone else who doesn't have any background in this attack: memcached is an open source general purpose cache that uses sockets to cache data. From what I gather, the attack here was possible because Github engineers accidentally left the memcached port open. So the attackers were able to spam memcached with large requests, and memcached responds immediately with the full contents of the cached memory (ass…

> From what I gather, the attack here was possible because Github engineers accidentally left the memcached port open.

That is incorrect.

The attackers made requests that were forged to have the sender IP address of Github to multiple public memcached instances. Memcached then responds back to Github instead of the attacker.

This is documented in more detail in the Cloudflare vulnerability report[0]

https://blog.cloudflare.com/memcrashed-major-amplification-a...

Re: February 28th DDoS Incident Report

#13

Post by r1ch »

This is a great example of why it's important to pick secure defaults when writing software, especially software that is often deployed on high bandwidth servers or cloud instances. If no listening interfaces are specified then the default should be to exit with an error, not listen on everything!

I also wonder if you can store something in a memcached cache that looks like a valid request, then reflect that with the source IP of another memcached server and let them burn each other out...

Re: February 28th DDoS Incident Report

#14

Post by flafla2 »

kevinconaway wrote: post #12
flafla2 wrote: post #9

Earlier quoted context omitted.

I see. For anyone else who doesn't have any background in this attack: memcached is an open source general purpose cache that uses sockets to cache data. From what I gather, the attack here was possible because Github engineers accidentally left the memcached port open. So the attackers were able to spam memcached with large requests, and memcached responds immediately with the full contents of the cached memory (ass…

> From what I gather, the attack here was possible because Github engineers accidentally left the memcached port open. That is incorrect. The attackers made requests that were forged to have the sender IP address of Github to multiple public memcached instances. Memcached then responds back to Github instead of the attacker. This is documented in more detail in the Cloudflare vulnerability report[0] https://blog.clou…

Ah ok. This makes much more sense -- leaving a port open seems like an amateur mistake for a firm like Github. Thanks for the link.

Re: February 28th DDoS Incident Report

#16

Post by theojulienne »

toomuchtodo wrote: post #8
voltagex_ wrote: post #4

Earlier quoted context omitted.

memcached reflection: https://blogs.akamai.com/2018/03/memcached-fueled-13-tbps-at...

Who is exposing their memcached instances on the public internet? Who is not filtering outbound UDP traffic from their memcached instances? Rhetorical of course. Akamai should have logs of their offenders. Off to scan for offenders and notify their providers!

During some analysis we did notice that at least some cloud providers default to having instances with public IPs (with no network-level ACLs) by default, and some Linux distributions default to having memcached listening for UDP traffic and binding to `0.0.0.0` by default as soon as it's installed. The unfortunate combination of these result in the machine being vulnerable to being used as an amplification vector in these attacks.

Re: February 28th DDoS Incident Report

#17

Post by randall »

flafla2 wrote: post #3
hyperpower wrote: post #2

Wow, 1.35Tbps? That's a lot for a DoS attack, right?

According to [0] that is around 1/400th of total internet traffic per second. This begs the question: who has that kind of botnet at their disposal and why are they targeting Github? Edit: The attacker didn't need nearly that kind of bandwidth to execute this attack. See [1] Edit: 1/50th -> 1/400th (bits vs bytes) [0] http://www.internetlivestats.com/one-second/#traffic-band [1] https://news.ycombinator.com/item?id=1…

Probably to get press. If you have a giant botnet and get press for it, it's probably easier to get clients.

Re: February 28th DDoS Incident Report

#19

Post by toast0 »

toomuchtodo wrote: post #8
voltagex_ wrote: post #4

Earlier quoted context omitted.

memcached reflection: https://blogs.akamai.com/2018/03/memcached-fueled-13-tbps-at...

Who is exposing their memcached instances on the public internet? Who is not filtering outbound UDP traffic from their memcached instances? Rhetorical of course. Akamai should have logs of their offenders. Off to scan for offenders and notify their providers!

> Who is not filtering outbound UDP traffic from their memcached instances?

This is of course the wrong way to do it -- you need to filter inbound UDP to your memcached instances so you don't waste your resources generating the responses, and also so you don't accidentally fragment the responses and only drop the first fragment outbound.

Re: February 28th DDoS Incident Report

#20

Post by jwilk »

gjtorikian wrote: post #5
hyperpower wrote: post #2

Wow, 1.35Tbps? That's a lot for a DoS attack, right?

It's the largest recorded attack: https://www.wired.com/story/github-ddos-memcached

The GitHub post says the amplification factor is 51000, while the Wired article says it's 50. So which is it?
Post reply on HN