Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

February 28th DDoS Incident Report

githubengineering.com

Post reply on HN
Print view View original

91–100 of 114 posts

Re: February 28th DDoS Incident Report

#91

Post by toomuchtodo »

hueving wrote: post #87
toomuchtodo wrote: post #24

Earlier quoted context omitted.

I disagree, due to seperation of responsibilities. Having run both an ISP and a hosting company, you have to filter traffic at your edge that can impact external resources (just as ISPs block outbound NetBios and SMTP traffic on port 25/tcp). Yes, the server or instance customer should be doing this. But they’re not, because poor security practices are an externality, not a cost they sustain. Security is more importa…

Your confused if you think it's the clouds that are misconfigured here. The issue is the ISPs allowing the spoofed traffic going towards the memcached servers.

If you’re a service provider allowing your equipment to participate in an amplification attack, you’re the fool trashing the commons.

It’s an ISPs job to filter outbound udp on arbitrary ports? Shall we only let 443 tcp outbound from eyeball networks?

Re: February 28th DDoS Incident Report

#92

Post by [deleted] »

arkadiyt wrote: post #65
r1ch wrote: post #13

This is a great example of why it's important to pick secure defaults when writing software, especially software that is often deployed on high bandwidth servers or cloud instances. If no listening interfaces are specified then the default should be to exit with an error, not listen on everything! I also wonder if you can store something in a memcached cache that looks like a valid request, then reflect that with the…

Shortly after Cloudflare's blog post, memcached pushed a commit that disabled UDP by default: https://github.com/memcached/memcached/commit/dbb7a8af90054b...

[deleted]

Re: February 28th DDoS Incident Report

#93

Post by fulafel »

dec0dedab0de wrote: post #26

Is there any legitimate reason to spoof a source IP? I don't think there is, why don't ISPs block any traffic with a source IP that isn't in their network. And then the rest of us block any ISPs that don't do that.

Spoofing is in the eye of the beholder. A router first and foremost routes packets toward the right destination, there is no such thing as a "spoofed source IP" without context. Policy about what traffic is allowed to come from what pipe is always error prone and increased complexity.

Re: February 28th DDoS Incident Report

#94

Post by chrisper »

r1ch wrote: post #13

This is a great example of why it's important to pick secure defaults when writing software, especially software that is often deployed on high bandwidth servers or cloud instances. If no listening interfaces are specified then the default should be to exit with an error, not listen on everything! I also wonder if you can store something in a memcached cache that looks like a valid request, then reflect that with the…

This is why I dislike that Ubuntu starts services by default after installing them.

Re: February 28th DDoS Incident Report

#95

Post by scurvy »

scurvy wrote: post #74
mirashii wrote: post #71

Earlier quoted context omitted.

A snarky reply like this comes up every time there's discussion of a DDOS, but it ignores the fact that there is some point that has to filter that UDP traffic, and if that point is saturated, the DDOS still worked. Mitigating attacks of this size isn't a firewall rule or a support ticket with your ISP.

Snark or not, the traffic is filtered upstream before your handoff. If you pick your carriers well, there's not a problem. Many carriers have turned upstream filters into a product. NTT's DPS Lite springs to mind. This just comes down to experience and knowing how to build a network. I'd think that Github would have people knowing how to architect this. They've been through a few DDoS before. Edit: It looks like Gith…

Whiners can downvote all you want, but you can't say I'm wrong. So either admit you're wrong or go pound sand.

Such sensitive.

Re: February 28th DDoS Incident Report

#96

Post by scurvy »

mirashii wrote: post #71
scurvy wrote: post #69

Earlier quoted context omitted.

Meh, or just block UDP to your networks that have no reason to run UDP. Every carrier will do upstream ACL's these days. 5 years ago that wasn't the case. These days, they all do. Some free, some charge. re: chat ops vs a web page. It's just a single BGP advertisement -- big whoop. Chatops is just hipster famous right now.

A snarky reply like this comes up every time there's discussion of a DDOS, but it ignores the fact that there is some point that has to filter that UDP traffic, and if that point is saturated, the DDOS still worked. Mitigating attacks of this size isn't a firewall rule or a support ticket with your ISP.

Also lol at "support ticket with your ISP." We're talking about major tier 1 carriers here.

Re: February 28th DDoS Incident Report

#98

Post by smsm42 »

tetha wrote: post #63
bhauer wrote: post #60

Am I old-fashioned to raise an eyebrow when I discover that Memcached servers are running visible to the public Internet? This strikes me as approximately as bizarre as having a database server that accepts connections from the public Internet. In my day, such back-end services were either simply not connected to the Internet (connected via a private network to the application services), firewalled, or at the very le…

It's simpler to just click services on AWS and get a public IP to connect to. Drop-policy Firewalls like AWS security groups are hard to configure and debug. Managing network interfaces and binding to specific interfaces instead of others is hard and causes hanging connections. Those are the excuses I dealt with when I took over the current IT department. By now, only haproxy accepts public connections. Everything el…

I wonder if it's time for providers like Amazon to provide configs by default that block all ports besides TCP 22, 80 and 443. You want to do other stuff? Configure a firewall. Don't know how? Hire somebody who does. This scenario with cheap insecure things being put out on the internet repeats again and again. IoT, PaaS, etc.

Re: February 28th DDoS Incident Report

#99

Post by techman9 »

bhauer wrote: post #60

Am I old-fashioned to raise an eyebrow when I discover that Memcached servers are running visible to the public Internet? This strikes me as approximately as bizarre as having a database server that accepts connections from the public Internet. In my day, such back-end services were either simply not connected to the Internet (connected via a private network to the application services), firewalled, or at the very le…

If you don't know what you're doing, it's easy to misconfigure iptables or whatever and inadvertently allow traffic from the internet to Memcached.

A quick Shodan search[1] shows like 90k boxes publicly accessible Memcached. Misconfiguration of firewalls is a serious problem.

[1] https://www.shodan.io/search?query=11211

Post reply on HN