Live data from Hacker News

February 28th DDoS Incident Report

githubengineering.com

31–40 of 114 posts

Re: February 28th DDoS Incident Report

#31

DDoS is a reminder of how broken the internet is. How many times are we going to see the HN comment that says "lol why do so many people use Cloudflare? I don't need it for my blog!" Naive decentralization (naive trust) doesn't work.

Now that we are moving away from net neutrality, can we not get ISPs to do DDOS protection so that we don't need specialised services like Cloudflare to be layered on top of simple sites?

Re: February 28th DDoS Incident Report

#32

Is there any legitimate reason to spoof a source IP? I don't think there is, why don't ISPs block any traffic with a source IP that isn't in their network. And then the rest of us block any ISPs that don't do that.

There even is a dedicated homepage for that problem of providers not filtering their egress traffic: http://www.bcp38.info/index.php/Main_Page

Re: February 28th DDoS Incident Report

#33
post #30
post #27

Earlier quoted context omitted.

This is one area where I really disagree with Debian and derivatives default behavior of starting a service immediately after installation and not having a firewall enabled by default. If I install a service on CentOS/RHEL/Fedora it is disabled by default, if I start the service firewalld will block traffic until I have explicitly enabled a rule to allow it (or explicitly stopped and disabled the firewalld service).…

As a compromise, Debian also comes with a secure-by-default configuration. In the case of memcached, the service is configured to listen on 127.0.0.1.

Thankfully that is one thing they do pretty well in most cases, though there are some (apache2, for example) that do listen on all interfaces by default. While even services like Apache may have secure configurations by default, it can often be installed by other programs that link or copy their own configuration files into the apache config directory - and then all you need is a PHP vulnerability or whatever.

Re: February 28th DDoS Incident Report

#34

Is there any legitimate reason to spoof a source IP? I don't think there is, why don't ISPs block any traffic with a source IP that isn't in their network. And then the rest of us block any ISPs that don't do that.

A couple of reasons:

1. It may be difficult/expensive to arrange for the correct set of source subnets to be available at the points where filtering needs to be done. Motivation to perform egress filtering fails to overcome this cost threshold.

2. Fear that some customers are actually (probably without realizing) relying on alien source address traffic being routed. Therefore filtering that traffic would result in unhappy customers and support workload.

In our network over the years I've come across several instances where it turned out we were (erroneously) relying on one of our upstream providers routing traffic with source IP from another provider's network. Since policy-based source IP selection on outbound traffic is quite tricky to setup and get right, I can imagine that ISPs would take the easy way out and just pass the traffic.

Re: February 28th DDoS Incident Report

#35

DDoS is a reminder of how broken the internet is. How many times are we going to see the HN comment that says "lol why do so many people use Cloudflare? I don't need it for my blog!" Naive decentralization (naive trust) doesn't work.

Now that we are moving away from net neutrality, can we not get ISPs to do DDOS protection so that we don't need specialised services like Cloudflare to be layered on top of simple sites?

ISPs absolutely could, but having worked near this space previously, it really isn't as easy as it sounds, both the detection and the mitigation, and ISPs are not particularly equipped to handle it themselves right now. There's a lot of money to be made there, though.

Re: February 28th DDoS Incident Report

#36

DDoS is a reminder of how broken the internet is. How many times are we going to see the HN comment that says "lol why do so many people use Cloudflare? I don't need it for my blog!" Naive decentralization (naive trust) doesn't work.

Lol, CloudFlare is what's breaking the web if you need to have a stupidly complicated JavaScript engine enabled and accessible to a webpage you don't trust (and can't trust) to be able to access the said webpage.

Based on how it's done, you can't check first if the page hidden behind clouflare is something you'd want to enable javascript for, because clouflare will not let you see the HTML code of the page, without enabling javascript for it first.

That is broken.

Re: February 28th DDoS Incident Report

#38

Is there any legitimate reason to spoof a source IP? I don't think there is, why don't ISPs block any traffic with a source IP that isn't in their network. And then the rest of us block any ISPs that don't do that.

A couple of reasons: 1. It may be difficult/expensive to arrange for the correct set of source subnets to be available at the points where filtering needs to be done. Motivation to perform egress filtering fails to overcome this cost threshold. 2. Fear that some customers are actually (probably without realizing) relying on alien source address traffic being routed. Therefore filtering that traffic would result in un…

That sounds like a negative externality that ISPs get to be lazy about and save money on by shoving the burden onto the cloudflares of the world. It’s really hard to dispose of hazardous waste when manufacturing things, but we force manufacturers to pay for the negative externalities. We should probably start thinking about the internet in the same way we think about the environment.

Re: February 28th DDoS Incident Report

#39
post #29

Is there any legitimate reason to spoof a source IP? I don't think there is, why don't ISPs block any traffic with a source IP that isn't in their network. And then the rest of us block any ISPs that don't do that.

Tragedy of the commons situation. It is advantageous for an individual ISP to wait out for the other ISPs to block source IP spoofing.

Note everyone seems to agree:

https://www.internetsociety.org/blog/2014/07/anti-spoofing-b...

If I understand the article's point, essentially, carriers pay for the egress traffic that causes DDoSes, that cost and the cost of the generated ill-will outweighs that of filtering, whose price has fallen and continues to fall.

Personally, I think that if the article author is correct, then I wonder if this is one of those high-level long-term decisions that companies appear absolutely incapable of making. (In my experience, short-term gains are way overvalued at the cost of long-term loss, generally, especially when it is hard to directly determine the costs/benefits involved.)

Re: February 28th DDoS Incident Report

#40
post #14

Earlier quoted context omitted.

> From what I gather, the attack here was possible because Github engineers accidentally left the memcached port open. That is incorrect. The attackers made requests that were forged to have the sender IP address of Github to multiple public memcached instances. Memcached then responds back to Github instead of the attacker. This is documented in more detail in the Cloudflare vulnerability report[0] https://blog.clou…

Ah ok. This makes much more sense -- leaving a port open seems like an amateur mistake for a firm like Github. Thanks for the link.

If Github had no open ports they wouldn't have much of a website.
Post reply on HN