Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

February 28th DDoS Incident Report

githubengineering.com

Post reply on HN
Print view View original

61–70 of 114 posts

Re: February 28th DDoS Incident Report

#61

Post by smudgymcscmudge »

bhauer wrote: post #60

Am I old-fashioned to raise an eyebrow when I discover that Memcached servers are running visible to the public Internet? This strikes me as approximately as bizarre as having a database server that accepts connections from the public Internet. In my day, such back-end services were either simply not connected to the Internet (connected via a private network to the application services), firewalled, or at the very le…

It is very much out of favor. But it just takes a few dozen misconfigured servers with big pipes to launch an attack like this.

Re: February 28th DDoS Incident Report

#62

Post by WJW »

Dirlewanger wrote: post #52
always_good wrote: post #18

DDoS is a reminder of how broken the internet is. How many times are we going to see the HN comment that says "lol why do so many people use Cloudflare? I don't need it for my blog!" Naive decentralization (naive trust) doesn't work.

Better than a private company playing gatekeeper for a massive amount of websites.

Please define better. If (say) an IPFS item is cached by 3-5 nodes each of which is on a private home internet connection it is trivial to DDOS each of those and thus effectively censor the content. On the other hand, it is decidedly nontrivial to DDOS Cloudflare.

Re: February 28th DDoS Incident Report

#63

Post by tetha »

bhauer wrote: post #60

Am I old-fashioned to raise an eyebrow when I discover that Memcached servers are running visible to the public Internet? This strikes me as approximately as bizarre as having a database server that accepts connections from the public Internet. In my day, such back-end services were either simply not connected to the Internet (connected via a private network to the application services), firewalled, or at the very le…

It's simpler to just click services on AWS and get a public IP to connect to. Drop-policy Firewalls like AWS security groups are hard to configure and debug. Managing network interfaces and binding to specific interfaces instead of others is hard and causes hanging connections.

Those are the excuses I dealt with when I took over the current IT department. By now, only haproxy accepts public connections. Everything else is firewalled to the office at most.

Re: February 28th DDoS Incident Report

#64

Post by lol768 »

feelin_googley wrote: post #47
megous wrote: post #36

Earlier quoted context omitted.

Lol, CloudFlare is what's breaking the web if you need to have a stupidly complicated JavaScript engine enabled and accessible to a webpage you don't trust (and can't trust) to be able to access the said webpage. Based on how it's done, you can't check first if the page hidden behind clouflare is something you'd want to enable javascript for, because clouflare will not let you see the HTML code of the page, without e…

"... you need to have a stupidly complicated JavaScript engine enabled and accessible to a webpage..." Does anyone have an example of this webpage? Unless I am engaging in e-commerce, I do not run a browser JavaScript engine. I rarely if ever encounter a webpage that truly "requires" one. GitHub certainly does not require JavaScript for me to use it via www.

> Does anyone have an example of this webpage?

It's a requirement when a CloudFlare'd site is in "I'm under attack" mode.

Re: February 28th DDoS Incident Report

#65

Post by arkadiyt »

r1ch wrote: post #13

This is a great example of why it's important to pick secure defaults when writing software, especially software that is often deployed on high bandwidth servers or cloud instances. If no listening interfaces are specified then the default should be to exit with an error, not listen on everything! I also wonder if you can store something in a memcached cache that looks like a valid request, then reflect that with the…

Shortly after Cloudflare's blog post, memcached pushed a commit that disabled UDP by default:

https://github.com/memcached/memcached/commit/dbb7a8af90054b...

Re: February 28th DDoS Incident Report

#66

Post by giodamelio »

corndoge wrote: post #15

Very cool that they are able to change BGP advertisements from ChatOps, achieve convergence and mitigate the attack in all of 4 minutes, that is some insane engineering.

I had a similar reaction. I had to double check the timestamps when I first read them. That this was all handled so fast is extremely impressive to me.

Re: February 28th DDoS Incident Report

#67

Post by vlan0 »

bhauer wrote: post #60

Am I old-fashioned to raise an eyebrow when I discover that Memcached servers are running visible to the public Internet? This strikes me as approximately as bizarre as having a database server that accepts connections from the public Internet. In my day, such back-end services were either simply not connected to the Internet (connected via a private network to the application services), firewalled, or at the very le…

> firewalled, or at the very least, configured to listen for and respond exclusively to connections from known front-end or application services.

Combine this with staying on top of vulnerabilities, this is really all you can hope for from a host standpoint. What is changing are the days of perimeter defense. The Zero Trust model is really the best path forward, and the only way to implement security in relation to the IoT.[1][2]

[1]https://www.youtube.com/watch?v=k80jOH2H10U [2]https://www.safaribooksonline.com/library/view/zero-trust-ne...

Re: February 28th DDoS Incident Report

#68

Post by always_good »

megous wrote: post #36
always_good wrote: post #18

DDoS is a reminder of how broken the internet is. How many times are we going to see the HN comment that says "lol why do so many people use Cloudflare? I don't need it for my blog!" Naive decentralization (naive trust) doesn't work.

Lol, CloudFlare is what's breaking the web if you need to have a stupidly complicated JavaScript engine enabled and accessible to a webpage you don't trust (and can't trust) to be able to access the said webpage. Based on how it's done, you can't check first if the page hidden behind clouflare is something you'd want to enable javascript for, because clouflare will not let you see the HTML code of the page, without e…

Well, that's too bad. Bad actors ruined it for you.

We make things more annoying for VPN traffic because it's 99% bad actors. Every time someone is up to no good on our services, they're behind Tor/VPN.

It's simple cost/benefit analysis. If you think a business should bend to every single whim someone might have, then you haven't built much of one.

Making someone run Javascript so they can click on a captcha? Worth the loss of a few pennies because someone's angry about it on HN.

You need to conveniently ignore why people use Cloudflare to say that Cloudflare is breaking the internet. Ideally, nobody would have to use it, but that isn't reality.

Re: February 28th DDoS Incident Report

#69

Post by scurvy »

corndoge wrote: post #15

Very cool that they are able to change BGP advertisements from ChatOps, achieve convergence and mitigate the attack in all of 4 minutes, that is some insane engineering.

Meh, or just block UDP to your networks that have no reason to run UDP. Every carrier will do upstream ACL's these days. 5 years ago that wasn't the case. These days, they all do. Some free, some charge.

re: chat ops vs a web page. It's just a single BGP advertisement -- big whoop. Chatops is just hipster famous right now.

Re: February 28th DDoS Incident Report

#70

Post by megous »

always_good wrote: post #68
megous wrote: post #36

Earlier quoted context omitted.

Lol, CloudFlare is what's breaking the web if you need to have a stupidly complicated JavaScript engine enabled and accessible to a webpage you don't trust (and can't trust) to be able to access the said webpage. Based on how it's done, you can't check first if the page hidden behind clouflare is something you'd want to enable javascript for, because clouflare will not let you see the HTML code of the page, without e…

Well, that's too bad. Bad actors ruined it for you. We make things more annoying for VPN traffic because it's 99% bad actors. Every time someone is up to no good on our services, they're behind Tor/VPN. It's simple cost/benefit analysis. If you think a business should bend to every single whim someone might have, then you haven't built much of one. Making someone run Javascript so they can click on a captcha? Worth t…

This happens over normal cable internet connection (as someone mentioned below, it is probaby when a website is in IUAM) and it is not related to captchas.

Cloudflare provides some page with JS code that computes something and then procedes to the correct page if it is computed correctly.

So I can either enable JS both for the cloudflare interstitial page and for the target website, or I'm not able to access the website.

I'm not angry, I just close the website/go back to search results. I will not allow the browser to run random JS code from the target website for no reason, just because the interstitial page requires it.

Still, if someone requires computing random challenges in javscript in order to gain access to a web page, they're breaking the web. Javascript is still an optional addon.

Post reply on HN