Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

February 28th DDoS Incident Report

githubengineering.com

Post reply on HN
Print view View original

101–110 of 114 posts

Re: February 28th DDoS Incident Report

#101

Post by jcims »

scaryclam wrote: post #86
bhauer wrote: post #60

Am I old-fashioned to raise an eyebrow when I discover that Memcached servers are running visible to the public Internet? This strikes me as approximately as bizarre as having a database server that accepts connections from the public Internet. In my day, such back-end services were either simply not connected to the Internet (connected via a private network to the application services), firewalled, or at the very le…

No, it's not out of favor. There are a lot of unqualified people out there pushing buttons on cloud providers dashboards and not caring about security (or not even understanding that it's an issue) though. When it's easier to just open up a server to the wide world than it is to learn how to connect safely, you'll always get a lot of people doing it.

When you have infrastructure as code, you get infrastructure by coders.

(I’ll be here all week)

Re: February 28th DDoS Incident Report

#102

Post by jack9 »

scurvy wrote: post #74
mirashii wrote: post #71

Earlier quoted context omitted.

A snarky reply like this comes up every time there's discussion of a DDOS, but it ignores the fact that there is some point that has to filter that UDP traffic, and if that point is saturated, the DDOS still worked. Mitigating attacks of this size isn't a firewall rule or a support ticket with your ISP.

Snark or not, the traffic is filtered upstream before your handoff. If you pick your carriers well, there's not a problem. Many carriers have turned upstream filters into a product. NTT's DPS Lite springs to mind. This just comes down to experience and knowing how to build a network. I'd think that Github would have people knowing how to architect this. They've been through a few DDoS before. Edit: It looks like Gith…

It's interesting that on HN, some experience doesn't get rewarded, just because there's some rather opinionated language. The majority of voters are barely exposed to these kinds of overwhelming attacks, leaving practical analysis to be buried unless it's got some big company name drop to legitimize it.

Re: February 28th DDoS Incident Report

#103

Post by lozenge »

bhauer wrote: post #60

Am I old-fashioned to raise an eyebrow when I discover that Memcached servers are running visible to the public Internet? This strikes me as approximately as bizarre as having a database server that accepts connections from the public Internet. In my day, such back-end services were either simply not connected to the Internet (connected via a private network to the application services), firewalled, or at the very le…

This is the entire Internet we're talking about, of course there will be a few misconfigured servers. It's more surprising that there are only a thousand.

Re: February 28th DDoS Incident Report

#104

Post by deoxxa »

smsm42 wrote: post #98
tetha wrote: post #63

Earlier quoted context omitted.

It's simpler to just click services on AWS and get a public IP to connect to. Drop-policy Firewalls like AWS security groups are hard to configure and debug. Managing network interfaces and binding to specific interfaces instead of others is hard and causes hanging connections. Those are the excuses I dealt with when I took over the current IT department. By now, only haproxy accepts public connections. Everything el…

I wonder if it's time for providers like Amazon to provide configs by default that block all ports besides TCP 22, 80 and 443. You want to do other stuff? Configure a firewall. Don't know how? Hire somebody who does. This scenario with cheap insecure things being put out on the internet repeats again and again. IoT, PaaS, etc.

It's interesting you say this, as that's pretty much exactly how Lightsail (Amazon's easy-mode VM thing) works by default. Public IP, ports 22 and 80 open. I'm guessing for a good chunk of users, that default config is all they need.

Re: February 28th DDoS Incident Report

#105

Post by hueving »

toomuchtodo wrote: post #91
hueving wrote: post #87

Earlier quoted context omitted.

Your confused if you think it's the clouds that are misconfigured here. The issue is the ISPs allowing the spoofed traffic going towards the memcached servers.

If you’re a service provider allowing your equipment to participate in an amplification attack, you’re the fool trashing the commons. It’s an ISPs job to filter outbound udp on arbitrary ports? Shall we only let 443 tcp outbound from eyeball networks?

Any UDP service can be used in an amplification attack. It's not the responsibility of AWS or other hosting companies to shut down all UDP traffic.

The problem is the ISPs allowing spoofed IPs.

Re: February 28th DDoS Incident Report

#106

Post by [deleted] »

scurvy wrote: post #74
mirashii wrote: post #71

Earlier quoted context omitted.

A snarky reply like this comes up every time there's discussion of a DDOS, but it ignores the fact that there is some point that has to filter that UDP traffic, and if that point is saturated, the DDOS still worked. Mitigating attacks of this size isn't a firewall rule or a support ticket with your ISP.

Snark or not, the traffic is filtered upstream before your handoff. If you pick your carriers well, there's not a problem. Many carriers have turned upstream filters into a product. NTT's DPS Lite springs to mind. This just comes down to experience and knowing how to build a network. I'd think that Github would have people knowing how to architect this. They've been through a few DDoS before. Edit: It looks like Gith…

[deleted]

Re: February 28th DDoS Incident Report

#107

Post by consumer451 »

bhauer wrote: post #60

Am I old-fashioned to raise an eyebrow when I discover that Memcached servers are running visible to the public Internet? This strikes me as approximately as bizarre as having a database server that accepts connections from the public Internet. In my day, such back-end services were either simply not connected to the Internet (connected via a private network to the application services), firewalled, or at the very le…

A bit off-topic, but I had the same thought when I read about Jenkins servers on the public Internet in this piece.[0]

But when I read that he had found a public facing Jenkins server owned by Google, I figured I must be missing something.

I run a 2 man shop, but I still keep things like Jenkins behind OpenVPN. Why would anyone leave Jenkins open? There must be a reason, right?

https://emtunc.org/blog/01/2018/research-misconfigured-jenki... [0]

Re: February 28th DDoS Incident Report

#108

Post by mnw21cam »

hueving wrote: post #105
toomuchtodo wrote: post #91

Earlier quoted context omitted.

If you’re a service provider allowing your equipment to participate in an amplification attack, you’re the fool trashing the commons. It’s an ISPs job to filter outbound udp on arbitrary ports? Shall we only let 443 tcp outbound from eyeball networks?

Any UDP service can be used in an amplification attack. It's not the responsibility of AWS or other hosting companies to shut down all UDP traffic. The problem is the ISPs allowing spoofed IPs.

So the problem here is that a number of UDP packets were sent from somewhere (with a small bandwidth) that had a spoofed source address. They were then sent to the reflection servers which produced more/bigger UDP packets that did not have a spoofed source address.

So the attacker only needs to find somewhere on the internet that is capable of generating spoofed packets. They needed a lot of places that had a reflection server, but the requirements for the spoofing was much smaller.

In other words, you would have to prevent 99.9% of the internet from being able to spoof source addresses before you fixed this problem.

Re: February 28th DDoS Incident Report

#109

Post by alex_duf »

jack9 wrote: post #102
scurvy wrote: post #74

Earlier quoted context omitted.

Snark or not, the traffic is filtered upstream before your handoff. If you pick your carriers well, there's not a problem. Many carriers have turned upstream filters into a product. NTT's DPS Lite springs to mind. This just comes down to experience and knowing how to build a network. I'd think that Github would have people knowing how to architect this. They've been through a few DDoS before. Edit: It looks like Gith…

It's interesting that on HN, some experience doesn't get rewarded, just because there's some rather opinionated language. The majority of voters are barely exposed to these kinds of overwhelming attacks, leaving practical analysis to be buried unless it's got some big company name drop to legitimize it.

Maybe the experience isn't rewarded because of said opinionated language?

I mean someone can be right, but that doesn't give that person the right to be dismissive. Treating ignorance with disdain isn't going to make anyone smarter.

Re: February 28th DDoS Incident Report

#110

Post by fps »

consumer451 wrote: post #107
bhauer wrote: post #60

Am I old-fashioned to raise an eyebrow when I discover that Memcached servers are running visible to the public Internet? This strikes me as approximately as bizarre as having a database server that accepts connections from the public Internet. In my day, such back-end services were either simply not connected to the Internet (connected via a private network to the application services), firewalled, or at the very le…

A bit off-topic, but I had the same thought when I read about Jenkins servers on the public Internet in this piece.[0] But when I read that he had found a public facing Jenkins server owned by Google, I figured I must be missing something. I run a 2 man shop, but I still keep things like Jenkins behind OpenVPN. Why would anyone leave Jenkins open? There must be a reason, right? https://emtunc.org/blog/01/2018/researc…

Google's "Beyond Corp" initiative [1] discourages trusted networks and VPNs in favor of secure services on public networks. By trusting the network to provide a level of security, you are more likely to be vulnerable to escalation attacks by bad actors that are able to access your private networks. You're also more likely to encourage legitimate users to set up workarounds that result in secure network breaches. Typically they use an Identity aware proxy in front of the service, but services can have a public view as well.

To answer your second question, I work for an open source non-profit software company, and we run some of our jenkins servers, which do continuous integration builds, publicly available so that community contributors and users can see build failures. Google has a number of open source projects that probably have similar goals.

1. https://cloud.google.com/beyondcorp/

Post reply on HN