Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Route Leak Impacting Cloudflare

cloudflarestatus.com

Post reply on HN
Print view View original

141–150 of 164 posts

Re: Route Leak Impacting Cloudflare

#141

Post by knowsmorsecode »

delfinom wrote: post #119
q3k wrote: post #87

Earlier quoted context omitted.

It's a chain. You first contact the leaker and their upstream, and then if that doesn't work then their upstream, etc. At some point you reach a company that's large enough that they must cooperate because they want to remain in business of being an actual responsible ISP. And then there's Verizon, who can safely ignore any ISP etiquette because they have a de-facto monopoly.

In this case Verizon seems to have absolutely no functioning NOC at night (if even at day).

Having had to deal with verizon, I can vouch that this is true.

Re: Route Leak Impacting Cloudflare

#142

Post by nikisweeting »

Hikikomori wrote: post #130
nikisweeting wrote: post #118

Earlier quoted context omitted.

It has authentication for only one hop , if routes propagated all the way up the chain with signatures, it would be much easier to block/limit bad AS behavior.

Your peering relationship is only for one hop. What it lacks is prefix/path validation, not authentication.

But authentication of every advertised range all the way up the chain would allow upstream providers to easily differentiate valid large prefix announcements that were done intentionally (e.g. big ISP announcing some routes) from crazy nonsense done by an unknown party that isn't a big ISP. We definitely need prefix filtering, but there needs to be some easily verifiable source of identity tied to each announcement to be able to automate the process of accepting and rejecting large prefix announcements.

Re: Route Leak Impacting Cloudflare

#143

Post by nikisweeting »

cntlzw wrote: post #110
nikisweeting wrote: post #71

Earlier quoted context omitted.

You're not going to be able to get a solid list, this is a different category of problem than something like CloudBleed, and even then the list wasn't solid. This issue is affecting AWS, Cloudflare, Cloudflare DNS, Google DNS, and the tens of thousands of other services that depend on them, but it's region specific and will break different things for different users as the leak propagates.

One source: https://twitter.com/atoonk/status/1143143943531454464 90 AS 13335 Cloudflare, Inc. 18 AS 7018 AT&T Services, Inc. 8 AS 63949 Linode, LLC 8 AS 2828 MCI Communications Services, Inc. d/b/a Verizon Business 6 AS 26769 Bandcon 6 AS 16509 Amazon.com, Inc. 4 AS 6428 CDM 4 AS 2914 NTT America, Inc. 2 AS 9808 Guangdong Mobile Communication Co.Ltd. 2 AS 6939 Hurricane Electric LLC 2 AS 62904 Eonix Corporation 2 AS…

I think it's like 2.4k ASNs at this point, each with 10s-1000s of IPs, I guess you can make a list from that but it's going to be as unreliable as the Cloudbleed list was. Also not always easy to do reverse hostname lookups from the IPs to see the site names.

Re: Route Leak Impacting Cloudflare

#144

Post by chimeracoder »

mindslight wrote: post #134
ceejayoz wrote: post #132

Earlier quoted context omitted.

You haven't identified the "bait" bit of the bait and switch. At no point has Google promised to respond to pings on 8.8.8.8, nor are they obliged to ever do so. Rejecting ICMP isn't "a new standard".

The promise is implicit when competing for mindshare with 4.2.2.2. Typing an IP address into a router setup is quite infrequent, compared to "let's check connectivity by ping x.x.x.x". Setting expectations that 8.8.8.8 can fill this role is the bait. As I said, it's much easier to respond to a ping than even a cached DNS query. Or it would also be consistent to simply never respond to ping. Now obviously in the moder…

> The promise is implicit when competing for mindshare with 4.2.2.2. Typing an IP address into a router setup is quite infrequent, compared to "let's check connectivity by ping x.x.x.x". Setting expectations that 8.8.8.8 can fill this role is the bait.

4.2.2.2 is not even meant to be used as a public DNS server (and has sometimes hijacked DNS requests at times to remind people of that). So it's weird to use 4.2.2.2 to criticize Google for blocking ICMP on their actually-public DNS server.

Re: Route Leak Impacting Cloudflare

#145

Post by kbirkeland »

nikisweeting wrote: post #98

There's one thing I don't understand about this all, it looks like Allegheny Technologies Incorporated (AS396531, a suspected original leaker) was originally announcing 192.92.159.0/24. How the heck did their peers not manage to filter a sudden announcement for a range big enough that it managed to snag both 8.8.8.8 and 1.1.1.1. Do upstreams really allow a tiny /24 AS to randomly announce a /4 and get away with it? O…

Leaking a /4 into BGP would do basically nothing unless the originator was originally advertising a /4. IP forwarding is based on the longest-prefix match. Since allocations are sized from /8 to /24, anybody actually advertising their space would not get hijacked by a /4. The leaker would just get traffic destined toward non-advertised networks.

Re: Route Leak Impacting Cloudflare

#146

Post by jiveturkey »

Interesting, isn't it, that when it's a US based steel plant, it's a route leak. When it's China Telecom, it's a route hijack.

The description of it as a leak AFAICT seems to be due to CF getting first dibs on the announcement[†] and positioned it as such. However, I firmly believe that had the general tech press gotten ahead of it first, it still would be treated much more generously than we treat China leaks.

[†] grin

Re: Route Leak Impacting Cloudflare

#147

Post by nikisweeting »

kbirkeland wrote: post #145
nikisweeting wrote: post #98

There's one thing I don't understand about this all, it looks like Allegheny Technologies Incorporated (AS396531, a suspected original leaker) was originally announcing 192.92.159.0/24. How the heck did their peers not manage to filter a sudden announcement for a range big enough that it managed to snag both 8.8.8.8 and 1.1.1.1. Do upstreams really allow a tiny /24 AS to randomly announce a /4 and get away with it? O…

Leaking a /4 into BGP would do basically nothing unless the originator was originally advertising a /4. IP forwarding is based on the longest-prefix match. Since allocations are sized from /8 to /24, anybody actually advertising their space would not get hijacked by a /4. The leaker would just get traffic destined toward non-advertised networks.

Then my next question is: If they didn't leak a massive range, then why was it a big problem? I assume if they leaked a bad /24 it surely wouldn't be enough to take down Cloudflare and Google for everyone... no? Did they just leak tons of bad /24s or was it something else?

Re: Route Leak Impacting Cloudflare

#148

Post by nikisweeting »

x86_64Ubuntu wrote: post #140
throwawayflower wrote: post #137

Earlier quoted context omitted.

Oh, and the country's train and public transport infrastructure is experiencing some major problems too due to the phone service outage.

You have to wonder if these outages aren't the result of hostile states laying the groundwork and testing the viability of certain attacks.

Heh I think based on BGP's track-record, if a state-level actor wanted to mess up everyone's BGP routes they wouldn't have to try very hard...

Re: Route Leak Impacting Cloudflare

#150

Post by kingbirdy »

nikisweeting wrote: post #147
kbirkeland wrote: post #145

Earlier quoted context omitted.

Leaking a /4 into BGP would do basically nothing unless the originator was originally advertising a /4. IP forwarding is based on the longest-prefix match. Since allocations are sized from /8 to /24, anybody actually advertising their space would not get hijacked by a /4. The leaker would just get traffic destined toward non-advertised networks.

Then my next question is: If they didn't leak a massive range, then why was it a big problem? I assume if they leaked a bad /24 it surely wouldn't be enough to take down Cloudflare and Google for everyone... no? Did they just leak tons of bad /24s or was it something else?

My understanding is they had an optimizer that broke the /4 down in to /24s and those got announced
Post reply on HN