Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Route Leak Impacting Cloudflare

cloudflarestatus.com

Post reply on HN
Print view View original

101–110 of 164 posts

Re: Route Leak Impacting Cloudflare

#101

Post by portman »

jgrahamc wrote: post #30
holycowzer wrote: post #24

Earlier quoted context omitted.

Thanks for the updates. I wish I could get this information somewhere other than hacker news though. :(

The team is updating the status page but not with granular detail because they'd have to spend time discussing what to say. I'm giving you the blow by blow.

I have a year-old startup and this is the first major Internet outage we've had to deal with... was really awesome to have your play-by-play and definitely changed our incident response (for the better!). Thank you so much.

Re: Route Leak Impacting Cloudflare

#102

Post by nikisweeting »

nikisweeting wrote: post #67
jgrahamc wrote: post #56

Earlier quoted context omitted.

We're definitely still working on it. Sorry you're affected by this. We're talking with the network providers involved. If anyone from the Verizon NOC is online... call me!

Confirming widespread FiOS issues in NYC as well, not limited to CloudFlare IPs.

If I had the guess, the leak was probably for a huge range, maybe a /4 or something. Verizon is also notoriously bad about dealing with BGP stuff, so I wouldn't be surprised if they have particularly bad filtering.

Re: Route Leak Impacting Cloudflare

#103

Post by joepie91_ »

purerandomness wrote: post #46
nikisweeting wrote: post #39

Earlier quoted context omitted.

At 3-4 major leaks per year it seems like we should probably fix BGP one of these days...

The way I understand it it's not BGP, it's mostly human error, or malicious intent. The protocol is fine.

A protocol that allows "human error" or "malicious intent" to take down entire swathes of the internet due to being entirely unauthenticated, is not "fine".

What you are describing is a protocol problem.

Re: Route Leak Impacting Cloudflare

#104

Post by mbell »

emilstahl wrote: post #59

“AS396531 "Allegheny Technologies Incorporated" is leaking a better-reachable route for AS13335 "Cloudflare, Inc." towards AS701 "Verizon Business/UUnet" explaining the current LSE going on.” https://twitter.com/OhNoItsFusl/status/1143117619106652160

> AS396531 - Allegheny Technologies Incorporated

That appears to be a steel/alloys company. Why are they operating BGP equipment?

Re: Route Leak Impacting Cloudflare

#105

Post by joepie91_ »

swixm wrote: post #73

What a great idea it is to have half the internet behind Crimeflare! It shows!

While I agree with the general sentiment - and I've certainly publicly and loudly expressed it in the past - this particular incident can't actually be blamed on that.

It's a route leak, which can affect any arbitrary amount of ISPs, because the BGP protocol is totally unauthenticated.

Re: Route Leak Impacting Cloudflare

#106

Post by RKearney »

nikisweeting wrote: post #45

What's weird is that 8.8.8.8 is also intermittently down for me. Are other people having issues with Google DNS too? https://i.imgur.com/3ySmVLW.png

Google rate limits ICMP to 8.8.8.8. It’s not meant to be used as your personal “is the internet up” test.

Re: Route Leak Impacting Cloudflare

#107

Post by Hikikomori »

nikisweeting wrote: post #54
purerandomness wrote: post #46

Earlier quoted context omitted.

The way I understand it it's not BGP, it's mostly human error, or malicious intent. The protocol is fine.

An unauthenticated protocol that allows unsigned routes to be blindly accepted is not a good protocol, that's why Cloudflare has been pushing RPKI for a while https://blog.cloudflare.com/rpki/ https://blog.cloudflare.com/rpki-details/

It has authentication and requires explicit configuration to form a neighbor relationship.

BGP was designed for operators to implement a routing policy. In most implementations it allows everything by default with no modifications to route metadata, so if you do not set up your policy correctly you'll have issues like this.

Re: Route Leak Impacting Cloudflare

#109

Post by nikisweeting »

RKearney wrote: post #106
nikisweeting wrote: post #45

What's weird is that 8.8.8.8 is also intermittently down for me. Are other people having issues with Google DNS too? https://i.imgur.com/3ySmVLW.png

Google rate limits ICMP to 8.8.8.8. It’s not meant to be used as your personal “is the internet up” test.

I use my own server or 1.1.1.1 for uptime checks, but 8.8.8.8 was my DNS fallback when 1.1.1.1 went down, which then meant I had no DNS working at all, which is why I noticed and tried pinging them.

Re: Route Leak Impacting Cloudflare

#110

Post by cntlzw »

nikisweeting wrote: post #71
filistar wrote: post #35

Does anyone know which global sites were unavailable because of Cloudflare crash?

You're not going to be able to get a solid list, this is a different category of problem than something like CloudBleed, and even then the list wasn't solid. This issue is affecting AWS, Cloudflare, Cloudflare DNS, Google DNS, and the tens of thousands of other services that depend on them, but it's region specific and will break different things for different users as the leak propagates.

One source: https://twitter.com/atoonk/status/1143143943531454464

90 AS 13335 Cloudflare, Inc. 18 AS 7018 AT&T Services, Inc. 8 AS 63949 Linode, LLC 8 AS 2828 MCI Communications Services, Inc. d/b/a Verizon Business 6 AS 26769 Bandcon 6 AS 16509 Amazon.com, Inc. 4 AS 6428 CDM 4 AS 2914 NTT America, Inc. 2 AS 9808 Guangdong Mobile Communication Co.Ltd. 2 AS 6939 Hurricane Electric LLC 2 AS 62904 Eonix Corporation 2 AS 55081 24 SHELLS 2 AS 54113 Fastly 2 AS 46606 Unified Layer 2 AS 45899 VNPT Corp 2 AS 4246 New Jersey Institute of Technology 2 AS 3257 GTT Communications Inc. 2 AS 27695 EDATEL S.A. E.S.P 2 AS 22781 Strong Technology, LLC. 2 AS 20473 Choopa, LLC 2 AS 16625 Akamai Technologies, Inc. 2 AS 12129 123.Net, Inc.

Post reply on HN