Route Leak Impacting Cloudflare

#1

Post by xPaw » Mon, Jun 24, 2019, 11:08 AM UTC

Route Leak Impacting Cloudflare
cloudflarestatus.com

Re: Route Leak Impacting Cloudflare

#2

Post by dajonker » Mon, Jun 24, 2019, 11:15 AM UTC

"Cloudflare is observing network performance issues." not just performance issues, our entire website is unavailable because of it.

edit: availability has been alternating between available and unavailable

Re: Route Leak Impacting Cloudflare

#3

Post by markplindsay » Mon, Jun 24, 2019, 11:17 AM UTC

This morning I'm finding out just how many of our supporting services rely on Cloudflare as well.

Re: Route Leak Impacting Cloudflare

#4

Post by sudhirj » Mon, Jun 24, 2019, 11:17 AM UTC

Isn't HN on Cloudflare? How are we reading about a CF outage on a site that runs behind CF?

Re: Route Leak Impacting Cloudflare

#5

Post by fluxsauce » Mon, Jun 24, 2019, 11:20 AM UTC

I was going back and forth about whether to post this; we're clearly experiencing some sort of partial outage, New Relic Synthetics shows everything offline (except for Sydney for whatever reason), but the actual webserver logs indicate healthy traffic and the sites load when I visit it. "Cloudflare is observing" is pretty opaque.

Re: Route Leak Impacting Cloudflare

#6

Post by [deleted] » Mon, Jun 24, 2019, 11:21 AM UTC

markplindsay wrote: post #3

This morning I'm finding out just how many of our supporting services rely on Cloudflare as well.

[deleted]

Re: Route Leak Impacting Cloudflare

#7

Post by monkin » Mon, Jun 24, 2019, 11:22 AM UTC

dajonker wrote: post #2

"Cloudflare is observing network performance issues." not just performance issues, our entire website is unavailable because of it. edit: availability has been alternating between available and unavailable

It's not their fault as accidents happen to everyone. You should be prepared for such a scenario and for many others too.

Re: Route Leak Impacting Cloudflare

#8

Post by tech5000 » Mon, Jun 24, 2019, 11:22 AM UTC

dajonker wrote: post #2

"Cloudflare is observing network performance issues." not just performance issues, our entire website is unavailable because of it. edit: availability has been alternating between available and unavailable

Same here.

Re: Route Leak Impacting Cloudflare

#9

Post by jgrahamc » Mon, Jun 24, 2019, 11:22 AM UTC

This appears to be a routing problem. All our systems are running normally but traffic isn't getting to us for a portion of our domains.

1128 UTC update Looks like we're dealing with a route leak and we're talking directly with the leaker and Level3 at the moment.

1131 UTC update Just to be clear this isn't affecting all our traffic or all our domains or all countries. A portion of traffic isn't hitting Cloudflare. Looks to be about an aggregate 10% drop in traffic to us.

1134 UTC update We are now certain we are dealing with a route leak.

@dang etc.: could someone update the title to reflect the status page "Route Leak Impacting Cloudflare"

1147 UTC update Staring at internal graphs looks like global traffic is now at 97% of expected so impact lessening.

1204 UTC update This leak is wider spread that just Cloudflare.

1208 UTC update Amazon Web Services now reporting external networking problem https://status.aws.amazon.com/

1230 UTC update We are working with networks around the world and are observing network routes for Google and AWS being leaked at well.

1239 UTC update Traffic levels are returning to normal.

Re: Route Leak Impacting Cloudflare

#10

Post by nnx » Mon, Jun 24, 2019, 11:22 AM UTC

Seems to be BGP/routing related, some networks can access CloudFlare networks normally.