Experiencing around 60% traffic drop on customer's sites.
Route Leak Impacting Cloudflare
61–70 of 164 posts
Re: Route Leak Impacting Cloudflare
#62Earlier quoted context omitted.
At 3-4 major leaks per year it seems like we should probably fix BGP one of these days...
The way I understand it it's not BGP, it's mostly human error, or malicious intent. The protocol is fine.
No system failure is ever "human error". It's faulty system design.
Re: Route Leak Impacting Cloudflare
#63Earlier quoted context omitted.
Thanks for the updates. I wish I could get this information somewhere other than hacker news though. :(
The team is updating the status page but not with granular detail because they'd have to spend time discussing what to say. I'm giving you the blow by blow.
Thanks for the extra detail here
Re: Route Leak Impacting Cloudflare
#64What is a route leak?
The Border Gatway Protocol is what network providers use to announce which IP ranges they can route traffic for. The problem is it's almost totally unauthenticated, so rogue ISPs and network operators can suddenly take over parts of the internet by "leaking" routes for ranges they shouldn't be able to control.
They do this by announcing something like "send me all traffic for 1.1.1.1 - 1.1.1.255", and if their peers don't verify it, they'll just start routing that traffic to them. Peer by peer, the route then propagates and a larger portion of the internet, and as routers learn the new bad route, more of the traffic to those IPs gets sent to incorrect network.
Re: Route Leak Impacting Cloudflare
#65Earlier quoted context omitted.
I guess we are in that 3% then! But 50% of our traffic has gone! Hopefully you are still working on it!
We're definitely still working on it. Sorry you're affected by this. We're talking with the network providers involved. If anyone from the Verizon NOC is online... call me!
I've had FiOS issues in PA since around 6:30AM impacting a decent fraction of sites. YouTube was down for about 5 minutes. Slack still very flaky. A client who hosts a 5-figure number of domains on Cloudflare still unavailable to me, but the pagers didn't start going off so presumably it is isolated. Thanks for all the updates.
Re: Route Leak Impacting Cloudflare
#66“AS396531 "Allegheny Technologies Incorporated" is leaking a better-reachable route for AS13335 "Cloudflare, Inc." towards AS701 "Verizon Business/UUnet" explaining the current LSE going on.” https://twitter.com/OhNoItsFusl/status/1143117619106652160
They aren't the original leaker.
Update: sorry, I may have been wrong. Hard to see clearly in the fog of BGP.
Re: Route Leak Impacting Cloudflare
#67Earlier quoted context omitted.
I guess we are in that 3% then! But 50% of our traffic has gone! Hopefully you are still working on it!
We're definitely still working on it. Sorry you're affected by this. We're talking with the network providers involved. If anyone from the Verizon NOC is online... call me!
Confirming widespread FiOS issues in NYC as well, not limited to CloudFlare IPs.
Re: Route Leak Impacting Cloudflare
#68One of the weirder leaks I've seen, 8.8.8.8 and 1.1.1.1 are both down for me, but everything else is working fine.
8.8.8.8 is google’s DNS, though.
Exactly, that's why it's weird. It would have to be a huge range leaked to get both 8.8.8.8 and 1.1.1.1, surprising that a peer didn't filter it before it worked it's way up the chain.
Re: Route Leak Impacting Cloudflare
#69We've been evaluating Cloudflare mainly for doing failovers faster than DNS. This morning I ran some tests to generate graphs to show the typical delay incurred in preparation for a show-and-tell with some key people.
I started seeing delays of up to 300 seconds! At best there was a 1 second delay. I wondered if I was going to have present "Why we've decided not to go with Cloudflare!"
Any longtime Cloudflare users comment on how rare an event this sort of thing is? It seems rare from eyeballing the recent alert history.
Re: Route Leak Impacting Cloudflare
#70Isn't HN on Cloudflare? How are we reading about a CF outage on a site that runs behind CF?
Doesn't seem so http://www.doesitusecloudflare.com/?url=https%3A%2F%2Fnews.y...
FWIW, this site doesn't seem accurate. I used it against my site that is definitely using cloudflare and it incorrectly reported it as not :p