Does anyone know which global sites were or still are unavailable because of Cloudflare crash? Maybe some media sites?
Discord for one
Route Leak Impacting Cloudflare
81–90 of 164 posts
Re: Route Leak Impacting Cloudflare
#82We've been evaluating Cloudflare mainly for doing failovers faster than DNS. This morning I ran some tests to generate graphs to show the typical delay incurred in preparation for a show-and-tell with some key people. I started seeing delays of up to 300 seconds! At best there was a 1 second delay. I wondered if I was going to have present "Why we've decided not to go with Cloudflare!" Any longtime Cloudflare users c…
Things like this are not unique to CF and actually originate from outside their network. It does happen every once in a while, but I have far more confidence in CF's ability to resolve it than my own. They have the clout in the industry, the connections and the expertise to deal with this kind of thing. I've been with CF since late 2011 and am quite satisfied with their services.
That's a good point, and I must admit I didn't know what a route leak was or that it could inflict this kind of damage. I appreciate now it's not CloudFlare's fault, and my hat is off to the CTO for posting more detail here.
On the plus side, I did get to test the "Pause CloudFlare" button in a real-world scenario!
Re: Route Leak Impacting Cloudflare
#83Isn't HN on Cloudflare? How are we reading about a CF outage on a site that runs behind CF?
The most surefire way to know if a site is behind Cloudflare (orange cloud is on) is by hitting /cdn-cgi/trace (e.g. https://news.ycombinator.com/cdn-cgi/trace) which is the debug output from Cloudflare’s HTTP server. There’s no way to my knowledge that route can be disabled or overriden.
Anyway, no, HN is not on Cloudflare, at least at the moment.
Re: Route Leak Impacting Cloudflare
#84Hi Shachar from Peer5 here, we're operating a MultiCDN. Cloudflare is actually one of the best performing CDNs. All CDNs encounter issues small to big - that's why using multiple providers and intelligently routing between them is critical for high resilience.
Right now we're seeing issues in the following ASNs:
9,541 .
59,257 .
38,264 .
132,165 .
23,888 .
55,714 .
45,773 .
45,669 .
9,260 .
58,895 .
17,557 .
38,547 .
38,193 .
135,407 .
23,966 .
7,590 .
136,525 .
Re: Route Leak Impacting Cloudflare
#85This appears to be a routing problem. All our systems are running normally but traffic isn't getting to us for a portion of our domains. 1128 UTC update Looks like we're dealing with a route leak and we're talking directly with the leaker and Level3 at the moment. 1131 UTC update Just to be clear this isn't affecting all our traffic or all our domains or all countries. A portion of traffic isn't hitting Cloudflare. L…
Are you depending on the leaker to fix the issue on their side? What happens in case of non-cooperative or non-responsive leaker?
Re: Route Leak Impacting Cloudflare
#86Earlier quoted context omitted.
I guess we are in that 3% then! But 50% of our traffic has gone! Hopefully you are still working on it!
We're definitely still working on it. Sorry you're affected by this. We're talking with the network providers involved. If anyone from the Verizon NOC is online... call me!
Having connectivity issues with Verizon FIOS in Massachusetts this morning as well
Re: Route Leak Impacting Cloudflare
#87This appears to be a routing problem. All our systems are running normally but traffic isn't getting to us for a portion of our domains. 1128 UTC update Looks like we're dealing with a route leak and we're talking directly with the leaker and Level3 at the moment. 1131 UTC update Just to be clear this isn't affecting all our traffic or all our domains or all countries. A portion of traffic isn't hitting Cloudflare. L…
Are you depending on the leaker to fix the issue on their side? What happens in case of non-cooperative or non-responsive leaker?
It's a chain. You first contact the leaker and their upstream, and then if that doesn't work then their upstream, etc.
At some point you reach a company that's large enough that they must cooperate because they want to remain in business of being an actual responsible ISP.
And then there's Verizon, who can safely ignore any ISP etiquette because they have a de-facto monopoly.
Re: Route Leak Impacting Cloudflare
#88This appears to be a routing problem. All our systems are running normally but traffic isn't getting to us for a portion of our domains. 1128 UTC update Looks like we're dealing with a route leak and we're talking directly with the leaker and Level3 at the moment. 1131 UTC update Just to be clear this isn't affecting all our traffic or all our domains or all countries. A portion of traffic isn't hitting Cloudflare. L…
Are you depending on the leaker to fix the issue on their side? What happens in case of non-cooperative or non-responsive leaker?
The upstream provider, if cooperative, could filter out their announcement as a quick fix. It's surprising it happened though, most upstreams put filters in place already.
Re: Route Leak Impacting Cloudflare
#89This appears to be a routing problem. All our systems are running normally but traffic isn't getting to us for a portion of our domains. 1128 UTC update Looks like we're dealing with a route leak and we're talking directly with the leaker and Level3 at the moment. 1131 UTC update Just to be clear this isn't affecting all our traffic or all our domains or all countries. A portion of traffic isn't hitting Cloudflare. L…
Are you depending on the leaker to fix the issue on their side? What happens in case of non-cooperative or non-responsive leaker?
It's sort of a network of trust thing, every time this happens everyone has to scramble to add route filters to ignore the leaked route on all their routers, and then they try to contact the leaker in parallel get them to fix it as well (and their upstream routers).