Cloudflare crawl endpoint

Post by pqdbr » Wed, Mar 11, 2026, 12:52 PM UTC

weird-eye-issue wrote: post #94

pqdbr wrote: post #65

Off-topic, but I'm having a terrible experience with Cloudflare and would love to know if someone could offer some help. All of a sudden, about 1/3 of all traffic to our website is being routed via EWR (New York) - me included -, even tough all our users and our origin servers are in Brazil. We pay for the Pro plan but support has been of no help: after 20 days of 'debugging' and asking for MTRs and traceroutes, they…

Do you think cloudflare is responsible for all of the network traffic routing in the entire world and can simply fix any problem even if it's on somebody else's network?

Post by lathiat » Wed, Mar 11, 2026, 1:15 PM UTC

kelvinjps10 wrote: post #140

Could they collaborate with the website's creators that have websites behind cloudfare to allow their content to be accessed via an API in exchange of a compensation?. This could be a way to compensate creators and AI companies be able to access content that's unreachable as it's protected by cloudfare

Post by carloslfu » Wed, Mar 11, 2026, 1:24 PM UTC

Post by dewey » Wed, Mar 11, 2026, 1:33 PM UTC

rendaw wrote: post #102

ljm wrote: post #10

Is cloudflare becoming a mob outfit? Because they are selling scraping countermeasures but are now selling scraping too. And they can pull it off because of their reach over the internet with the free DNS.

I think the simple explanation is that they weren't selling scraping countermeasures, they were selling web-based denial of service protection (which may be caused by scrapers).

Post by Macha » Wed, Mar 11, 2026, 2:05 PM UTC

gruez wrote: post #55

Macha wrote: post #53

Earlier quoted context omitted.

So what’s the user agent for their bot? They don’t seem to specify the default in the docs and it looks like it’s user configurable. So yet another opt out bot which you need your web server to match on special behaviour to block

>So yet another opt out bot which you need your web server to match on special behaviour to block Given that malicious bots are allegedly spoofing real user agents, "another user agent you have to add to your list" seems like the least of your problems.