Is cloudflare becoming a mob outfit? Because they are selling scraping countermeasures but are now selling scraping too. And they can pull it off because of their reach over the internet with the free DNS.
[flagged]
Any kind of source for the claim?
Cloudflare crawl endpoint
21–30 of 194 posts
Re: Cloudflare crawl endpoint
#22Re: Cloudflare crawl endpoint
#23Does this bypass their own anti-AI crawl measures? I'll need to test it out, especially with the labyrinth.
If this does bypass their own (and others') anti-AI crawl measures, it'd basically mean that the only people who can't crawl are those without money. We're creating an internet that is becoming self-reinforcing for those who already have power and harder for anyone else. As crawling becomes difficult and expensive, only those with previously collected datasets get to play. I certainly understand individual sites want…
Common Crawl has free egress
Re: Cloudflare crawl endpoint
#24Selling the cure (DDoS protection) and creating the poison (Authorized AI crawling) against their customers.
Re: Cloudflare crawl endpoint
#25Is cloudflare becoming a mob outfit? Because they are selling scraping countermeasures but are now selling scraping too. And they can pull it off because of their reach over the internet with the free DNS.
Cloudflare has been trying to mediate publishers & AI companies. If publishers are behind Cloudflare and Cloudflare's bot detection stops scrapers at the request of publishers, the publishers can allow their data to be scraped (via this end point) for a price. It creates market scarcity. I don't believe the target audience is you and me. Unless you own a very popular blog that AI companies would pay you for.
Re: Cloudflare crawl endpoint
#26Didn't they just throw a (very public) fit over Perplexity doing the exact same thing?
Re: Cloudflare crawl endpoint
#27I'm surprised that Cloudflare hasn't started hosting a pre-scraped version of websites that use Cloudflare's proxy - something like https://www.example.com/cdn-cgi/cached-contents.json They already have the website content in their cache, so why not just cut out the middle man of scraping services and API's like this and publish it? Obviously there's good reasons NOT to, but I am surprised they haven't started offeri…
It’s a bit more complicated than that. This is their product Browser Rendering, which runs a real browser that loads the page and executes JavaScript. It’s a bit more involved than a simple curl scraping.
Re: Cloudflare crawl endpoint
#28Will this crawler be run behind or infront of their bot blocker logic?
Re: Cloudflare crawl endpoint
#29[deleted]
Re: Cloudflare crawl endpoint
#30All what was expected, first they do a huge campaign to out evil scrapers. We should use their service to ensure your website block LLMs and bots to come scraping them. Look how bad it is.
And once that is well setup, and they have their walled garden, then they can present their own API to scrape websites. All well done to be used by your LLM. But as you know, they are the gate keeper so that the Mafia boss decide what will be the "intermediary" fee that is proper for itself to let you do what you were doing without intermediary before.