Cloudflare getting all the cool toys. AWS, anyone awake over there?
Cloudflare crawl endpoint
11–20 of 194 posts
Re: Cloudflare crawl endpoint
#12Is cloudflare becoming a mob outfit? Because they are selling scraping countermeasures but are now selling scraping too. And they can pull it off because of their reach over the internet with the free DNS.
[flagged]
Re: Cloudflare crawl endpoint
#13Is cloudflare becoming a mob outfit? Because they are selling scraping countermeasures but are now selling scraping too. And they can pull it off because of their reach over the internet with the free DNS.
For a long time cloudflare has proudly protected DDoS-as-a-service sites (but of course, they claim they don't "host" them)
Re: Cloudflare crawl endpoint
#14Does this bypass their own anti-AI crawl measures? I'll need to test it out, especially with the labyrinth.
If this does bypass their own (and others') anti-AI crawl measures, it'd basically mean that the only people who can't crawl are those without money.
We're creating an internet that is becoming self-reinforcing for those who already have power and harder for anyone else. As crawling becomes difficult and expensive, only those with previously collected datasets get to play. I certainly understand individual sites wanting to limit access, but it seems unlikely that they're limiting access to the big players - and maybe even helping them since others won't be able to compete as well.
Re: Cloudflare crawl endpoint
#15I'm surprised that Cloudflare hasn't started hosting a pre-scraped version of websites that use Cloudflare's proxy - something like https://www.example.com/cdn-cgi/cached-contents.json They already have the website content in their cache, so why not just cut out the middle man of scraping services and API's like this and publish it? Obviously there's good reasons NOT to, but I am surprised they haven't started offeri…
> I'm surprised that Cloudflare hasn't started hosting a pre-scraped version of websites that use Cloudflare's proxy
It's entirely possible that they're doing this under the hood for cases where they can clearly identify the content they have cached is public.
Re: Cloudflare crawl endpoint
#16Does this bypass their own anti-AI crawl measures? I'll need to test it out, especially with the labyrinth.
They say it doesn't: https://developers.cloudflare.com/browser-rendering/faq/#wil...
Further down they also mention that the requests come from CFs ASN and are branded with identifying headers, so third party filters could easily block them too if they're so inclined. Seems reasonable enough.
Re: Cloudflare crawl endpoint
#17Is cloudflare becoming a mob outfit? Because they are selling scraping countermeasures but are now selling scraping too. And they can pull it off because of their reach over the internet with the free DNS.
no? it takes 10 seconds to check:
> The /crawl endpoint respects the directives of robots.txt files, including crawl-delay. All URLs that /crawl is directed not to crawl are listed in the response with "status": "disallowed".
You don't need any scraping countermeasures for crawlers like those.
Re: Cloudflare crawl endpoint
#18Is cloudflare becoming a mob outfit? Because they are selling scraping countermeasures but are now selling scraping too. And they can pull it off because of their reach over the internet with the free DNS.
Their free DNS is only a small piece of the pie.
The fact that 30%+ of the web relies on their caching services, routablility services and DDoS protection services is the main pull.
Their DNS is only really for data collection and to front as "good will"
Re: Cloudflare crawl endpoint
#19This is actually really amazing. Cloudflare is just skating to where the puck is going to be on this one.
Re: Cloudflare crawl endpoint
#20This might be really great! I had the idea after buying https://mirror.forum recently (which I talked in discord and archiveteam irc servers) that I wanted to preserve/mirror forums (especially tech) related [Think TinyCoreLinux] since Archive.org is really really great but I would prefer some other efforts as well within this space. I didn't want to scrape/crawl it myself because I felt like it would feel like yet a…
"I didn't want to scrape/crawl it myself because I felt like it would feel like yet another scraping effort for AI and strain resources of developers"
You feel better paying someone to do the same thimg?